Not content to play second fiddle to its investment, partner, and apparent competitor OpenAI which is holding its second “DevDay” today, Microsoft announced its own series of major AI updates to its Copilot chatbot, Windows 11 operating system, and Bing search engine.
With few exceptions, the updates all leverage AI in an effort to create a more personalized and powerful experience for users, from AI generated search results to AI conversational voice interfaces and AI-powered suggestions for actions to take next.
As Microsoft rolls out these updates, the company says privacy and responsible AI remain central to the company’s vision — following the rapping it took from security experts and users earlier this year after its constantly screenshotting Windows 11 Recall feature was poorly received.
Here’s a rundown of what Microsoft announced today:
Copilot can now talk to you and watch your screen
Microsoft’s Copilot, initially launched as Bing Chat back in February 2023 before being rebranded in November 2023, has been updated with a conversational voice mode and the ability to watch a user’s activity alongside their screen — with their permission and opt-in consent, of course, launched from the Microsoft Edge web browser and a new program called Copilot Labs (reminiscent of Google’s Search Labs and Gmail Labs products), but which is only available to Copilot Pro subscribers ($20 per month).
As Microsoft revealed in a blog post today: “Before releasing our most advanced tools to all users, we are trialing them for a small subset to gather feedback, learn, and then applying these lessons back into the product – making them at once, we hope, better and safer. Available to Copilot Pro users, think of it as offering a glimpse into “work-in-progress” projects just around the corner.”
Mustafa Suleyman, the controversial former Google DeepMinder and current Executive Vice President and CEO of Microsoft AI, posted on X: “I truly believe we can deliver a calmer, more helpful and supportive era of technology, with a Copilot that is now more intuitive, more personalized, and secure. Learn more, download, and enjoy. At Microsoft AI, we are creating an AI companion for everyone. This is the first step.”
Newly introduced features include:
• Copilot Voice, which allows users to communicate with the AI via natural speech, offering four different voice options. It is rolling out in English-speaking countries such as Australia, Canada, New Zealand, the UK, and the US, with more regions and languages to follow.
For the voice mode, a Microsoft spokesperson told VentureBeat via email that the underlying technology is fine-tuned versions of OpenAI models: “Copilot Voice is built upon the latest models from OpenA, but fine-tuned by Microsoft. Copilot Voice responds differently due to this fine tuning. It has 4 unique voices with additional voices coming soon. Copilot Voice will be available for both Pro and free users, with longer time limits on Pro accounts.”
• Copilot Daily, a feature designed to provide users with a digestible overview of their day, including news and weather summaries, tailored to their interests. Information is pulled from authorized content providers including Reuters and The Financial Times, offering a curated, simplified start to the day.
• Copilot Vision, an experimental feature available to select Copilot Pro users, brings a visual understanding capability. Users can interact with images or web pages, and Copilot will interpret what’s on the screen to assist in making decisions, such as comparing product options or suggesting next steps.
Microsoft underscores that none of the data from these sessions is stored or used for training, emphasizing privacy and security.
Also available through Copilot Labs is Think Deeper, a feature built to handle more complex questions, providing detailed, step-by-step answers to challenging queries. The goal is to offer deeper insights into topics such as life decisions, financial comparisons, or practical dilemmas like deciding on a new car or place to live. It seems reminiscent of OpenAI’s new o1 models which are also designed to reason and perform “chain-of-thought” before responding.
Copilot also integrates creative tools such as Visual Search, enabling users to upload images for the AI to analyze, whether identifying dog breeds or offering recipe ideas based on a dish’s photo.
The AI even helps refine writing, providing suggestions for tone and style, and encouraging creativity with tools to generate stories, poems, and images.
As for what models are powering the new Copilot experience under the hood, a Microsoft spokesperson told VentureBeat via email: “We continue to use the latest models from OpenAI. Additionally, we are also beginning to apply our own in-house technology to the experience. This combination gives us more flexibility to apply the best model to each customer’s interaction with Copilot.”
Windows has been updated with new AI-powered suggestions and AI-powered search
With the new updates, Windows 11 continues to evolve into a more AI-powered operating system, particularly for users with Copilot+ PCs. These AI-enabled features are designed to increase productivity while maintaining a streamlined, easy-to-use interface.
• Recall allows users to instantly retrieve previously viewed content on their PC. This feature is opt-in and secured by Windows Hello, ensuring user privacy is protected. Sensitive information, such as credit card details, is kept confidential with additional security filters.
• Click to Do helps users quickly perform actions by overlaying interactive suggestions on the screen. Whether rewriting text, editing images, or launching web searches, this tool is aimed at speeding up workflows without the need for manual navigation. Accessible with just a click, Click to Do is poised to make multitasking on a Copilot+ PC even more intuitive.
• Improved Search in Windows 11 takes advantage of AI to offer more intuitive file search. Users can now describe the content they’re looking for in natural language, like “BBQ party photos,” without needing to remember specific file names or locations. This feature extends to OneDrive searches, providing a powerful way to find photos and documents with ease.
The AI models powering the improved search are apparently all in-house models, according to a Microsoft spokesperson, who told VentureBeat via email that: “Improved Windows search models are downloaded to your Copilot+ PC and leverage the NPU to power the improved Windows search experience across settings, File Explorer, and Windows Search Box.”
These enhancements also extend to creative applications. Super resolution in Photos allows users to upscale low-resolution images, while generative fill and erase in Paint offers precision in editing photos, such as removing unwanted objects. These features leverage the onboard AI to provide quick, high-quality results.
Bing gets AI-powered search summaries and deep search
Microsoft’s AI-powered Bing continues to evolve with the rollout of Bing Generative Search. This feature moves beyond simple keyword-based queries, offering detailed, context-rich answers for more complex questions.
Launched in beta in the U.S., Bing Generative Search allows users to try out its capabilities by typing “Bing generative search” into the search bar and selecting from a carousel of demo queries displayed as image “cards” with captions over them.
Unlike traditional search engines that deliver a list of results, Bing Generative Search aims to create a seamless experience by synthesizing information from multiple sources and presenting it in a cohesive, easily digestible format.
This is especially helpful for queries like “how to effectively run a one-on-one meeting” or “how to remove background noise from a podcast.” The AI analyzes millions of data points and surfaces not just the answer, but relevant insights and deeper explanations.
Bing will generate its own generated summary of information from multiple sources — similar to Google’s controversial “AI Overviews” feature. However, Microsoft’s look a bit more interactive containing pop-overs with source links and text, as well as an interactive table of contents that users can click to rapidly view different sections of a summary.
One interesting new feature is a Deep Search button, which users can click to get a more profound and thorough explanation of a topic. The experience is designed to handle informational and complex queries, such as solving detailed problems or exploring unfamiliar subjects. The platform’s goal is to increase traffic to content creators and publishers by maintaining a balanced web ecosystem. Microsoft cautions it’s still in beta.
As for the models under the hood, Jordi Ribas, Microsoft’s Corporate Vice President and Head of Search, told me when I asked on X that Bing is relying on a “combination of LLM and SLMs” or small language models (which I take to mean the Phi series) “including advanced GPT models” (which I take to be OpenAI’s GPT-4o series unveiled back in May and updated several times since then).
Trying to take privacy seriously after Windows Recall backlash
As part of its ongoing efforts to prioritize responsible AI, Microsoft has integrated privacy and safety measures across all its AI tools. According to the Copilot FAQ, users have control over their data, with options to opt out of certain data collection and AI training features. Particularly in regions like the European Economic Area (EEA) and the UK, Microsoft is fully compliant with local privacy laws.
Copilot is also built to foster responsible use. The AI provides citations for the sources it references, allowing users to fact-check information. Microsoft has incorporated feedback loops into the system, with users encouraged to report inaccurate or inappropriate content through built-in reporting tools.
Microsoft’s AI emphasis on responsible AI extends to its collaboration with OpenAI and its commitment to safeguarding users from potential misuse. Features including content filtering and abuse detection are designed to prevent harmful interactions, ensuring that Copilot and Bing remain safe and constructive tools for users.
The emphasis on privacy and safety comes after Microsoft faced a vocal user backlash when it announced its Windows 11 Recall feature for Copilot+ PCs back in May 2024, which takes a series of screenshots stored on a user’s device and which is meant to allow the PC to return to a previous state when asked by the user. However, security researchers and engineers pointed out the feature behaved a lot like spyware, especially since Microsoft initially debuted it as a stock feature installed and turned on by default, which the user had to jump through multiple hoops to disable. It’s since been made opt-in only and did not ship with some initial Copilot+ PCs from Microsoft’s original equipment manufacturer (OEM) partners.
With its latest updates to Copilot, Windows 11, and Bing, Microsoft continues to position itself at the forefront of AI innovation.
Author: Carl Franzen
Source: Venturebeat
Reviewed By: Editorial Team
