Microsoft supercharges Copilot with Google integration, smarter vision

Copilot Vision’s eyesight is improving, as the integrated Windows AI technology will soon be able to see entire documents, plus link to apps like Google Drive via a new connectors function.

Separately, Microsoft is adding Copilot to the Windows 11 taskbar and making “Hey Copilot” a wake word for the Windows AI app. It’s part of the company’s effort to expand its presence across your PC. The biggest update, though, is Copilot Actions, which will let the assistant actually modify files and system settings.

Copilot Vision is the “visual” aspect of Copilot, able to “see” your PC’s screen and guide you through troubleshooting steps. In my hands-on testing, I found it more than capable of assisting with certain tasks, though it struggled consistently to recognize on-screen symbols in simple games like Solitaire. Microsoft plans to improve it in one key area: right now, it can only see what you see. Soon, in Office apps (Word, PowerPoint, and Excel), it will be able to view the entire document, even sections not currently visible on your screen.

More from Microsoft on the big changes coming to Copilot:

Copilot Vision is also rolling out to all markets where Copilot is available, Microsoft said. You won’t need a specialized PC either; if you have Copilot, you’ll have Vision capabilities.

Microsoft’s basic Copilot functions are also being beefed up, widening their scope as well.

To date, Copilot has been a tool to surf the web. Microsoft sees it as a source of knowledge, a way to browse, pull in contextual information, and eventually serve as a platform for agents that can perform tasks like shopping.

Originally, Microsoft indicated that Copilot would be able to adjust Windows settings, such as turning your PC to dark mode. The company later backed off that plan but added a feature called Direct Settings Access, which lets users ask Copilot to guide them to the proper Windows setting.

Copilot will soon be able to connect to other services as well, including OneDrive files and Outlook contacts, emails, and calendar events. You can also link it to Google services like Google Drive, Gmail, Google Calendar, and Google Contacts. In addition, you’ll be able to take a Copilot response and automatically drop it into a formatted document, spreadsheet, or presentation.

Microsoft tipped a version of this feature on October 10th when it announced the feature for Windows Insiders.

The connectors allow for more useful, broader queries, including hunting down a particular paper or simply asking Copilot when your dentist appointment is. Of course, you’ll have the choice to turn on these connections yourself or else leave Copilot alone.

If Statcounter’s metrics are right, ChatGPT’s usage far surpasses Copilot’s, with its 78 percent share outweighing the latter’s 10 percent share by about eight to one. Still, Windows and Copilot are among the first things most people see each day, and the company is banking on making it more useful and prominently integrated.