Google Lens getting big MUM upgrade that lets you take a photo and ask questions about it

At I/O in May, Google announced Multitask Unified Model (MUM) as a big improvement to Search. MUM is now set to come to Google Lens “early next year.”

After taking a picture in Lens or analyzing an existing image/screenshot, swiping up will continue to show “Visual matches.” There will soon be a new “Add questions” button at the very top of the sheet that opens a very slick UI. These “Related searches” or any entered query let you refine the search.

Google showed off two demos at Search On 2021 today. Both are intended to let you combine visual and text input.

The most impressive example involves taking a picture of a broken bike part and asking, “how do you fix this.” Google will correctly identify what exactly needs to be fixed and how to do so through videos, forum results, and other help articles on the web.

By combining images and text into a single query, we’re making it easier to search visually and express your questions in more natural ways.

Another example involves capturing an image of a style — e.g. on a shirt — and asking for other pieces of clothing (“socks with this pattern”) that share the same design.

MUM in Google Lens will be available in the coming months. The company is actively testing and conducting “rigorous testing and evaluation” of the AI model before launching.

Other Search On 2021 announcements today include:

• Google Lens coming to Chrome and will soon be integrated into the Google app browser on iOS
• Maps adds ‘Fire’ as a top-level layer, bringing ‘Tree Canopy’ tool to 100 cities
• Search launches redesign with more images, suggestions, and ‘Things to know’
• Google’s new ‘Address Maker’ app will let communities create functional addresses quickly
• ‘About this result’ panel will explain why each link appeared in your search

More about Google Lens:

• Lens gets a touch of Material You, Search widget & Pixel Launcher shortcut [U]
• Google’s first ads on TikTok heavily promote Google Lens [Video]
• Google Lens coming to desktop Chrome as new integrated image search tool
• Lens ‘app’ redesign emphasizes screenshot analysis over live viewfinder