Bringing multimodal search to AI Mode
Bullet points
- Google's AI Mode is getting a multimodal search upgrade, powered by Lens and Gemini.
- AI Mode now understands images, allowing you to ask questions about what you see.
- Snap a photo or upload an image and get a comprehensive response with links to learn more.
- AI Mode uses Gemini to understand the entire scene, including objects, materials, colors, and shapes.
- Try out this new feature in the Google app (Android and iOS) and share your feedback.
Since launching AI Mode to Google One AI Premium subscribers, we’ve heard incredibly positive feedback from early users about its clean design, fast response time and ability to understand complex and nuanced questions. On average, AI Mode queries are twice as long as traditional Search queries on Google. People are using AI Mode for help with exploratory and open-ended questions, along with more complicated tasks — like comparing two products, exploring how-tos and planning a trip.
With this positive feedback, we’ve now started to make AI Mode available to millions more Labs users in the U.S., and you can sign up in Labs now to try it. We’re also continuing to improve the experience and today we’re bringing the powerful multimodal capabilities in Lens to AI Mode.
Search what you see in AI Mode
With AI Mode’s new multimodal understanding, you can snap a photo or upload an image, ask a question about it and get a rich, comprehensive response with links to dive deeper. This experience brings together powerful visual search capabilities in Lens with a custom version of Gemini, so you can easily ask complex questions about what you see.
AI Mode builds on our years of work on visual search and takes it a step further. With Gemini’s multimodal capabilities, AI Mode can understand the entire scene in an image, including the context of how objects relate to one another and their unique materials, colors, shapes and arrangements. Drawing on our deep visual search expertise, Lens precisely identifies each object in the image. Using our query fan-out technique, AI Mode then issues multiple queries about the image as a whole and the objects within the image, accessing more breadth and depth of information than a traditional search on Google. The result is a response that’s incredibly nuanced and contextually relevant, so you take the next step.
In this example, AI Mode grasps the subtleties in the image, precisely identifies each book on the shelf and issues queries to learn about the books and similar recommendations that are highly rated. It provides a list of recommended books with links to learn more and purchase, and allows you to ask follow up questions to narrow down your search.
We’re continuing to test and improve AI Mode with feedback from people in Labs. Sign up in Labs now to try out this new capability in AI Mode in the Google app (Android and iOS) and let us know what you think.