How Pinterest built a more representative shopping experience with AI

Artificial intelligence and augmented reality can work hand in hand to support innovations that can change the shopping and advertising game.

Pinterest combined artificial intelligence with augmented reality and incorporated diversity and inclusion in a way that helped the company’s bottom line, Jeremy King, head of engineering at Pinterest, said in a conversation with Matt Marshall, founder and CEO of VentureBeat, at the Transform 2021 virtual conference.

With more than 80 million people searching for beauty items every month, Pinterest had to innovate how people could see themselves in the products they were interested in. The company created the Try On tool, which lets users use their phones to see how they would look with different types of makeup. Offering a palette of skin tone ranges increased engagement, which increased searches, which led to platform growth.

“AR Try On is our augmented reality feature that allows you to use the lens camera on your phone to try on lipsticks and eyeshadows and filter makeup,” King said. “You pick up your phone, and essentially point it right at your face and you can do a search for something like red lipstick or makeup ideas and you’ll see not only a palette of skin tone ranges, but also a button to try on the shades.”

Above: Pinterest’s AR Try On is an augmented reality tool that allows customers to try on makeup.

Image Credit: Volley

Pinterest worked to prevent bias from being introduced into the data by diversifying the data sources and testing computer vision and machine learning algorithms on all kinds of skin tone ranges and makeup colors. Pinterest is able to serve diverse recommendations across the board, and the tool is able to understand and process skin tone ranges in complex lighting scenarios.

“Building for diversity and representation isn’t just the right thing to do. For us, it’s really good for our business,” King said.

As for the payoff? When using the skin tone range feature with AR, customers were five times more likely to show purchase intent.

Connecting customers to products

Pinterest has been working to improve customer and vendor experiences in other ways as well, turning its values of diversity and inclusion into action.

“We frankly heard from pinners that we weren’t naturally relevant or diverse in our recommendations, and they had to add more descriptors to their searches to filter it,” King said. “We took that as an action item to create these capabilities not only to allow more diverse content to show up, but to integrate new features like AR Try on. And once we had that, then we really got the engagement that we’re looking for.”

The company has also worked with a diverse set of creators to produce content while also working with retail partners to ingest hundreds of thousands of products from around the world in Pinterest’s catalog.

“Creators are coming in creating unique content,” King said. “We want to be able to not only use creators, but also our computer vision technology, to identify every single picture and every one of those billions of times so that you can make a really great experience.”