AI and computer vision powers growing shop-and-go platform

AI and computer vision were not necessarily top-of-mind for Sodexo, a food and facilities management company that runs over 400 university dining programs, which was looking for a future-forward, seamless experience to offer students instead of the usual buffet meal options.

All the company knew is that they wanted something like Amazon Go’s cashierless, shop-and-go stores. That is, where shoppers can walk in, pick items off the shelves, and leave without standing in line at the cashier or suffering through swiping codes at the self-checkout. 

“Students today want things they can partially or fully prepare in their room or apartment, with organic, highly-local options,” said Kevin Rettle, global vice president product development and digital innovation at Sodexo. “We also wanted to remove friction, but many solutions still require the interaction of the guest with a cashier – this generation really doesn’t want to talk to a lot of people in their service interactions.” 

For the University of Denver, Sodexo chose the San Jose-based AiFi, which offers a frictionless and cashierless AI-powered retail solution. Its flexibility (the company says it can deploy two stores per week) and diverse locations (sports stadiums, music festivals, grocery store chains, college campuses and more) make it unique, explained Steve Gu, who cofounded AiFi in 2016 with his wife, Ying Zheng. Both Gu and Zheng have Ph.D.s in computer vision and spent time at Apple and Google.

AiFi, which is powered only by cameras and computer vision technology, announced today that it now boasts a total of 80 checkout-free stores worldwide, partnering with retailers including Carrefour, Aldi, Loop and Verizon. It has also opened 53 Zabka stores in Poland and 2 NFL stores. Gu maintains this is an industry benchmark for how this technology can scale in a way that Amazon Go, which has more than 42 stores, cannot.

Cameras and computer vision, not sensors

Amazon Go’s stores are retrofitted with specialized cameras, sensors, and weighted shelves, Gu explained. “That makes the solution very expensive and hard to scale,” he said. Instead, AiFi uses the “cheapest-possible off-the-shelf cameras,” combined with what he says is the real power: Computer vision. 

AiFi deploys sophisticated AI models through a large number of cameras placed across the ceiling, Gu said, in order to understand everything happening in the shop. Cameras track customers throughout their shopping journey, while computer vision recognizes products and detects different activities, including putting items onto or grabbing items off the shelves.

Beneath the platform’s hood are neural network models specifically developed for people-tracking as well as activity and product recognition. AiFi also developed advanced calibration algorithms that allow the company to re-create the shopping environment in 3D.

AiFi also leverages simulated datasets. “We spend quite a lot of effort building those simulated environments so we can train the AI algorithms and the models inside them,” Gu said. “That really helps us develop those models faster and make them more scalable.” 

In a simulated world, he explained, you can easily adjust human shapes and characteristics, as well as the shelf layout and the look of the product. You can create a cluttered, crowded store environment or one that is neat and orderly. “Things that cannot be done in the real world can be easily done in a simulated world,” he said. “The AI can learn about those scenarios and will then be able to perform or outperform in a real setting.” 

Computer vision that is constantly evolving

AiFi’s system is evolving and will improve over time, Gu continued, citing current challenges including the ability for the platform to recognize small items such as gum or lipstick.

“If they are not placed in the right place, it’s very hard for the computer vision to discern what it is,” he said. There are also issues related to items with similar looks and textures. “If they are placed together in adjacent spaces it sometimes causes confusion for the cameras and computer vision to recognize these products,” he said. “But the good thing is that it’s not purely based on the visual texture – you also have the 3D scene geometry, the location, the context as well.”  

There also are current limitations to the size of the store and the number of people it can track. “The question is can the solution also be scalable to super centers of 100,000 square feet?” he said. “Also, the system is able to track hundreds of people shopping simultaneously in a shop environment. But in order for that to further scale, to track thousands of people, with very complex shopping behavior, that’s something that is still a work in progress.”

To enter an AiFi-powered store, shoppers don’t need a biometric scan or an AiFi app — they can swipe a credit card or use the retailer’s app. At the University of Denver, for example, Sodexo wanted a partner that was agnostic to the front end. “We were able to use our wallet and payment processing, and tie the AiFi technology, the cameras, and the AI into our system,” said Rettle.

Consumer adoption is key

“From a product ownership perspective, you always kind of hold your breath. Is it going to work?” he said. But ultimately, at the University of Denver the students immediately took to the AiFi concept.

“We didn’t have to teach any of the students what to do,” he said. “They get it without having a bunch of prompts.”

Critics in the retail space also predicted the AiFi technology would be a “loss-prevention nightmare — that the students will figure out how to game the system,” Rettle said. Instead, the current accuracy rate for the AiFi solution is 98.3% and the shrink rate (what shoppers walk out without paying for) has actually declined, he said.

Some products don’t quite work yet with AiFi’s solution, Rettle admits, including college and fan “swag.” “The platform still has to understand consumer behavior around that, which will certainly evolve with the technology,” he said.

Rettle also said he doesn’t envision a campus or stadium that could shift to 100% autonomous retail. “For us it’s something that complements,” he said. “But I see a strong future in terms of being able to continue to deploy and drive ubiquity with the solution based on consumer acceptance.”

For Gu, AiFi’s potential is “huge,” with over a dozen new stores in the works and a growing partnership with Microsoft as an independent software vendor partner (AiFi runs its solution on Azure). “You’re going to see a lot of autonomous retail in a variety of verticals — not just stadiums, festivals and universities, but offices, movie theaters and other spaces,” he said.