How to assess your AI projects’ ROI as recession hits

As companies scramble to protect themselves against the economic downturn, all sorts of projects are being impacted. And applied artificial intelligence (AI) is no exception.

Before the downturn, the AI industry was enjoying a gold rush, with companies pouring plenty of cash into machine learning (ML) talent, research and projects. While these efforts have borne fruit and can be seen in applications we use every day, much of this investment was prompted by unjustified hype surrounding AI. 

As organizations adjust their AI initiatives to the new market conditions, here’s what to expect.

Measuring ROI for AI projects

“Even before the downturn, we have been talking about ROI in AI projects,” Anand Rao, global artificial intelligence lead at PwC, told VentureBeat. “While ROI is a concern for the adoption of any technology, the difference with AI, as opposed to other technologies such as cloud, is that you’re talking about prediction.”

How do you measure the value of prediction? Most companies use supervised machine learning models, which means they train their models on examples that are labeled by human experts. The model’s accuracy percentage is then measured by comparing its predictions against the ground truth specified by human annotators. However, not all accuracy measures are made equal.

“In many organizations, it is not the best person who is doing the labeling,” Rao said. For example, when a financial institution is creating an ML model for underwriting decisions, having the best underwriters label the training examples will result in a better outcome than having an intern do the labeling during their spare time. This is critical because a model with (say) 95% accuracy is more valuable than one with a lower accuracy percentage. 

“There is also a complication where you don’t measure human performance as rigorously as you measure AI performance,” Rao said. “You really don’t know whether all your underwriters match the accuracy of your AI system. If they don’t, your AI is far better than you originally assumed.”

And finally, organizations must also account for the cost of wrong predictions, Rao said, which depends on the application, environment, customers and many other factors.

“The challenge of measuring the ROI of AI/ML algorithms has existed all the time,” Rao said. “We need more rigorous measures. Now, with the downturn, it becomes even more critical that we have a good sense of ROI on ML/AI algorithms.”

With a clearer picture of the profitability of their AI projects, organizations will be in a better position to decide whether to continue or stop them.

The AI portfolio approach

AI will remain important for maintaining a competitive edge in many industries, even during the recession. But companies need to adjust their AI strategies to economic conditions. And this can start with a change in how a company perceives AI projects.

“Executives like to look at every project and ask, what is the ROI for this recommendation engine or this NLP technology?” Rao said. “Measuring ROI at that level, project by project, is not the right approach. You’re going to say, ‘This project didn’t have any ROI so let’s stop doing any of that kind of work in the future.’”

Rao recommends what he calls a “portfolio approach.” Instead of measuring the success of AI projects on a project-by-project basis, companies should look at their AI initiative as a portfolio that includes a variety of AI projects. 

Some projects will be based on ML models that have been tested by competitors and proven to work. These are the low-hanging fruits of applied AI. They have a high chance of success and are easy to adopt. Rao calls them “ROI-generating AI projects.”

Other projects will focus on experimenting with state-of-the-art AI, such as large language models, exploring new technology, and keeping your data scientists motivated to push the boundaries. These kinds of projects have a lower chance of success but can have higher returns if they succeed. 

“You need a portfolio in which some projects are new, some are just maintenance types, some are things that others have done,” Rao said. “You run many experiments, and maybe out of 10, three will succeed. And those will give far more return than the entire 10 put together.”

Executives must also be mindful of the risk/return tradeoffs of their AI projects. This means that instead of selecting models based on accuracy, AI managers should look at a broad range of characteristics, including fairness, explainability, robustness and safety. For example, facial recognition technology comes with privacy and ethical risks, which need to be weighed against the technology’s benefits.

“I think the portfolio approach will start to take hold, especially in the downturn where people are enquiring about the value of AI,” Rao said. “We’re almost maturing from a ‘cool’ technology with hype to meeting the reality and becoming more entrenched with traditional technology and getting the rigor needed to become widely adopted.”

The tech talent bubble

The past few years have seen a great inflow of data scientists and machine learning engineers into various sectors. The growing demand for AI talent has created a bubble where tech companies are offering huge salaries. As companies grapple with the recession, there will be an adjustment.

“People were paid a lot for their AI talent, not only from outside but also from within the tech industry. They went from one company to another and back, and constantly change with bigger offers. Salaries and compensations were constantly increasing,” Rao said. “In the past year, there has been tremendous pressure on the tech industry. In addition to shedding jobs, there is a tech talent freeze. We’re seeing the tech talent bubble burst.”

With the slowdown in the economy, many organizations are beginning to question whether they are getting the return they want on the huge investments they are making in acquiring and keeping AI talent. Are they getting a measurable revenue increase? Would their revenue be cut by half if they retained just half of their AI/ML engineers?

“The question being asked is: What exactly is the value they are adding?” Rao said. “There is an intense focus on ROI as well as the productivity of AI/ML folks with respect to their salaries.”

As senior executives start asking questions about AI/ML engineering productivity, there will be a slowdown in hiring, Rao believes. At the same time, companies will need to go back to the drawing board and figure out ways to measure the ROI of their AI projects and determine how much of their revenue is the result of AI/ML.

The bright side of the bursting tech talent bubble is that the AI talent pool will become much more accessible to other industries.

“Previously, being a product manager at a big tech company would be a dream job for someone with a CS or MBA background. Now they’re looking beyond tech companies because there is not much intake from tech companies,” Rao said. “The brain drain from other sectors to tech is reversing. In some sense, it’s good to have that correction. We were in an inflationary bubble previously. Now it’s becoming a more rational model of compensation across the board.”