Arize AI goes self-service with ML observability platform

Arize AI, which makes a machine learning (ML) observability and monitoring platform, today introduced a new version of the software at its Arize:Observe Summit 2022. 

The Berkeley, California-based startup claims to be the industry’s only full-stack ML observability and model performance monitoring platform built specifically to solve troubleshooting bottlenecks and pain points experienced by thousands of ML engineers, data scientists and other practitioners responsible for deploying and maintaining ML models. 

Fundamentally, Arize AI makes older black box-type systems transparent in order to ensure that more effective and responsible AI models move smoothly from research to production, Chief Product Officer and co-founder Aparna Dhinakaran told VentureBeat.

Automated model monitoring and analytics platforms help ML teams detect issues when they emerge, troubleshoot why they happened and improve overall model performance, Dhinakaran said. By connecting offline training and validation datasets to online production data in a central inference store, ML teams can speed up model validation, data quality checks and model performance management, she said.

Keeping ML models on track and performing optimally

Arize AI effectively serves as a guardrail on deployed AI applications, Dhinakaran said.

“In machine learning, you can test a model (with controlled data) and it can work fine when you build it,” Dhinakaran told VentureBeat. “But then when you put it into the real world, you can’t control the data. And that’s actually a pretty big difference. For example, maybe when you built your (bank loan-application) model, you built it on data from a few segments of the population; historically, white men have had access to that kind of loan and have been approved more often. Now you see the changing demographics – your model is seeing the data that it was not trained on. So when it (the model) starts to behave poorly on that type of data, you have no way of knowing about it, you don’t even know it’s doing anything negative. 

“So what we’re trying to do is put that guardrail in place as that line between the model and the impact it has on the real world and to make sure that we can catch those types of problems. Then we give ML engineers and practitioners the tools to go and fix it.”

Dhinakaran pointed out another use case in which a merchant kept coming up with a high number of fraud cases – far more than normal. Arize AI was able to pinpoint the cause of the fraud incidents (it turned out to be the merchant himself), and the culprit was removed from the marketplace in which he was doing business.

What’s in the new ML release

With this new release, 2-year-old Arize claimed a milestone in its evolution, becoming the first ML observability company to offer a full complement of self-serve signup options for every customer – including a free offering.  

Included in the release are improvements to platform features used often by ML engineers, according to Dhinakaran. 

• Monitors and identifies drift: Drift is a term used by ML practitioners to identify when data changes significantly from how the model was built. Arize pinpoints drift across model dimensions and values by tracking for prediction, data, and concept drift across model dimensions and values, and comparing across training, validation and production environments.
• Ensures data integrity: Guarantee the quality of model data inputs and outputs with automated checks for missing, unexpected, or extreme values.
• Responds to model performance problems: Unmask the exact features and dimensions that are pulling performance down with automated monitors and flexible dashboards to begin troubleshooting. 
• Improves explainability: See how a model dimension affects prediction distributions, and leverage SHAP to explain feature importance for specific cohorts.

Arize’s self-serve options

Arize’s self-serve options enable users to detect root causes of issues and resolve model performance issues fast, regardless of the number of models deployed in production, Dhinakaran said. 

With its integration via an SDK or file ingestion from major cloud storage providers, ML teams can begin monitoring and troubleshooting model performance in minutes, Dhinakaran said. 

“The reality today is that most teams are only doing ‘red light; green light’ model monitoring and haven’t yet embraced true ML observability with ML performance tracing to pinpoint the source of model performance problems before they impact customers or the bottom line,” Dhinakaran said. “We are changing that with a platform that is purpose-built to tackle the toughest ML observability challenges of the world’s most respected organizations.”

Free version available

In a recent survey of more than 900 data scientists, engineers and executives, Dhinakaran said Arize found that 84% of data scientists and ML engineers say the time it takes to detect and diagnose problems with a model is an issue for their teams at least some of the time. This challenge is more significant when teams are reliant upon solutions that are not optimized to detect, root cause, and quickly resolve model performance issues, she said.

Arize has three versions – Free, Pro and Enterprise – of its platform that map directly to the number of models used in production. The free version of Arize delivers access to the full version of the platform for up to two models, 500 features per model and 500K production prediction. Arize AI competes in a growing market that includes IBM Watson Studio, FiddlerAI, Databricks Lakehouse, InRule, V7, MLOps, Algorithmia, Neptune.ai, Comet.ML, Determined AI and DVC.