Report: 68% of orgs rely on in-house evaluation to tune AI models

Natural language processing (NLP), business intelligence (BI), data integration and data annotation were considered foundational AI technologies, according to a new report by Gradient Flow. The addition of data annotation to the AI technologies respondents plan to have in place by the end of 2022 indicates more sophisticated use cases as the technology matures.

As AI use cases advance, another interesting trend is taking place: practitioners are shifting from exclusively data scientists to domain experts. More than half (61%) of respondents identified clinicians as their target users, followed by healthcare providers (45%) and health IT companies (38%). 

As low- and no-code solutions gain traction, it’s likely this trend will continue in healthcare and beyond. Take building a website, for example — what once was a major software engineering effort is today mostly a graphic design project. This will be a huge step in further democratizing important technologies throughout different industries. 

There are several other factors at play with lowering the barriers to entry for AI; one is the availability of, and interest in, open-source technologies. Locally installed commercial software (37%) and open-source software (35%) were the most popular forms of software being used to build healthcare AI applications, respondents indicated. This shows a 12% decline in use of cloud services (30%) from last year’s survey (42%).

A potential reason for this shift from public cloud providers to reliance on open source technologies could be a sign that users are taking security and data privacy more seriously. In light of recent breaches and stringent laws and regulations unique to the healthcare industry, this is a move in the right direction. 

In fact, a majority of respondents (53%) chose to rely on their own data to validate models, rather than on third-party or software vendor metrics. Mature organizations (68%) relied even more heavily on using in-house evaluation and tuning models themselves — a step critical to prevent model degradation over time. 

While training and tuning models remains a priority for users, this year, both technical leaders and respondents from mature organizations cited the availability of healthcare-specific models and algorithms as the most important requirement when evaluating locally installed software libraries or SaaS solutions. This is further proof of how healthcare AI is being refined.

Gradient Flow’s survey ran online for 50 days, from January 13 to March 4, 2022, generating a total of 321 respondents from 41 countries. A quarter of respondents held technical leadership roles, a fifth of whom worked in organizations that have had AI models in production for over two years (referred to as “mature organizations” throughout the survey).

Read the full report by Gradient Flow.