Nvidia advances medical AI and digital twin capabilities

Nvidia has been a leader in providing AI and digital twin infrastructure for the medical community. Its various offerings improve diagnostics, the development of new medical devices, medical research and drug development. At the Fall GTC Conference, Nvidia announced various new medical tools, partnerships and workflows. 

“GTC is a really unique healthcare conference, where we learn how AI and accelerated computing are advancing the field, from things like surgery all the way through to pharmaceutical research,” Nvidia’s VP of healthcare, Kimberly Powell, said in a press conference.

Highlights include:

• Release of MONAI 1.0, a new domain-specific AI framework that improves AI imaging workflows to medical diagnostics and robotics.
• Migration of Clara Holoscan from MGX to IGX to simplify medical tool and robot development, deployment and management. 
• BioNeMo extends Nvidia’s large language model (LLM) to support protein, DNA and chemical analysis workflows. 
• Partnership with MIT’s and Harvard’s Broad Institute to accelerate human genomics research. 

These various announcements build on and extend each other. Let’s walk through them one at a time. 

MONAI simplifies medical imaging

Nvidia and King’s College London introduced MONAI in April 2020 to simplify AI medical imaging workflows. This helps transform raw imaging data into interactive digital twins to improve analysis or diagnostics, or guide surgical instruments. The development and adoption of the platform have picked up steam with over 600,000 downloads, half of these in the last six months. 

They are now officially rolling out Monai 1.0. It comes with several critical capabilities baked in. Interactive labeling can reduce by 75% the time required to label data for training AI models. Auto3D adapts AutoML techniques for automatically choosing machine learning models for 3D segmentation and interpretation. Monai Flare supports federated learning to enhance the privacy of medical data. Model Zoo comes with over 15 pretrained models. Native support for streaming imaging applications like endoscopy, ultrasound and surgical video helps streamline medical imaging workflows. 

IGX industrializes the medical metaverse 

Nvidia introduced Clara Holoscan MGX earlier this year as a reference design for a medical device platform. Clara Holoscan IGX builds on Nvidia’s experience to further streamline and industrialize medical device development on top of Nvidia’s new IGX platform for robotics. This reduces the effort it takes to integrate Holoscan into new products with integrated security and management capabilities.

Over 70 leading companies have been developing equipment on top of Clara Holoscan MGX, including Siemens Healthineers for MRI, Olympus for endoscopy, and Intuitive Ion for better lung biopsies. New Clara IGX products include Activ Surgical’s hyperspectral blood flow imager, Moon Surgical’s robotic-assisted surgeon, and Proximie’s telepresence surgery system. 

“We learned that what we’re building for these medical device use cases is actually applicable to a much broader market,” said Powell. “Industrial automation and smart factories all have a similar robotics pipeline that needs to be executed on the far edges of the network and incorporate things like functional safety so that humans and robots can be in the same place.”

The platform also helps minimize new applications’ latency to ensure patient safety. Powell said they set the goal of keeping latency down to 50 milliseconds. The latest version of Holoscan can do straight-up video processing in less than 10 milliseconds and supports more than 30 simultaneously running AI algorithms at less than 50 milliseconds. 

Powell said they are aligning Clara with Nvidia’s Isaac platform for robotics and Omniverse platform for industrial digital twins. “We’re leveraging everything the company makes, and we’re connecting these platforms together because robotics isn’t unique in healthcare as it is in other domains,” Powell said. “And we take all the lessons learned and the necessary interconnections between these platforms to provide it back to the medical device industry.”

BioNeMo speaks proteins

Nvidia’s new BioNeMo Framework helps medical researchers train and develop large biomolecular language models at supercomputing scales. It extends efforts like the Nvidia NeMo Megatron framework and research projects like AlphaFold that use large language models to analyze proteins to support DNA, protein and chemical research.  

Each domain has its own unique way of encoding data into strings. DNA uses nucleic acid sequences, proteins use amino acid sequences, and chemicals use simplified molecular-input line-entry system (SMILES) strings.

We have over 10,000 diseases and only 500 cures,” said Powell. “We need to boost numerical and experimental methods with AI to explore the nearly infinite chemistry and protein space. Nvidia BioNeMo LLM framework and cloud services will accelerate the development of AI that understands chemistry and biology.” 

The new framework comes with four pretrained models. ESM-1, introduced by Meta AI Labs, processes amino acid sequences to predict properties and functions. OpenFold helps visualize proteins. MegaMolBART can help predict chemical reactions, optimize mixtures or generate new ones. ProtT5 helps extend the capabilities of protein large language models to sequence generation. 

Powell said Nvidia is providing BioNeMo as both a framework and a service. The framework will help researchers develop new pre-trained language models at any scale for chemistry, protein, DNA and RNA. It also supports data transformations necessary for biomolecules. Nvidia plans to provide early access to the BioNeMo service in October. 

Nvidia-Broad partnership accelerates innovations

Nvidia has also announced an extensive partnership with the Broad Institute of MIT and Harvard, a top genetics research group and tools provider.

Nvidia is porting Clara Parabricks computational genomics framework to the Broad Institute’s Terra cloud platform used by 25,000 leading medical researchers. Initially, they plan to support six new workflows. For example, a new whole genome sequencing workflow running on GPUs shortens the process from a day to an hour and cuts the cost in half compared to a CPU approach. 

The two will also partner on building large language models for analyzing DNA and RNA. Nvidia is also contributing a new deep learning model to the Broad Institute’s genome analysis toolkit that more than 100,000 researchers use. 

Powell said combining the Broad Institute’s deep domain expertise with Nvidia technology expertise could accelerate the deployment of new AI medical innovations from years to months.