Ray, the machine learning tech behind OpenAI, levels up to Ray 2.0

Over the last two years, one of the most common ways for organizations to scale and run increasingly large and complex artificial intelligence (AI) workloads has been with the open-source Ray framework, used by companies from OpenAI to Shopify and Instacart.

Ray enables machine learning (ML) models to scale across hardware resources and can also be used to support MLops workflows across different ML tools. Ray 1.0 came out in September 2020 and has had a series of iterations over the last two years.

Today, the next major milestone was released, with the general availability of Ray 2.0 at the Ray Summit in San Francisco. Ray 2.0 extends the technology with the new Ray AI Runtime (AIR) that is intended to work as a runtime layer for executing ML services.  Ray 2.0 also includes capabilities designed to help simplify building and managing AI workloads.

Alongside the new release, Anyscale, which is the lead commercial backer of Ray, announced a new enterprise platform for running Ray. Anyscale also announced a new $99 million round of funding co-led by existing investors Addition and Intel Capital with participation from Foundation Capital.

“Ray started as a small project at UC Berkeley and it has grown far beyond what we imagined at the outset,” said Robert Nishihara, cofounder and CEO at Anyscale, during his keynote at the Ray Summit.

OpenAI’s GPT-3 was trained on Ray

It’s hard to understate the foundational importance and reach of Ray in the AI space today.

Nishihara went through a laundry list of big names in the IT industry that are using Ray during his keynote. Among the companies he mentioned is ecommerce platform vendor Shopify, which uses Ray to help scale its ML platform that makes use of TensorFlow and PyTorch. Grocery delivery service Instacart is another Ray user, benefitting from the technology to help train thousands of ML models. Nishihara noted that Amazon is also a Ray user across multiple types of workloads.

Ray is also a foundational element for OpenAI, which is one of the leading AI innovators, and is the group behind the GPT-3 Large Language Model and DALL-E image generation technology.

“We’re using Ray to train our largest models,” Greg Brockman, CTO and cofounder of OpenAI, said at the Ray Summit. “So, it has been very helpful for us in terms of just being able to scale up to a pretty unprecedented scale.”

Brockman commented that he sees Ray as a developer-friendly tool and the fact that it is a third-party tool that OpenAI doesn’t have to maintain is helpful, too.

“When something goes wrong, we can complain on GitHub and get an engineer to go work on it, so it reduces some of the burden of building and maintaining infrastructure,” Brockman said.

More machine learning goodness comes built into Ray 2.0

For Ray 2.0, a primary goal for Nishihara was to make it simpler for more users to be able to benefit from the technology, while providing performance optimizations that benefit users big and small.

Nishihara commented that a common pain point in AI is that organizations can get tied into a particular framework for a certain workload, but realize over time they also want to use other frameworks. For example, an organization might start out just using TensorFlow, but realize they also want to use PyTorch and HuggingFace in the same ML workload. With the Ray AI Runtime (AIR) in Ray 2.0, it will now be easier for users to unify ML workloads across multiple tools.

Model deployment is another common pain point that Ray 2.0 is looking to help solve, with the Ray Serve deployment graph capability.

“It’s one thing to deploy a handful of machine learning models. It’s another thing entirely to deploy several hundred machine learning models, especially when those models may depend on each other and have different dependencies,” Nishihara said. “As part of Ray 2.0, we’re announcing Ray Serve deployment graphs, which solve this problem and provide a simple Python interface for scalable model composition.”

Looking forward, Nishihara’s goal with Ray is to help enable a broader use of AI by making it easier to develop and manage ML workloads.

“We’d like to get to the point where any developer or any organization can succeed with AI and get value from AI,” Nishihara said.