Confluent expands Kafka Steams capabilities, acquires Apache Flink vendor

The ability to move, manage and process data in real time is the domain of data streaming, which is largely dominated by a series of open-source technologies.

The ability to stream data is a core capability of the open-source Apache Kafka technology. Among the leading vendors that supports Kafka with commercial products and services is Confluent, which is led by the original creators of Kafka. Confluent had its initial public offering (IPO) in June 2021, with demand for real time data streaming continuing to grow.

However, the ability to stream data is only one component of enabling enterprise applications, as organizations also need to be able process streams so that they can be useful for business operations, business intelligence (BI) and data analytics. The Kafka Streams technology that Confluent uses enables some stream processing, but according to Confluent CEO and Kafka co-creator Jay Kreps, there is more that enterprises need.

“Streaming data has become very mainstream, and Kakfa is a platform now in almost every company,” Kreps told VentureBeat. “So, the next kind of logical problem to solve is, ‘What do you do with the streams of data and how can you make it easier and easier to build applications around?’”

To help expand its data stream processing capabilities, Confluent today announced its acquisition of privately held vendor Immerok, a leading contributor to the Apache Flink project. The financial terms of the deal have not been publicly disclosed.

Apache Flink is a popular open-source stream processing supported by multiple commercial vendors including Aiven and Alibaba, which owns Vervetica.

Have no fear, Apache Flink is here

Vervetica was formerly known as DataArtisans and was acquired by Alibaba in January 2019 for $103 million. The DataArtisans founders were the original core creators of Apache Flink.

A number of the original team members from Vervetica left the company in 2022 to join startup Immerok, which at the time had raised approximately $17 million in funding. While the company is relatively new, Kreps said is has done substantial work on building a real cloud native service around Flink.

In Kreps’ view, the basic idea with a cloud native interface is having an open, well established interface that developers can build applications with. This is the direction in which Confluent had already been taking Kafka with its Confluent Cloud data in motion platform. And, this is a direction that Immerok and Flink will fit into.

Kreps explained that Confluent’s direction will help enable a hub of streaming data, then provide users with different ways for working with that data.

What does Flink do anyway?

A database typically has two fundamental components: a storage layer where data resides and a query processing layer for executing queries against data. Kreps explained that in the traditional database world, storage and query processing are part of the same product. 

With Kafka, data streaming can be stored in different ways, including with open source ksqlDB technology led by Confluent. Querying data, which is what stream processor Apache Flink does, is often a separate technology.

“Kafka is kind of central exchange of streaming data and we think Flink can be the processing layer that people build applications against,” said Kreps.

The idea of separating the data layer from the query engine is also common in the data warehouse and data lakehouse markets. In Kreps’ view, the model that Confluent will build out with Flink can be thought of as a real-time data version of a data warehouse or data lakehouse.

Confluent’s existing Kafka Streams capability also enables stream processing, though it works in a different way. Kreps explained that Kafka Streams is an embedded software library that developers can integrate into applications. In contrast, Flink runs as a service at the cloud layer, enabling developers to build SQL and other real time applications.

Looking forward, Confluent’s plan is to integrate the Immerok Flink capabilities into its Stream Designer, which was announced in Oct. 2022. Stream Designer provides a point and click approach to help developers connect different Kafka data streaming sources with enterprise applications.

“We definitely think there are lots of ways of using streams of data and we want them all to grow and thrive,” said Kreps.