This was my second time attending the two-day event at the Austin Convention Center. At the conference, I reconnected with familiar faces and made plenty of new friends from the Kafka, Flink, and real-time data community.

Visionaries Unite: Shaping Tomorrow’s Data Landscape with Real-Time Insights

Day 1 of the event kicked off with an impactful keynote featuring Jay Kreps, CEO and Co-Founder of Confluent, alongside industry leaders like Seema Acharya (Mercedes Benz R&D), Richie Artoul (WarpStream by Confluent), Shaun Clowes (Confluent), Kushal Khandelwal (Viacom18), and Adam Tybor (Accenture).

Each speaker shared insights on the future of data, AI, and real-time streaming, highlighting their experiences and innovations across various industries.

Engineering the Future: The Data Streaming Revolution and Its Pioneers

Day 2's keynote explored the transformative impact of data streaming on the industry, highlighting the emergence of the Data Streaming Engineer role that unifies the work of application developers, data engineers, data scientists, and architects. Speakers included David Anderson, Tim Berglund, Lucia Cerchie, Danica Fine, Sandon Jacobs, Adi Polak, Anna Povzner, and Matthew Seal, all from Confluent.

Attendees gained valuable insights into the current technologies shaping the Data Streaming Platform and learned how to prepare for their implementation in the coming year.

Breaking New Ground with BYOC

Confluent formalized the announcement of acquiring WarpStream, a Kafka-compatible streaming solution with innovative architecture, to expand its cloud-native streaming offerings. This addition enhances Confluent's ability to meet diverse customer needs with advanced features like offset-preserving replication, cluster quotas and next-gen BYOC architecture.

In just 13 months, WarpStream grew its tea,, launched new products, expanded cloud regions, achieved SOC2 Type II certification, and onboarded major customers like Grafana Labs, Zomato, and CharacterAI.

During the keynote, WarpStream founder Richie Artoul discussed the adoption of BYOC (Bring Your Own Cloud) as a cloud service, emphasizing its ability to ensure data sovereignty and security by keeping all data securely within the customer's account.

Confluent Acquires WarpStream

Spotlighting the Innovators

At the conference, I reconnected with several friends from various companies and made some great new connections.

DeltaStream

As one of the founding engineers, I've been part of DeltaStream’s journey since the very beginning, joining the team 3.5 years ago. After raising an initial $10 million in seed funding from NEA, I played a key role in building out our cloud-based infrastructure, which includes multi-tenant, dedicated, and BYOC offerings.

DeltaStream is founded by Hojjat Jaffarpour, the original creator of KSQL. Our platform is a serverless, unified stream processing solution designed to simplify the development of stream processing applications by eliminating unnecessary complexity. At the conference, DeltaStream unveiled their Series A funding round raising $15 million, led by NEA, Galaxy Interactive and Sanabil Investments.

StreamNative

I met Sijie Guo, the founder of StreamNative, at their booth and had an engaging conversation about the importance of BYOC and serverless solutions in their go-to-market strategy. Sijie is a member of the Apache Software Foundation (ASF), serving as a PMC member, committer, and contributor for projects like Apache Pulsar, Apache Bookeeping and Apache Doris.

StreamNative has re-architected data streaming from the ground up for cloud-native environments using Apache Pulsar. Their ONE StreamNative Platform, powered by the Ursa engine, provides a scalable, high-performance, and cost-efficient data streaming solution that is compatible with the Kafka API and can run on any cloud.

Sijie Guo, creator of Apache Pulsar, Founder @ StreamNative

RisingWave

RisingWave is on my favorite stream processing runtime, that unified both batch and streaming data and is postgres compatible. RisingWave aims to democratize stream processing by making it simple, affordable, and accessible.

Yingjun, the talented founder of Rising Wave, previously was a software engineer on the Redshift team at AWS and a researcher at IBM Almaden Research Center, holds a PhD from the National University of Singapore and was a visiting PhD at Carnegie Mellon University.

RisingWave has 2 product lines, the Rising Wave Cloud and the Rising Wave streaming database, all built in RUST to provide continuous analytics, build event-driven applications, real-time ETL pipelines, and feature stores in minutes.

I met up with Rayees Pasha at the conference, who leads product at RisingWave.

Rayees Pasha @ RisingWave

Restate

Stephan Ewen, Till Rohrmann and Igal Shilman are the co-founders of Restate. Till also serves as an advisor for various companies using Apache Flink, including DeltaStream.

Restate at the core has build a durable state execution engine to provide the simplest way to build resilent applications. Restate is on a mission to simplify modern distributed applications by enhancing their resilience, reliability, and fault tolerance.

Their team features co-creators of Apache Flink and distributed processing experts from renowned companies like Apple, Stripe, Amazon, and Meta.

Restate provides durable state execution engine using kafka

Responsive

Responsive is yet another company emerging from the Confluent mafia cohort, and if you ever found KStreams to be challenging to work with, they’ve created a product that makes it incredibly simple.

I met the three remarkable founders—Apurva Mehta, Rohan Desai, Almog Garva—at their booth. In their previous roles at Confluent, they were pivotal in the development of KStreams and KSQL.

AntiThesis

This was my top discovery product on the floor. In the past, we relied heavily on system tests at Confluent to evaluate Kafka, but these tests often fell short in catching edge cases across the wide range of Kafka configurations.

AntiThesis addresses this challenge by offering autonomous testing for distributed systems. Their product is utilized for black box testing by notable companies such as Ethereum, Palantir, and MongoDB. Discover how WarpStream leverages AntiThesis in this blog post.

Timeplus

Timeplus is a SQL stream processing and analytics engine that combines columnar and row stores within a single binary. It enables the creation of real-time applications for both edge and cloud environments. I had a fantastic time chatting with Jove Zhong, Co-founder and Head of Product at TimePlus, about the inner workings of TimePlus and especially their Proton engine.

TimePlus Team - Sarwar Bhuiyan, Wei Q

FactorHouse

FactorHouse aims to empower engineers with tools that simplify working with Apache Kafka and Apache Flink, making real-time streaming compute a seamless and enjoyable experience. They provide an integration and observability platform for stream processing with products like Flow Integration platform, KPow for Apache Kafka and Flex for Apache Flink.

Founded by Derek and Kylie Troy-West in 2019, FactorHouse leverages extensive experience with Apache Kafka, Cassandra, and Storm to deliver data platforms for global banking, insurance, and fintech.

Their tools are immensely popular, with hundreds of thousands of installations worldwide. Fun fact: they’ve bootstrapped the company since 2019, and a little birdie told me they’re about to announce their seed funding round soon.

Datorios

Coming from Israel, Datorios aims to democratize real-time data processing with a development console for Apache Flink, empowering engineering teams to make stream processing standard and unlock their companies' real-time potential.

While visiting the booth, I learned Datorios builds and develops their own distribution of Apache Flink.

LittleHorse

I had a great lunch with Colt McNealy at the conference. If you've worked with Kafka Streams, you're likely familiar with Colt or LittleHorse!

LitteHorse enables the creation of workflow-driven applications by offering durable state execution with Kafka Streams, allowing for the rapid composition and deployment of microservices to enhance developer agility in meeting business demands.

ThatDot

Have you ever considered writing stream processing applications using Graph instead of Apache Kafka? Enter thatDot, the creators of Quine OSS, who present the world’s first streaming graph analytics engine, developed over eight years for advanced persistent threat detection at Darpa and funded by Crowdstrike.

Ryan Wright, the CTO and Founder of thatDot, always offers engaging conversations about graph-based stream processing, threat detection, and fraud analysis.

Platformatory

Based out of Bengaluru, Platformatory provides consulting, system integration, and platform engineering for streaming data solutions. Pavan Keshavamurthy, the founder of Platformatory is a quite popular speaker in the streaming circles.

His talk Events at the (API) Horizon: How to synthesize domain events & changes from your HTTP/REST APIs at the conference introduced an alternative approach that utilizes standard API proxies or gateway solutions to process and push events to Kafka directly from the API data plane, reducing the need for extensive code changes and complex CDC implementations.

Feldera

Lalith Suresh, CEO and Founder of Feldera, has developed an incremental computational data engine that enables rapid analysis of live and historical data with full SQL processing, handling millions of events per second and scaling to datasets larger than memory for AI use cases.

An incremental computational data engine is essential for RAG and AI use cases as it allows for real-time analysis and processing of vast amounts of data, enabling timely insights and decisions.

ShadowTraffic

You might have seen a walking QR code invite for a free Kafka test stream, encouraging you to check out ShadowTraffic, created by the talented Michael Drogralis, who previously managed the KStreams and KSQL products at Confluent.

Follow his journey as a solo bootstrapped entrepreneur running ShadowTraffic (yet another company by Confluent Mafia #yacbcm), achieving around $100k in ARR, on LinkedIn.

InfinyOn

InfinyOn, a real-time event streaming company, has secured $5 million in seed funding led by Gradient Ventures and Fly Ventures, with participation from Bessemer Venture Partners and others.

The company aims to enhance customer experiences through its modern cloud-native event streaming platform, which features in-line computation powered by WebAssembly, enabling large-scale multi-channel dataflows.

With strong traction across various industries and recognition on G2’s list for Event Stream Processing Software, InfinyOn leverages its Fluvio Open Source platform, built in Rust, to simplify data architectures and improve efficiency while maintaining high security and low memory usage.

Debadyuti Roy Chowdhury, who is a great friend, runs the product organization at InfinyOn.

Final Thoughts

The data streaming world gathered in Austin, Texas for Current 2024, uniting the community in an exhilarating celebration of all things Apache Kafka® and Apache Flink®. Attendees were treated to insights from tech leaders, industry giants, and startups, who filled the air with a wealth of knowledge about real-time event streaming and beyond.

What an exciting time to connect and learn together on topics like private saas, serverless, byoc, durarable state execution, streaming lineage, streaming observability, making kafka protocol more efficient! Innovation in real time data opens up exciting opportunities that inspire creativity to drive transformative changes across industries.

Today marks approximately 10 years of my experience building infrastructure with Kafka and its ecosystem.

In 2014, I began using the Apache distribution of Kafka at SAP Ariba while building cloud infrastructure as the organization transitioned from a SOA architecture to a microservices-based architecture. Soon, we were deploying Confluent's Kafka distribution to production, onboarding teams to leverage Kafka as the central nervous system for the microservices. The reason for this transition was that Confluent's distribution included Schema Registry and several other proprietary features that were more effective for running Kafka in production.

Witnessing the rise in Kafka adoption both internally and externally, I joined Confluent in 2017. There, I had the opportunity to build developer productivity and platform infrastructure for Kafka, Connect, KSQL, Cloud, and the rest of the engineering organization. At Confluent, I witnessed firsthand the rise in adoption of the overall Kafka ecosystem.

The Evolution of Database Models from 1960 to 2010

In the 1960s, network-based and hierarchical models for computing and storing databases rapidly gained popularity among organizations. Examples include CODASYL, IMS, and SABRE.

In 1970, E.F. Codd's seminal paper introduced the relational database, revolutionizing database thinking by separating the database schema—the logical organization—from physical data storage. This innovation set the standard for modern database systems.

Between 1974 and 1977, two significant prototypes of relational database systems emerged: Ingres, developed at UBC, and System R, created at IBM San Jose. Ingres employed the QUEL query language, paving the way for systems such as Ingres Corp., MS SQL Server, Sybase, Wang’s PACE, and Britton-Lee.

System R introduced the SEQUEL query language, shaping the development of SQL/DS, DB2, Allbase, Oracle, and Non-Stop SQL. This era also marked the recognition and adoption of the term Relational Database Management System (RDBMS).

In 1976, Peter Chen proposed a new database model called Entity-Relationship, or ER. This model enabled designers to prioritize data application over the traditional focus on logical table structure.

Structured Query Language, known as SQL, was adopted as the standard query language by the American National Standards Institute in 1986 and by the International Organization for Standardization in 1987.

Since then, we have witnessed a significant evolution as the popularity of RDBMS rose in contrast to existing network-based and hierarchical-based data management tools. In subsequent years, a multitude of products emerged, including DB2, PARADOX, RBASE 5000, RIM, dBASE III and IV, OS/2 Database Manager, and Watcom SQL.

The 1990s played a crucial role in advancing databases and database software, particularly towards application development. New client tools like Oracle Developer, MySQL, PowerBuilder, and Visual Basic (VB) were released. Additionally, tools enhancing developer productivity such as ODBC, Excel, and Access were developed. Prototypes for Object Database Management Systems (ODBMS) emerged in the early 1990s. By 2010, there was a notable shift in how data was generated and utilized, both within organizations and externally, with row-based and column-based databases finding market acceptance with NoSQL solutions.

Modern Data Stack

From 2010 to the present day, the modern data stack embodies a comprehensive ecosystem of products crafted for efficient data management and analytics. Products in this category typically encompass robust components such as cloud storage solutions like Amazon S3 or Google Cloud Storage, providing scalable data storage. Data pipelines are orchestrated using tools such as Apache Kafka or Apache Airflow, ensuring smooth data ingestion and processing. Platforms like Snowflake or Google BigQuery provide scalable and high-performance solutions for data warehousing and analytics.

The stack is agile and scalable, accommodating diverse data sources and formats, including structured, semi-structured, and unstructured data. Real-time analytics capabilities are enhanced through technologies like Apache Flink or Apache Spark Streaming. Overall, the modern data stack empowers organizations to extract actionable insights, drive innovation, and maintain competitive advantage in today's data-driven world.

There is an evolutionary surge of products in this category, with solutions like ClickHouse, Atlan, MonteCarlo, Telmai, AirByte, RockSet, Materialize and numerous others gaining significant popularity.

Revolutionizing Data Efficiency: Stream Processing and Streaming Stores

Stream processing holds the promise of efficiently handling data with low latency and compute costs, even at massive volumes and high cardinality. With the aid of streaming stores, they eliminate the necessity for additional computations during data ingestion and can operate directly on the data source across multiple streams.

In addition to Kafka, emerging streaming stores like Redpanda and Warpstream are increasingly gaining traction. Today, companies like DeltaStream, RisingWave, Decodable, Arroyo etc are developing products aimed at simplifying the complexity involved in building stream processing applications. With each new evolution of a software stack, it commoditizes the stack that came before it.

As Stream processing improves efficiency by processing data closer to its source with low latency and handling high cardinality, it will surpass existing use cases currently served by the modern data stack by the end of this decade.

The adoption of AI/ML across organizations will drive this transformation. AI/ML applications demand real-time data processing for timely decisions and predictions. Stream processing facilitates handling continuous data streams with low latency, crucial for dynamic environments such as financial trading, IoT networks, and real-time recommendations.

In summary, stream processing is indispensable for AI/ML applications to achieve real-time analytics, adaptive decision-making, and operational efficiency across various industries.

“Data streaming is the central nervous system for data, while AI/ML algorithms are the brain. ~ Confluent”