Streaming Data is Undergoing a Paradigm Shift

Change is coming.

Kafka Tiered Storage for the Masses

If you’re a Kafka fan you may have heard that Apache Kafka 3.6 RC 0 implements tiered storage for Kafka. This means a preview will be available in the opensource Kafka.

Until now tiered storage was not supported natively in Kafka and has only been available using third-party tools or modules, most of them proprietary, such as the Confluent tiered storage module, and Amazon Managed Streaming for Apache Kafka. This means that if you wanted tiered storage, you were almost certainly locked into a vendor paying for it.

With the release of 3.6 RC 0, tiered storage is now available to the masses. This may sound small but in reality, it’s a pretty big deal, and I believe that we’re on the precipice of a true paradigm shift in how we approach streaming data.

Why This is Important

Kafka has become a huge part of the streaming data ecosystem and is arguably one of, if not the, most common entry point of all data into a distributed system. Data is commonly consumed in real-time, but there’s also the capability to fetch historic data as well, as long as it’s within the retention policy parameters. Here is where Kafka has traditionally shown its weakness. To…