Consider the scenario where you aim to read data from a Kafka topic, apply transformations using Flink, and then push the results back to another Kafka topic. The process entails a substantial amount of boilerplate code. Initially, you’d need to install Kafka and Flink, potentially on your local machine for development purposes. Next steps involve creating a Kafka consumer to read data, developing code for data transformation, and establishing a Kafka sink for writing results back to Kafka.
But hold on, what format is the data stored in on Kafka? How will Flink interpret it? Typically, data on Kafka is in JSON format, requiring the addition of two functions: a deserialization function to enable Flink to manipulate the data and a serialization function to push the data back to the Kafka sink in JSON format. It’s also a good idea to include metadata, such as processing time, at this stage.
Now, your pipeline is set up. Or is it? Have you tested it? Running a quick test involves launching the Flink job, checking the result on the output Kafka topic, stopping the job, and cleaning up the output Kafka topic. Likely, you’ll need to go through this cycle several times before your code runs as expected.
Now, let’s consider a more complex scenario. What if you have two Kafka topic sources that you want to join, ensuring both topics have the same format? This requires good processes and code for managing Kafka topics in a standardized way or writing custom code in Flink for specific deserialization logic. For the join operation, you’ll need to manage Flink state, configure the appropriate backend, handle checkpoints, allocate processing resources, and more.
As the complexity grows, so do considerations about testing, deploying to production, governance, data retention, data access, scaling, monitoring, schema changes, and managing upgrades to new Kafka and Flink versions. This becomes a substantial job in itself.
To address these challenges, we created PipeStore, a no-code data streaming platform that seamlessly combines the capabilities of Kafka and Flink, enabling the creation of pipelines in just a few minutes. Testing complex pipelines, involving multiple sources and transformations, is as straightforward as clicking the ‘Deploy’ button and reviewing results on the screen. Agile and iterative code modification becomes hassle-free without the need to manage Kafka topics, Flink jobs, and states. PipeStore’s approach significantly simplifies maintenance, and our case studies indicate a productivity increase of a factor of 100 compared to traditional pipelines creation using Kafka and Flink.
Too good to be true? Try it out!