Kafka Connect Made Convenient
Working with connectors can be easier with kcctl
This post assumes that you already have the knowledge to get a working Kafka environment, e.g., you have at least a working Kafka cluster with Kafka Connect, or you have a working service hosted somewhere. For simplicities sake, you can sign up for a free trial account at Aiven. Setting up a Kafka service there is as trivial as a few mouse clicks.
I have recently begun working with data pipelines using Kafka to move data between various sinks and sources. For example, you might use a MySQL DB as a source of data and use Kafka to propagate that data into S3 for long-term storage, sending Kafka metrics to Influx for use by operations in tracking the server's health. I’m sure you can come up with scenarios that would make sense to you personally without much help. All of these sources and sinks are most conveniently tied together using Kafka Connect.
Kafka Connect relies on a REST API for the configuration of connectors. I love REST API’s. They can be convenient to use, allow a clean separation of architectural boundaries, and are conveniently agnostic when it comes to interfacing. However, when we’re talking about ease of development, I personally don’t like them, primarily because I either have to construct URIs by hand for every call or come up with some way of avoiding that task like pulling…
