Kafka Connect Made Convenient
--
Working with connectors can be easier with kcctl
This post assumes that you already have the knowledge to get a working Kafka environment, e.g., you have at least a working Kafka cluster with Kafka Connect, or you have a working service hosted somewhere. For simplicities sake, you can sign up for a free trial account at Aiven. Setting up a Kafka service there is as trivial as a few mouse clicks.
I have recently begun working with data pipelines using Kafka to move data between various sinks and sources. For example, you might use a MySQL DB as a source of data and use Kafka to propagate that data into S3 for long-term storage, sending Kafka metrics to Influx for use by operations in tracking the server's health. I’m sure you can come up with scenarios that would make sense to you personally without much help. All of these sources and sinks are most conveniently tied together using Kafka Connect.
Kafka Connect relies on a REST API for the configuration of connectors. I love REST API’s. They can be convenient to use, allow a clean separation of architectural boundaries, and are conveniently agnostic when it comes to interfacing. However, when we’re talking about ease of development, I personally don’t like them, primarily because I either have to construct URIs by hand for every call or come up with some way of avoiding that task like pulling up the closest command from shell command history and tweaking it. Nothing insanely painful but it would be much easier if I had a CLI tool.
This is where kcctl comes in. This tool provides a CLI wrapper around the REST API which allows you to interact with Kafka Connect more conveniently. It takes the responses and prints them in an easily readable format to the terminal. The installation instructions for kcctl can be found at the GitHub repo link above. I will caveat the installation instructions with one item; as of the time of this writing, the brew install method was not working on my Mac M1 laptop. I suspect this is another M1 issue because installing with homebrew works fine on my Linux laptop. My suggestion for anyone having difficulty on a Mac M1 machine would be to either build from the source or try the precompiled pre-release version. You can find a link to the precompiled release in the repo README file.
