In a previous blog I talked about building a standalone executable from a Java based Kafka Streams application. Subject to a modification to the Kafka code, this worked well. Today I’m going to introduce an alternative way of getting there using Quarkus and its support for Microprofile.
Using Quarkus, I was able to make a streaming application consuming from and producing to kafka topics. Docker image sizes reduced to less than half of the previous size. Application memory consumption and CPU usage both dropped to perhaps 1/8th of what it was. But beware: This did not use Kafka Streams (future blog to cover this option) and the quarkus framework has to be used – existing codebases would need work.
I looked at Quarkus a short time ago but did not come across any support for Kafka. Things are moving fast however, and there are now a couple of approaches to using Kafka. In this blog, I am showing an approach based on the Quarkus Kafka Guide (https://quarkus.io/guides/kafka-guide).
These are not the Streams you thought you were looking for!
Kafka Streams is undoubtedly a tremendous API: KSQL has been built on it after all, alongside plentiful bespoke solutions. Close examination of the Quarkus Kafka Guide will, however, show that there is no streaming taking place. However, there is a “Price Generator” that writes to Kafka, and a “Price Converter” that reads from Kafka. It seemed like a simple operation to combine these, giving an app that could read from Kafka, perform a transformation, and write back to another Kafka topic.
This is akin to using a Consumer and a Producer, rather than Kafka Streams. What we’re not getting by doing this are the other features that Kafka Streams bring, such as the ability to join streams. While doing this may still be possible, we’d quickly find ourselves trying to write a new implementation of Kafka Streams – not something that most of us would want to undertake. So, the approach outlined here will be based on the assumption that you’re wanting something pretty simple. Many Kafka Streams based apps certainly fit into this category, but not all.
That aside: this approach is also worth considering if you need to create an app that will either produce to, or consume from, Kafka.
Like the previous blog using Kafka Streams with GraalVM, the app is intended to read XML input from a Kafka topic and write the corresponding JSON to another topic.
You can find the source code here.
Creating the project
Following the Kafka Guide, I created a new project using the maven archetype (note that the version changes quite frequently – at the time of writing this blog the version from the Kafka Guide is now 0.19.1):
mvn io.quarkus:quarkus-maven-plugin:0.16.1:create \ -DprojectGroupId=com.aimyourtechnology.quarkus.kafka \ -DprojectArtifactId=xmlJsonConverter -Dextensions="reactive-kafka,vert.x"
Adding an Acceptance Test
I had already created a useful automated acceptance test which writes to the input topic and reads from the output, asserting the Json received is correct. Adding this was to be my 2nd commit.
As I’d been through an implementation before, this was perhaps the easiest part. The application.properties file in a Quarkus project is quite key. The Kafka Guide gave good guidance on what I’d need, and by pointing some values to environment variables, it was pretty simple to set the app up to be quite generic.
I had some issues getting a native image to work, so I ended up creating a new Dockerfile using the following base image:
I then added a few scripts to help, and documented them in the README so I wouldn’t forget!
Native Image on Mac
Building a native OSX image was simple:
mvn package -Pnative
Running it simply needed the environment variables to be set, which I did through a bash script:
#!/usr/bin/env bash export KAFKA_BROKER_SERVER="localhost" export KAFKA_BROKER_PORT="9092" export INPUT_KAFKA_TOPIC="incoming.op.msgs" export OUTPUT_KAFKA_TOPIC="modify.op.msgs" export APP_NAME="sns-incoming-operator-messages-converter3" export MODE="xlToJson" ./target/xmlJsonConverter-1.0-SNAPSHOT-runner -Xmx48M
Native Image on Docker
Building a docker image was now pretty simple:
./mvnw package -Pnative -Dnative-image.docker-build=true docker build -f src/main/docker/Dockerfile.tiny -t sns/quarkus-xml-json-converter-tiny .
Running it was as simple as you’d expect, just providing the required environment variables:
docker run -i --rm --network=sns2-system-tests_default -e APP_NAME=linuxXmlToJsonConverter -e MODE=xmlToJson -e KAFKA_BROKER_SERVER=sns2-system-tests_kafka01.internal-service_1 -e KAFKA_BROKER_PORT=9092 -e INPUT_KAFKA_TOPIC=incoming.op.msgs -e OUTPUT_KAFKA_TOPIC=modify.op.msgs sns/quarkus-xml-json-converter-tiny
After development, I upgraded the version to 0.18.0 simply by changing the quarkus.version in the project pom.xml. It did cause the application.properties file to change, but there were no other issues.
In my previous blog, I compared a normal JVM application against the GraalVM native image. As I’m effectively working on the same application, here I have added on stats where I built the native executable using Quarkus. I’ve placed these below the previously garnered stats to make it easier to compare all approaches.
|Arguments||Memory Usage||Physical Footprint||CPU Usage|
|JVM||-Xmx48m||Real: 370MB; Private: 337MB; Shared: 25MB||343M||0.6%->3.7%|
|GraalVM Native Image||-Xmx48m||Real: 22MB; Private: 8MB; Shared: 1MB||10M||0.4%|
|Quarkus Native Image||-Xmx48m||Real: 23MB; Private: 12MB; Shared: 3MB||16M||0.2%|
|Arguments||Docker Image Size||Memory Usage||CPU Usage|
|GraalVM Native Image||-Xmx48m||32.5MB||8MiB||1.5%|
|Quarkus Native Image||-Xmx48m||47.9MB||8.9MiB||0.42%->0.84%|
While we see that the Quarkus native image is still considerable smaller than the JVM equivalent in all aspects, we note that the GraalVM option is even smaller. This is undoubtedly down to the fact that with the Quarkus build, we’re getting more of a framework thrown in, and we’re able to do a lot more with it should we want to – I encourage you to browse the Guides on the Quarkus site to get an idea.
This said, we did not have to rely on a modification to the Kafka codebase to make it work! Until this issue is addressed, Quarkus may be your best route to a small footprint for streaming with Kafka.
Quarkus is also limited to using either Java or Kotlin – its the wrong place to look (for now at least) if you want to use any other JVM language.
With Quarkus moving quickly, a Kafka Streams extension has been added. While it is early days, progress is promising. My next blog in this series will take a closer look.