Overview

Problem: Integration tests using Kafka containers face high memory consumption and long startup times, slowing down the testing process and increasing resource costs.
Solution: Using a native image for EmbeddedKafka with GraalVM.
Status: Completed, article is published, currently using.

---

Native Embedded kafka in Docker Container

Utilizes org.springframework.kafka.test.EmbeddedKafkaBroker and GraalVM to provide native kafka in docker.

Article

In the article, I want to share my experience with creating a native image for EmbeddedKafka using GraalVM. Utilizing this image in integration tests not only speeds up the execution of test scenarios but also reduces memory consumption. Interestingly, when compared to using confluentinc/cp-kafka in Testcontainers, there is a noticeable difference in both speed and memory usage — and it’s not in favor of the latter.

Article on Habr - Как сократить потребление памяти в интеграционных тестах с Kafka с помощью GraalVM Article on Medium - How to Reduce Memory Consumption in Integration Tests with Kafka Using GraalVM

Usage

Add dependency

implementation 'pw.avvero:emk-testcontainers:1.0.0'

Create a EmbeddedKafkaContainer to use it in your tests:

EmbeddedKafkaContainer kafka = new EmbeddedKafkaContainer("avvero/emk-native:1.0.0"); // OR avvero/emk:latest

Improved Testcontainers Support in Spring Boot 3.1

The new @ServiceConnection annotation can be used on the container instance fields of your tests. See example in tests.

Refer to article in Spring blog to get more information.

---

Examples for using Kafka in integration tests

One can find examples for using Kafka in integration tests:

• EmbeddedKafka - module example-spring-embedded-kafka
• Testcontainers - module example-testcontainers
• Embedded Kafka Native - module example-embedded-kafka-container

---

Docker Image

Ready to use Docker image is hosted on Docker Hub and can be pulled using the following command:

docker pull avvero/emk

or native image for linux/arm64

docker pull avvero/emk-native

Build container with java

Run command: make emk-docker-build

Build container with native

Run command: emk-docker-build-native

Native build details

Include Reachability Metadata

0. Setup graalvm: https://www.graalvm.org/latest/docs/getting-started
1. Include Reachability Metadata Using the Native Image Gradle Plugin
2. Run make emk-run-with-agent
3. Run activity over broker