zeebe-hazelcast-exporter

Export records from Zeebe to Hazelcast. Hazelcast is an in-memory data grid which is used as a transport layer.

The records are transformed into Protobuf and added to one ringbuffer. The ringbuffer has a fixed capacity and will override the oldest entries when the capacity is reached.

Multiple applications can read from the ringbuffer. The application itself controls where to read from by proving a sequence number. Every application can read from a different sequence.

The Java module provide a convenient way to read the records from the ringbuffer.

Usage

Java Application

Add the Maven dependency to your pom.xml

<dependency>
	<groupId>io.zeebe.hazelcast</groupId>
	<artifactId>zeebe-hazelcast-connector</artifactId>
	<version>1.0.1</version>
</dependency>

Connect to Hazelcast and register a listener

ClientConfig clientConfig = new ClientConfig();
clientConfig.getNetworkConfig().addAddress("127.0.0.1:5701");
HazelcastInstance hz = HazelcastClient.newHazelcastClient(clientConfig);

final ZeebeHazelcast zeebeHazelcast = ZeebeHazelcast.newBuilder(hz)
    .addProcessInstanceListener(processInstance -> { ... })
    .readFrom(sequence) / .readFromHead() / .readFromTail()
    .build();

// ...

zeebeHazelcast.close();

Install

Docker

A docker image is published to GitHub Packages that is based on the Zeebe image and includes the Hazelcast exporter (the exporter is enabled by default).

docker pull ghcr.io/camunda-community-hub/zeebe-with-hazelcast-exporter:1.1.2-1.0.1

For a local setup, the repository contains a docker-compose file. It starts a Zeebe broker with the Hazelcast exporter. The version of the exporter is defined in the .env file.

mvn clean install -DskipTests
cd docker
docker-compose up

Manual

1. Download the latest Zeebe distribution (zeebe-distribution-%{VERSION}.tar.gz )

2. Download the latest exporter JAR (zeebe-hazelcast-exporter-1.0.1-jar-with-dependencies.jar)

3. Copy the exporter JAR into the broker folder ~/zeebe-broker-%{VERSION}/exporters.

   cp exporter/target/zeebe-hazelcast-exporter-1.0.1-jar-with-dependencies.jar ~/zeebe-broker-%{VERSION}/exporters/
   

4. Add the exporter to the broker configuration ~/zeebe-broker-%{VERSION}/config/application.yaml:

   zeebe:
     broker:  
       exporters:
         hazelcast:
           className: io.zeebe.hazelcast.exporter.HazelcastExporter
           jarPath: exporters/zeebe-hazelcast-exporter-1.0.1-jar-with-dependencies.jar
   

5. Start the broker ~/zeebe-broker-%{VERSION}/bin/broker

Configuration

In the Zeebe configuration file, you can change

• the Hazelcast port
• the Hazelcast cluster name
• the value and record types which are exported
• the ringbuffer's name
• the ringbuffer's capacity
• the ringbuffer's time-to-live
• the record serialization format

Default values:

zeebe:
  broker:
    exporters:
      hazelcast:
        className: io.zeebe.hazelcast.exporter.HazelcastExporter
        jarPath: exporters/zeebe-hazelcast-exporter.jar
	args:
	      # Hazelcast port
    	  port = 5701
    	  
    	  # Hazelcast cluster name
    	  clusterName = "dev"
    
          # comma separated list of io.zeebe.protocol.record.ValueType to export or empty to export all types 
          enabledValueTypes = ""
    
          # comma separated list of io.zeebe.protocol.record.RecordType to export or empty to export all types
          enabledRecordTypes = ""
        
          # Hazelcast ringbuffer's name
          name = "zeebe"
    
          # Hazelcast ringbuffer's capacity
          capacity = 10000 

          # Hazelcast ringbuffer's time-to-live in seconds. Don't remove the records until reaching the capacity by setting it to 0.  
          timeToLiveInSeconds = 0

          # record serialization format: [protobuf|json]
          format = "protobuf"

The values can be overridden by environment variables with the same name and a ZEEBE_HAZELCAST_ prefix (e.g. ZEEBE_HAZELCAST_PORT).

Connect to an External/Remote Hazelcast Cluster

By default, the exporter creates an in-memory Hazelcast instance and publishes the records to it. But it can also be configured to export the records to a remote/external Hazecast instance by setting the argument remoteAddress or the environment variable ZEEBE_HAZELCAST_REMOTE_ADDRESS to the address of the remote Hazelcast instance.

• the remote Hazelcast address
• the remote Hazelcast cluster name
• the connection timeout

Default values:

zeebe:
  broker:
    exporters:
      hazelcast:
        className: io.zeebe.hazelcast.exporter.HazelcastExporter
        jarPath: exporters/zeebe-hazelcast-exporter.jar
	args:
	      # remote Hazelcast address
    	  remoteAddress = 127.0.0.1:5702
    	  
    	  # Hazelcast cluster name
    	  clusterName = "dev"
    	  
    	  # connection timeout
    	  remoteConnectionTimeout = "PT30S"

Full docker-compose.yml with external Hazelcast

version: "2"

networks:
  zeebe_network:
    driver: bridge

services:
  zeebe:
    container_name: zeebe_broker
    image: camunda/zeebe:1.1.2
    environment:
      - ZEEBE_LOG_LEVEL=debug
      - ZEEBE_HAZELCAST_REMOTE_ADDRESS=hazelcast:5701
    ports:
      - "26500:26500"
      - "9600:9600"
    volumes:
      - ../exporter/target/zeebe-hazelcast-exporter-1.0.1-jar-with-dependencies.jar:/usr/local/zeebe/exporters/zeebe-hazelcast-exporter.jar
      - ./application.yaml:/usr/local/zeebe/config/application.yaml
    networks:
      - zeebe_network
    depends_on:
      - hazelcast

  hazelcast:
    container_name: hazelcast
    image: hazelcast/hazelcast:4.2
    ports:
      - "5701:5701"
    environment:
      - JAVA_OPTS="-Dhazelcast.local.publicAddress=hazelcast:5701"
    networks:
      - zeebe_network
      
  hazelcast-management:
    container_name: hazelcast-management
    image: hazelcast/management-center:4.2
    ports:
      - "8083:8080"
    networks:
      - zeebe_network
    depends_on:
      - hazelcast    

Check out the Hazelcast documentation on how to deploy and configure Hazelcast on Kubernetes.

Build it from Source

The exporter and the Java connector can be built with Maven

mvn clean install

Code of Conduct

This project adheres to the Contributor Covenant Code of Conduct. By participating, you are expected to uphold this code. Please report unacceptable behavior to [email protected].