Follow the instructions from Get Docker CE for Ubuntu page.
Follow the instructions from Post-installation steps for Linux page.
- Go to your terminal.
- Clone this repository and go inside it
git clone https://github.com/mjaglan/docker-kafka-zookeeper.git cd docker-kafka-zookeeper
- Run the following script
# Here, N = number of kafka & zookeeper nodes to create (default value is 3). . ./restart-all.sh N
- Zookeeper Nodes are available at
ZK-1 Accepting Requests at 0.0.0.0:2181 ZK-2 Accepting Requests at 0.0.0.0:2182 ZK-3 Accepting Requests at 0.0.0.0:2183 ...
- Broker Nodes are available at
Broker-1 Accepting Requests at 0.0.0.0:9092 Broker-2 Accepting Requests at 0.0.0.0:9093 Broker-3 Accepting Requests at 0.0.0.0:9094 ...
-
The zk-services.sh starts zookeeper on
N
nodes.testbed-1 - Start Native zkServer PID: 65
-
The zk-health.sh checks zookeeper status.
testbed-1 - ZOOKEEPER HEALTH STATUS: imok Zookeeper version: 3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT Latency min/avg/max: 0/0/0 Received: 2 Sent: 1 Connections: 1 Outstanding: 0 Zxid: 0x100000000 Mode: follower Node count: 4
-
The kafka-services.sh starts kafka on the same
N
nodes.testbed-1 - Start Kafka Server PID: 1488
-
The kafka-health.sh checks for kafka broker ids in zookeeper.
testbed-1 - KAFKA HEALTH STATUS /brokers/ids/2 /brokers/ids/1 /brokers/ids/0
-
The kafka-benchmarks.sh will create some topics and run a few benchmark tests.
-
Create and list all the topics -
testbed-1 - KAFKA TOPICS: test-topic-rep-3 test-topic-rep-one topic-testbed-1
-
Kafka producer performance test -
PRODUCER PROCESS @ testbed-1 -
3-thread, async 3 times replication, no compression 15000000 records sent, 63406.716039 records/sec (6.05 MB/sec), 8642.51 ms avg latency, 29022.00 ms max latency, 64 ms 50th, 9341 ms 95th, 12387 ms 99th, 13204 ms 99.9th.
1-thread, no replication, no compression 15000000 records sent, 130646.088456 records/sec (12.46 MB/sec), 3887.82 ms avg latency, 7980.00 ms max latency, 3749 ms 50th, 6582 ms 95th, 7583 ms 99th, 7913 ms 99.9th.
-
Kafka consumer performance test -
CONSUMER PROCESS @ testbed-3 -
1-thread, no replication, no compression start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec 2017-12-12 22:45:02:571, 2017-12-12 22:45:29:852, 1430.5115, 52.4362, 15000000, 549833.2173
-
- Yahoo Kafka Manager can be accessed on browser at http://CONTAINER-IP:9000/
- List of Clusters
- Summary of a Cluster
- List of Brokers
- List of Topics
- List of Consumers
Docker version 17.06.0-ce
Ubuntu Trusty 14.04 Host OS
Eclipse IDE for Java EE Developers Oxygen (4.7.0)
Eclipse Docker Tooling 3.1.0