Add support for kafka 3 with zookeeper support (adding images for 3.0.1, 3.1.1 and 3.2.0) #709

lunarfs · 2022-05-12T15:25:49Z

What is it good for
Support for Kafka 3.0.1, 3.1.1 and 3.2.0 with the regular ZooKeeper setup.
What I did
@Jack12816 did some approach of getting to kafka 3.0.0 I have worked with his commits as basis for getting to 3.0.1,2.1.1 and 3.2.0
the PR is inspired by Add support for kafka 2.8.1 #688 and Add support for Kafka 3.0.0. #691 I had to change some Kafka connection options (--zookeeper is deprecated since multiple versions, and since Kafka 3.0 it was removed -- replaced by the --bootstrap-server option. I did a fix for some tests for the new version/outputs
I have added code to report the no longer supported KAFKA_ADVERTISED_HOST_NAME KAFKA_ADVERTISED_PORT and KAFKA_PORT and KAFKA_HOST_NAME (see KAFKA-12945: Remove port, host.name and related configs in 3.0 apache/kafka#10872 for more information).
I have updated README.md to reflect that < 3.0 no longer supports KAFKA_ADVERTISED_HOST_NAME etc.
So i have done some changes to start-kafka.sh to reflect the removed options.. it is a syntactic nightmare of if then else.. and there might be some edge cases i have not yet covered.. feel free to comment
A picture of a cute animal (not mandatory but encouraged)

# Conflicts: # .travis.yml # CHANGELOG.md # Dockerfile # README.md # test/0.10/test.create-topics.kafka.sh # test/docker-compose.yml

jimbogithub · 2022-05-12T21:06:58Z

@lunarfs Like #707 I'm a +1 for adding 3.0.1 support (3.1.0 doesn't have the bug fix I need). Could you add that as well?

lunarfs · 2022-05-13T08:56:04Z

@lunarfs Like #707 I'm a +1 for adding 3.0.1 support (3.1.0 doesn't have the bug fix I need). Could you add that as well?

I would be happy to supply 3.0.1 support, but I think I would like @wurstmeister to comment a bit on the format of how the approach to the 3.x is done in this PR, is it ok to do it like this or should we do some other approach with the tests. are there new corner cases we need to test.

lunarfs · 2022-05-18T12:48:43Z

@lunarfs Like #707 I'm a +1 for adding 3.0.1 support (3.1.0 doesn't have the bug fix I need). Could you add that as well?

added support for 3.0.1, 3.1.1 and 3.2.0

create-topics.sh

start-kafka.sh

test/0.0/test.start-kafka-bug-312-kafka-env.kafka.sh

test/0.0/test.start-kafka-bug-313-kafka-opts.kafka.sh

… KAFKA_ADVERTISED_PORT KAFKA_PORT and KAFKA_HOST_NAME

test/scenarios/jmx/docker-compose-kafka3.yml

test/scenarios/jmx/docker-compose-kafka.yml

…versions

jimbogithub · 2022-06-29T03:12:50Z

@lunarfs I'm testing this out in my local environment. Looks pretty good except for a couple of issues if I have KAFKA_CREATE_TOPICS enabled:

KAFKA_PORT is used by create-topics.sh in order to poll for Kafka to be ready, hence I've had to patch out the startup reject of this setting.
create-topics.sh also requires BROKER_LIST for the 3.x style CONNECT_OPTS. This is easily added to the docker-compose environment, so probably just needs to be documented.

So far so good after making those changes.

jimbogithub · 2022-06-29T03:18:14Z

@lunarfs I'm testing this out in my local environment. Looks pretty good except for a couple of issues if I have KAFKA_CREATE_TOPICS enabled:

KAFKA_PORT is used by create-topics.sh in order to poll for Kafka to be ready, hence I've had to patch out the startup reject of this setting.

create-topics.sh also requires BROKER_LIST for the 3.x style CONNECT_OPTS. This is easily added to the docker-compose environment, so probably just needs to be documented.

So far so good after making those changes.

Maybe BROKER_LIST could be examined to determine the port instead of re-enabling KAFKA_PORT?

…s alos where the internal-broker-list.sh is

…eflect this in tests, where it was not propperly mocked

lunarfs · 2022-06-30T10:11:31Z

@lunarfs I'm testing this out in my local environment. Looks pretty good except for a couple of issues if I have KAFKA_CREATE_TOPICS enabled:

KAFKA_PORT is used by create-topics.sh in order to poll for Kafka to be ready, hence I've had to patch out the startup reject of this setting.

create-topics.sh also requires BROKER_LIST for the 3.x style CONNECT_OPTS. This is easily added to the docker-compose environment, so probably just needs to be documented.

So far so good after making those changes.

Maybe BROKER_LIST could be examined to determine the port instead of re-enabling KAFKA_PORT?

@jimbogithub thanks, this was a nice catch, I did not properly modify the tests to catch this scenario.. I have modified tests such that i could capture the scenario. I think the BROKER_LIST is not a valid source for this, but the KAFKA_LISTNERS has the internal port captured, so getting it from there should be fine I think.. and then we also do not need to give extra params in the compose file as KAFKA_LISTNERS is already available in create-topics.sh. For the test container however BROKER_LIST is the better option, so if KAFKA_LISTNERS is not available we will use BROKER_LIST.

jimbogithub · 2022-07-15T04:32:03Z

The topic creation is still problematic. It's not working with the example config:

HOSTNAME_COMMAND: curl http://169.254.169.254/latest/meta-data/public-hostname
KAFKA_ADVERTISED_LISTENERS: INSIDE://:9092,OUTSIDE://_{HOSTNAME_COMMAND}:9094
KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:9094
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE

I use very similar config myself and it results in:

kafka0_1     | Exception in thread "main" org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
kafka0_1     | 	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:538)
kafka0_1     | 	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:470)
kafka0_1     | 	at org.apache.kafka.clients.admin.Admin.create(Admin.java:133)
kafka0_1     | 	at kafka.admin.TopicCommand$TopicService$.createAdminClient(TopicCommand.scala:205)
kafka0_1     | 	at kafka.admin.TopicCommand$TopicService$.apply(TopicCommand.scala:209)
kafka0_1     | 	at kafka.admin.TopicCommand$.main(TopicCommand.scala:50)
kafka0_1     | 	at kafka.admin.TopicCommand.main(TopicCommand.scala)
kafka0_1     | Caused by: org.apache.kafka.common.config.ConfigException: Invalid url in bootstrap.servers: OUTSIDE
kafka0_1     | 	at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:59)
kafka0_1     | 	at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:48)
kafka0_1     | 	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:490)

Maybe prefer BROKER_LIST when available so that it can always be overridden?
Or look for KAFKA_INTER_BROKER_LISTENER_NAME to help find the internal address in the KAFKA_LISTENERS?

dswiecki · 2022-07-22T07:53:53Z

@lunarfs @jimbogithub are there available any preview docker images of this change?

jimbogithub · 2022-07-26T19:42:36Z

@dswiecki I have an image at jimbodock/kafka:2.13-3.1.1. It is slightly stripped down (removed the embedded Docker which is only needed for the CI tests). It is based on the build at the time of #709 (comment) so supports KAFKA_PORT and BROKER_LIST to be able to create topics.

mattathompson · 2022-08-03T17:19:00Z

🙏 Excited for this.

lunarfs · 2022-08-31T13:03:17Z

The topic creation is still problematic. It's not working with the example config:

HOSTNAME_COMMAND: curl http://169.254.169.254/latest/meta-data/public-hostname
KAFKA_ADVERTISED_LISTENERS: INSIDE://:9092,OUTSIDE://_{HOSTNAME_COMMAND}:9094
KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:9094
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE

I use very similar config myself and it results in:

kafka0_1     | Exception in thread "main" org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
kafka0_1     | 	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:538)
kafka0_1     | 	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:470)
kafka0_1     | 	at org.apache.kafka.clients.admin.Admin.create(Admin.java:133)
kafka0_1     | 	at kafka.admin.TopicCommand$TopicService$.createAdminClient(TopicCommand.scala:205)
kafka0_1     | 	at kafka.admin.TopicCommand$TopicService$.apply(TopicCommand.scala:209)
kafka0_1     | 	at kafka.admin.TopicCommand$.main(TopicCommand.scala:50)
kafka0_1     | 	at kafka.admin.TopicCommand.main(TopicCommand.scala)
kafka0_1     | Caused by: org.apache.kafka.common.config.ConfigException: Invalid url in bootstrap.servers: OUTSIDE
kafka0_1     | 	at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:59)
kafka0_1     | 	at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:48)
kafka0_1     | 	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:490)

Maybe prefer BROKER_LIST when available so that it can always be overridden? Or look for KAFKA_INTER_BROKER_LISTENER_NAME to help find the internal address in the KAFKA_LISTENERS?

OK, I will take a look at this one of the followin days

jozefbarcin · 2022-10-06T13:47:44Z

🙏 Excited for this.

nise-wg2 · 2022-10-12T09:16:43Z

🙏 Looking forward to this!

jimbogithub · 2022-12-05T19:26:45Z

The topic creation is still problematic. It's not working with the example config:

HOSTNAME_COMMAND: curl http://169.254.169.254/latest/meta-data/public-hostname
KAFKA_ADVERTISED_LISTENERS: INSIDE://:9092,OUTSIDE://_{HOSTNAME_COMMAND}:9094
KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:9094
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE

I use very similar config myself and it results in:

kafka0_1     | Exception in thread "main" org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
kafka0_1     | 	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:538)
kafka0_1     | 	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:470)
kafka0_1     | 	at org.apache.kafka.clients.admin.Admin.create(Admin.java:133)
kafka0_1     | 	at kafka.admin.TopicCommand$TopicService$.createAdminClient(TopicCommand.scala:205)
kafka0_1     | 	at kafka.admin.TopicCommand$TopicService$.apply(TopicCommand.scala:209)
kafka0_1     | 	at kafka.admin.TopicCommand$.main(TopicCommand.scala:50)
kafka0_1     | 	at kafka.admin.TopicCommand.main(TopicCommand.scala)
kafka0_1     | Caused by: org.apache.kafka.common.config.ConfigException: Invalid url in bootstrap.servers: OUTSIDE
kafka0_1     | 	at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:59)
kafka0_1     | 	at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:48)
kafka0_1     | 	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:490)

Maybe prefer BROKER_LIST when available so that it can always be overridden? Or look for KAFKA_INTER_BROKER_LISTENER_NAME to help find the internal address in the KAFKA_LISTENERS?

OK, I will take a look at this one of the followin days

@lunarfs I've added a PR to your PR that resolves the above by simply making the KAFKA_LISTENERS parsing smarter. If you can ripple that through to here then I suspect this is good to go.

jimbogithub · 2022-12-05T19:36:57Z

Preview available here: jimbodock/kafka:2.13-3.3.1. (with the embedded docker cli stripped out)

jozefbarcin · 2023-01-04T15:06:25Z

Preview available here: jimbodock/kafka:2.13-3.3.1. (with the embedded docker cli stripped out)

Works fine for me

Changed base image to eclipse-temurin

frank-vissing-se added 4 commits May 11, 2022 15:46

Jack12816 initial 3.0 approach

ac64f78

Merge remote-tracking branch 'origin/master' into going_to_3

07b56eb

# Conflicts: # .travis.yml # CHANGELOG.md # Dockerfile # README.md # test/0.10/test.create-topics.kafka.sh # test/docker-compose.yml

make test independent on how sort works

a576cbf

add support for kafka 3.1.0

f267754

lunarfs changed the title ~~Going to 31~~ Add support for kafka 3.1.0 May 12, 2022

OneCricketeer mentioned this pull request May 12, 2022

support latest kafka 3.0.1, 3.1.0 #707

Open

frank-vissing-se added 2 commits May 18, 2022 14:36

add newer and better versions

5d7a593

make sure user removes obsolete config

8be81b0

lunarfs changed the title ~~Add support for kafka 3.1.0~~ Add support for kafka 3.2.0 May 18, 2022

OneCricketeer reviewed May 18, 2022

View reviewed changes

create-topics.sh Outdated Show resolved Hide resolved

start-kafka.sh Outdated Show resolved Hide resolved

test/0.0/test.start-kafka-bug-312-kafka-env.kafka.sh Outdated Show resolved Hide resolved

test/0.0/test.start-kafka-bug-313-kafka-opts.kafka.sh Outdated Show resolved Hide resolved

frank-vissing-se added 15 commits May 18, 2022 15:16

support 3.x in compose

ecef794

fix typo

c527cb5

fix missing $ (typo)

289836a

re add BROKER_LIST, propper default KAFKA_VERSION

426638f

make shell test happy

823e024

propper default

c3c455f

do not require BROKER_LIST

57af850

refactor removed arguments for kafka

750ebb1

start kafka with propper options

01fa192

supply entire list of bootstrap servers

fe19311

various test cleanup to support removal of KAFKA_ADVERTISED_HOST_NAME…

8a8daf4

… KAFKA_ADVERTISED_PORT KAFKA_PORT and KAFKA_HOST_NAME

fix shell check

8914a83

make MAJOR_VERSION accessible for test scripts

724866b

propper naming of versions.sh

a97e332

fix selection of kafka args

8d80440

OneCricketeer reviewed May 20, 2022

View reviewed changes

test/scenarios/jmx/docker-compose-kafka3.yml Outdated Show resolved Hide resolved

OneCricketeer reviewed May 20, 2022

View reviewed changes

test/scenarios/jmx/docker-compose-kafka.yml Outdated Show resolved Hide resolved

frank-vissing-se added 4 commits May 20, 2022 19:30

revievers comment about wrong hostname, changing to propper name kafka

13933fd

reviewers comment about not needing seperate test setup for previous …

14f089d

…versions

trigger build

b3e54b0

no need to specify compose file

704e9ab

lunarfs changed the title ~~Add support for kafka 3.2.0~~ Add support for kafka 3 with zookeeper support (adding images for 3.0.1, 3.1.1 and 3.2.0) May 23, 2022

frank-vissing-se added 2 commits May 23, 2022 14:41

totally revert this file

0698081

trigger build

58dfb8f

frank-vissing-se added 2 commits June 30, 2022 11:30

typo in readme, we are in the test folder when executing this, this i…

8922faa

…s alos where the internal-broker-list.sh is

examine KAFKA_LISTNERS to determine the port we are runing on, also r…

2abb7db

…eflect this in tests, where it was not propperly mocked

frank-vissing-se added 2 commits June 30, 2022 12:16

Double quote to prevent globbing

001df7d

make create topics work both for KAFKA_LISTNERS and BROKER_LIST

722fc5c

StephenSorriaux mentioned this pull request Mar 10, 2023

Add 3.X images StephenSorriaux/kafka-docker#1

Merged

sharninder and others added 2 commits April 17, 2023 11:36

changed image

19d8c52

Merge pull request #3 from sharninder/upgrade-image

3b705db

Changed base image to eclipse-temurin

andrewchoi5 approved these changes May 31, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for kafka 3 with zookeeper support (adding images for 3.0.1, 3.1.1 and 3.2.0) #709

Add support for kafka 3 with zookeeper support (adding images for 3.0.1, 3.1.1 and 3.2.0) #709

lunarfs commented May 12, 2022 •

edited

Loading

jimbogithub commented May 12, 2022

lunarfs commented May 13, 2022

lunarfs commented May 18, 2022

jimbogithub commented Jun 29, 2022

jimbogithub commented Jun 29, 2022

lunarfs commented Jun 30, 2022 •

edited

Loading

jimbogithub commented Jul 15, 2022

dswiecki commented Jul 22, 2022

jimbogithub commented Jul 26, 2022

mattathompson commented Aug 3, 2022

lunarfs commented Aug 31, 2022

jozefbarcin commented Oct 6, 2022 •

edited

Loading

nise-wg2 commented Oct 12, 2022

jimbogithub commented Dec 5, 2022

jimbogithub commented Dec 5, 2022 •

edited

Loading

jozefbarcin commented Jan 4, 2023

Add support for kafka 3 with zookeeper support (adding images for 3.0.1, 3.1.1 and 3.2.0) #709

Are you sure you want to change the base?

Add support for kafka 3 with zookeeper support (adding images for 3.0.1, 3.1.1 and 3.2.0) #709

Conversation

lunarfs commented May 12, 2022 • edited Loading

jimbogithub commented May 12, 2022

lunarfs commented May 13, 2022

lunarfs commented May 18, 2022

jimbogithub commented Jun 29, 2022

jimbogithub commented Jun 29, 2022

lunarfs commented Jun 30, 2022 • edited Loading

jimbogithub commented Jul 15, 2022

dswiecki commented Jul 22, 2022

jimbogithub commented Jul 26, 2022

mattathompson commented Aug 3, 2022

lunarfs commented Aug 31, 2022

jozefbarcin commented Oct 6, 2022 • edited Loading

nise-wg2 commented Oct 12, 2022

jimbogithub commented Dec 5, 2022

jimbogithub commented Dec 5, 2022 • edited Loading

jozefbarcin commented Jan 4, 2023

lunarfs commented May 12, 2022 •

edited

Loading

lunarfs commented Jun 30, 2022 •

edited

Loading

jozefbarcin commented Oct 6, 2022 •

edited

Loading

jimbogithub commented Dec 5, 2022 •

edited

Loading