Kafka Zookeeper Docker

Standalone in Docker

docker run -d --name zoo1 --restart=always -v /etc/localtime:/etc/localtime:ro -p 2181:2181 zookeeper:3.5
docker run -d --name kafka1 --restart=always -v /etc/localtime:/etc/localtime:ro -p 9092:9092 -e KAFKA_BROKER_ID=0 -e KAFKA_ZOOKEEPER_CONNECT=zoo1:2181 --link zoo1 -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka_ip:9092 -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 -e JMX_PORT=1099 -t wurstmeister/kafka

Standalone in Docker Compose

zookeeper-kafka-standalone.yml

kafka-topics.sh --create --bootstrap-server 127.0.0.1:9092 --replication-factor 1 --partitions 1 --topic kafkazookeeper
kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic kafkazookeeper
kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --topic kafkazookeeper --from-beginning
kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --topic kafkazookeeper --group kafkazookeepergroup --from-beginning

kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --list
kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --topic kafkazookeeper --describe
kafka-consumer-groups.sh --bootstrap-server 127.0.0.1:9092 --list
kafka-consumer-groups.sh --bootstrap-server 127.0.0.1:9092 --group kafkazookeepergroup --describe
kafka-consumer-groups.sh --bootstrap-server 127.0.0.1:9092 --group kafkazookeepergroup --describe --members
kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 127.0.0.1:9092 --topic kafkazookeeper
kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --delete --topic kafkazookeeper
# bin/kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --delete --topic kafkazookeeper.*

kafka-topics.sh --version
kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --topic __consumer_offsets --describe
kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --topic kafkazookeeper --partitions 2 --alter
kafka-reassign-partitions.sh --bootstrap-server 127.0.0.1:9092 --reassignment-json-file replica.json --execute

vi replica.json

{
    "version":1,
    "partitions":[
        {"topic":"kafkazookeeper","partition":0,"replicas":[1,2,3]},
        {"topic":"kafkazookeeper","partition":1,"replicas":[1,2,3]},
        {"topic":"kafkazookeeper","partition":2,"replicas":[1,2,3]}
    ]
}

Cluster in Docker Compose

zookeeper-kafka.yml

kafka-topics.sh --create --zookeeper 172.18.0.201:2181,172.18.0.202:2181,172.18.0.203:2181 --replication-factor 1 --partitions 1 --topic kafkazookeeper
kafka-topics.sh --zookeeper 172.18.0.201:2181,172.18.0.202:2181,172.18.0.203:2181 --list

Tests

zkCli.sh -server localhost:2181
ls /brokers/ids
ls /controller
ls /brokers/topics/__consumer_offsets/partitions/0/state

Runtime Environment

Tips

java.rmi.server.ExportException: Port already in use: 1099

unset JMX_PORT;

UnrecognizedOptionException: bootstrap-server is not a recognized option

The bin/kafka-topics.sh command line tool is now able to connect directly to brokers with --bootstrap-server instead of zookeeper. The old --zookeeper option is still available for now.

# Kafka 2.2.X
bin/kafka-topics.sh --list --bootstrap-server localhost:9092
# Kafka 2.1.X
bin/kafka-topics.sh --list --zookeeper localhost:2181

Kafka 2.3.0

There have been several improvements to the Kafka Connect REST API

Kafka 3.0.0

The deprecation of support for Java 8 and Scala 2.12

Kafka 3.1.0

Apache Kafka supports Java 17

Introduction

https://kafka.apache.org/images/streams-and-tables-p1_p4.png