How to install Kafka

Posted on by admin

You can find Apache Kafka download servers as below;

For older/all versions

https://archive.apache.org/dist/kafka

For current/recent versions

https://downloads.apache.org/kafka

Setup Java with ASDF

# Install asdf version manager for multiple runtime 
brew install asdf

# Install wget command-line download tool
brew install wget
# Add Java plugin to ASDF
asdf plugin add java

# Install Amazon Corretto Java 8.452.09.1
asdf install java corretto-8.452.09.1

# Install AdoptOpenJDK Java 24.0.1+9
asdf install java adoptopenjdk-24.0.1+9



# Set global Java version to AdoptOpenJDK 24.0.1+9
asdf set -u java adoptopenjdk-24.0.1+9

# Refresh ASDF shims for all tools(In case terminal can't find java)
asdf reshim

Setup Kafka

mkdir kafka

cd kafka

Setup Kafka 1.1.0 with Java 8

wget "https://archive.apache.org/dist/kafka/1.1.0/kafka_2.11-1.1.0.tgz"

tar -xzf kafka_2.11-1.1.0.tgz

cd kafka_2.11-1.1.0

# Set local Java version to corretto-8.452.09.1(Amazon)
asdf set java corretto-8.452.09.1

# Start Zookeeper in background mode
bin/zookeeper-server-start.sh -daemon config/zookeeper.properties

# or Start Zookeeper in foreground mode (see logs in terminal)
bin/zookeeper-server-start.sh config/zookeeper.properties

# Start Kafka Server in background mode
bin/kafka-server-start.sh -daemon config/server.properties

# Start Kafka Server in background mode
bin/kafka-server-start.sh config/server.properties

# Check if Zookeeper is running
jps -l | grep zookeeper

# Check if Kafka Server is running
jps -l | grep kafka

# Stop Kafka Server
bin/kafka-server-stop.sh

# Stop Zookeeper
bin/zookeeper-server-stop.sh

# Kill whatever is on port 9092 (Kafka)
kill -9 $(lsof -t -i:9092)

# Kill whatever is on port 2181 (Zookeeper)
kill -9 $(lsof -t -i:2181)

Create second Kafka Server(Broker)

# Step 1: Create a copy of the server properties for the second broker
cp config/server.properties config/server-1.properties

# Step 2: Modify the second broker's configuration
# You need to change these properties in server-1.properties:
# - broker.id (must be unique)
# - port (must be different from first broker)
# - log.dirs (must be different to avoid conflicts)

# Using sed to modify the properties
sed -i '' 's/broker.id=0/broker.id=1/' config/server-1.properties
sed -i '' 's/#listeners=PLAINTEXT:\/\/:9092/listeners=PLAINTEXT:\/\/:9093/' config/server-1.properties
sed -i '' 's/log.dirs=\/tmp\/kafka-logs/log.dirs=\/tmp\/kafka-logs-1/' config/server-1.properties

# Step 3: Start the second Kafka server(broker)
echo "Starting Kafka Broker 1 on port 9093..."
bin/kafka-server-start.sh -daemon config/server-1.properties

Create topic

# Basic topic creation
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic my-topic

Verify topic creation

# List all topics
bin/kafka-topics.sh --list --zookeeper localhost:2181

# Describe topic details
bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-topic

Send some messages to Kafka

# Send single message
echo "Hello Kafka" | bin/kafka-console-producer.sh --broker-list localhost:9092 --topic my-topic

# Send multiple lines using heredoc
cat << EOF | bin/kafka-console-producer.sh --broker-list localhost:9092 --topic my-topic
Hello Kafka
This is message 2
Message 3
Final message
EOF

Consume messages from Kafka

# Consume from the beginning of the topic
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my-topic --from-beginning

# Consume only new messages (from current point forward)
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my-topic

# If you have multiple brokers
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092,localhost:9093 --topic my-topic --from-beginning

# Consume as part of a consumer group
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my-topic --group my-consumer-group

# Check consumer group status
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-consumer-group

# Consume only 10 messages then exit
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my-topic --max-messages 10

# Consume only from partition 0
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my-topic --partition 0 --from-beginning

Setup Kafka 3.7.2 with Java 8

wget "https://archive.apache.org/dist/kafka/3.7.2/kafka_2.13-3.7.2.tgz"

tar -xzf kafka_2.13-3.7.2.tgz

cd kafka_2.13-3.7.2

bin/zookeeper-server-start.sh config/zookeeper.properties

bin/kafka-server-start.sh config/server.properties

Starting Zookeeper and kafka

bin/zookeeper-server-start.sh config/zookeeper.properties
bin/kafka-server-start.sh config/server.properties

Create topic

bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test

List topics

bin/kafka-topics.sh --list --bootstrap-server localhost:9092

or

bin/kafka-topics.sh --list --zookeeper localhost:2181

Send some messages to Kafka

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test

or

bin/kafka-console-producer.sh --zookeeper localhost:2181 --topic test

Consume messages from Kafka

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning

Leave a Reply