What is Apache Kafka?
LinkedIn began designing Kafka in 2010 and it was subsequently open-sourced in early 2011.

The platform was created to address LinkedIn's growing needs for real-time data processing and to overcome the limitations of their existing systems.

Kafka is a distributed streaming platform comprised of multiple servers (brokers) that work together to store, process, and transmit large volumes of real-time messages.

For someone unfamiliar with Kafka jargon, Kafka is a system made up of multiple servers that can store and send messages immediately as they arrive.

What is development language?
Apache Kafka is primarily developed in Scala and Java. While its core components are written in Scala, client APIs and many other parts of Kafka are written in Java.

Core Concepts:
– Producer: Sends messages to Kafka.
– Consumer: Reads messages from Kafka.
– Topic: A category or feed name to which messages are stored and published, similar to a table in database.
– Broker/Server: A single Kafka server that stores data and serves client requests.
– Cluster: A group of brokers working together.
– Partition: Topics are split into partitions, which are the basic unit of parallelism in Kafka.
– Offset: A unique identifier for each record within a partition.

Notes:

Messages in Kafka are categorized into topics. The closest analogies for a topic are a database table or a folder in a filesystem. Topics are additionally broken down into a number of partitions.

The term stream is often used when discussing data within systems like Kafka. Most often, a stream is considered to be a single topic of data, regardless of the number of partitions. This represents a single stream of data moving from the producers to the consumers.

Producers -->                           KAFKA CLUSTER       <-- Consumers/Consumer Groups
+---------------------------------------------------------------------------------------+
|                   Broker 1                 |                 Broker 2                 |
|                                            |                                          |
| +----------------------------------------+ | +--------------------------------------+ |
| |               Topic: Orders_Placed     | | |         Topic: Orders_Placed         | |
| +----------------------------------------+ | +--------------------------------------+ |
| | Partition 1 (Leader)                   | | | Partition 1 (Follower)               | |
| | - orderID: 123456                      | | | - orderID: 123456 (replica)          | |
| | - orderID: 123457                      | | | - orderID: 123457 (replica)          | |
| +----------------------------------------+ | +--------------------------------------+ |
| | Partition 2 (Follower)                 | | | Partition 2 (Leader)                 | |
| | - orderID: 123459 (replica)            | | | - orderID: 123459                    | |
| | - orderID: 123460 (replica)            | | | - orderID: 123460                    | |
| +----------------------------------------+ | +--------------------------------------+ |
|                                            |                                          |
| +----------------------------------------+ | +--------------------------------------+ |
| |                Topic: Payments         | | |           Topic: Payments            | |
| +----------------------------------------+ | +--------------------------------------+ |
| | Partition 1 (Follower)                 | | | Partition 1 (Leader)                 | |
| | - paymentID: PAY00123 (replica)        | | | - paymentID: PAY00123                | |
| | - paymentID: PAY00124 (replica)        | | | - paymentID: PAY00124                | |
| +----------------------------------------+ | +--------------------------------------+ |
| | Partition 2 (Leader)                   | | | Partition 2 (Follower)               | |
| | - paymentID: PAY00125                  | | | - paymentID: PAY00125 (replica)      | |
| | - paymentID: PAY00126                  | | | - paymentID: PAY00126 (replica)      | |
| +----------------------------------------+ | +--------------------------------------+ |
+---------------------------------------------------------------------------------------+
                        ^                                             ^
                        |                                             |
                        v                                             v
                        ------------  ZooKeeper Cluster  --------------

Producers -->                                           KAFKA CLUSTER                           <-- Consumers/Consumer Groups
+---------------------------------------------------------------------------------------------------------------------------+
|               Broker 1                 |                 Broker 2                 |                     Broker 3          |
|                                        |                                          |                                       |
| +------------------------------------+ | +--------------------------------------+ | +-----------------------------------+ |
| |           Topic: Orders_Placed     | | |         Topic: Orders_Placed         | | |      Topic: Orders_Placed         | |
| +------------------------------------+ | +--------------------------------------+ | +-----------------------------------+ |
| | Partition 1 (Leader)               | | | Partition 1 (Follower)               | | | Partition 1 (Replica)             | |
| | - orderID: 123456                  | | | - orderID: 123456 (replica)          | | | - orderID: 123456 (replica)       | |
| | - orderID: 123457                  | | | - orderID: 123457 (replica)          | | | - orderID: 123457 (replica)       | |
| +------------------------------------+ | +--------------------------------------+ | +-----------------------------------+ |
| | Partition 2 (Follower)             | | | Partition 2 (Leader)                 | | | Partition 2 (Replica)             | |
| | - orderID: 123459 (replica)        | | | - orderID: 123459                    | | | - orderID: 123459 (replica)       | |
| | - orderID: 123460 (replica)        | | | - orderID: 123460                    | | | - orderID: 123460 (replica)       | |
| +------------------------------------+ | +--------------------------------------+ | +-----------------------------------+ |
|                                        |                                          |                                       |
| +------------------------------------+ | +--------------------------------------+ | +-----------------------------------+ |
| |            Topic: Payments         | | |           Topic: Payments            | | |       Topic: Payments             | |
| +------------------------------------+ | +--------------------------------------+ | +-----------------------------------+ |
| | Partition 1 (Follower)             | | | Partition 1 (Leader)                 | | | Partition 1 (Replica)             | |
| | - paymentID: PAY00123 (replica)    | | | - paymentID: PAY00123                | | | - paymentID: PAY00123 (replica)   | |
| | - paymentID: PAY00124 (replica)    | | | - paymentID: PAY00124                | | | - paymentID: PAY00124 (replica)   | |
| +------------------------------------+ | +--------------------------------------+ | +-----------------------------------+ |
| | Partition 2 (Leader)               | | | Partition 2 (Follower)               | | | Partition 2 (Replica)             | |
| | - paymentID: PAY00125              | | | - paymentID: PAY00125 (replica)      | | | - paymentID: PAY00125 (replica)   | |
| | - paymentID: PAY00126              | | | - paymentID: PAY00126 (replica)      | | | - paymentID: PAY00126 (replica)   | |
| +------------------------------------+ | +--------------------------------------+ | +-----------------------------------+ |
+---------------------------------------------------------------------------------------------------------------------------+
                            ^                                      ^                                      ^
                            |                                      |                                      |
                            v                                      v                                      v
                            ----------------------------  ZooKeeper Cluster  ------------------------------

No Replication

Producers -->                                                   KAFKA CLUSTER                                            <-- Consumers/Consumer Groups
+----------------------------------------------------------------------------------------------------------------------------------------------------+
|                        Broker 1                 |                  Broker 2                   |                       Broker 3                     |
|                                                 |                                             |                                                    |
| +---------------------------------------------+ | +-----------------------------------------+ | +------------------------------------------------+ |
| |                Topic: Orders                | | |           Topic: Orders                 | | |                  Topic: Payments               | |
| +---------------------------------------------+ | +-----------------------------------------+ | +------------------------------------------------+ |
| | Partition 0 (Leader)                        | | | Partition 1 (Follower)                  | | | Partition 0 (Leader)                           | |
| |                                             | | |                                         | | |                                                | |
| |  +----------------------+                   | | |  +----------------------+               | | |  +----------------------+                      | |
| |  |"id": "O100"          |                   | | |  |"id": "O101"          |               | | |  |"id": "P100"          |                      | |
| |  |"item": "Laptop"      |                   | | |  |"item": "Phone"       |               | | |  |"order_id": "O100"    |                      | |
| |  |"qty": 1              |                   | | |  |"qty": 2              |               | | |  |"amount": 1200        |                      | |
| |  +----------------------+                   | | |  +----------------------+               | | |  +----------------------+                      | |
| |                                             | | |                                         | | |                                                | |
| |  +----------------------+                   | | |  +----------------------+               | | |  +----------------------+                      | |
| |  |"id": "O102"          |                   | | |  |"id": "O103"          |               | | |  |"id": "P101"          |                      | |
| |  |"item": "Tablet"      |                   | | |  |"item": "Headphones"  |               | | |  |"order_id": "O101"    |                      | |
| |  |"qty": 3              |                   | | |  |"qty": 1              |               | | |  |"amount": 300         |                      | |
| |  +----------------------+                   | | |  +----------------------+               | | |  +----------------------+                      | |
| +---------------------------------------------+ | +-----------------------------------------+ | +------------------------------------------------+ |
| |                Topic: Payments              | | |           Topic: Shipments              | | |                 Topic: Shipments               | |
| +---------------------------------------------+ | +-----------------------------------------+ | +------------------------------------------------+ |
| | Partition 1 (Follower)                      | | | Partition 0 (Leader)                    | | | Partition 1 (Follower)                         | |
| |                                             | | |                                         | | |                                                | |
| |  +----------------------+                   | | |  +----------------------+               | | |  +----------------------+                      | |
| |  |"id": "P102"          |                   | | |  |"id": "S100"          |               | | |  |"id": "S101"          |                      | |
| |  |"order_id": "O102"    |                   | | |  |"order_id": "O100"    |               | | |  |"order_id": "O101"    |                      | |
| |  |"amount": 450         |                   | | |  |"status": "Shipped"   |               | | |  |"status": "Packed"    |                      | |
| |  +----------------------+                   | | |  +----------------------+               | | |  +----------------------+                      | |
| |                                             | | |                                         | | |                                                | |
| |  +----------------------+                   | | |  +----------------------+               | | |  +----------------------+                      | |
| |  |"id": "P103"          |                   | | |  |"id": "S102"          |               | | |  |"id": "S103"          |                      | |
| |  |"order_id": "O104"    |                   | | |  |"order_id": "O102"    |               | | |  |"order_id": "O104"    |                      | |
| |  |"amount": 750         |                   | | |  |"status": "Delivered" |               | | |  |"status": "Dispatched"|                      | |
| |  +----------------------+                   | | |  +----------------------+               | | |  +----------------------+                      | |
| +---------------------------------------------+ | +-----------------------------------------+ | +------------------------------------------------+ |
+----------------------------------------------------------------------------------------------------------------------------------------------------+
                                    ^                                                                                  ^
                                    |                                                                                  |
                                    v                                                                                  v
                                    --------------------------------  ZooKeeper Cluster  -------------------------------

No replication

Producers -->                                                       KAFKA CLUSTER                                        <-- Consumers/Consumer Groups
+----------------------------------------------------------------------------------------------------------------------------------------------------+
|                        Broker 1                 |                  Broker 2                   |                       Broker 3                     |
|                                                 |                                             |                                                    |
| +---------------------------------------------+ | +-----------------------------------------+ | +------------------------------------------------+ |
| |                Topic: Orders                | | |           Topic: Orders                 | | |                  Topic: Payments               | |
| +---------------------------------------------+ | +-----------------------------------------+ | +------------------------------------------------+ |
| | Partition 0 (Leader)                        | | | Partition 1 (Follower)                  | | | Partition 0 (Leader)                           | |
| |                                             | | |                                         | | |                                                | |
| |  +--------------------------------------+   | | |  +-----------------------------------+  | | |  +------------------------------------------+  | |
| |  | {                                    |   | | |  | {                                 |  | | |  | {                                        |  | |
| |  |   "id": "O100",                      |   | | |  |   "id": "O101",                   |  | | |  |   "id": "P100",                          |  | |
| |  |   "item": "Laptop",                  |   | | |  |   "item": "Phone",                |  | | |  |   "order_id": "O100",                    |  | |
| |  |   "qty": 1                           |   | | |  |   "qty": 2                        |  | | |  |   "amount": 1200                         |  | |
| |  | }                                    |   | | |  | }                                 |  | | |  | }                                        |  | |
| |  +--------------------------------------+   | | |  +-----------------------------------+  | | |  +------------------------------------------+  | |
| |                                             | | |                                         | | |                                                | |
| |  +--------------------------------------+   | | |  +-----------------------------------+  | | |  +------------------------------------------+  | |
| |  | {                                    |   | | |  | {                                 |  | | |  | {                                        |  | |
| |  |   "id": "O102",                      |   | | |  |   "id": "O103",                   |  | | |  |   "id": "P101",                          |  | |
| |  |   "item": "Tablet",                  |   | | |  |   "item": "Headphones",           |  | | |  |   "order_id": "O101",                    |  | |
| |  |   "qty": 3                           |   | | |  |   "qty": 1                        |  | | |  |   "amount": 300                          |  | |
| |  | }                                    |   | | |  | }                                 |  | | |  | }                                        |  | |
| |  +--------------------------------------+   | | |  +-----------------------------------+  | | |  +------------------------------------------+  | |
| +---------------------------------------------+ | +-----------------------------------------+ | +------------------------------------------------+ |
| |                Topic: Payments              | | |           Topic: Shipments              | | |                 Topic: Shipments               | |
| +---------------------------------------------+ | +-----------------------------------------+ | +------------------------------------------------+ |
| | Partition 1 (Follower)                      | | | Partition 0 (Leader)                    | | | Partition 1 (Follower)                         | |
| |                                             | | |                                         | | |                                                | |
| |  +--------------------------------------+   | | |  +-----------------------------------+  | | |  +------------------------------------------+  | |
| |  | {                                    |   | | |  | {                                 |  | | |  | {                                        |  | |
| |  |   "id": "P102",                      |   | | |  |   "id": "S100",                   |  | | |  |   "id": "S101",                          |  | |
| |  |   "order_id": "O102",                |   | | |  |   "order_id": "O100",             |  | | |  |   "order_id": "O101",                    |  | |
| |  |   "amount": 450                      |   | | |  |   "status": "Shipped",            |  | | |  |   "status": "Packed",                    |  | |
| |  | }                                    |   | | |  |   "date": "2024-02-25"            |  | | |  |   "date": "2024-02-26"                   |  | |
| |  +--------------------------------------+   | | |  | }                                 |  | | |  | }                                        |  | |
| |                                             | | |  +-----------------------------------+  | | |  +------------------------------------------+  | |
| |                                             | | |                                         | | |                                                | |
| |  +--------------------------------------+   | | |  +-----------------------------------+  | | |  +------------------------------------------+  | |
| |  | {                                    |   | | |  | {                                 |  | | |  | {                                        |  | |
| |  |   "id": "P103",                      |   | | |  |   "id": "S102",                   |  | | |  |   "id": "S103",                          |  | |
| |  |   "order_id": "O104",                |   | | |  |   "order_id": "O102",             |  | | |  |   "order_id": "O104",                    |  | |
| |  |   "amount": 750                      |   | | |  |   "status": "Delivered",          |  | | |  |   "status": "Dispatched",                |  | |
| |  | }                                    |   | | |  |   "date": "2024-02-27"            |  | | |  |   "date": "2024-02-28"                   |  | |
| |  +--------------------------------------+   | | |  | }                                 |  | | |  | }                                        |  | |
| |                                             | | |  +-----------------------------------+  | | |  +------------------------------------------+  | |
| +---------------------------------------------+ | +-----------------------------------------+ | +------------------------------------------------+ |
+----------------------------------------------------------------------------------------------------------------------------------------------------+
                                    ^                                                                                  ^
                                    |                                                                                  |
                                    v                                                                                  v
                                    --------------------------------  ZooKeeper Cluster  -------------------------------

One consumer group with four partitions

Continue reading →

You can find Apache Kafka download servers as below;

For older/all versions

https://archive.apache.org/dist/kafka

For current/recent versions

https://downloads.apache.org/kafka

Setup Java with ASDF

# Install asdf version manager for multiple runtime 
brew install asdf

# Install wget command-line download tool
brew install wget

# Add Java plugin to ASDF
asdf plugin add java

# Install Amazon Corretto Java 8.452.09.1
asdf install java corretto-8.452.09.1

# Install AdoptOpenJDK Java 24.0.1+9
asdf install java adoptopenjdk-24.0.1+9



# Set global Java version to AdoptOpenJDK 24.0.1+9
asdf set -u java adoptopenjdk-24.0.1+9

# Refresh ASDF shims for all tools(In case terminal can't find java)
asdf reshim

Setup Kafka

mkdir kafka

cd kafka

Setup Kafka 1.1.0 with Java 8

wget "https://archive.apache.org/dist/kafka/1.1.0/kafka_2.11-1.1.0.tgz"

tar -xzf kafka_2.11-1.1.0.tgz

cd kafka_2.11-1.1.0

# Set local Java version to corretto-8.452.09.1(Amazon)
asdf set java corretto-8.452.09.1

# Start Zookeeper in background mode
bin/zookeeper-server-start.sh -daemon config/zookeeper.properties

# or Start Zookeeper in foreground mode (see logs in terminal)
bin/zookeeper-server-start.sh config/zookeeper.properties

# Start Kafka Server in background mode
bin/kafka-server-start.sh -daemon config/server.properties

# Start Kafka Server in background mode
bin/kafka-server-start.sh config/server.properties

# Check if Zookeeper is running
jps -l | grep zookeeper

# Check if Kafka Server is running
jps -l | grep kafka

# Stop Kafka Server
bin/kafka-server-stop.sh

# Stop Zookeeper
bin/zookeeper-server-stop.sh

# Kill whatever is on port 9092 (Kafka)
kill -9 $(lsof -t -i:9092)

# Kill whatever is on port 2181 (Zookeeper)
kill -9 $(lsof -t -i:2181)

Create second Kafka Server(Broker)

# Step 1: Create a copy of the server properties for the second broker
cp config/server.properties config/server-1.properties

# Step 2: Modify the second broker's configuration
# You need to change these properties in server-1.properties:
# - broker.id (must be unique)
# - port (must be different from first broker)
# - log.dirs (must be different to avoid conflicts)

# Using sed to modify the properties
sed -i '' 's/broker.id=0/broker.id=1/' config/server-1.properties
sed -i '' 's/#listeners=PLAINTEXT:\/\/:9092/listeners=PLAINTEXT:\/\/:9093/' config/server-1.properties
sed -i '' 's/log.dirs=\/tmp\/kafka-logs/log.dirs=\/tmp\/kafka-logs-1/' config/server-1.properties

# Step 3: Start the second Kafka server(broker)
echo "Starting Kafka Broker 1 on port 9093..."
bin/kafka-server-start.sh -daemon config/server-1.properties

Create topic

# Basic topic creation
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic my-topic

Verify topic creation

# List all topics
bin/kafka-topics.sh --list --zookeeper localhost:2181

# Describe topic details
bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-topic

Send some messages to Kafka

# Send single message
echo "Hello Kafka" | bin/kafka-console-producer.sh --broker-list localhost:9092 --topic my-topic

# Send multiple lines using heredoc
cat << EOF | bin/kafka-console-producer.sh --broker-list localhost:9092 --topic my-topic
Hello Kafka
This is message 2
Message 3
Final message
EOF

Consume messages from Kafka

# Consume from the beginning of the topic
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my-topic --from-beginning

# Consume only new messages (from current point forward)
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my-topic

# If you have multiple brokers
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092,localhost:9093 --topic my-topic --from-beginning

# Consume as part of a consumer group
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my-topic --group my-consumer-group

# Check consumer group status
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-consumer-group

# Consume only 10 messages then exit
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my-topic --max-messages 10

# Consume only from partition 0
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my-topic --partition 0 --from-beginning

Setup Kafka 3.7.2 with Java 8

wget "https://archive.apache.org/dist/kafka/3.7.2/kafka_2.13-3.7.2.tgz"

tar -xzf kafka_2.13-3.7.2.tgz

cd kafka_2.13-3.7.2

bin/zookeeper-server-start.sh config/zookeeper.properties

bin/kafka-server-start.sh config/server.properties

Starting Zookeeper and kafka

bin/zookeeper-server-start.sh config/zookeeper.properties

bin/kafka-server-start.sh config/server.properties

Create topic

bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test

List topics

bin/kafka-topics.sh --list --bootstrap-server localhost:9092

bin/kafka-topics.sh --list --zookeeper localhost:2181

Send some messages to Kafka

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test

bin/kafka-console-producer.sh --zookeeper localhost:2181 --topic test

Consume messages from Kafka

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning

Kenan Hançer Blog

Software Adventures

Apache Kafka

Notes about Apache Kafka

No Replication

No replication

One consumer group with four partitions

How to install Kafka

For older/all versions

For current/recent versions

Setup Java with ASDF

Setup Kafka

Setup Kafka 1.1.0 with Java 8

Create second Kafka Server(Broker)

Create topic

Verify topic creation

Send some messages to Kafka

Consume messages from Kafka

Setup Kafka 3.7.2 with Java 8

Starting Zookeeper and kafka

Create topic

List topics

Send some messages to Kafka

Consume messages from Kafka