Notes about Apache Kafka

What is Apache Kafka?
LinkedIn began designing Kafka in 2010 and it was subsequently open-sourced in early 2011.

The platform was created to address LinkedIn's growing needs for real-time data processing and to overcome the limitations of their existing systems.

Kafka is a distributed streaming platform comprised of multiple servers (brokers) that work together to store, process, and transmit large volumes of real-time messages.

For someone unfamiliar with Kafka jargon, Kafka is a system made up of multiple servers that can store and send messages immediately as they arrive.

What is development language?
Apache Kafka is primarily developed in Scala and Java. While its core components are written in Scala, client APIs and many other parts of Kafka are written in Java.

Core Concepts:
Producer: Sends messages to Kafka.
Consumer: Reads messages from Kafka.
Topic: A category or feed name to which messages are stored and published, similar to a table in database.
Broker/Server: A single Kafka server that stores data and serves client requests.
Cluster: A group of brokers working together.
Partition: Topics are split into partitions, which are the basic unit of parallelism in Kafka.
Offset: A unique identifier for each record within a partition.

Notes:

Messages in Kafka are categorized into topics. The closest analogies for a topic are a database table or a folder in a filesystem. Topics are additionally broken down into a number of partitions.

The term stream is often used when discussing data within systems like Kafka. Most often, a stream is considered to be a single topic of data, regardless of the number of partitions. This represents a single stream of data moving from the producers to the consumers.

Producers -->                           KAFKA CLUSTER       <-- Consumers/Consumer Groups
+---------------------------------------------------------------------------------------+
|                   Broker 1                 |                 Broker 2                 |
|                                            |                                          |
| +----------------------------------------+ | +--------------------------------------+ |
| |               Topic: Orders_Placed     | | |         Topic: Orders_Placed         | |
| +----------------------------------------+ | +--------------------------------------+ |
| | Partition 1 (Leader)                   | | | Partition 1 (Follower)               | |
| | - orderID: 123456                      | | | - orderID: 123456 (replica)          | |
| | - orderID: 123457                      | | | - orderID: 123457 (replica)          | |
| +----------------------------------------+ | +--------------------------------------+ |
| | Partition 2 (Follower)                 | | | Partition 2 (Leader)                 | |
| | - orderID: 123459 (replica)            | | | - orderID: 123459                    | |
| | - orderID: 123460 (replica)            | | | - orderID: 123460                    | |
| +----------------------------------------+ | +--------------------------------------+ |
|                                            |                                          |
| +----------------------------------------+ | +--------------------------------------+ |
| |                Topic: Payments         | | |           Topic: Payments            | |
| +----------------------------------------+ | +--------------------------------------+ |
| | Partition 1 (Follower)                 | | | Partition 1 (Leader)                 | |
| | - paymentID: PAY00123 (replica)        | | | - paymentID: PAY00123                | |
| | - paymentID: PAY00124 (replica)        | | | - paymentID: PAY00124                | |
| +----------------------------------------+ | +--------------------------------------+ |
| | Partition 2 (Leader)                   | | | Partition 2 (Follower)               | |
| | - paymentID: PAY00125                  | | | - paymentID: PAY00125 (replica)      | |
| | - paymentID: PAY00126                  | | | - paymentID: PAY00126 (replica)      | |
| +----------------------------------------+ | +--------------------------------------+ |
+---------------------------------------------------------------------------------------+
                        ^                                             ^
                        |                                             |
                        v                                             v
                        ------------  ZooKeeper Cluster  --------------
Producers -->                                           KAFKA CLUSTER                           <-- Consumers/Consumer Groups
+---------------------------------------------------------------------------------------------------------------------------+
|               Broker 1                 |                 Broker 2                 |                     Broker 3          |
|                                        |                                          |                                       |
| +------------------------------------+ | +--------------------------------------+ | +-----------------------------------+ |
| |           Topic: Orders_Placed     | | |         Topic: Orders_Placed         | | |      Topic: Orders_Placed         | |
| +------------------------------------+ | +--------------------------------------+ | +-----------------------------------+ |
| | Partition 1 (Leader)               | | | Partition 1 (Follower)               | | | Partition 1 (Replica)             | |
| | - orderID: 123456                  | | | - orderID: 123456 (replica)          | | | - orderID: 123456 (replica)       | |
| | - orderID: 123457                  | | | - orderID: 123457 (replica)          | | | - orderID: 123457 (replica)       | |
| +------------------------------------+ | +--------------------------------------+ | +-----------------------------------+ |
| | Partition 2 (Follower)             | | | Partition 2 (Leader)                 | | | Partition 2 (Replica)             | |
| | - orderID: 123459 (replica)        | | | - orderID: 123459                    | | | - orderID: 123459 (replica)       | |
| | - orderID: 123460 (replica)        | | | - orderID: 123460                    | | | - orderID: 123460 (replica)       | |
| +------------------------------------+ | +--------------------------------------+ | +-----------------------------------+ |
|                                        |                                          |                                       |
| +------------------------------------+ | +--------------------------------------+ | +-----------------------------------+ |
| |            Topic: Payments         | | |           Topic: Payments            | | |       Topic: Payments             | |
| +------------------------------------+ | +--------------------------------------+ | +-----------------------------------+ |
| | Partition 1 (Follower)             | | | Partition 1 (Leader)                 | | | Partition 1 (Replica)             | |
| | - paymentID: PAY00123 (replica)    | | | - paymentID: PAY00123                | | | - paymentID: PAY00123 (replica)   | |
| | - paymentID: PAY00124 (replica)    | | | - paymentID: PAY00124                | | | - paymentID: PAY00124 (replica)   | |
| +------------------------------------+ | +--------------------------------------+ | +-----------------------------------+ |
| | Partition 2 (Leader)               | | | Partition 2 (Follower)               | | | Partition 2 (Replica)             | |
| | - paymentID: PAY00125              | | | - paymentID: PAY00125 (replica)      | | | - paymentID: PAY00125 (replica)   | |
| | - paymentID: PAY00126              | | | - paymentID: PAY00126 (replica)      | | | - paymentID: PAY00126 (replica)   | |
| +------------------------------------+ | +--------------------------------------+ | +-----------------------------------+ |
+---------------------------------------------------------------------------------------------------------------------------+
                            ^                                      ^                                      ^
                            |                                      |                                      |
                            v                                      v                                      v
                            ----------------------------  ZooKeeper Cluster  ------------------------------

No Replication

Producers -->                                                   KAFKA CLUSTER                                            <-- Consumers/Consumer Groups
+----------------------------------------------------------------------------------------------------------------------------------------------------+
|                        Broker 1                 |                  Broker 2                   |                       Broker 3                     |
|                                                 |                                             |                                                    |
| +---------------------------------------------+ | +-----------------------------------------+ | +------------------------------------------------+ |
| |                Topic: Orders                | | |           Topic: Orders                 | | |                  Topic: Payments               | |
| +---------------------------------------------+ | +-----------------------------------------+ | +------------------------------------------------+ |
| | Partition 0 (Leader)                        | | | Partition 1 (Follower)                  | | | Partition 0 (Leader)                           | |
| |                                             | | |                                         | | |                                                | |
| |  +----------------------+                   | | |  +----------------------+               | | |  +----------------------+                      | |
| |  |"id": "O100"          |                   | | |  |"id": "O101"          |               | | |  |"id": "P100"          |                      | |
| |  |"item": "Laptop"      |                   | | |  |"item": "Phone"       |               | | |  |"order_id": "O100"    |                      | |
| |  |"qty": 1              |                   | | |  |"qty": 2              |               | | |  |"amount": 1200        |                      | |
| |  +----------------------+                   | | |  +----------------------+               | | |  +----------------------+                      | |
| |                                             | | |                                         | | |                                                | |
| |  +----------------------+                   | | |  +----------------------+               | | |  +----------------------+                      | |
| |  |"id": "O102"          |                   | | |  |"id": "O103"          |               | | |  |"id": "P101"          |                      | |
| |  |"item": "Tablet"      |                   | | |  |"item": "Headphones"  |               | | |  |"order_id": "O101"    |                      | |
| |  |"qty": 3              |                   | | |  |"qty": 1              |               | | |  |"amount": 300         |                      | |
| |  +----------------------+                   | | |  +----------------------+               | | |  +----------------------+                      | |
| +---------------------------------------------+ | +-----------------------------------------+ | +------------------------------------------------+ |
| |                Topic: Payments              | | |           Topic: Shipments              | | |                 Topic: Shipments               | |
| +---------------------------------------------+ | +-----------------------------------------+ | +------------------------------------------------+ |
| | Partition 1 (Follower)                      | | | Partition 0 (Leader)                    | | | Partition 1 (Follower)                         | |
| |                                             | | |                                         | | |                                                | |
| |  +----------------------+                   | | |  +----------------------+               | | |  +----------------------+                      | |
| |  |"id": "P102"          |                   | | |  |"id": "S100"          |               | | |  |"id": "S101"          |                      | |
| |  |"order_id": "O102"    |                   | | |  |"order_id": "O100"    |               | | |  |"order_id": "O101"    |                      | |
| |  |"amount": 450         |                   | | |  |"status": "Shipped"   |               | | |  |"status": "Packed"    |                      | |
| |  +----------------------+                   | | |  +----------------------+               | | |  +----------------------+                      | |
| |                                             | | |                                         | | |                                                | |
| |  +----------------------+                   | | |  +----------------------+               | | |  +----------------------+                      | |
| |  |"id": "P103"          |                   | | |  |"id": "S102"          |               | | |  |"id": "S103"          |                      | |
| |  |"order_id": "O104"    |                   | | |  |"order_id": "O102"    |               | | |  |"order_id": "O104"    |                      | |
| |  |"amount": 750         |                   | | |  |"status": "Delivered" |               | | |  |"status": "Dispatched"|                      | |
| |  +----------------------+                   | | |  +----------------------+               | | |  +----------------------+                      | |
| +---------------------------------------------+ | +-----------------------------------------+ | +------------------------------------------------+ |
+----------------------------------------------------------------------------------------------------------------------------------------------------+
                                    ^                                                                                  ^
                                    |                                                                                  |
                                    v                                                                                  v
                                    --------------------------------  ZooKeeper Cluster  -------------------------------

No replication

Producers -->                                                       KAFKA CLUSTER                                        <-- Consumers/Consumer Groups
+----------------------------------------------------------------------------------------------------------------------------------------------------+
|                        Broker 1                 |                  Broker 2                   |                       Broker 3                     |
|                                                 |                                             |                                                    |
| +---------------------------------------------+ | +-----------------------------------------+ | +------------------------------------------------+ |
| |                Topic: Orders                | | |           Topic: Orders                 | | |                  Topic: Payments               | |
| +---------------------------------------------+ | +-----------------------------------------+ | +------------------------------------------------+ |
| | Partition 0 (Leader)                        | | | Partition 1 (Follower)                  | | | Partition 0 (Leader)                           | |
| |                                             | | |                                         | | |                                                | |
| |  +--------------------------------------+   | | |  +-----------------------------------+  | | |  +------------------------------------------+  | |
| |  | {                                    |   | | |  | {                                 |  | | |  | {                                        |  | |
| |  |   "id": "O100",                      |   | | |  |   "id": "O101",                   |  | | |  |   "id": "P100",                          |  | |
| |  |   "item": "Laptop",                  |   | | |  |   "item": "Phone",                |  | | |  |   "order_id": "O100",                    |  | |
| |  |   "qty": 1                           |   | | |  |   "qty": 2                        |  | | |  |   "amount": 1200                         |  | |
| |  | }                                    |   | | |  | }                                 |  | | |  | }                                        |  | |
| |  +--------------------------------------+   | | |  +-----------------------------------+  | | |  +------------------------------------------+  | |
| |                                             | | |                                         | | |                                                | |
| |  +--------------------------------------+   | | |  +-----------------------------------+  | | |  +------------------------------------------+  | |
| |  | {                                    |   | | |  | {                                 |  | | |  | {                                        |  | |
| |  |   "id": "O102",                      |   | | |  |   "id": "O103",                   |  | | |  |   "id": "P101",                          |  | |
| |  |   "item": "Tablet",                  |   | | |  |   "item": "Headphones",           |  | | |  |   "order_id": "O101",                    |  | |
| |  |   "qty": 3                           |   | | |  |   "qty": 1                        |  | | |  |   "amount": 300                          |  | |
| |  | }                                    |   | | |  | }                                 |  | | |  | }                                        |  | |
| |  +--------------------------------------+   | | |  +-----------------------------------+  | | |  +------------------------------------------+  | |
| +---------------------------------------------+ | +-----------------------------------------+ | +------------------------------------------------+ |
| |                Topic: Payments              | | |           Topic: Shipments              | | |                 Topic: Shipments               | |
| +---------------------------------------------+ | +-----------------------------------------+ | +------------------------------------------------+ |
| | Partition 1 (Follower)                      | | | Partition 0 (Leader)                    | | | Partition 1 (Follower)                         | |
| |                                             | | |                                         | | |                                                | |
| |  +--------------------------------------+   | | |  +-----------------------------------+  | | |  +------------------------------------------+  | |
| |  | {                                    |   | | |  | {                                 |  | | |  | {                                        |  | |
| |  |   "id": "P102",                      |   | | |  |   "id": "S100",                   |  | | |  |   "id": "S101",                          |  | |
| |  |   "order_id": "O102",                |   | | |  |   "order_id": "O100",             |  | | |  |   "order_id": "O101",                    |  | |
| |  |   "amount": 450                      |   | | |  |   "status": "Shipped",            |  | | |  |   "status": "Packed",                    |  | |
| |  | }                                    |   | | |  |   "date": "2024-02-25"            |  | | |  |   "date": "2024-02-26"                   |  | |
| |  +--------------------------------------+   | | |  | }                                 |  | | |  | }                                        |  | |
| |                                             | | |  +-----------------------------------+  | | |  +------------------------------------------+  | |
| |                                             | | |                                         | | |                                                | |
| |  +--------------------------------------+   | | |  +-----------------------------------+  | | |  +------------------------------------------+  | |
| |  | {                                    |   | | |  | {                                 |  | | |  | {                                        |  | |
| |  |   "id": "P103",                      |   | | |  |   "id": "S102",                   |  | | |  |   "id": "S103",                          |  | |
| |  |   "order_id": "O104",                |   | | |  |   "order_id": "O102",             |  | | |  |   "order_id": "O104",                    |  | |
| |  |   "amount": 750                      |   | | |  |   "status": "Delivered",          |  | | |  |   "status": "Dispatched",                |  | |
| |  | }                                    |   | | |  |   "date": "2024-02-27"            |  | | |  |   "date": "2024-02-28"                   |  | |
| |  +--------------------------------------+   | | |  | }                                 |  | | |  | }                                        |  | |
| |                                             | | |  +-----------------------------------+  | | |  +------------------------------------------+  | |
| +---------------------------------------------+ | +-----------------------------------------+ | +------------------------------------------------+ |
+----------------------------------------------------------------------------------------------------------------------------------------------------+
                                    ^                                                                                  ^
                                    |                                                                                  |
                                    v                                                                                  v
                                    --------------------------------  ZooKeeper Cluster  -------------------------------

One consumer group with four partitions

Continue reading

How to install Kafka

Downloading

wget "https://archive.apache.org/dist/kafka/1.1.0/kafka_2.11-1.1.0.tgz"

tar -xzf kafka_2.11-1.1.0.tgz

Starting Zookeeper and kafka

bin/zookeeper-server-start.sh config/zookeeper.properties
bin/kafka-server-start.sh config/server.properties

Create topic

bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test

List topics

bin/kafka-topics.sh --list --bootstrap-server localhost:9092

or

bin/kafka-topics.sh --list --zookeeper localhost:2181

Send some messages to Kafka

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test

or

bin/kafka-console-producer.sh --zookeeper localhost:2181 --topic test

Consume messages from Kafka

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning

C# and RabbitMQ PubSub Example

Before you run the following C# code, run the below docker code to have a RabbitMQ applicaton locally.

docker run -d –hostname my-rabbit –name some-rabbit -p 8080:15672 -p 5672:5672 rabbitmq:3-management

After run Docker RabbitMQ container, you can reach from http://localhost:8080/ as shown in following picture.