Notes about Apache Kafka

What is Apache Kafka?
LinkedIn began designing Kafka in 2010 and it was subsequently open-sourced in early 2011.

The platform was created to address LinkedIn's growing needs for real-time data processing and to overcome the limitations of their existing systems.

Kafka is a distributed streaming platform comprised of multiple servers (brokers) that work together to store, process, and transmit large volumes of real-time messages.

For someone unfamiliar with Kafka jargon, Kafka is a system made up of multiple servers that can store and send messages immediately as they arrive.

What is development language?
Apache Kafka is primarily developed in Scala and Java. While its core components are written in Scala, client APIs and many other parts of Kafka are written in Java.

Core Concepts:
Producer: Sends messages to Kafka.
Consumer: Reads messages from Kafka.
Topic: A category or feed name to which messages are stored and published, similar to a table in database.
Broker/Server: A single Kafka server that stores data and serves client requests.
Cluster: A group of brokers working together.
Partition: Topics are split into partitions, which are the basic unit of parallelism in Kafka.
Offset: A unique identifier for each record within a partition.

Notes:

Messages in Kafka are categorized into topics. The closest analogies for a topic are a database table or a folder in a filesystem. Topics are additionally broken down into a number of partitions.

The term stream is often used when discussing data within systems like Kafka. Most often, a stream is considered to be a single topic of data, regardless of the number of partitions. This represents a single stream of data moving from the producers to the consumers.

Producers -->                           KAFKA CLUSTER       <-- Consumers/Consumer Groups
+---------------------------------------------------------------------------------------+
|                   Broker 1                 |                 Broker 2                 |
|                                            |                                          |
| +----------------------------------------+ | +--------------------------------------+ |
| |               Topic: Orders_Placed     | | |         Topic: Orders_Placed         | |
| +----------------------------------------+ | +--------------------------------------+ |
| | Partition 1 (Leader)                   | | | Partition 1 (Follower)               | |
| | - orderID: 123456                      | | | - orderID: 123456 (replica)          | |
| | - orderID: 123457                      | | | - orderID: 123457 (replica)          | |
| +----------------------------------------+ | +--------------------------------------+ |
| | Partition 2 (Follower)                 | | | Partition 2 (Leader)                 | |
| | - orderID: 123459 (replica)            | | | - orderID: 123459                    | |
| | - orderID: 123460 (replica)            | | | - orderID: 123460                    | |
| +----------------------------------------+ | +--------------------------------------+ |
|                                            |                                          |
| +----------------------------------------+ | +--------------------------------------+ |
| |                Topic: Payments         | | |           Topic: Payments            | |
| +----------------------------------------+ | +--------------------------------------+ |
| | Partition 1 (Follower)                 | | | Partition 1 (Leader)                 | |
| | - paymentID: PAY00123 (replica)        | | | - paymentID: PAY00123                | |
| | - paymentID: PAY00124 (replica)        | | | - paymentID: PAY00124                | |
| +----------------------------------------+ | +--------------------------------------+ |
| | Partition 2 (Leader)                   | | | Partition 2 (Follower)               | |
| | - paymentID: PAY00125                  | | | - paymentID: PAY00125 (replica)      | |
| | - paymentID: PAY00126                  | | | - paymentID: PAY00126 (replica)      | |
| +----------------------------------------+ | +--------------------------------------+ |
+---------------------------------------------------------------------------------------+
                        ^                                             ^
                        |                                             |
                        v                                             v
                        ------------  ZooKeeper Cluster  --------------
Producers -->                                           KAFKA CLUSTER                           <-- Consumers/Consumer Groups
+---------------------------------------------------------------------------------------------------------------------------+
|               Broker 1                 |                 Broker 2                 |                     Broker 3          |
|                                        |                                          |                                       |
| +------------------------------------+ | +--------------------------------------+ | +-----------------------------------+ |
| |           Topic: Orders_Placed     | | |         Topic: Orders_Placed         | | |      Topic: Orders_Placed         | |
| +------------------------------------+ | +--------------------------------------+ | +-----------------------------------+ |
| | Partition 1 (Leader)               | | | Partition 1 (Follower)               | | | Partition 1 (Replica)             | |
| | - orderID: 123456                  | | | - orderID: 123456 (replica)          | | | - orderID: 123456 (replica)       | |
| | - orderID: 123457                  | | | - orderID: 123457 (replica)          | | | - orderID: 123457 (replica)       | |
| +------------------------------------+ | +--------------------------------------+ | +-----------------------------------+ |
| | Partition 2 (Follower)             | | | Partition 2 (Leader)                 | | | Partition 2 (Replica)             | |
| | - orderID: 123459 (replica)        | | | - orderID: 123459                    | | | - orderID: 123459 (replica)       | |
| | - orderID: 123460 (replica)        | | | - orderID: 123460                    | | | - orderID: 123460 (replica)       | |
| +------------------------------------+ | +--------------------------------------+ | +-----------------------------------+ |
|                                        |                                          |                                       |
| +------------------------------------+ | +--------------------------------------+ | +-----------------------------------+ |
| |            Topic: Payments         | | |           Topic: Payments            | | |       Topic: Payments             | |
| +------------------------------------+ | +--------------------------------------+ | +-----------------------------------+ |
| | Partition 1 (Follower)             | | | Partition 1 (Leader)                 | | | Partition 1 (Replica)             | |
| | - paymentID: PAY00123 (replica)    | | | - paymentID: PAY00123                | | | - paymentID: PAY00123 (replica)   | |
| | - paymentID: PAY00124 (replica)    | | | - paymentID: PAY00124                | | | - paymentID: PAY00124 (replica)   | |
| +------------------------------------+ | +--------------------------------------+ | +-----------------------------------+ |
| | Partition 2 (Leader)               | | | Partition 2 (Follower)               | | | Partition 2 (Replica)             | |
| | - paymentID: PAY00125              | | | - paymentID: PAY00125 (replica)      | | | - paymentID: PAY00125 (replica)   | |
| | - paymentID: PAY00126              | | | - paymentID: PAY00126 (replica)      | | | - paymentID: PAY00126 (replica)   | |
| +------------------------------------+ | +--------------------------------------+ | +-----------------------------------+ |
+---------------------------------------------------------------------------------------------------------------------------+
                            ^                                      ^                                      ^
                            |                                      |                                      |
                            v                                      v                                      v
                            ----------------------------  ZooKeeper Cluster  ------------------------------

No Replication

Producers -->                                                   KAFKA CLUSTER                                            <-- Consumers/Consumer Groups
+----------------------------------------------------------------------------------------------------------------------------------------------------+
|                        Broker 1                 |                  Broker 2                   |                       Broker 3                     |
|                                                 |                                             |                                                    |
| +---------------------------------------------+ | +-----------------------------------------+ | +------------------------------------------------+ |
| |                Topic: Orders                | | |           Topic: Orders                 | | |                  Topic: Payments               | |
| +---------------------------------------------+ | +-----------------------------------------+ | +------------------------------------------------+ |
| | Partition 0 (Leader)                        | | | Partition 1 (Follower)                  | | | Partition 0 (Leader)                           | |
| |                                             | | |                                         | | |                                                | |
| |  +----------------------+                   | | |  +----------------------+               | | |  +----------------------+                      | |
| |  |"id": "O100"          |                   | | |  |"id": "O101"          |               | | |  |"id": "P100"          |                      | |
| |  |"item": "Laptop"      |                   | | |  |"item": "Phone"       |               | | |  |"order_id": "O100"    |                      | |
| |  |"qty": 1              |                   | | |  |"qty": 2              |               | | |  |"amount": 1200        |                      | |
| |  +----------------------+                   | | |  +----------------------+               | | |  +----------------------+                      | |
| |                                             | | |                                         | | |                                                | |
| |  +----------------------+                   | | |  +----------------------+               | | |  +----------------------+                      | |
| |  |"id": "O102"          |                   | | |  |"id": "O103"          |               | | |  |"id": "P101"          |                      | |
| |  |"item": "Tablet"      |                   | | |  |"item": "Headphones"  |               | | |  |"order_id": "O101"    |                      | |
| |  |"qty": 3              |                   | | |  |"qty": 1              |               | | |  |"amount": 300         |                      | |
| |  +----------------------+                   | | |  +----------------------+               | | |  +----------------------+                      | |
| +---------------------------------------------+ | +-----------------------------------------+ | +------------------------------------------------+ |
| |                Topic: Payments              | | |           Topic: Shipments              | | |                 Topic: Shipments               | |
| +---------------------------------------------+ | +-----------------------------------------+ | +------------------------------------------------+ |
| | Partition 1 (Follower)                      | | | Partition 0 (Leader)                    | | | Partition 1 (Follower)                         | |
| |                                             | | |                                         | | |                                                | |
| |  +----------------------+                   | | |  +----------------------+               | | |  +----------------------+                      | |
| |  |"id": "P102"          |                   | | |  |"id": "S100"          |               | | |  |"id": "S101"          |                      | |
| |  |"order_id": "O102"    |                   | | |  |"order_id": "O100"    |               | | |  |"order_id": "O101"    |                      | |
| |  |"amount": 450         |                   | | |  |"status": "Shipped"   |               | | |  |"status": "Packed"    |                      | |
| |  +----------------------+                   | | |  +----------------------+               | | |  +----------------------+                      | |
| |                                             | | |                                         | | |                                                | |
| |  +----------------------+                   | | |  +----------------------+               | | |  +----------------------+                      | |
| |  |"id": "P103"          |                   | | |  |"id": "S102"          |               | | |  |"id": "S103"          |                      | |
| |  |"order_id": "O104"    |                   | | |  |"order_id": "O102"    |               | | |  |"order_id": "O104"    |                      | |
| |  |"amount": 750         |                   | | |  |"status": "Delivered" |               | | |  |"status": "Dispatched"|                      | |
| |  +----------------------+                   | | |  +----------------------+               | | |  +----------------------+                      | |
| +---------------------------------------------+ | +-----------------------------------------+ | +------------------------------------------------+ |
+----------------------------------------------------------------------------------------------------------------------------------------------------+
                                    ^                                                                                  ^
                                    |                                                                                  |
                                    v                                                                                  v
                                    --------------------------------  ZooKeeper Cluster  -------------------------------

No replication

Producers -->                                                       KAFKA CLUSTER                                        <-- Consumers/Consumer Groups
+----------------------------------------------------------------------------------------------------------------------------------------------------+
|                        Broker 1                 |                  Broker 2                   |                       Broker 3                     |
|                                                 |                                             |                                                    |
| +---------------------------------------------+ | +-----------------------------------------+ | +------------------------------------------------+ |
| |                Topic: Orders                | | |           Topic: Orders                 | | |                  Topic: Payments               | |
| +---------------------------------------------+ | +-----------------------------------------+ | +------------------------------------------------+ |
| | Partition 0 (Leader)                        | | | Partition 1 (Follower)                  | | | Partition 0 (Leader)                           | |
| |                                             | | |                                         | | |                                                | |
| |  +--------------------------------------+   | | |  +-----------------------------------+  | | |  +------------------------------------------+  | |
| |  | {                                    |   | | |  | {                                 |  | | |  | {                                        |  | |
| |  |   "id": "O100",                      |   | | |  |   "id": "O101",                   |  | | |  |   "id": "P100",                          |  | |
| |  |   "item": "Laptop",                  |   | | |  |   "item": "Phone",                |  | | |  |   "order_id": "O100",                    |  | |
| |  |   "qty": 1                           |   | | |  |   "qty": 2                        |  | | |  |   "amount": 1200                         |  | |
| |  | }                                    |   | | |  | }                                 |  | | |  | }                                        |  | |
| |  +--------------------------------------+   | | |  +-----------------------------------+  | | |  +------------------------------------------+  | |
| |                                             | | |                                         | | |                                                | |
| |  +--------------------------------------+   | | |  +-----------------------------------+  | | |  +------------------------------------------+  | |
| |  | {                                    |   | | |  | {                                 |  | | |  | {                                        |  | |
| |  |   "id": "O102",                      |   | | |  |   "id": "O103",                   |  | | |  |   "id": "P101",                          |  | |
| |  |   "item": "Tablet",                  |   | | |  |   "item": "Headphones",           |  | | |  |   "order_id": "O101",                    |  | |
| |  |   "qty": 3                           |   | | |  |   "qty": 1                        |  | | |  |   "amount": 300                          |  | |
| |  | }                                    |   | | |  | }                                 |  | | |  | }                                        |  | |
| |  +--------------------------------------+   | | |  +-----------------------------------+  | | |  +------------------------------------------+  | |
| +---------------------------------------------+ | +-----------------------------------------+ | +------------------------------------------------+ |
| |                Topic: Payments              | | |           Topic: Shipments              | | |                 Topic: Shipments               | |
| +---------------------------------------------+ | +-----------------------------------------+ | +------------------------------------------------+ |
| | Partition 1 (Follower)                      | | | Partition 0 (Leader)                    | | | Partition 1 (Follower)                         | |
| |                                             | | |                                         | | |                                                | |
| |  +--------------------------------------+   | | |  +-----------------------------------+  | | |  +------------------------------------------+  | |
| |  | {                                    |   | | |  | {                                 |  | | |  | {                                        |  | |
| |  |   "id": "P102",                      |   | | |  |   "id": "S100",                   |  | | |  |   "id": "S101",                          |  | |
| |  |   "order_id": "O102",                |   | | |  |   "order_id": "O100",             |  | | |  |   "order_id": "O101",                    |  | |
| |  |   "amount": 450                      |   | | |  |   "status": "Shipped",            |  | | |  |   "status": "Packed",                    |  | |
| |  | }                                    |   | | |  |   "date": "2024-02-25"            |  | | |  |   "date": "2024-02-26"                   |  | |
| |  +--------------------------------------+   | | |  | }                                 |  | | |  | }                                        |  | |
| |                                             | | |  +-----------------------------------+  | | |  +------------------------------------------+  | |
| |                                             | | |                                         | | |                                                | |
| |  +--------------------------------------+   | | |  +-----------------------------------+  | | |  +------------------------------------------+  | |
| |  | {                                    |   | | |  | {                                 |  | | |  | {                                        |  | |
| |  |   "id": "P103",                      |   | | |  |   "id": "S102",                   |  | | |  |   "id": "S103",                          |  | |
| |  |   "order_id": "O104",                |   | | |  |   "order_id": "O102",             |  | | |  |   "order_id": "O104",                    |  | |
| |  |   "amount": 750                      |   | | |  |   "status": "Delivered",          |  | | |  |   "status": "Dispatched",                |  | |
| |  | }                                    |   | | |  |   "date": "2024-02-27"            |  | | |  |   "date": "2024-02-28"                   |  | |
| |  +--------------------------------------+   | | |  | }                                 |  | | |  | }                                        |  | |
| |                                             | | |  +-----------------------------------+  | | |  +------------------------------------------+  | |
| +---------------------------------------------+ | +-----------------------------------------+ | +------------------------------------------------+ |
+----------------------------------------------------------------------------------------------------------------------------------------------------+
                                    ^                                                                                  ^
                                    |                                                                                  |
                                    v                                                                                  v
                                    --------------------------------  ZooKeeper Cluster  -------------------------------

One consumer group with four partitions

Four partitions split to two consumers in a consumer group

Four consumers in a consumer group with one partition each

More consumers in a consumer group than partitions means idle consumers

Adding a new consumer group, both groups receive all messages