What is Apache Kafka?
LinkedIn began designing Kafka in 2010 and it was subsequently open-sourced in early 2011.
The platform was created to address LinkedIn's growing needs for real-time data processing and to overcome the limitations of their existing systems.
Kafka is a distributed streaming platform comprised of multiple servers (brokers) that work together to store, process, and transmit large volumes of real-time messages.
For someone unfamiliar with Kafka jargon, Kafka is a system made up of multiple servers that can store and send messages immediately as they arrive.
What is development language?
Apache Kafka is primarily developed in Scala and Java. While its core components are written in Scala, client APIs and many other parts of Kafka are written in Java.
Core Concepts:
– Producer: Sends messages to Kafka.
– Consumer: Reads messages from Kafka.
– Topic: A category or feed name to which messages are stored and published, similar to a table in database.
– Broker/Server: A single Kafka server that stores data and serves client requests.
– Cluster: A group of brokers working together.
– Partition: Topics are split into partitions, which are the basic unit of parallelism in Kafka.
– Offset: A unique identifier for each record within a partition.
Notes:
Messages in Kafka are categorized into topics. The closest analogies for a topic are a database table or a folder in a filesystem. Topics are additionally broken down into a number of partitions.
The term stream is often used when discussing data within systems like Kafka. Most often, a stream is considered to be a single topic of data, regardless of the number of partitions. This represents a single stream of data moving from the producers to the consumers.
Producers --> KAFKA CLUSTER <-- Consumers/Consumer Groups
+---------------------------------------------------------------------------------------+
| Broker 1 | Broker 2 |
| | |
| +----------------------------------------+ | +--------------------------------------+ |
| | Topic: Orders_Placed | | | Topic: Orders_Placed | |
| +----------------------------------------+ | +--------------------------------------+ |
| | Partition 1 (Leader) | | | Partition 1 (Follower) | |
| | - orderID: 123456 | | | - orderID: 123456 (replica) | |
| | - orderID: 123457 | | | - orderID: 123457 (replica) | |
| +----------------------------------------+ | +--------------------------------------+ |
| | Partition 2 (Follower) | | | Partition 2 (Leader) | |
| | - orderID: 123459 (replica) | | | - orderID: 123459 | |
| | - orderID: 123460 (replica) | | | - orderID: 123460 | |
| +----------------------------------------+ | +--------------------------------------+ |
| | |
| +----------------------------------------+ | +--------------------------------------+ |
| | Topic: Payments | | | Topic: Payments | |
| +----------------------------------------+ | +--------------------------------------+ |
| | Partition 1 (Follower) | | | Partition 1 (Leader) | |
| | - paymentID: PAY00123 (replica) | | | - paymentID: PAY00123 | |
| | - paymentID: PAY00124 (replica) | | | - paymentID: PAY00124 | |
| +----------------------------------------+ | +--------------------------------------+ |
| | Partition 2 (Leader) | | | Partition 2 (Follower) | |
| | - paymentID: PAY00125 | | | - paymentID: PAY00125 (replica) | |
| | - paymentID: PAY00126 | | | - paymentID: PAY00126 (replica) | |
| +----------------------------------------+ | +--------------------------------------+ |
+---------------------------------------------------------------------------------------+
^ ^
| |
v v
------------ ZooKeeper Cluster --------------Producers --> KAFKA CLUSTER <-- Consumers/Consumer Groups
+---------------------------------------------------------------------------------------------------------------------------+
| Broker 1 | Broker 2 | Broker 3 |
| | | |
| +------------------------------------+ | +--------------------------------------+ | +-----------------------------------+ |
| | Topic: Orders_Placed | | | Topic: Orders_Placed | | | Topic: Orders_Placed | |
| +------------------------------------+ | +--------------------------------------+ | +-----------------------------------+ |
| | Partition 1 (Leader) | | | Partition 1 (Follower) | | | Partition 1 (Replica) | |
| | - orderID: 123456 | | | - orderID: 123456 (replica) | | | - orderID: 123456 (replica) | |
| | - orderID: 123457 | | | - orderID: 123457 (replica) | | | - orderID: 123457 (replica) | |
| +------------------------------------+ | +--------------------------------------+ | +-----------------------------------+ |
| | Partition 2 (Follower) | | | Partition 2 (Leader) | | | Partition 2 (Replica) | |
| | - orderID: 123459 (replica) | | | - orderID: 123459 | | | - orderID: 123459 (replica) | |
| | - orderID: 123460 (replica) | | | - orderID: 123460 | | | - orderID: 123460 (replica) | |
| +------------------------------------+ | +--------------------------------------+ | +-----------------------------------+ |
| | | |
| +------------------------------------+ | +--------------------------------------+ | +-----------------------------------+ |
| | Topic: Payments | | | Topic: Payments | | | Topic: Payments | |
| +------------------------------------+ | +--------------------------------------+ | +-----------------------------------+ |
| | Partition 1 (Follower) | | | Partition 1 (Leader) | | | Partition 1 (Replica) | |
| | - paymentID: PAY00123 (replica) | | | - paymentID: PAY00123 | | | - paymentID: PAY00123 (replica) | |
| | - paymentID: PAY00124 (replica) | | | - paymentID: PAY00124 | | | - paymentID: PAY00124 (replica) | |
| +------------------------------------+ | +--------------------------------------+ | +-----------------------------------+ |
| | Partition 2 (Leader) | | | Partition 2 (Follower) | | | Partition 2 (Replica) | |
| | - paymentID: PAY00125 | | | - paymentID: PAY00125 (replica) | | | - paymentID: PAY00125 (replica) | |
| | - paymentID: PAY00126 | | | - paymentID: PAY00126 (replica) | | | - paymentID: PAY00126 (replica) | |
| +------------------------------------+ | +--------------------------------------+ | +-----------------------------------+ |
+---------------------------------------------------------------------------------------------------------------------------+
^ ^ ^
| | |
v v v
---------------------------- ZooKeeper Cluster ------------------------------
No Replication
Producers --> KAFKA CLUSTER <-- Consumers/Consumer Groups
+----------------------------------------------------------------------------------------------------------------------------------------------------+
| Broker 1 | Broker 2 | Broker 3 |
| | | |
| +---------------------------------------------+ | +-----------------------------------------+ | +------------------------------------------------+ |
| | Topic: Orders | | | Topic: Orders | | | Topic: Payments | |
| +---------------------------------------------+ | +-----------------------------------------+ | +------------------------------------------------+ |
| | Partition 0 (Leader) | | | Partition 1 (Follower) | | | Partition 0 (Leader) | |
| | | | | | | | | |
| | +----------------------+ | | | +----------------------+ | | | +----------------------+ | |
| | |"id": "O100" | | | | |"id": "O101" | | | | |"id": "P100" | | |
| | |"item": "Laptop" | | | | |"item": "Phone" | | | | |"order_id": "O100" | | |
| | |"qty": 1 | | | | |"qty": 2 | | | | |"amount": 1200 | | |
| | +----------------------+ | | | +----------------------+ | | | +----------------------+ | |
| | | | | | | | | |
| | +----------------------+ | | | +----------------------+ | | | +----------------------+ | |
| | |"id": "O102" | | | | |"id": "O103" | | | | |"id": "P101" | | |
| | |"item": "Tablet" | | | | |"item": "Headphones" | | | | |"order_id": "O101" | | |
| | |"qty": 3 | | | | |"qty": 1 | | | | |"amount": 300 | | |
| | +----------------------+ | | | +----------------------+ | | | +----------------------+ | |
| +---------------------------------------------+ | +-----------------------------------------+ | +------------------------------------------------+ |
| | Topic: Payments | | | Topic: Shipments | | | Topic: Shipments | |
| +---------------------------------------------+ | +-----------------------------------------+ | +------------------------------------------------+ |
| | Partition 1 (Follower) | | | Partition 0 (Leader) | | | Partition 1 (Follower) | |
| | | | | | | | | |
| | +----------------------+ | | | +----------------------+ | | | +----------------------+ | |
| | |"id": "P102" | | | | |"id": "S100" | | | | |"id": "S101" | | |
| | |"order_id": "O102" | | | | |"order_id": "O100" | | | | |"order_id": "O101" | | |
| | |"amount": 450 | | | | |"status": "Shipped" | | | | |"status": "Packed" | | |
| | +----------------------+ | | | +----------------------+ | | | +----------------------+ | |
| | | | | | | | | |
| | +----------------------+ | | | +----------------------+ | | | +----------------------+ | |
| | |"id": "P103" | | | | |"id": "S102" | | | | |"id": "S103" | | |
| | |"order_id": "O104" | | | | |"order_id": "O102" | | | | |"order_id": "O104" | | |
| | |"amount": 750 | | | | |"status": "Delivered" | | | | |"status": "Dispatched"| | |
| | +----------------------+ | | | +----------------------+ | | | +----------------------+ | |
| +---------------------------------------------+ | +-----------------------------------------+ | +------------------------------------------------+ |
+----------------------------------------------------------------------------------------------------------------------------------------------------+
^ ^
| |
v v
-------------------------------- ZooKeeper Cluster -------------------------------
No replication
Producers --> KAFKA CLUSTER <-- Consumers/Consumer Groups
+----------------------------------------------------------------------------------------------------------------------------------------------------+
| Broker 1 | Broker 2 | Broker 3 |
| | | |
| +---------------------------------------------+ | +-----------------------------------------+ | +------------------------------------------------+ |
| | Topic: Orders | | | Topic: Orders | | | Topic: Payments | |
| +---------------------------------------------+ | +-----------------------------------------+ | +------------------------------------------------+ |
| | Partition 0 (Leader) | | | Partition 1 (Follower) | | | Partition 0 (Leader) | |
| | | | | | | | | |
| | +--------------------------------------+ | | | +-----------------------------------+ | | | +------------------------------------------+ | |
| | | { | | | | | { | | | | | { | | |
| | | "id": "O100", | | | | | "id": "O101", | | | | | "id": "P100", | | |
| | | "item": "Laptop", | | | | | "item": "Phone", | | | | | "order_id": "O100", | | |
| | | "qty": 1 | | | | | "qty": 2 | | | | | "amount": 1200 | | |
| | | } | | | | | } | | | | | } | | |
| | +--------------------------------------+ | | | +-----------------------------------+ | | | +------------------------------------------+ | |
| | | | | | | | | |
| | +--------------------------------------+ | | | +-----------------------------------+ | | | +------------------------------------------+ | |
| | | { | | | | | { | | | | | { | | |
| | | "id": "O102", | | | | | "id": "O103", | | | | | "id": "P101", | | |
| | | "item": "Tablet", | | | | | "item": "Headphones", | | | | | "order_id": "O101", | | |
| | | "qty": 3 | | | | | "qty": 1 | | | | | "amount": 300 | | |
| | | } | | | | | } | | | | | } | | |
| | +--------------------------------------+ | | | +-----------------------------------+ | | | +------------------------------------------+ | |
| +---------------------------------------------+ | +-----------------------------------------+ | +------------------------------------------------+ |
| | Topic: Payments | | | Topic: Shipments | | | Topic: Shipments | |
| +---------------------------------------------+ | +-----------------------------------------+ | +------------------------------------------------+ |
| | Partition 1 (Follower) | | | Partition 0 (Leader) | | | Partition 1 (Follower) | |
| | | | | | | | | |
| | +--------------------------------------+ | | | +-----------------------------------+ | | | +------------------------------------------+ | |
| | | { | | | | | { | | | | | { | | |
| | | "id": "P102", | | | | | "id": "S100", | | | | | "id": "S101", | | |
| | | "order_id": "O102", | | | | | "order_id": "O100", | | | | | "order_id": "O101", | | |
| | | "amount": 450 | | | | | "status": "Shipped", | | | | | "status": "Packed", | | |
| | | } | | | | | "date": "2024-02-25" | | | | | "date": "2024-02-26" | | |
| | +--------------------------------------+ | | | | } | | | | | } | | |
| | | | | +-----------------------------------+ | | | +------------------------------------------+ | |
| | | | | | | | | |
| | +--------------------------------------+ | | | +-----------------------------------+ | | | +------------------------------------------+ | |
| | | { | | | | | { | | | | | { | | |
| | | "id": "P103", | | | | | "id": "S102", | | | | | "id": "S103", | | |
| | | "order_id": "O104", | | | | | "order_id": "O102", | | | | | "order_id": "O104", | | |
| | | "amount": 750 | | | | | "status": "Delivered", | | | | | "status": "Dispatched", | | |
| | | } | | | | | "date": "2024-02-27" | | | | | "date": "2024-02-28" | | |
| | +--------------------------------------+ | | | | } | | | | | } | | |
| | | | | +-----------------------------------+ | | | +------------------------------------------+ | |
| +---------------------------------------------+ | +-----------------------------------------+ | +------------------------------------------------+ |
+----------------------------------------------------------------------------------------------------------------------------------------------------+
^ ^
| |
v v
-------------------------------- ZooKeeper Cluster -------------------------------
One consumer group with four partitions
Four partitions split to two consumers in a consumer group
Four consumers in a consumer group with one partition each
More consumers in a consumer group than partitions means idle consumers
Adding a new consumer group, both groups receive all messages
