I mostly use these tools below so this post will be updated frequently.
Continue readingNotes about Apache Kafka
What is Apache Kafka?
LinkedIn began designing Kafka in 2010 and it was subsequently open-sourced in early 2011.
The platform was created to address LinkedIn's growing needs for real-time data processing and to overcome the limitations of their existing systems.
Kafka is a distributed streaming platform comprised of multiple servers (brokers) that work together to store, process, and transmit large volumes of real-time messages.
For someone unfamiliar with Kafka jargon, Kafka is a system made up of multiple servers that can store and send messages immediately as they arrive.
What is development language?
Apache Kafka is primarily developed in Scala and Java. While its core components are written in Scala, client APIs and many other parts of Kafka are written in Java.
Core Concepts:
– Producer: Sends messages to Kafka.
– Consumer: Reads messages from Kafka.
– Topic: A category or feed name to which messages are stored and published, similar to a table in database.
– Broker/Server: A single Kafka server that stores data and serves client requests.
– Cluster: A group of brokers working together.
– Partition: Topics are split into partitions, which are the basic unit of parallelism in Kafka.
– Offset: A unique identifier for each record within a partition.
Notes:
Messages in Kafka are categorized into topics. The closest analogies for a topic are a database table or a folder in a filesystem. Topics are additionally broken down into a number of partitions.
The term stream is often used when discussing data within systems like Kafka. Most often, a stream is considered to be a single topic of data, regardless of the number of partitions. This represents a single stream of data moving from the producers to the consumers.
Producers --> KAFKA CLUSTER <-- Consumers/Consumer Groups
+---------------------------------------------------------------------------------------+
| Broker 1 | Broker 2 |
| | |
| +----------------------------------------+ | +--------------------------------------+ |
| | Topic: Orders_Placed | | | Topic: Orders_Placed | |
| +----------------------------------------+ | +--------------------------------------+ |
| | Partition 1 (Leader) | | | Partition 1 (Follower) | |
| | - orderID: 123456 | | | - orderID: 123456 (replica) | |
| | - orderID: 123457 | | | - orderID: 123457 (replica) | |
| +----------------------------------------+ | +--------------------------------------+ |
| | Partition 2 (Follower) | | | Partition 2 (Leader) | |
| | - orderID: 123459 (replica) | | | - orderID: 123459 | |
| | - orderID: 123460 (replica) | | | - orderID: 123460 | |
| +----------------------------------------+ | +--------------------------------------+ |
| | |
| +----------------------------------------+ | +--------------------------------------+ |
| | Topic: Payments | | | Topic: Payments | |
| +----------------------------------------+ | +--------------------------------------+ |
| | Partition 1 (Follower) | | | Partition 1 (Leader) | |
| | - paymentID: PAY00123 (replica) | | | - paymentID: PAY00123 | |
| | - paymentID: PAY00124 (replica) | | | - paymentID: PAY00124 | |
| +----------------------------------------+ | +--------------------------------------+ |
| | Partition 2 (Leader) | | | Partition 2 (Follower) | |
| | - paymentID: PAY00125 | | | - paymentID: PAY00125 (replica) | |
| | - paymentID: PAY00126 | | | - paymentID: PAY00126 (replica) | |
| +----------------------------------------+ | +--------------------------------------+ |
+---------------------------------------------------------------------------------------+
^ ^
| |
v v
--------------- ZooKeeper Cluster -----------
Producers --> KAFKA CLUSTER <-- Consumers/Consumer Groups
+---------------------------------------------------------------------------------------------------------------------------+
| Broker 1 | Broker 2 | Broker 3 |
| | | |
| +------------------------------------+ | +--------------------------------------+ | +-----------------------------------+ |
| | Topic: Orders_Placed | | | Topic: Orders_Placed | | | Topic: Orders_Placed | |
| +------------------------------------+ | +--------------------------------------+ | +-----------------------------------+ |
| | Partition 1 (Leader) | | | Partition 1 (Follower) | | | Partition 1 (Replica) | |
| | - orderID: 123456 | | | - orderID: 123456 (replica) | | | - orderID: 123456 (replica) | |
| | - orderID: 123457 | | | - orderID: 123457 (replica) | | | - orderID: 123457 (replica) | |
| +------------------------------------+ | +--------------------------------------+ | +-----------------------------------+ |
| | Partition 2 (Follower) | | | Partition 2 (Leader) | | | Partition 2 (Replica) | |
| | - orderID: 123459 (replica) | | | - orderID: 123459 | | | - orderID: 123459 (replica) | |
| | - orderID: 123460 (replica) | | | - orderID: 123460 | | | - orderID: 123460 (replica) | |
| +------------------------------------+ | +--------------------------------------+ | +-----------------------------------+ |
| | | |
| +------------------------------------+ | +--------------------------------------+ | +-----------------------------------+ |
| | Topic: Payments | | | Topic: Payments | | | Topic: Payments | |
| +------------------------------------+ | +--------------------------------------+ | +-----------------------------------+ |
| | Partition 1 (Follower) | | | Partition 1 (Leader) | | | Partition 1 (Replica) | |
| | - paymentID: PAY00123 (replica) | | | - paymentID: PAY00123 | | | - paymentID: PAY00123 (replica) | |
| | - paymentID: PAY00124 (replica) | | | - paymentID: PAY00124 | | | - paymentID: PAY00124 (replica) | |
| +------------------------------------+ | +--------------------------------------+ | +-----------------------------------+ |
| | Partition 2 (Leader) | | | Partition 2 (Follower) | | | Partition 2 (Replica) | |
| | - paymentID: PAY00125 | | | - paymentID: PAY00125 (replica) | | | - paymentID: PAY00125 (replica) | |
| | - paymentID: PAY00126 | | | - paymentID: PAY00126 (replica) | | | - paymentID: PAY00126 (replica) | |
| +------------------------------------+ | +--------------------------------------+ | +-----------------------------------+ |
+---------------------------------------------------------------------------------------------------------------------------+
^ ^ ^
| | |
v v v
---------------------------- ZooKeeper Cluster ------------------------------
One consumer group with four partitions
Designing SocialMediaLink in OOP
Beginning of Software Design
Requirement: Design an Actor class which has personal details and social media links.
A note on design:
The current design, with direct properties for each social media link and separate methods for updating/removing each, is more rigid and might lead to more boilerplate code as the number of social media platforms increases.
Continue readingThis design has some potential disadvantages:
- Rigidity:
- If you wanted to add support for another social media platform, you'd have to modify the
Actorclass, adding a new property and accompanying methods to update and remove the link. This breaks the Open/Closed Principle (OCP) of SOLID, which states that a class should be open for extension but closed for modification.- Duplication:
- The
IsValidLinkmethod is invoked in everyUpdateXXXLinkmethod. If the validation logic changes, it will be correctly applied across all methods, but the repetitive structure is a red flag that there might be a more elegant design solution.- No Single Responsibility:
- The
Actorclass is currently handling both the data representation of an actor and the management and validation of links. This can be seen as a violation of the Single Responsibility Principle (SRP) of SOLID. Ideally, anActorshouldn't have to be concerned with the intricacies of URL validation.- Null State Ambiguity:
- Using
nullto represent the absence of a link can lead to potential null reference exceptions if not handled properly elsewhere in the code. While setting the value tonulldoes represent the absence of a value, it requires other parts of your application to consistently check for null before using a link.- Lack of History/Tracking:
- In the current design, there's no way to keep track of changes to an actor's social media links. If link history or auditing is a requirement (either now or in the future), this design would need to be significantly refactored.
- Potential for Incomplete Removals:
- If a developer forgets to call the removal method, old data might remain in the system. In the design where links were contained in a list, you could have just cleared the list or removed specific items, ensuring all links of that type were removed.
- Scalability Concerns:
- As new properties or methods are added, this class will grow, making it harder to maintain. A larger class tends to be more error-prone and harder to debug.
Recommendation:
A more flexible approach would involve encapsulating the behaviour and data of social media links into their own classes (as shown in the next designs). It allows for easier additions of new link types, centralized validation logic, and a clearer separation of concerns.
Notes about IP Address and Subnet Mask
Everything is working on binary number system in computers, so it is valid in IP addresses as well.
Assume that IP address is 192.168.1.0 in dotted-decimal format is equal to 11000000.10101000.00000001.00000000 in dotted-binary format.
Assume that Subnet Mask is 255.255.255.0 in dotted-decimal format is equal to 11111111.11111111.11111111.00000000 in dotted-binary format. Subnet Mask is shown in /24 in slash notation(known as CIDR notation)
IP Address with Subnet Mask can be shown in CIDR notation 192.168.1.0/24
CIDR gives a concise way to represent both an IP network and its size using a combination of an IP address and a number after a slash.
Subnet Mask is 24 after slash in decimal representation. In order to find out 24, Subnet mask should be represented in binary number system like 11111111.11111111.11111111.00000000. So, just count how many 1 bit there are in Subnet Mask.
IP Address and Subnet Mask in Decimal format:
IP Address: 192.168.1.0
Subnet Mask: 255.255.255.0
IP Address and Subnet Mask in CIDR format:
192.168.1.0/24
IP Address and Subnet Mask in Binary format:
IP Address: 11000000.10101000.00000001.00000000
Subnet Mask: 11111111.11111111.11111111.00000000
How to create AWS Lambda with AWS SAM CLI
According to my experience, if you want to create an AWS Lambda without interactive mode, 15 templates below don't exists for each runtime, for example,
Data processingtemplate only exists fordotnetandnodejs.
Notice that third template name isHello World Example with Powertools for AWS Lambdain below output, but if you want to create a project with python runtime and third template in the belowsam initoutput, you have to usehello-world-powertools-pythonas a template name. But it is not mentioned in help. So, i just usedsam initin interactive mode and output of this command showed me template name.sam init --name my-data-processing-app --runtime python3.9 --dependency-manager pip --app-template hello-world-powertools-python
$ sam init
You can preselect a particular runtime or package type when using the `sam init` experience.
Call `sam init --help` to learn more.
Which template source would you like to use?
1 - AWS Quick Start Templates
2 - Custom Template Location
Choice: 1
Choose an AWS Quick Start application template
1 - Hello World Example
2 - Data processing
3 - Hello World Example with Powertools for AWS Lambda
4 - Multi-step workflow
5 - Scheduled task
6 - Standalone function
7 - Serverless API
8 - Infrastructure event management
9 - Lambda Response Streaming
10 - Serverless Connector Hello World Example
11 - Multi-step workflow with Connectors
12 - Full Stack
13 - Lambda EFS example
14 - DynamoDB Example
15 - Machine Learning
Template: 1
Use the most popular runtime and package type? (Python and zip) [y/N]: How to create AWS Lambda, Azure Function, Google Cloud Function with Serverless Framework CLI
Serverless framework is a CLI to build cloud applications. There are more than 10 Serverless Infrastructure Providers like AWS, Azure and GCP as shown below screenshot.
Find more Serverless Framework CLI command in Azure Serverless TypeScript project in GitHub
According to my experience, Serverless Framework gives a better support for AWS provider, for example, if you see project template below, there are many project templates for AWS but very less support for other providers.
Find more details in https://www.serverless.com/framework/docs/getting-started
Continue readingVisitor Design Pattern Notes
Visitor classes are commonly used to perform operations on a set of objects without modifying the structure of their underlying classes.
Visitor Design Pattern allows you to add new operations(behaviours) via Visitor classes to existing classes without modifying their structure. Instead of adding the operation within the existing class, you create a visitor class named as operation you want to have in existing class, so that the operation is externalised from a set of objects without modifying the structure of their underlying classes.
Visitor Design Pattern is more about adding new operations to existing classes via Visitor classes without modifying their structure, rather than adding new fields or methods directly to the classes themselves.
Continue readingConsider a system where you have a hierarchy of classes representing different shapes, like
Circle,Rectangle, andTriangle. Each of these shapes implementsVisitableinterface(or its equivalent) to allow visitor classes to operate on them.
