Kafka thematic architecture – replication, failover, and parallel processing

This article discussed the architecture of the Kafka topic and discussed how partitioning can be used for failover and parallel processing.

Kafka stores topics in logs. The topic log is divided into multiple partitions. Kafka partitions logs across multiple servers or disks. Treat topics as categories, stream names, or feeds.

The topic itself is publish and subscribe style messaging. A topic can have zero or more subscribers named consumer groups. Topics are divided into partitions for speed, scalability, and size.

Kafka breaks the topic log into partitions. Records are stored on a partition. If this key exists, records are usually stored. If the key is missing, it is stored in a loop (the default behavior). By default, the record key determines the partition to which the producer sends the record.

Kafka uses partitions to extend topics on multiple servers for producer writes. In addition, Kafka uses partitioning to facilitate parallel consumers. Records used by consumers are parallel to the number of partitions.

Order in which each partition is guaranteed. If the key is partitioned, all records of the key will be on the same partition, which is useful if you have to replay the logs. Kafka can copy partitions to multiple agents for failover.

Kafka Maintains the order of records in a partition. A partition is an ordered, unchanging sequence of records. Kafka uses partitions to be continuously attached to partitions as structured commit logs. Records in a partition are assigned a sequential ID number called an offset. Offsets identify each record location in a partition. Topic partitioning allows Kafka logs to expand beyond the size appropriate for a single server. A topic partition must fit the server hosting it, but a topic can span multiple partitions hosted on many servers. In addition, topic partitioning is a unit of parallelism — a partition can only be handled by one consumer in a consumer group at a time. Consumers can run in their own processes or threads. If the consumer stops, Kafka partitions over the remaining consumers in the same consumer group.

Kafka theme partition replication

Followers copy the leader, and if the leader dies, they take over. Kafka also uses partitions for parallel consumer processing in groups. Kafka partitions the topic log across servers in a Kafka cluster. Each server handles the sharing of its data and requests through the shared partition leadership.

Kafka uses ZooKeeper to select a copy of the broker partition as the leader.

An agent with partition leadership handles all read and write operations for partitioned records. Kafka copies write operations to subsequent partitions (node/partition pairs) of the primary partition. The followers of synchronization are called ISRs (synchronization replicas). If the partition leader fails, Kafka selects a new ISR as the new leader.

When all isRs of a partition are written to its log, the record is considered “committed”. Only the submitted record can be read from the consumer. Another partition can be owned by the leader of another Kafka agent.

Kafka Theme architecture review

What is ISR?

The ISR is a synchronous copy. If the leader fails, the ISR will be chosen as the new leader.

How does Kafka scale its customers?

Kafka extends consumers by partitions so that each consumer gets a share of the partition. Consumers can have multiple partitions, but a partition can only be used by one consumer in a consumer group at a time. If there is only one partition, there can only be one consumer.

What is a leader? Followers?

The leader performs all reads and writes to a particular topic partition. Followers copy leaders.

How does Kafka perform failover for consumers?

If a consumer in a consumer group dies, the partition assigned to that consumer is divided among the remaining consumers in that group.

How does Kafka perform failover for agents?

If an agent dies, Kafka assigns leadership of its topic partition to the remaining agents in the cluster.

Easter egg bonus

Click on free Java learning notes, interviews, documentation, and videos

Some information is as follows: