Kafka is a popular message-oriented middleware. This article mainly introduces the basic components of Kafka and its related principles

Basic architecture

  • Broker: Message-oriented middleware processes nodes. A Kafka node is a Broker. One or more brokers can form a Kafka cluster
  • Topic: Kafka categorizes messages by Topic, and each message published to the Kafka cluster needs to be assigned a Topic
  • Producer: A message Producer, a client that sends messages to the Broker
  • Consumer: Message Consumer, a client that reads messages from the Broker
  • ConsumerGroup: Each Consumer belongs to a specific ConsumerGroup. A message can be sent to multiple Consumer groups, but only one Consumer in a ConsumerGroup can consume the message
  • Partition: A physical concept. A topic can be divided into multiple partitions, each of which is internally ordered

The offset

Kafka uses offsets to ensure the order of messages within a partition. The order of offsets does not cross partitions. After Kafka0.10, use a special topic __consumer_offset to save offsets. The __consumer_offset log is kept compact, which means that the topic collates messages with the same key

__consumer_offset holds three types of messages:

  • Consumer Group group metadata messages
  • Consumer Group shift message
  • After the news

kafka log

storage

Each Partition corresponds to a log directory: {topicName}-{partitionID}/. There are multiple log segments under the directory. The LogSegment file consists of an. Index file and a. Log file

send

Using the Page cache to read files sequentially, the operating system prereads data to the Page cache. At the same time, log files are mapped directly to the virtual address space using Mmap

Zero copy: Message data is sent directly from the Page cache to the network. The normal file reading process is shown below, with two copies of memory between user and kernel mode

A copy of the

  • Each partition has an IN-sync Replicas (ISR)

  • Each replica in the ISR collection is synchronized with the leader, and those not in the ISR collection cannot be synchronized

  • Only replicas in the ISR are eligible to be elected leader

  • Messages written by Producer are considered “committed” only if they are received by all replicas in the ISR.

  • Log End Offset: The Offset of the latest piece of data written to Kafka by Producer

  • High Watermark: The latest data in the other REPLICas has been successfully backed up to the offset, that is, the data between the Log End offset and the High Watermark has been written to the leader of the partition. But it has not been successfully backed up to another Replicas

Copy synchronization process:

Controller

A Controller is similar to a cluster master and manages the following blocks:

  • Broker on-line and offline processing
  • Partition expansion of topic, handling partition copy allocation and leader election

The Controller is elected by preempting zK temporary nodes through brokers, and the Controller establishes long connections to all brokers

Controller Manages the partition leader election in the following ways:

Election way instructions
OfflinePartitionLeaderSelector Triggered when the leader is offline
ReassignedPartitionLeaderSelector Partition copy redistribution triggered after data synchronization is complete
PreferredReplicaPartitionLeaderSelector Optimal leader election, triggered when manual or automatic leader balance scheduling is triggered
ControlledShutdownLeaderSelector Triggered when the broker sends a ShutDown request to actively ShutDown the service

Message idempotent

Question:

  • Before 0.11.0, producer guaranteed at least once
  • The Server does not know whether the request has been processed or not (no previous status information is recorded) when the request is retried. Therefore, repeated data requests may be sent. This is due to the duplication of data caused by Kafka’s own mechanism (request retry on exceptions)

Solution:

  • PID (Producer ID) : identifies each Producer client
  • Sequence numbers: Each message sent by the client carries the corresponding sequence number. The Server checks whether the data is repeated based on the sequence number

Rebalance

5 things that happen in Kafka Rebalance:

  1. New consumers have joined the Consumer Group.
  2. Some customers went offline. A consumer does not need to be logged out. For example, if a consumer does not send a HeartbeatRequest to a GroupCoordinator for a long time due to a long GC or network delay, the GroupCoordinator considers the consumer logged out.
  3. Some consumers voluntarily quit the Consumer Group.
  4. The number of partitions for any Topic subscribed to by the Consumer Group changed.
  5. The consumer calls unsubscrible() to unsubscribe from a Topic.

Kafka manages rebalance operations through the GroupCoordinator

  • A GroupCoordinator is a component in KafkaServer that manages Consumer groups
  • GroupCoordinator Adds a Watcher to ZooKeeper
  • Obtaining a GroupCoordinator: The consumer sends the ConsumerMetadataRequest to any Broker in the Kafka cluster
  • A consumer connects to a GroupCoordinator and periodically sends a HeartbeatRequest
  • If IllegalGeneration is abnormal in the HeartbeatResponse, the GroupCoordinator initiates a Rebalance operation. The Rebalance process is divided into two processes.

Join Group:

  1. The Consumer first sends a JoinGroup quest request to the GroupCoordinator, which contains information about the Consumer
  2. A GroupCoordinator selects a consumer to become the Group Leader, encapsulates it as a JoinGroupResponse, and returns it to each consumer
  3. Only the JoinGroupResponse received by the Group Leader encapsulates the information of all consumers. The Group Leader allocates partitions based on the information of consumers and the selected partition allocation policy.

Sync Group:

  • Each consumer sends a SyncGroupRequest to the GroupCoordinator, but only the Group Leader’s SyncGroupRequest request contains the partition allocation results
  • A GroupCoordinator generates a SyncGroupResponse based on the partition allocation result of the Group Leader and sends the SyncGroupResponse to all consumers
  • After consumers receive SyncGroupResponse, they parse it and get the partition assigned to them