In my recent study of Kafka, I have found that its core concepts are somewhat different from RocketMQ. Let me talk about the differences between Kafka partitions and RocketMQ queues.

RocketMQ queue

RocketMQ has several queues for each topic, distributed across brokers in the cluster as follows:

The queue is abstracted as a consumer queue in the broker. In clustered mode, only one consumer can subscribe to each queue and each consumer group, but one consumer can consume multiple queues, which ensures that messages are not consumed twice in clustered mode, as shown in the following figure:

In RocketMQ open source, when creating a topic, the cluster creation mode is used to specify the number of queues for the topic in the cluster. For example, if there are 2 brokers in the cluster, we select 4 queues to create the topic, and 4 queues will be created for the topic in each broker. The topic would have 4 x 2 queues in the cluster, and the downside is that you can’t control the exact number of queues, but that’s not a problem.

RocketMQ implements message redundancy in Master/Slave mode. In the production environment, the multi-master/Slave mode is also used to build clusters. There are two types of queue data synchronization between Master and Slave: synchronous replication and asynchronous replication.

Thus, RocketMQ is consumed by queues, and queue data is made redundant through master-slave synchronization.

Kafka partitions and replicas

Partition is one of the core concepts of Kafka. Partition mechanism makes Kafka have the ability of horizontal expansion. On its partition, Kafka can also set a copy of the partition, which greatly improves the reliability of Kafka messages.

In Kafka, a topic can have more than one partition in a cluster, and each partition can have only one consumer in each consumer cluster to subscribe to, but a consumer can consume multiple queues, just like the RocketMQ queue:

We can improve the message throughput by adjusting the number of theme zones. We can also set the replica factor for the zones, that is, how many replicas does the zone have in the cluster. The replicas are divided into leader replica and follower replica. Data is synchronized between them through the IN-sync Replica (ISR) and the leader Replica.

When creating a topic topic-demo, you can specify the number of partitions for the topic in the cluster, as well as the replica factor size:

--partitions 4 --replication-factor 2Copy the code

The above parameters create 4 partitions for this topic with a copy factor of 2. I now have a cluster with 3 brokers:

nodel brokerid=O node2 brokerid=l node3 brokerid=2Copy the code

According to Kafka’s default allocation:

Node1: topic - demo - 0, topic - demo - 1Node2: topic-demo-1, topic-demo-2, and topic-demo-3Node3: topic-demo-0, topic-demo-2, and topic-demo-3Copy the code

Have you noticed that each partition is allocated a copy, and the distribution of partitions is as balanced as possible, and the copies of partitions are not on the same node as possible. If we set the copy factor to 3, the principle is the same.

Unlike RocketMQ queues, where Kafka partitions can be set exactly as many as they want in a cluster and then distributed randomly and evenly across the cluster, and can define the number of copies, RocketMQ’s master-slave mode appears to have only one copy. Of course, in order to save storage space and improve performance, the general replica factor setting of 2 is sufficient.

Kafka’s partitioning and replication mechanism is more flexible and reasonable than RocketMQ’s queue and master-slave synchronization.

The recent hot,

Why does RocketMQ guarantee consistency of subscriptions?

High availability design for RocketMQ message sending

I have a few more thoughts on the mechanics of creating the RocketMQ Topic

In-depth analysis of the creation mechanism of RocketMQ Topic

RocketMQ source code analysis routing center

RocketMQ’s consumption model

Long press to subscribe

I’m looking at it. It’s full of air