In common

  • The underlying principles are similar, and RocketMQ borrows from Kafka’s design.
  • Both make use of the Page Cache mechanism of the operating system, and reduce the randomness of read and write as much as possible by sequential I/O, so as to concentrate the read and write in a very small range and reduce the Page missing interruption, thus reducing the access to the disk and improving the performance.

The difference

Storage form

  • Kafka uses partitions, one file per partition per topic. Sequential writing, timing brush disk. However, if there are too many partitions in a single broker, sequential writes degenerate into random writes, and too many dirty pages in the Page Cache trigger frequent Page miss interrupts, resulting in significant performance degradation.
  • RocketMQ uses CommitLog+ConsumeQueue. All topics of a single broker are written in CommitLog order, and the Page Cache only needs to keep the latest Page. Meanwhile, each queue in each topic has a corresponding ConsumeQueue file as an index. ConsumeQueue occupies very little Page Cache and has little impact on disk flushing.

Storage Reliability

  • RocketMQ supports asynchronous flush, synchronous flush, synchronous Replication, and asynchronous Replication.
  • Kafka uses asynchronous flush, asynchronous Replication.

The order message

Both Kafka and RocketMQ support single-topic partitioning order only. RocketMQ officially claims to support strict ordering, but by using a single partition.

Delay message

  • RocketMQ supports delay messages with fixed delay levels that can be configured.
  • Kfaka does not support delayed messages.

The message to repeat

  • RocketMQ supports only At Least Once.
  • Kafka supports At Least Once, Exactly Once.

The message filter

  • RocketMQ performs filtering on the Broker side, supporting tag filtering and custom filtering logic.
  • Kafka does not support message filtering on the Broker side and requires custom implementation on the consumer side.

Message failed retry

  • RocketMQ supports periodic retries, with each retry interval gradually increasing.
  • Kafka does not support retry.

DLQ (Dead Letter Queue)

  • RocketMQ logs all consumption failures via DLQ.
  • Kafka DLQ. Third-party tools such as Spring implement this by writing failure messages to a dedicated topic.

Back in the consumption

  • RocketMQ supports backtracking consumption in the same way that Kafka does.
  • Kafka needs to find the offset based on the timestamp and start consuming from the offset.

The transaction

  • RocketMQ supports transactional messaging, using a two-phase commit +broker timed lookup. However, only producer and broker consistency can be guaranteed, and only one-way retries between broker and consumer can be ensured. The final consistency is guaranteed.
  • Kafka has support for transactional messages since version 0.11, implementing the Exactly Once message semantics (single partition) in addition to final consistency.

Service discovery

  • RocketMQ implements namesRV itself.
  • Kafka uses ZooKeeper.

High availability

  • RocketMQ controls granularity only at the Broker on a high availability design. High availability is ensured through master-slave replication.
  • Kafka controls the granularity of high availability on partitions. The Leader and Replica partitions of each topic can load balance storage across all brokers.
  • This Kafka design has the following advantages over a master-slave replication design like RocketMQ:
    • There is no need to set up slave brokers in Kafka. All brokers can send and receive messages. Load balancing is also better.
    • Partition elections in Kafka are automated, and RocketMQ needs to specify its own master-slave relationship.
    • If the number of copies in the Kafka partition is set to N, n-1 node failures can be tolerated. When a fault occurs, only the zone leader election is required, which is highly efficient.

The resources

  • Kafka and RocketMQ storage differences
  • RocketMQ high-performance underlying storage design
  • Kafka transaction feature analysis
  • Kafka design Analysis (8) – Exactly Once semantics and transaction mechanism principle
  • Messaging middleware – RocketMQ vs. Kafka features
  • A quick comparison between Kafka and RocketMQ
  • Problems to be solved with RocketMQ’s messaging middleware