Product | technology drops

The author | message service team



Didi Chuxing messaging service team recently opened source its widely used distributed messaging middleware product DDMQ, which is an enterprise-level message queue product dedicated to providing low latency, high concurrency, high availability and high reliability messaging services.

1. Product features

DDMQ has the following excellent features:

  • Low Latency High Throughput: the delay is millisecond, and a single machine can swallow millions of messages.

  • Rich message types: real-time messages, delayed messages, and distributed transaction messages.

  • Massive message storage and message backtracking consumption: RocketMQ and Kafka are supported as real-time message storage engines, and RocksDB is used as delayed message storage engines.

  • Second-level delay message: The delay time for a single message can be set to the level of seconds, and the delay time for ordinary messages and cyclic messages can be set.

  • Multi-language client, provides mainstream development language SDK, including PHP, Java, Go, C/C++, Python, maintains the most easy-to-use High Level form on THE API.

  • Multiple consumption modes: Supports Thrift RPC pull, HTTP push, and third-party storage direct write to consume messages.

  • Support flexible message filtering and conversion function: By using Groovy script to transform and filter the message body on the server side, it can effectively reduce the amount of data transmission between the client and the server, and reduce the load of the client to process messages.

  • Unified Web console: it is convenient for users to manage resources such as Topic. Through the console, users can configure production and consumption limit value, consumption mode, Groovy script, start and stop consumption, reset consumption schedule and other functions.

  • Complete monitoring: Provides the functions of health check and message accumulation alarm.

2. Application scenarios

Message queue, as a necessary infrastructure for modern distributed applications, has a wide range of application scenarios.

  • Peak peel

    In scenarios such as seckilling, traffic spikes in a short period of time, and downstream systems become overloaded or even crash due to lack of protection. DDMQ provides massive stacking capacity and consumption limiting to ensure the smooth operation of downstream systems.

  • Asynchronous decoupling

    Through the loose coupling design of the upstream and downstream systems, the upstream system will not be unavailable due to the downtime of the downstream system. Ensure the normal and stable operation of main process.

  • The order message

    In reality, there are many scenarios that need to ensure order, such as order creation, payment, refund and other processes in order system, all need to ensure order. The sequential consumption capability provided by DDMQ ensures that messages are first in, first out.

  • Transaction message

    In the microservice scenario, the ultimate consistency of distributed transactions can be achieved through DDMQ transaction messages.

3. Architecture design

The following diagram depicts the overall architecture of DDMQ. Mainly including Broker Cluster, Producer Proxy Cluster (hereinafter referred to as PProxy), Consumer Proxy Cluster (hereinafter referred to as CProxy), SDK, Console and other modules.



The Broker Cluster is DDMQ’s message storage layer. RocketMQ was used as the real-time message storage engine (Kafka was also supported), and Chronos was our own delayed message storage engine based on RocksDB.

PProxy is the production proxy service of DDMQ. The Thrift RPC Server is built in. The production SDK sends messages to PProxy through RPC calls, and then PProxy produces messages to specific brokers. In PProxy, we have realized the functions of production flow limiting, retry and message mass production.

CProxy is the consuming proxy service of DDMQ and has built-in Thrift RPC Server. When SDK consumption is selected, the consumer pulls messages from CProxy in pull mode. The PullBuffer in CProxy caches a certain number of messages to be consumed in advance. So the delay in consumption is low. If HTTP consumption is selected, CProxy pushes messages to the callback URL specified by the service. In CProxy, we implemented message filtering (by writing Groovy scripts), message body transformation (Transit), retry, consumption limiting, internal ordering of sequential consumption, and so on.

The Console is the DDMQ Console. Users can apply for resources such as topics and groups on the Console. Data such as Topic will be persisted to MySQL and pushed to Zookeeper; PProxy and CProxy control message production and consumption logic in real time by reading and listening to Topic and Group data on Zookeeper.

The Proxy+SDK architecture selected by DDMQ has the following advantages:

  • It is convenient to realize the multi-language SDK. Since didi uses a large number of technology stacks internally, putting the main logic on Proxy is conducive to reducing the complexity of SDK and greatly speeding up the development of SDK. Currently, Didi internally supports SDK implementation of PHP, Go, C/C++, Java, Python, Node.js and other languages.

  • The storage layer is not aware of services. Because the Proxy layer shields RocketMQ or Kafka, the storage layer can be switched without service awareness.

  • The speed of new function iteration is accelerated, and the development of new function is realized in Proxy layer, which reduces the upgrade frequency of SDK.

4. To summarize

DDMQ has been in stable operation within Didi for more than two years, supporting the stable operation of online ride-hailing, Xiaoju car service, mapping, finance, intelligent driving, intelligent transportation, food delivery and other businesses. Daily message flow reached 100 billion level, and the overall service availability exceeded 5 9.


GitHub storage address:

github.com/didi/DDMQ

Welcome to make more issues