1. Introduction:

Apache TubeMQ is a new generation of MQ that Tencent donated to Apache after open source in 2019. It originates from the actual production environment of Tencent and focuses on high-performance storage and transmission of massive data. Today, MQ is the Red Sea (only Apache has 5 MQ), compared with many open source MQ components, What are some of the features of Apache TubeMQ, in what situations do we apply it, and what are the benefits of using this product? This article would like to introduce from this point.

2. When is Apache TubeMQ applicable?

To sum up, the main scenarios are as follows:

  1. The amount of data reported is too large for normal MQ to support;
  2. Regular MQ can resist, but consumes too much machine resources, or the system is unstable and difficult to maintain
  3. Pure JAVA technology stack, easy to self-transformation and version maintenance, even if there is a problem with the version can quickly coordinate resources immediately stop loss;
  4. As long as the function of production and consumption, do not need transaction message, Exactly Once and other advanced functions;
  5. The system is highly automated and tolerates small data loss in extreme cases.

3. Why is Apache TubeMQ suitable for these scenarios?

Personally, I think there are three main points:

3.1. Due to actual production environment requirements:

Online 8 years over 40 trillion data magnitude service precipitation, Apache TubeMQ from Tencent’s actual production environment, initially we also use Apache Kafka as most business samples for data services, but because Kafka server using Scala implementation, reading, maintenance is more difficult, The problem can not be solved immediately, at the same time there are some unreasonable Kafka design, the use of more complex; Under the pressure of the live network, based on the idea of Kafka and the actual needs, we have developed a distributed message middleware idea of high reliability, high performance, low latency and low cost, which is based on server-side management and control to provide external services in SAAS mode. We have also built and continuously improved the product centering on this project positioning.

Apache TubeMQ has been serving big data scenarios for nearly 8 years, and has reached a throughput level of 40 trillion + per day. It has formed a stable and easy to maintain time-tested product, serving businesses including wide access, PCG, wechat, etc. Our largest cluster size is over 300 brokers. Each Broker is configured with 800 topics, and the consumer group is nearly 3K in size.

Based on the needs of various businesses within Tencent, we are sure that MQ will be suitable for use in similar business scenarios.

3.2. Strong enough high throughput performance indicators:

Tubemq_perf_test_vs_Kafka. In general, in the TS60 model (10 GIGABit network card, 64 GB memory, 24 core CPU, 12TSATA hard disk), 1000 topics are configured, each topic is configured with 10 partitions, and each message is 1K in size. On the premise of producing two copies for consumption, a single Broker can achieve more than 10W TPS (Short for TransactionsPerSecond). Number of successful request responses per second), with an end-to-end delay of less than 10ms. This is just a conservative metric, and I recommend that you run your own comparison tests in the same scenario. You can run the comparison tests against any external MQ, and the results will be different.

Many MQ have said that the system can achieve tens of millions of TPS, even 1ms end-to-end delay, we give this index we think too low? I would like to express that the provision of indicators should be based on the supporting test premise. Under the given clear system configuration and production and consumption load, if TPS of ten million level is to be achieved, the cluster only needs less than 100 brokers to reach the horizontal expansion under the premise of more than 10W TPS per machine. Without the above system configuration, production and consumption load, tens of millions of TPS, machine magnitude would be even lower.

3.3. Sufficiently transparent openness:

Apache TubeMQ can achieve the characteristics we described, mainly from its TubeMQ architecture built according to business scenarios. We not only use this system to support services internally, but also open source it, incubate projects according to Apache rules, so that more external companies can use it, and reduce costs through it. Improve system performance and stability. The system is stable enough, students who have MQ experience can build it according to the guidance on the official website. The system is completely open source and built in pure JAVA. Even if the original team does not maintain it, there are enough technical talents in the market to support its improvement. Operate the community according to the Apache community specifications, as long as you have any suggestions for improvement, verification is effective can be integrated into the system, and the original team is domestic personnel, communication is more convenient.

4. What is the follow-up development roadmap of Apache TubeMQ?

One-stop flow data service platform, we want the whole open source data reporting platform in this project, the data reporting involves the collection, gathering, storage, forwarding the module in the form of a plug-in, such as organic integrate (even TubeMQ, also can undertake replacement) in this platform, based on this system, the business only need to publish and subscribe to data, You can easily build analytics and applications based on streaming data. We are building each part of the module, welcome everyone to build together.

5. How to join the project?

It can be co-built according to the Apache project. For details, please refer to how to join TubeMQ