RocketMQ vomit blood summary

architecture

 

The conceptual model

The basic conceptual model and the extended back-end conceptual model

The storage model

 

RocketMQ vomit blood summary

User Guide

  • RocketMQ is a distributed messaging middleware, which was originally developed by Alibaba messaging middleware team and applied to production systems on a large scale to meet the demand of massive online message accumulation. RocketMQ was donated to Apache Open Source Foundation at the end of 2016 as an incubation project, and became Apache’s top project in less than a year. In the early stage, Ali used to develop message system based on ActiveMQ. As the scale of business message increased, the bottleneck gradually appeared. Later, It also considered Kafka. RocketMQ and Kafka are so similar in concept and principle that they are often compared; RocketMQ uses the long polling pull mode by default and supports tens of millions of message stacks on a single machine, making it an excellent application in massive message systems.

  • Multiple NameserVers can be deployed independently of each other. Other roles report status information to multiple Nameservers at the same time to achieve hot backup. NameServer itself is stateless, that is, the status information of brokers and topics in NameServer is not permanently stored, but is periodically reported and stored in memory by each role (NameServer supports persistence of configuration parameters, which is generally not needed).

  • Why not ZooKeeper? ZooKeeper provides powerful features, including automatic Master elections. RocketMQ’s architecture eliminates the need for Master elections and requires a lightweight metadata server. It’s worth noting that NameServer doesn’t offer a ZooKeeper-like watcher mechanism, but instead uses a heartbeat mechanism every 30 seconds.

  • heartbeat

    • A single Broker keeps heartbeat requests to all NamesRVs at an interval of 30 seconds. Heartbeat requests contain information about all the current topics of the Broker. Namesrv checks Broer’s heartbeat. If a Broker has not had a heartbeat for 2 minutes, it takes the Broker offline and adjusts the relationship between Topic and Broker. Namesrv does not actively notify producers and consumers that brokers are down.
    • The Consumer is connected to the Broker over a long period of time and sends heartbeat messages to the Broker every 30 seconds. The Broker checks live consumers every 10 seconds. If a Consumer has no heartbeat for 2 minutes, it disconnects from the Consumer and sends notifications to other instances of the Consumer group to rebalance the Consumer cluster.
    • The producer gets the Topic and Broker mappings from Namesrv every 30 seconds and updates them into local memory. Establish a long connection to all brokers involved in your Topic, with a heartbeat every 30 seconds. The Broker also scans the currently registered Producer every 10 seconds. If a Producer does not send a heartbeat for more than 2 minutes, the connection is disconnected.

  • Namesrv is not too stressful, and the main overhead is maintaining heartbeat and providing topi-broker relational data. However, it should be noted that when the Broker sends a heartbeat to Namesrv, it will bring all the Topic information it is currently responsible for. If there are too many topics (ten thousand levels), the data of each Topic will be dozens of meters in a heartbeat. If the network condition is poor, the network transmission will fail and the heartbeat will fail. Causes Namesrv to mistake the Broker heartbeat for failure.

  • Each theme can set the queue number, automatically create theme by default when four, consumer need to order message sent to the same queue, such as a few of the same order number related consumer need to order message sent to the same queue, order consumption characteristics, there won’t be any queue, two common consumer spending and when consumer amount is less than the queue number, Consumers will consume multiple queues. As for message duplication, it is handled at the consumer end. RocketMQ 4.3+ supports transactional messages that can be used in distributed transaction scenarios (ultimate consistency).

  • About queueNums:

    • The client creates it automatically, the math.min algorithm determines that a maximum of 8 (BrokerConfig) queues will be created, more than 8 can be created/modified via the console, and the Topic configuration is saved in store/config/topics. Json
    • The minimum granularity of consumption load balancing is queue, and the number of consumers should not be larger than the number of queues
    • Read/write queue number (writeQueueNums/readQueueNums) is RocketMQ unique concept, through the console modification. What happens when readQueueNums is not equal to writeQueueNums?
topicRouteData = this.mQClientAPIImpl.getDefaultTopicRouteInfoFromNameServer(defaultMQProducer.getCreateTopicKey(), 1000 * 3);
    if (topicRouteData != null) {
        for (QueueData data : topicRouteData.getQueueDatas()) {
            int queueNums = Math.min(defaultMQProducer.getDefaultTopicQueueNums(), data.getReadQueueNums());
            data.setReadQueueNums(queueNums);
            data.setWriteQueueNums(queueNums);
        }
    }
Copy the code

  • Brokers store Topic information, and topics are composed of queues evenly distributed across multiple brokers. The sending mechanism of Producer ensures that messages are distributed equally across all queues, so that all messages fall equally on each Broker.

  • RocketMQ messages are stored in a ConsumeQueue with a CommitLog. ConsumeQueue stores very little data. Message bodies are read and written via CommitLog. If a message has data only in CommitLog but not in ConsumeQueue, the consumer cannot consume it, and RocketMQ’s transaction message implementation takes advantage of this.

    • CommitLog: Set up a ConsumeQueue for the CommitLog. Each ConsumeQueue corresponds to a MessageQueue (in the conceptual model), so if there is a CommitLog, ConsumeQueue even if data is lost, You can still recover.
    • ConsumeQueue: a logical Queue of messages that stores the start offset, log size, and MessageTag hashCode of the Queue in the CommitLog. Each Queue in a Topic has a corresponding ConsumeQueue file. For example, if there are three queues in a Topic, the message index in each Queue will have a number starting from 0 and increasing upwards. From this point, the concept of offset can be used to define the consumption of the Consumer side.

  • High performance of RocketMQ lies in CommitLog, zero copy, and jump read (try to hit PageCache). High reliability lies in disk flush and Master/Slave. In addition, the failure of all Nameservers does not affect running brokers, producers, and consumers.

  • Sending message load balancing, thread safe sending messages (multiple instances can send messages in an infinite loop), and consumer load balancing in a clustered consumption mode, combined with the aforementioned high performance read and write capabilities, contribute to RocketMQ’s high concurrent read and write capabilities.

  • When flush and master/slave synchronization are both asynchronous (the default), if the broker process hangs (such as a restart), messages are still not lost because persist is performed when the broker is shutdown. When the physical machine is down, there is a risk of message loss. In addition, after the master hangs, the consumer consumes messages from the slave, but the slave cannot write messages.

  • RocketMQ has good dynamic scaling (non-sequential messages), which is reflected in both the Topic and Broker dimensions.

    • Topic dimension: If the message volume of a Topic is very large, but the water pressure of the cluster is still very low, the number of queues of this Topic can be expanded. The number of queues of this Topic is proportional to the sending and consumption speed.
    • Broker dimension: If the cluster water level is high and needs to be expanded, you can simply deploy the Broker on machines. The Broker registers with Namesrv. Producer and Consumer discover a new Broker through Namesrv, and immediately connect to the Broker to send and receive messages.

  • Producer: fails and tries twice by default. The sync/async; ProducerGroup. In transactional messaging, if the producer that sent the message fails before committing/ROLLBACK, the broker will look back at other instances of the ProducerGroup some time later to confirm that the message should commit/rollback

  • Consumer: DefaultPushConsumer/DefaultPullConsumer, push and pull with the, USES a long polling way; With CLUSTERING, a message is consumed by one instance of the ConsumerGroup, but can be consumed by many different Consumergroups. With BROADCASTING, a message is consumed by all instances of the ConsumerGroup.

  • DefaultPushConsumer: The Broker receives a request for a new message and if there is no new message in the queue and is in no hurry to return, checks the status continuously through a loop, waitForRunning for a while (5s) at a time, then checking at check. When there is no new message and the third check exceeds suspendMaxTimeMills(15s), the null result is returned. While waiting, the Broker calls notifyMessageArriving when it receives a new message to return the requested result. The core of “long polling” is that the Broker holds (suspends) incoming requests from clients for a short period of time. When new messages arrive, the Broker immediately returns the message to the Consumer using the existing connection. The initiative of “long polling” is still in the hands of the Consumer, the Broker will not actively push the Consumer even if there is a large backlog of messages. The limitation of the long polling approach is that it requires resources to Hold Consumer requests and is suitable for situations where the number of client connections is controlled, such as message queues.

  • DefaultPullConsumer: The PullConsumer has more autonomy and flexibility because the user needs to iterate through the MessageQueue and save the Offset themselves.

  • For non-sequential messages in cluster mode, consumption failures are retried 16 times by default, with a delay of 3 to 18. (messageDelayLevel = “1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h”)

  • MQClientInstance is the underlying class for the various types of Consumer and Producer clients that interact with NameServer and Broker. If you do not manually specify instanceName when creating a Consumer or Producer type, there will only be one MQClientInstance object in the process. That is, when a Java application needs to connect to multiple MQ clusters, you must manually specify different instanceName. It should be mentioned that when consumers (different JVM instances) are all on the same physical machine, if instanceName is specified, consumption load balancing fails (each instance will consume all messages). In addition, when simulating cluster consumption in a JVM, you must specify a different instanceName, or the ConsumerGroup will be prompted to exist at startup.

More

  • RocketMQ architecture module parsing
  • RocketMQ has high concurrent reads and writes
  • RocketMQ consumption failure retry in-depth analysis
  • RocketMQ Field and Principles douban.com
  • Code combat github.com/javahongxi/…