In fact, the answer to this kind of question, to put it bluntly, at least do not ask you to read the source code of that technology, at least you probably know the basic principle of that technology, core components, basic architecture, and then refer to some open source technology to design a system to say it is good

For example, with the message queue system, let’s think about it from the following perspectives

(1) First of all, mq has to support scalability, which means that throughput and capacity can be increased quickly when needed. Broker -> topic -> partition, each partition is a machine that stores a portion of the data. If resources are not enough now, it is easy to add partition to topic, and then do data migration, add machine, not can store more data, provide higher throughput.

(2) Secondly, you have to consider whether the MQ data should be landed on disk, right? That must be, to drop the disk, in order to ensure that other processes die data is lost. How do you drop the disk when you drop it? Sequential write, so that there is no addressing overhead of random disk reads and writes, sequential disk read and write performance is very high, this is the idea of Kafka.

(3) Second, do you think about the availability of your MQ? For this, refer to kafka’s high availability guarantee mechanism as discussed in the usability section. Multiple copies -> leader & follower -> Broker hangs and re-elects the leader to serve.

(4) Can support the loss of data 0? Yes, refer to the Kafka zero data loss scheme we talked about earlier

In fact, an MQ must be very complex, in fact, this is an open question, is to see if you have the overall thinking and design ability from an architectural perspective.

If you are still not clear, please refer to the previous article, “the purpose of the message queue, advantages and disadvantages, technology selection, how to guarantee the high availability of the message queue,” how to ensure that information is not repeated consumption “, “how to prevent loss of data queue,” “how to ensure the message in order to perform the backlog of messages in the message queue?”