In the previous article we analyzed how to use message-oriented middleware to decouple the two systems. XIAOXI
At the same time, we mentioned that using message-oriented middleware also makes it easier for a piece of data to be subscribed to simultaneously by multiple systems for different purposes.
One of the current architectures is shown below.
In this figure, we can clearly see that the real-time computing platform publishes a piece of data to the messaging middleware, and then the following:
The data query platform subscribes to this data and stores it in its local database cluster and cache cluster, and then provides the data query service externally
The data quality monitoring system will monitor the calculation results according to certain business rules. If there is any data calculation error, it will immediately alarm
The data link tracking system will collect the calculation result as a link node, and collect the whole calculation link of a data and assemble a series of data calculation link to land storage. Finally, if a certain data calculation error, the problem can be immediately backtracked through the calculation link
From the review above, it has become clear that in the above scenario, using message-oriented middleware can decouple and implement the message “Pub/Sub” model for publishing and subscribing messages.
How to implement a Pub/Sub model in which data is subscribed to by multiple systems based on RabbitMQ messaging middleware?
2. Queue consumption model based on message middleware
RabbitMQ’s basic queue-consumption model is used to support RabbitMQ’s basic queue-consumption model. You can see that RabbitMQ has a queue in which producers send data and messages are queued in the order in which they are sent.
Now let’s say we have four pieces of data in the queue, and we have two consumers consuming the data in the queue.
At this point, each consumption will be evenly allocated to 2 pieces of data, that is to say, 4 pieces of data will be evenly allocated to each consumer, each consumer is only processing part of the data, this is a typical queue consumption model.
This article provides the RabbitMQ code implementation of the basic queue-consumption model, as well as how to ensure that data is not lost when consumers are down, and how to persist queues and messages in RabbitMQ clusters.
3. “Pub/Sub” model based on message middleware
In addition to the basic model mentioned above, message-oriented middleware can also implement a “Pub/Sub” model, which is a “Publish/Subscribe” model. Pub is Publish and Sub is Subscribe.
This model allows multiple systems to consume a single piece of data simultaneously, which means that every piece of data you publish is broadcast to each system, as shown below:
That is to say, we want to achieve the effect of the above: real-time computing platform release a series of data into the message middleware, and data query platform, data quality monitoring system, the data link tracking system, will subscribe to the data, will complete consumption to the same data, each system can according to their own needs to use the data.
So what about the so-called “Pub/Sub” model based on RabbitMQ?
4. What exactly is Exchange in RabbitMQ?
In fact, producers are not allowed to post messages directly to a queue in RabbitMQ, but to a special component within RabbitMQ called an exchange, which you can probably interpret as a message routing component.
That is, any message sent from the real-time computing platform to RabbitMQ is received by an Exchange.
The Exchange then decides which queue to forward the message to according to certain rules, which is actually a core message model in RabbitMQ.
Look at the picture below to get a sense of it.
5. Default Exchange
You might say, when I send messages to RabbitMQ I don’t use any exchange, but why send messages to the queue anyway?
That’s because you’re using the default Exchange, which routes messages directly to the queue you specify, so simply using the queue consumption model eliminates the need for exchange.
This is one of the ways to make a message persistent that I showed you earlier.
Notice that the first argument is an empty string, which means that the default exchange will send the message, and then it will route the message to our queue.
6. Post the message to fanout Exchange
There are many types of Exchange components in RabbitMQ, such as direct, Topic, headers and fanout, with the last fanout in this article.
The Exchange component is actually very simple. You can create a Fanout exchange, bind multiple queues to that exchange, and then once you send a message to that Exchange, it will route the message to all its bound queues.
To create an Exchange, use the following code. For example, in the real time computing platform (producer) code, add the following section to create a Fanout exchange.
The first parameter is called “RT_compute_data”, which is the name of Exchange. Rt is short for “RealTime”, which means the result data of real-time computing system.
The second parameter defines the exchange type as “fanout”.
channel.exchangeDeclare(rt_compute_data, fanout);
We then use the following code to post messages to the Exchange component we created:
You’ll notice that the message is sent to the exchange, but which queue is it routed to? At this point, we are not sure yet, but let consumers bind their queues to the exchange themselves.
7. Bind your own queue to Exchange
The consumer code is also modified. We turned off the autoAck mechanism here and then manually ack it each time.
In the above code, each consumer system will be different in that each consumer will need to define its own queue and bind it to Exchange.
For example, the queue of the data query platform is Rt_compute_data_query, that of the data quality monitoring platform is Rt_compute_data_monitor, and that of the data link tracing system is Rt_compute_data_link.
Each system that subscribes to this data actually has its own queue, which is then routed by Exchange to calculate all the data produced by the platform in real time.
And because of the multi-queue mode, each system can deploy consumer clusters for data consumption and processing, which is very convenient.
8. Overall architecture diagram
As shown in the figure above, the real-time computing platform posts messages to “rT_compute_data”, an “exchange”, but it does not specify which queue the exchange should route the messages to because it does not know.
Data query platforms, data quality monitoring systems, and data link tracking systems can then declare their own queues and bind them to Exchange.
Because of the binding between queue and Exchange, it is specified here by the platform that is subscribing to the data itself. And because the Exchange is of fanout type, as soon as it receives data, it routes it to all queues bound to it, so that each queue has the same copy of data for consumption by the corresponding platform.
Moreover, for each platform’s own queue, it can also deploy a consumption service cluster to consume its own queue, and the data in its queue will be evenly distributed to each consumption service instance for processing, and each consumption service instance will obtain part of the data.
Is this the implementation of a “Pub/Sub” model of different systems subscribing to a copy of data?
Of course, RabbitMQ also supports a variety of different types of exchange, which can implement a variety of complex functions, and we will show you the use of message-middleware technology through a practical case study of online system architecture.