Message queues – What exactly are producers and consumers

What is a message queue?

I don’t know if message queuing sounds like a very advanced technology when you look at it, but it sounds pretty cool to me.

Message queues, which are often referred to as MQ(Message Queue), are pretty straightforward. Forget the word Message and look at queues. This is a queue that you’re all familiar with.

A queue is a first-in, first-out data structure.

In Java, quite a few queues have been implemented.

So why message queue (MQ) middleware?? In fact, this problem is very similar to when I learned Redis before. Redis is an in-memory database stored in key-value format. Why Redis when we can use an implementation class like HashMap to achieve similar results? Is Redis Really That Fast?

At this point, you can guess why message queuing (MQ) middleware is used, which will be added below. A message queue can be simply interpreted as putting data to be transmitted in a queue.

Science:

  • Putting data into a message queue is called a producer
  • Fetching data from a message queue is called a consumer

There are many message queue products in the market, such as the old ActiveMQ, RabbitMQ, Kafka which is the most popular in my view, ZeroMQ, RocketMQ donated by Alibaba to Apache, Even NoSQL databases such as Redis support MQ functionality. All in all, there are more than a dozen well-known products.

What are the benefits of using message queues? Let’s look at the following scenario

The decoupling

Take the common order system as an example, after the user clicks the “order” button, the business logic may include: deducting inventory, generating corresponding documents, sending red envelopes, sending SMS notification. In the early stage of service development, these logics may be executed synchronously. As the number of orders increases, the system service performance needs to be improved. In this case, some operations that do not take effect immediately can be separated and executed asynchronously, such as sending red packets and sending SMS notifications. MQ can be used in this scenario, where a message is sent to MQ after the main process of placing an order (such as inventory reduction and receipt generation) is complete to allow the main process to complete quickly, and MQ’s message is pulled by a separate thread (or pushed by MQ), and when a red envelope or SMS message is found in MQ, Execute the corresponding business logic.

To put it simply, a service needs to call the interface or method of B service to transfer data. At this time, if message queue is used, A service only needs to send data to the message queue, and B service can take out the corresponding data from the message queue, thus realizing decoupling

asynchronous

In fact, asynchrony means that after service A sends data to the message queue, it can return data or perform other processes without waiting for service B to process data, thus improving the efficiency of some business scenarios using asynchrony

Peak clipping/current limiting

Let’s do another scenario. Now we do a big rush once a month, and the concurrency during the rush can be quite high, like 3000 requests per second. Suppose we now have two machines processing requests, and each machine can only handle 1,000 requests at a time.

Peak clipping scene

That extra 1,000 requests could have crashed our entire system… So, there is a way we can write to the message queue:

Write to the message queue, from which the system receives the request

System B and SYSTEM C get data from the message queue according to the number of requests they can handle, so that even if there are 8000 requests per second, they just put the requests in the message queue, and the messages to the message queue are controlled by the system itself, so they don’t crash the whole system.

What’s the problem with using message queues?

As you can see from our scenario above, message queues can do quite a lot of things.

Let’s go back to the beginning of the article: “The JDK already has quite a few queue implementations, but we still need message queue middleware.” In fact, very simple, JDK implementation of queue types although there are many kinds, but are simple memory queue. Why do I say the JDK is a simple memory queue? Let’s take a look at what might be considered to implement message queues (middleware).

High availability

Whether we use message queues for decoupling, async, or peak-shaving, message queues certainly cannot be stand-alone. Think about it, if it’s a stand-alone message queue, if that machine goes down, then our entire system is almost unusable.

In case the standalone queue fails in case the standalone queue fails so when we use message queues in our project, it has to be clustered/distributed. To cluster/distribute, you want the message queue to be supported off the shelf, rather than writing your own code to implement it manually.

Data loss problem

We write data to the message queue, and systems B and C hang up before they can fetch the data from the message queue. If nothing is done, our data is lost.

Data loss As anyone who has studied Redis knows, Redis can persist data to disk and recover data from disk if Redis fails. Similarly, the data in the message queue needs to be stored elsewhere to minimize data loss.

Where does that exist?

Disk? Database? Redis? Distributed file system? Synchronous or asynchronous storage?

How do consumers get the message queue data?

How do consumers get data from message queues? There are two ways:

  • The producer puts the data into the message queue, the message queue has the data, and actively asks the consumer to fetch it.
  • The consumer constantly rotates the message queue to see if there is new data, and if there is, consumes it (commonly known as pull).
  • other

In addition to these, we have to consider various issues when using:

  • What if messages are consumed repeatedly?
  • I want to make sure that the messages are absolutely sequential how do I do that?
  • ……

Despite the benefits of message queues, we also found that introducing message queues increased the complexity of the system. There are already several message queue wheels on the market, each with its own characteristics, and which MQ to choose is a matter of choice.

This time I’ll start with RabbitMQ

RabbitMQ

RabbitMQ is an open source implementation of AMQP developed in the Erlang language.

  • AMQP: Advanced Message Queue. It is an open standard of application layer protocol, designed for message-oriented middleware. The client and message-oriented middleware based on this protocol can transfer messages, regardless of product and development language conditions.

RabbitMQ originated in the financial system and is used to store and forward messages on distributed systems. RabbitMQ is very easy to use, scalable, and highly available. Specific features include:

  • RabbitMQ uses mechanisms to ensure Reliability, such as persistence, transport confirmation, and release confirmation.

  • Flexible Routing Is used to route messages through Exchange before they are queued. For typical routing, RabbitMQ already provides some built-in Exchange. For more complex routing capabilities, you can bind multiple Exchanges together or implement your own Exchange through a plug-in mechanism.

  • Message Clustering (Clustering) RabbitMQ servers can form a cluster to form a logical Broker.

  • Highly Available Queues can be mirrored on machines in a cluster, making them usable even if some nodes fail.

  • RabbitMQ supports multiple message queuing protocols, such as STOMP and MQTT.

  • RabbitMQ supports almost all common languages, such as Java,.NET, Ruby, and so on.

  • RabbitMQ provides an easy-to-use user interface that enables users to monitor and manage many aspects of a message Broker.

  • If a message is abnormal, RabbitMQ provides a Tracing mechanism for the user to find out what happened.

  • RabbitMQ provides many plug-ins to extend RabbitMQ in many ways, or you can write your own.

Conceptual model in RabbitMQ

A message model

All MQ products are the same process in model abstraction: consumers subscribe to a queue. Producers create messages, publish them to queues, and then send the messages to listening consumers.

RabbitMQ basic concepts

This is the simplest and most abstract description, but RabbitMQ can be explained in more detail. As mentioned above RabbitMQ is an open source implementation of the AMQP protocol, so it is actually a basic concept within AMQP:

“Message” is an anonymous Message that consists of a header and a body. The body of the message is opaque, and the header consists of a set of optional attributes, including routing-key, priority (priority over other messages), delivery-mode (indicating that the message may require persistent storage), and so on.

Publisher is a producer of messages and a client application that publishes messages to the exchange.

An Exchange Exchange that receives messages sent by producers and routes them to queues in the server.

Binding The association between message queues and switches. A binding is a routing rule that connects a switch to a message queue based on a routing key, so a switch can be thought of as a routing table made up of bindings.

Queue Message Queue, used to hold messages until they are sent to consumers. It is the container and destination of the message. A message can be put into one or more queues. The message remains in the queue, waiting for the consumer to connect to the queue to pick it up.

Connection A network Connection, such as a TCP Connection.

Channel an independent two-way data Channel in a multiplexing connection. The channel is built on the real TCP connection and the virtual connection, AMQP command is sent through the channel, whether it is to publish messages, subscribe queue or receive messages, these actions are completed through the channel. Because it is very expensive for an operating system to establish and destroy TCP, the concept of a channel was introduced to reuse a TCP connection.

Consumer, the Consumer of a message, represents a client application that retrieves a message from a message queue.

Virtual Host A Virtual Host that represents a batch of switches, message queues, and related objects. A virtual host is a separate server domain that shares the same authentication and encryption environment. Each Vhost is essentially a mini RabbitMQ server with its own queue, switch, binding and permission mechanism. Vhost is the basis of the AMQP concept and must be specified at connection time. The default vhost for RabbitMQ is /.

The Broker represents the message queue server entity.

The last

This article focuses on what a message queue is, what benefits a message queue can bring, and what issues a message queue might involve. Hope to bring you certain help.

If you want to know more about Java dry goods, you can follow my official account :Garnett’s Way of Java, reply information to receive a lot of learning resources, looking forward to your arrival!