Start thinking about
- Why is MQ used in the system? Must it be used in distributed systems?
- What middleware does MQ have? What are their characteristics?
- Are there any problems with the benefits MQ brings to the system? How to solve it?
During ali’s interview, the interviewer asked several questions about MQ:
- What is the role of MQ in your project?
- Why choose this MQ as messaging middleware?
- What about repeated consumption?
- How do YOU ensure that messages are consumed?
- Do you have any other problems? Then think about it first with a question. If you have a good idea, you can leave it in the comments section and share it with us.
The use of message-oriented middleware in systems
I previously wrote an article on rocketMQ for distributed locking, “Springcloud + rocketMQ for Distributed Transactions”, but this is not a basic feature of MQ. Not all MQ features are available.
What role does MQ play in a system? Leaving aside the basics of publishing and subscribing, here are a few things:
- Distributed system decoupling
- Asynchronous processing of business that does not require immediate return
- Peak load reduction, no direct access to services, ease service pressure, increase performance
- logging
Distributed system decoupling
In distributed systems, either calls are made through REST or RPC calls such as Dubbo, but some scenarios require a decoupled design and cannot be called directly. For example, in a message-driven system, the message sender completes the local business and sends the message, and the multi-platform message consumer service needs to receive the pushed message and continue to process other business.
Looking at these two architecture diagrams, the first type of BC is directly dependent on A service, so if the interface in A is modified, BC must be modified, and the coupling degree is high. Second, MQ is used as the middleware for sending and receiving messages. The BC only relies on the received messages, not the specific interface, so that even if the A service modifies or adds other services, it simply subscribes to MQ.
Asynchronous processing of real-time services is not required
Take the user registration service process as an example.
- Users register for the library
- The user authenticates email sending
- The user authenticates sending SMS messages
In the original system design, such service process would be handled in sequence, namely 1-2-3; But here you can think about, if a single service in the case of a single machine, a large number of registered users, can the system withstand?
Suppose the time of each phase is 1 = 50ms, 2 = 50ms, 3 = 50ms, then a request comes down to all = 150ms; Let’s assume that the server CPU is equal to 1 and can only handle single threads. QPS = 1000/150 ≈ 7 Now I’m going to triple this QPS * 3 by introducing MQ services as middleware
As you can see in the figure, I returned directly after the user registration of service A. At this time MQ is used to send asynchronous processing messages, while service B and C process them separately. A does not have to wait for the result of B and C, so the user experience is only 50ms waiting time. However, in the stage of email and SMS, users can accept waiting for a certain period of time due to network delay.
Peak peel
General services, our requests to access the system are direct requests, such a mode in the case of small user visits, the problem is not very big. But if user requests hit a certain bottleneck or create problems, we need to consider optimizing our architectural design, and MQ middleware is one of the solutions.
The following seckill system as an example to analyze the problem of seckill system instant millions of concurrent, how to deal with? Generally, the second kill system will filter requests. Invalid and repeated requests will be filtered once, and the rest will enter the second kill service and order service. But even with this high concurrency, if the gateway were to forward all requests to the downstream order service, it would overwhelm the downstream system, causing service unavailability or even an avalanche.
The solution is to add the faster tasks that are processed upstream to the queue for processing, and then consume the queue one by one until all the queue consumption is completed. If the spsec service processes 1000 requests per second and the downstream order service processes 10 requests per second, the order service can calmly process ten requests per second, rather than simply blocking 1000 requests regardless of whether the downstream order service wants to process them or not.
Here, you can summarize the filtering mode of the second kill system:
- Page button click once set gray
- Limit the number of requests per second, such as 100/s, can be used with Nginx, Sentinel
- Filter repeated requests from the same user, through user unique identification, product information,
- The successful kill information is stored through the message queue and processed by the downstream order system
The log
All services send logs to MQ services for log storage. MQ is used as middleware to persist logs, forward big data services to read MQ, and analyze logs
MQ how to choose
Someone comes up with a performance comparison and says RabbitMQ is the best MQ in the world… You compare the selection of MQ to the selection of a wife, come up to the full set, beautiful white skin, front and back, sexy hot, hardworking… Is really lack of social education ah, brother can afford? Always a set of maintenance package, 1W/ month to keep it? Next door Lao Wang often come to your home for dinner, crazy imagination… Is the food ok? Jujube + medlar + kidney treasure tablet, I am afraid the heart has insufficient power
Anyway, actually I think this is a question, the first thing we should look at is what are the conditions?
- Purpose? Is it for logging, decoupling, or asynchronous processing
- What about the company? Whether the staff is sufficient, the existing staff technology stack situation, the strength of the staff technology stack
- Project status? Project period, staff, number of users, architectural design, whether old project
- Mainstream MQ status? Reliability, community activity, documentation comprehensiveness, cloud service support
In the example above, log messages are kafka. Why Kafka? Kafka is an open source distributed publish-subscribe messaging system for LinkedIn. Kafka is a top-level project of Apache with an active community. Kafka’s main feature is a Pull based approach to message consumption and high throughput. Kafka was originally intended for log collection and transmission. Later versions began to support replication, do not support transactions, there is no strict requirements for message duplication, loss, error, suitable for the data collection business of Internet services that generate a large amount of data. But Kafka is relatively heavy and relies on ZooKeeper, which can be used in large companies and requires maintenance.
RocketMQ is an open source reliable messaging system from Alibaba that has been donated to Apache as a top-level project. Initially positioned as non-logging reliable message transport, it actually performs well in log processing. Currently supported clients include Java, c++, GO, the community is more active, the document is still comprehensive. But it is difficult to modify the core, because Ali Cloud makes money by selling this service. So if the strength of the company is not confident or carefully choose it, really can not directly buy cloud services, save worry and effort, or that sentence, depending on the actual situation.
Features of mainstream MQ
Below is the picture of the source network, some of the description is out of date, but basically not bad, just for reference:
How do I ensure that messages are not re-consumed
Here is a brief description, and then a confession is written specifically for this issue. For some special reasons, such as network reasons, message consumption is not recorded due to service restart, resulting in the possibility of repeated consumption. The general approach is to ensure that the interface design is idempotent, and the existence of a theme is determined by a unique identifier.
- Redis cache is used, unique token is saved redis, token is deleted after each consumption
- The database determines whether the primary key record exists. If the primary key record exists, it will be updated. If the primary key record does not exist, it will be inserted
Like articles please follow me
Application field
Click follow + forward, send [interview] or [information] to get more resources
