Recently, and re-learn the next Redis, deeply Redis charm, I just know Redis can not only fast but also slow (I think so good O (╥﹏╥) O), it is a sharp weapon.
Hey hey hey, don’t get me wrong, this article is serious!
Well, back to our topic, we all know that Redis is a memory-based single-process, single-thread database (multi-threading has been supported since Redis6.0 started!). , processing speed is very fast. So why is Redis slow? By slow, we mean that Redis can set some parameters to achieve the result of slow processing. (That’s why Redis can be both fast and slow!)
Let’s talk about how our model Redis implements delay in queues:
In our daily life, we can find that,
- Orders placed on taobao, JD.com and other shopping platforms will be automatically cancelled if payment is not made after a certain period of time.
- When you take a taxi, the platform will cancel your order and remind you that there is no owner to take the order.
- When ordering food, if the business does not receive the order within 10 minutes, it will automatically cancel the order.
- When we receive a package, if we don’t click to confirm receipt, the program will automatically complete the order after a certain period of time.
- After the platform completes the order, if we do not review the product within the specified time, the buyer will automatically default not to review the product.
- . There are many more scenarios like this.
At this point, we can think about why are we doing this?
Because it ensures that the inventory of goods can be released for others to buy, you don’t have to wait for a taxi with no reply, you can move to another place in time to order takeout.
So how does all of this work?
At this point, we can look at this diagram to see how message latency is handled:
When a user sends a message request to the server when the background, the server will detect whether you need the message delay processing, if you need to add to the delay in the queue, the detector for detecting and processing delay task, for don’t need to delay processing tasks, the server will immediately to deal with the message, and the processed results corresponding to return to the user.
For within the delay task detector, do you have a query latency and task execution time delay task two functions, tasks detector will go first to read information delay task queue in the queue, judge what tasks have time expired in the current queue and will be expired output task execution (with real time, there are certain time error, because this is timing task).
At this point, we can think about the Redis data structure in the time setting flag command.
Zset is not thought of this command, with reorder (fractional sorting) function. Yes, you’re right!
We can use zset (sortedset) to sort with the set timestamp as score, using zadd score1 value1…. Command to keep producing messages into memory. Then zrangeBySocre is used to query all the tasks that meet the conditions to be processed, and the queue tasks can be executed through a loop. You can also query the earliest task through zrangebyscore key min Max withscores limit 0 1 to conduct consumption.
In general, you can do this in two ways ((^▽^) if you think of other ways, let me know ~) :
(1) Use ZrangeByScore to query all tasks in the current delay queue, find out all the delayed tasks that need to be processed, and perform operations in sequence.
(2) find the earliest one task, judging by the score value of task execution time is greater than the current system, such as: the earliest task execution time at 3 PM, 58 points system time at 2 o ‘clock), said this should need to be performed immediately, the time is coming soon (one unit, when he comes he comes, he came with the pace of death).
What are the advantages of Redis to implement delayed queues?
In fact, Redis is used to implement the delay queue with these advantages:
(1) Redis ZSet supports high performance score sorting.
(2) Redis operates in memory, and the speed is very fast.
(3) Redis can build clusters. When there are many messages, we can use clusters to improve the speed of message processing and availability.
(4) Redis has a persistence mechanism. When failure occurs, data can be recovered by AOF and RDB to ensure data reliability
At this time, there will be a small partner asked there is no other way to realize the delay queue! emmm…. Of course there is, only the unexpected can not do (O(∩_∩) ha ha ~ kidding)
One, using message middleware to realize delay queue
(1) Use RabbitMQ to implement delay queuing, that is, use message middleware to implement delay queuing
Method 1: In MQ we can set x-expires to Queue or x-message-TTL to Message.
(Note here: messages with the same delay are thrown into the same queue, and a queue is created for each delay – this is because MQ’s expiration detection is lazy.)
Method 2: We can use RabbitMQ plugin rabbitmq-delayed-message-exchange to implement delayed queuing. When the deliverable time is reached and it is delivered to the destination queue by the switch type marked with type X-delayed -type.
(2) RocketMQ implements delay queue
Rocketmq when delay messages sent, is the first message according to the delay time sent to the specified queue (put the same message delay time in the same queue, guarantee the order of message processing, can make the same queue the message delay time is the same, the whole rocketmq delay news sorted in increasing order, Ensure that information is processed sequentially. . After that, a timer is used to poll and process the information in the queue to determine whether it expires. Expired messages are sent to the corresponding processing queue for processing.
Lol RocketMQ currently only supports specific delay periods, 1s,5s,10s… 2H: the delay cannot be set for any time period. Interested partners can go to understand it is relevant knowledge ah ~
Kafka implements delay queue
Kafka uses a customized SystemTimer to implement the delay function. The TimingWheel in Kafka is a circular queue that stores scheduled tasks. You can set the delay queue.
Netty realizes delay queue
Netty also implements delay queuing based on the time wheel algorithm. Netty mainly uses HashedWheelTimer to construct the delay queue. The underlying data structure of HashedWheelTimer uses DelayedQueue and adopts the algorithm of time wheel to realize it.
DelayQueue to implement DelayQueue
Java has its own DelayQueue data type, which we can use to implement delayed queues. DelayQueue encapsulates a PriorityQueue. When adding an element to the DelayQueue, a Delay is given to the element as the sorting condition. The smallest element in the queue is placed first. Elements in the queue can only be removed from the queue after Delay. In this way, the data is stored in memory and may be lost, and it cannot support distributed systems.
