Blog: bugstack.cn

Precipitation, share, grow, let yourself and others can gain something! 😄

One, foreword

Don't roll it, just use it!

Ha, ha, ha, ha, ha, ha, ha, ha, ha, ha, ha. However, every time I received a new demand, I was itching to update iteratively on this demand based on the previous architectural design and landing experience, or find a better solution to completely overturn the previous one. It’s always refreshing to roll out the code

In fact, most of the pure code farmers like to write code, are relatively volume, for example, a requirement in the implementation is about P5, if this function is not only used but also very good is P6, in addition to good but also condensed common requirements developed into a common component service is P7. Every growing up code farmers, are in the road to build the wheel again and again to verify their ideas and practice, absolutely not a piece of eight-part essay can be tired out of a senior technical cattle.

Second, delayed task scenario

What is a deferred task?

In our actual business demand scenarios, some state changes before the start of activities, T+1 reconciliation after order settlement, and the generation of interest charges on loan bills need to be reached by using delayed tasks. The actual operation usually includes Quartz and Schedule to scan and process the data in your library table periodically. When the conditions are met, change the data state or generate new data to insert into the table.

Such a simple requirement is the initial requirement of delayed task. If there are few contents and few users in the early stage of the requirement, it may be completed by a single machine directly facing the table for a round training in actual development. However, with the development of business requirements and the increasing complexity of functions, it is often not so simple to feedback to r&d design and implementation, such as: You need to ensure as low delay as possible to complete the large-scale data scanning processing, otherwise, just like the generation of loan interest fees, the next day users have not seen their interest information or the re-reconciliation after repayment, may be this time to generate customer complaints.

So how do you design a scenario like this?

3. Delayed task design

The usual processing flow of the task center is mainly that scheduled tasks scan the task library table, scan the task information that is about to reach the timeout time to the processing queue (memory /MQ message), and then the service system processes the task. After the processing is complete, the task status in the library table is updated.

Question:

  1. Massive data Large-scale task list data needs to be quickly scanned in the sub-database sub-table.
  2. Task scanning service and business logic processing are coupled together, so it has no generality and reusability.
  3. Some subdivision task systems require low latency processing and cannot wait too long.

1. Task list mode

In addition to some of the smaller state change scenario, in their respective business library table, for example, it contains a status field, the field on the one hand has the program logic to handle change state, has arrived at the designated maturity by task service is changed automatically processing operation, general this kind of function, design directly to your repository in the table.

There are also large and frequently used scenarios that would be redundant and difficult to maintain if such fields were added to each of the N tables required by each system. Therefore, it is very suitable to make a general task delay system for such scenarios. Each business system submits the actions that need to be delayed to the delay system, and then the delay system calls back at a specified time. The callback action can be interface or MQ message for contact. For example, you can design a task scheduling table like this:

  1. The task scheduling table extracted is mainly about what tasks to get and when to initiate actions. The specific action processing is still handed over to business engineering.
  2. For centralized processing of tasks of a large number of respective businesses, a sub-database and sub-table should be designed to meet the growth of subsequent business volume.
  3. House number design, for the scanning of a table, if the amount of data is large, and do not want to scan a table for only one task, you can scan a table for multiple tasks, add to the volume of scanning. In this case, a door number is needed to isolate the scanning scope of different tasks and avoid the scanning of duplicate task data.

2. Low latency mode

The low delay processing scheme is based on the task list mode, and the new time control processing is added. It can put the tasks that are about to expire in the previous period of time into the Redis cluster team, and pop them out of the queue at the time of consumption. In this way, the processing time of tasks can be closer to faster, and the task execution can be avoided due to the large sweep interval.

  • When receiving the delayed tasks submitted by the business system, they can be placed into the task library or synchronized to the Redis cluster according to the length of execution time. Some tasks with late execution time can be put into the task library first, and then added to the timeout task execution queue by scanning.
  • So the core of this design lies in the use of Redis queue, and in order to ensure the reliability of consumption, it is necessary to introduce two-stage consumption and register ZK registry to guarantee at least one consumption.This paper mainly focuses on the design of Redis queue, and other more logical processing can be expanded and improved according to business requirements

Redis consumption queue

  • Calculate the slot where the data resides based on the message bodyindex = CRC32 & 7
  • StoreQueue uses Slot according to SlotKey =#{topic}_#{index} And Sorted by task score to store task execution information.Timed messages treat the timestamp as a score, and each time a message with a score less than the current timestamp is popped up for consumption
  • To ensure that each message can be consumed at least once, rather than pop the elements in the ordered collection directly, the consumer moves the elements from StoreQueue to PrepareQueue and returns the message to the consumer. If the consumption succeeds, it is deleted from PrepareQueue, and if the consumption fails, it is moved from PreapreQueue to StoreQueue again, which is processed in a two-phase consumption manner.
  • Reference document: 2021 Ali Technical Staff hundred Treasures black book PDF,Low-latency time-out center implementation

A simple case

@Test
public void test_delay_queue(a) throws InterruptedException {
    RBlockingQueue<Object> blockingQueue = redissonClient.getBlockingQueue("TASK");
    RDelayedQueue<Object> delayedQueue = redissonClient.getDelayedQueue(blockingQueue);
    new Thread(() -> {
        try {
            while (true){
                Object take = blockingQueue.take();
                System.out.println(take);
                Thread.sleep(10); }}catch (InterruptedException e) {
            e.printStackTrace();
        }
    }).start();
    int i = 0;
    while (true){
        delayedQueue.offerAsync("Test" + ++i, 100L, TimeUnit.MILLISECONDS);
        Thread.sleep(1000L); }}Copy the code

The test data

2022-02-13  WARN 204760 --- [      Finalizer] i.l.c.resource.DefaultClientResources    : io.lettuce.core.resource.DefaultClientResources was not shut down properly, shutdown() was not called before itGarbage-collected. Call shutdown() or shutdown(long,long,TimeUnit) Test 1 Test 2 Test 3 Test 4 Test 5 Process finished with exit code -1Copy the code
  • Source: github.com/fuzhengwei/…
  • Description: Use DelayedQueue in Redisson as message queue, wait for consumption time after writing to POP consumption.

Four,

  • Scheduling tasks are frequently used in actual scenarios. For example, we often use XXL-Job, and some distributed task scheduling components developed by large companies. These functions may originally be very small and simple, but after abstraction, integration and refinement, they become core universal middleware services.
  • When we consider the use of task scheduling, no matter which way of design and implementation, we need to consider the use of this feature for iteration and maintenance, if only a very small scenario, and few people use, then on their own machine can be used.Excessive design and use can sometimes throw r&d resources into the mud
  • In fact, all technical knowledge points are like tools, knives, guns, cudgels, tomahawk hooks. How to combine their own characteristics and use these weapons is the process of a programmer’s continuous growth. If you want to learn more about this kind of in-depth technical content, you can join the Lottery distributed second kill system to learn more valuable and more effective combat methods.

Five, series recommendation

  • Before the interview, make yourself into a roll king!
  • Scheme design: Based on IDEA plug-in development and bytecode staking technology, realize automatic analysis of r&d delivery quality
  • Have you been working for two or three years and have no idea what the architecture diagram is?
  • Work two years resume to write like this, who want you!
  • BATJTMD, large factory recruitment, are recruiting what kind of Java programmers?