Preface:

Recently, I read the content of thread pool, combined with the source code to learn its internal principles, wondering whether I have actually used thread pool in the project? On second thought, thread pools are used in many parts of the project; Here’s a quick look at the recent use of multithreading in logging:

  • Asynchronous thread pooling of service interface logs
  • Multithreading is used to clear logs in scheduled tasks

Main line of this article:

Interpreting the basic principle of thread pool;

② Practical application examples of thread pool

  • Thread pool application Demo project structure description

  • Asynchronous thread pooling of service interface logs

  • Multithreading is used to clear logs in scheduled tasks

Basic principles of thread pool

Don’t say anything, first paste a brain map, through the brain map of the thread pool quickly understand;

In addition to the picture, you can also read this article “AVA Thread Pool implementation principle and its Practice in Meituan Business” for a detailed understanding of the thread pool;

Examples of thread pools in action:

Let’s talk about the recent use of thread pools in project logging;

Thread pool application Demo description:

Demo address: github.com/leishen6/sp…

The two logging applications mentioned above have been Demo written as a SpringBoot project. The project structure is shown below:

Asynchronous thread pooling of service interface logs:

In the background service interface project, it is often necessary to store the logs of request messages and response messages of the interface.

Let’s talk about why thread-pool entries are more elegant by comparing them to normal entries.

Common warehousing operation:

Common warehousing is to directly complete the business logic processing and build a response at the same time log into the database, after the success of warehousing and then return the response;

The flow chart is as follows:

However, there is a big disadvantage of this is that there is an extra database operation (log entry), which may lead to slow response speed.

Let’s talk about optimizing log entry through thread pools to improve interface responsiveness.

Thread pool entry operation:

In the thread pool way, the log can be directly put into the queue, and then directly return the response. Finally, the thread in the thread pool can take out the log data in the queue to do the library operation asynchronously.

The flow chart is as follows:

In the thread pool mode, the main thread that processes the request can queue the log and return the response directly, and then use the thread in the thread pool to fetch the log data from the queue and asynchronously store it. Because of the reduction of one database operation, the interface response speed is greatly improved.

Let’s look at the code implementation:

1. LinkedBlockingDeque, the queue that stores the request and response logs

// Two-way blocking queues based on linked lists, where elements can be inserted and removed at both ends of the queue, are thread-safe and more efficient with multi-threaded concurrency
BlockingQueue<TestLogBean> queue = new LinkedBlockingDeque<TestLogBean>(MAX_QUEUE_SIZE);
Copy the code

In addition to the LinkedBlockingDeque blocking queue, there are several other blocking queues that are commonly used, as shown below:

2. Thread pool for logging operation in project: single-thread thread pool + thread pool with fixed number of threads

  • Single-threaded thread pool: used to loop and listen to the number of logs in the queue and decide when to remove the logs from the queue to a fixed number of threads in the pool.
  • A thread pool with a fixed number of threads is mainly used for log entry operations.

Part of the code is as follows:

/** * initializes */
public void init(a){
    // Two-way blocking queues based on linked lists, where elements can be inserted and removed at both ends of the queue, are thread-safe and more efficient with multi-threaded concurrency
    queue = new LinkedBlockingDeque<TestLogBean>(MAX_QUEUE_SIZE);
    lastExecuteTime = System.currentTimeMillis();

    logger.info("LogPoolManager init successfully......");

    logManagerThreadPool.execute(new Runnable() {
        @Override
        public void run(a) {
            while (run.get()){
                try {
                    // Thread sleep. The specific time is set according to the actual situation of the project
                    Thread.sleep(SLEEP_TIME);
                } catch (InterruptedException e) {
                    logger.error("log Manager Thread sleep fail ", e);
                }
                // Log insertion is performed when 10 logs are stored or the interval is greater than the maximum value
                if (logCount.get() >= BATCH_SIZE || (System.currentTimeMillis() - lastExecuteTime) > MAX_EXE_TiME) {
                    if (logCount.get() > 0) {
                        logger.info("begin drain log queue to database...");
                        List<TestLogBean> list = new ArrayList<TestLogBean>();
                        /** * drainTo (): Get all available data objects from BlockingQueue at once (you can also specify how many data objects to get). There is no need to batch lock or release lock multiple times. * Puts fetched data into the specified list collection */
                        queue.drainTo(list);
                        // Set the number of tasks in the task queue to 0
                        logCount.set(0);
                        // Thread is removed from the thread pool to perform log insertion
                        logWorkerThreadPool.execute(new InsertThread(testLogService, list));
                        logger.info("end drain log queue to database...");
                    }
                    // Get the current execution time
                    lastExecuteTime = System.currentTimeMillis();
                }
            }
            logger.info("LogPoolManager shutdown successfully"); }}); }Copy the code

In this project, the asynchronous thread pooling processing of service interface log is tested. After the project is started, enter the following URL in the browser and refresh the page:

http://127.0.0.1:8081/v1/api/log/test

Using multiple threads to clear logs in scheduled tasks:

If the amount of data in the log table occupies too much disk space, disk alarms are generated constantly. In this case, you need to thin the log table.

At this time, you can use multithreading to clear some data in the log table to release disk space.

What data in the log table needs to be cleaned up? The following scenario might appear in a requirement:

The leader said he needed to keep the latest year’s data in the log table, that is, 365 days forward from the current date; For example: today is 2020-12-30, the date pushed forward 365 days is 2019-12-30, so the logs generated before 2019-12-30 need to be cleared;

The following code, due to the need to ensure the flexibility of the program as far as possible, so it is necessary to delete the table name, according to the deleted fields for flexible configuration; The configuration parameters are shown as follows: application-cleanlog.properties configuration file

## Thread pool size
threads.pool.num=8
## Tables that need to be cleaned
log.clean.table=t_test_log
## The field by which the cleanup is performed
log.clean.filed=createts
The amount of data to be cleaned each time
log.clean.batchCount=1000
## The number of times of cyclic cleaning during each scheduled cleaning
log.clean.batchNum=6
## How many days of current data need to be retained before the rest of the data can be cleared
log.clean.dateNum=1
Copy the code

Note:

  • The size of the thread pool must be properly configured based on the server hardware configuration and actual service log volume. If the setting is too large, it may occupy too much memory and perform frequent context switching, which may lead to low efficiency.

  • In this project, data is cleaned according to the date field. If the table has been partitioned according to the date, it can be cleaned directly according to the partition. The partition cleaning speed is faster, but deletion by partition has not been implemented in this project.

  • Data volume for each delete: Indicates the amount of data to be deleted during a delete. Do not set the value too large. If the amount of data to be deleted is too large, the table may be locked, affecting normal query and new operations.

  • Number of delete operations performed during scheduled task execution: Indicates the number of delete operations performed during scheduled task execution. The amount of data to delete each time should not be too large. If the total amount of data to be deleted remains the same, the number of times to be deleted should be larger.

    Total amount of data to be cleared = Amount of data to be cleared each time x Number of data to be cleared

The flow chart of log clearing by multi-threading is as follows:

Let’s look at the code implementation:

Part of the code implementation is as follows:

/** * multithread cleanup log start */
public void cleanLogStart(a){

    // The number of times that the log is cleared
    int whileNum = props.getInt("log.clean.batchNum");
    LogCleanBean logClean = null;

    while (whileNum > 0) {// Query the time period that matches the amount of data to be deleted each time
        List<String> list = logCleanService.selectTime(logCleanBean);
        if(list ! =null && list.size() > 0){
            logClean = new LogCleanBean();
            logClean.setTableName(logCleanBean.getTableName());
            logClean.setFieldName(logCleanBean.getFieldName());
            // Gets the minimum generation time for logs that can be deleted
            logClean.setMinTime(list.get(list.size()-1));
            // Gets the maximum generation time for logs that can be deleted
            logClean.setMaxTime(list.get(0));
            logCleanBean.setMinTime(logClean.getMinTime());

            // This query does not meet the specified data volume, indicating that the data is cleared
            if (list.size() < logCleanBean.getBatchCleanCount()){
                whileNum = 0;
            }else {
                // The number of cleanup times is decremented--whileNum; }}else {
            break;
        }
        // multithreading
        cleanManagerThreadPool.execute(new CleanThread(this.logCleanService, logClean)); }}Copy the code

Extension:

In this project, delete is used and the data is deleted according to the time field. If the MySql database is used, the available disk space is not increased after deleting the data, and the available disk space may be reduced. Why?

  • In the InnoDB storage engine, delete does not actually delete data. Mysql actually marks the deleted data as deleted. Therefore, when DELETE deletes data from a table, the disk space of the table file will not be reduced and storage space will not be freed. Just make the deleted rows invisible. Disk space is not freed, but it can still be reused the next time data is inserted (reuse → overwrite).

  • If large fields such as Text and BLOB exist in the deleted data, the log file may become extra large and occupy part of the disk space. As a result, the free disk space is further reduced.

Solution:

  • If the deleted data does not contain large fields such as Text and BLOB, you can directly overwrite the deleted data and wait for the MySql automatic cleanup operation, but it takes a certain amount of time.

  • You can use the optimize Table table_name after the delete operation to immediately free disk space. However, because the optimize execution locks the table, don’t use it at peak times or often; Because it blocks normal queries, updates, etc.

You can also use TRUNCate and DROP to quickly release disk space. Specific selection based on the actual situation of the current project.

— END —

This article introduces the use of thread pools in logging scenarios, in addition to many other scenarios using thread pools, and in many frameworks also use thread pools, you can also read the framework source code to learn how to use thread pools.

Like + comment + forward yo

If this article is helpful to you, please wave your love to make a fortune of the little hand under the praise ah, your support is my continuous creation of power, thank you!

You can wechat search [Muzilei] public number, a large number of Java learning articles, you can have a look at yo!