“This article has participated in the good article call order activity, click to see: back end, big front end double track submission, 20,000 yuan prize pool for you to challenge!”
What is disk IO high BUSY
High DISK I/O busy Occurs on some application servers with extremely frequent write/write operations. For example, a server with a database installed, such as the mysql or Oracle database, is prone to high I/O busy in a high concurrency environment. The disk I/O high busy is usually caused by a large number of write operations. Oracle, for example, has redo and undo operations, the transaction log, for rollback and commit, in addition to normal write and write operations. Both of these can easily lead to high BUSY IO during a large number of UPDATE or INSERT operations. In general, we think that IO busy above 80% should be a warning. This problem is common at high concurrency. We can easily simulate it with our stress testing tools.
As shown in the figure above, the red box has reached 85% of the disk IO (nMON tool).
Why is the disk IO high busy
Disk I/O is busy due to high write/write frequency, scattered storage, and disk pointer hopping back and forth. A large number of small files will increase the disk head search time. “How to efficiently process IO” is actually a relatively large topic, excellent data processing framework, for this piece have done a lot of optimization processing.
Lying flat processing Disk IO high BUSY
Disk I/O high BUSY Usually occurs on the database server and file processing server. The disk I/O of the database server is high busy.
1. Find a large number of writer-thread commands using Linux commands
There are several tools for seeing disk IO load in Linux. Iostat Displays the I/O load from the system dimension. Iostat and iotop are not common tools provided with Linux distributions. They are included in the sysstat package. You need to install them using yum or apt-get before using them. Command reference: Iostat -x 1 12 Viewing disk I/O information iotop -op Viewing process operations IO pidstat -d 1 Collects I/O statistics. Update iotop -p $PID -d 1 update iotop -p $PID -d 1 Update iotop -p $PID -d 1
2, check out what the thread is doing
The jstack command allows you to view the stack information, that is, to see what the thread is doing. In fact, in the first step, most of the time you’ll see what’s causing high BUSY IO.
3. Find out why this operation is done
Once you’ve figured out what’s causing the high disk busy, you can actually consider optimizing.
4. Optimize the results of 3
There are many optimization schemes, if it is a disk problem, you can consider changing the disk. If it’s because of code or business logic, it’s time to adjust.
5. Consider optimization
Code optimization
Code optimizations are simple optimizations, mostly due to some bad code, which usually gets sifted out during testing.
Optimization of technical scheme
Technical optimization needs to consider using other technologies instead. For example, spring Session has multiple solutions, redis, JDBC, HashMap… If you are using JDBC, you will be writing to the database for every back-end operation (including queries). If you are using Oracle, you will be generating a large amount of undo and redo information. So this also shows how important the optimization step is after the program is developed, and it also shows the importance of different technology types and different scenarios for performance.
Business direction optimization
In terms of business optimization, it’s more complicated, and you need to be good at this area.
A chestnut
The disk I/O on the Oracle server is high busy under high concurrency.
Environment: Oracle19 WebLogic12c…… Once upon a time, an IO alert went off in production, and a review of the log revealed that there were only a large number of access requests that were causing the database disk I/O usage to exceed 80%. What the hell is going on here? I exported the AWR report of Oracle database for this time period, and found that there were a lot of update operations in it. What the hell is this? After checking SQL, it was found that it was caused by JDBC configuration of Spring Session, and then optimized, and used Redis to do session storage operation.
This little chestnut shows high disk busy are easy to troubleshoot (is not biased, the author also deals with many problems of high IO busy, or performance test, or production, in general, actually in handling this kind of problem is relatively easy), determine the good mentality, guaranteed plan, step by step operation, screening out soon. Consider not only the database server, but also the overall Web service and code implementation.