Background of the problem

From a business system of the server monitoring system warning notice, disk space utilization rate has reached 90%, and then login server search under larger log file, delete all the buried here (pit), released some disk space, was negligence, not to determine whether the detect and remove the file size of space has been released in full. Did not lead a few days, the server was warned again, compare wonder, how the log growth of so fast, after the investigation found that the original is the last operation after deleting files, there is a larger file space did not release caused by.

Problem reduction and solution

Find data files that take up a lot of space

#View the disk space usage
$ df -h
#Query files that occupy large space in the/TMP directory
$ du -sh /tmp/*|sort -nr|head -3
#Example Query files that occupy large space in the /home directory
$ du -sh /home/*|sort -nr|head -3
#After the file is found, delete it. Then run the df -h command to check whether the file is released
Copy the code

/ TMP /* / TMP /* / TMP /*

Delete policy in Linux: Linux does not provide the recycle bin function. Therefore, files to be deleted are moved to/TMP, and data in/TMP is deleted periodically.

Many servers do not partition/TMP during system installation. Therefore, data in/TMP occupies a large space. You can delete files in/TMP to release space.

A dubbo service log file in the /home directory was deleted.

Deleting a file does not free space. Procedure

In most cases, space is not freed after a file is deleted. However, it can occur when the file is locked by a process or a process writes data to the file. Understand the file storage mechanism and storage structure of Linux to understand this problem.

File storage in Linux system is divided into two parts: pointer part and data part.

  • Pointer part: it exists in the meta-data of the file system. After we run the rm command to delete the data, the pointer is cleared from the meta-data.

  • Data: The data is stored directly on disk. When the pointer is removed from meta-data, the space occupied by the data can be overwritten and new content can be written.

The dubbo log file still occupies space because the dubbo process is still writing data to the file. The pointer is not deleted from the meta-data when the file is deleted, so the log file still occupies space.

How do I find such files

You can use the lsof command to get a list of files that have been deleted but are still being used by the program:

lsof | grep delete
Copy the code

How to free up such space

To solve this problem, you can release space by restarting occupied processes, restarting the operating system, and running commands. Non-production environments are most convenient to use the first two methods, but for production environments, or try to use the command mode, in fact, the command is very simple:

echo " " >/home/dubbo/log/xxx.log
Copy the code

In this way, the occupied disk space is released and the process is not affected.

If you have any questions, please send me a private message. Let’s discuss and learn together