I have a Linux server on a cloud with Centos7 installed. Play around with Docker and Docker-compose containers and container choreography on this server. Use docker – compose installed a Nginx and redis, mongo’s environment, ran a call interface of the internal development. It has been working normally. Since it was an internal test server, the installation and use of the system did not give much thought to whether it was reasonable. I didn’t think it was a big deal.

But one day, I went back to the container and tried to install a guns version of Git. During the installation process, I was suddenly prompted to run out of disk space. My mind is still on a Windows server, checking who’s using the space and seeing if I can delete anything. So I started looking for commands in Linux, df and du, with that in mind. There are many log files in /var/log/. On closer inspection, looks like it’s taking up a lot of space. I thought it wouldn’t matter if I deleted a few logs, so I chose the big ones and deleted them all at once. Then, I check the disk again, ha ha space again. So I’m going to go back to my data disk, where my data disk is mounted separately under a directory called /data.

However, then the problem began to appear: the directory on the display disk suddenly failed to access, I think there is something abnormal, restart the machine, Windows is not so why. If you’re familiar with Linux, you might chuckle. I wiped out all the important logs, the memory data on the swap disk will probably not write back to disk correctly, so the shutdown operation is suspended. How could you delete the log file? Unfortunately I have to tell you that I have been executing commands as root. Alas, this is ignorance.

Later, the problem became more serious. I could only click power off on the cloud host to shut it down due to the delay in normal shutdown. Although the power off operation on the cloud host is also virtualized operation mode. After the power failure, Linux finally started up again, but errors started to appear during the boot process, and when it finally started up, I found that the automatic mounting of the data disk failed. So, I tried to manually mount the data disk, but unexpectedly also mount not, I began to panic, the procedures and data inside but useful ah, there is no backup? The mind starts racing to assess what the worst outcome is.

I also started contacting customer service staff at cloud hosting companies to ask them what to do about it. Speaking of this, do you see what feelings?

Through a series of queries and questions, I also uploaded the error message:

By viewing the system log, you can see the red error log alert:

Finally, the cloud host staff advised me to unmount the data disk and then remount the disk partition. If not, use the xfs_repair command to repair the file system. Although I repeatedly asked if such a fix would result in data loss, the result was that there was a high probability of data loss. Helpless and unable, I followed such instructions to operate, the following is a screenshot:

Xfs_repair -l: xfs_repair -l: xfs_repair -l: xfs_repair -l: xfs_repair Still, some important configuration files and front-end code are missing. Fortunately, there are backups in the development environment, but the configuration has to be reconfigured and debugged again, which is very troublesome.

I’ve been reviewing the lessons of the accident. I think the root cause is why the system disk is full. Why deleting log files has such a big impact. With these questions in mind I searched the Internet for a while. I couldn’t find the answer to the question, so I rummaged through a copy of the book birdman’s Linux Home Dishes that I had bought. I decided to take a closer look at the Linux disk and log section, hoping to find the answer. As a result, after carefully reading the relevant chapters, I suddenly realized that I had made a lot of misunderstandings and pitfalls in the process of using Linux. Therefore, finishing up the second half of this article, in order to remind themselves and later people, must pay attention to the principles of Linux and the knowledge of the bottom, do not just want to find the command to solve the problem, so eager to get results and immediate benefits will give you a lesson.

= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =

The following is the summary of the accident after the replay:

(1) Do not always use the root command to do everything.

Since root is a god, you can delete almost anything you want, otherwise you can try rm -rf from the root directory. Try it, hehe. Because I have been using root to operate, deleting logs can only be deleted by root, but ordinary users cannot delete logs.

(2) Why is the system disk space used up?

My system disk has 20 GIGABytes, although it is not large, and it should not fill up quickly when used with a 40GB data disk. I concluded the following reasons:

A. My data disk does not have A partition to mount to several common Linux directories, such as /var/usr/home. /usr is a common directory for installing software, which is similar to the Program Files directory in Windows. /var is a variable data directory, which is also the main directory for data inflation

Ali Cloud ECS server mount disk to existing home directory blog.csdn.net/zhou_fan_xi…

B. Container is a large space, container and container data are also in /var/lib/ directory, container should also be placed on the data disk

7 changes in CentOS Docker default image and the position of the container www.linuxidc.com/Linux/2017-…

C. System logs are closely related to disks and virtual memory swap disks. Before the shutdown, sync will write the data on the ram disk back to the disk. If you want to keep the log files from crowding the system, you should also set a rotation deletion task for the log data, or save it to another disk.

(3) If a disk file system problem occurs, the repair process is as follows:

www.cnblogs.com/yuzhaoxin/p…

(4) Linux system monitoring, in fact, the automatic analysis of the log is very important, in bird brother’s book gave a monitoring program he wrote himself, download address is:

Linux.vbird.org/download/in…

In a word, this is a major lesson I have learned from Linux, but also urged me to have some in-depth understanding of Linux, I hope that the future use of Linux will be more and more smooth. And we share!

2019-08-17 At Hangzhou FeigoCopy the code

/dev/vdb5 10 gb, ext4 filesystem; /dev/vdb5 10 GB, ext4 filesystem; /dev/vdb5 10 GB, ext4 filesystem; /dev/vdb5 10 GB, ext4 filesystem; /dev/vdb6: XFS file system: /dev/vdb6: XFS file system: XFS /dev/vdb5 is mounted to the /home directory, and /dev/vdb6 is mounted to the /var directory. Then, the Docker and docker-compose software was quickly restored. When the container was created, the following capacity ratio was seen after executing df -h. It can be seen that the increased occupation of the container is in the /var directory, which is also placed on the data disk, does not affect the space of the system disk. In the future, if /usr becomes too large, you can mount it on an additional data disk and move things from /usr to the new data disk, so that the system disk remains unaffected by space changes.

At this point, my CENtos7 system is reinstalled and adjusted!