Problems arise

Recently, the machine that uses Docker container in the production environment obviously has too fast disk growth, and the monitoring is red. At first, I didn’t care because these machines were old machines and the initial disk space itself was not large. Normally, it would need to be cleaned up at intervals.

But later, I took the trouble to clean it up once a week. After all, it was not professional operation and maintenance, so I took the time to look at it carefully.

To solve the process

Run commands to check disk usage

du -h --max-depth=1 /
Copy the code

Found after screening the directory/var/lib/docker/containers and/var/lib/docker overlay2 takes up a lot of space, also on the network search, most is to make the docker clear command, do not work.

  • /var/lib/docker/containers

    Pick a larger directory to continue

    ls -lht
Copy the code

    Finding that the space hog is actually a log file, I searched for a solution:

    cat /dev/null > *-json.log
Copy the code

    Conditional restart, do not restart OK.

  • /var/lib/docker/overlay2

    The rocketMQ client logs in the directory that occupies large space are also found

    Du - h - Max - the depth = 1 / var/lib/docker overlay2 / c9934ca80d32e8c8a6108bbe914258ed41f17b87f751d3ed5811a38bf8535a8f 1.1 G /var/lib/docker/overlay2/c9934ca80d32e8c8a6108bbe914258ed41f17b87f751d3ed5811a38bf8535a8f/diff 0 / var/lib/docker/overlay2 / c9934ca80d32e8c8a6108bbe914258ed41f17b87f751d3ed5811a38bf8535a8f / 1.7 G work The/var/lib/docker overlay2 / c9934ca80d32e8c8a6108bbe914258ed41f17b87f751d3ed5811a38bf8535a8f/merged 2.8 G /var/lib/docker/overlay2/c9934ca80d32e8c8a6108bbe914258ed41f17b87f751d3ed5811a38bf8535a8fCopy the code

    One copy exists in both the diff and merged directories. Continue to explore:

    The overlay2 directory is the docker’s hierarchical storage directory, storing the actual hierarchical content. Created after the completion of the container in image storage directory/var/lib/Docker overlay2 / containers will be generated initial layer and reading and writing, and both use the same logo, much more behind the initial layer – init. The initialization layer mainly stores container-related environment information, such as container host name, host host information and domain name service files, when the container environment is initialized. The read-write layer is used to read and write containers. Processes in Docker containers only have write permission on the read-write layer, and only have read permission on the file contents of other layers.

    With RocketMQ, a large number of MQ trace logs are generated in the {user.home}\logs\ Rocketmqlogs directory by default

    Then I select a RocketMQ client application and enter the container:

    docker exec -it xx bash
Copy the code

    I looked in the root directory and I didn’t find the log directory

    find -name rocketmq_client.log

./root/logs/rocketmqlogs/rocketmq_client.log
Copy the code

    Finally, logs directory found under root… The user is root.

    When Docker starts, I only do volume mapping for /logs and ignore the rest of the logs, thus causing this problem.

    Write it down and give some inspiration to other students who have similar problems.