Welcome to Tencent cloud technology community, get more Tencent mass technology practice dry goods oh ~

Author: Tencent Cloud Container Service team

Kubernetes does not provide a native solution for logging container instances. The kubectl logs command is provided to view the running logs of the container instance. The kubectl logs command is implemented as follows: the container runs the standard output and standard error log output to disk by default. Save to host directory: / var/lib/docker/containers/container_id/directory. When the user calls kubectl logs, kubelet reads the data in the corresponding log file and sends the data back to the master, which then returns it to the user. In this way, users can view logs.

Tencent cloud container service uses kubectl logs command to view the logs of corresponding container instances on the console and provide the function of viewing the logs of a specific period, which greatly facilitates users to locate and track the programs in container instances. However, because the container instance logs are stored locally, it is easy to consume a lot of disk space on the host when a large number of logs are printed in the program. After the log service has been online for a period of time, users need to manually clear logs. Let’s consider whether there is an easier way to periodically clean up and roll back logs on cluster nodes.

Log cleaning and rewinding are implemented by logrotate service

Logrotate is a very useful tool that automatically truncates (or rounds) logs, compresses, and deletes old log files. For example, you can set logrotate to rotate the /var/log/foo log file every 30 days and delete logs that are older than 6 months. Once configured, Logrotate is fully automated without any further human intervention.

However, if you use the previous deployment mode, manually install and configure the Logrotate tool on each node. If through Kubernetes container service choreography ability, logrotate through Kubernetes service way deployed to each node, so that can achieve only one deployment, deployment to all nodes. The consistency of logrotate configuration is guaranteed by means of container. The specific implementation plan is shown in the figure below:

The following is an example of creating DaemonSet:

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: logrotate
spec:
  template:
    metadata:
      labels:
        app: logging
        id: logrotate
      name: logrotate
    spec:
      containers:
      - name: logrotate-es
        image: blacklabelops/logrotate
        securityContext:
          privileged: true
        volumeMounts:
         - name: containers
           mountPath: /var/lib/docker/containers
         - name: varlog
           mountPath: /var/log/docker
         - name: logs
           mountPath: /logs
        env:
        - name: LOGS_DIRECTORIES
          value: "/var/lib/docker/containers /var/log/docker"
        - name: LOGROTATE_INTERVAL
          value: "hourly"
        - name: LOGROTATE_OLDDIR
          value: "/logs"
      volumes:
         - hostPath:
             path: /var/lib/docker/containers
           name: containers
         - hostPath:
             path: /var/log/docker
           name: varlog
         - hostPath:
             path: /var/log/containers/
           name: logs
Copy the code

Using this YAML file, you can deploy directly in Kubernetes.

# kubectl create -f logrotate_ds.yaml
daemonset "logrotate" created
Copy the code

In the example YAML file, the Logrotate service rolls back logs at regular intervals (one hour) and cleans up logs after more than five copies are rolled back. You can modify parameters and set different rollback rules and clearing rules as required. For details, see github.com/blacklabelo…

Rewind and clean up by modifying dockerd parameters

Because of Kubernetes’ log collection, the bottom layer is achieved through Docker. However, Docker provides some log rollback and clearing functions. You can add log-opts() to the startup parameter of dockerd to rollback and clear logs. Max-size sets the maximum number of copies of a log, and max-file sets the maximum number of copies of a log. If the number of copies exceeds this, the log will be deleted.

Create /etc/dockerd/daemon.json

{
  "log-driver":"json-file"."log-opts": {"max-size" :"10m"."max-file":"3"}}Copy the code

Parameter Description: If the log volume of a container exceeds 10 MB, the log volume will be rolled back. If the log volume number exceeds 3, the log volume will be cleared.

2. Modify the Dockerd service configuration file

In the/etc/systemd/system/multi – user. Target. Wants/dockerd serviced adding dockerd boot parameters – the config file – the file = / etc/docker/daemon. Json

3. Restart the Dockerd service

systemctl daemon-reload
service dockerd restart
Copy the code

reading

The latest kubernetes-based application orcheography practice 5 Docker log best practices to help you quickly solve the distributed transaction XA consistency problem —— https://cloud.tencent.com/community/article/579587