There are several ways to output logs inside a container: In the process of solving nginx-Ingress-Controller log persistence, we try to solve various problems non-invassively with k8S ‘ability of arranging and organizing, including directory permissions and mount problems, NGINx configuration problems, log rotation problems. Hope these solutions can provide ideas for readers to container application of log falling disk.
Start with requirements
The Canoe micro service platform relies heavily on Nginx-Ingress-Controller for traffic management and load balancing, so log maintenance for this component is very important.
The K8S team designed a relatively non-invasive solution, through which the log output of nginx-Ingress-Controller, whether the log output of accesslog or klog library, can be redirected to drop disk, rotation, and cleaning. We found that this log persistence solution basically meets the log persistence requirements of most container applications.
Nginx log – ingress – controller
The nginx-Ingress-controller log consists of three parts:
-
Controller logs: Output to stdout. The output to a file can be configured through -log_DIR in the startup parameter. After being redirected to a file, the file is automatically rotated but not automatically cleared
-
Nginx-ingress-controller configmap: nginx-configuration configmap: nginx-configuration configmap: nginx-configuration configmap: nginx-configuration Output to a file does not automatically rotate or clean up
-
Errorlog: Output to stderr in a similar way to accesslog.
Unload controller logs
- /data/log/nginx_ingress_controller/ map to /var/log/nginx_ingress_controller/,
- Configure log-dir and logtostderr for nginx-ingress-controller to redirect logs to /var/log/nginx_ingress_controller/.
The controller logs need to be cleared periodically. Because the controller’s logs are output through klog(k8s.io/klog), which will carry out log scrolling, we can periodically clear the log files of a certain time ago through the script.
Drop disks for nginx logs
-
Example Change configMap to nginx-configuration. Configure the output paths for accessLog and errorlog, replacing the default stdout and stderr. The output path can be the same as the controller for easy search.
-
Both accessLog and ErrorLog have only one log file, and we can use Logrotate for log rotation to rotate and clean up the logs output to the host. Configuration such as:
$ cat /etc/logrotate.d/nginx.log /data/log/nginx_ingress_controller/access.log { su root list rotate 7 daily maxsize 50M copytruncate missingok create 0644 www-data root } 12345678910
-
By default, nginx-ingress-controller uses user 33 to log in to the startup container in the official template provided by nginx-Ingress-controller. Therefore, there is permission problem when you mount the hostPath. We need to manually execute chown -r 33:33 /data/log/nginx_ingress_controller on the machine.
Automation ops
Nginx logs fall into disk, the 2nd and 3rd points need to be operated by people, is there any solution?
initContainer
The question is: is there a way to add a hook to the host directory before the nginx-Ingress-controller starts?
You can use initContainer. Initcontainer must run and exit successfully before containers in containers can run. Use this k8s characteristics, we develop a docker image:hub.c.163.com/combk8s/adddirperm:1.0.0, inside only execute the following script:
#! /bin/bash logdir=$LOG_DIR userID=$USER_ID echo "try to set dir: $logdir 's group as $userID" chown -R $userID:$userID $logdir 12345Copy the codeThe script reads some environment variables to determine which directory to change and what user group to change to.
Package the script as a Dockerimage and place it in deploy YAMl of nginx-Ingress-Controller as initcontainers. Note That environment variables and volumeMount are configured for initContainer.
sidecar
Second, we notice that the base image of nginx-Ingress-Controller does not include Logrotate, so our initial idea is to run and configure Logrotate on the host. However, installing and running Logrotate on machines has scripts that require bulk operation, and some environments may not allow us to install these things on machines. So think of a containerized solution.
So we designed a sidecars, use our own build a good image: hub.c.163.com/combk8s/logrotate:v1.1. Logrotate is executed every 6 hours in the container where the image is started. The scheduled time can be injected using the environment variable CRON_EXPR. For example, CRON_EXPR= */3 * * * * indicates that the logrotate is executed every 3 minutes.
We save the rotation policy for nginx accessLog and ErrorLog logs to configMap and mount the volume to the Sidecar container. The sidecar container should also mount the nginx-ingress-controller log directory.
inductive
Let’s summarize the solutions to these problems:
- Directory mounting problems caused by mirroring user permission: Use initContainer to chown the directory user group before starting the service container
- For stdout and stderr logs, you need to directly adjust service configurations to implement log landing. Otherwise, such logs can only be viewed in the container output file saved by the Docker, which is not readable
- For logs that need to be rotated, a sidecar container can be used for periodic rotation.
Thinking about other schemes
The mirror cover
Some people suggest removing initContainer and adding a layer based on the original Nginx-Ingress-Controller image, where the script to configure path permissions is executed. Personally, I think this method is neither beautiful nor convenient. The only benefit is that deploy YAMl is still simple (without configurations like volumeMount).
Still see personal use feeling however ~
Author: Huang Yang