preface

Logstash is an open source data collector for unified log processing, which belongs to L in ELK and is widely used in the field of log collection.

Docker default log drive is json – file, every container locally generated a/var/lib/docker/containers/containerID/containerID – json. The log, and log drive is to support the extension, this chapter mainly explain is used Logstash collects docker logs.

Docker does not have a Logstash driver, but you can collect logs of the GELF driver through the Logstash Gelf Input plugin.

The premise

  1. docker

  2. Understand the Logstash configuration

  3. docker-compose

Preparing configuration Files

docker-compose.yml

Version: '3.7' X-logging: &default-logging driver: gelf options: Gelf-address: "udp://localhost:12201" mode: Non-blocking max-buffer-size: 4m tag: "kafeidou.{{.Name}}" # configure the container tag to kafeidou, as shown in figure 2. Docker-compose adds a copy suffix to the docker-compose container, such as logstash_1services: logstash: ports: -12201: 12201/udp image: Docker. Elastic. Co/logstash/logstash: 7.5.1 volumes: - ./logstash.yml:/usr/share/logstash/config/logstash.yml - /var/log/logstash:/var/log/logstash - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf logstash-worker: image: Docker. Elastic. Co/logstash/logstash: 7.5.1 depends_on: - logstash logging: driver: "gelf" options: gelf - address: "udp://localhost:12201"Copy the code

logstash.yml

HTTP. Host: "0.0.0.0"Copy the code

logstash.conf

input { gelf{  use_udp => true  port_tcp => 12202 }} output {   file {        path => "/var/log/logstash/%{+yyyy-MM-dd-HH}/%{container_name}.log"   } }Copy the code

Since logstash requires write permission in the configured directory, you need to prepare the directory for storing logs and grant the permission. Create a directory

mkdir /var/log/logstashCopy the code

Grant access, here for experimental demonstration, direct access to 777

chmod -R 777 /var/log/logstashCopy the code

The docker – compose. Yml, logstash. Conf and logstash yml file directory execute the command: docker – compose up – d

[root@master logstash]# docker-compose up -dWARNING: The Docker Engine you're using is running in swarm mode.Compose does not use swarm mode to deploy services to multiple nodes in a swarm. All containers will be scheduled on the current node.To deploy your application across the swarm, use `docker stack deploy`.Starting logstash_logstash_1 ... doneStarting logstash_logstash-worker_1 ... doneCopy the code

The logstash startup is slow. The result of my experiment is about 90 seconds, so fluentd is more recommended to collect logs

Check the log directory and you should have the corresponding container log file:

[root@master logstash]# ls /var/log/logstash/2020-02-16[root@master logstash]# ls /var/log/logstash/2020-02-16/logstash_logstash-worker_1.logCopy the code

You can also download my files directly:

  1. docker-compose.yml

  2. logstash.conf

  3. logstash.yml

conclusion

Fluentd is more recommended for technical selection, why?

Fluentd is lighter and more flexible, and currently owned by CNCF, with a higher level of activity and reliability.

Why introduce LogStash to collect Docker logs?

If a company or business is already using the ELK stack, it may not be necessary to introduce another Fluentd and continue to use LogStash to break through the Docker log. Here is mainly to do a share, so that students who meet this situation can have a choice.

Recommended reading:

Use Fluentd as docker log driver to collect logs