preface

ELK is gradually being replaced by EFK due to the large memory footprint of Logstash and relatively poor flexibility. The EFK mentioned in this article is Elasticsearch+Fluentd+Kfka. In fact, K should be Kibana’s log display. This section is not for demonstration, this article only describes the data collection process.

The premise

  1. docker

  2. docker-compose

  3. Apache kafka service

architecture

Data Collection process

Data generation uses CAdvisor to collect monitoring data from the container and transfer the data to Kafka.

Cadvospr->Kafka->Fluentd-> ElasticSearch

Each service can be scaled horizontally to add services to the logging system.

The configuration file

docker-compose.yml

Version: "3.7" Services: ElasticSearch: image: elasticSearch :7.5.1 environment: Cadvisor: image: Google/cAdvisor command: -discovery. type=single-node # -storage_driver=kafka -storage_driver_kafka_broker_list=192.168.1.60:9092(Kafka service IP:PORT) -storage_driver_kafka_topic=kafeidou depends_on: - elasticsearch fluentd: image: Lypgcs/fluentd - es - kafka: v1.3.2 volumes: -) /, / etc/fluent - / var/log/fluentd: / var/log/fluentdCopy the code

Among them:

  1. Data generated by cAdvisor is transmitted to the kafka service on 192.168.1.60,topic kafeidou

  2. Elasticsearch is specified to start in single-machine mode (discovery.type=single-node environment variable) to test the overall effect

fluent.conf

#<source>#  type http#  port 8888#</source><source>  @type kafka  brokers 192.168.1.60:9092  format json  <topic>    topic     kafeidou  </topic></source><match **>  @type copy#  <store>#   @type stdout#  </store>  <store>  @type elasticsearch  host 192.168.1.60  port 9200  logstash_format true  #target_index_key machine_name  logstash_prefix kafeidou  logstash_dateformat %Y.%m.%d  flush_interval 10s  </store></match>Copy the code

Among them:

  1. The type copy plug-in is used to copy the data received by FluentD. It is used to print the data in the console or store it in a file for debugging purposes. This configuration file is closed by default and only the necessary ES output plug-in is provided. If necessary, you can turn on the @type stdout section to debug whether data is received.

  2. The input source is also configured with an HTTP input configuration, which is off by default and is also used for debugging, putting data into Fluentd. The following command can be executed on Linux:

    curl -i -X POST -d 'json={"action":"write","user":"kafeidou"}' http://localhost:8888/mytagCopy the code

  3. The target_index_key parameter indexes es with the value of a field in the data. For example, the configuration file uses the value of machine_name as the index of es.

Start the deployment

Docker-compose up -d in the directory that contains the docker-compose. Yml file and the fluent.conf file

Select * from es; select * from es; select * from es; select * from es;

Bash: - : command was not found [root @ master kafka] # curl http://192.168.1.60:9200/_cat/indices? vhealth status index uuid pri rep docs.count docs.deleted store.size pri.store.sizeyellow open 55a4a25feff6 Fz_5v3suRSasX_Olsp-4tA 1 1 1 0 4kb 4kbCopy the code

May also directly in the browser enter http://192.168.1.60:9200/_cat/indices? V Viewing the results will be more convenient.

The result of the query is 55a4a25feff6, which generates a single index (docs. Count).

So far kafka-> Fluentd -> ES is a log collection process.

Of course, the architecture is not fixed. You can also use fluentd-> Kafka -> ES to collect data. The fluentd.conf configuration file is modified to switch es and Kafka-related configurations.

Fluentd-es plugin and Fluentd-kafka plugin can be found on Github or fluentd website.

“Recommended Reading”

Deploy standalone Kafka using Docker and Docker-compose

Use Fluentd as docker log driver to collect logs

Create fluentd images with Kafka and ES plug-ins