Video address of Station B:How did ELK and Kafka get together as a log collection solution

Video address of public account:How did ELK and Kafka get together as a log collection solution

Zhihu Video Address:How did ELK and Kafka get together as a log collection solution

Video text edition

I’m going to do a video on ELK and Kafka using a graph and some commands to understand how ELK is configured for log collection and Kafka is configured for message queue distribution. And the communication mechanism between them hope to help you learn, thank you! Our target is a host whose services generate logs, for example, the /var/log directory constantly generates various log records. We call this collection range a log collection source. At this point, we deploy fileBeat, ELK’s file collector, on this host. Filebeat can also push logs directly to ELK’s Elasticsearch search engine as well as to ELK’s Logstash stash tool. The most important step is that Filebeat can push logs directly as messages Send to Kafka cluster.

I’m going to use PWD to look at the directory where fileBeat is located. My installation directory is /opt/filebeat. Then I’m going to use ls to look at the files under that directory. Find the fileBeat configuration file filebeat.yml.

Filebeat. Yml (); / / output (); / / output (); / / output () Filebeat will not export data to Elastisearch. You can also see that the address of Elasticsearch is node1 and port 9200. Good! Then press n to search for the next output. At this time, we find the second configuration point: Logstash output. We can also see the Logstash output that has been # noted, which shows that the data collected by FileBeat will not be pushed to the Logstash. Node1, port 5044Ok, we continue to press n to search for the last data, at which point we find the last output point: Node2, node3, node4, node2, node3, node4 The ports are 9092 and the topic pushed to Kafka is testT3

Using the configuration shown in the command demo, let’s find the most reasonable solution for pipelining logs. First, FileBeat pushes data to the Kafka cluster, which creates a large buffer of data. And Kafka can receive more fileBeat collection points and then Logstash acts as Kafka’s consumer client to subscribe to testT3 Topics. Logstash has all kinds of filters, codecs, data cleaning, This provides higher quality log data for the next stage of output. Finally, Logstash writes the log data to the index of elasticSearch cluster to complete the log landing.

I’m a “Read Bytes” creator who delves into big data technology and interprets distributed architectures

Head over to Byte Read Zhihu — learn more about big data

Public id Read-byte “Read bytes” Distributed, big data, depth of software architecture, professional interpretation