Introduction to the

ELK stands for Elasticsearch, Logstash, and Kibana, all of which are open source software. A new FileBeat is added, which is a lightweight log collection and processing tool (Agent). FileBeat consumes less resources and is suitable for collecting logs from various servers and transferring them to Logstash. This tool is also recommended by the official.

ElasticsearchIt is an open source distributed search engine that collects, analyzes and stores data. Its features are: distributed, zero configuration, automatic discovery, index automatic sharding, index copy mechanism, restful interface, multi-data sources, automatic search load and so on.

LogstashIt is used to collect, analyze, and filter logs, and supports a large number of data acquisition methods. The client is installed on the host that needs to collect logs. The server filters and modifies the received node logs and sends them to ElasticSearch at the same time.

KibanaAlso an open source and free tool, Kibana provides a log analysis friendly Web interface for Logstash and ElasticSearch that helps aggregate, analyze and search important data logs.

The Logstash data processing process mainly consists of Inputs, Filters and Outputs:

  • Inputs: Inputs for obtaining data from data sources, such as Beats, file, Kafka, RabbitMQ, log4j, redis, etc. Reference: official website address
  • Filters: Filters are data handlers in the Logstash pipeline. When input triggers an event, the filter triggers the transport of data, that is, transforming and parsing data in various formats. Common filter plug-ins are as follows:
    • Grok: Parses and constructs arbitrary text. Grok is the basis of the Logstash filter and is widely used to export structure from unstructured data. Currently, Grok is the best Logstash method for parsing unstructured log data into structured and queriable content.
    • Mutate: Performs general conversions on event fields. Supports renaming, deleting, replacing, and modifying fields in events.
    • Date: Converts a string time field to a timestamp
    • Drop: Completely deletes events, such as debugging events.
    • Clone: A replication event. Fields may be added or deleted.
    • Geoip: Adds information about the geographic location of an IP address.
  • Outputs: for data output, common plug-ins such as:
    • Elasticsearch: The most efficient, convenient, easy to query memory, the most selective, official recommended!
    • File: Writes output data to the disk as a file.

Beats: A general term for an open source, lightweight logging collector based on golang.

Official website Introduction address

  • Lightweight: Small size, single function, developed based on the GO language, with inherent performance advantages, independent of the Java environment.
  • High performance: Consumes minimal CPU, memory, and I/O resources.
  • Open source: Hundreds of beats are maintained in the community.

In terms of functions, Beats is the younger brother. Thanks to the ecological advantages of Java, Logstash functions are obviously more powerful. However, The Logstash Forwarder has been criticized for its performance in data collection. Beats is designed to replace the Logstash Forwarder

Yml configuration (FileBeat example)

# can be set to multiple, - symbols are arrays. Log paths: - /IDEAproject/its_library/library-log/log_total.log output.logstash: hosts: [" 127.0.0.1:5044 "]Copy the code

Start the beta

./filebeat.exe -c localhost.yml
Copy the code

Logstash configuration

  • Log demo
The 2020-12-24 17:10:39. 591 [HTTP - nio - 4040 - exec - 10] INFO com. Its. If business. Controller. TagController - getIeltsTeacher  : userid == 10Copy the code

Yml configuration

input { beats { port => 5044 } } filter { grok { match => { "message" => "%{DATA:dateTime} \[%{NOTSPACE:thread}\] %{NOTSPACE:level} %{NOTSPACE:className} - %{NOTSPACE:methodName} : %{NOTSPACE:userid} == %{NOTSPACE:uid}" } } if "_grokparsefailure" in [tags] { drop {} } } output { elasticsearch { hosts  => ["http://localhost:9200"] index => "%{[@metadata][beat]}-%{[@metadata][version]}" } }Copy the code

Start the command

./bin/logstash -f confing/lcoalhost.yml
Copy the code

Kibana

{" _index ":" filebeat - 7.10.1 ", "_type" : "_doc", "_id" : "qSkDlHYBEJNmxDg2vB - C", "_score" : 1.0, "_source" : { "userid" : "userid", "message" : "The 2020-12-24 17:10:39. 591 [HTTP - nio - 4040 - exec - 10] INFO com. Its. If business. Controller. TagController - getIeltsTeacher : userid == 10", "log" : { "file" : { "path" : """D:\IDEAproject\its_library\library-log\log_total.log""" }, "offset" : 3022704 }, "@version" : "1", "ecs" : {"version" : "1.6.0"}, "methodName" : "getIeltsTeacher", "uid" : "10", "className" : "com.its.library.business.controller.TagController", "input" : { "type" : "log" }, "host" : { "name" : "JT-116400-1" }, "agent" : { "ephemeral_id" : "0e82c3f4-6f00-4cd8-b903-28703bd91bac", "type" : "filebeat", "id" : "D5f521aa-2ba2-4a7a-8c76-3661f54e9fb9 ", "hostname" : "JT-116400-1", "version" : "7.10.1", "name" : "Jt-116400-1"}, "dateTime" : "2020-12-24 17:10:39.591", "thread" : "http-NIO-4040-exec-10 ", "tags" : ["beats_input_codec_plain_applied"], "level" : "INFO", "@timestamp" : "2020-12-24T09:10:40.394z"}Copy the code

ElasticSearch complete directory

Elasticsearch is the basic application of Elasticsearch.Elasticsearch Mapping is the basic application of Elasticsearch.Elasticsearch is the basic application of Elasticsearch Elasticsearch tF-IDF algorithm and advanced search 8.Elasticsearch ELK