Elasticsearch

Distributed search and analysis engine. It has the characteristics of high scalability, high reliability and easy management. Built based on Apache Lucene, it can store, search and analyze large amount of data in near real time.

Logstash

Log collector. Collect various data sources, filter, analyze, and format the data, and store it in Elasticsearch.

Kibana

Data analysis and visualization platform. Use with Elasticsearch to search, analyze, and chart your data.

Filebeat

A lightweight open source log file data collector, Filebeat reads file contents and sends them to Logstash for parsing into Elasticsearch, or directly to Elasticsearch for centralized storage and analysis.

Architecture is introduced

Based on the usage of ELK, Logstash serves as the log collector, Elasticsearch stores logs, and Kibana serves as the log presentation.

Architecture is a

The reason why there are multiple Logstash files in the figure is that in the case of distributed architecture, each machine needs to deploy one Logstash. If it is really a single server, one Logstash can be deployed.

As mentioned above, Logstash analyzes, filters and formats data. These operations consume high CPU and memory resources of the server. Therefore, this architecture will affect the performance of each server, so it is not recommended.

Architecture 2

Compared with architecture 1, an MQ and a Logstash are added. The output and input of the Logstash support common message queues such as Kafka, Redis, and RabbitMQ. The Logstash before MQ is only used for collecting and transmitting logs, not parsed or filtered. Parsing and filtering is continued by the Logstash behind MQ so that each server does not consume too many resources.

Architecture three

This architecture is based on the simplification of architecture 2, and can be adopted in the actual use process, the log directly into MQ, Logstash consumption MQ data.

Architecture of four

This architecture adds Beats in logging data sources and Logstash (or Elasticsearch). Beats is a collection of a variety of single-purpose data collectors, each of which is based on libbeat, the universal library for forwarding data. Beat occupies almost negligible system CPU and memory. Libbeat also provides a detection mechanism. It automatically slows down the rate of occurrence. In the following example, we use Filebeat to collect file logs, other beats can be ignored.

Compared to Architecture 2, architecture 4 would be more ideal if the Logstash deployed on each server were replaced with the corresponding Beats.

However, the Logstash resource consumption for log parsing and filtering is high, so you can use distributed Logstash deployment if necessary, and cluster Elasticsearch deployment to enhance the entire log system.

The deployment of

The JDK, Java version 8, needs to be installed before deployment. Then download the installation package of the corresponding operating system. If Docker deployment is used, you can directly use the provided image.

Unzip the download package and you can start directly.

Logstash
Bin /logstash -f logstash. Conf # logstash. Conf is the log processing configuration file that you need to create yourself

Copy the code

The basic format of the configuration file is as follows:

# enter

input { 

}

# Analyze, filter

filter { 

}

# output

output { 

}

Copy the code
Elasticsearch
bin/elasticsearch

Copy the code

If root is not allowed to start, create a new user:

  1. Creating a User Group

    groupadd elsearch

    Copy the code

  2. Create a user

    useradd elsearch -g elsearch -p elsearch

    Copy the code

  3. Log in to the root user and change the permission of user elsearch on the elasticSearch folder (extracted from the elasticsearch directory) in the elsearch group

    chown -R elsearch:elsearch elasticsearch

    Copy the code

  4. Switch to user elsearch and restart ElasticSearch

Kibana
bin/kibana

Copy the code
Filebeat
filebeat -e -c filebeat.yml

Copy the code

Yml key configuration: all /var/log/. log files are output to port 5044 of the Logstash file

filebeat.prospectors:

- input_type: log

  paths:

	- /var/log/*.log

output.logstash:

  hosts: ["localhost:5044"]

Copy the code
Logstash examples:
Configuration file contents:
input {

  beats {

    port => 5044

    codec => "json"

  }

}



filter{

   if [logtype] {

     mutate { replace => { type => "%{logtype}" }}

   }else{

     mutate { replace => { type => 'unknow' }}

   }

   date {

     match => [ "createTime" , "yyyy-MM-dd HH:mm:ss" ]

   }

}



output {

  elasticsearch {

    hosts => ["localhost:9200"]

    index => "logstash-%{type}-%{+YYYY.MM.dd}"

  }

  stdout { codec => rubydebug }

}

Copy the code
Configuration file description:

Using Filebeat as the input to the Logstash file, the Logstash file listens on port 5044. In this example, we format the received log with json. If the json contains logtype, we set the logtype to logtype; if not, we set it to unknow. Format createTime as YYYY-MM-DD HH: MM :ss. Select * from elasticSearch; select * from elasticSearch; select * from elasticSearch;

Kibana using renderings in production environment