We’ve talked about that before
Open source component ELK logging system configuration and management

Today we’ll also talk about how ELK + FileBeat builds a log system

Elasticsearch

Distributed search and analysis engine. It has the characteristics of high scalability, high reliability and easy management. Based on Apache Lucene, it can store, search and analyze large volumes of data in near real time.

Logstash

Log collector. Collect data from ElasticSearch, filter, analyze, format, and store it in ElasticSearch.

Kibana

Data analysis and visualization platform. Use with ElasticSearch to search, analyze and chart data.

Filebeat

FileBeat is a lightweight open source data collector for log files. FileBeat reads the contents of the files, sends them to Logstash for parsing and then to ElasticSearch, or sends them directly to ElasticSearch for centralized storage and analysis.

Architecture is introduced

Based on the use of Elk, Logstash is used as a log collector, Elasticsearch is used for log storage, and Kibana is used for log rendering. The following architectures are generally used.

Architecture is a

The reason why there are more than one Logstash in the figure is that given the distributed architecture of the application, each machine needs to deploy one Logstash, or one Logstash if it is truly a single server.

As mentioned earlier, Logstash does data analysis, filtering, formatting, and other operations, which are very high on the CPU and memory resources of the server. Therefore, this architecture affects the performance of each server, so it is not recommended.

Architecture 2

Compared with the architecture I, an MQ and Logstash are added. The output and input of Logstash support common message queues such as Kafka, Redis and RabbitMQ. Logstash before MQ is only used for log collection and transmission, and does not parse and filter. Logstash behind MQ continues parsing and filtering so that each server is not consuming too much resources.

Architecture three

This architecture is simplified based on the second architecture, and can be adopted during the actual use of the log directly into MQ, Logstash can consume the MQ data.

Architecture of four

This architecture adds Beats to the log data source and Logstash (or Elasticsearch). BEATS sets a variety of single-purpose data collectors. Each collector is based on Libbeat, a common library used for forwarding data. The CPU and memory of the system occupied by Beat are almost negligible. It will automatically slow down the rate of occurrence. In the following example we use FileBeat to collect file logs, other beats can be ignored.

Architecture 4 would be preferable if Logstash was deployed on each server with the corresponding BEATS compared to Architecture 2.

However, Logstash is expensive for log parsing and filtering, so you can distribute your Logstash deployments if needed, and cluster your Elasticsearch deployments to enhance the entire logging system.

The deployment of

The JDK, Java 8 version, needs to be installed before deployment. Then the official download the corresponding operating system installation package, if using Docker deployment, directly use the provided image.

After downloading the package, you can start it directly.

Logstash
Bin/logstash-f logstash.conf # logstash.conf is a log processing configuration file that you need to create yourself

The basic format of the configuration file is as follows:

# input {} # filter {} # output {} # input {} # filter {} # output {}
Elasticsearch

bin/elasticsearch

If root is not allowed to start at startup, create a new user:

  1. Create a User Group

groupadd elsearch

  1. Create a user
useradd elsearch -g elsearch -p elsearch
  1. Go to root and change the Elsearch permissions on Elasticsearch from the Elsearch user group
chown -R elsearch:elsearch elasticsearch
  1. Go to the ElSearch user and restart ElasticSearch

Kibana

bin/kibana

Filebeat

filebeat -e -c filebeat.yml

Filebeat. yml key configuration, all /var/log/.log files will be exported to Logstash port 5044

 filebeat.prospectors:
 - input_type: log
   paths:
     - /var/log/*.log
 output.logstash:
   hosts: ["localhost:5044"]

Logstash examples:

Content of configuration file:

input {
  beats {
    port => 5044
    codec => "json"
  }
}
 filter{
    if [logtype] {
      mutate { replace => { type => "%{logtype}" }}
    }else{
      mutate { replace => { type => 'unknow' }}
    }
    date {
      match => [ "createTime" , "yyyy-MM-dd HH:mm:ss" ]
    }
 }
 output {
   elasticsearch {
     hosts => ["localhost:9200"]
     index => "logstash-%{type}-%{+YYYY.MM.dd}"
   }
   stdout { codec => rubydebug }
 }

Configuration file description:

Using FileBeat as Logstash’s input, Logstash listens on port 5044. In this example, we will format the received log with JSON. If the JSON contains logtype, we will set the log’s type to logtype. If not, we will set the log’s type to unknow. Then format the createTime in the log into the time format yyyy-mm-dd HH: MM :ss. The final output to elasticsearch, the index is the index name elasticsearch, we according to the different type create different indexes.

Renderings of Kibana using in a production environment

Reference:
http://beckjin.com/2017/12/10…