The environment that

  • CentOS Linux Release 7.5.1804 (Core)
  • Docker docker version 1.13.1
  • elk

    sebp/elk latest
  • Filebeat filebeat – 6.4.0

Elk is on the same machine as FileBeat

architecture

  • Elasticsearch is a full-text search engine for near real-time queries. Elasticsearch is designed to handle and search huge amounts of log data.

  • Logstash reads the raw log, analyzes and filters it, and then forwards it to other components (such as Elasticsearch) for indexing or storage. Logstash supports rich Input and Output types and handles logs for various applications.

  • Kibana is a JavaScript based Web graphical interface application designed to visualize Elasticsearch data. Kibana can query Elasticsearch and display the results with rich charts. Users can create dashboards to monitor system logs.

  • Filebeat Introduces Filebeat as a log collector to solve the problem of high Logstash overhead. Compared to Logstash, Filebeat takes up almost nothing of the system’s CPU and memory.

Log processing: Filebeat sends logs to The Logstash for analysis and filtering, and then the Logstash forwards logs to Elasticsearch. Finally, Kibana visualizes Elasticsearch data

Install the ELK suite

ELK deployment solutions can be very flexible, and in larger production systems, ELK has its own cluster, enabling high availability and load balancing. Our goal is to learn and practice ELK in the shortest possible time, so we will adopt a minimal deployment solution: build ELK in a container.

  • To run an ELK image, vm. Max_map_count requires at least 262144 memory
Conf vi /etc/sysctl.conf Add vm.max_map_count=262144 to the end of the file and run the sysctl -p commandCopy the code

Elk may start with the following error: Max virtual memory areas VM. Max_map_count [65530] is too low, increase to at least [262144] Docker logs container ID reference links: blog.csdn.net/jiankunking…

  • Install the docker
Yum install docker install systemctl start docker install systemctl start dockerenable docker
Copy the code
  • Run the ELK mirror
sudo docker run -p 5601:5601 -p 9200:9200 -p 5044:5044 -it --name elk sebp/elk
Copy the code
  • Configuration logstash
View container information docker ps-aEnter the container Sudo Dockerexec-it elk /bin/bash or sudo dockerexec-it container ID /bin/bash Modified 02-beats-input.confcd /etc/logstash/conf.d/
vi 02-beats-input.conf
Copy the code

/etc/logstash/conf.d/02-beats-input.conf

There is something wrong with the vi command here. I deleted the three lines line by line using the DEL key

Delete the following three lines. This example does not use a certificate. If you need a certificate, copy the logstash. CRT file to the client and add a path to filebeat.yml

ssl => true 
ssl_certificate => "/pki/tls/certs/logstash.crt"
ssl_key => "/pki/tls/private/logstash.key"
Copy the code

Note: Sebp/ELk docker creates a certificate logstash. CRT, which uses * wildcard by default. If you use a certificate, the server address used by FileBeat. yml must bea domain name, not an IP address

If you do not remove these three lines, you will receive the following error when you start FileBeat:

2018-09-12T10:01:29.770+0800 ERROR logstash/async.go:252 Failed to publish events caused by: Lumberjack protocol error 2018-09-12T10:01:29.775+0800 Error logstash/async. Go :252 Failed to publish events caused by: 2018-09-12T10:01:30.775+0800 ERROR pipeline/output.go:109 Failed to publish events: client is not connectedCopy the code
  • Restart the ELK container
Docker restart Container IDCopy the code
  • Kibana visual page

Enter http://ip:5601 in your browser and wait for a while to see the successful kibana startup management page

JSON interface for Elasticsearch: http://[Host IP]:9200/_search? pretty

Install Filebeat

There are many ways to install FileBeat. Here I use the RPM package to install FileBeat, which can be automatically registered as systemd service

  • Download the RPM package of FileBeat
cd/ opt/softwares wget HTTP: / / https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.4.0-x86_64.rpmCopy the code

Or to look at directly download the latest version of the website: www.elastic.co/downloads/b…

  • Install filebeat
The RPM - the ivh filebeat - 6.4.0 - x86_64. RPMCopy the code
  • Configuration filebeat
cd /etc/filebeat
vi filebeat.yml
Copy the code

Change the configuration to the following:

#=========================== Filebeat inputs =============================

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /opt/datas/logs/*/*.log
  tags: ["Test environment"]
  multiline:
    pattern: '^\s*(\d{4}|\d{2})\-(\d{2}|[a-zA-Z]{3})\-(\d{2}|\d{4})'
    # pattern: '^\s*("{)'
    negate: true
    match: after
    max_lines: 1000
    timeout: 30s
Copy the code

Paths: Multiline for paths where you want to fetch and analyze logs: If the merging operation is not carried out, when the log collection is very long or the log output is in XML format, the collection will be incomplete or the pattern will be divided into multiple parts: The configured regular expression specifies the expression to match (string that starts with 2017-11-15 08:04:23:889 time format). If no match is found, the rows will be merged. Refer to the link

Change the configuration to the one shown above. Comment out Elasticsearch Output and enable Logstash Output. Hosts: IP address of the elK host. If you want to send logs to Elasticsearc directly, edit this line. If you want to send logs to Logstash directly, edit this line. Only one line of Logstash output can be used

  • Start the FileBeat service
Start filebeat systemctl start filebeat.service Check the filebeat status. systemctl status filebeat.service Check the fileBeat log tail-f /var/log/filebeat/filebeat
Copy the code

Refer to the link: www.jianshu.com/p/7ca38fa88…

Kibana configuration

Click the Discover button in the upper left corner, as shown in the picture below, prompting to create “Index pattern” :

Reference link: blog.csdn.net/qq_39284787… Blog.csdn.net/boling_cava… www.cnblogs.com/CloudMan6/p… Blog.csdn.net/boling_cava…