introduce

The so-called elk respectively elasticsearch, logstash, kibana.

Elasticsearch is a distributed open source search and analysis engine for all types of data, including text, digital, structured and unstructured data. Elasticsearch is a near-real-time full-text search engine designed to process and search huge amounts of log data

Logstash: Reads raw logs, analyzes and filters them, and forwards them to other components (such as Elasticsearch) for indexing or storage. Logstash supports rich Input and output types to handle logs for a variety of applications

Kibana: A JavaScript based Web graphical interface application designed to visualize Elasticsearch data. Kibana can query Elasticsearch and display the results with rich charts. Users can create dashboards to monitor system logs

Filebeat: The introduction of Filebeat as a log collector is mainly to solve the problem of high overhead of logstash. Compared with this problem, Filebeat occupies almost negligible amount of CPU and memory

Log Processing Process

Filebeat sends logs to LogStash for analysis and filtering, which then forwards them to ElasticSearch, and Kibana displays elasticSearch data

Environment and software version

  1. System: centos7.7

  2. Host: 192.168.0.192

  3. Filebeat: Filebeat 6.2.4 – x86_64. RPM

  4. Elk: sebp/Elk: the latest

Docker installation configuration

Docker installation configuration here is not repeated, remember to configure ali cloud accelerator ok

Elk installation and configuration

Setting the memory size

Running an ELK image requires at least 262144 memory for vm.max_map_count

  • Edit the system configuration file and add the following content

    vim /etc/sysctl.conf
Copy the code
    vm.max_map_count = 262144
Copy the code

  • Wq Save and exit

  • Check whether it takes effect.

    Sysctl -pCopy the code

Install the elk

The integration version of Docker-ELk is used here, instead of downloading each image

  • Pull the mirror

    docker pull sebp/elk
Copy the code

  • Look at mirror

    docker images
Copy the code

  • Start the elk

    docker run -dit --name elk \
    -p 5601:5601 \
    -p 9200:9200 \
    -p 5044:5044 \
    -v /home/elk:/var/lib/elasticsearch \
    -v /etc/localtime:/etc/localtime \
sebp/elk
Copy the code

    – p: three port corresponding Kibana, elasticsearch, logstash port

    -v: mounts the container directory to the host directory for data synchronization

access

The above is the elK installation start process, the following access, test whether the installation is successful

  • Elasticsearch

    http://192.168.0.192:9200/

  • Kibana

    http://192.168.0.192:5601/

Modify the

The image above shows that the installation started successfully. Here are some simple changes. In the demo and test environment, SSL is not necessary

  • Enter the ELK container to edit the configuration file and delete the CONFIGURATIONS related to SSL authentication

    docker exec -it elk bash
Copy the code
    vim /etc/logstash/conf.d/02-beats-input.conf
Copy the code

    Delete the following

    The netizens’ experience is shown in the picture below

  • Exit the container and restart

    Docker restart elkCopy the code

Filebeat installation and configuration

Here there are two ways to install FileBeat, one is directly installed online, the other is to use docker image pull configuration,docker advantage is completely isolated from the host, to delete directly delete the image is done, here I use direct online installation

Install filebeat

The curl - L - O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.2.4-x86_64.rpmCopy the code

Note: the above way, affinity, slow, suggest direct reproduction artifacts. Elastic. Co/downloads/b… Link to the browser, load to the desktop, and finally upload to the server

Delete and uninstall half of fileBeat before uploading

  • View the installed FileBeat

    rpm -qa | grep filebeat
Copy the code

  • Uninstall filebeat

    Yum -y remove filebeat 6.2.4-1. X86_64Copy the code

  • upload

    Here I upload to the /root directory

  • The installation

    Sudo RPM - vi filebeat - 6.2.4 - x86_64. RPMCopy the code

  • configuration

    By default, the configuration file is in the directory shown in the following figure

    vim /etc/filebeat/ filebeat.yml
Copy the code

  • Modify the

    Enabled: Change fasle to true (fileBeat 6.0 is disabled by default and must be changed to true)

    Paths: Indicates the path where you want to capture the logs for analysis. I don’t want to change this

    Open elasticSearch,logstash and configure host IP + port as shown below

    Change the configuration to the one shown above. Comment out Elasticsearch Output and enable Logstash Output. If you want to send a log directly to Elasticsearc, edit this line: Elasticsearch Output If you want to send a log directly to Logstash, edit this line: Logstash Output

    Save the Settings and exit

  • Restart filebeat

    /etc/init.d/filebeat restart
Copy the code

Access Kibana and create the default index

Above the default index has been created, the console will display all of the log file under/var/log/log, this is our in/etc/filebeat/filebeat yml configuration

test

Now we manually write a log file to /etc/log/, as shown below

echo 'Liucong big fool...... ' >> /var/log/info.log
Copy the code

Go back to the console, hit Refresh and it comes out