What is ELK

ELK is an acronym for three open source projects: Elasticsearch, Logstash, and Kibana. Elasticsearch is a search and analysis engine. Logstash is a server-side data processing pipeline that captures data from multiple sources at the same time, transforms it, and sends it to a “repository” such as Elasticsearch. Kibana allows users to visualize data using graphs and charts in Elasticsearch.

SpringBoot can collect logs and write them to the ELK by integrating the Logstash – Logback-encoder plug-in. Of course, if you’re using Log4j2 or some other logging framework, you just need to integrate the plugin.

Ii. ELK construction

This paper uses docker-compose to deploy ES+Logstash+Kibana

New Project Directory

$ mkdir elk-demo
Copy the code

Docker-compose create docker-compose file as follows:

version: '3.5'

services: 
  elasticsearch:
    container_name: elk-es
    image: Elasticsearch: 6.5.0
    restart: always
    ports: 
      - 9200: 9200
      - 9300: 9300
    environment:
      - TZ=Asia/Shanghai
    volumes: 
      - ./es/data:/usr/share/elasticsearch/data
      - ./es/es-single.yml:/usr/share/elasticsearch/config/elasticsearch.yml
  kibana:
    container_name: elk-kibana
    image: Kibana: 6.5.0
    restart: always
    ports:
      - 5601: 5601
    environment: 
      - elasticsearch.hosts=http://elasticsearch:9200
    depends_on: 
      - elasticsearch
  logstash:
    container_name: elk-logstash
    image: Logstash: 6.5.0
    restart: always
    ports: 
      - 5044: 5044
      - 9600: 9600
Copy the code

Create the elasticSearch directory

$ mkdir -p es/data
Copy the code

Place the elasticSearch configuration file in the es folder with the contents of es-single.yml:

cluster.name: elasticsearch-single
node.name: es-single-node-1
network.bind_host: 0.0. 0. 0
http.port: 9200
transport.tcp.port: 9300
http.cors.enabled: true
http.cors.allow-origin: "*"
node.master: true
node.data: true
action.auto_create_index: true
Copy the code

When finished, go back to the project root directory and start ELK

$ docker-compose up -d
Copy the code

Install ik word dividers

$ docker exec -it elk-es /bin/bash
$ cd plugins/
Select elasticSearch from version 6.5.0$wget HTTP: / / http://github.com/medcl/elasticsearch-analysis-ik/releases/download/v6.5.0/elasticsearch-analysis-ik-6.5.0.zipCopy the code

The installation process may be slow or fail due to network problems. For details, see the manual installation document to install the package locally. If cloning git repositories is not possible, you can choose to use code cloud speed download

Once the download is complete, unzip to the specified directory

$mkdir ElasticSearch -analysis-ik $unzip ElasticSearch - analysis-IK-6.5.0.zip-d elasticsearch-analysis-ik
Copy the code

Other plug-ins can be installed as needed, and then restart the container to see if the IK splitter is loaded

$ docker restart elk-es
$ docker logs elk-es | grep ik
Copy the code

Configuration logstash

$ docker exec -it elk-logstash /bin/bash
$ cd /usr/share/logstash/config/
# modified xpack. Monitoring. Elasticsearch. Url value to http://elasticsearch:9200
$ vi logstash.yml
# modified logstash. Conf
$ cd /usr/share/logstash/pipeline/
$ vi logstash.conf
Copy the code

logstash.conf

input {
    tcp {
        port => 5044
        codec => json_lines
    }
}
output {
    elasticsearch {
        hosts => ["http://elasticsearch:9200"]
        index => "log-%{+YYYY.MM.dd}"
    }
    stdout {
        codec => rubydebug
    }
}
Copy the code

Restart the logstash

$ docker restart elk-logstash
Check whether the es connection is normal
$ docker logs -f elk-logstash
Copy the code

Elasticsearch may be slow to start, kibana will not connect when it first starts, just wait a few minutes until ElasticSearch is fully started.

SpringBoot access

Introduce logstuck-logback-encoder in POM files

<dependency>
    <groupId>net.logstash.logback</groupId>
    <artifactId>logstash-logback-encoder</artifactId>
    <version>6.3</version>
</dependency>
Copy the code

Add the log configuration file logback-spring. XML

<?xml version="1.0" encoding="UTF-8"? >
<configuration>
    <include resource="org/springframework/boot/logging/logback/base.xml" />

    <appender name="LOGSTASH" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
        <destination>localhost:5044</destination>
        <encoder charset="UTF-8" class="net.logstash.logback.encoder.LogstashEncoder" />
    </appender>

    <root level="INFO">
        <appender-ref ref="LOGSTASH" />
        <appender-ref ref="CONSOLE" />
    </root>
</configuration>
Copy the code

The destination value in the configuration file is changed according to the actual environment. Then start the SpringBoot project and view the logs through Kibana.

Conf file. The log collection format is log-%{+ YYYY.mm. Dd} and index patter is log-*. After creation, check the log through Discover menu.

At this point, the SpringBoot+ELK environment is built.