Note: the original is not easy, this article is the original article, without permission cannot be reproduced, please contact the author before reprinting

The correct way to open ElasticSearch, Kibana, logStash

Elast = Elast = Elast = Elast = Elast = Elast = Elast = Elast = Elast = Elast = Elast = Elast = Elast = Elast = Elast = Elast = Elast = Elast = Elast = Elast = Elast = Elast = Elast = Elast = Elast = Elast = Elast = Elast = Elast = Elast = Elast = Elast = Elast = Elast Configure the Elasticsearch cluster. Configure the Elasticsearch cluster. Configure the Elasticsearch cluster

preface

What is Elasticsearch? Why is it that every time you learn a technology, you learn it in English? It’s a real pain in the head. Since it’s in English, let’s take a peek at Elasticsearch with The help of Youdao. There are two separate words, search and elastic. Let’s take a look at brainless Youdao and get the following explanation:

Elasticsearch is a very elastic, flexible, elastic, and searchable tool. * o (≧ del ≦) ツ ┏ ━ ┓

Baidu Encyclopedia explains it as follows:

ElasticSearch is a Lucene-based search server. It provides a distributed multi – user – capable full – text search engine based on RESTful Web interface. Elasticsearch, developed in the Java language and released as open source under the Apache license, is a popular enterprise-level search engine. ElasticSearch for cloud computing is stable, reliable, fast, easy to install and use. Official clients are available in Java,.net (C#), PHP, Python, Apache, Groovy, Ruby, and many other languages. Elasticsearch is the most popular enterprise search engine, according to DB-Engines’ rankings, followed by Apache Solr, also based on Lucene.

Elasticsearch is a real-time, distributed storage search engine. In the process of development, we often put data into the Elasticsearch engine, and then extract the data from the engine. There are a lot of apis for retrieving data, such as sorting, conditional query, etc. The most powerful feature of Elasticsearch is fuzzy search. For those of you who have experience with Mysql but have not yet worked with Elasticsearch, you may wonder if the Mysql method is not unambiguable. Where and cannot be searched conditionally? Can’t orderby sort data? I randomly randomly pick up a Sql statement can not achieve business requirements?

select department_name, count(*) Number of employees

from departments d, employees e

where d.department_id = e.department_id

group by d.department_id

having count(*) >5

order by count(*) desc;

Copy the code

Yes, the above SQL code can fulfill the actual requirements, but as our business becomes more and more complex and large, and our user base grows, we have to think from the user’s point of view. Imagine what kind of picture it would be if every day taobao er opened Taobao to search for the data they want to wait for dozens of seconds. Again, for example, in our common file open to retrieve the data when we need, TXT, word, excel, for example, we can quickly open the commonly, because these files take up the actual space is too small, most of these files are a few KB, assuming that we open a to G the log files for the unit, the system can also be as normal as usual? In other words, Elasticsearch is a search by index, it has a great search capability, it is stable, reliable, fast and installs.

Elasticsearch works with a data collection and log resolution engine Logstash and an analysis and visualization platform called Kibana, also known as the ELK system. The text will mainly introduce the following aspects

  • Build ELK based on Docker container
  • Creating the Elasticsearch cluster
  • IK word splitter plugin for Elasticsearch
  • Import and export Elasticsearch data based on Logstash

I. Establishment of ELK system

Elasticsearch is a real-time full-text search and analysis engine that collects, analyzes, and stores data. Is a set of open REST and JAVA API architecture to provide efficient search capabilities, scalable distributed system. It is built on top of the Apache Lucene search engine library.

Logstash is a tool for collecting, analyzing, and filtering logs. It supports almost any type of logging, including system logging, error logging, and custom application logging. It can receive logs from many sources, including syslog, messaging (such as RabbitMQ), and JMX, and it can output data in a variety of ways, including email, WebSockets, and Elasticsearch. 1 Kibana is a Web-based graphical interface for searching, analyzing, and visualizing log data stored in Elasticsearch metrics. It utilizes Elasticsearch’s REST interface to retrieve data, allowing users not only to create custom dashboard views of their own data, but also to query and filter data in special ways

Summary: Elasticsearch for search, Kibana for visualization, And Logstash for collection. Next, we will build an ELK system based on Docker. The installation and basic use of Docker have been recorded in the previous article, so there is no more introduction here. In addition, it is worth noting the versioning relationship between the three, (if you use other versions of the tool, building as follows may cause other problems) :

  • Elasticsearch: 5.6.8
  • Kibana: 5.6.8
  • Logstash:lastest

The installation of the Elasticsearch

  • Docker pulls the Elasticsearch image
Docker pull elasticsearch: 5.6.8

Copy the code
  • Create the configuration file and data directory mapped by the ElasticSearch container locally

Set http.host to 0.0.0.0 and write the configuration to the elasticSearch. yml configuration file in the config directory

#Create a configuration file on the centos and configure it

The mkdir -p/resources/elasticsearch/config # create config directory

The mkdir -p/resources/elasticsearch # / data to create a data directory

#Set http.host to 0.0.0.0 and write the configuration to the elasticSearch. yml configuration file in the config directory

Echo "HTTP. Host: 0.0.0.0" > > / resources/elasticsearch/config/elasticsearch yml

Copy the code
  • Create a elasticSearch container and start it

For elasticSearch, you need to create a container and start it up. For ElasticSearch, you need to create a container and start it up. Parameters for wearing an ES container are described as follows:

  1. –name: Alias the container
  2. -p: maps the running port of the container to the local port
  3. “Discovery. type=single-node” : indicates that the node is created in single-node mode. This section describes how to set up the cluster mode later
  4. -v: maps the configuration file and data file in the container to the file created locally for later configuration
# create a single node for elasticSearch and start it.

# Note: in docker, \ means newline

docker run --name elasticsearch -p 9200:9200 \

-e "discovery.type=single-node" \

-e ES_JAVA_OPTS="-Xms256m -Xmx256m" \

-v /resources/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml \

-v /resources/elasticsearch/data:/usr/share/elasticsearch/data -d elasticsearch:5.68.

# Parameter description:

# --name: Give the container an alias

# -p: Map the container's run port to the local port

# -e "discovery.type=single-node" : indicates that the discovery mode is created in single-node mode

# -v: indicates that the configuration file and data file in the container are mapped to the file created locally above for later configuration



Select elasticSearch from elasticSearch

docker update new-elasticsearch --restart=always

Copy the code

You can use curl to test elasticSearch:

Delete from elasticSearch's runtime port using curl

curl localhost:9200

If the following output is displayed, the installation is successful

{

  "name" : "XwmNOpR".

  "cluster_name" : "elasticsearch".

  "cluster_uuid" : "yB3VNHxmQzevk1vXUQTkcg".

  "version" : {

    "number" : "5.6.8".

    "build_hash" : "688ecce".

    "build_date" : "The T16 2018-02-16: now. 010 z".

    "build_snapshot" : false.

    "lucene_version" : "6.6.1"

  },

  "tagline" : "You Know, for Search"

}

Copy the code

If the Aliyun or Tencent cloud server is used, you need to configure a security group on the console. Otherwise, the vm cannot be accessed

Install kibana

  • Docker pulls the Kibana image
#Docker pulls the Kibana image

Docker pull kibana: 5.6.8

Copy the code
  • Create the container and set it to boot
#Create the Kibana container and specify the address of ElasticSearch

docker run --name kibana -e ELASTICSEARCH_URL=http://[ip]:9200 -p 5601:5601 \

- d kibana: 5.6.8

#Configure automatic startup upon startup

docker update new-kibana --restart=always

Copy the code

To curl up with kibana, you can download http://[IP]:5601. To curl up with Kibana: http://[IP]:5601.

[root@iZm5eei156c9h3hrdjpe77Z ~]# curl localhost:5601

<script>var hashRoute = '/app/kibana';

var defaultRoute = '/app/kibana';



var hash = window.location.hash;

if (hash.length) {

  window.location = hashRoute + hash;

} else {

  window.location = defaultRoute;

}</script>[root@iZm5eei156c9h3hrdjpe77Z ~]#

Copy the code

Install the logstash

  • Pull the Logstash image
#Pull the mirror

docker pull logstash

Copy the code
  • Create a configuration file and configure input and output

Create the logstash. Conf file in /resources/logstash and use vim to edit it as follows:

Create the logstash. Conf file in /resources/logstash and edit it using vim

mkdir /resources/logstash

# Edit using Vim

vim logstash.conf

# change the IP address of Elasticsearch to the IP address of Elasticsearch

input {

    tcp {

        port => 4560

        codec => json_lines

    }

}

output{

  elasticsearch { 

    hosts => ["[ip]:9200"] 

    index => "applog"

    }

  stdout { codec => rubydebug }

}

Copy the code
  • Create a container and boot it up
#Create a container, specifying Elasticsearch for the link

docker run -d -p 4560:4560 \

-v /resources/logstash/logstash.conf:/etc/logstash.conf \

--link elasticsearch:elasticsearch \

--name logstash logstash \

logstash -f /etc/logstash.conf



#Self-start upon startup

docker update new-logstash --restart=always

Copy the code

This is the setup of the Logstash container. You can use it in a simple way.

Go to the Logstash container and CD to the bin directory

docker exec -it logstash /bin/bash

cd /usr/share/logstash/bin

Copy the code

To execute the logstash command, note that the –path.data parameter needs to be specified here, otherwise the running process will fail

#Note: the --path.data parameter needs to be specified here, otherwise the running process will fail

logstash -e 'input { stdin { } } output { stdout {} }' --path.data=/root/

Copy the code

After running, we type Hello World on the console and the following results appear

Import and export Elasticsearch database data based on Logstash

Application requirements: 192.168.220.202 es does not have the info index library. In this case, we can export the Skuinfo index library from 192.168.220.202 to a JSON file with a Logstash stash. Then import the file to the ES index library at 192.168.220.201 with a logstash stash. How can such a requirement be realized?

Use logstash to export es data from the index repository

Create a temporary folder to hold the exported data and configuration files, then use vim to create and configure an export.conf configuration file

mkdir /resources/mydata/logstash_temp

vim export.conf

Copy the code

The configuration of the export.conf file is as follows

# will be192.168220.202.The info index library of Elasticsearch is exported as an info.json file

input{

     elasticsearch {

        hosts => ["192.168.220.202:9200"] # specify the address of Elasticsearch that contains the target data

                index => "info"# specify the index library to export

                size => 200# specify the size of each batch of exported data. Do not set it too large, otherwise it will fail

            scroll => "5m"

                docinfo => false

    }

}

output{

   file {

     path => "info.json"# specify the path to save the data and the name of the JSON file

   }

 }

Copy the code

Use the docker cp command to copy the exported configuration file to the bin directory of the Logstash container

#Cp the export. Conf file to the logstash container

docker cp ./export.conf logstash:/usr/share/logstash/bin

Copy the code

Go to the Logstash container and execute the configuration file. Note: Be sure to specify the path.data property, otherwise an error will be reported

#Go to the Logstash and execute the configuration file

docker exec -it logstash /bin/bash

cd /usr/share/logstash/bin

/logstash -f./export. Conf --path.data=/root/

Copy the code

When this is done, an info.json file is generated in the current directory, which we export to centos

#Json file is generated in the current directory and exported to centos

docker cp logstash:/usr/share/logstash/bin/info.json /resources/mydata/

Copy the code

Can complete the index in the library data export into a json file, the file exists in the/resources/mydata/info. The json

Logstash Import of Elasticsearch index library data

  • The JSON data is imported and the CONF file is configured
  1. Use the XFTP tool to import the info.json file in 192.168.220.202 to the Windows PC
  2. The file through XFTP tools imported into 192.168.220.201 / resources/mydata/logstash_temp
  3. Edit the import.conf configuration file using vim
#1. Use the XFTP tool to import the info.json file in 192.168.220.202 to the Windows PC

#2. Will the file by XFTP tools imported into 192.168.220.201 / resources/mydata/logstash_temp

cd /mydata/mysources/logstash_temp

#Configure import.conf using vim

vim import.conf

Copy the code
  • The configuration file of import.conf is as follows
# Read json file

input {

  file {

# set json file path, multiple file paths can be set as array [], fuzzy match with *

    path= > "/root/info.json"

    start_position => "beginning"

# set encoding

    codec => json {charset => "UTF-8"}

Use type to specify input/output paths when multiple files exist

    type => "json_index"

  }

}



Filter formatted data

filter {

    mutate{

# delete invalid fields

        remove_field => ["@version"."message"."host"."path"]

    }

# add timestamp field, add @timestamp time8hours

    ruby { code => "event.set('timestamp', event.get('@timestamp').time.localtime + 8*60*60)" }



}



Data output to ES

output {

# log output format json_lines; Rubydebug etc.

    stdout {

        codec => rubydebug

    }

# output to es

    if[type] == "json_index"{

# Unparsed JSON is not logged in elasticSearch

        if "_jsonparsefailure" not in [tags] {

            elasticsearch {

#es address IP port

                hosts => "192.168.220.201:9200"

The configuration data is transferred to es

                index => "info"

# type

                document_type => "info"

            }

        }

    }

}

Copy the code
  • After configuring the import.conf file, we can actually implement the data import
#Import the import.conf and info.json files into the Logstash container

Docker cp. / import. Conf logstash: / usr/share/logstash/bin # import conf configuration file

Docker cp./skuinfo.json logstash:/root/ # Import the JSON data file. Ensure that the file path is the same as that in the CONF configuration file

#Enter the logstash container and run the logstash command to import the data

docker exec -it logstash /bin/bash

cd /usr/share/logstash/bin

./logstash -f ./import.conf --path.data=/root/

Copy the code

After the command is executed, access 192.168.220.201:5601 to access the data in the INFO database

Install IK Chinese word divider

Installation of IK Chinese word segmentation plug-in

When installing ik tokenizer, you can download zip files from Github, upload them to centos, and then upload them to elasticserch. However, github has a different version from the actual version. On the other hand, when we downloaded different versions of ElasticSearch, some versions of the container ran incorrectly and some didn’t. Therefore, we will install version 5.6.8 when we build ELK in the future (be careful).

Pit 2: Github is usually very slow to download zip files (whatever they are), so you can use Gitee to download zip files from Github. When creating a repository on github, choose to import the existing repository. Then copy and paste the original Github Git connection to create git. Once created, we can indirectly download any file from the Github repository through the code cloud, and download it much faster

The installation process and code examples are as follows:

  1. Download 5.6.8 ik participle: https://gitee.com/tianxingjian123/elasticsearch-analysis-ik
  2. Package it using Maven: CD to IK directory, then MVN package: MVN package-pdist,native dskiptests-dtar
  3. Once packaged using Maven, you can create a target folder with the./ Releases/ElasticSearch-Analysis-IK-5.6.8.zip zip file
  4. Create an IK folder on the VM and upload the ZIP file to the IK folder. Then run the unzip command to decompress the ZIP file and delete the ZIP file
  5. Then use the Docker to upload the IK folder to the elasticSearch container’s plugins
  6. Enter the elasticSearch container and run the ls command to check whether the IK folder is successfully uploaded

Ik participle link: gitee.com/tianxingjia…

# ik word-segmentation link: HTTPS://gitee.com/tianxingjian123/elasticsearch-analysis-ik

# download5.68.After the IK segmentation, we need to package it using Maven

cd C:\Users\M\Desktop\code-demo\elasticsearch-analysis-ik

mvn package -Pdist,native -DskipTests -Dtar

/ Releases/ElasticSearch-analysis-ik./releases/ ElasticSearch-analysis-ik5.68..zip

Create an IK folder in the virtual machine

mkdir ik

# Then upload the zip file to the IK folder using XFTP, unzip the zip file, and delete the zip file after unzip

unzip elasticsearch-analysis-ik5.68..zip

rm -rf elasticsearch-analysis-ik5.68..zip

Then use docker to upload the IK folder to elasticSearch's plugins

docker cp ./ik elasticsearch:/usr/share/elasticsearch/plugins

Enter the elasticSearch container

docker exec -it new-elasticsearch /bin/bash

To check whether the ik folder was successfully uploaded, run the following command

root@78f36ce60b3f:/usr/share/elasticsearch# cd plugins/

root@78f36ce60b3f:/usr/share/elasticsearch/plugins# ls

ik

root@78f36ce60b3f:/usr/share/elasticsearch/plugins# cd ik

root@78f36ce60b3f:/usr/share/elasticsearch/plugins/ik# ls

commons-codec1.9.jar             httpclient4.52..jar

commons-logging1.2.jar             httpcore4.44..jar

config                     plugin-descriptor.properties

elasticsearch-analysis-ik5.68..jar

root@78f36ce60b3f:/usr/share/elasticsearch/plugins/ik# 

Then go to the bin directory and view the installed IK tokenizer plug-in

root@78f36ce60b3f:cd /usr/share/elasticsearch/bin

root@78f36ce60b3f:/usr/share/elasticsearch/bin# elasticsearch-plugin list

ik

Copy the code

The ik Chinese word splitter plugin for Elasticsearch has been added to Elasticsearch. The ik Chinese word splitter plugin for Elasticsearch has been added to Elasticsearch.

Word segmentation results test

# open chrom browser, visit: http://192.168.220.201:5601/, if appear kibana interface, kibana install normally

Go to the Kibana Dev Tools screen and test whether the IK Chinese word segmentation plugin is properly installed

GET bank/_analyze

{

  "text": "Now it is 1:30 in the morning, a little cold, I finish writing this article to sleep!".

  "analyzer": "ik_smart"

}

Copy the code

After running, the result of test IK Chinese word segmentation is as follows, indicating that Chinese sentence segmentation has been completed

Create Elasticsearch cluster

Create the configuration files and data files required by the cluster for container mapping

During each build, you need to ensure that the data directory is empty, otherwise it will cause errors during the actual operation

mkdir /mydata

cd /mydata

mkdir elasticsearch1

cd elasticsearch1

mkdir data Make sure that the data directory is empty, otherwise it will fail in the actual run

mkdir config

cd conf

vim elasticsearch.yml

The configuration information of the # elasticSearch. yml file is as follows

Copy the code

The configuration information of elasticSearch. yml file is as follows:

  1. Enable cross-domain. In order for es-Head to be accessible, you need to install an additional header plug-in here
  2. Set the name of the cluster (the same for each cluster node)
  3. Specifies whether this node is eligible to be elected as the master node. Default is true. Es is the first machine in the default cluster to be the master
  4. Allows the node to store data (enabled by default) and allow any IP address to access it
In order for es-head to be accessible, you need to install the header plugin

http.cors.enabled: true

http.cors.allow-origin: "*"



# cluster name (same)

cluster.name: elasticsearch

# name of node (different, according to alias)

node.name: es1

The default value is true. Es indicates that the first machine in the cluster is the master. If this machine fails, the master will be reelected

node.master: true

# Allow this node to store data (default enabled)

node.data: true

Allow any IP access

network.host: 0.0. 0. 0

# Node discovery through this IP list, I have configured the IP of each container

discovery.zen.ping.unicast.hosts: ["192.168.220.200:9300"."192.168.220.200:9301"."192.168.220.200:9302"]

# Without this setting, a cluster suffering from a network failure could split the cluster into two separate clusters -- resulting in a split brain -- which would result in data loss

discovery.zen.minimum_master_nodes: 2

Copy the code

After the first ElasticSearch configuration file is created, create the other two nodes

# configuration es2

cd /mydata

cp -r ./elasticsearch1 ./elasticsearch2

Yml/elasticSearch.yml/elasticSearch.yml/elasticSearch.yml/elasticSearch.yml

node.name=es2



# configuration es3

cd /mydata

cp -r ./elasticsearch1 ./elasticsearch3

Yml/elasticSearch.yml/elasticSearch.yml/elasticSearch.yml/elasticSearch.yml

node.name=es3

Copy the code

Create elasticSearch container and start it

# create es1 container and start

docker run --name es1 -p 9200:9200 -p 9300:9300 \

-e ES_JAVA_OPTS="-Xms256m -Xmx256m" \

-v /mydata/elasticsearch1/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml \

-v /mydata/elasticsearch1/data:/usr/share/elasticsearch/data -d elasticsearch:5.68.



# introduce ik segmentation

docker cp ./ik es1:/usr/share/elasticsearch/plugins



Create an ES2 container and start it

docker run --name es2 -p 9201:9200 -p 9301:9300 \

-e ES_JAVA_OPTS="-Xms256m -Xmx256m" \

-v /mydata/elasticsearch2/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml \

-v /mydata/elasticsearch2/data:/usr/share/elasticsearch/data -d elasticsearch:5.68.



Create an ES3 container and start it

docker run --name es3 -p 9202:9200 -p 9302:9300 \

-e ES_JAVA_OPTS="-Xms256m -Xmx256m" \

-v /mydata/elasticsearch3/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml \

-v /mydata/elasticsearch3/data:/usr/share/elasticsearch/data -d elasticsearch:5.68.

Copy the code

You can set up the Elasticsearch cluster

  • Access, respectively,http://192.168.220.200:9200, http://192.168.220.200:9201, http://192.168.220.200:9202, it can be found that the construction has been completed, and the corresponding information of each ES node is displayed.
  • accesshttp://192.168.220.200:9200/_cat/nodesYou can view cluster nodes.
  • accesshttp://192.168.220.200:9200//_cat/healthViewing health Status (Green, Yellow, red)

Meet the problem

  • After instantiating the container, we may be limited by the number of processes, and we need to increase the number of processes in the virtual machine
vim /etc/sysctl.conf

Add the following configuration

vm.max_map_count=655360

Exit and restart the configuration

sysctl -p

Copy the code
  • After the above three ES containers are instantiated, there may be some memory problems, in which case, we need to increase the memory of the virtual machine

After running the above command, may find that the current configuration available only 50, and when we visit http://192.168.220.200:9200 will find request failed at this time we open the corresponding virtual machine Settings, set the memory 3 gb, in a short time back into the virtual machine, Using the free -m command, you can find that the available memory is more than 1000

After the above three ES containers are instantiated, check the current available memory

free -m

# after running the above command, may find that the current configuration available only 50, and when we visit http://192.168.220.200:9200 will find request failed at this time we open the corresponding virtual machine Settings, set the memory 3 gb, in a short time back into the virtual machine, Using the free -m command, you can find that the available memory is more than 1000



After the above configuration, we restart the three ES containers

docker restart es1 es2 es3 || docker start es1 es2 es3



After the container restarts, use chrom to access es

http://192.168220.200.:9200

http://192.168220.200.:9201

http://192.168220.200.:9202

It can be found that the construction has been completed, and the corresponding information of each ES node is displayed



# View cluster nodes under Kibana dev Tools

GET /_cat/nodes

# Check health status (green, yellow, red)

GET /_cat/health

Copy the code
2020-03-08, By Tao Tao, Yu Shangrao