ElasticSearch Logstash Kibana ElasticSearch Logstash Kibana

I am interested in Elasticsearch, so I will record the configuration process why I use ELK

ELK is actually three tools, Elastricsearch + LogStash + Kibana. Through ELK, it is used to collect logs and analyze logs. Finally, it is displayed through visual UI. At the beginning, when the service volume is small, simple SLF4J+Logger prints logs on the server, and grep is used for simple query. However, as the service volume increases, the data volume also increases, so ELK can be used to collect and analyze a large number of logs

Briefly draw the architecture diagram

“In the environment configuration, mainly introduces the Mac and Linux configuration, Windows system is roughly the same, of course, if everyone has installed JDK1.8 or later ~”

[root@VM_234_23_centos ~]# java -version
java version "1.8.0_161"
Java(TM) SE Runtime Environment (build 1.8.0_161-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)
1234
Copy the code

Note that older VERSIONS of ELK also require older JDK support. The ELK version configured in this document is 6.0+, so you need a JDK version at least 1.8

ElasticSearch

Elasticsearch is a distributed, RESTful search and data analysis engine that addresses a growing variety of use cases. At the heart of the Elastic Stack, it stores your data centrally, helping you find what you expect and what you don’t expect.

Mac installation and running

Elasticsearch: brew install ElasticSearchCopy the code

Linux: Download the Elasticsearch file from the official address of Elasticsearch (you can also download the file using an FTP tool), decompress the gz file using tar, and go to the bin directory to run the software

[root @ VM_234_23_centos app] # curl - L - O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.4.tar.gz [root@VM_234_23_centos app]# tar -zxvf ElasticSearch-6.2.4.tar. gz [root@VM_234_23_centos app]# CD elasticSearch-6.2.4 [root @ VM_234_23_centos elasticsearch - 6.2.4] #. / bin/elasticsearch 1234Copy the code

Note that a new user group is required to run ElasticSearch on a Linux machine, and there is a dent record of Elastic installing on Linux at the end of this article

Logstash

Logstash is an open source server-side data processing pipeline that enables you to capture data from multiple sources simultaneously, transform it, and then send it to your favorite “repository.” (Our repository is, of course, Elasticsearch.) – Official cute

1. Software Installation Mac Installation:

brew install logstash
1
Copy the code

Linux installation:

[root @ VM_234_23_centos app] # curl - L - O % of Total % at https://artifacts.elastic.co/downloads/logstash/logstash-6.3.2.tar.gz Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 137M 100 137M 0 0 5849k 0 0:00:24 0:00:24 --:--:-- 6597k [root@VM_234_23_centos app]# tar -zxvf logstuck-6.3.2.tar.gz 12345Copy the code

Vim /etc/logstash. Conf Conf file to specify the elasticSearch plug-in and configure the hosts for elasticSearch.

input { stdin { } }
output {
  elasticsearch { hosts => ["localhost:9200"] }
  stdout { codec => rubydebug }
}
12345
Copy the code

“3. Run”

bin/logstash -f logstash.conf
1
Copy the code

4. Visit http://localhost:9600/

{" host ":" = - = ", "version" : "6.2.4", "http_address" : "127.0.0.1:9600", "id" : "5b47e81f-bdf8-48fc-9537-400107a13bd2", "name": "=-=", "build_date": "2018-04-12T22:29:17Z", "build_sha": "a425a422e03087ac34ad6949f7c95ec6d27faf14", "build_snapshot": false } 12345678910Copy the code

“In elasticSearch log, you can also see the logstash log.”

[the 2018-08-16 T14:08:36, 436] [INFO] [O.E.C.M.M etaDataIndexTemplateService] [f2s1SD8] adding template [logstash] for the index patterns [logstash-*] 1Copy the code

“Seeing this return value indicates successful installation and startup.” At that stage of the run, it is possible to encounter memory allocation errors

Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000c5330000, 986513408, 0) failed; Error = 'Cannot allocate memory' (errno=12) 1Copy the code

This error is obviously insufficient memory, because I bought a server with 1GB memory of Tencent Cloud (if it is a server, please buy a higher configuration =-=), I have already run ElasticSearch, so the Logstash cannot allocate enough memory, so I need to modify the JVM configuration at last.

[root@VM_234_23_centos logstash-6.3.2]# CD config/ [root@VM_234_23_centos config]# ll total 28 -rw-r--r-- 1 root root 1846 Jul 20 14:19 jvm.options -rw-r--r-- 1 root root 4466 Jul 20 14:19 log4j2.properties -rw-r--r-- 1 root root 8097 Jul  20 14:19 logstash.yml -rw-r--r-- 1 root root 3244 Jul 20 14:19 pipelines.yml -rw-r--r-- 1 root root 1696 Jul 20 14:19 startup.options [root@VM_234_23_centos config]# vim jvm.options 123456789Copy the code

“Change -xms1g -XMx1g to”

-Xms256m
-Xmx256m
12
Copy the code

“Then it starts normally ~~” Kibana

1. Software Installation

Kibana allows you to visualize data in Elasticsearch and manipulate ElasticStack, so you can solve any questions here: why you’re pold at 2am, for example, or how rain affects quarterly data. (And the ICONS are really cool)

Mac installation

brew install kibana
1
Copy the code

Linux installation, official download address:

www.elastic.co/downloads/k…

[root @ VM_234_23_centos app] # curl - L - https://artifacts.elastic.co/downloads/kibana/kibana-6.3.2-linux-x86_64.tar.gz O %  Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 195M 0 271k 0 0 19235 0 2:57:54 0:00:14 2:57:40 26393 1234Copy the code

“At this stage, the download may be very slow, so I will download it locally and transfer it to the server through the Rz command.”

[root@VM_234_23_centos app]# rz rz waiting to receive. Starting zmodem transfer. Press Ctrl+C to cancel. Transferring Kibana - 6.3.2 - Linux - x86_64. Tar. Gz... Errors [root@VM_234_23_centos app]# tar -zxvf kibana-6.3.2-linux-x86_64.tar.gz [root@VM_234_23_centos app]# tar -zxvf kibana-6.3.2-linux-x86_64.tar.gz 123456Copy the code

2. Modify the Configuration

Modify the config/kibana.yml file to specify elasticSearch. url to point to elasticSearch instance. If you use the default configuration like I did, you don’t need to modify this file

“3. Startup”

[root@VM_234_23_centos kibana]# ./bin/kibana
1
Copy the code

“4. Visithttp://localhost:5601/app/kibana#/home?\_g= ()”

With so much functionality shown in the interface, let’s integrate Spring+Logstash by integrating SLF4J+LogBack

“1. After modifying logstash. Conf, restart the logstash”

Input {# stdin {} TCP {# host:port is the destination of the appender. Enable port 9250 to receive messages from LogBack host => "127.0.0.1" port => 9250 mode => "server" tags => ["tags"] COdec => jSON_lines}} output  { elasticsearch { hosts => ["localhost:9200"] } stdout { codec => rubydebug } } 123456789101112Copy the code

“2. Reference dependencies in Java applications”

<dependency> <groupId>net.logstash.logback</groupId> <artifactId>logstash-logback-encoder</artifactId> The < version > 5.2 < / version > < / dependency > 12345Copy the code

“3. Configure log Output in logback. XML”

<! - log export to Logstash - > < appender name = "stash" class = "net. Logstash. Logback. Appender. LogstashTcpSocketAppender" > <destination>localhost:9250</destination> <! - the encoder must be configured with a variety of optional - > < encoder charset = "utf-8" class = "net. Logstash. Logback. Encoder. LogstashEncoder" > <! -- "appName ":"ye_test" <customFields>{" appName ":"ye_test"}</customFields> </encoder> </appender> <root level="INFO"> <appender-ref ref="stash"/> </root> 123456789101112131415Copy the code

“Since I did not specify the corresponding index in the first step, when the service started, the log collector Logstash automatically created the index of logstuck-timestamp.”

“4. Add index to Kibana”

“5. Check index information in Discover on the left”

“6. Adding a Visualization Chart Visualize”

“There are still more functions to explore, first of all, the environment will be built to continue to learn with power ~” Tread pit record

Start the error

uncaught exception in thread [main] org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: can not run elasticsearch as root

Cause: User Root cannot be used to log in. Solution: Switch users

[root@VM_234_23_centos ~]# groupadd es [root@VM_234_23_centos ~]# useradd es -g es -p es [root@VM_234_23_centos ~]# [root@VM_234_23_centos ~]# sudo su - es 12345 [root@VM_234_23_centos ~]# sudo su - es 12345Copy the code

The Exception in the thread “main” Java. Nio. File. AccessDeniedException:

Cause of error: Start ES as a non-root user and the user does not have sufficient file permissions. Solution: chown -r User name: user name File (or directory) name Example: chown -r ABC: ABC SearchEngine starts ES

[2018-07-13T10:19:44.775][INFO][O.E.P.luginsService][f2s1SD8] Loaded Module [aggs-matrix-stats] [2018-07-13T10:19:44.779][INFO][O.E.p.luginsService][f2s1SD8] Loaded Module [analysis-common] [2018-07-13T10:19:444,780][INFO][O.e.p.luginsService][f2s1SD8] Loaded Module [ingest-common] [2018-07-13T10:19:44.780][INFO][O.E.P.luginsService][f2s1SD8] Loaded Module [lang-expression] [2018-07-13T10:19:44.780][INFO][O.E.P.luginsService][f2s1SD8] Loaded Module [mustache] [2018-07-13T10:19:444,780][INFO][O.E.p.luginsService][f2s1SD8] Loaded Module [lang-painless] [2018-07-13T10:19:444,780][INFO][O.E.p.luginsService][f2s1SD8] Loaded Module [2018-07-13T10:19:444,780][INFO][o.e.p.pluginsService][f2s1SD8] Loaded Module [parent-join] [2018-07-13T10:19:44.780][INFO][o.e.p.luginsService][f2s1SD8] Loaded Module [percolator] [2018-07-13T10:19:44.780][INFO][o.e.p.luginsService][f2s1SD8] Loaded Module [rank-eval] [2018-07-13T10:19:444,781][INFO][O.E.p.luginsService][f2s1SD8] Loaded Module [reindex] [2018-07-13T10:19:444,781][INFO [2018-07-13T10:19:444,781][o.e.p.luginsService][f2s1SD8] loaded Module [repository-url] [2018-07-13T10:19:444,781][INFO][O.e.p.luginsService [2018-07-13T10:19:444,781][INFO][O.e.p.luginsService][f2s1SD8] loaded Module [transport-netty4] [2018-07-13T10:19:444,781 [tribe] [2018-07-13T10:19:444,781][INFO][O.e.p.luginsService][f2s1SD8] Loaded Module [X-pack-core] [2018-07-13T10:19:44.781][INFO][O.e.p.pluginsService][f2s1SD8] Loaded Module [X-pack -deprecation] [2018-07-13T10:19:44.781][INFO][o.e.p.luginsService][f2s1SD8] Loaded Module [x-pack-graph] [2018-07-13T10:19:44.781][INFO][O.E.P.luginsService][f2s1SD8] Loaded Module [X-pack-logstash] [2018-07-13T10:19:44.782][INFO][o.e.p.luginsService][f2s1SD8] Loaded Module [x-pack-ml] [2018-07-13T10:19:44.782][INFO][O.E.P.luginsService][f2s1SD8] Loaded Module [x-pack-monitoring] [2018-07-13T10:19:44.782][INFO][o.e.p.luginsService][f2s1SD8] Loaded Module [x-pack-rollup] [2018-07-13T10:19:44.782][INFO][O.E.P.luginsService][f2s1SD8] Loaded Module [x-pack-Security] [2018-07-13T10:19:44.782][INFO][o.e.p.luginsService][f2s1SD8] Loaded Module [X-pack-sql] [2018-07-13T10:19:44.782][INFO][O.E.p.luginsService][f2s1SD8] Loaded Module [x-pack-upgrade] [2018-07-13T10:19:44.782][INFO][O.e.p.luginsService][f2s1SD8] Loaded Module [x-pack-watcher] [2018-07-13T10:19:444,783][INFO][o.E.p.pluginsService][f2s1SD8] No plugins loaded on load 123456789101112131415161718192021222324252627Copy the code

Modify jvm.options in the config directory to make the heap size smaller

#Xms represents the initial size of total heap space
#Xmx represents the maximum size of total heap space
-Xms512m
-Xmx512m
1234
Copy the code

Insufficient virtual memory

max virtual memory areas vm.max_map_count [65530] is too low, increaseto at least [262144]

[2018-07-13T14:02:06.749][DEBUG][O.E.A. ctionModule] Using REST Wrapper from plugin Org. Elasticsearch. Xpack. Security. Security [the 2018-07-13 T14:02:07, 249] [INFO] [O.E.D.D iscoveryModule] [f2s1SD8] using Discovery Type [Zen][2018-07-13T14:02:09.173][INFO][O.E.n.ode][f2s1SD8] Initialized [2018-07-13T14:02:09.174][INFO ][o.e.n.Node ] [f2s1SD8] starting ... [2018-07-13T14:02:09.539][INFO][O.E.T. ransportService][f2s1SD8] publish_address {10.105.234.23:9300}, Bound_addresses {0.0.0.0:9300} [2018-07-13T14:02:09.575][INFO][O.E.B. bootstrapchecks][f2s1SD8] bound or publishing to  a non-loopback address, enforcing bootstrap checks ERROR: [1] bootstrap checks failed [1]: max virtual memory areas vm.max_map_count [65530] is too low, Increase at least [262144] [2018-07-13T14:02:09,621][INFO][o.e.n.ode][f2s1SD8] stopping... [2018-07-13T14:02:09,726][INFO][O.E.n.ode][f2s1SD8] Stopped [2018-07-13T14:02:09,726][INFO][O.E.n.ode][f2s1SD8] closing ... [2018-07-13T14:02:09.744][INFO][O.E.n.ode][f2s1SD8] 123456789101112Copy the code

Need to change the size of virtual memory (under root permission)

[root@VM_234_23_centos elasticsearch]# vim /etc/sysctl.conf # insert the following code and save to exit vm. Max_map_count =655360 [root@VM_234_23_centos Elasticsearch]# sysctl -p restart elastricSearch 12345Copy the code

Some interview questions for 2020 are summarized. The interview questions are divided into 19 modules, which are: Java Basics, Containers, Multithreading, Reflection, Object copy, JavaWeb exceptions, Networking, Design Patterns, Spring/SpringMVC, SpringBoot/SpringCloud, Hibernate, MyBatis, RabbitMQ, Kafka, Zookee Per, MySQL, Redis, JVM.

Access to information above information: concern public number: programmers with stories, access to learning materials.

Remember to click follow + comment oh ~