Personal environment
(1) Centos 7.0
(2) ELK – 5.2.0 – Linux – x64
(3) filebeat – 6.2.3 – Linux – x86
(4) the JDK 8 u141 – Linux – x64
(5) kafka_2. 11-2.2.1
Note: ELK must use the same version, there will be various pits between different versions. Other software can be used in different versions. I’ve shared ELK’s version here, and you can use your own for the rest. Share here to give you a time travel: https://download.csdn.net/download/D_Janrry/12409286
The topology
Architecture interpretation: (The entire architecture is divided into 5 layers from left to right)
The first layer, data acquisition layer
On the left is the business server cluster, where FileBeat is installed to collect logs and send the collected logs to the two Logstash services.
The second layer, data processing layer, data cache layer
The Logstash service formats the received logs and saves them to the local Kafka + ZooKeeper cluster.
Third layer, data forwarding layer
This single Logstash node pulls data from the Kafka + ZooKeeper cluster in real time and forwards it to the ES cluster.
The fourth layer, persistent data storage
ES cluster will receive the data, write disk, build index library.
The fifth layer, data retrieval, data display
ES Master + Kibana mainly coordinates ES cluster, handles data retrieval request and data display.
Environment initialization
1. Plan nodes
Due to the limited number of virtual machines, I had to deploy all the software on one machine. Sure, two, three, four, five will do.
2. Configure the host name
hostnamectl set-hostname bigdata1su -l....Copy the code3. Handle the firewall
systemctl stop firewalld
systemctl disable firewalld
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
setenforce 0Copy the code4. Synchronize the clock source
yum install ntpdate ntp -y
ntpdate cn.pool.ntp.org
hwclock --systohcCopy the code5. Install common software
yum install net-tools bash-completion wget vim -yCopy the code6. The JDK installation
Installation is too simple, we will skip over, I believe everyone will
All the preceding steps need to be completed.
ELK installation
1. ES installation
Elasticsearch5. X is a standalone Web App, so it doesn’t need to integrate with Elasticsearch. So the older versions might be a little hard to install.
I have posted an article about installing the current version of ES-7.6.2 and installing plug-ins, which I have filled out. Time travel: https://blog.csdn.net/D_Janrry/article/details/105461236
However, I do mind if you don’t install the latest one. The latest one is really cool, but there are too many pits. Finally, when it is compatible with Logstash, the display cannot get the image of ES, so it is not compatible.
Version 6.6.2 is also not compatible.
Version 6.6.1 is compatible and ELK can be implemented. However, when it comes to the final log transfer, its Logstash does not allow multiple. Conf files to be opened.
See here, you can imagine my mood of collapse again and again? It’s annoying, it’s annoying, it’s a lot of thieves. So, I gave up. If you do, use version 5.2.0. It can be installed as per 7.6.2. Good luck.
2. The Logstash installation
Logstash event processing has three stages: Inputs -→filters→outputs. Is a tool for receiving, processing and forwarding logs.
Support for system logs, Webserver logs, error logs, application logs, and all types of logs that can be thrown.
Unzip the installation is good, in fact, you can do not modify. Here I have modified it. And then we look down
[root@bigdata1 logstash]# pwd
/data/program/software/logstash
[root@bigdata1 logstash]# mkdir conf.d // Directory for the configuration file
[root@bigdata1 config]# pwd
/data/program/software/logstash/config
[root@bigdata1 config]# vim logstash.ymlCopy the code
3. Kibana installation
Install, omit, because install everybody can right, too simple to repeat, we won’t go into detail
After installation, modify the configuration file
[root@bigdata1 config]# pwd
/data/program/software/kibana/config
[root@bigdata1 config]# vim kibana.yml
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.url: "http://192.168.239.7:9200"
kibana.index: ".kibana"Copy the codeUse the browser open http://192.168.239.7:5601
Verify that the installation succeeded.
Filebeat installation
Filebeat.prospectors: // File detector -type: log// Probe type, log file enabled:false// This must be changedfalsePaths: // path - /var/log/*.log // Note that the presence of the wildcard * makes project logs with the same directory structure collectedCopy the codeAs shown in Figure 1:
The default is ES, which needs to be changed to logstash. // This means that when the log is collected, it is handed over to the logstash.
Note: The above services are all installed on Bigdata1, but the ES and Kafka + ZooKeeper clusters are installed on three nodes.
Kafka + zookeeper installation
1. The zookeeper configuration
We know that Kafak clusters rely on ZooKeeper. Zookeeper is already included in the Kafka package. In view of issues between versions, it is recommended to use Kafka’s built-in ZooKeeper.
Why is that? Because I stepped on the pit again, but this is for the computer, my computer installed ZooKeeper in addition, and finally received no messages from the producer from the consumer. So I used Kafka’s built-in ZooKeeper.
Its configuration is as follows:
[root@bigdata2 config]# pwd
/data/program/software/kafka/config
[root@bigdata2 config]# vim zookeeper.propertiesCopy the codeSame operation for three nodes.
Note: If you install ZooKeeper externally and have the problems I mentioned above before using Kafka embedded ZooKeeper, be sure to turn the previous ZooKeeper off.
2. Kafka configuration
[root@bigdata2 config]# pwd
/data/program/software/kafka/config
[root@bigdata2 config]# vim server.properties
broker.id=1
listeners=PLAINTEXT://bigdata1:9092
log.dirs=/var/kafka
zookeeper.connect=bigdata1:2181,bigdata2:2181,bigdata3:2181Copy the code
To pass the configuration files to the other two nodes using SCP technology, simply change the value of broker-id to 2 and 3. Since kafka applications on BigdatA2 and BigdatA3 are copied directly, the contents of the logs directory need to be emptied.
[root@bigdata2 kafka]# pwd
/data/program/software/kafka
[root@bigdata2 kafka]# ls
bin config libs LICENSE logs NOTICE site-docs
[root@bigdata2 kafka]# cd logs
[root@bigdata2 logs]# rm -rf ./*Copy the codeAt this point, the service deployment is complete. Next, the various services work together.
Log Collection and Analysis
If you are still confused by the previous flow chart, let me briefly repeat the workflow:
- The fileBeat configuration file specifies the directory in which you want to analyze the logs. When FileBeat starts working, it will collect the logs in the corresponding directory. Once it’s collected, it’s handed over to the Logstash, so that’s why we set the direction of it earlier. See Figure 1
- Configure a file under conf.d of the logstash to receive fileBeat log messages and send the logs to the Kafka + ZooKeeper cluster (that is, message queues), as shown in Figure 2. Once received by the Kafka + ZooKeeper cluster, configure a file to forward from the Kafka + ZooKeeper cluster to the ES cluster, as shown in Figure 3.
- The ES cluster receives it and then sends it to Kibana. Finally, it is withdrawn on the Web interface, as shown in Figure 4 and Figure 5.
1. The deployment of kafka logstash + + zookeeper
[root@bigdata1 conf.d]# vim filebeat_to_logstash.conf
input {
beat {
port => 5044
}
}
output{
kafka{
bootstrap_servers => "192.168.239.8:9092192168 239.9:9092"
topic_id => "ecplogs"}}Copy the codeAs shown in the figure:
[root@bigdata1 conf.d]# vim logstash_to_elasticsearch.conf
input {
kafka{
bootstrap_servers => "192.168.239.8:9092192168 239.9:9092"
topic_id => "ecplogs"
}
}
output {
elasticsearch {
hosts => ["192.168.239.8:9200"."192.168.239.9:9200"]
index => "ecp-log-%{+YYYY.MM.dd}"}}Copy the codeAs shown in the figure:
2. Service startup and data tracking for each link
To start fileBeat, you need to install the following software:
1. Start ZooKeeper +kafka
A little…
(2) Start logstash to receive logs
A little… It’s filebeat_to_logstash. Conf
(3) Log consumption test on Kafka terminal
[root@bigdata1 bin]# / data/program/software/kafka/bin/kafka - the console - consumer. Sh - the bootstrap - server 192.168.239.8:9092 - topic ecplogs --from-beginning// Ecologs is the value of topic_id in the logstash fileCopy the codeThe log is as follows:
Therefore, the final log content on Kibana should be the same as here.
(4) Enable logstash forwarding
A little… Logstash_to_elasticsearch. conf file
(5) Test results on ES
As shown in the figure:
(6) Present the data on Kibana
You can see that the logs collected here are consistent with those collected on Kafka, which proves that the logs were collected successfully. Good luck to you all!
Author: Janrry, Long Long
The original link: https://blog.csdn.net/D_Janrry/article/details/106061610