Management Platform Process

Environment to prepare

  • Elasticsearch – 6.0.0. Tar. Gz
  • Filebeat – 7.0.1 – Linux – x86_64. Tar. Gz
  • Kibana – 6.0.0 – Linux – x86_64. Tar. Gz
  • Logstash – 6.0.0. Tar. Gz
  • Kafka_2. 11-2.1.1. TGZ

Except for Kafka, the other four are available for download on The Elastic website, where you can select the software and version to download. Kafka 6.0.0 is available for download on Apache

This document is installed and deployed on a CentOS Linux release 7.2.1511 (Core) 64-bit OS. To check the version and bits, run the following command

  • Cat /etc/centos-release: Check the version number
  • Getconf LONG_BIT: View the number of bits

Installation and Deployment

Since it is java-based, make sure you have Jdk installed on your system before deploying it. Jdk 1.8 is installed on this system, and all files are placed in /usr/elk/

Kibana deployment

  1. Unzip the files
[root@localhost elk]# tar -zxvf kibana-6.0.0-linux-x86_64.tar.gz
Copy the code
  1. Modify the configuration by modifying the /conf/kibana.yml file
Elasticsearch: "http://localhost:9200" server.host: 0.0.0.0Copy the code

Elasticsearch deployment

  1. Unzip the files
[root@localhost elk]# tar -zxvf elasticsearch-6.0.0.tar.gz
Copy the code
  1. You cannot use root to start the project, so create a new user to start it
[root@localhost elk]# groupAdd elkgroup # Add user to this group useradd user name -g group name -p password [root@localhost elk]# useradd elkuser [root@localhost elk]# chown -r elsearch:elsearch elasticSearch -6.0.0 # Elasticsearch [root@localhost elk]# su elkuser [elkuser@localhost elasticSearch -6.0.0]$./bin/ elasticSearchCopy the code

Logstash deployment

  1. Unzip the files
[root@localhost elk]# tar -zxvf logstash-6.0.0.tar.gz
Copy the code
  1. Create a configuration file logstash. Conf in the conf directory
input{ file{ type=>"log" path=>"/usr/logs/*.log" start_position=>"beginning" } } output{ stdout{ codec=>rubydebug{} } Elasticsearch {hosts = > "127.0.0.1 index" = > "log - % {+ YYYY. MM. Dd}"}}Copy the code
  1. Start according to the configuration file
[root@localhost elk]# ./logstash -f .. /config/logstash.confCopy the code

Kafka deployment

  1. Install the glibc
[root@localhost elk]# yum -y install glibc.i686
Copy the code
  1. Configuration zookeeper
[root @ localhost elk] # vi config/zookeeper. # properties configuration contents dataDir = / data/designed/kafka_2. 11-0.10.0.0 / data DataLogDir =/data/programs/kafka_2.11-0.10.0.0/logs clientPort=2181 maxClientCnxns=100 tickTime=2000 initLimit=10 # Save the configuration and exit. Then according to the configuration starts the zookeeper [root @ localhost kafka_2. 11-2.1.1] #. / bin/zookeeper - server - start. Sh config/zookeeper. PropertiesCopy the code
  1. Configure the Kafka file
[root@localhost kafka_2.11-2.1.1]# vi config/server.properties # configure content broker.id=0 port=9092 host.name=127.0.0.1 num.network.threads=3 num.io.threads=8 socket.send.buffer.bytes=102400 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 log.dirs=/data/logs/kafka num.partitions=2 num.recovery.threads.per.data.dir=1 Log. The retention. Check. Interval. Ms = 300000 zookeeper. Connect = localhost: 2181 zookeeper. Connection. A timeout. Ms = 6000 # configuration is complete keep out, Kafka [root@localhost kafka_2.11-2.1.1]#./bin/kafka-server-start.sh config/server.propertiesCopy the code

During startup, you may be prompted to run out of memory,

Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000c0000000, 1073741824, 0) failed; error='Cannot allocate memory' (errno=12) # # There is insufficient memory for the Java Runtime Environment to continue.  # Native memory allocation (mmap) failed to map 1073741824 bytes for committing reserved memory. # An error report file  with more information is saved as:Copy the code

The default kafka configuration is 1 gb, so change the startup configuration

Change export KAFKA_HEAP_OPTS=" -xmx1g-xMS1g "to export KAFKA_HEAP_OPTS=" -xmx256m -xMS128m" in /home/kafka-server-start. shCopy the code

Then restart it

FileBeat deployment

  1. Example Modify the filebeat.yml configuration
[root@localhost filebeat-7.0.1-linux-x86_64]# vi filebeat.yml So configuration to qualify for yml specification # -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- kafka output -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- - the output. Kafka: enabled: True hosts: ["127.0.0.1:9092"] topic: test # And modify the configuration enabled: true # - /home/elk/log/access.log ["nginx-accesslog"] # start service [root@localhost filebeat-7.0.1-linux-x86_64]#Copy the code

Questions and points to note

Connection to node-1 could not be established. Broker may not be available. This address is the server address. The logstash address for kafka is 127.0.0.1.

  • Logstash configuration
  • Kafka queue names must correspond

As for the detailed configuration of each software, it will be introduced in the subsequent article. For details, you can also refer to the official document as follows: Kibana Logstash Elasticsearch Filebeat