I found that Graylog was actually a very good idea when I was running for IGP project abroad. At that time, It was Robert who proposed to build this log management platform. I saw his demo privately. The real power of Graylog, it turns out, is that it gives the log management system google-like power to do a full-text search of logs, which he describes several times: “Anything can be… , Anything…” . However, concepts that are too far ahead of their time can always turn out to be unfortunate if they are not understood in your environment.

I read on a blog that “Graylog is the open source version of Splunk,” a well-known enterprise super log management system that costs $30,000. Once again to prove a truth: knowledge is wealth.

In fact, the requirements for background operation and maintenance are nothing more than “visualization/alarm/report”. Some related startups in China are growing rapidly, such as “Light Cloud” and “One”, which provide “probe” services for server-related businesses, checking performance and background logs. I know of zabbix as well, but it’s a bit more difficult to configure than Graylog. So I don’t know yet.

With the rapid development of the Internet wave, business types are iterating layer by layer, and the back-end operation and maintenance requirements for servers are also increasing. If back-end operation and maintenance is not taken into account in the early stage, problems are gradually exposed as the business develops in the future, the efficiency of operation and maintenance can sometimes determine the speed of product development.

Graylog diagram

  • Deployment diagram

Web page screenshot:

Minimum installation:

<br>

Production tool installation:

  • Architecture diagram

    A screenshot from the official website:

Installation manual

Software listing

  1. Directing a 3.0
  2. Elasticsearch 1.5
  3. Graylog – server 1.3
  4. Graylog – web 1.3

Installation environment

CentOS6.6, server IP: Linux host address of ITStudio, Oracle JDK 7u71 installed

Installation steps

1. mongodb

Mongodb official website

[root@logserver yum.repos.d]# vim The/etc/yum. Repos. D/mongo - org - 3.0. Repo
---
[mongo - org - 3.0]
name=MongoDB Repository
baseurl=http://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/3.0/x86_64/
gpgcheck=0
enabled=1
---

[root@logserver yum.repos.d]# yum install -y mongodb-org

[root@logserver yum.repos.d]# vi /etc/yum.conf
The last line adds:
---
exclude=mongodb-org,mongodb-org-server,mongodb-org-shell,mongodb-org-mongos,mongodb-org-tools
---

[root@logserver yum.repos.d]# service mongod start
[root@logserver yum.repos.d]# chkconfig mongod on

[root@logserver yum.repos.d]# vi /etc/security/limits.conf
The last line adds:
---
*                soft    nproc           65536
*                hard    nproc           65536
mongod           soft    nproc           65536

*                soft    nofile          131072
*                hard    nofile          131072
---

[root@logserver ~] # vi /etc/init.d/mongod
ulimit -f unlimited Pre-line insertion:
---
  if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
    echo never > /sys/kernel/mm/transparent_hugepage/enabled fi if test -f /sys/kernel/mm/transparent_hugepage/defrag; then echo never > /sys/kernel/mm/transparent_hugepage/defrag fi --- [root@logserver ~]# /etc/init.d/mongod restartCopy the code

2. elasticsearch

The latest version of ElasticSearch is 2.3. Graylog does not support 2.x. 1.7.3 is recommended. By chance, version 1.5 was used but the configuration was successful. For the record, version 1.5 is still used because of the compatibility of various components.

[root@logserver ~] # rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
[root@logserver ~] # vi /etc/yum.repos.d/elasticsearch.repo
---
[elasticsearch-1.5]
name=Elasticsearch repository for 1.5.x packages
Baseurl = http://packages.elastic.co/elasticsearch/1.5/centos
gpgcheck=1
gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1
---

[root@logserver ~] # yum install elasticsearch
[root@logserver ~] # chkconfig --add elasticsearch

[root@logserver ~] # vi /etc/elasticsearch/elasticsearch.yml
  32 cluster.name: graylog

[root@logserver ~] # /etc/init.d/elasticsearch start
[root@logserver ~] # curl localhost:9200Copy the code

3. graylog

// Install the latest version of graylog by entering the following command:
$ sudo rpm -Uvh https://packages.graylog2.org/repo/packages/graylog-1.3- repository-el6_latest.rpm
$ sudo yum install graylog-server graylog-web
[root@logserver ~] # /etc/init.d/graylog-server start
Starting graylog-server:                                   [sure]
Startup failed!
[root@logserver ~] # cat /var/log/graylog-server/server.log
2015- 05- 22T15:53:14. 962 + 08:00 INFO  [CmdLineTool] Loaded plugins: []
2015- 05- 22T15:53:15. 032 + 08:00 ERROR [Server] No password secret set. Please define password_secret in your graylog2.conf.
2015- 05- 22T15:53:15. 033 + 08:00 ERROR [CmdLineTool] Validating configuration file failed - exiting.

[root@logserver ~] # yum install pwgen
[root@logserver ~] # pwgen -N 1 -s 96
zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz
[root@logserver ~] # echo -n 123456  sha256sum 
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx  -

[root@logserver ~] # vi /etc/graylog/server/server.conf
11 password_secret = zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz
.
22 root_password_sha2 = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
.
152 elasticsearch_cluster_name = graylog

[root@logserver ~] # /etc/init.d/graylog-server restart
Startup successful!

[root@logserver ~] # /etc/init.d/graylog-web start
Starting graylog-web:                                      [sure]
Startup failed!
[root@logserver ~] # cat /var/log/graylog-web/application.log
2015- 05- 22T15:53:22. 960 + 08:00 - [ERROR] - from lib.Global in main 
Please configure application.secret in your conf/graylog-web-interface.conf

2015- 05- 22T16: for 343 + 08:00 - [ERROR] - from lib.Global in main 
Please configure application.secret in your conf/graylog-web-interface.conf

[root@logserver ~] # pwgen -N 1 -s 96
yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
[root@logserver ~] # vi /etc/graylog/web/web.conf
---
2 Graylog2 - server uris, = "http://127.0.0.1:12900/"
12 application.secret="yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy"
---

Note: / etc/graylog/web/web conf graylog2 - server. In the uris, value must be with the/etc/graylog/server/server rest_listen_uri consistent in the conf
---
36 rest_listen_uri = http://127.0.0.1:12900/
---
[root@logserver ~] # /etc/init.d/graylog-web startCopy the code

Enter the URL http://localhost:9000/ to go to the graylog login page. The administrator account and password are admin and 123456

4. Add a log collector

Log in to http://IP_adress:9000/ as admin

Note: IP_adress is the server’s real IP address. The following the same

4.1

Enter System > Inputs > Inputs in Cluster > Raw/Plaintext TCP Launch new input named “TCP 5555

On any Linux machine with nc installed:

echo `date`  nc IP_adress 5555Copy the code

After logging in to http://IP_adress:9000/, click the green search button in the third line and see a new message:

Timestamp Source Message 
2016- 04-10 0849:: 15.280 5970.156.6.Sunday, April 10, 2016 1648:28: CSTCopy the code

Screenshots:

<br>

The installation is successful!!

4.2

Enter System > Inputs > Inputs in Cluster > GELF HTTP Launch new input

Name it “HTTP 12201”

On any Linux machine with curl installed:

curl -XPOST http://IP_adress/gelf  -p0 -d '{"short_message":"Hello there"."host":"example.org"."facility":"test"."_foo":"bar"} 'Copy the code

After logging in to your browser at http://IP_adress:9000/, click the third green search button and see a new message:

Timestamp Source Message 
2016- 04-10 20: 50: 42.936 5970.156.6.  Hello Graylog.From Bei.Copy the code

Screenshots:

<br>

GELF HTTP Input set successfully!! !

5. Time zone and highlighting Settings

Time zone of the admin account:

[root@logserver ~] # vi /etc/graylog/server/server.conf
---
30 root_timezone = Asia/Shanghai
---
[root@logserver ~] # /etc/init.d/graylog-server restartCopy the code

Default time zone for other accounts:

[root@logserver ~] # vi /etc/graylog/web/web.conf
---
18 timezone="Asia/Shanghai"
---
[root@logserver ~] # /etc/init.d/graylog-web restartCopy the code

Allows query results to be highlighted:

[root@logserver ~] # vi /etc/graylog/server/server.conf
---
147 allow_highlighting = true
---
[root@logserver ~] # /etc/init.d/graylog-server restartCopy the code

The Graylog server deployment is now complete. Follow the article about configuring corresponding services in the future.

reference

  1. C’s official website
  2. Graylog – A rising star in log aggregation tools