In the previous article “Logstash: Enabling Monitoring and Centralized Management”, I described in detail how to use Elasticsearch to self-monitor Logstash. In today’s exercise, we rehearse previous articles:

  • Elastic: Monitors Elasticsearch and Kibana
  • Elastic: Monitors Beats and APM Servers

I’ll use Metricbeat to monitor the Logstash. I’ll use an example from the “Logstash: Enabling Monitoring and centralized Management” article to illustrate this. Elasticsearch: Monitoring Elasticsearch and Kibana before we start monitoring Elasticsearch, we must start monitoring Elasticsearch as we did in the previous article “Elastic: Monitoring Elasticsearch and Kibana”.

How do I monitor Logstash?

There are two ways to monitor Logstash:

Use the monitoring APIs provided by Logstash for monitoring

By monitoring the use of APIs, they can provide Logstash metrics. We don’t need to do any configuration and it works by default.

Node Info API

This API gives you details about the Logstash node:

curl -XGET 'localhost:9600/_node? pretty'Copy the code

Plugins info API

All installed plugins are available via this API:

curl -XGET 'localhost:9600/_node/plugins? pretty'Copy the code

Node Stats API

The runtime state of the Logstash node can be obtained using this API:

curl -XGET 'localhost:9600/_node/stats? pretty'Copy the code

Hot Threads API

With this API we can get the current hot Threads of the Logstash file:

curl -XGET 'localhost:9600/_node/hot_threads? pretty'Copy the code

Use the Elastic monitoring function

Using this method, we can send data directly to the monitored Elasticsearch cluster just as we did in the previous article monitoring Beats, Elasticsearch, and Kibana. We can use Kibana to visualize the data. In this way, we must use Metricbeat to collect the required indicators.

In today’s article, I’ll focus on how to use Elastic monitoring to monitor Logstash.

Start the Logstash

Let’s start with the Logstash example described in the “Logstash: Enabling Monitoring and centralized Management” article:

apache.conf

input { file { path => "/Users/liuxg/data/multi-pipeline/apache.log" start_position => "beginning" sincedb_path => "/dev/null" # ignore_older => 100000 type => "apache" } } filter { grok { match => { "message" => '%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response:int} (? :-|%{NUMBER:bytes:int}) %{QS:referrer} %{QS:agent}' } } } output { stdout { codec => rubydebug } elasticsearch { hosts => ["localhost:9200"] user => "elastic" password => "password" index => "apache_log" template => "/Users/liuxg/data/multi-pipeline/apache_template.json" template_name => "apache_elastic_example" template_overwrite => true } }Copy the code

We use the following command to start the Logstash:

sudo ./bin/logstash -f apache.conf 
Copy the code

Start the logstash-xpack module

We use the following command to start the logstash-xpack module:

./metricbeat modules enable logstash-xpack
Copy the code
$ ./metricbeat modules enable logstash-xpack
Enabled logstash-xpack
Copy the code

We can check whether the module has been started by using the following command:

 ./metricbeat modules list
Copy the code

You can see from the above that the Logstash – Xpack module has been successfully started.

We next configure the modules.d/logstash. Yml file.

modules.d/logstash-xpack.yml

# Module: logstash
# Docs: https://www.elastic.co/guide/en/beats/metricbeat/7.x/metricbeat-module-logstash.html

- module: logstash
  xpack.enabled: true
  period: 10s
  hosts: ["localhost:9600"]
  #username: "user"
  #password: "secret"
Copy the code

We need to configure it according to your Logstash address.

Let’s start Metricbeat:

./metricbeat -e
Copy the code

Monitor the Logstash

We can exit the Settings mode. Click Exit Setup Mode above:

We can see that there is a Pipeline. Click on the Pipeline.

We can see some statistics displayed above. Since we had processed all the documents and started Metricbeat again, all the data we saw was zero. We append a file to apache-daily-access.log using the following command:

$cat << EOF >> apache.log > 84.149.9.216 -- [17/May/2015:10:05:57 +0000] "GET / presentations/logstash - monitorama - 2013 / CSS/fonts/Roboto - Bold. The vera.ttf HTTP / 1.1 ", 200, 38720 "Http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla / 5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36" > EOFCopy the code

Let’s look again:

Let’s click Logstash again: