Nginx log monitoring and alarm using Logstash

1. Implementation idea

First, the Nginx logs are saved to a file in JSON format, and then the log data is pushed to the Logstash node using the Rsyslog service. Finally, the Logstash service is configured to format the logs in JSON format, and the occurrence frequency of the corresponding abnormal state is calculated according to the status code and the threshold alarm is triggered. This section uses 504 error statistics and alarms as an example.

2. Configure the client

2.1 Nginx Log Configuration

Edit /etc/nginx/nginx.conf to add the following

log_format json '{"@timestamp":"$time_iso8601",' '"host":"$host",' '"scheme":"$scheme",' '"server_addr":"$server_addr",'  '"client_ip":"$remote_addr",' '"server_protocol":"$server_protocol",' '"method":"$request_method",' '"query_string":"$query_string",' '"body_bytes_sent":$body_bytes_sent,' '"bytes_sent":$bytes_sent,' '"request_length":$request_length,' '"request_time":$request_time,' '"upstream_time":"$upstream_response_time",' '"upstream_host":"$upstream_addr",' '"upstream_status":"$upstream_status",' '"server_name":"$server_name",' '"url":"$uri",' '"request_url":"$request_uri",' '"http_x_forwarded_for":"$http_x_forwarded_for",' '"referer":"$http_referer",' '"agent":"$http_user_agent",' '"status":$status}'; access_log /home/nginx/log/access.log json;Copy the code

2.2 rsyslog configuration

Use rsyslog push logs to Logstash, edit/etc/rsyslog. D/nginx. Conf, enter the following, remember to restart after the completion of rsyslog service.

$ModLoad imfile $Modload mmjsonparse action(type="mmjsonparse") template(name="nginx-json" type="list") { property(name="$! all-json") } $InputFileName /home/nginx/log/access.log $InputFileTag nginx-access $InputFileStateFile nginx-accessfile $InputFileSeverity info $InputFileFacility local7 $InputRunFileMonitor $InputFilePersistStateInterval 1 if $syslogtag Startswith 'nginx then @ @ 10.160.209.10:514; nginx-jsonCopy the code

3. Configure the server

3.1 Logstash configuration

Create a new Logstash configuration called metric_nginx.conf and look like this

Input {syslog {host => "10.160.209.10" port => "514" codec => "json"}} filter {json {source => "MSG"} mutate { convert => ["request_time",float] } mutate { remove_field => ["msg"] remove_field => ["@version"] remove_field => ["port"] remove_field => ["facility"] remove_field => ["priority"] remove_field => ["severity"] remove_field => ["severity_label"] remove_field => ["facility_label"]} metrics {meter => "error.%{status}" # Setup counter add_tag with different status values => "metric" ignore_older_than => 10}} output {if "metric" in [tags] {if [error.504][rate_1m] > 0.0 {# Alarm stdout {codec => line {format => "alarm: %{[error.504][rate_1m]}" # Analog alarm}}}}}Copy the code

Then use the following command to start the Logstash service

 logstash -f metric_nginx.conf
Copy the code

About the above [rate_1m] counter please refer to the definition of: www.elastic.co/guide/en/lo…

4. Automatic operation and maintenance model based on Logstash model

After introducing the above examples, I believe that you can see the role of Logstash is far more than that. After you are familiar with the Logstsh log processing process, you can realize a set of automatic operation and maintenance platform based on log drive. As shown in the following figure, logs of Mon, OSD, and other services can be connected to Logtash. After processing the logs with a Logstash stash, different output plugins can be used to generate Alarm alarms or trigger Ansible playbook operations. Or it can be stored in message-oriented middleware such as Redis and Kafka to channel data to other business systems. Especially when you are familiar with Ceph log exceptions, you can do some online fault automation.

The output plug-in is introduced: www.elastic.co/guide/en/lo…