This is a follow-up to my previous post “Elastic: Creating Your First Elastic Watcher.” In that article, we explained how to use the Watcher API to create an alert. We used only a simple demonstration of logging. Maybe you don’t like it, because we rarely sit in our seats and stare at our screens and log. The other thing is that we might have a situation, the situation that we want to have is that there’s an error message and we’re not nervous about it, but if by a certain amount of time, our error or exception is at a certain number, then we want to send the alert that we want in that situation. What should we do about it?

In today’s exercise, we’ll show you how to use email to send notifications. Are you excited about that? After all, this is closer to our actual application scenario.

Elastic Alerts and Watchers got hands-on

Configure Elasticsearch

Elasticsearch config/ elasticSearch.yml to support mail sending, you have to configure Elasticsearch in config/ elasticSearch.yml. About other actions, please refer to our official website www.elastic.co/guide/en/el Elastic… .

For email configuration, we can refer to www.elastic.co/guide/en/el… . We need to do different configurations for different email addresses. In my case, I was using hotmail, so at the end of my config/ elasticSearch.yml file I added the following configuration:

xpack.notification.email.account:
    exchange_account:
        profile: outlook
        email_defaults:
            from: 'Alert <[email protected]>'
        smtp:
            auth: true
            starttls.enable: true
            host: smtp-mail.outlook.com
            port: 587
            user: [email protected]
Copy the code

On top of that, we configured my email account information. Careful developers may notice that my password is not included in the information above. Since password information is sensitive information, you may not want to keep it in your configuration file. To do this, type the following command in the Elasticsearch installation directory:

bin/elasticsearch-keystore add xpack.notification.email.account.exchange_account.smtp.secure_password
Copy the code

We just paste in the password of our own account number. Please note that the commands are different for different email accounts, such as Gmail, etc.

After we have modified the above configuration, we need to restart our Elasticsearch.

Modify the Watcher

Next, we need to modify our Watcher interface. If you have completed my previous tutorial, we can do a search like this in our Kibana:

The GET filebeat 7.6.0 / _search {" size ": 0," query ": {" bool" : {" must ": [{" match" : {" process. The name ": "JUSTME" } }, { "match": { "message": "error" } } ], "filter": [ { "range": { "@timestamp": { "from": "now-1m", "to": "now" } } } ] } }, "aggs": { "failed_messages": { "terms": { "field": "process.name" } } } }Copy the code

On top of that, we did a aggregation. Process. name is JUSTME and message contains error in the past 1 minute. If the number of statistics exceeds one of our thresholds, say five, we expect Watcher to send me an alert. If it’s below 5, then we consider it normal. The output of the above search results is in the following format:

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 4,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  },
  "aggregations" : {
    "failed_messages" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 0,
      "buckets" : [
        {
          "key" : "JUSTME",
          "doc_count" : 4
        }
      ]
    }
  }
}
Copy the code

The format of the output above is very important. This provides basic information for writing search and condition below.

Based on the above search and output, we tried to modify our Watcher.

PUT _watcher/watch/my-justme-watcher
{
  "trigger": {},
  "input": {},
  "condition": {},
  "actions": {}
}
Copy the code

A Watcher format is shown above. Let’s describe them separately:

trigger

This definition is practiced multiple times watcher runs once. For example, we can define it as follows:

  "trigger": {
    "schedule" : {
      "interval": "1h"
    }
  },
Copy the code

As it says, we want Watcher to run every hour. In our case, we define it as:

"trigger": { "schedule": { "cron": "0 0/1 * * * ?" }},Copy the code

It says we want to run it every minute.

input

This defines the index that an alert wants to search for, and how long the index is searched for. According to the above search criteria, we write as follows:

  "input": {
    "search": {
      "request": {
        "indices": [
          "filebeat-*"
        ],
        "body": {
          "query": {
            "bool": {
              "must": [
                {
                  "match": {
                    "process.name": "JUSTME"
                  }
                },
                {
                  "match": {
                    "message": "error"
                  }
                }
              ],
              "filter": [
                {
                  "range": {
                    "@timestamp": {
                      "from": "{{ctx.trigger.scheduled_time}}||-1m",
                      "to": "{{ctx.trigger.triggered_time}}"
                    }
                  }
                }
              ]
            }
          },
          "aggs": {
            "failed_messages": {
              "terms": {
                "field": "process.name"
              }
            }
          }
        }
      }
    }
  }
Copy the code

Our time is to find all JUSTME statistics that meet the criteria in the past minute.

condition

We define it as follows:

 "condition": {
    "compare": {
      "ctx.payload.aggregations.failed_messages.buckets.0.doc_count": {
        "gt": 5
      }
    }
  }
Copy the code

Please note that the above CTX. Content. Aggregations. Failed_messages. Buckets. 0. The doc_count way. It is consistent with the format of our query results. Above, we compare the results of the query and pull out the data for the first bucket. If doc_count is greater than 5, the event will be emitted.

actions

This defines the actions that need to be taken when the condition above is satisfied. These actions can be found at the address. Watcher currently supports the following actions: Email, Webhook, Index, Logging, Slack, and PagerDuty.

In today's exercise, we use email:

  "actions": {
    "send_email": {
      "throttle_period": "2m",
      "email": {
        "to": "[email protected]",
        "subject": "Watcher Notification",
        "body": "{{ctx.payload.hits.total}} errors found"
      }
    }
  }
Copy the code

Above, we send triggered notifications to one of our mailboxes under the heading Watcher Notification. It shows “XX Errors Found”. Also note the use of throttle_period. Because our Watcher runs every minute. In some cases, Watcher will send a notification each time the conditions are met. This creates spam. We don’t want Watcher to send every time it runs. Set it to 2m, or 2 minutes. Send one notification within two minutes.

After our modification, the complete Watcher configuration is as follows:

PUT _watcher/watch/my-justme-watcher
{
  "trigger": {
    "schedule": {
      "cron": "0 0/1 * * * ?"
    }
  },
  "input": {
    "search": {
      "request": {
        "indices": [
          "filebeat-*"
        ],
        "body": {
          "query": {
            "bool": {
              "must": [
                {
                  "match": {
                    "process.name": "JUSTME"
                  }
                },
                {
                  "match": {
                    "message": "error"
                  }
                }
              ],
              "filter": [
                {
                  "range": {
                    "@timestamp": {
                      "from": "{{ctx.trigger.scheduled_time}}||-1m",
                      "to": "{{ctx.trigger.triggered_time}}"
                    }
                  }
                }
              ]
            }
          },
          "aggs": {
            "failed_messages": {
              "terms": {
                "field": "process.name"
              }
            }
          }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.aggregations.failed_messages.buckets.0.doc_count": {
        "gt": 5
      }
    }
  },
  "actions": {
    "send_email": {
      "throttle_period": "2m",
      "email": {
        "to": "[email protected]",
        "subject": "Watcher Notification",
        "body": "{{ctx.payload.hits.total}} errors found"
      }
    }
  }
}
Copy the code

Run the command above.

Test the Watcher

In many cases, if we set the time longer, we may not be able to wait that long to do a test. Elastic provides the _execute interface for our testing:

PUT _watcher/watch/my-justme-watcher/_execute
Copy the code

Through the interface above, we don’t have to wait as long as we set in schedule to run our Watcher. The command above will run Watcher directly for us.

At the same time, enter the following commands in our Ubuntu terminal:

sudo logger `who`  -t JUSTME error
Copy the code

If we receive 5 or more error messages within a minute, we will receive a notification message in our mail:

We open the contents of the email:

We can remove a Wacher as follows:

DELETE _watcher/watch/<watch_id>
Copy the code

In our case:

DELETE _watcher/watch/my-justme-watcher
Copy the code

 

The next step

If you want to know how to create a dynamic threshold alert, read my article “Elastic: How to Create a Dynamic threshold alert.”