This is a follow-up to my previous post “Elastic: Creating Your First Elastic Watcher.” In that article, we explained how to use the Watcher API to create an alert. We used only a simple demonstration of logging. Maybe you don’t like it, because we rarely sit in our seats and stare at our screens and log. The other thing is that we might have a situation, the situation that we want to have is that there’s an error message and we’re not nervous about it, but if by a certain amount of time, our error or exception is at a certain number, then we want to send the alert that we want in that situation. What should we do about it?
In today’s exercise, we’ll show you how to use email to send notifications. Are you excited about that? After all, this is closer to our actual application scenario.
Elastic Alerts and Watchers got hands-on
Configure Elasticsearch
Elasticsearch config/ elasticSearch.yml to support mail sending, you have to configure Elasticsearch in config/ elasticSearch.yml. About other actions, please refer to our official website www.elastic.co/guide/en/el Elastic… .
For email configuration, we can refer to www.elastic.co/guide/en/el… . We need to do different configurations for different email addresses. In my case, I was using hotmail, so at the end of my config/ elasticSearch.yml file I added the following configuration:
xpack.notification.email.account:
exchange_account:
profile: outlook
email_defaults:
from: 'Alert <[email protected]>'
smtp:
auth: true
starttls.enable: true
host: smtp-mail.outlook.com
port: 587
user: [email protected]
Copy the codeOn top of that, we configured my email account information. Careful developers may notice that my password is not included in the information above. Since password information is sensitive information, you may not want to keep it in your configuration file. To do this, type the following command in the Elasticsearch installation directory:
bin/elasticsearch-keystore add xpack.notification.email.account.exchange_account.smtp.secure_password
Copy the codeWe just paste in the password of our own account number. Please note that the commands are different for different email accounts, such as Gmail, etc.
After we have modified the above configuration, we need to restart our Elasticsearch.
Modify the Watcher
Next, we need to modify our Watcher interface. If you have completed my previous tutorial, we can do a search like this in our Kibana:
The GET filebeat 7.6.0 / _search {" size ": 0," query ": {" bool" : {" must ": [{" match" : {" process. The name ": "JUSTME" } }, { "match": { "message": "error" } } ], "filter": [ { "range": { "@timestamp": { "from": "now-1m", "to": "now" } } } ] } }, "aggs": { "failed_messages": { "terms": { "field": "process.name" } } } }Copy the codeOn top of that, we did a aggregation. Process. name is JUSTME and message contains error in the past 1 minute. If the number of statistics exceeds one of our thresholds, say five, we expect Watcher to send me an alert. If it’s below 5, then we consider it normal. The output of the above search results is in the following format:
{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 4,
"relation" : "eq"
},
"max_score" : null,
"hits" : [ ]
},
"aggregations" : {
"failed_messages" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"key" : "JUSTME",
"doc_count" : 4
}
]
}
}
}
Copy the codeThe format of the output above is very important. This provides basic information for writing search and condition below.
Based on the above search and output, we tried to modify our Watcher.
PUT _watcher/watch/my-justme-watcher
{
"trigger": {},
"input": {},
"condition": {},
"actions": {}
}
Copy the codeA Watcher format is shown above. Let’s describe them separately:
trigger
This definition is practiced multiple times watcher runs once. For example, we can define it as follows:
"trigger": {
"schedule" : {
"interval": "1h"
}
},
Copy the codeAs it says, we want Watcher to run every hour. In our case, we define it as:
"trigger": { "schedule": { "cron": "0 0/1 * * * ?" }},Copy the codeIt says we want to run it every minute.
input
This defines the index that an alert wants to search for, and how long the index is searched for. According to the above search criteria, we write as follows:
"input": {
"search": {
"request": {
"indices": [
"filebeat-*"
],
"body": {
"query": {
"bool": {
"must": [
{
"match": {
"process.name": "JUSTME"
}
},
{
"match": {
"message": "error"
}
}
],
"filter": [
{
"range": {
"@timestamp": {
"from": "{{ctx.trigger.scheduled_time}}||-1m",
"to": "{{ctx.trigger.triggered_time}}"
}
}
}
]
}
},
"aggs": {
"failed_messages": {
"terms": {
"field": "process.name"
}
}
}
}
}
}
}
Copy the codeOur time is to find all JUSTME statistics that meet the criteria in the past minute.
condition
We define it as follows:
"condition": {
"compare": {
"ctx.payload.aggregations.failed_messages.buckets.0.doc_count": {
"gt": 5
}
}
}
Copy the codePlease note that the above CTX. Content. Aggregations. Failed_messages. Buckets. 0. The doc_count way. It is consistent with the format of our query results. Above, we compare the results of the query and pull out the data for the first bucket. If doc_count is greater than 5, the event will be emitted.
actions
This defines the actions that need to be taken when the condition above is satisfied. These actions can be found at the address. Watcher currently supports the following actions: Email, Webhook, Index, Logging, Slack, and PagerDuty.
In today's exercise, we use email:
"actions": {
"send_email": {
"throttle_period": "2m",
"email": {
"to": "[email protected]",
"subject": "Watcher Notification",
"body": "{{ctx.payload.hits.total}} errors found"
}
}
}
Copy the codeAbove, we send triggered notifications to one of our mailboxes under the heading Watcher Notification. It shows “XX Errors Found”. Also note the use of throttle_period. Because our Watcher runs every minute. In some cases, Watcher will send a notification each time the conditions are met. This creates spam. We don’t want Watcher to send every time it runs. Set it to 2m, or 2 minutes. Send one notification within two minutes.
After our modification, the complete Watcher configuration is as follows:
PUT _watcher/watch/my-justme-watcher
{
"trigger": {
"schedule": {
"cron": "0 0/1 * * * ?"
}
},
"input": {
"search": {
"request": {
"indices": [
"filebeat-*"
],
"body": {
"query": {
"bool": {
"must": [
{
"match": {
"process.name": "JUSTME"
}
},
{
"match": {
"message": "error"
}
}
],
"filter": [
{
"range": {
"@timestamp": {
"from": "{{ctx.trigger.scheduled_time}}||-1m",
"to": "{{ctx.trigger.triggered_time}}"
}
}
}
]
}
},
"aggs": {
"failed_messages": {
"terms": {
"field": "process.name"
}
}
}
}
}
}
},
"condition": {
"compare": {
"ctx.payload.aggregations.failed_messages.buckets.0.doc_count": {
"gt": 5
}
}
},
"actions": {
"send_email": {
"throttle_period": "2m",
"email": {
"to": "[email protected]",
"subject": "Watcher Notification",
"body": "{{ctx.payload.hits.total}} errors found"
}
}
}
}
Copy the codeRun the command above.
Test the Watcher
In many cases, if we set the time longer, we may not be able to wait that long to do a test. Elastic provides the _execute interface for our testing:
PUT _watcher/watch/my-justme-watcher/_execute
Copy the codeThrough the interface above, we don’t have to wait as long as we set in schedule to run our Watcher. The command above will run Watcher directly for us.
At the same time, enter the following commands in our Ubuntu terminal:
sudo logger `who` -t JUSTME error
Copy the code
If we receive 5 or more error messages within a minute, we will receive a notification message in our mail:
We open the contents of the email:
We can remove a Wacher as follows:
DELETE _watcher/watch/<watch_id>
Copy the codeIn our case:
DELETE _watcher/watch/my-justme-watcher
Copy the code
The next step
If you want to know how to create a dynamic threshold alert, read my article “Elastic: How to Create a Dynamic threshold alert.”