This chapter focuses on how to integrate alert notifications using open source components and the Alertmanager component. Kubernetes alert integration will be explained directly in the configuration file, the principle is similar, here only for the relevant alert notification integration.

Alarm notification receiver

I used to check the alarm information in the Web UI, but now I start to use the receiver to integrate with Alertmanager and send the alarm information to Email, enterprise wechat and the pinning robot. For students who have high requirements for alarm, You can configure fly-book, SMS, voice call and other alerts according to the open source component PrometheusAlert bucket mentioned below.

Email

As mentioned earlier, Alertmanager supports Email by default and is the most common way to configure Email. SMTP is built into Alertmanager. You can directly capture the SMTP part of alertManager.yml and adjust and configure it

global:
  resolve_timeout: 5m
  # SMTP configuration
  smtp_from: "[email protected]" # Send email subject
  smtp_smarthost: 'smtp.qq.com:465' SMTP host configuration for the mail server
  smtp_auth_username: "[email protected]" # login user name
  smtp_auth_password: "auth_pass" # Auth password here is the third-party login authorization password of mailbox, not the user password. Try to use QQ to test.
  smtp_require_tls: false # Some mailboxes need to be enabled with this function. Here we use mailbox 163 for testing only. This function is not required.
route:
  receiver: ops
  group_wait: 30s # Wait for the configured time in the group. If the same alarm occurs within 30 seconds in the same group, it will appear in the same group.
  group_interval: 5m # if the contents in the group do not change, merge into an alert message, send after 5m.
  repeat_interval: 24h # Send alarm interval, if there is no repair within the specified time, then send alarm again.
  group_by: [alertname]  # Alarm grouping
  routes:
      - match:
          team: operations
        group_by: [env.dc]
        receiver: 'ops'
      - receiver: ops If the rule's lable contains an alertname, use ops to send it
        group_wait: 10s
        match:
          team: operations
The receiver specifies the sender and the sending channel
receivers:
# Definition of OPS grouping
- name: ops
  email_configs:
  - to: '[email protected],[email protected]' # If you want to send more than one person, you can split it with ','.
    send_resolved: true
    headers:
      from: "Alarm Center"
      subject: "[operations] Alarm email"
      to: "Little Wolf King"
Copy the code

After the configuration is complete, restart the Alertmanager component for the configuration to take effect, and then trigger an alert using the previous memory threshold to see the result of sending.

Alert messages received:

The recovery message received after the alarm has been contacted.

Enterprise WeChat

First of all, you have to have the authority of the enterprise wechat administrator, if not you can register a test, I have a self-registered enterprise wechat here

Step 1 After logging in, create an application in Application Management.

Step 2: Create an app. Fill in the following information and upload the app logo at will.

The following figure shows the successful creation.

At this time, you need to record AgentId and Secret. For your Secret information, it is best to manage it well. Mine will be deleted once used, so there is no need to worry about security risks.

ID key
AgentId 1000004
Secret F-fzpgsabmfiFt7_4QRQwWEl8eyx7evO12sRYe_Q5vA

Step 3, now we use the new enterprise wechat application in Alertmanager configuration, can be configured globally, can also need to send a separate receiver, because the alarm needs to be hierarchical, so it needs to be handled separately, the use of the separate configuration here, need to know the enterprise ID, and department ID.

Enterprise ID Pass

The department ID can be obtained from the address book

# Enterprise wechat configuration
  wechat_configs:
  - corp_id: 'wwxxxxx' The enterprise ID is the unique identifier
    api_url: 'https://qyapi.weixin.qq.com/cgi-bin/' # Enterprise wechat API interface, unified definition
    send_resolved: true Set to send alarm recovery message
    to_party: '2' If you are in DB group, it might be 3. If you are in WEB group, it might be 4. In addition, the relevant person who needs to receive the alert must be in this department.
    agent_id: '1000004' The agent_id of the new application
    api_secret: 'F-fzpgsabmfiFt7_4QRQwWEl8eyx7evO12sRYe_Q5vA' # Create app Secret
Copy the code

At this time, we restart the Alertmanager and use the previous method to trigger the simulated alarm to see if there is no problem sending. At this time, our enterprise wechat and Email can receive the alarm, and I have processed the alarm with the module. It will be more readable.

cat wechat.tmpl
# # wechat template
{{ define "wechat.default.message" }}
{{ if gt (len .Alerts.Firing) 0 -}}
Alerts Firing:
{{ range .Alerts }}
Alarm level: {{ .Labels.status }}

Alarm type: {{ .Labels.alertname }}

Faulty host: {{ .Labels.instance }}

Alert Subject: {{ .Annotations.summary }}

Alert Details: {{ .Annotations.description }}

⏱ : {{ (.StartsAt.Add 28800e9).Format "The 2006-01-02 15:04:05" }}
{{- end }}
{{- end }}

{{ if gt (len .Alerts.Resolved) 0 -}}
Alerts Resolved:
{{ range .Alerts }}
Alarm level: {{ .Labels.status }}

Alarm type: {{ .Labels.alertname }}

Faulty host: {{ .Labels.instance }}

Alert Subject: {{ .Annotations.summary }}

Alert Details: {{ .Annotations.description }}

⏱ : {{ (.StartsAt.Add 28800e9).Format "The 2006-01-02 15:04:05" }}
⏲ : {{ (.EndsAt.Add 28800e9).Format "The 2006-01-02 15:04:05" }}
{{- end }}
{{- end }}
{{- end }}
Copy the code

Firing warning:

Here’s Resolve’s alert:

Webhook

We are all familiar with Dingding, most enterprises have started dingding office, and its free Webhook robot is also very popular with everyone. Let’s look at how you can integrate the alert functionality with the help of third-party open source components.

First of all, you need to create robots in the nail nail, and then add keyword information and IP restrictions and other security Settings in the white list, as long as you have a group, you can build in the group, very simple, here will not do the demonstration

Install the Prometheus- Webhook-dingTalk component first.

mkdir -p /etc/prometheus-webhook-dingtalk/template/
cd/etc/prometheus-webhook-dingtalk/ wget https://github.com/timonwong/prometheus-webhook-dingtalk/releases/download/v1.4.0/prometheus-webhook-dingtalk-1.4.0.linu Gz tar xf Prometheus -webhook-dingtalk-1.4.0.linux-amd64.tar.gz mv /* /etc/ Prometheus -webhook-dingtalk/ mv Prometheus -webhook-dingtalk/ bin/ cat <<EOF> /lib/systemd/system/prometheus-webhook-dingtalk.service [Unit] Description=prometheus-webhook-dingding Documentation=https://prometheus.io/ After=network.target [Service] Type=simple User=prometheus ExecStart=/bin/prometheus-webhook-dingtalk --web.listen-address=":8070" --web.enable-ui --config.file="/etc/prometheus-webhook-dingtalk/config.yml" Restart=on-failure [Install] WantedBy=multi-user.target EOF
## Start service
systemctl enable prometheus-webhook-dingtalk.service
systemctl start prometheus-webhook-dingtalk.service
Copy the code

The configuration file

## Request timeout
# timeout: 5s

## Customizable templates path
## Customizable templates path
templates:
        # - contrib/templates/legacy/template.tmpl
  # Custom template path
  - /etc/prometheus-webhook-dingtalk/template/default.tmpl

You can also override the default template with 'default_message'
## The following example uses the "Legacy" template in V0.3.0
# default_message:
# title: '{{ template "legacy.title" . }}'
# text: '{{ template "legacy.content" . }}'

## Targets, previously was known as "profiles"
# define webhook, nail create webhook token
targets:
You can define multiple interfaces here if you have multiple groups
  ops:
    url: https://oapi.dingtalk.com/robot/send?access_token=a4feed2322222222222222222222222
  web:
    url: https://oapi.dingtalk.com/robot/send?access_token=a4feed2325c1333333333333333333333
Copy the code

Define template:

cd /etc/prometheus-webhook-dingtalk/template
cat default.tmpl
{{ define "__subject" }}[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .GroupLabels.SortedPairs.Values | join "" }} {{ if gt (len .CommonLabels) (len .GroupLabels) }}({{ with .CommonLabels.Remove .GroupLabels.Names }}{{ .Values | join "" }}{{ end }}){{ end }}{{ end }}
{{ define "__alertmanagerURL" }}{{ .ExternalURL }}/#/alerts? receiver={{ .Receiver }}{{ end }}

{{ define "__text_alert_list" }}{{ range . }}
**Labels**
{{ range .Labels.SortedPairs }}> - {{ .Name }}: {{ .Value | markdown | html }}
{{ end }}
**Annotations**
{{ range .Annotations.SortedPairs }}> - {{ .Name }}: {{ .Value | markdown | html }}
{{ end }}
**Source:** [{{ .GeneratorURL }}]({{ .GeneratorURL }})
{{ end }}{{ end }}

{{/* Firing */}}

{{ define "default.__text_alert_list" }}{{ range . }}

**Trigger Time:** {{ dateInZone "2006.01.02 15:04:05" (.StartsAt) "Asia/Shanghai"}} the Summary: * * * * {{. Annotations. The Summary}} Description: * * * * {{. Annotations. The Description}} Graph: * * * * [📈] ({{ .GeneratorURL }}) **Details:** {{ range .Labels.SortedPairs }}{{if and (ne (.Name) "severity") (ne (.Name) "summary") }}> - {{ .Name }}: {{ .Value | markdown | html }}
{{ end }}{{ end }}
{{ end }}{{ end }}

{{/* Resolved */}}

{{ define "default.__text_resolved_list" }}{{ range . }}

**Trigger Time:** {{ dateInZone "2006.01.02 15:04:05" (.StartsAt) "Asia/Shanghai" }}

**Resolved Time:** {{ dateInZone "2006.01.02 15:04:05" (.EndsAt) "Asia/Shanghai"Summary:}} * * * * {{. Annotations. The Summary}} Graph: * * * * [📈] ({{. GeneratorURL}}) * * Details: * * {{range .Labels.SortedPairs }}{{if and (ne (.Name) "severity") (ne (.Name) "summary") }}> - {{ .Name }}: {{ .Value | markdown | html }}
{{ end }}{{ end }}
{{ end }}{{ end }}

{{/* Default */}}
{{ define "default.title" }}{{ template "__subject" . }}{{ end }}
{{ define "default.content" }}#### \[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}\] **[{{ index .GroupLabels "alertname" }}]({{ template "__alertmanagerURL" . }})**
{{ ifgt (len .Alerts.Firing) 0 -}} ! [Firing-img](https://p3-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/8cfb86e5056441e59f57b4153bdc287b~tplv-k3u1fbpfcp-zoom-1.i mage) **Alerts Firing** {{ template"default.__text_alert_list" .Alerts.Firing }}
{{- end }}
{{ ifgt (len .Alerts.Resolved) 0 -}} ! [Resolved-img](https://p3-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/2d591992a0e04f578e3180e2468bbfbd~tplv-k3u1fbpfcp-zoom-1 .image) **Alerts Resolved** {{ template"default.__text_resolved_list" .Alerts.Resolved }}
{{- end }}
{{- end }}

{{/* Legacy */}}
{{ define "legacy.title" }}{{ template "__subject" . }}{{ end }}
{{ define "legacy.content" }}#### \[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}\] **[{{ index .GroupLabels "alertname" }}]({{ template "__alertmanagerURL" . }})**
{{ template "__text_alert_list" .Alerts.Firing }}
{{- end }}

{{/* Following names for compatibility */}}
{{ define "ding.link.title" }}{{ template "default.title" . }}{{ end }}
{{ define "ding.link.content" }}{{ template "default.content" . }}{{ end }}
Copy the code

Configure alerts in Aertmanager

The receiver specifies the sender and the sending channel
receivers:
# Definition of OPS grouping
- name: ops
  email_configs:
  - to: '[email protected],[email protected]'
    send_resolved: true
    headers: { Subject: "[operations] Alarm email"} The header of the received message
  # Nailing configuration
  webhook_configs:
  - url: http://localhost:8070/dingtalk/ops/send # Here is the interface in the pin open source component, if a separate defined receiver needs to correspond to your grouping with the Pin robot's Webhook token
    # Enterprise wechat configuration
  wechat_configs:
  - corp_id: 'ww5421dksajhdasjkhj'
    api_url: 'https://qyapi.weixin.qq.com/cgi-bin/'
    send_resolved: true
    to_party: '2'
    agent_id: '1000002'
    api_secret: 'Tm1kkEE3RGqVhv5hO-khdakjsdkjsahjkdksahjkdsahkj'
# web
- name: web
  email_configs:
  - to: '[email protected]'
    send_resolved: true
    headers: { Subject: "[Web] Alarm Email"} The header of the received message
  webhook_configs:
  - url: http://localhost:8070/dingtalk/web/send
Copy the code

Continue to use the above trigger analog alarm, at this time, three alarms will receive the alarm information, we are just for debugging, often one alarm notification can meet the needs, for important businesses still need to use telephone and SMS reminder.

9. Is the Firing of nails alarm:

Nail Resolve alert:

Alert notification template

Prometheus creates alerts and forwards them to Alertmanager, which sends alert notifications to various receivers based on different labels, such as Email, pins, enterprise wechat, fee-book, SMS, etc. All receivers have a receive template, which is then formatted to send alerts to the Receiver. The templates provided by Alertmanager are templates based on the Go language. Users can define templates based on their own requirements. The templates provided above are sufficient for basic use.

The following describes the parameters commonly used in custom templates

The name of the The data type describe
Receiver string The name of the receiver that receives the alert notification
Status string Alert status, such as an announcement of Firing or resolving
Alert Alert The actual content of the alert notification, all the lists in the alert
GroupLables KV The group label that contains the alert notification
CommandLabels KV Common tag for all alerts, containing all tags for group LABS
CommandAnnotations KV Comments, such as custom strings
ExternalURL string The address of the Alertmanager in the alert message

The KV type mentioned above is a set of Key/Value string pairs that use unlabeled tags and comments, as defined in the default template in Alertmanager. default.tmpl

The definitions of View In AlertManager, Receiver, and ExternalURL displayed In the email are.externalURL and.receiver In the template.

{{ define "__alertmanager" }}AlertManager{{ end }}

{{ define "__alertmanagerURL"}} {{.ExternalURL }}/#/alerts? receiver={{ .Receiver | urlquery }} {{ end }}
.
Copy the code

You can see the View In the received email alerts In AlertManager link address is: http://192.168.1.220:19093/#/alerts? Receiver = ops.

For the type of Alert, the fields in the Alert list also contain the following parameters, definitions, and descriptions

The name of the The data type describe
Status string Defines the alert state as resolved or triggered
Label KV Contains the tags in the alert
Annotations KV A set of comments for alerts
StartsAt time.Time Alarm trigger time
EndsAt time.Time Alarm end time, set only at the end of the alarm time
GeneratorURL string The link URL for the alert rule, which is the Rules lookup address in Prometheus

For the alarm notification template must first be familiar with the language of the template in the grammar and the basic knowledge of simple HTML, then put some of the information above related metadata know, can modify the template, if you are lazy, I templates can be adjusted to use directly, the corresponding index, the change can use the tag name.

The following is a modified version of the template alert format, which you can check out using the official default.tmpl.

Open source alert components are recommended

  • Prometheus-Webhook-Dingtalk

An open source third-party alert plug-in, for Dingpin robot Webhook integration, Go language writing, now the iteration has been very good, there may be some limitations, such as markdown@ a person can not be implemented, because dingpin’s own API does not support this function.

  • Alertmanager-wechatrobot-webhook

This open source component converts Alertmanger Webhook messages into enterprise wechat robots that can receive messages. Also written in go language, Alertmanager has integrated enterprise wechat configuration by default. If you need to use enterprise wechat robots, you can check this out.

  • PrometheusAlert buckets

If there is demand for SMS, telephone alarm and other classmates, recommend this open source alarm components, the language writing, Web framework is Beego, support will receive these messages to the nail, WeChat, fly, tencent SMS, tencent’s telephone, ali cloud SMS, ali cloud telephone, SMS, huawei let cloud telephones, spoke not fine here, Please feel free to consult me if you have any problems with the configuration.