This is the fourth day of my participation in the First Challenge 2022

preface

Configuring alarm distribution to different spike groups in Prometheus describes the configuration of distributing the same alarm to multiple spike groups. The following is a template of a nail alarm that I usually use. I can’t remember where TO find the original template, and it took quite a bit of time. Later, I made some adjustments by myself, and it became the main form I use now.

Nail the alarm component

Use of Prometheus – webhook – dingtalk

The template configuration

If you want to visualize custom templates, you can start dingTalk with the following parameters :–web.enable-ui:

./prometheus-webhook-dingtalk  --web.enable-ui
Copy the code

After doing this, you can go to http://localhost:8060/ui to preview, which is the template style I want to provide:

​

To define a template, use go template: golang.org/pkg/text/te… , can be customized according to the situation

Template configuration code

{{ define "__subject" }}[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .GroupLabels.SortedPairs.Values | join " " }} {{ if gt (len .CommonLabels) (len .GroupLabels) }}({{ with .CommonLabels.Remove .GroupLabels.Names }}{{ .Values | join " " }}{{ end }}){{ end }}{{ end }} {{ define "__alertmanagerURL" }}{{ $alertURL := "http://localhost:9093" }} {{- $alertURL -}}/#/alerts? receiver={{ .Receiver }}&tmp={{ .ExternalURL }}{{ end }} {{ define "__text_alert_list" }}{{ range . }} **Labels** {{ range .Labels.SortedPairs }}> - {{ .Name }}: {{ .Value | markdown | html }} {{ end }} **Annotations** {{ range .Annotations.SortedPairs }}> - {{ .Name }}: {{ .Value | markdown | html }} {{ end }} **Source:** [{{ .GeneratorURL }}]({{ .GeneratorURL }}) {{ end }}{{ end }} {{/* Firing */}} {{define "default.__text_alert_list"}}{{range.}} ** (. StartsAt) "Asia/Shanghai}}" abstract: * * * * {{. Annotations. The summary}} description: * * * * {{. Annotations. The description}} monitoring: * * * * [grafana](http://localhost:3000) ** 表 表 :** {{range.labels.sortedpairs}}{{if and (ne (.name) "severity") (ne (.name)) "summary") }}> - {{ .Name }}: {{ .Value | markdown | html }} {{ end }}{{ end }} {{ end }}{{ end }} {{/* Resolved */}} {{ define "Default.__text_resolved_list"}}{{range.}} ** dateInZone "2006.01.02 15:04:05" (.startsat) "Asia/Shanghai" } ** {{dateInZone "2006.01.02 15:04:05" (.endsat) "Asia/Shanghai"}} ** ** * [grafana](http://localhost:3000) ** * {{range.labalogs.sortedpairs}}{{if and (ne (.name) "severity") (ne) (.Name) "summary") }}> - {{ .Name }}: {{ .Value | markdown | html }} {{ end }}{{ end }} {{ end }}{{ end }} {{/* Default */}} {{ define "default.title" }}{{ template "__subject" . }}{{ end }} {{ define "default.content" }}#### [{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] **[{{ index .GroupLabels "alertname" }}]({{ template "__alertmanagerURL" . }})**  {{ if gt (len .Alerts.Firing) 0 -}} ! [Firing-img](https://is3-ssl.mzstatic.com/image/thumb/Purple20/v4/e0/23/cf/e023cf56-0623-0cdf-afce-97ae90eabfda/mzl.uplm Rpgi.png / 320x0W.jpg) ** Alarm notification ** {{template "default.__text_alert_list".Alerts.Firing}} {{-end}} {{if gt (len) .Alerts.Resolved) 0 -}} ! [Resolved-img](https://is3-ssl.mzstatic.com/image/thumb/Purple18/v4/41/72/99/4172990a-f666-badf-9726-6204a320c16e/mzl.dy Pdixoy.png /320x0w.png) ** Alarm cleared ** {{template "default.__text_resolved_list".Alerts.Resolved}} {{-end}} {{-end}} {{/* Legacy */}} {{ define "legacy.title" }}{{ template "__subject" . }}{{ end }} {{ define "legacy.content" }}#### [{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] **[{{ index .GroupLabels "alertname" }}]({{ template "__alertmanagerURL" . }})** {{ template "__text_alert_list" .Alerts.Firing }} {{- end }} {{/* Following names for compatibility */}} {{ define "ding.link.title" }}{{ template "default.title" . }}{{ end }} {{ define "ding.link.content" }}{{ template "default.content" . }}{{ end }}Copy the code

You can simply copy and paste the preceding alarms or monitor urls based on your scenario.

The alarm URL in the template is faulty

Click the URL of an alarm to go to the alarm management page. Silent alarms are allowed. Some alarms can be ignored temporarily. The alarm management page is as follows:

​

However, the alarm URL is incorrect and an incorrect address is displayed. My solution is to specify alertURL and ignore ExternalURL.

​

ExternalURL can be used at github.com/timonwong/p…

My implementation scheme is because I can operate only this dingTalk or AlertManager, Prometheus is integrated with others, it is not convenient to restart the configuration, I want such a scheme, just for reference.