Longhorn, Enterprise Cloud Native Container Distributed Storage – Monitoring (Prometheus+AlertManager+Grafana)

The content is from the official Longhorn 1.1.2 English technical manual.

A series of

• What’s a Longhorn?
• Longhorn enterprise cloud native container distributed storage solution design architecture and concepts
• Longhorn Enterprise Cloud Native Container Distributed storage – Deployment
• Longhorn Enterprise Cloud Native Container Distributed Storage – Volume and Node
• Longhorn, enterprise cloud native Container Distributed storage -K8S resource configuration example

directory

1. Set up thePrometheus 和 GrafanaTo monitorLonghorn
2. willLonghornMetrics integration intoRancherIn the monitoring system
3. LonghornMonitoring indicators
4. supportKubelet Volumeindicators
5. LonghornExample alert rule

Set up thePrometheus 和 GrafanaTo monitorLonghorn

An overview of

Longhorn natively exposes metrics in Prometheus text format at REST endpoint http://LONGHORN_MANAGER_IP:PORT/metrics. For a description of all the metrics available, see Longhorn’s Metrics. You can capture these metrics using any collection tool such as Prometheus, Graphite, Telegraf, etc., and then visualize the collected data using tools such as Grafana.

This document provides a sample setup for monitoring Longhorn. Monitoring systems used Prometheus to collect data and alerts, and Grafana to visualise /dashboarding the collected data. In a high-level overview, monitoring systems include:

• PrometheusServer fromLonghornIndicator endpoints capture and store time series data.PrometheusIt is also responsible for generating alerts based on configured rules and collected data.PrometheusThe server then sends the alert toAlertmanager.
• AlertManagerThen manage these alerts (alerts), including silence (silencing), restrain (inhibition), aggregation (aggregation) and send notifications via email, call notification systems and chat platforms.
• Grafana 向 PrometheusThe server queries the data and draws dashboards for visualization.

The following figure describes the detailed architecture of the monitoring system.

There are two unmentioned components in the figure above:

• The Longhorn backend service is pointingLonghorn manager podsSet of services.LonghornThe index is at the endpointhttp://LONGHORN_MANAGER_IP:PORT/metrics 的 Longhorn manager podsIn the public.
• Prometheus operatorMade inKubernetesRunning on thePrometheusIt becomes very easy.operatormonitoring3Five custom resources:ServiceMonitor,Prometheus 和 AlertManager. When users create these custom resources,Prometheus OperatorDeploy and manage using user-specified configurationsPrometheus server.AlerManager.

The installation

Follow these instructions to install all components into the Monitoring namespace. To install them in different namespaces, change the field namespace: OTHER_NAMESPACE

createmonitoringThe namespace

apiVersion: v1
kind: Namespace
metadata:
  name: monitoring
Copy the code

The installationPrometheus Operator

Deploy the Prometheus Operator and its required ClusterRole, ClusterRoleBinding, and Service Account.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/name: prometheus-operator
    app.kubernetes.io/version: v0.38.3
  name: prometheus-operator
  namespace: monitoring
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus-operator
subjects:
- kind: ServiceAccount
  name: prometheus-operator
  namespace: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/name: prometheus-operator
    app.kubernetes.io/version: v0.38.3
  name: prometheus-operator
  namespace: monitoring
rules:
- apiGroups:
  - apiextensions.k8s.io
  resources:
  - customresourcedefinitions
  verbs:
  - create
- apiGroups:
  - apiextensions.k8s.io
  resourceNames:
  - alertmanagers.monitoring.coreos.com
  - podmonitors.monitoring.coreos.com
  - prometheuses.monitoring.coreos.com
  - prometheusrules.monitoring.coreos.com
  - servicemonitors.monitoring.coreos.com
  - thanosrulers.monitoring.coreos.com
  resources:
  - customresourcedefinitions
  verbs:
  - get
  - update
- apiGroups:
  - monitoring.coreos.com
  resources:
  - alertmanagers
  - alertmanagers/finalizers
  - prometheuses
  - prometheuses/finalizers
  - thanosrulers
  - thanosrulers/finalizers
  - servicemonitors
  - podmonitors
  - prometheusrules
  verbs:
  - The '*'
- apiGroups:
  - apps
  resources:
  - statefulsets
  verbs:
  - The '*'
- apiGroups:
  - ""
  resources:
  - configmaps
  - secrets
  verbs:
  - The '*'
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - list
  - delete
- apiGroups:
  - ""
  resources:
  - services
  - services/finalizers
  - endpoints
  verbs:
  - get
  - create
  - update
  - delete
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - namespaces
  verbs:
  - get
  - list
  - watch
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/name: prometheus-operator
    app.kubernetes.io/version: v0.38.3
  name: prometheus-operator
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/component: controller
      app.kubernetes.io/name: prometheus-operator
  template:
    metadata:
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/name: prometheus-operator
        app.kubernetes.io/version: v0.38.3
    spec:
      containers:
      - args:
        - --kubelet-service=kube-system/kubelet
        - --logtostderr=true
        - - config - reloader - image = jimmidyson/configmap - reload: v0.3.0
        - - Prometheus - config - reloader = quay. IO/Prometheus - operator/Prometheus - config - reloader: v0.38.3
        image: Quay. IO/Prometheus - operator/Prometheus - operator: v0.38.3
        name: prometheus-operator
        ports:
        - containerPort: 8080
          name: http
        resources:
          limits:
            cpu: 200m
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        securityContext:
          allowPrivilegeEscalation: false
      nodeSelector:
        beta.kubernetes.io/os: linux
      securityContext:
        runAsNonRoot: true
        runAsUser: 65534
      serviceAccountName: prometheus-operator
---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/name: prometheus-operator
    app.kubernetes.io/version: v0.38.3
  name: prometheus-operator
  namespace: monitoring
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/name: prometheus-operator
    app.kubernetes.io/version: v0.38.3
  name: prometheus-operator
  namespace: monitoring
spec:
  clusterIP: None
  ports:
  - name: http
    port: 8080
    targetPort: http
  selector:
    app.kubernetes.io/component: controller
    app.kubernetes.io/name: prometheus-operator
Copy the code

The installationLonghorn ServiceMonitor

Longhorn ServiceMonitor has a label selector app: Longhorn-Manager to select the Longhorn backend service. Later, Prometheus CRD could include the Longhorn ServiceMonitor so that Prometheus Server could discover all Longhorn Manager Pods and their endpoints.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: longhorn-prometheus-servicemonitor
  namespace: monitoring
  labels:
    name: longhorn-prometheus-servicemonitor
spec:
  selector:
    matchLabels:
      app: longhorn-manager
  namespaceSelector:
    matchNames:
    - longhorn-system
  endpoints:
  - port: manager
Copy the code

Installation and configurationPrometheus AlertManager

1. Create a highly available Alertmanager deployment with three instances:

   apiVersion: monitoring.coreos.com/v1
   kind: Alertmanager
   metadata:
     name: longhorn
     namespace: monitoring
   spec:
     replicas: 3
   Copy the code

2. An Alertmanager instance will not start unless a valid configuration is provided. For more instructions on Alertmanager configuration, see here. The following code shows an example configuration:

   global:
     resolve_timeout: 5m
   route:
     group_by: [alertname]
     receiver: email_and_slack
   receivers:
   - name: email_and_slack
     email_configs:
     - to: <the email address to send notifications to>
       from: <the sender address>
       smarthost: <the SMTP host through which emails are sent>
       # SMTP authentication information.
       auth_username: <the username>
       auth_identity: <the identity>
       auth_password: <the password>
       headers:
         subject: 'Longhorn-Alert'
       text: |- {{ range .Alerts }} *Alert:* {{ .Annotations.summary }} - `{{ .Labels.severity }}` *Description:* {{ Annotations. Description}} *Details:* {{range.attractions.sortedpairs}} • *{{{.name}}:* '{{.value}}' {{end}} {{end} }}  slack_configs:
     - api_url: <the Slack webhook URL>
       channel: <the channel or user to send notifications to>
       text: |- {{ range .Alerts }} *Alert:* {{ .Annotations.summary }} - `{{ .Labels.severity }}` *Description:* {{ Annotations. Description}} *Details:* {{range.attractions.sortedpairs}} • *{{{.name}}:* '{{.value}}' {{end}} {{end} }}Copy the code

   Save the above Alertmanager configuration in a file named AlertManager.yaml and create a secret from it using Kubectl.

   The Alertmanager instance requires that the secret resource name follow the Alertmanager -{ALERTMANAGER_NAME} format. In the previous step, the name of Alertmanager was Longhorn, so the secret name must be AlertManager-Longhorn

   $ kubectl create secret generic alertmanager-longhorn --from-file=alertmanager.yaml -n monitoring
   Copy the code

3. To be able to view the Web UI of the Alertmanager, expose it through a Service. A simple way to do this is to use a Service of type NodePort:

   apiVersion: v1
   kind: Service
   metadata:
     name: alertmanager-longhorn
     namespace: monitoring
   spec:
     type: NodePort
     ports:
     - name: web
       nodePort: 30903
       port: 9093
       protocol: TCP
       targetPort: web
     selector:
       alertmanager: longhorn
   Copy the code

   After the preceding services are created, you can access the Web UI of Alertmanager using the IP address and port 30903 of the node.

   Use the NodePort service above for quick validation, as it does not communicate over a TLS connection. You might want to change the service type to ClusterIP and set up an Ingress-Controller to expose the Alertmanager web UI over TLS connections.

Installation and configurationPrometheus server

1. Create a custom PrometheusRule resource that defines alert conditions.

   apiVersion: monitoring.coreos.com/v1
   kind: PrometheusRule
   metadata:
     labels:
       prometheus: longhorn
       role: alert-rules
     name: prometheus-longhorn-rules
     namespace: monitoring
   spec:
     groups:
     - name: longhorn.rules
       rules:
       - alert: LonghornVolumeUsageCritical
         annotations:
           description: Longhorn volume {{$labels.volume}} on {{$labels.node}} is at {{$value}}% used for
             more than 5 minutes.
           summary: Longhorn volume capacity is over 90% used.
         expr: 100 * (longhorn_volume_usage_bytes / longhorn_volume_capacity_bytes) > 90
         for: 5m
         labels:
           issue: Longhorn volume {{$labels.volume}} usage on {{$labels.node}} is critical.
           severity: critical
   Copy the code

   For more information on how to define the alarm rules, please see Prometheus. IO/docs/promet…

2. If RBAC authorization is enabled, create a ClusterRole and ClusterRoleBinding for Prometheus Pod:

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: prometheus
     namespace: monitoring
   Copy the code

   apiVersion: rbac.authorization.k8s.io/v1beta1
   kind: ClusterRole
   metadata:
     name: prometheus
     namespace: monitoring
   rules:
   - apiGroups: [""]
     resources:
     - nodes
     - services
     - endpoints
     - pods
     verbs: ["get"."list"."watch"]
   - apiGroups: [""]
     resources:
     - configmaps
     verbs: ["get"]
   - nonResourceURLs: ["/metrics"]
     verbs: ["get"]
   Copy the code

   apiVersion: rbac.authorization.k8s.io/v1beta1
   kind: ClusterRoleBinding
   metadata:
     name: prometheus
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: ClusterRole
     name: prometheus
   subjects:
   - kind: ServiceAccount
     name: prometheus
     namespace: monitoring
   Copy the code

3. Create a Prometheus custom resource. Note that we selected the Longhorn Service Monitor and Longhorn rules in the spec.

   apiVersion: monitoring.coreos.com/v1
   kind: Prometheus
   metadata:
     name: prometheus
     namespace: monitoring
   spec:
     replicas: 2
     serviceAccountName: prometheus
     alerting:
       alertmanagers:
         - namespace: monitoring
           name: alertmanager-longhorn
           port: web
     serviceMonitorSelector:
       matchLabels:
         name: longhorn-prometheus-servicemonitor
     ruleSelector:
       matchLabels:
         prometheus: longhorn
         role: alert-rules
   Copy the code

4. To be able to view the Web UI of the Prometheus server, expose it through a Service. A simple way to do this is to use a Service of type NodePort:

   apiVersion: v1
   kind: Service
   metadata:
     name: prometheus
     namespace: monitoring
   spec:
     type: NodePort
     ports:
     - name: web
       nodePort: 30904
       port: 9090
       protocol: TCP
       targetPort: web
     selector:
       prometheus: prometheus
   Copy the code

   After the service is created, you can access the Web UI of Prometheus Server using the IP address and port 30904 of the node.

   At this point, you should be able to see all Longhorn Manager Targets and Longhorn Rules in the Targets and Rules section of the Prometheus Server UI.

   Use the above NodePort service for quick validation because it does not communicate over a TLS connection. You may want to change the service type to ClusterIP and set up an Ingress-Controller to expose Prometheus Server’s Web UI over TLS connections.

Install Grafana

1. Create Grafana data source configuration:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: grafana-datasources
     namespace: monitoring
   data:
     prometheus.yaml: |- { "apiVersion": 1, "datasources": [ { "access":"proxy", "editable": true, "name": "prometheus", "orgId": 1, "type": "prometheus", "url": "http://prometheus:9090", "version": 1 } ] }Copy the code

2. Creating a Grafana deployment:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: grafana
     namespace: monitoring
     labels:
       app: grafana
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: grafana
     template:
       metadata:
         name: grafana
         labels:
           app: grafana
       spec:
         containers:
         - name: grafana
           image: Grafana/grafana: 7.1.5
           ports:
           - name: grafana
             containerPort: 3000
           resources:
             limits:
               memory: "500Mi"
               cpu: "300m"
             requests:
               memory: "500Mi"
               cpu: "200m"
           volumeMounts:
             - mountPath: /var/lib/grafana
               name: grafana-storage
             - mountPath: /etc/grafana/provisioning/datasources
               name: grafana-datasources
               readOnly: false
         volumes:
           - name: grafana-storage
             emptyDir: {}
           - name: grafana-datasources
             configMap:
                 defaultMode: 420
                 name: grafana-datasources
   Copy the code

3. Grafana exposure on NodePort 32000:

   apiVersion: v1
   kind: Service
   metadata:
     name: grafana
     namespace: monitoring
   spec:
     selector:
       app: grafana
     type: NodePort
     ports:
       - port: 3000
         targetPort: 3000
         nodePort: 32000
   Copy the code

   Use the above NodePort service for quick validation because it does not communicate over a TLS connection. You might want to change the service type to ClusterIP and set up an Ingress-Controller to expose Grafana over TLS connections.

4. Access the Grafana dashboard using any node IP on port 32000. The default credentials are:

   User: admin
   Pass: admin
   Copy the code

5. Install the Longhorn dashboard

   After entering Grafana, import the preset panel: grafana.com/grafana/das…

   For instructions on how to import the Grafana Dashboard, see grafana.com/docs/grafan…

   Upon success, you should see the following dashboards:

willLonghornMetrics integration intoRancherIn the monitoring system

aboutRancherThe monitoring system

With Rancher, you can monitor the status and progress of cluster nodes, Kubernetes components, and software deployments through integration with Prometheus, a leading open source monitoring solution.

For instructions on how to deploy/enable the Rancher monitoring system, see rancher.com/docs/ranche…

willLonghornAdd metrics toRancherThe monitoring system

If you use Rancher to manage your Kubernetes and have Rancher monitoring enabled, you can add Longhorn metrics to Rancher monitoring by simply deploying the following ServiceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: longhorn-prometheus-servicemonitor
  namespace: longhorn-system
  labels:
    name: longhorn-prometheus-servicemonitor
spec:
  selector:
    matchLabels:
      app: longhorn-manager
  namespaceSelector:
    matchNames:
    - longhorn-system
  endpoints:
  - port: manager
Copy the code

After the ServiceMonitor is created, Rancher will automatically discover all Longhorn metrics.

You can then set up the Grafana dashboard for visualization.

LonghornMonitoring indicators

Volume (Volume)

The Node (the Node)

Disk (Disk)

Instance Manager(Instance Manager)

Manager

supportKubelet Volumeindicators

aboutKubelet Volumeindicators

Kubelet discloses the following metrics:

1. kubelet_volume_stats_capacity_bytes
2. kubelet_volume_stats_available_bytes
3. kubelet_volume_stats_used_bytes
4. kubelet_volume_stats_inodes
5. kubelet_volume_stats_inodes_free
6. kubelet_volume_stats_inodes_used

These metrics measure information related to the PVC file system within the Longhorn Block device.

They differ from the Longhorn_volume_ * metric, which measures information specific to a Longhorn block device.

You can set up a monitoring system to grab Kubelet metric endpoints to get the status of the PVC and set up alerts for abnormal events, such as the PVC running out of storage space.

A popular monitoring setup is promethy-operator/kube-Promethy-Stack, which grabs the kubelet_volume_stats_* metrics and provides them with dashboards and alert rules.

Longhorn CSI plugin support

In V1.1.0, the Longhorn CSI plug-in supports NodeGetVolumeStats RPC according to the CSI spec.

This allows Kubelet to query the Longhorn CSI plug-in to get the status of the PVC.

Kubelet then exposes this information in the kubelet_volume_stats_* metric.

LonghornExample alert rule

We have provided several sample Longhorn alert rules below for your reference. See here for a list of all the Longhorn metrics available and to build your own alert rules.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  labels:
    prometheus: longhorn
    role: alert-rules
  name: prometheus-longhorn-rules
  namespace: monitoring
spec:
  groups:
  - name: longhorn.rules
    rules:
    - alert: LonghornVolumeActualSpaceUsedWarning
      annotations:
        description: The actual space used by Longhorn volume {{$labels.volume}} on {{$labels.node}} is at {{$value}}% capacity for
          more than 5 minutes.
        summary: The actual used space of Longhorn volume is over 90% of the capacity.
      expr: (longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes) * 100 > 90
      for: 5m
      labels:
        issue: The actual used space of Longhorn volume {{$labels.volume}} on {{$labels.node}} is high.
        severity: warning
    - alert: LonghornVolumeStatusCritical
      annotations:
        description: Longhorn volume {{$labels.volume}} on {{$labels.node}} is Fault for
          more than 2 minutes.
        summary: Longhorn volume {{$labels.volume}} is Fault
      expr: longhorn_volume_robustness = = 3
      for: 5m
      labels:
        issue: Longhorn volume {{$labels.volume}} is Fault.
        severity: critical
    - alert: LonghornVolumeStatusWarning
      annotations:
        description: Longhorn volume {{$labels.volume}} on {{$labels.node}} is Degraded for
          more than 5 minutes.
        summary: Longhorn volume {{$labels.volume}} is Degraded
      expr: longhorn_volume_robustness = = 2
      for: 5m
      labels:
        issue: Longhorn volume {{$labels.volume}} is Degraded.
        severity: warning
    - alert: LonghornNodeStorageWarning
      annotations:
        description: The used storage of node {{$labels.node}} is at {{$value}}% capacity for
          more than 5 minutes.
        summary:  The used storage of node is over 70% of the capacity.
      expr: (longhorn_node_storage_usage_bytes / longhorn_node_storage_capacity_bytes) * 100 > 70
      for: 5m
      labels:
        issue: The used storage of node {{$labels.node}} is high.
        severity: warning
    - alert: LonghornDiskStorageWarning
      annotations:
        description: The used storage of disk {{$labels.disk}} on node {{$labels.node}} is at {{$value}}% capacity for
          more than 5 minutes.
        summary:  The used storage of disk is over 70% of the capacity.
      expr: (longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes) * 100 > 70
      for: 5m
      labels:
        issue: The used storage of disk {{$labels.disk}} on node {{$labels.node}} is high.
        severity: warning
    - alert: LonghornNodeDown
      annotations:
        description: There are {{$value}} Longhorn nodes which have been offline for more than 5 minutes.
        summary: Longhorn nodes is offline
      expr: longhorn_node_total - (count(longhorn_node_status{condition="ready"}==1) OR on() vector(0))
      for: 5m
      labels:
        issue: There are {{$value}} Longhorn nodes are offline
        severity: critical
    - alert: LonghornIntanceManagerCPUUsageWarning
      annotations:
        description: Longhorn instance manager {{$labels.instance_manager}} on {{$labels.node}} has CPU Usage / CPU request is {{$value}}% for
          more than 5 minutes.
        summary: Longhorn instance manager {{$labels.instance_manager}} on {{$labels.node}} has CPU Usage / CPU request is over 300%.
      expr: (longhorn_instance_manager_cpu_usage_millicpu/longhorn_instance_manager_cpu_requests_millicpu) * 100 > 300
      for: 5m
      labels:
        issue: Longhorn instance manager {{$labels.instance_manager}} on {{$labels.node}} consumes 3 times the CPU request.
        severity: warning
    - alert: LonghornNodeCPUUsageWarning
      annotations:
        description: Longhorn node {{$labels.node}} has CPU Usage / CPU capacity is {{$value}}% for
          more than 5 minutes.
        summary: Longhorn node {{$labels.node}} experiences high CPU pressure for more than 5m.
      expr: (longhorn_node_cpu_usage_millicpu / longhorn_node_cpu_capacity_millicpu) * 100 > 90
      for: 5m
      labels:
        issue: Longhorn node {{$labels.node}} experiences high CPU pressure.
        severity: warning
Copy the code

The Prometheus. IO/docs/promet… See more information on how to define alert rules.