Video tutorial

• Tutorial addresses

preface

Why was Prometheus built for cloud-native monitoring

• You’ve heard more than once that Prometheus was built for cloud-native monitoring, but have you ever wondered what that means?
• We know how to use containers and the K8S environment as the base running environment in cloud native

• The integrated architecture is divided into many scattered micro-services, and the changes and expansion of micro-services are particularly frequent, which leads to frequent changes in the collected target information. This puts forward two requirements for the timing monitoring system:

  • A large number of POD containers running across multiple hosts need to be collected
  • At the same time, be aware of their changes
  • At the same time to build a complete K8S monitoring ecology, can have
• To put it bluntly, monitoring in the cloud native environment has become more difficult and complex. So you need a system that is designed from the start for cloud native monitoring scenarios, and Prometheus is designed to do just that.

What improvements has Prometheus made to merit the K8?

What indicators should be paid attention to in K8s

In the table below, I briefly list the four major indicators that we need to pay attention to in K8s:

The index type	Collecting source	Application, for example,	Found type	Grafana screenshots
Container-based resource metrics	Kubelet has a built-in CAdvisor Metrics interface	Look at the container CPU, MEM utilization, etc	The k8s_sd node level accesses node_ip directly
K8S resource indicators	kube-stats-metrics(KSM)	You can see thatA look at the many components of K8S from container monitoring kube-stats-metrics Look at why the pod state is like the pod waiting state For example, check the distribution of Node Pod according to namespace	Domain names are accessed through coredns
K8S service component metrics	Service component Metrics interface	View Apiserver, Scheduler, ETC, Coredns Request Delay, etc	K8s_sd endpoint level
Deploy business buried point metrics in POD	The POD Metrics interface	Based on business metrics scenarios	K8s_sd pod level, access pod IP metricspath

Adaptation 1. SDK + index self-exposure +pull model: build the entire ecology monitored by K8S

• In the table listed above, we see four major indicators that need to be paid attention to in K8S
• In fact, we can simply divide K8S users into two roles: K8S cluster administrators and ordinary users. Each role has different metrics to focus on

Is it not tiring to gather by yourself?

• Since there are so many demands, if it is only collected by the monitoring system itself, the first is very tired, and the second is not able to build such a complete ecology

secret

• Prometheus is the pull model collection, each monitored source only needs to expose its own metrics in the local HTTP port, Prometheus can access the interface to collect metrics
• The same is true for Prometheus in K8S, where components need to expose their own metrics. As we mentioned in Container Base Resource metrics, Kubelet’s built-in CAdvisor metrics are exposed to /metrics/ CAdvisor on port 10250.
• Prometheus discovers these indicators through the K8S service to complete the collection

Adaptation 2. K8S service discovery

For example,

• Endpoint level service discovery: Examples include collecting Apiserver, Kube-Controller-Manager, etc

kubernetes_sd_configs:
   role: endpoints

• Node level service discovery: For example, when collecting metrics for CAdvisor and Kubelet themselves

  kubernetes_sd_configs:
  - role: node

• 3. Node level service discovery: for example, when collecting POD self-firing point indicators

  kubernetes_sd_configs:
  - role: pod
    follow_redirects: true

Interpretation: Watch updates instantly

• The real-time discovery of resource changes through Watch meets one of the challenges of monitoring under cloud native conditions that we proposed at the very beginning. It is necessary to timely perceive the changes of the acquisition source.
• At the same time, Prometheus can access the discovered endpoint, node and pod in the K8S large layer environment

3. Collection authentication: Token & certificate

Many interfaces in K8S need authentication, and even require TLS bi-directional authentication

• At the same time, we know that many interfaces in K8S have access authentication. For example, if we directly access Kubelet’s /metrics/ CAdvisor interface on K8S node, we will return unauthorized access. As shown below

[root@Kubernetes-node01 logs]# curl -k https://localhost:10250/metrics/cadvisor
Unauthorized

• Prometheus also faces authentication problems when collecting CAdvisor indicators

The solution

• The clever Prometheus developers solved this problem by supporting the associated tokens and certificates in the configuration in the collection, as shown in the configuration below which represents a token file and a certificate file.

bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true

• As we know, when using k8s k8s through service account, clusterrolebinding to solve the problem of certificate of token, mount
• Prometheus is using this, create Prometheus container is related to the service account, clusterrolebinding example configuration are as follows:

ApiVersion: rbac authorization. K8s. IO/v1 # API version kind: ClusterRole # type metadata: name: Prometheus rules: -apigroups: [""] resources: # -nodes-nodes/proxy-services-endpoints-pods verbs: ["get", "list", "watch"] - apiGroups: - extensions resources: - ingresses verbs: ["get", "list", "watch"] - nonResourceURLs: ["/metrics"] verbs: ["get"] --- apiVersion: v1 kind: ServiceAccount metadata: name: Prometheus # namespace: kube-system # namespace -- apiVersion: . Rbac authorization. K8s. IO/v1 kind: ClusterRoleBinding metadata: name: Prometheus roleRef: # choose need binding Role apiGroup: Rbac. Authorization. K8s. IO kind: ClusterRole name: cluster - admin the subjects: # object - kind: ServiceAccount name: prometheus namespace: kube-system

• We need to configure the corresponding serviceAccountName in Prometheus YAML when we create the statsfulset related to PrometheusserviceAccountName: prometheus
• Once configured, Kubernetes will mount the corresponding token and certificate files into POD.
• When we exec into POD of Prometheus, we can view relevant files in/var/run/secrets/kubernetes.io/serviceaccount/, as shown in the figure below:

/ # ls /var/run/secrets/kubernetes.io/serviceaccount/ -l total 0 lrwxrwxrwx 1 root root 13 Jan 7 20:54 ca.crt -> .. data/ca.crt lrwxrwxrwx 1 root root 16 Jan 7 20:54 namespace -> .. data/namespace lrwxrwxrwx 1 root root 12 Jan 7 20:54 token -> .. Data/token / # df -h | grep service TMPFS 7.8 G 7.8 G 0% 12.0 K/var/run/secrets/kubernetes. IO/serviceaccount / #

• The secret of the associated certificate needs to be configured when collecting the ETCD

  kubectl create secret generic etcd-certs --from-file=/etc/kubernetes/pki/etcd/healthcheck-client.crt --from-file=/etc/kubernetes/pki/etcd/healthcheck-client.key --from-file=/etc/kubernetes/pki/etcd/ca.crt -n kube-system
  

4. Powerful Relabel ability to intercept, transform and static sharding labels

Prometheus relabel instructions

• The document address https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config

Application 1: LabelMap intercepts the key name of the service discovery label when collecting CAdvisor metrics

• You can see that the service discovery source has been added a lot when you harvest the CAdvisor__meta_kubernetes_node_label_Opening label

• But these label names are too long and need to be pared down. We use the relabel configuration as follows

relabel_configs:
- separator: ;
regex: __meta_kubernetes_node_label_(.+)
replacement: $1
action: labelmap

• Take this tag, for example,__meta_kubernetes_node_label_kubernetes_io_os="linux"The above configuration represents the key that matches the beginning of _meta_kubernetes_node_label_, keeping only the rest of the key, so what you see in the final tag isbeta_kubernetes_io_os="linux".
• LabelMap assigns a value to the target label on behalf of the matched label

Use 2: REPLACE to assign value to the label while collecting POD custom index

• When we use pod custom indicators in pod yaml spec. The template. The metadata. The need to define three in the annotations to Prometheus. IO configuration in the beginning, The distribution represents whether Prometheus collection is required, the ports exposed by Metrics, and the HTTP path information of Metrics. The details are as follows:

  spec:
  selector:
    matchLabels:
      name: fluentd-elasticsearch
  template:
    metadata:
      labels:
        name: fluentd-elasticsearch
      annotations:
        prometheus.io/scrape: 'true'
        prometheus.io/port: '9102'
        prometheus.io/path: 'metrics'

• The POD custom indicators are collected as follows

  relabel_configs: - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] separator: ; regex: "true" replacement: $1 action: keep - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path] separator: ; regex: (.+) target_label: __metrics_path__ replacement: $1 action: replace - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port] separator: ; regex: ([^:]+)(? ::\d+)? ; (\d+) target_label: __address__ replacement: $1:$2 action: replace

• That is to__meta_kubernetes_pod_annotation_prometheus_io_pathAssigned to__metrics_path__
• It means will be related__meta_kubernetes_pod_annotation_prometheus_io_portAssigned to__address__Port behind

Application 2: keep does the filtering when collecting the service component’s endpoint

• The Endpoint resource is a list of IP addresses and ports that expose a service
• It means to use the K8S service to discover endpoints. There are a lot of endpoints, so you need to filter the Apiserver

kubernetes_sd_configs:
- role: endpoints

• The filter method matches default for the tag __meta_kubernetes_namespace and __meta_kubernetes_service_name matches kubernetes and __meta_kubernetes_endpoint_port_name matches HTTPS.keep

relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] separator: ; regex: default; kubernetes; https replacement: $1 action: keep

• K8S will create a service for the APIServer in the Default Namespace

  443 $kubectl get SVC - A | grep default kubernetes ClusterIP 10.96.0.1 < none > 443 / TCP 9 d

• The final acquired endpoint is converted to the collection path:https://masterip:6443/metrics

Prometheus for K8S monitoring adaptation work