This is the 13th day of my participation in Gwen Challenge.
Prometheus is known for monitoring microservices in the K8S cluster, but its unique form of data is derived from a number of sources. What are those sources? With cAdvisor, Heapster, and Metry-Server in place, you have access to almost all metrics for container operations, but you can’t do anything about this:
Scheduling N Replicas? There are now N available, right? N Pod is running/stopped/terminated status? Did the Pod restart N times? I have N jobs runningCopy the code
These are provided by Kube-state-Metrics, which is developed based on client-Go, polls the Kubernetes API and converts Kubernetes’ structured information to Metrics. Kube-state-metrics is an open-source plugin for Kubernetes.
Without further ado, go straight to the tutorial…
Deployment tutorial
download
- Release1.9.7 kube-state-metrics (v1.9.7) : kube-state-metrics
- CD/kube – state – metrics/examples/standard, you can see several files:
cluster-role-binding.yaml
cluster-role.yaml
deployment.yaml
prometheus-configmap.yaml
service-account.yaml
service.yaml
Copy the code
If Prometheus is already deployed in kube-system, the namespace in the source code does not need to be changed; otherwise, you can customize it to Monitoring.
update
- First modify service.yaml
apiVersion: v1 kind: Service metadata: annotations: prometheus.io/scrape: "true" labels: app.kubernetes.io/name: Kube - state - metrics app. Kubernetes. IO/version: v1.9.7 name: kube - state - metrics namespace: kube - system spec: clusterIP: None ports: - name: http-metrics port: 8080 targetPort: http-metrics - name: telemetry port: 8081 targetPort: telemetry selector: app.kubernetes.io/name: kube-state-metricsCopy the code
Very simple, added annotations for later use
Pit: The role authorization in the source code is bound to the resource whose KIND is ClusterRole written by it. However, it was later found that when kube-state-metrics service was deployed, it could not access the API-Server of K8S successfully, so it needed to be modified and its ClusterRole should be abandoned. Use cluster-admin, the highest permission on the K8S system.
- Changing access Rights
vi cluster-role-binding.yaml
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: labels: app.kubernetes.io/name: Kube - state - metrics app. Kubernetes. IO/version: v1.9.7 name: kube - state - metrics roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-admin #kube-state-metrics subjects: - kind: ServiceAccount name: kube-state-metrics namespace: kube-systemCopy the code
The deployment of
cd /kube-state-metrics/examples/standard
kubectl create -f .
Copy the code
At this point, Prometheus’s mounted configMap needs to be updated, as it was stated that only the endpoint with Prometheus. IO /scrape: “true” is captured
vi prometheus-configmap.yaml
apiVersion: v1 kind: ConfigMap metadata: name: prometheus-config namespace: kube-system data: prometheus.yaml: | global: scrape_interval: 15s evaluation_interval: 15s scrape_configs: - job_name: 'kubernetes-apiservers' kubernetes_sd_configs: - role: endpoints scheme: https tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: default; kubernetes; https - job_name: 'kubernetes-nodes' kubernetes_sd_configs: - role: node scheme: https tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - action: labelmap regex: __meta_kubernetes_node_label_(.+) - target_label: __address__ replacement: kubernetes.default.svc:443 - source_labels: [__meta_kubernetes_node_name] regex: (.+) target_label: __metrics_path__ replacement: /api/v1/nodes/${1}/proxy/metrics - job_name: 'kubernetes-cadvisor' kubernetes_sd_configs: - role: node scheme: https tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - action: labelmap regex: __meta_kubernetes_node_label_(.+) - target_label: __address__ replacement: kubernetes.default.svc:443 - source_labels: [__meta_kubernetes_node_name] regex: (.+) target_label: __metrics_path__ replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor - job_name: 'kubernetes-service-endpoints' kubernetes_sd_configs: - role: endpoints relabel_configs: - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape] action: keep regex: true - action: labelmap regex: __meta_kubernetes_service_label_(.+) - source_labels: [__meta_kubernetes_namespace] action: replace target_label: kubernetes_namespace - source_labels: [__meta_kubernetes_service_name] action: replace target_label: service_name - job_name: 'kubernetes-services' kubernetes_sd_configs: - role: service metrics_path: /probe params: module: [http_2xx] relabel_configs: - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe] action: keep regex: true - source_labels: [__address__] target_label: __param_target - target_label: __address__ replacement: blackbox-exporter.example.com:9115 - source_labels: [__param_target] target_label: instance - action: labelmap regex: __meta_kubernetes_service_label_(.+) - source_labels: [__meta_kubernetes_namespace] target_label: kubernetes_namespace - source_labels: [__meta_kubernetes_service_name] target_label: kubernetes_name - job_name: 'kubernetes-ingresses' kubernetes_sd_configs: - role: ingress relabel_configs: - source_labels: [__meta_kubernetes_ingress_annotation_prometheus_io_probe] action: keep regex: true - source_labels: [__meta_kubernetes_ingress_scheme,__address__,__meta_kubernetes_ingress_path] regex: (.+); (. +); (.+) replacement: ${1}://${2}${3} target_label: __param_target - target_label: __address__ replacement: blackbox-exporter.example.com:9115 - source_labels: [__param_target] target_label: instance - action: labelmap regex: __meta_kubernetes_ingress_label_(.+) - source_labels: [__meta_kubernetes_namespace] target_label: kubernetes_namespace - source_labels: [__meta_kubernetes_ingress_name] target_label: kubernetes_name - job_name: 'kubernetes-pods' kubernetes_sd_configs: - role: pod relabel_configs: - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] action: keep regex: true - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path] action: replace target_label: __metrics_path__ regex: (.+) - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port] action: replace regex: ([^:]+)(?::\d+)?; (\d+) replacement: $1:$2 target_label: __address__ - action: labelmap regex: __meta_kubernetes_pod_label_(.+) - source_labels: [__meta_kubernetes_namespace] action: replace target_label: kubernetes_namespace - source_labels: [__meta_kubernetes_pod_name] action: replace target_label: kubernetes_pod_nameCopy the code
After the ConfigMap is updated, restart Prometheus for it to take effect. If the ConfigMap is not deployed, create the ConfigMap and run the script to deploy it.
Import the template
Finally, download the state-metrics monitoring template import template from grafana.com
After importing grafana, you can see the effect:
