preface

A few years ago, telegraf+ Infludb +Grafana was deployed for performance testing and monitoring, which was convenient to view and archive performance monitoring data. With the popularity of K8s, Prometheus began to be favored by various major factories. Recently, Prometheus+Grafana was deployed. With a variety of exports, rich performance test monitoring means.

Prometheus profile

concept

Prometheus is an open source monitoring alarm system and time sequence database (TSDB) developed by SoundCloud. Prometheus, developed in the Go language, is an open source version of Google’s BorgMon monitoring system.
In 2016, Google launched the Linux Foundation’s Cloud Native Computing Foundation, which included Prometheus as its second open source project.
Prometheus is currently active in the open source community.
Prometheus and Heapster(Heapster is a subproject of K8S for obtaining cluster performance data.) Compared with the function more perfect, more comprehensive. Prometheus is also capable of supporting tens of thousands of clusters.

Official website: Prometheus. IO /

The characteristics of

Multidimensional data model
Efficient and flexible query statements
Individual server nodes are independent of distributed storage
Time series data is collected in PULL mode based on HTTP
Temporal sequence data can be pushed through an intermediate gateway
Discover target service objects through service discovery or static configuration
Supports a wide variety of charts and interfaces, such as Grafana

component

Promethues Server: Obtains and stores time series data
Exporters: Mainly collects data as the agent and sends it to Prometheus Server. Exporters are responsible for various data collection activities, such as Node-Exporters and Redis.
- For more exporters see Exporters AND INTEGRATIONS
- Corresponding port number Default port Allocations · Prometheus/Prometheus Wiki · GitHub
Pushgateway: allows ephemeral and batch jobs to push their data to Prometheus; Since this type of work does not exist for very long, it requires them to actively push data to PushGateway, which then sends it to Prometheus.
Alertmanager: provides the alarm function of Prometheus.

architecture

Grafana

Grafana is a cross-platform open source metric analysis and visualization tool that enables you to query and visually present collected data with timely notifications.

The characteristics of

Display mode: fast and flexible client chart, panel plug-in has many different ways of visualization indicators and logs, the official library has rich dashboard plug-in, such as heat map, line chart, chart and other display modes;
Data sources: Graphite, InfluxDB, OpenTSDB, Prometheus, Elasticsearch, CloudWatch, KairosDB, etc.
Notification alerts: Visually define alert rules for the most important metrics. Grafana will continuously calculate and send notifications, and get notifications through Slack, PagerDuty, etc., when the data reaches a threshold;
Mixed presentation: Mix different data sources in the same chart, you can specify data sources based on each query, or even customize data sources;
Annotation: Using rich event annotation charts from different data sources, hovering over events shows the full event metadata and markup;
Filters: Ad-hoc filters allow the dynamic creation of new key/value filters that are automatically applied to all queries that use the data source

Install the deployment

prometheus

Download it manually and start the service according to the YML configuration file

download the latest release

wget https://github.com/prometheus/prometheus/releases/download/v*/prometheus-*.*-amd64.tar.gz
tar xvf prometheus-*.*-amd64.tar.gz
cd prometheus-*
nohup ./prometheus --config.file=./prometheus.yml &
Copy the code

grafana

Download it manually and start the service according to the YML configuration file

download the latest release

Wget https://dl.grafana.com/oss/release/grafana-7.5.6-1.x86_64.rpm sudo yum install grafana 7.5.6-1. X86_64. RPMCopy the code

Docker installation

Docker-compose installs Prometheus and Grafana

cd /opt
mkdir -p prometheus/config/
mkdir -p grafana/data
chmod 777 grafana/data
mkdir -p /data/prometheus
chmod 777 /data/prometheus
Copy the code

createprometheus.ymlThe configuration file

cd /opt/prometheus/config/
touch prometheus.yml
Copy the code

The editorprometheus.ymlThe configuration file

#my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  #scrape_timeout is set to the global default (10s).

#Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      #- alertmanager:9093

#Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  #- "first_rules.yml"
  #- "second_rules.yml"

#A scrape configuration containing exactly one endpoint to scrape:
#Here it's Prometheus itself.
scrape_configs:
  #The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    #metrics_path defaults to '/metrics'
    #scheme defaults to 'http'.

    static_configs:
    - targets: ['192.168.9.140:9090']


  - job_name: "node"
    static_configs:
    - targets: ["192.168.9.140:9100"]

  - job_name: "qianmingyanqian"
    static_configs:
    - targets: ["11.12.108.226:9100"."11.12.108.225:9100"]

  ## config for the multiple Redis targets that the exporter will scrape
  - job_name: "redis_exporter_targets"
    scrape_interval: 5s
    static_configs:
      - targets:
        - Redis: / / 192.168.9.140:6379
        - Redis: / / 192.168.9.140:7001
        - Redis: / / 192.168.9.140:7004
    metrics_path: /scrape
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 192.1689.140.: 9121
Copy the code

createdocker-compose_prometheus_grafana.ymlThe configuration file

cd /opt
mkdir docker-compose
touch docker-compose_prometheus_grafana.yml
Copy the code

The editordocker-compose_prometheus_grafana.ymlFile and type

version: '2'

networks:
  monitor:
    driver: bridge

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    hostname: prometheus
    restart: always
    volumes:
      - /opt/prometheus/config:/etc/prometheus
      - /data/prometheus:/prometheus
    ports:
      - "9090:9090"
    expose:
      - "8086"
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--log.level=info'
      - '-- web. Listen - address = 0.0.0.0:9090'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention=15d'
      - '--query.max-concurrency=50'
    networks:
      - monitor

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    hostname: grafana
    restart: always
    volumes:
      - /opt/grafana/data:/var/lib/grafana
    ports:
      - "3000:3000"
      - "For"
    networks:
      - monitor
    depends_on:
      - prometheus
Copy the code

Docker-compose runs the Docker container

docker-compose -p prometheus_grafana -f docker-compose_prometheus_grafana.yml up -d
Copy the code

Access the Prometheus service through port 9090 of the browser

Access the Grafana service through port 3000 of the browser

Monitoring server Nodes

Install and run node_exporter

Download and install Node_EXPORTER on the server to be monitored

download the latest release

wget https://github.com/prometheus/node_exporter/releases/download/v*/node_exporter-*.*-amd64.tar.gztar xvfz node_exporter-*.*-amd64.tar.gzcd node_exporter-*.*-amd64nohup ./node_exporter &
Copy the code

Or use Docker to install and createdocker-compose_node-exporter.ymlThe configuration file

cd /opt/docker-composetouch docker-compose_node-exporter.yml
Copy the code

The editordocker-compose_node-exporter.ymlThe configuration file is as follows:

---version: '3.8'services:  node_exporter:    image: quay.io/prometheus/node-exporter:latest    container_name: node_exporter    command:      - '--path.rootfs=/host'    network_mode: host    pid: host    restart: unless-stopped    volumes:      - '/:/host:ro,rslave'
Copy the code

Docker-compose runs the Docker container

docker-compose -p node_exporter -f docker-compose_node-exporter.yml up -d
Copy the code

Try accessing the following

The curl http://192.168.9.140:9100/metricsCopy the code

If the following data is displayed, the service is normal

# HELP node_xfs_read_calls_total Number of read(2) system calls made to files in a filesystem.# TYPE node_xfs_read_calls_total counternode_xfs_read_calls_total{device="dm-1"} 10196node_xfs_read_calls_total{device="dm-2"} 17401node_xfs_read_calls_total{device="dm-3"} 970node_xfs_read_calls_total{device="dm-4"} 10node_xfs_read_calls_total{device="dm-5"} 19node_xfs_read_calls_total{device="dm-6"} 132node_xfs_read_calls_total{device="sda2"} 16378node_xfs_read_calls_total{device="sda3"} 2.67817784 e+09 node_xfs_read_calls_total 1.053587 e+06 {device = "sda6"}
Copy the code

Configuration Prometheus

prometheus.ymlAppend the following

  - job_name: "node"    static_configs:    - targets: ["192.168.9.140:9100"]
Copy the code

After modifying the configuration file, restart the systemprometheusThe container

docker restart CONTAINER ID
Copy the code

After success, see the following figure

Configuration Grafana

Add data source after login

Choose the Prometheus

Type in the URL fieldhttp://192.168.9.140:9090, as shown in the figure, and save

Add Dashboards

Import the templatehttps://grafana.com/grafana/dashboards/1860And the load succeeds

More Template References

On the detailed monitoring page, you can select a monitoring period as required

Monitor the redis

Install and run redis_exporter

Download and install Redis_EXPORTER on the Redis server to be monitored

download the latest release

Wget HTTP: / / https://github.com/oliver006/redis_exporter/releases/download/v1.23.1/redis_exporter-v1.23.1.linux-386.tar.gztar ZXVF redis_tuppy-v1.23.1.linux-386.tar.gznohup./ redis_tuppy-redisis.addr 192.168.9.140:6379-redis.password 111111 &Copy the code

Or install using Docker

docker run-d --name redis_exporter -P 9121:9121 oliver006/redis_exporter --redis.addr=192.168.9.140:6379 --redis. Password =111111
Copy the code

Configuration Prometheus

Redis single instance,prometheus.ymlAppend the following

## config for scraping the exporter itself- job_name: 'redis_exporter'  scrape_interval: 5s  static_configs:    - targets:[192.168.9.140:9121]
Copy the code

Redis cluster,prometheus.ymlAppend the following

  ## config for the multiple Redis targets that the exporter will scrape - job_name: "redis_exporter_targets" scrape_interval: 5s static_configs: - targets: - redis: / / 192.168.9.140:6379 - redis: / / 192.168.9.140:7001 - redis: / / 192.168.9.140:7004 metrics_path: /scrape relabel_configs: - source_labels: [__address__] target_label: __param_target - source_labels: [__param_target] target_label: instance-target_label: __address__ replacement: 192.168.9.140:9121
Copy the code

After modifying the configuration file, restart the systemprometheusThe container

docker restart CONTAINER ID
Copy the code

After success, see the following figure

Configuration Grafana

Add Dashboards and import templateshttps://grafana.com/grafana/dashboards/11835And the load succeeds

On the detailed monitoring page, you can select a monitoring period as required

Monitoring the mysql

Install and run mysqLD_exporter

First, authorize the user

root@localhost 14:43:  [(none)]>CREATE USER 'exporter'@'localhost' IDENTIFIED BY 'mysql_exporter'; Query OK,0 rows affected (0.04 sec)root@localhost 14:43:  [(none)]>GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'exporter'@'localhost'; Query OK,0 rows affected (0.03 sec)
Copy the code

createmy.cnfThe configuration file

cd /opttouch .my.cnfvim .my.cnf
Copy the code

Configuration file Type the following

[ client ]user = exporterpassword = mysql_exporter
Copy the code

Download and install mysqLD_EXPORTER on the mysql server to be monitored

download the latest release

wget https://github.com/prometheus/mysqld_exporter/releases/download/v*/mysqld_exporter-*.*-amd64.tar.gztar xvfz mysqld_exporter-*.*-amd64.tar.gzcd mysqld_exporter-*.*-amd64nohup ./mysqld_exporter --config.my-cnf=/opt/.my.cnf &
Copy the code

Or use Docker to install and createdocker-compose_mysqld-exporter.ymlThe configuration file

cd /opt/docker-composetouch docker-compose_mysqld-exporter.yml
Copy the code

The editordocker-compose_mysqld-exporter.ymlThe configuration file is as follows:

version: '2'networks:    monitor:        driver: bridgeservices:    mysql-exporter:        image: prom/mysqld-exporter        container_name: mysql-exporter        hostname: mysql-exporter        restart: always        ports:            - "9104:9104"        networks:            - my-mysql-network        environment:            DATA_SOURCE_NAME: "The exporter: mysql_exporter @ (192.168.9.140:3306)/"networks:    my-mysql-network:        driver: bridge
Copy the code

Docker-compose runs the Docker container

docker-compose -p mysql_exporter -f docker-compose_mysqld-exporter.yml up -d
Copy the code

Configuration Prometheus

prometheus.ymlAppend the following

- job_name: 'mysql'     static_configs: - targets: ['192.168.9.140:9104']         labels: instance: mysql
Copy the code

After modifying the configuration file, restart the systemprometheusThe container

docker restart CONTAINER ID
Copy the code

After success, see the following figure

Configuration Grafana

Add Dashboards and import templateshttps://grafana.com/grafana/dashboards/11323And the load succeeds

On the detailed monitoring page, you can select a monitoring period as required

If you just want to monitor MySQL or MongoDB, consider the Percona Monitoring and Management (PMM) tool, which adds additional features such as slow query collection

The problem to repair

Prometheus

Symptom The following error information is displayed when the Prometheus service is started and the browser accesses port 9090

Warning: Error fetching server time: Detected 785.6099998950958 seconds time difference between your browser and the server.
Copy the code

Go to the Prometheus machine and synchronize the clock

ntpdate time3.aliyun.com
Copy the code

redis_exporter

Too many Redis Instances may be displayed if Prometheus. Yml is incorrectly configured
After being started by redis_exporter, a major error may occur on the Targets web page of the Prometheus service. In this case, the password configuration is incorrect when redis_exporter is running. For example, some Instances of Redis do not require a password to be forcibly configured. Or the Redis instance requires a password and is not configured

- redis_exporter_last_scrape_error{err="dial redis: unknown network redis"1}Copy the code

mysqld_exporter

The following error may be displayed when creating a mysql user

ERROR 1290 (HY000): The MySQL server is running with the --skip-grant-tables option so it cannot execute this statement
Copy the code

Refresh the permission and run the command again

flush privileges;
Copy the code

Afterword.

Besides the Linux server, Redis, Mysql, etc., you can also monitor the corresponding components through other exporters. The tool is not a panacea, and it is important to be able to quickly locate performance bottlenecks.

mo4tech.com (Moment For Technology) is a global community with thousands techies from across the global hang out!Passionate technologists, be it gadget freaks, tech enthusiasts, coders, technopreneurs, or CIOs, you would find them all here.

Prometheus&Grafana performance monitoring

preface

Prometheus profile

concept

The characteristics of

component

architecture

Grafana

The characteristics of

Install the deployment

prometheus

grafana

Docker installation

Monitoring server Nodes

Install and run node_exporter

Configuration Prometheus

Configuration Grafana

Monitor the redis

Install and run redis_exporter

Configuration Prometheus

Configuration Grafana

Monitoring the mysql

Install and run mysqLD_exporter

Configuration Prometheus

Configuration Grafana

The problem to repair

Prometheus

redis_exporter

mysqld_exporter

Afterword.

Prometheus&Grafana performance monitoring

preface

Prometheus profile

concept

The characteristics of

component

architecture

Grafana

The characteristics of

Install the deployment

prometheus

grafana

Docker installation

Monitoring server Nodes

Install and run node_exporter

Configuration Prometheus

Configuration Grafana

Monitor the redis

Install and run redis_exporter

Configuration Prometheus

Configuration Grafana

Monitoring the mysql

Install and run mysqLD_exporter

Configuration Prometheus

Configuration Grafana

The problem to repair

Prometheus

redis_exporter

mysqld_exporter

Afterword.

Related Posts

Go Modules step pit summary

Apollo-client initialization process parsing

Part23: Golang buffer channel and coroutine pool