How to choose the right monitoring platform

  • 1. How many monitoring types are supported (database, operating system, middleware, developed application monitoring (Java, PHP,c#), etc.)?
  • 2. Rich monitoring report + custom monitoring report + warning notification?
  • 3. Extensible custom monitoring plug-in definition?

Monitoring platform construction

Prometheus use

Prometheus: Collects and stores various types of data in the form of time series.

architecture

Install the configuration

  • 1. Download
Wget HTTP: / / https://github.com/prometheus/prometheus/releases/download/v2.13.0/prometheus-2.13.0.linux-amd64.tar.gzCopy the code
  • 2. The configuration
Gz CD Prometheus -2.13.0. Linux-amd64.tar. gz CD Prometheus -2.13.0. Linux-amd64 / Prometheus --config.file=' Prometheus. Yml '-- Start configuration access address :http://localhost:9090Copy the code
  • 2.1 Configuration File Details

Prometheus. Yml reference

Rule_files: #job Scrape_configs: # task name - job_name: Scrape_timeout :/metrics (default :/metrics) metrics_path: Scheme: # HTTP Authorization basic_auth: username: password: password_file: # static_configs: # target host-targets :[] # tag configuration -labels :[]Copy the code

Grafana use

Grafana: Open source analysis and monitoring system

architecture

Use (Linux)

  • 1. Install
Wget https://dl.grafana.com/oss/release/grafana-6.4.2.linux-amd64.tar.gz tar - ZXVF grafana - 6.4.2. Linux - amd64. Tar. GzCopy the code
  • 2. Run
CD grafana - 6.4.2. Linux - amd64. / bin/grafana - server http://localhost:3000Copy the code
  • 2.1 Configuration Description
Defaults. Ini: the default configuration ustom. Ini or grafana. Ini: custom configuration -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- paths: store related path server: the server configuration Database: storage database configuration remote_cache: cache configuration Security: security configuration users: user configuration dashboards: panel configuration SMTP: mailbox configuration log: log configuration Alerting: alarm configurationCopy the code

Monitoring platform construction

Prometheus early warning use

Early warning Provides reports on monitoring servers, applications, and data for final monitoring purposes

For details, see AlertManager alarm notification configurations

  • 1. Usage instructions for alertManager. yml parameters
# configure the notification template: Receivers are configured to send messages to whom at what point in time and in what way route: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- reference: global: resolve_timeout: 5 m smtp_smarthost: 'smtp.exmail.qq.com:465' # email SMTP server agent smtp_from: 'XXXXX' # Smtp_require_tls: false # disable TLS route: group_by: [' alertName '] group_wait: 10s group_interval: 10s REPEAT_interval: 1h Receiver: 'forEmail' receivers: # 'forEmail' email_configs: - to: 'xxxx' send_resolved: true inhibit_rules: - source_match: severity: 'critical' target_match: severity: 'warning' equal: ['alertname', 'dev', 'instance']Copy the code

Early Warning Collection configuration

Propmetheus enable alert configuration alerting: AlertManagers: - static_configs: -targets: ["localhost:9093"] 2. Rule_files: - "/data/monitor/ Prometheus -2.13.0. Linux -amd64/rule/*. Rules "groups: - name: # -alert: expr: # define the rule condition using the PromQL expression # Reach, or how long it takes to trigger the rule for: Labels: Severity: team: # Summary: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- reference: server downtime monitoring groups: - name: node - up rules: - alert: node - up expr: Up {job="node"} == 0 for: 15s labels: Severity: 1 team: node annotations: summary: "{{$labels. Instance}} has stopped running for more than 15s!"Copy the code

Prometheus common exporters use

Java Applications (Spring Boot)
  • 1. On the Spring Boot application end, invoke dependencies and configure them
<dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-actuator</artifactId> </dependency> <dependency> <groupId>io.micrometer</groupId> <artifactId>micrometer-registry-prometheus</artifactId> <version>1.1.3</version> </dependency> @bean MeterRegistryCustomizer<MeterRegistry> Configurer ( @Value("${spring.application.name}") String applicationName) { return (registry) -> registry.config().commonTags("application", applicationName); } management: endpoints: web: exposure: include: '*' metrics: tags: application: ${spring.application.name}Copy the code
  • 2. Configure the Prometheus collection terminal
- job_name: 'springboot_prometheus' scrape_interval: 5s metrics_PATH: '/actuator/ Prometheus' -Targets: ['192.168.0.106:8080','192.168.0.106:8081'] # Configure multiple applicationsCopy the code
Server Monitoring (Linux/ Windows)
1. Download the wget https://github.com/prometheus/node_exporter/releases/download/v0.18.1/node_exporter-0.18.1.linux-386.tar.gz 2. Start tar -zxvf node_extude-0.18.1.linux-386.tar.gz./ node_extude-3. - job_name: 'node' static_configs: -targets: ['192.168.0.106:9100']Copy the code
The database
MongoDB Monitoring configuration [TBD]
https://github.com/percona/mongodb_exporter/releases https://devconnected.com/mongodb-monitoring-with-grafana-prometheus/#b_Installing_the_MongoDB_exporter 1. Download the wget HTTP: / / https://github.com/percona/mongodb_exporter/releases/download/v0.10.0/mongodb_exporter-0.10.0.linux-amd64.tar.gz  2. -job_name: 'mongoDb' static_configs: -targets: ['192.168.0.106:9001']Copy the code
Redis
1. Download and configure https://github.com/oliver006/redis_exporter 2. Monitoring configuration 3 at https://grafana.com/grafana/dashboards/763. -job_name: 'Redis' static_configs: -targets: ['192.168.0.106:9121']Copy the code
Mysql
1. Download the https://github.com/prometheus/mysqld_exporter 1.1 introducing the mysql database creation to collect user and authorization: CREATE USER 'exporter'@'localhost' IDENTIFIED BY 'exporter' WITH MAX_USER_CONNECTIONS 3; GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'exporter'@'localhost'; [client] host=192.168.0.102 #mysql server IP address port=3306 user=exporter# / mysqLD_exports --config.my-cnf=/ In the specified directory /my.conf. 2. Dashboard configuration https://grafana.com/grafana/dashboards/7362 https://grafana.com/grafana/dashboards/6239 3. -job_name: 'Mysql' static_configs: -targets: ['192.168.0.106:9104']Copy the code
PostgreSQL
# 1. Postgresql collector download WGET https://github.com/prometheus-community/postgres_exporter/releases/download/v0.9.0/postgres_exporter-0.9.0.linux-amd64.t Ar.gz # 2. Postgresql database configuration CREATE USER postgres_exporter PASSWORD '123456'; ALTER USER postgres_exporter SET SEARCH_PATH TO postgres_exporter,pg_catalog; CREATE SCHEMA postgres_exporter AUTHORIZATION postgres_exporter; CREATE FUNCTION postgres_exporter.f_select_pg_stat_activity() RETURNS setof pg_catalog.pg_stat_activity LANGUAGE sql SECURITY DEFINER AS $$ SELECT * from pg_catalog.pg_stat_activity; $$; CREATE FUNCTION postgres_exporter.f_select_pg_stat_replication() RETURNS setof pg_catalog.pg_stat_replication LANGUAGE sql SECURITY DEFINER AS $$ SELECT * from pg_catalog.pg_stat_replication; $$; CREATE VIEW postgres_exporter.pg_stat_replication AS SELECT * FROM postgres_exporter.f_select_pg_stat_replication(); CREATE VIEW postgres_exporter.pg_stat_activity AS SELECT * FROM postgres_exporter.f_select_pg_stat_activity(); GRANT SELECT ON postgres_exporter.pg_stat_replication TO postgres_exporter; GRANT SELECT ON postgres_exporter.pg_stat_activity TO postgres_exporter; # pg_export.sh # pg_export.sh # pg_export.sh # pg_export.sh /bin/sh export DATA_SOURCE_NAME="user=postgres_exporter host=127.0.0.1 password=123456 port=5432 dbname=test_monitor /postgres_exporter # 4. Prometheus job configuration -job_name: 'postgres' static_configs: -targets: [' 192.168.3.32:9187] http://192.168.3.32:9187/metrics # to check the data # 5. Postgres_exporter custom query configuration instructions: 1. Modified based on postgres_EXPORTER source code, add query statement 2. Source code package compile upload and update queries. Yaml fileCopy the code
Message middleware
  • RabbitMq
Premise: install the RabbitMq (if it is a test environment, use the docker simplified operation) 1 at https://www.cnblogs.com/yufeng218/p/9452621.html. Download and configure https://github.com/kbudde/rabbitmq_exporter 1.1-1.2 through the environment variables through a configuration file (more parameters reference documentation) such as: RABBIT_USER = admin RABBIT_PASSWORD = admin RABBIT_URL SKIP_QUEUES = = http://192.168.0.105:15672 "RPC_. *" MAX_QUEUES = 5000. / rabbitmq_exporter 2. Monitor the template 2 at https://grafana.com/grafana/dashboards/4279. -job_name: 'Redis' static_configs: -targets: ['192.168.0.106:9419']Copy the code

Other Middleware

  • docker
Copy the code

Application server

  • Nginx
1. Install nginx module - VTS provides information about collecting nginx (nginx installed) on the basis of source/monitor premise https://github.com/vozlt/nginx-module-vts nginx. HTTP {that is configured in the conf vhost_traffic_status_zone; server { location /status { vhost_traffic_status_display; vhost_traffic_status_display_format html; }} check whether /status/format/json can access data. If yes, the configuration is successful. 2. Download the nginx - VTS - exporter and monitor configuration reference: https://hnlq715.github.io/nginx-vts-exporter/ / nginx - VTS - exporter -- help - > check the detailed parameters . / nginx - VTS - exporter - nginx. Scrape_uri = http://localhost/status/format/json nginx. Scrape_uri: Indicator path (based on the access path configured in nginx.config) 3. -job_name: 'nginx' static_configs: -targets: ['192.168.0.106:9913']Copy the code