Docker-compose quickly built Prometheus+Grafana monitoring system

A, description,

Prometheus collected the data, and Grafana presented it. Which uses the Exporter from Prometheus:



1)Node Exporter is responsible for collecting host hardware and operating system data. It will run as a container on all hosts.



2)CAdvisor, responsible for collecting container data. It will run as a container on all hosts.



3)Alertmanager, responsible for alarms. It will run as a container on all hosts.

Docker-compose: docker-compose

2.1 installation docker

Start by installing a 64-bit Linux host with a kernel higher than 3.10 and at least 1GB of memory. Install Docker on this host.

Install dependencies
yum install -y yum-utils device-mapper-persistent-data lvm2
# Add Docker package source
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
Install Docker CE
yum install docker-ce -y
# start
systemctl start docker
# Boot
systemctl enable docker
# Check Docker information
docker infoCopy the code

2.2 installation docker – compose

The curl -l https://github.com/docker/compose/releases/download/1.23.2/docker-compose- ` uname-s`-`uname -m` -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-composeCopy the code

Add a configuration file

mkdir -p /usr/local/src/config
cd /usr/local/src/configCopy the code

2.1 Adding the Promethes. yml configuration file

vim prometheus.yml

# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets: ['192.168.159.129:9093']
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  - "node_down.yml"
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'
    static_configs:
- targets: ['192.168.159.129:9090']

  - job_name: 'cadvisor'
    static_configs:
    - targets: ['192.168.159.129:8080']

  - job_name: 'node'
    scrape_interval: 8s
    static_configs:
      - targets: ['192.168.159.129:9100']Copy the code

2.2 Adding an Email Alarm Configuration File

Add the alertManager. yml configuration file to configure the mailbox for sending and receiving emails


vim alertmanager.yml

global:
  smtp_smarthost: 'smtp.163.com:25'　　# 163 server
  smtp_from: '[email protected]'　　　　　　　　# Email address
  smtp_auth_username: '[email protected]'　　This is your email address
  smtp_auth_password: 'TPP***'　　　　　　　　# Email password
  smtp_require_tls: false　　　　　　　　TLS authentication is not enabled

route:
  group_by: ['alertname']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 10m
  receiver: live-monitoring

receivers:
- name: 'live-monitoring'
  email_configs:
  - to: '[email protected]'　　　　　　　　# Email address to receive emailCopy the code

2.3 Adding alarm Rules

Add node_down.yml for Prometheus targets monitoring


vim node_down.yml

groups:
- name: node_down
  rules:
  - alert: InstanceDown
    expr: up == 0
    for: 1m
    labels:
      user: test
    annotations:
      summary: "Instance {{ $labels.instance }} down"
      description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 1 minutes."Copy the code

Docker-compose

vim docker-compose-monitor.yml

version: '2'

networks:
    monitor:
        driver: bridge

services:
    prometheus:
        image: prom/prometheus
        container_name: prometheus
        hostname: prometheus
        restart: always
        volumes:
            - /usr/local/src/config/prometheus.yml:/etc/prometheus/prometheus.yml
            - /usr/local/src/config/node_down.yml:/etc/prometheus/node_down.yml
        ports:
            - "9090:9090"
        networks:
            - monitor

    alertmanager:
        image: prom/alertmanager
        container_name: alertmanager
        hostname: alertmanager
        restart: always
        volumes:
            - /usr/local/src/config/alertmanager.yml:/etc/alertmanager/alertmanager.yml
        ports:
            - "9093:9093"
        networks:
            - monitor

    grafana:
        image: grafana/grafana
        container_name: grafana
        hostname: grafana
        restart: always
        ports:
            - "3000:3000"
        networks:
            - monitor

    node-exporter:
        image: quay.io/prometheus/node-exporter
        container_name: node-exporter
        hostname: node-exporter
        restart: always
        ports:
            - "9100:9100"
        networks:
            - monitor

    cadvisor:
        image: google/cadvisor:latest
        container_name: cadvisor
        hostname: cadvisor
        restart: always
        volumes:
            - /:/rootfs:ro
            - /var/run:/var/run:rw
            - /sys:/sys:ro
            - /var/lib/docker/:/var/lib/docker:ro
        ports:
            - "8080:8080"
        networks:
            - monitorCopy the code

5. Start docker-compose

# start container:
docker-compose -f /usr/local/src/config/docker-compose-monitor.yml up -d
# delete container:
docker-compose -f /usr/local/src/config/docker-compose-monitor.yml down
# restart container:
docker restart idCopy the code

The container starts as follows:

Prometheus Targets interface is as follows:

Note: If State is Down, the firewall is faulty. For details, see firewall configuration.

The Prometheus Graph interface is as follows:

Note: If no data is available, synchronize the time.

Six, firewall configuration

6.1 close the selinux

setenforce 0
vim /etc/sysconfig/selinuxCopy the code

6.2 configure iptables

# Delete the built-in firewall
systemctl stop firewalld.service
systemctl disable firewalld.serviceCopy the code

# iptables installation
yum install -y iptables-servicesCopy the code

# configurationvim /etc/sysconfig/iptables *filter :INPUT ACCEPT [0:0] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [24:11326] -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT -A INPUT -p icmp -j ACCEPT -A INPUT -i lo -j ACCEPT -A INPUT -p tcp -m state  --state NEW -m tcp --dport 22 -j ACCEPT -A INPUT -p tcp -m state --state NEW -m tcp --dport 9090 -j ACCEPT -A INPUT -p tcp -m state --state NEW -m tcp --dport 8080 -j ACCEPT -A INPUT -p tcp -m state --state NEW -m tcp --dport 3000 -j ACCEPT -A INPUT -p tcp -m state --state NEW -m tcp --dport 9093 -j ACCEPT -A INPUT -p tcp -m state --state NEW -m tcp --dport 9100 -j ACCEPT -A INPUT -j REJECT --reject-with icmp-host-prohibited -A FORWARD -j REJECT --reject-with icmp-host-prohibited COMMITCopy the code

# start
systemctl restart iptables.service
systemctl enable iptables.serviceCopy the code

Configure Grafana

7.1 Adding a Prometheus Data source

7.2 configuration dashboards

Description: can use the built-in template, can also go to https://grafana.com/dashboards and download the corresponding template.

7.3 Viewing Data

I downloaded the Docker-related template from the website: Docker and System Monitoring, 893






Type 893 and the following information is loaded

After importing, go to the home page to view the data

Appendix: Separate command to start each container

# start Prometheus
docker run -d -p 9090:9090 --name=prometheus \
-v /usr/local/src/config/prometheus.yml:/etc/prometheus/prometheus.yml \
-v /usr/local/src/config/node_down.yml:/etc/prometheus/node_down.yml \
prom/prometheus

# start grafana
docker run -d -p 3000:3000 --name=grafana grafana/grafana

Start the AlertManager container
docker run -d -p 9093:9093 -v /usr/local/src/config/config.yml:/etc/alertmanager/config.yml --name alertmanager prom/alertmanager

# Start Node Exporter
docker run -d \
  -p 9100:9100 \
  -v "/:/host:ro,rslave" \
  --name=node_exporter \
  quay.io/prometheus/node-exporter \
  --path.rootfs /host

# start cadvisor
docker run                                    \
--volume=/:/rootfs:ro                         \
--volume=/var/run:/var/run:rw                 \
--volume=/sys:/sys:ro                         \
--volume=/var/lib/docker/:/var/lib/docker:ro  \
--publish=8080:8080                           \
--detach=true                                 \
--name=cadvisor                               \
google/cadvisor:latestCopy the code