This post celebrates the release of Node Exporter to V1.0.0.

Prometheus is the first open source monitoring and alarm solution from SoundCloud. And has grown into the second CNCF graduate program after Kubernetes. Prometheus also made significant strides in monitoring with the adoption of cloud native concepts and the development of technologies such as Kubernetes.

Major components include Prometheus, Alertmanager, Node Exporter, Blackbox Exporter, and Pushgateway.

This article is a celebration of Node Exporter’s release of V1.0.0, so the focus is on the notoriously security-related aspects of TLS and Basic Auth.

background

Node Exporter is an official release by Prometheus, and is used to collect Node system information, such as CPU, memory, disk, and network information. Normally, if we were using Prometheus as a monitoring solution, Node Would use almost all of them.

In Promethues’ monitoring system, one of the persistent arguments in the community is that Metrics don’t contain too personal information. So as you can see, most of the /metrics interfaces are directly exposed, with no special security measures.

However, with the extensive use of Prometheus in production, safety issues became more important.

The first thought was to enable TLS for the connection between Prometheus and the monitoring target. However, since each type of exporter does not support TLS connection natively, we usually choose to cooperate with a reverse proxy.

This method meets the requirements, but it is a bit complicated. Prometheus recently revised its security model to include SUPPORT for TLS and Basic Auth from Node, which runs at least half of its components, as well as a list of the latest security benchmarks (TLS V1.2 and above are supported by default).

The use of TLS

Here’s a hands-on look at how to enable TLS.

To prepare the certificate

(MoeLove) ➜ ~ mkdir -p Prometheus - TLS (MoeLove) ➜ ~cdPrometheus - TLS (MoeLove) ➜ Prometheus - TLS openssl req-new-newkey RSA :2048 -days 365 -nodes -x509 -keyout node_exporter.key -out node_exporter.crt -subj"/C=CN/ST=Beijing/L=Beijing/O=Moelove.info/CN=localhost"Generating a RSA private key ........................................... + + + + +.. +++++ writing new private key to'node_exporter.key'----- (MoeLove) ➜ Prometheus - TLS node_export. CRT node_export.keyCopy the code

By following the above steps, we get the node_classies.crt and node_classies.key files.

Node Exporter uses TLS

Download v1.0.0 Node Exporter and decompress it

(MoeLove) ➜ / TMP tar -zxvf node_extude-1.0.0. linux-amd64.tar.gz node_extude-1.0.0. linux-amd64/ Node_exporter - 1.0.0. Linux - amd64 / node_exporter node_exporter - 1.0.0. Linux - amd64 / NOTICE Node_exporter - 1.0.0. Linux - amd64 / LICENSE (MoeLove) ➜ / TMPcdLinux-amd64 (MoeLove) ➜ node_exporters -1.0.0. Linux-amd64 LS LICENSE node_exporters NOTICECopy the code

Copy the generated node_classie. CRT and node_classie. key files to the current directory.

Linux - AMd64 cp ~/ Prometheus - TLS /node_exporter.*. (MoeLove) ➜ Linux-amd64 LS LICENSE node_exporters node_exporters. CRT node_exporters. Key NOTICECopy the code

Write a configuration file and save it as config.yaml (optionally named) :

tls_server_config:
  cert_file: node_exporter.crt
  key_file: node_exporter.key
Copy the code

Next, use –web.config to pass the configuration file to Node Exporter

(MoeLove) ➜ node_exporters -1.0.0.linux-amd64./ node_exporters --web.config=config.yaml level=info Ts = 2020-05-26 T17: hindermost. 123 zcaller=node_exporter.go:177 msg="Starting node_exporter" version=(version = "1.0.0, branch = HEAD, revision = b9c96706a7425383902b6143d097cf6d7cfd1960)"Level = info ts = 2020-05-26 T17: hindermost. 124 zcaller=node_exporter.go:178 msg="Build context" build_context="(go = go1.14.3, user = root @ 3 e55cc20ccc0, date = 20200526-06:01:48)"Level = info ts = 2020-05-26 T17: hindermost. 130 zcaller=node_exporter.go:105 msg="Enabled collectors". Level = info ts = 2020-05-26 T17: hindermost. 135 zcaller=tls_config.go:200 msg="TLS is enabled and it cannot be disabled on the fly." http2=true
Copy the code

When you see this last log, your Node Exporter has ENABLED A TLS connection.

Of course, we can also choose to manually verify:

# direct curl request(MoeLove) ➜ Prometheus - TLS curl localhost:9100/metrics Client sent an HTTP request to an HTTPS server. (MoeLove) ➜ prometheus-tls curl https://localhost:9100/metrics curl: (60) SSL certificate problem: self signed certificate More details here: https://curl.haxx.se/docs/sslcerts.html curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above.Copy the code

Curl: $curl: $curl: $curl: $curl: $curl: $curl: $curl: $curl: $curl: $curl: $curl: $curl: $curl: $curl: $curl: $curl: $curl: $curl: $curl: $curl

The curl (MoeLove) ➜ Prometheus - TLS-s  --cacert node_exporter.crt https://localhost:9100/metrics  |grep node_exporter_build_info
# HELP node_exporter_build_info A metric with a constant '1' value labeled by version, revision, branch, and goversion from which node_exporter was built.
# TYPE node_exporter_build_info gauge
node_exporter_build_info{branch="HEAD",goversion="go1.14.3",revision="b9c96706a7425383902b6143d097cf6d7cfd1960",version="1.0.0"1}Copy the code

Of course, in addition to passing the certificate to curl with the –cacert argument, you can also ignore the certificate check with the -k argument.

The curl (MoeLove) ➜ Prometheus - TLS-s  -k https://localhost:9100/metrics  |grep node_exporter_build_info        
# HELP node_exporter_build_info A metric with a constant '1' value labeled by version, revision, branch, and goversion from which node_exporter was built.
# TYPE node_exporter_build_info gauge
node_exporter_build_info{branch="HEAD",goversion="go1.14.3",revision="b9c96706a7425383902b6143d097cf6d7cfd1960",version="1.0.0"1}Copy the code

Configure Prometheus to use TLS

Next, we need to configure Prometheus to get metrics from Node via HTTPS. The installation process is simple, whether directly download the latest binary version, or directly use the Docker image.

Notice that I have copied the certificate issued above into the current directory.

Linux - AMd64 cp ~/ Prometheus - TLS/node_main.crt. (MoeLove) ➜ Linux - AMd64 LS console_libraries Consoles LICENSE node_export. CRT NOTICE Prometheus Prometheus. Yml promtool tsdbCopy the code

Next, the configuration file needs to be modified so that Prometheus can fetch metrics from Node’s exposure.

global:
  scrape_interval:     15s 
  evaluation_interval: 15s 

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
    - targets: ['localhost:9090']

  - job_name: 'node_exporter'
    scheme: https
    tls_config:
      ca_file: node_exporter.crt
    static_configs:
    - targets: ['localhost:9100']
Copy the code

Scheme: HTTPS indicates that the connection is established through HTTPS. Tls_config specifies the certificate file to be used. For details about tls_config configuration, see the official documentation.

Finally, start the Prometheus and access/the targets in the browser, if you see in the endpoint https://localhost:9100/metrics, so congratulations you, Prometheus and Node Exporter are already connected via TLS.

Add Basic Auth

I’ve shown you how to use TLS between Prometheus and Node Exporter, and I’ll show you how to add Basic Auth.

It is important to note here that Basic Auth and TLS are not strongly dependent. You can use Basic Auth without TLS enabled, but I personally recommend doing it thoroughly and with TLS enabled.

Configure the password for Node Exporter

We can use htpasswd directly to generate the bcrypt password hash.

(MoeLove) ➜ Prometheus - TLS htpasswd -nBC 12' ' | tr -d ':\n'       
New password:
Re-type new password:                                              
$2y$12$WLw2sYa.NYZoBVoCOE84qe3xNm7kbSoKVIBXP.PvqNDna60vnZhEW
Copy the code

Here I’m just using it to generate a password hash, without passing a user name.

Next, modify the configuration file used by the Node Exporter mentioned above as follows:

tls_server_config:
  cert_file: node_exporter.crt
  key_file: node_exporter.key
basic_auth_users:
  User name Prometheus
  prometheus: $2y$12$WLw2sYa.NYZoBVoCOE84qe3xNm7kbSoKVIBXP.PvqNDna60vnZhEW
Copy the code

Start Node again and use curl to request the Metrics interface to see that 401 is returned.

(MoeLove) ➜ Prometheus - TLS curl - Ik https://127.0.0.1:9100/metrics HTTP / 1.1 401 Unauthorized the content-type: text/plain. charset=utf-8 Www-Authenticate: Basic X-Content-Type-Options: nosniff Date: Wed, 27 May 2020 11:45:16 GMT Content-Length: 13Copy the code

When you open the Targets page for Prometheus, you will also see the current prompt 401, unable to fetch metrics.

Configure Prometheus to use Basic Auth

Next, modify the Prometheus configuration file and add a basic_auth configuration item to it.

global:
  scrape_interval:     15s 
  evaluation_interval: 15s 

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
    - targets: ['localhost:9090']

  - job_name: 'node_exporter'
    scheme: https
    tls_config:
      ca_file: node_exporter.crt
    basic_auth:
      username: prometheus
      password: moelove.info
    static_configs:
    - targets: ['localhost:9100']
Copy the code

After modifying the configuration file, simply have Prometheus reload the configuration file:

(MoeLove) ➜  killall -HUP prometheus
Copy the code

Now refresh the Targets page for Prometheus to see that metrics are being fetched normally.

conclusion

This article describes how to enable THE TLS connection between Prometheus and Node Exporter and Basic Auth authentication for Node Exporter. Prior to that, you might have been able to maintain a partnership by adding an inverse proxy, such as Basic to Node

In production, we recommend more standardized operations, such as CA selection and password management. For example, Basic Auth of Node Exporter supports multiple user names and passwords.

Support for the security features mentioned in this article will be progressively advanced in the official Prometheus base components, including Prometheus, Alertmanager, Pushgateway, and Official Exporter, Follow-up community to provide a large probability of my friend will gradually follow up.

Finally, congratulations to Node Exporter on the release of V1.0.0.

Please feel free to subscribe to my official account [MoeLove]