For many people, the unknown, uncertain, out of control, there is a subconscious escape.

I had a similar feeling when I first encountered Prometheus. Prometheus contains too many concepts and the bar is too high for beginners.

Concept: Instance, Job, Metric, Metric Name, Metric Label, Metric Value, Metric Type (Counter, Gauge, Histogram, Summary), DataType (Instant) Vector, Range Vector, Scalar, String), Operator, Function

“Although Alibaba is the world’s largest retail platform, alibaba is not a retail company, it is a data company,” Ma said.

Prometheus, too, is essentially a data-based monitoring system.

Daily monitoring

Assume that the request volume of each WebServerA API needs to be monitored. The dimensions to be monitored include service name (Job), instance IP address (Instance IP address), API name (Handler), method, return code, and Request volume (Value).

SQL is used as an example to illustrate common query operations:

Query method=put and code=200 (red box)

SELECT * from http_requests_total WHERE code= "200" AND method= "put" AND created_at BETWEEN 1495435700 AND 1495435710;Copy the code

Query handler= Prometheus and method= POST requests (green box)

SELECT * from http_requests_total WHERE handler= "Prometheus" AND method= "post" AND created_at BETWEEN 1495435700 AND 1495435710;Copy the code

Query instance=10.59.8.110 where handler starts with query (green box)

SELECT * from http_requests_total WHERE handler= "query" AND instance= "10.59.8.110" AND created_at BETWEEN 1495435700 AND 1495435710;Copy the code

As shown in the preceding examples, daily monitoring is mainly used to perform combined queries based on monitoring dimensions and time in terms of common queries and statistics.

If 100 services are monitored, each service deploys an average of 10 instances, each service has 20 apis, 4 methods, and data is collected every 30 seconds and retained for 60 days. Then the total number of data items is: 100(services) 10 (instances) 20 (API) 4 (methods) 86400 (one day seconds) * 60(days) / 30 (seconds) = 13.824 billion data items, writing, storing, and querying data of such magnitude is impossible to be completed in the relational database of Mysql class.

Thus Prometheus uses TSDB as a storage engine.

The storage engine

TSDB, as the storage engine of Prometheus, perfectly fits the application scenarios of monitoring data:

  • The amount of data stored is enormous
  • Most of the time it’s writing
  • Writes are added almost sequentially, and most of the time the data is sorted by time when it arrives
  • Writes rarely write long-ago data and rarely update data. In most cases, data is written to the database seconds or minutes after it has been collected
  • The deletion operation is usually a block deletion, with the historical time to start selected and subsequent blocks specified. Rarely delete data at a single time or separate random times
  • Basic data is large, generally exceeding memory size. Generally, only a small portion of the selection is randomly selected, and caching does little or nothing
  • Read operations are typically sequential reads in ascending or descending order
  • Highly concurrent read operations are common

So how does TSDB do this?

"Labels" : [{" latency ":" 500 "}] "samples" : [{" timestamp ": 1473305798," value ": 0.9}]Copy the code

The original data is divided into two parts: Label and samples. The former records the monitoring dimension (label: label value), the indicator name and the optional key value pair of the label uniquely determine a time series (represented by series_id). The latter contains timestamp and index value.

The series ^ │............ server {latency = "500"} │............. server {latency = "300"} │........ ... Server {} │............ v < -- -- -- -- -- -- -- -- time -- -- -- -- -- -- -- -- -- -- >Copy the code

TSDB stores values for keys using timeseries:doc::. To speed up common query operations: combine label and time range.

TSDB builds three additional indexes: Series, Label Index, and Time Index.

Take label latency as an example:

Series

Store two pieces of data. One is a series of all label key-value pairs arranged in lexicographical order. The other part is the index of the time line to the data file, which cuts the specific location information of the stored data block records according to the time window, so that a large number of non-query window records can be quickly skipped in the query

Label Index

Each pair of labels is a list of all values of the label with index:label: as the key, and refers to the starting position of the value of Series by reference.

Time Index

The index:timeseries:: key is used to point to the data file for the corresponding time period

Data calculation

The powerful storage engine that powers data computing sets Prometheus apart from other monitoring services.

Prometheus queries various data sequences and then adds basic operators and powerful functions to perform metric series matrix operations (see figure below).

As such, Promtheus’ system is no less capable than the “data warehouse” + “computing platform” of the monitoring world. Therefore, when big data is applied in the industry at the beginning, we can understand that this is the direction of monitoring in the future.

One calculation, search everywhere

Of course, with so much computing power, the resources consumed are also terrifying.

Therefore, it is usually much faster to query the expected result than to require the original expression each time. This is especially true for dashboards and alarm rules, where the same expression needs to be queried repeatedly each time the dashboard is refreshed, as well as each operation of alarm rules.

For this reason, Prometheus provides Recoding rules that precompute frequently needed or highly computable expressions and store the results as a new set of time series for one calculation and multiple queries.