preface

This series focuses on Prometheus and how it and its surrounding ecosystem built its own real-time monitoring and alarm platform.

Intended for users of Prometheus for the first time, this series is intended to focus on manipulation and execution, but highlights important concepts as well. The series is mainly divided into the following pieces

  • PrometheusIntroduction and construction of various concepts, how to capture data (Prometheus: Fire of God, The Rise of Prometheus)
  • How do I push data toPrometheus, what kind of scenarios are push and pull respectively used for (this sharing content)
  • PrometheusData structure and query languagePromQLThe use of
  • How do Java applications work togetherPrometheusIntegration, how to enable service discovery, and how to customize business metrics
  • PrometheusHow andGrafanaVisual suite for integration and alarm setting
  • How to write a Java suite that integrates monitoring Dubbo metrics
  • Practical case sharing, how to do each business side and system side monitor the market

Grab and push

Pull mode:

Prometheus obtains data only by pulling, which means that Prometheus will request the HTTP URL provided by each target at a fixed frequency to obtain data. This requires each service endpoint to provide an HTTP interface to retrieve real-time data.

Push mode:

Prometheus also implemented a variant of the push data approach.

Why is it disguised. Since Prometheus used pull and pull methods to obtain data, there was no official push feature. However, a PushGateway component has been added to support push.

This component acts as a proxy service and is deployed independently. It has no data capture function, can only passively wait for data push. After the application pushes data to PushGateway, Prometheus fetches data from PushGateway.

Points to note in push mode

Even if the client pushes the full amount of data inPushGateway.PrometheusNor is it possible to pull all the data the user pushes up during the period.

In fact, Prometheus only pulls data from the user’s last push.

In the first part of this series, it was mentioned that Prometheus did not need every precise data, but kept medium and low precision data for a long time. It just captures one piece of data at a time, at a fixed frequency. It can also form some kind of trend in the data.

If the client does not push a new indicator toPushGateway, thenPrometheusWill always pull data on the last push until the indicator disappears, default is 5 minutes.

Pushgateway is not supposed to store metrics, but we added a feature to store metrics locally for the time being in the event of an unexpected Pushgateway restart. The parameter –persistence.interval=5m — is kept for 5 minutes by default. Five minutes later, indicators on the local storage will be deleted. You can adjust this value to correct the timing of the discovery of the exception.

Through a singlePushgatewayWhen monitoring multiple instances,PushgatewaySingle points of failure and potential bottlenecks

If Pushgateway is used, multi-point deployment is recommended. Then the front through nginx reverse proxy multiple nodes, load balancing.

Scenarios applicable to the push mode

  • PrometheusWhen the scheduled pull mode is used, each node cannot be pulled directly due to the sub-network or firewallTargetIndex data, can be used at this timeTargettoPushGatewayPush data up and thenPrometheusGo to thePushGatewayUpper timing pull
  • When monitoring service data, you can collect different service data in a unified mannerPushGatewayTo unify the collection and thenPrometheusTo unify the pull

Set up

Pushgateway includes a Docker installation and a normal installation

Go to the Github release page for Prometheus

Github.com/prometheus/…

Download the corresponding package as required, I need to deploy on the Linux server, so download this

Download it and unzip it. Run:

nohup ./pushgateway &
Copy the code

The default port is 9091

You can access the following page based on IP +port in the browser, and the startup is successful:

In addition, set Target in the Prometheus configuration file:

- job_name: 'pushgateway'
    scrape_interval: 10s # Pull every 10 seconds
    honor_labels: true
    static_configs:
    - targets: ['localhost:9091']
      labels:
          instance: pushgateway
Copy the code

After setting, restart Prometheus and see Pushgateway in the UP state in the Target TAB.

The setup phase is complete.

URL push test

I used postman software to push the test, and the push URL format is: /metrics/job/

{/

/

}


Push an indicator AAA labeled BBB =BBB, CCC =CCC, 111.1 to a group named job= PUSHGateway,instance=demo.

In fact, you can simply understand that aaa has four labels: Job, instance, BBB and CCC. Only job and instance belong to group labels.

For the same index in a group, Prometheus takes only the latest one at a time; the same index may exist in different groups.

This will be covered in detail in the next article in the series on data structures and tag structures.

In summary, after you submit the POST request, you can see the following data at http://ip:9091:

As you can see, the aaa tag has been successfully submitted to Pushgateway.

Next, we look up this metric in Prometheus:

As you can see, Prometheus was also successful in pulling this indicator.

The Java side uses SDK to push

While we can use tools such as HttpClient to submit on the Java server side, we need to assemble many request bodies ourselves. Prometheus officially provides an SDK.

First we introduce dependency packages in Maven:

<dependency>
  	<groupId>io.prometheus</groupId>
  	<artifactId>simpleclient_pushgateway</artifactId>
  	<version>0.9.0</version>
</dependency>
Copy the code

On the Gauge, Timer, Counter, four common examples of indicators for push the Summary:

public void run(String... args) throws Exception {
      Gauge guage = Gauge.build("my_custom_metric"."This is my custom metric.")
        								 .labelNames("aaa"."bbb").register();
      Gauge.Child child = guage.labels("AAA"."BBB");
      child.set(334.5);

      Gauge timerGauge = Gauge.build("my_timer_metric"."this is my timer metric.").register();
      Gauge.Timer timer = timerGauge.startTimer();
      Thread.sleep(3000L);

      Counter counter = Counter.build("my_count_metric"."this is my count metric.").register();
      counter.inc();
      counter.inc();

      Summary summary = Summary.build("my_summary_metric"."this is my summary metric.").register();
      summary.observe(45.6);
      summary.observe(54.5);

      String url = "xxx.xxx.xxx.xxx:9091";
      PushGateway pg = new PushGateway(url);
      Map<String, String> groupingKey = new HashMap<>();
      groupingKey.put("instance"."my_instance");
      pg.pushAdd(CollectorRegistry.defaultRegistry, "my_job", groupingKey);
}
Copy the code

This code demonstrates a batch submission scenario for four metrics. By registering to CollectorRegistry. DefaultRegistry, finally pushAdd together.

We can query the submitted metrics in Pushgateway:

These four indicators can also be found in Prometheus, but the specific graph is not attached. Try it out for yourself.

The last

This series is designed to teach you step by step to build your own system and business monitoring. This will be updated later. The next section examines data format analysis in Prometheus and the use of PromQL.

Pay attention to the author

If you like the author’s article, welcome to follow “Yuanren Tribe” on wechat, a technology sharing account that only does original work

After concern, reply to “data” to obtain 50G technical data