One of the biggest problems in using microservices is the screening and monitoring costs caused by the increase in the number of services. A lot of work has been done to solve these costs. Of course, Istio also integrates these components well, and these components are already installed by default (Zipkin + Jaeger, Prometheus + Grafana), how to use these components

Attach:

A meow blog :w-blog.cn

Istio official address :preliminary.istio. IO /zh

Istio Chinese document: preliminary. Istio. IO/useful/docs /

PS: This section is based on the latest ISTIO version 1.0.3

Clean up bookInfo and create again

Reset the official example bookInfo to its original state:

Run the clean BookInfo script and recreate it

>. / istio - 1.0.3 / samples/bookinfo/platform/kube/cleanup. Sh namespace? [default] istio-test using NAMESPACE=istio-test Deleted config: destinationrules details ... Kubectl apply-n istio-test-f kubectl apply-n istio-test-f Istio 1.0.3 / samples/bookinfo/platform/kube/bookinfo yaml > kubectl apply - n istio - test - f Istio 1.0.3 / samples/bookinfo/networking/bookinfo - gateway. Yaml > kubectl apply - n istio - test - f Istio 1.0.3 / samples/bookinfo/networking/destination - rule - all. YamlCopy the code

2. Link monitoring

Is often a request in micro service will do their best to N many services, so each service response status after which services to the business development or problem screen appears important additional, link monitoring is one of these solutions, the micro service invocation chain in records and through the visual way to display, in the industry is relatively mature solutions zipkin, But because Zipkin’s interface wasn’t that user-friendly, we used it with Jaeger, and Istio integrated it.

2.1 Access using Jaeger

Map to native by internal mapping

> kubectl port-forward -n istio-system $(kubectl get pod -n istio-system -l app=jaeger -o jsonpath='{.items[0].metadata.name}') 16686:16686Copy the code

Alternatively, it can be changed to nodeport:

> kubectl edit svc jaeger-query -n istio-system
  ports:
  - name: query-http
    port: 16686
    protocol: TCP
    targetPort: 16686
    nodePort: 30686
  selector:
    app: jaeger
  sessionAffinity: None
  type: NodePortCopy the code

In the Jaeger Dashboard, select ProductPage from Service and click the Find Traces button to see trace information:

Moving down to the next level, you can see the invocation level of each service and the distribution of total elapsed time:

You can see more about this in the expansion

2.2 Prerequisite of Link Monitoring Headers transmission

Why do YOU need to pass the specified Headers after using the service grid? Here will from link monitoring mechanism, before service grid monitoring need to link every program needs to link the monitoring server sends the message, initiated by the first program to find links to monitor the ID, the following each procedure is called monitoring system needs to be told that link when I am in the link ID, to link the whole link at this time.

Although the Istio agent can automatically send Span information, some assistance is needed to unify the entire tracing process. The application should propagate the HTTP headers associated with the trace itself so that the same trace process is properly unified when the agent sends a Span message.

To complete the propagation of the trace, the application should collect the following HTTP headers from the request source and propagate them to outgoing requests:

  • x-request-id
  • x-b3-traceid
  • x-b3-spanid
  • x-b3-parentspanid
  • x-b3-sampled
  • x-b3-flags
  • x-ot-span-context

If you look at the sample service, you can see that the ProductPage service (Python) extracts the required headers from the HTTP request:

def getForwardHeaders(request):
    headers = {}

    if 'user' in session:
        headers['end-user'] = session['user']

    incoming_headers = [ 'x-request-id',
                         'x-b3-traceid',
                         'x-b3-spanid',
                         'x-b3-parentspanid',
                         'x-b3-sampled',
                         'x-b3-flags',
                         'x-ot-span-context'
    ]

    for ihdr in incoming_headers:
        val = request.headers.get(ihdr)
        if val is not None:
            headers[ihdr] = val
            #print "incoming: "+ihdr+":"+val

    return headersCopy the code

2.3 Acquisition control

Istio captures tracing of all requests by default. For example, when to use the Bookinfo sample application/productPage above every time you visit you see the corresponding trace dashboard in Jaeger. Link monitoring every time and link server communication is also a performance overhead, under a daily pv business must to collect all the link is inappropriate, regardless of from the CPU or disk space is easy to appear the bottleneck, and link monitoring is not a log is a means of screening, so we need to collect in a production environment frequency restrictions:

Find the PILOTTRACESAMPLING environment variable changed from 100% to 10% in pilot:

> kubectl -n istio-system edit deploy istio-pilot
...
        - name: PILOT_TRACE_SAMPLING
          value: "10"
...
> :wqCopy the code

If you refresh the page 10 more times, you will only see one call in JaegerUI, where the minimum accuracy is 0.01% and the valid value is 0.0~100.0(you don’t need this function, you can turn it off completely).

Third, data acquisition

Another advantage of Istio integration is Prometheus + Grafana, which provides customisable report display and alarm mechanisms for Prometheus as a basic data collection and storage method for Grafana.

> kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app=prometheus -o jsonpath='{.items[0].metadata.name}') 9090:9090Copy the code

> kubectl edit svc prometheus -n istio-system
  ports:
  - name: http-prometheus
    port: 9090
    protocol: TCP
    targetPort: 9090
    nodePort: 30090
  selector:
    app: prometheus
  sessionAffinity: None
  type: NodePortCopy the code

You can query a variety of metrics

Example 1:

Istio_requests_total {destination_service=" productPage.istio-test.svc.cluster. local"}Copy the code

Example 2:

Destination_service ="reviews.istio-test.svc.cluster.local", destination_version="v3"}Copy the code

Example 3:

Rate (istio_requests_total{destination_service=~" productPage.*", response_code="200"}[5m])Copy the code

Fourth, monitoring visualization

Of course, using Prometheus directly is not very convenient and is not particularly user friendly, and does not do a lot of presetting, which is generally used with Grafana

> kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app=grafana -o jsonpath='{.items[0].metadata.name}') 3000:3000Copy the code

> kubectl edit svc grafana -n istio-system
  ports:
  - name: http
    port: 3000
    protocol: TCP
    targetPort: 3000
    nodePort: 33000
  selector:
    app: grafana
  sessionAffinity: None
  type: NodePortCopy the code

You can then specify a variety of metrics