The cover image shows a lock-screen desktop getting up at 3:30 a.m.

preface

From 7th last Saturday to 11th today, I have been in the hospital. My child has been hospitalized because of pneumonia. I need to take care of my baby during the day and night. By the way, the hospital doesn’t have broadband or WiFi, so my phone is on a hot spot

The main content of this article

This article is mainly a combination of theory and practice. The practical part involves how to add link tracking Sleuth + Zipkin to my Spring Cloud “Jiabibian” open source project.

This article focuses on:

  • Link tracing fundamentals
  • How to easily incorporate link-tracing middleware into your project
  • How to use link tracing to troubleshoot problems?

First, why use link tracing?

1.1 Cause: Split service units

Microservices architecture is actually a distributed architecture, divided into service units according to business.

Because the number of service units is large, perhaps thousands, and the business is more complex, it is difficult to locate errors and exceptions if they occur.

1.2 Cause: Complicated logic

For example, a request needs to invoke multiple services to complete the entire business closed loop, and the code logic and business logic of internal services are complicated. If a service has problems, it is difficult to quickly determine the problem of that service.

1.3 Result: Fast positioning

If we add distributed link tracing to track which services are involved in a request and in what order, then we know the details of each request and can locate problems quickly.

Second, the core of link tracing

Link tracking components include Twitter’s Visual link tracking component Zipkin, Google’s Dapper, Ali’s Eagleeye, etc. Sleuth is a component of Spring Cloud. Spring Cloud Sleuth borrowed Dapper’s terminology.

This paper mainly explains the use of Sleuth + Zipkin to better achieve link tracking.

Why is it possible to trace the entire link? It’s just a Trace ID that connects a bunch of spans. You can obtain the information of the entire link by integrating the information recorded by the Span. The following

2.1span

  • In plain English: Remote calls and Span one-to-one.

  • As a basic unit of work, a Span is generated each time a remote invocation service is sent.

  • Span is a 64-bit unique ID.

  • By calculating the start and end times of Span, you can count the time taken for each service invocation.

2.2 Trace

  • One Trace corresponds to multiple spans.

  • It consists of a series of spans, a tree structure.

  • 64-BIT unique ID.

  • Each time a client accesses the API interface of the microservice system, it may call multiple microservices in between. Each call will generate a new Span, and multiple spans form the Trace

2.3 Annotations

The link tracing system defines core annotations that define the start and end of a request, between microservices, not devices such as browsers or mobile phones. Notes include:

  • cs– Client Sent: Indicates that the Client sends a request, describing what the request is calledSpanThe start time of. Note: The client here refers to the caller of the microservice, not the browser or mobile client as we understand it.
  • sr– Server Received: The Server receives the request and is ready to start processing it if it is sentsrMinus thecsTime stamp, you can get network transmission time.
  • ss– Server Sent: The Server sends a response and records the time when the request is processed.ssTimestamp minussrTime stamp, you can get the server request time.
  • cr– Client Received: indicates the end time of the Span Received by the Client. IfcrTime stamp of minuscsTime stamp, you can get the time consumed by a microservice invocation, that is, oneSpanThe total time consumed by.

2.4 Mechanism of Link Tracing

Suppose that the links of three microservice invocations are as follows: Service 1 calls Service 2, and Service 2 calls Service 3 and Service 4.

Link tracing will add a Trace ID and a Span ID to each service call. As shown below:

Explanation in plain English:

  • Note the color above. The same color represents the same Span ID, indicating a node in the link tracing.

  • Step 1: The client calls Service 1 and generates a Request with the Trace ID and Span ID empty before the Request reaches Service 1.

  • Service 1: Trace ID = X, Span ID = A

  • Service 1 sends a request to Service 2. The Span ID is equal to B. This is called Client Sent.

  • Step 4: The request reaches Service 2, the Span ID is equal to B, the Trace ID does not change, and it is called Server Received, that is, the Server gets the request and is ready to start processing it.

  • Step 5: Service 2 starts processing the request. After processing, the Trace ID remains unchanged, and Span ID = C.

  • Service 2 sends the request to Service 3. The Trace ID remains the same. The Span ID = D.

  • Step 7: Service 3 receives the request, Span ID = D, and is called Server Received.

  • Step 8: Service 3 starts processing the request, and when it is finished, Span ID = E.

  • Service 3 sends a response to Service 2. Span ID = D.

  • Step 10: Service 3 receives the response from Service 2. The Span ID = D is called Client Received.

  • Step 11: Service 2 returns a response to Service 1 with a Span ID equal to B, which is called Client Received.

  • Step 12: After Service 1 processes the response, the Span ID = A, which is the same as the Span ID of step 2.

  • Span ID = A, Span ID = A,

  • Service 3 sends a request to Service 4. The Span ids of Service 3 are F and G. Refer to steps 6 to 10 above.

Simplify the above steps in the same color to the following link tracing diagram:

  • Span ID = A, Parent ID = null,Service 1The request was received.
  • Span ID= B, Parent ID= A,Service 1Send a request toService 2Return a response toService 1In the process.
  • Span ID= C, Parent ID= B,Service 2The intermediate process of.
  • Span ID= D, Parent ID= C,Service 2Send a request toService 3Return a response toService 2In the process.
  • Span ID= E, Parent ID= D,Service 3The intermediate process of.
  • Span ID= F, Parent ID= C,Service 3Send the request to Service 4 and return the response toService 3In the process.
  • Span ID= G, Parent ID= F, Service 4

Using the Parent ID to find the Parent node, the entire link can be traced.

Spring Cloud integration Sleuth

You can refer to my GitHub open source project PassJava.

3.1 Introducing Spring Cloud dependencies

Introduce Spring Cloud dependencies in passjava-common

Because the link tracing component we used, Sleuth, is a component of Spring Cloud, we need to introduce Spring Cloud dependencies.

<dependencyManagement> <dependencies> <! - Spring Cloud depend on - > < the dependency > < groupId > org. Springframework. Cloud < / groupId > <artifactId>spring-cloud-dependencies</artifactId> <version>Hoxton.SR3</version> <type>pom</type> <scope>import</scope> </dependency> </dependencies> </dependencyManagement>Copy the code

3.2 Introduction of Sleuth dependency

Importing the link tracing component Sleuth is as simple as importing the Sleuth dependency in the PEM. XML file.

Introduce the Sleuth dependency in passjava-common:

<! -- Link Tracing Component -->
<dependency>
	<groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-sleuth</artifactId>
</dependency>
Copy the code

3.3 Observing link Tracing using Logs

Instead of integrating the Zipkin link tracing visual component, we will view the link tracing information in a log.

File path: \ passjava-platform \passjava-question\ SRC \main\resources\application.properties
Add configuration:
logging.level.org.springframework.cloud.openfeign=debug
logging.level.org.springframework.cloud.sleuth=debug
Copy the code

3.4 Starting microservices

Enable the following microservices:

  • Passjava-gateway service (gateway)

  • Passjava-question service (Title central micro service)

  • Renren Service (Admin background management service)

    The following figure is displayed after successful startup:

3.5 Test trace requests

Open the Admin background, visit the topic center -> topic configuration page, you can see the following request:

http://localhost:8060/api/question/v1/admin/question/list?t=1605170539929&page=1&limit=10&key=
Copy the code

When you open the console, you can see that the trace log is printed.

Description:

  • When Sleuth link tracing is not configured, the INFO message is [passjava-question,,,] followed by three empty strings.
  • When configuring the Sleuth link tracking, tracking information is [passjava – question, 504 a5360ca906016 e55ff064b3941956, false], the first is the Trace ID, the second is the Span ID.

4. Zipkin link tracking principle

Above, we can obtain the call link by simply introducing the Sleuth component, but it is not convenient to see only through the output information of the console.

Zipkin was born, a graphical tool. Zipkin is an open source distributed tracking system for Twitter. It is mainly used to collect time series data for tracking system invocation problems.

And with the introduction of the Zipkin component, there is no need to introduce the Sleuth component because the Zipkin component already introduced it for us.

Zipkin official: zipkin.io

4.1 Zipkin infrastructure

Zipkin consists of four components:

  • Collection (collector component), which is responsible for collecting external system trace information.
  • Storage the collected trace information is stored in memory by default and can be stored in MySQL and ElasticSearch.
  • API (query component), provides interface to query trace information, used for UI components.
  • UI (Visual Web UI component) allows you to visually view trace information based on service, time, and annotations. Note: The Web UI does not require authentication.

4.2 Zipkin tracking process

Process explanation:

  • Step 1: The user code initiates an HTTP Get request at /foo.
  • Step 2: After the request reaches the tracing tool, the request is intercepted and two pieces of information are recorded: the label and the timestamp. And HTTP Headers will add trace Headers.
  • Step 3: Send the wrapped request to the HTTP client containing x-B3-tracEID and X-B3-SPANID request headers.
  • Step 4: Send the request by the HTTP client.
  • Step 5: After the Http client returns a 200 OK response, the tracing tool records the elapsed time.
  • Step 6: The trace tool sends 200 OK to the client.
  • Step 7: Asynchronously report Span information to the Zipkin collector.

Integrate Zipkin visual components

5.1 Starting and Connecting VMS

vagrant up
Copy the code

Use the Xshell tool to connect to the VM.

5.2 Docker Installs the Zipkin service

  • Use the following command to start pulling the Zipkin image and starting the Zipkin container.
docker run -d -p 9411:9411 openzipkin/zipkin
Copy the code
  • After the command is executed, the system downloads and starts the command.

  • Using the Docker ps command, you can see that the Zipkin container has been started successfully. As shown below:

  • Open Zipkin UI in a browser window

Access the service address: http://192.168.56.10:9411/zipkin.

5.3 Introduction of Zipkin dependencies

Introduce zipkin dependencies in the public module

<! Zipkin -->
<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-zipkin</artifactId>
</dependency>
Copy the code

Since the SLEUTH component is already introduced in the Zipkin package, you can delete the previously introduced SLEUTH component.

5.4 Adding Zipkin Configuration

Add the Zipkin configuration under the microservice module you want to track.

# Zipkin server address
spring.zipkin.base-url=http://192.168.56.10:9411/
Turn off service discovery, or Spring Cloud will treat the Zipkin URL as the service name.
spring.zipkin.discovery-client-enabled=false
Use RabbitMQ or Kafka to transfer data over HTTP.
spring.zipkin.sender.type=web
# set sampling rate to 100 %, default to 0.1 (10%)
spring.sleuth.sampler.probability=1
Copy the code

5.5 Test whether Zipkin works

Here I wrote an API in the passJava-member microservice:

Passjava – member service API: getMemberStudyTimeListTest, access path for/studytime/list/test / {id}.

Passjava – member service remote call passjava – study service API: getMemberStudyTimeListTest.

I used the Postman tool to test the MEMBER service API:

Open Zipkin tool and search passjava-member link tracking log, you can see a record, as shown in the following figure:

You can see from the figure that the Passjava-member microservice invokes the Passjava-Study microservice, as shown in the left half of the figure.

Moreover, the detailed call time of passJava-Study microservice is clearly recorded, as shown in the right half of the figure.

Time calculation:

  • Request transfer time: Server start-client Start = 2.577s-2.339s = 0.238s
  • Server processing time: Server finish-server Start = 2.863s – 2.577s = 0.286s
  • Total request time: Client finish-client Start = 2.861s – 2.331s = 0.522s
  • Total elapsed time of passjava-member service: 3.156 seconds
  • Total elapsed time of the passjava-study service: 0.521 seconds
  • As you can see, the PassJava-member service took a long time and performed poorly.

You can also view it as an icon:

Zipkin data persistence

6.1 Zipkin supported databases

Zipkin stores data in memory by default, and monitoring data is lost if Zipkin restarts. In a build environment, data loss can be a big problem, so you need to persist Zipkin’s monitoring data. Zipkin supports storing data in the following databases:

  • Memory (default, not recommended)
  • MySQL (not recommended for large data volume, slow query)
  • Elasticsearch (recommended)
  • Cassandra (there are few domestic companies using Cassandra and few relevant documents)

6.2 Using Elasticsearch as the Storage Medium

  • Configure ElasticSearch as the zipkin data storage medium using Docker.
Docker run --env STORAGE_TYPE=elasticsearch --env ES_HOSTS=192.168.56.10:9200 openZipkin /zipkin-dependenciesCopy the code
  • ES as storage media configuration parameters:

Seven,

This article explains the core principles of link tracing and Sleuth + Zipkin’s components, as well as how they were added to my open source project, Gabugo.

Open source project address: github.com/Jackson0714…

Write in the last

This week really exhausted, Eva is also suffering, after leaving the hospital, Eva eat is not as active as before, see the doctor that kind of clothes are afraid, even see the photo printer are afraid. I am afraid to give him an injection, take medicine, do atomization. Have not married baby to grasp the time to learn, refueling young ~

I am Wukong, strive to become stronger, become super Saiya people! I have written a set of Spring Cloud advanced tutorial and PMP brush small procedures.

Welcome to our public account: Wukong chat structure