Spring Cloud Sleuth+Zipkin for Distributed link tracing

Hello, everyone. I’m Chen

This is the ninth article in Spring Cloud Advancements. The previous article is as follows:

How strong is Nacos, the soul ferryman of microservices, in 55 images?
OpenFeign kills series 9 asks, who can stand that?
Ali interview asked: Nacos, Apollo, Config configuration center how to select? These 10 dimensions tell you!
Ali failed in the interview: How to select 5 kinds of microservice registry? These dimensions tell you!
Ali’s Sentinel-killing Device 17 questions?
Compare 7 distributed transaction schemes, or prefer Ali open source Seata, really fragrant! (Principle + actual combat)
Spring Cloud Gateway kills 10 questions?
Spring Cloud Gateway integrated Ali Sentinel Gateway flow limiting actual combat!

In this article, Chen introduces the knowledge related to link tracking, which is mainly composed of Spring Cloud Sleuth and Zipkin, and the following article introduces another one.

The contents of the article are as follows:

Why do YOU need link tracing?

In a large-scale distributed micro-service system, a system is divided into N multiple modules, which are responsible for different functions and combined into a set of systems, ultimately providing rich functions. In this distributed architecture, a request often involves multiple services, as shown below:

The call between services is complex, and the cost of maintenance increases exponentially, which inevitably leads to the following problems:

How can dependencies and dependencies between services be clearly seen?
How can I quickly locate the abnormal service when exceptions occur?
How can you quickly locate which service is affected when a performance bottleneck occurs?

In order to locate problems quickly in distributed architecture, distributed link tracing comes into being. Restore a distributed request to an invocation link for logging, performance monitoring, and centralized presentation of the invocation of a distributed request. For example, the time spent on each service node, the specific machine on which the request arrives, the request status of each service node, and so on.

What are the common link tracing techniques?

There are many link tracing projects on the market, some of which are excellent, as follows:

Cat: Real-time application monitoring platform developed based on Java and open source by Dianping, including real-time application monitoring and business monitoring. The integration solution is to implement monitoring through code burying points, such as interceptors, filters, etc. The code is very intrusive, the integration cost is high, the risk is large.
Zipkin: An open source distributed tracking system developed by Twitter to collect timed data from services to address latency issues in microservices architecture, including data collection, storage, lookup, and presentation. The product is relatively simple to use in combination with Spring-Cloud-sleUTH, with easy integration but simple functionality.
Pinpoint: Korea open source based on bytecode injection call chain analysis, and application monitoring analysis tools. Feature is to support a variety of plug-ins, UI powerful, access end without code intrusion
Skywalking: SkyWalking is a native open source call chain analysis and application monitoring analysis tool based on bytecode injection. Feature is to support a variety of plug-ins, strong UI function, access end without code intrusion. You have joined the Apache incubator.
Sleuth: Link tracing solution for distributed systems provided by SpringCloud. Unfortunately, Ali department does not have open source projects related to link tracking. We can use Spring Cloud Sleuth+Zipkin to do link tracking solutions.

What is Spring Cloud Sleuth?

Spring Cloud Sleuth implements a distributed service link tracing solution that allows us to quickly locate problems with a service. In simple terms, Sleuth is the client of the call chain monitoring tool, which is integrated on each microservice and responsible for generating the call chain monitoring data.

Spring Cloud Sleuth is only responsible for generating monitoring data and displaying it in the form of logs, without providing a visual UI interface.

Before learning Sleuth, you must understand several concepts:

Span: A basic unit of work, equivalent to a node in a linked list, whose start, progress, and end are marked by a unique ID. We can count the elapsed time of the service invocation from the start and end timestamps stored in it. You can also get the name of the event, request information, and so on.
Trace: A tree structure formed by a series of spans that creates a unique ID (traceId) that uniquely identifies a link when a request arrives at a system entry. This traceId is always passed between services until the request is returned, so you can use this traceId to concatenate the entire request to form a complete link.
Annotations: Some core annotations are used to annotate events between microservice calls. The important ones are as follows:
- Client Send (CS) : The Client sends a request to start the life cycle of a request
- Server Received (SR) : the Server receives the request and processes it. Sr-cs = network latency = time of service invocation
- Server Send (SS) : the Server is ready to Send the packet to the client. Ss-sr = Request processing time on the server
- Cr (Client Reveived) : the Client receives the response from the server, and the request ends. Cr-sr = Total time of request

How does Spring Cloud integrate Sleuth?

Integrating Spring Cloud Sleuth doesn’t have to be difficult. To do this, you need to prepare the following three services:

Gateway-sleuth9031: serves as the gateway service
Sleuth-product9032: Commodity microservices
Sleuth-order9033: Order microservice

The invocation relationship of the three services is shown as follows:

The client requests the gateway to initiate a request to query the order, the gateway routes the order service, the order service obtains the order details and calls the goods service to obtain the goods details.

Add the dependent

Add the SLEuth dependency to the parent module as follows:

<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-starter-sleuth</artifactId>
</dependency>
Copy the code

The above is just the dependency of Spring Cloud Sleuth, Nacos and openFeign, which is no longer detailed here. There are unclear can be combined with Chen’s previous several articles and case source code to fill in the gap.

Adjusting a Log Level

Since Sleuth does not have a UI, you need to adjust the log level to see more detailed link information on the console.

Add the following configuration to the configuration files of the three services:

Set the log level of openFeign and sleuth to DEBUG for viewing log information
logging:
  level:
    org.springframework.cloud.openfeign: debug
    org.springframework.cloud.sleuth: debug
Copy the code

Perfect demo interface

The following interfaces are only for demonstrating built data and do not integrate DB.

Sleuth-order9033 interface for querying order details, as shown below:

The interface for querying product details of SLEUth-product9032 is shown as follows:

Gateway-sleuth9031 Gateway route configuration is as follows:

test

Launch the three services, direct access to the browser: http://localhost:9031/order/get/12

Observe the console log output as shown below:

There are four parameters in the log format, which are defined as follows:

First: service name
The second traceId uniquely identifies a link
Third: spanId, the basic unit of work ID in the link
Fourth: indicates whether to output data to other services. True will output information to other visual services for observation. There is no zipkin integration, so false

Good, so far integration completed, can not help but in the heart to breathe a cool breath, directly see log that is not the eye to see blind……….

The source code of the case has been uploaded, the public number [code ape technology column] reply to the key word 9528.

What is ZipKin?

Zipkin is an open source Twitter project based on Google Dapper that collects timed data from the service,

To address latency issues in microservices architecture, including data collection, storage, lookup, and presentation.

The ZipKin infrastructure is shown below:

Zipkin is divided into four core components as follows:

Collector: A Collector component that processes trace information sent from external systems and converts it into a Span format processed internally by Zipkin for subsequent storage, analysis, presentation, and so on.
Storage: A Storage component that processes trace information received by the collector. By default, this information is stored in memory. We can also modify this Storage policy to store trace information to the database by using other Storage components
RESTful apis: API components that provide external access interfaces. For example, to display tracking information to clients, or external system access to achieve monitoring, etc.
UI: Upper level application based on API component implementation. Through UI components, users can easily and intuitively query and analyze tracking information

Zipkin is divided into server and client. The server is mainly used to collect tracking data and display it. The main function of the client is to send it to the server.

How to build zipkin server?

You first need to download the jar packages from the server, address: search.maven.org/artifact/io…

After downloading, you will get a JAR package as shown below:

To start the jar directly, run the following command:

Java jar zipkin - server - 2.23.4 - exec. JarCopy the code

If the following interface is displayed, the startup is complete:

At this point, you can visit zipkin’s UI at http://localhost:9411, which is as follows:

The above is to build the server by downloading jar, of course, there are other ways to install, such as Docker, to try it yourself, Chen will not demonstrate.

How to build zipKin client?

The server is only tracking the collection and presentation of data, the client is the generation and transmission of data, the following details how to build a client.

Add zipkin’s dependencies directly to the same three microservices as in the previous example, as follows:

<! -- Link-traced Zipkin dependencies, which include Sleuth dependencies -->
<dependency>
      <groupId>org.springframework.cloud</groupId>
      <artifactId>spring-cloud-starter-zipkin</artifactId>
</dependency>
Copy the code

Note: Since the Spring Cloud Sleuth dependency is already included in the Spring-Cloud-starter-Zipkin, only one of the above dependencies needs to be introduced.

The zipkin server address needs to be configured in the configuration file as follows:

spring:
  cloud:
  sleuth:
    sampler:
      # Log data sampling percentage, default 0.1(10%), set to 100% for testing purposes, production environment only needs 0.1
      probability: 1.0
  zipkin:
      #zipkin server request address
    base-url: http://127.0.0.1:9411
      Have nacos treat it as a URL, not as a service name
    discovery-client-enabled: false
Copy the code

After completion of the above configuration start the service can, at this time to visit: http://localhost:9031/order/get/12

After invoking the interface, visit the ZIPkin UI again as shown below:

It can be seen that the interface just called has been monitored. Click SHOW to view the details, as shown below:

You can see that a complete link is displayed on the left, including the service name and time, and information about the service invocation is displayed on the right, including the start time, end time, request URL, and request mode…..

In addition to the information related to the call link, you can also clearly see the dependency of each service as shown in the following figure:

How to switch data transmission mode of zipKin?

The default zipkin transport is HTTP, but there is a problem that if the client and server fail during transport, this trace log information will be lost.

Of course, Zipkin also supports MQ transport, which supports the following types of messaging middleware:

ActiveMQ
RabbitMQ
Kafka

MQ transmission can not only prevent message loss but also improve transmission efficiency. Therefore, MQ transmission is recommended in production.

So the question is, how do you switch?

In fact, the way is very simple, Chen to RabbitMQ as an example.

1. Connect the server to RabbitMQ

Run the server and connect to RabbitMQ as follows:

Java jar zipkin - server - 2.23.4 - exec. Jar -- zipkin. The collector. The rabbitmq. Addresses = localhost --zipkin.collector.rabbitmq.username=guest --zipkin.collector.rabbitmq.password=guestCopy the code

The command analysis is as follows:

zipkin.collector.rabbitmq.addressesAddress: MQ
zipkin.collector.rabbitmq.usernameUser name:
zipkin.collector.rabbitmq.passwordPassword:

2. Add RabbitMQ to the client

The RabbitMQ dependencies will need to be added as follows:

<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-amqp</artifactId>
</dependency>
Copy the code

Configure the IP address, user name, and password for MQ as follows:

spring:
  rabbitmq:
    addresses: 127.0. 01.
    username: guest
    password: guest
Copy the code

3. Switch the transmission mode in the configuration file

Spring. Cloud. Zipkin. Sender. This configuration is used to switch the type transmission mode, the values for the rabbit rabbitMQ is used for data transmission.

The configuration is as follows:

spring:
  cloud:
  zipkin:
    sender:
     Use rabbitMQ for data transfer
      type: rabbit
Copy the code

Note: using the MQ transmission, is spring. The cloud. Zipkin. Sender. The base – the url can be removed.

The complete configuration is shown below:

4, test,

Since using the MQ transmission, so we do not start the service side is also successfully transmission, browser visit: http://localhost:9031/order/get/12

RabbitMQ has received data from the zipkin queue, as shown in the following figure:

You can see that a message is not consumed, and you can see that the message content is Trace and Span information.

Ok, let’s start the server with the following command:

Java jar zipkin - server - 2.23.4 - exec. Jar -- zipkin. The collector. The rabbitmq. Addresses = localhost --zipkin.collector.rabbitmq.username=guest --zipkin.collector.rabbitmq.password=guestCopy the code

After starting the server, it found that the messages in the Zipkin queue were instantly consumed. After checking the UI interface of Zipkin, it found that link information had been generated, as shown in the following figure:

How does Zipkin persist?

Zipkin information is stored in memory by default and will be lost if the server restarts, but zipkin provides pluggable storage.

Zipkin supports the following four storage methods:

Memory: The service will be invalid after restart
MySQL: The performance deteriorates with a large amount of data
Elasticsearch: a mainstream solution, recommended
Cassandra: Technology is too awesome to use, few people use it, choose your own, but official recommendation

Zipkin (zipkin, zipkin, Zipkin, Zipkin, Zipkin)

1. Create a database

Zipkin table SQL server MySQL built in source zipkin – storage/MySQL – v1 / SRC/main/resources/MySQL. The SQL, the SQL file I will be in the case of source code.

Github address: github.com/openzipkin/…

Create database: zipkin (any name), import SQL, create a new database table as shown in the following figure:

2. Configure MySQL on server

The server configuration is simple, run the following command:

Java-jar zipkin-server-2.23.4-exec.jar --STORAGE_TYPE=mysql --MYSQL_HOST=127.0.0.1 --MYSQL_TCP_PORT=3306 --MYSQL_DB=zipkin --MYSQL_USER=root --MYSQL_PASS=Nov2014Copy the code

The preceding command parameters are analyzed as follows:

STORAGE_TYPE: specifies the storage mode. The default memory mode is memory
MYSQL_HOST: specifies the IP address of the MySQL database. The default value is localhost
MYSQL_TCP_PORT: port number of MySQL. The default port number is 3306
MYSQL_DB: the database name in MySQL. The default is zipkin
MYSQL_USER: indicates the user name
MYSQL_PASS: password

How did Chen remember these parameters? Nonsense, must remember, don’t have to always check for source code, these configurations are in source/zipkin – server/SRC/main/resources/zipkin – server – Shared. Yml this configuration file, such as the MySQL related configuration, the following figure:

All the configuration items on the Zipkin server are here, so check them out.

Github address: github.com/openzipkin/…

RabbitMQ transfer mode, MySQL persistence mode, complete command as follows:

Java-jar zipkin-server-2.23.4-exec.jar --STORAGE_TYPE=mysql --MYSQL_HOST=127.0.0.1 --MYSQL_TCP_PORT=3306 --MYSQL_DB=zipkin --MYSQL_USER=root --MYSQL_PASS=Nov2014 --zipkin.collector.rabbitmq.addresses=localhost --zipkin.collector.rabbitmq.username=guest --zipkin.collector.rabbitmq.password=guestCopy the code

Persistence is something that the server does, not the client, so that’s it. Chen won’t test it anymore.

conclusion

In front of the introduction of so much, I do not know if you have looked carefully, Chen summed it up:

As a component of link tracing, Spring Cloud Sleuth only provides log collection and log printing without a visual UI
Zipkin provides powerful log tracking analysis, visualization, service dependency analysis and other related functions, combined with Spring Cloud Sleuth as a mainstream solution
The Zipkin production environment recommends switching the MQ transport mode, which has two advantages
- Preventing data Loss
- MQ is decoupled asynchronously, resulting in a significant performance improvement
Zipkin is stored in memory by default, MySQL is also available, but the performance of Elasticsearch gets worse as the amount of data increases. This is why Elasticsearch is recommended for production environments, as described in the next article.