preface

Article content output source: Pull hook education Java high salary training camp;

Spring Cloud is a one-stop microservices solution. Many companies are using Spring Cloud components. If we want to learn about the Spring Cloud microservices architecture, we need to learn about their components. These include: registry, load balancing, fuse processing, procedure call, gateway services, configuration center, message bus, call link, data monitoring, and more.

This article leads you to understand the micro service link tracking technology? When we are the number of micro service more and more, service calls more and more complex, between when a service problems, troubleshoot problems is very troublesome, so you need to link tracking technology, can clearly record the service call link, where the be clear at a glance, this problem can be convenient we fast positioning problem, get it quickly.

In this tutorial you will learn:

  1. Application scenarios of distributed link tracing technology
  2. core idea
  3. Spring Cloud Sleuth
  4. Zipkin
  5. ZipKin data persistence

Application scenarios of distributed link tracing technology

To support the growing volume of business, we will use microservices architecture to design our systems so that they can not only withstand the impact of traffic through cluster deployment, but also flexibly scale according to the business.

Therefore, in the microservice architecture, a request can be completed at least three or four service invocations, and at most it can span dozens or even hundreds of service nodes. The questions then arise:

1. How to dynamically display the service invocation link? (such as what other services A service calls – dependencies)

2. How to analyze and tune the bottleneck nodes in the service invocation link? (For example, A — >B — >C, where the C service takes A very long time)

3. How to quickly discover faults of service links?

This is the purpose and significance of the existence of distributed link tracking technology. If we can log each link node during the invocation processing of a request, and finally display the logs centrally, then we can monitor some indicators in the invocation link, for example, which service instance does the request reach? What is the status of the request being processed? What is the processing time? These can all be analyzed…

The monitoring technology based on this idea in distributed environment is distributed link tracing (full link tracing).

Distributed link tracking technology is mature, and there are many distributed link tracking solutions in the market, both at home and abroad. For example, Spring Cloud Sleuth + Twitter Zipkin, Alibaba’s Eagle Eye, Dianping’s CAT, Meituan’s Mtrace, JD’s Hydra, Sina’s Watchman, There’s also Apache Skywalking, which has been mentioned a lot lately.

core idea

Nature: Logging, as a complete technology, distributed link tracing also has its own theory and concept

In the microservice architecture, the invocation chain for request processing can be represented as a tree, as shown below

image-20200820111746438

Above describes a common invocation scenario, a request to the downstream through the gateway service route of micro service – 1, then the micro service micro service – 2-1 call, call again after get the result of micro service – 3, the final assembling service – 2 and micro – 3 results, through the gateway returned to the user in order to track the whole call link, certainly need log, Logging is the foundation, and there must be some theoretical concepts on top of that, The current mainstream distributed link Tracing technology/system is based on the concept of the paper “Dapper, a Large-scaleDistributed Systems Tracing Infrastructure” by Google. What are the core concepts involved in this paper? Let’s take a look at the previous service invocation

A request link is a request link, namely Trance, on which all services have the same link ID. One service at a time is span.

Trace: The tracing unit of service tracing is a process from the time when a customer sends a request and arrives at the boundary of the tracked system until the tracked system returns a response to the customer

Trace ID: In order to achieve request tracing, when a request is sent to the entry endpoint of a distributed system, the service tracing framework only needs to create a unique Trace ID for the request. Meanwhile, during the flow within the distributed system, the framework will retain the unique Trace ID until it is returned to the requester. A Trace is composed of one or more spans, and each Span has a SpanId in which TraceId is recorded. At the same time, there is also a SpanId named ParentId pointing to the SpanId of another Span, indicating the parent-child relationship, which essentially expresses the dependency relationship

SpanID: In order to count the time delay of each processing unit, when a request arrives at each service component, it is also marked with a unique SpanID to mark its start, process, and end. For each Span, it must have two start and end nodes. By recording the timestamp of the start Span and the end Span, the time delay for that Span can be calculated. In addition to the timestamp record, it can also contain some other metadata, such as the time name, request information, etc.

Each Span has a unique trace identifier Span ID, and several ordered spans form a trace.

Span can be considered as a log data structure, which records some log information at some special time points, such as timestamp, spanId, TraceId, parentIde, etc. Another concept is abstracted from Span, called event, and the core event is as follows

  • CS: Client send/start A client/consumer sends a request that describes the start of a span

  • SR: Server received/start The sr-CS is the network delay of sending requests

  • SS: Server send/fifinish SS-SR indicates the time consumed by the server

  • CR: Client received/ fiFINISHED Cr-SS indicates the time required for the response (network latency of the response)

Spring Cloud Sleuth (Tracing service framework) can track the invocation between services. Sleuth can record which services a service request goes through and the service processing duration, etc. Based on these, we can clarify the invocation relationship between various micro-services and conduct problem tracking analysis.

  • Time analysis: Use Sleuth to understand the time of sampling requests and analyze service performance issues (which service calls are time-consuming)

  • Link optimization: Discover frequently invoked services and optimize them accordingly

Sleuth records trail data by logging

We often use Spring Cloud Sleuth and Zipkin together, send Sleuth data information to Zipkin for aggregation, and use Zipkin to store and display data.

Spring Cloud Sleuth is used to collect links.

image-20200820112539597

Zipkin is used for link analysis, statistics and display, and persistence.

image-20200820112610329

Spring Cloud Sleuth

Next we used link tracing technology in the project. How did you do that? First we introduce Spring Cloud Sleuth. In the project log collection, we built the micro service architecture demo on the basis of transformation.

Rely on

Add Spring Cloud Sleuth dependencies to the following services. Or we can add dependencies directly to the parent POM file.

image-20200820131801309
<! -- Link Tracing --><dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-sleuth</artifactId>
</dependency>
Copy the code

configuration

Each microservice modifies the application.yml configuration file to add a log level

# Distributed link tracing
logging:
  level:
    org.springframework.web.servlet.DispatcherServlet: debug
    org.springframework.cloud.sleuth: debug
Copy the code

This completes the link log collection.

Zipkin

Spring Cloud Sleuth is completed above, and then we integrate ZipKin to make link tracking convenient for us to analyze and view.

Zipkin consists of Zipkin Server and Zipkin Client. Zipkin Server is an independent service, and Zipkin Client is a specific micro-service

Zipkin the service side

Let’s finish the server side first. The server side is where we look and analyze link tracing.

Rely on

Again, we need to add a ZipKin server dependency

<! -- Zipkin-server dependencies --><dependency>
    <groupId>io.zipkin.java</groupId>
    <artifactId>zipkin-server</artifactId>
The < version > 2.12.3 < / version > <exclusions> <! Log4j2 = log4j2 = log4j2 = log4j2 = log4j2 = log4j2 <exclusion>  <groupId>org.springframework.boot</groupId>  <artifactId>spring-boot-starter-log4j2</artifactId>  </exclusion>  </exclusions> </dependency>  <! -- Zipkin-server UI depends on coordinates --><dependency>  <groupId>io.zipkin.java</groupId>  <artifactId>zipkin-autoconfigure-ui</artifactId> The < version > 2.12.3 < / version ></dependency> Copy the code

configuration

In the configuration file we specify the startup port and turn off auto detection.

server:
  port: 8771

management:
  metrics:
 web:  server:  request:  autotime:  enabled: false # Disable automatic detection Copy the code

Start the class

In the startup class we add the @enableZipkinServer annotation to start the server.

@SpringBootApplication
@enableZipkinServer // Enable the Zipkin server functionpublic class ZipkinServerHw8771Application {

    public static void main(String[] args) {
 SpringApplication.run(ZipkinServerHw8771Application.class, args);  }  } Copy the code

The server is now configured.

ZipKin client

The client side is our existing services.

Rely on

Increased reliance on

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-zipkin</artifactId>
</dependency>
Copy the code

configuration

We also need to add configuration that points to our Zipkin server and in what form the message is sent to the server.

spring
  zipkin:
Base - url: http://127.0.0.1:8771# zipkin server request address
    sender:
      The Web client sends trace log data to the server via a network request, plus configuration
 # Kafka/Rabbit client passes trail log data to MQ for relay  type: web  sleuth:  sampler:  Sampling rate 1 means 100% of all data will be collected. By default, 0.1 means 10% of the requested trace data will be collected  # In the production environment, the number of requests is very large, so it is not necessary to collect and analyze the trace data of all requests. As the pressure of network including server side is relatively large, you can configure the sampling rate to collect a certain proportion of the trace data of requests for analysis  probability: 1 Copy the code

So we’re done with the client.

test

Now let’s start our microservices to test it out, starting the configuration center and then starting the other microservices.

At startup, my Zipkin startup kept reporting errors. Find a day of problems are not solved, and finally the Springboot version down to solve. Suspected version conflict. My final Zipkin POM file looks like this:

<? xml version="1.0" encoding="UTF-8"? ><project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
The < modelVersion > 4.0.0 < / modelVersion >    <parent>
 <groupId>org.springframework.boot</groupId>  <artifactId>spring-boot-starter-parent</artifactId> < version > 2.1.6. RELEASE < / version > </parent>  <groupId>cn.quellanan</groupId>  <artifactId>zipkin-server-hw-8771</artifactId> The < version > 1.0.0 < / version > <name>zipkin-server-hw-8771</name> <description> Link tracing zipkin server </description>  <dependencies> <! - web depend on -- -- > <dependency>  <groupId>org.springframework.boot</groupId>  <artifactId>spring-boot-starter-web</artifactId>  </dependency> <! -- Actuator helps you monitor and manage Spring Boot apps --> <dependency>  <groupId>org.springframework.boot</groupId>  <artifactId>spring-boot-starter-actuator</artifactId>  </dependency>   <dependency>  <groupId>io.zipkin.java</groupId>  <artifactId>zipkin-autoconfigure-ui</artifactId> The < version > 2.12.3 < / version > </dependency>  <dependency>  <groupId>io.zipkin.java</groupId>  <artifactId>zipkin-server</artifactId> The < version > 2.12.3 < / version > <exclusions>  <exclusion>  <groupId>org.springframework.boot</groupId>  <artifactId>spring-boot-starter-log4j2</artifactId>  </exclusion>  </exclusions>  </dependency>  </dependencies>   <build>  <plugins> <! Build plugins --> <plugin>  <groupId>org.apache.maven.plugins</groupId>  <artifactId>maven-compiler-plugin</artifactId>  <configuration>  <source< / > 8source>  <target>8</target>  <encoding>utf-8</encoding>  </configuration>  </plugin> <! -- Package plugin --> <plugin>  <groupId>org.springframework.boot</groupId>  <artifactId>spring-boot-maven-plugin</artifactId>  </plugin>  </plugins>  </build>  </project> Copy the code

The effect of the final full boot:

image-20200820202620968

Access:

http://localhost:8771/
Copy the code
image-20200820202728532

Let’s try accessing some interfaces.

http://localhost/api/code/create/[email protected]
Copy the code

Let’s look at the interface

image-20200820203215027
image-20200820203313274

You can also view details

image-20200820203547189

ZipKin data persistence

But there is a problem. Every time ZipKin restarts, the previous data is lost, so it needs to be persisted. ZipKin supports multiple ways of persisting.

Website:

https://github.com/openzipkin/zipkin
Copy the code

Find the script to create the table

https://github.com/openzipkin/zipkin/tree/master/zipkin-storage/mysql-v1/src/main/resources
Copy the code
image-20200820200522506
--
-- Copyright 2015-2019 The OpenZipkin Authors
--
Licensed under the Apache License, Version 2.0 (the"License"); you may not use this file except
-- in compliance with the License. You may obtain a copy of the License at
-- - http://www.apache.org/licenses/LICENSE-2.0-- -- Unless required by applicable law or agreed to in writing, software distributed under the License -- is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express -- or implied. See the License for the specific language governing permissions and limitations under -- the License. --  CREATE TABLE IF NOT EXISTS zipkin_spans (  `trace_id_high` BIGINT NOT NULL DEFAULT 0 COMMENT 'If non zero, this means the trace uses 128 bit traceIds instead of 64 bit'. `trace_id` BIGINT NOT NULL,  `id` BIGINT NOT NULL,  `name` VARCHAR(255) NOT NULL,  `remote_service_name` VARCHAR(255),  `parent_id` BIGINT,  `debug` BIT(1),  `start_ts` BIGINT COMMENT 'Span.timestamp(): epoch micros used for endTs query and to implement TTL'. `duration` BIGINT COMMENT 'Span.duration(): micros used for minDuration and maxDuration query'. PRIMARY KEY (`trace_id_high`, `trace_id`, `id`) ) ENGINE=InnoDB ROW_FORMAT=COMPRESSED CHARACTER SET=utf8 COLLATE utf8_general_ci;  ALTER TABLE zipkin_spans ADD INDEX(`trace_id_high`, `trace_id`) COMMENT 'for getTracesByIds'; ALTER TABLE zipkin_spans ADD INDEX(`name`) COMMENT 'for getTraces and getSpanNames'; ALTER TABLE zipkin_spans ADD INDEX(`remote_service_name`) COMMENT 'for getTraces and getRemoteServiceNames'; ALTER TABLE zipkin_spans ADD INDEX(`start_ts`) COMMENT 'for getTraces ordering and range';  CREATE TABLE IF NOT EXISTS zipkin_annotations (  `trace_id_high` BIGINT NOT NULL DEFAULT 0 COMMENT 'If non zero, this means the trace uses 128 bit traceIds instead of 64 bit'. `trace_id` BIGINT NOT NULL COMMENT 'coincides with zipkin_spans.trace_id'. `span_id` BIGINT NOT NULL COMMENT 'coincides with zipkin_spans.id'. `a_key` VARCHAR(255) NOT NULL COMMENT 'BinaryAnnotation.key or Annotation.value if type == -1'. `a_value` BLOB COMMENT 'BinaryAnnotation.value(), which must be smaller than 64KB'. `a_type` INT NOT NULL COMMENT 'BinaryAnnotation.type() or -1 if Annotation'. `a_timestamp` BIGINT COMMENT 'Used to implement TTL; Annotation.timestamp or zipkin_spans.timestamp'. `endpoint_ipv4` INT COMMENT 'Null when Binary/Annotation.endpoint is null'. `endpoint_ipv6` BINARY(16) COMMENT 'Null when Binary/Annotation.endpoint is null, or no IPv6 address'. `endpoint_port` SMALLINT COMMENT 'Null when Binary/Annotation.endpoint is null'. `endpoint_service_name` VARCHAR(255) COMMENT 'Null when Binary/Annotation.endpoint is null' ) ENGINE=InnoDB ROW_FORMAT=COMPRESSED CHARACTER SET=utf8 COLLATE utf8_general_ci;  ALTER TABLE zipkin_annotations ADD UNIQUE KEY(`trace_id_high`, `trace_id`, `span_id`, `a_key`, `a_timestamp`) COMMENT 'Ignore insert on duplicate'; ALTER TABLE zipkin_annotations ADD INDEX(`trace_id_high`, `trace_id`, `span_id`) COMMENT 'for joining with zipkin_spans'; ALTER TABLE zipkin_annotations ADD INDEX(`trace_id_high`, `trace_id`) COMMENT 'for getTraces/ByIds'; ALTER TABLE zipkin_annotations ADD INDEX(`endpoint_service_name`) COMMENT 'for getTraces and getServiceNames'; ALTER TABLE zipkin_annotations ADD INDEX(`a_type`) COMMENT 'for getTraces and autocomplete values'; ALTER TABLE zipkin_annotations ADD INDEX(`a_key`) COMMENT 'for getTraces and autocomplete values'; ALTER TABLE zipkin_annotations ADD INDEX(`trace_id`, `span_id`, `a_key`) COMMENT 'for dependencies job';  CREATE TABLE IF NOT EXISTS zipkin_dependencies (  `day` DATE NOT NULL,  `parent` VARCHAR(255) NOT NULL,  `child` VARCHAR(255) NOT NULL,  `call_count` BIGINT,  `error_count` BIGINT,  PRIMARY KEY (`day`, `parent`, `child`) ) ENGINE=InnoDB ROW_FORMAT=COMPRESSED CHARACTER SET=utf8 COLLATE utf8_general_ci;  Copy the code

The created table is as follows:

image-20200820200616841

Introduction of depend on

<! Mysql persistence dependency --><dependency>
    <groupId>io.zipkin.java</groupId>
    <artifactId>zipkin-autoconfigure-storage-mysql</artifactId>
The < version > 2.12.3 < / version ></dependency> <dependency>  <groupId>mysql</groupId>  <artifactId>mysql-connector-java</artifactId> </dependency> <dependency>  <groupId>com.alibaba</groupId>  <artifactId>druid-spring-boot-starter</artifactId> The < version > 1.1.10 < / version ></dependency> <! -- Transaction control required to operate database --><dependency>  <groupId>org.springframework</groupId>  <artifactId>spring-tx</artifactId>  </dependency> <dependency>  <groupId>org.springframework</groupId>  <artifactId>spring-jdbc</artifactId> </dependency> Copy the code

Increase the configuration

Added the configuration of connecting to a database

spring:
  application:
    name: zipkin-server-8771
  sleuth:
    enabled: false
 datasource:  driver-class-name: com.mysql.jdbc.Driver url: jdbc:mysql://localhost:3306/zipkin? serverTimezone=UTC&useUnicode=true&characterEncoding=utf-8&useSSL=false&allowMultiQueries=true  username: root  password: 123456  druid:  initialSize: 10  minIdle: 10  maxActive: 30  maxWait: 50000 # specify mysql as the zipkin persistence medium zipkin:  storage:  type: mysql Copy the code

test

Once configured, let’s reboot and try again

image-20200820204442475
image-20200820204505810

There is a lot of data in the table for link tracing, and after we restart the server, the previous data is still there, to achieve data persistence.

conclusion

Link tracing can play a very important role in microservice monitoring, which can quickly locate faulty services. We will also encounter them in actual projects, so let’s get on with it

This article is formatted using MDNICE