1. Background knowledge, requirements description, and common dependencies

1.1. Background & Requirement Description

According to the official documentation of Spring Cloud, it is a complete microservice system, and users can use Spring Cloud to quickly build their own microservice system. So how exactly does Spring Cloud work? What components does it have?

In the Spring-Cloud-Commons component, there is an abstract interface to all of the component functionality provided by Spring Cloud by default, and some default implementations. The current 2020.0.x (iiford according to the previous naming convention), also known as Spring-Cloud-Commons-3.0.x includes:

  • Service discovery:DiscoveryClientTo discover microservices from the registry.
  • The service registry:ServiceRegistry, register microservices to the registry.
  • Load balancing:LoadBalancerClientThe client invokes load balancing. Among them,Retry strategyfromSpring - the cloud - Commons - 2.2.6Added load balancing abstractions.
  • The circuit breaker:CircuitBreakerUnder what circumstances will the service be disconnected and degraded
  • Calling HTTP clients: Internal RPC calls are HTTP calls

Then, a complete microservice system generally includes:

  1. Unified gateway
  2. Configuration center
  3. Full link monitoring and monitoring center

In the previous series, we upgraded Spring Cloud to Hoxton with the following components:

  1. Registration Center: Eureka
  2. Client package: OpenFeign
  3. Client LoadBalancer: Spring Cloud LoadBalancer
  4. Circuit breakers and isolation: Resilience4J

And achieve the following functions:

Registry related:

  1. All clusters share the same common Eureka cluster.
  2. Implementation of instances quickly up and down.

Microservice instance related:

  1. Different clusters do not call each other through instancesmetamapIn thezoneConfiguration to distinguish between instances of different clusters. instance-onlymetamapIn thezoneInstances that are configured the same can call each other.
  2. Calls between microservices are still based on open-feign, with retries, only GET requests with 4xx and 5XX status codes are retried (retries on 4XX because there is no new API for the old instance during rolling upgrade, retries can send requests to the new instance)
  3. A microservice calls other microservices A and B from different thread pools. And the thread pools that call different instances are different. That is, instance level thread isolation
  4. Implement instance + method level fuses. The default instance level fuses are too rough. Problems with some interfaces on the instance, but not all interfaces.
  5. Load balancing polling algorithm, requests and requests need to be isolated, do not share the same position, so that the retry after a failed request is the same as the original failed instance.
  6. The same is true for asynchronous calls to WebFlux that are not servlets.

Gateway related:

  1. throughmetamapIn thezoneConfigure to identify the cluster and forward requests only to microservice instances in the same cluster
  2. Forwarding requests, with retries, only GET requests with status codes 4xx and 5XX are retried
  3. Different instances of different microservices are thread isolated
  4. Implement instance level fuses.
  5. Load balancing polling algorithm, requests and requests need to be isolated, do not share the same position, so that the retry after a failed request is the same as the original failed instance
  6. Implement the request body modification (possibly the request needs to be decrypted, the request body needs to print the log, so it will involve the request body modification)

In the process of subsequent use, development and online operation, we also encountered some problems:

  1. The rapid growth of business at certain moments, such as the 6.30 shopping spree, the Double 11 promotion, the Double 12 Shopping Festival, as well as during the legal holidays, is hard to predict. Although there is a capacity expansion strategy based on instance CPU load, there is still a lag, and there is still a surge of traffic that causes core services (such as orders) to be unavailable for a period of time (maybe 5 to 30 minutes). The main reason is that the system is under a lot of pressure and many requests are queued. After the queuing time is too long, the response time expires when these requests are processed. As a result, the requests that could be normally processed cannot be processed. And the user’s behavior is that the harder the order is, the more it needs to refresh and retry, which further increases the system stress, which is an avalanche. With instance-level thread isolation, we limit the maximum concurrency for each instance to call other microservices, but there are queues because of the wait queues. At the same time, due to the lack of traffic limiting in the API gateway and the asynchronous responsiveness of the Spring Cloud Gateway, many requests are backlogged, further exacerbating the avalanche. So here, we consider these cases, redesigning thread isolation and adding API gateway flow limiting.
  2. Microservice discovery, in order to be compatible with cloud native applications such as K8s features in the future, the best service discovery is multiple sources
  3. Link monitoring and indicator monitoring are two sets of systems, which are troublesome to use and cost is high. Can they be optimized into one set?

Next, we will upgrade the existing dependencies and expand and extend the existing functions to form a complete Spring Cloud microservice system and monitoring system.

1.2. Write common dependencies

This project code, please refer to: github.com/HashZhang/s…

This time we abstract out more concrete dependencies for various scenarios. Generally, our entire project will include:

  1. Common toolkit dependencies: Generally all projects rely on some third party tool library, such as Lombok, Guava, etc. Put common toolkit dependencies into these dependencies.
  2. Traditional Servlet Synchronous microservice dependencies: Dependency management for microservices that do not apply reactive programming but use the traditional Web Servlet pattern.
  3. Responsive microservice dependencies: Dependency management for microservices implemented based on responsive programming by the Project Reactor. Responsive programming is a big trend, and the Spring community is pushing it hard. This can be seen in various Spring components, especially Spring Cloud components. Spring-cloud-commons also provides synchronous and asynchronous interfaces for each component abstraction of microservices. Part of our project also uses responsive programming.

Why is microserver abstracting from reactive and traditional servlets?

  1. First of all, Spring officially advocates responsive programming, especially since the release of Hoxton, spring-Cloud-Commons abstracts all public interfaces from the traditional synchronous version as well as the Project Reactor based asynchronous version. And in terms of implementation, the bottom layer of the default synchronous version is also converted to synchronous implementation through the Project Reactor. As you can see, asynchrony is already a trend.
  2. However, asynchronous learning requires a certain threshold, and while traditional projects are mostly synchronous, some new components or microservices can be implemented using responsiveness.
  3. Reactive and synchronous dependencies are not completely compatible, and while synchronous and asynchronous co-exist within the same project, this is not officially recommended (it actually starts the WebServer as a Servlet WebServer). And projects like The Spring Cloud Gateway implementation are completely incompatible, so it’s best to keep them separate.
  4. Why isn’t reactive programming popular? Mainly because of database IO, not NIO. Whether it’s Java’s own Future framework, Spring WebFlux, or vert. x, they’re all non-blocking Ractor model-based frameworks (the latter two are both implemented using Netty). In blocking programming, every request needs to be handled by a thread, and if I/O is blocked, that thread will be blocked. But in non-blocking programming, based on reactive programming, threads are not blocked and can handle other requests. As a simple example, if there is only one thread pool, when the request comes in, the thread pool needs to read the DATABASE IO. This IO is NIO non-blocking IO. Then write the request data to the database connection and return it directly. Then the database returns the data, and the link’s Selector is ready with a Read event, and the data processing (equivalent to a callback) is Read through the thread pool, not necessarily on the same thread as before. That way, instead of waiting for the database to return, the thread can process other requests directly. In this case, even if the SQL execution of one service takes a long time, the execution of other services will not be affected. The foundation of all this, however, is that IO must be non-blocking IO, known as NIO (or AIO). There is no NIO in official JDBC, only BIO implementation (because Oracle officially provides the maintenance, but Oracle believes that the Project Loom mentioned below is a solution to the hardware inefficiencies of synchronous style code, so it has not been published). Instead of having the thread write the request to the link and return directly, it must wait for the response. The solution, however, is to use another thread pool to process the database request and wait for the callback to come back. In this case, business thread pool A passes the database BIO request to thread pool B for processing, reads the data, and then passes the rest of the business logic to A. So A doesn’t have to block and can handle other requests. However, there is still A case where all threads of B are blocked and the queue is full and requests of A are blocked because the execution of A business SQL takes A long time. This is not A perfect implementation. To be truly perfect, you need JDBC to implement NIO.
  5. What is the future of Java responsive programming? Could there be another solution? Personally, I would like to explore reactive programming for WebFlux if you are interested, but it is not necessary to use reactive programming. While asynchronous programming is the trend and responsive programming is gaining traction, Java has another solution to the performance bottleneck of synchronous coding: Project Loom. Project Loom lets you continue to write code in a synchronous style, using non-blocking lightweight virtual threads underneath. Network IO doesn’t block system threads, but sychronized and local file IO still do. The main problem was solved, however. So, this series will focus on synchronous style code and apis.

1.2.1. The parent

pom.xml

<? The XML version = "1.0" encoding = "utf-8"? > < project XMLNS: xsi = "http://www.w3.org/2001/XMLSchema-instance" XMLNS = "http://maven.apache.org/POM/4.0.0" Xsi: schemaLocation = "http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd" > < the parent > < the groupId > org. Springframework. Boot < / groupId > < artifactId > spring - the boot - starter - parent < / artifactId > < version > 2.4.4 < / version > < / parent > < modelVersion > 4.0.0 < / modelVersion > < groupId > com. Making. Hashjang < / groupId > < artifactId > spring - cloud - iiford < / artifactId > < packaging > pom < / packaging > < version > 1.0 - the SNAPSHOT < / version > < properties > < project version > 1.0 - the SNAPSHOT < / project. The version > < / properties > < dependencies > <! --> <dependency> <groupId> <artifactId>junit</artifactId> <scope>test</scope> </dependency> <! <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> <scope>test</scope> </dependency> <! - mockito extension, Mock final class --> <dependency> <groupId>org.mockito</groupId> <artifactId>mockito </artifactId> <version>3.6.28</version> <scope>test</scope> </dependency> </dependencies> <dependencyManagement> <dependencies> <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-dependencies</artifactId> <version>2020.0.2</version> <type> POm </type> <scope>import</scope> </dependency> </dependencies> </dependencies>  <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> The < version > 3.6.1 < / version > < configuration > <! Best use JDK 12 or later to compile, 11.0.7 is sometimes buggy for spring-cloud-gateway. -- Although the website says it has been resolved, <source>11</source> <target>11</target> </configuration> </plugin> </plugins> </build> </project>Copy the code

1.2.2. Common base dependency packages

pom.xml

<? The XML version = "1.0" encoding = "utf-8"? > < project XMLNS = "http://maven.apache.org/POM/4.0.0" XMLNS: xsi = "http://www.w3.org/2001/XMLSchema-instance" Xsi: schemaLocation = "http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd" > < the parent > < artifactId > spring - cloud - iiford < / artifactId > < groupId > com. Making. Hashjang < / groupId > < version > 1.0 - the SNAPSHOT < / version > < / parent > < modelVersion > 4.0.0 < / modelVersion > < artifactId > spring - the cloud - iiford - common < / artifactId > < properties > < guava version > 30.1.1 - jre < / guava. Version > < fastjson. Version > 1.2.75 < / fastjson version > < disruptor version > 3.4.2 < / disruptor version > < jaxb. Version > 2.3.1 < / jaxb version > < activation version > 1.1.1 < / activation. The version > < / properties > < dependencies > <! Integration of internal cache framework, Caffeine --> <! > <dependency> <groupId>com.github. Ben-manes. Caffeine </groupId> <artifactId>caffeine</artifactId> </dependency> <! Guava </groupId> <artifactId> Guava </artifactId> <version>${guava.version}</version> </dependency> <! <dependency> <groupId>com.alibaba</groupId> <artifactId>fastjson</artifactId> <version>${fastjson.version}</version> </dependency> <! > <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter</artifactId> <exclusions> <exclusion> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-logging</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-log4j2</artifactId> </dependency> <! --> <dependency> <groupId>org.projectlombok</groupId> <artifactId>lombok</artifactId> </dependency> <! --log4j2 dependency required for asynchronous logging, Lmax </groupId> <artifactId> Disruptor </artifactId> Disruptor </artifactId> <version>${disruptor.version}</version> </dependency> <! -- The modularity features of JDK 9 and later caused javax.xml not to load automatically, Javax.xml. bind</groupId> <artifactId> JAXB-api </artifactId> <version>${jaxb.version}</version> </dependency> <dependency> <groupId>com.sun.xml.bind</groupId> <artifactId>jaxb-impl</artifactId> <version>${jaxb.version}</version> </dependency> <dependency> <groupId>org.glassfish.jaxb</groupId> <artifactId>jaxb-runtime</artifactId> <version>${jaxb.version}</version> </dependency> <dependency> <groupId>com.sun.xml.bind</groupId> <artifactId>jaxb-xjc</artifactId> <version>${jaxb.version}</version> </dependency> <dependency> <groupId>javax.activation</groupId> <artifactId>activation</artifactId> <version>${activation.version}</version> </dependency> </dependencies> </project>Copy the code

Caffeine’s highly efficient local Cache framework has the same interface design as Guava-Cache and can be easily upgraded. Performance comparison tests for Guava-Cache, ConcurrentHashMap, ElasticSearchMap, Collision, Ehcache, etc., are available in Caffeine’s source code and are provided to Yahoo’s test library. Caffeine simulates a user scenario that is similar to real life. Furthermore, caffeine uses a number of papers to implement caching for different scenarios, such as:

  1. Adaptive Replacement Cache: www.cs.cmu.edu/~15-440/REA…

2. Quadruply – segmented LRU:www.cs.cornell.edu/~qhuang/pap… 3. 2 Queue:www.tedunangst.com/flak/post/2… 4. Segmented LRU: www.is.kyusan-u.ac.jp/~chengk/pub… 5. Filtering – -based Buffer Cache: storageconference. Us / 2017 / cca shut…

So we chose Caffeine as our local caching framework

Reference: github.com/ben-manes/c…

2. guava

Guava is Google’s Java library, and while we don’t use guava for native caches, there are many other elements of Guava that we use frequently.

Reference: guava. Dev/releases/sn…

3. Internal serialization changed from Fastjson to Jackson

Json libraries generally need to be warmed up, and we’ll see how. Some of the internal serialization in our project is Fastjson serialization, but fastjson has not been updated for a long time, and there are many issues. In order to avoid future problems (or vulnerabilities, or performance problems) and increase the possible problem points online, we have made compatibility in this version. Fastjson will be removed in the next version. Details on how to do this will follow.

4. The log format is Log4j2

Due to its asynchronous logging feature, printing a large number of service logs does not become a performance bottleneck. However, it is still not recommended that the online environment output location information such as lines of code, for reasons and solutions described below. Since the log4J2 asynchronous logging feature relies on Disruptor, you also need to add the disruptor dependency.

Reference:

  • logging.apache.org/log4j/2.x/
  • lmax-exchange.github.io/disruptor/

5. JDK 9+ compatibility needs to add some dependencies

The modularity of javax.xml since JDK 9 caused it to not load automatically, and many of the dependencies in the project required the module, so they were added manually.

1.2.3. Servlet microservice public dependencies

pom.xml

<? The XML version = "1.0" encoding = "utf-8"? > < project XMLNS = "http://maven.apache.org/POM/4.0.0" XMLNS: xsi = "http://www.w3.org/2001/XMLSchema-instance" Xsi: schemaLocation = "http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd" > < the parent > < artifactId > spring - cloud - iiford < / artifactId > < groupId > com. Making. Hashjang < / groupId > < version > 1.0 - the SNAPSHOT < / version > < / parent > < modelVersion > 4.0.0 < / modelVersion > < artifactId > spring - the cloud - iiford - service - common < / artifactId > < dependencies > <dependency> <groupId>com.github.hashjang</groupId> <artifactId>spring-cloud-iiford-common</artifactId> <version>${project.version}</version> </dependency> <! - registered to eureka - > < the dependency > < groupId > org. Springframework. Cloud < / groupId > <artifactId>spring-cloud-starter-netflix-eureka-client</artifactId> </dependency> <! -- No Ribbon, With Spring Cloud LoadBalancer - > < the dependency > < groupId > org. Springframework. Cloud < / groupId > <artifactId>spring-cloud-loadbalancer</artifactId> </dependency> <! Micro - service calls between mainly by openfeign encapsulation API - > < the dependency > < groupId > org. Springframework. Cloud < / groupId > <artifactId>spring-cloud-starter-openfeign</artifactId> </dependency> <! -- Resilience4j as a retry, circuit breaker, concurrency limit, Resilience4j </groupId> <artifactId> Resilience4j-spring-cloud2 </artifactId> </dependency> <! -- https://mvnrepository.com/artifact/io.github.resilience4j/resilience4j-feign --> <dependency> <groupId>io.github.resilience4j</groupId> <artifactId>resilience4j-feign</artifactId> </dependency> <! -- Operating interface --> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-actuator</artifactId> </dependency> <! - records invocation path - > < the dependency > < groupId > org. Springframework. Cloud < / groupId > <artifactId>spring-cloud-starter-sleuth</artifactId> </dependency> <! --> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-actuator</artifactId> </dependency> <! -- Expose HTTP interface, servlet framework adopts NIO undertow, pay attention to direct memory usage, <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> <exclusions> <exclusion> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-tomcat</artifactId>  </exclusion> </exclusions> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-undertow</artifactId> </dependency> </dependencies> </project>Copy the code

There are dependencies involved in this, which we’ll use later.

1.2.4. Webflux microservice-related dependencies

For Webflux responsive micro-services, you can replace spring-boot-starter-web with spring-boot-starter- Webflux

Reference: pom. XML