The following article is from Alibaba middleware, the author of Dong Yiquan

Note: this article is adapted from dong Yiquan’s speech at Dubbo Community Developer Day in Shanghai.

1, the origin

Why did Ctrip introduce Dubbo? In fact, since the end of 2013, Ctrip has mainly used THE SOA microservice framework based on HTTP protocol. This framework is internally developed by Ctrip, and the overall framework has not undergone major reconstruction in the past six years.

Limited by the original design, the framework itself is not very scalable, making it difficult for users to extend some functions by themselves. In addition, due to the HTTP protocol, only one request can be processed at a time for a connection. In the case of high concurrency, resources such as server connection count and thread pool are strained, which affects the performance of request processing.

Dubbo, as a high-performance RPC framework, is not only a well-known open source product in the industry, but also its overall excellent architecture design and data transmission mode can solve these problems mentioned above. Just in the second half of 2017, Alibaba announced the resumption of Dubbo maintenance. For these reasons, our team decided to introduce Dubbo to Ctrip.

Dubbo’s first step on the ground

The first step in implementing Dubbo as a new service framework in your company is to address the issues of service governance and monitoring.

Service governance

In terms of service governance, Ctrip’s existing SOA framework has a complete set of service registry and service governance system. For service registries, Apache Zookeeper is probably the most common. We use Artemis, a registry developed by Eureka based on Netflix’s open source.

Artemis’s architecture is a decentralized peer-to-peer cluster. Each node has the same status and no master or slave. The service instance maintains a long connection to any node in the cluster, sending registration and heartbeat messages. The nodes receiving the information distribute the information to other nodes to ensure data consistency between clusters. The client also receives a list of service instances pushed by the registry over a long connection.

In terms of service data models, we directly reused the data models of existing SOA services. As shown in the figure, the core service model corresponds to an interface in Dubbo. Multiple services can be contained within an application, and a service can be deployed on multiple servers. We refer to the service applications running on each server as service instances.

All services need to be registered with the governance system before going live. After registration, the system assigns a unique identifier, ServiceID, to the user. This ServiceID is sent to the registry to identify the instance when a service instance is registered, and the client needs this ID to obtain the list of instances of the specified service.

Since Dubbo itself does not have a ServiceID design, the problem here is how to pass the ServiceID information corresponding to an interface to the registry.

Our approach is to add a serviceId parameter to the Service and Reference configuration. The implementation of ArtemisServiceRegistry reads this parameter and passes it to the registry. This allows normal interaction with the registry.

Service monitoring

In terms of service monitoring, we have mainly done two parts: monitoring at the level of statistics and monitoring at the level of call chain.

Statistics refer to the periodic summary of various service invocation data, such as call volume, response time, request body and response body size, request exceptions, and so on. This data is summarized on the client side and the server side in minute granularity, and then output to the Dashboard kanban. We also add labels to the data, such as Service ID, server IP, method called, and so on. Users can easily query the monitoring data they need.

In the monitoring service invocation chain, we use CAT. CAT is a real-time application monitoring platform developed by Meituan-Dianping. It records the entire process of a request through a tree of Transaction and Event nodes.

We added burying points of CAT Transaction and Event on both the client and server of Dubbo, recording information such as called service, SDK version, service time, and caller id. In addition, the context information of CAT service invocation is passed to the server through Dubbo’s Attachment, so that the monitoring data of the client and the server can be connected.

It can be very convenient to query when removing obstacles. In the diagram, the outer layer we see is the monitoring data recorded by the client. After expanding at the invocation initiation point, we can see the corresponding monitoring data on the server side.

Release).

After solving the two problems of service governance and monitoring docking, we completed an initial localization of Dubbo on Ctrip. In March 2018, we released the first usable version of the customized version of Dubbo on Ctrip. We need a new name for the product before it can be officially released. Since it’s Ctrip plus Dubbo, we’ll call this custom version CDubbo.

3. CDubbo function expansion

In addition to the basic system docking, we also implemented a series of functional extensions to CDubbo, mainly including the following five points: Callback enhancement, serialization extension, fuse and request test tool. Let me introduce you one by one.

The Callback to enhance

First, let’s take a look at this code. Is there anything wrong with the code?

There is a DemoService in this code. The argument to the callbackDemo method is an interface. The following Demo class calls the callbackDemo method in foo and bar, respectively.

If you have used Callback, you should know that foo is called correctly, and bar returns an error when called repeatedly. The client can only create one instance of the same Callback interface.

But what’s wrong with that? Let’s look at a scenario like this.

A user made a request on the page to inquire about airline tickets. After receiving the request, the site server invokes the ticket query service at the back end. Given that this call can be time-consuming, callback is used on the interface to pass back and forth the actual query results. It is then pushed to the client by the site server via websocket-like technology.

So here’s the question. When the site server receives the callback data, it needs to know which user it corresponds to which call so that the data can be correctly pushed to the user. However, for a globally unique callback interface instance, obtaining this request context information is more difficult. The interface definition and implementation need to be prepared in advance. It may be necessary to introduce additional global objects to hold this part of context information.

To address this problem, we added the Stream feature in CDubbo. As before, let’s look at the code.

How is this code different from the previous code?

First, the argument to the callback interface is replaced with a StreamContext. Also, instead of receiving a callback from a globally unique instance, it receives a callback from an anonymous class, and instead of a single method, there are three methods, onNext, onError, and onCompleted. This allows the caller in the anonymous class to retrieve the context of the original request through the closure. Is the experience better?

So how is Stream implemented? Let’s take a look at this picture.

On the client side, when the client makes a call with a Stream, it creates a StreamContext with the streamContext.create method. Although it is said to create, it is actually in a global StreamContext with a unique StreamID and the actual processing logic corresponding to the callback. When a request is sent, this StreamID is sent to the server. The server also carries StreamID when it initiates a callback. This allows the client to know which StreamContext this callback corresponds to.

4. Serialization extension

Some business departments of Ctrip used the request data model written by the Contract of Google Protocol Buffer when they developed SOA services. Google PB requires that data models generated by contract be serialized using the PB serializer. Recommended: Everything you need to know about Java serialization.

To make it easier for them to migrate THEIR SOA services to Dubbo, we also added support for the GooglePB serialization approach to Dubbo. In order to facilitate users to expand, we added an extension interface to the implementation of PB serializer, allowing users to continue to add data compression functions in the periphery. It is not difficult to implement the overall serializer, but it is important to note that since the Dubbo service can only expose one serialization method to the outside world, this serialization method should be compatible with all Java data types.

PB happens to be one of those serializers that can only serialize data types generated by their own contracts. Therefore, we will fallback to the default hessian for serialization when encountering unsupported data types.

5. Request a circuit breaker

I believe you should be familiar with the circuit breaker. When a large number of requests fail or time out on the client or server, the system automatically executes the Fail-fast logic and directly returns an error message instead of continuing to send and receive requests. Recommended reading: Distributed service avalanche protection fuse, Hystrix theory + Practice.

Here we’re using one of the more established solutions in the industry: Netflix’s open source Hystrix. It not only contains the function of fusing, but also supports concurrency control, isolation between different calls and other functions. Errors in a single call do not affect other calls. You can customize each function as required. CDubbo’s servers and clients integrate Hystrix to handle requests for exceptions and avoid an avalanche effect.

6. Service testing tools

Dubbo is an RPC protocol that uses binary data stream to transmit. Service testing is a difficult problem to operate. To get testers to test a Dubbo service without writing code, we have to solve three problems: how to write test requests, how to send test requests, and how to view response data.

The first is how to structure the request. The question really falls into two parts. One is how the user constructs the request without writing any code. Considering that many testers are familiar with Restful Service testing, we decided to use JSON format to represent the request data.

Isn’t it difficult for a tester to construct a request from a blank JSON? So we still want to be able to make the user understand the data model of the request. Spring Boot returns JSON data in a minute.

Although we are using Dubbo 2.5.10, this functionality is already available in Dubbo 2.7.3. So we copied this part of the code and then extended it to store the metadata information for the service in a global context. We also added an internal operation, $serviceMeta, via Filter in CDubbo to expose the service metadata information. This metadata information includes method lists, parameter lists for each method, data models for parameters, and so on.

After the user retrieves the data model by calling internal operations, a basic JSON structure can be generated. The user can then easily construct a test request simply by populating the structure with the actual test data.

Then, how do you send the edited request to the server? Because there is no model code, the call cannot be made directly. Dubbo provides a great tool for GenericService, a generalization call.

The request body is sent to the server via a generalization call, and the Map returned by the server is serialized into JSON and displayed to the tester. The whole test process is complete. It also solves the problem of how to view the response data.

For the convenience of users, we developed a service test platform. It allows users to directly select services and instances, write and send test requests. In addition, in order to facilitate users to carry out automated tests, we also packaged this part of the function into a JAR package and released it.

In fact, in the process of doing testing tools, but also encountered a little problem. This can be done by converting from JSON to Map to POJO. However, as mentioned earlier, some objects are generated through contracts similar to Google Protobuf. They are not pure POJOs and cannot be translated directly.

So, we extend the generalization call. First, for this custom serializer, we allow the user to define the format conversion implementation from the data object to JSON. Second, we added the ability to convert JSON and Google PB objects to and from Dubbo when the server handles the generalization call. Both of these extensions are now incorporated into Dubbo’s codebase and released with version 2.7.3.

Fortress test gateway

Speaking of service-only testing, there are times when we want to test a service in a production environment, especially when an application is released. In Ctrip, there is a test method called Fortress test, which means that during the application release process, the publishing system will first select a server as fortress, and release the new version of the application to fortress.

The user then sends a request to the fortress via a specific test method to verify that the functionality of the new version of the application works. Since fortification tests are performed before the fortification machine is pulled into the cluster, it is necessary to enable the client to identify a fortification test request and forward it to the specified fortification service instance.

This can be done by route, but it requires the client to know a lot about the details of forwarding, and the functionality integrated into the SDK can cause some problems with subsequent upgrades and maintenance. So we developed a service gateway specifically for fortress testing.

When a client recognizes that the context of the current request contains the Fortress request identity, it forwards the Dubbo request to the pre-configured test gateway. The gateway parses the service request, determines which service it corresponds to, and then finds the service’s bastion machine and forwards the request. After the service completes processing the request, the gateway also forwards the response data back to the caller.

Unlike normal HTTP gateways, Dubbo’s service gateway takes into account an additional request method, the callback mentioned earlier. Because callback is a request from the server, the entire process is different from a normal request from the client.

The gateway binds the connection initiated by the client to the connection between the gateway and the server, and records the ID of the last request to be returned. This allows accurate routing of callback requests and responses.

8. Follow-up function planning

As of today, there are 27 releases of CDubbo. Many of Ctrip’s business units are already connected to Dubbo. In the future, CDubbo will extend more functions, such as request traffic limiting and authentication and authorization. We hope to contribute more features to the open source community in the future.

Author: Dong Yiquan, ctrip framework research and development department technical expert. Currently, I am responsible for the research and development of ctrip’s servitization framework.

Read more on my blog:

1.Java JVM, Collections, Multithreading, new features series tutorials

2.Spring MVC, Spring Boot, Spring Cloud series tutorials

3.Maven, Git, Eclipse, Intellij IDEA series tools tutorial

4.Java, backend, architecture, Alibaba and other big factory latest interview questions

Life is good. See you tomorrow