Service Mesh is a key part of cloud native today. Ant has implemented Service Mesh in the production environment on a large scale, but the overall Service Mesh transformation in the industry is still not high. In this process, the support of protocols is the most basic part. The multi-protocol extended development framework provided by MOSN aims to reduce the cost of Mesh reconstruction for scenarios using private protocols and help services to be implemented quickly.

MOSN is a high-performance network agent developed by Ant. MOSN is mainly used for Sidecar on the data side of Service Mesh. Service Mesh is a hot topic in the cloud native direction in recent years. Its main purpose is to build an infrastructure layer that is responsible for the communication between services. It is mainly to have a Sidecar co-deployed with service applications to realize the basic capabilities of various middleware, realize the standardization of infrastructure and the decoupling of business logic, and achieve the rapid evolution of basic capabilities without business awareness. At present, many domestic companies have begun to embrace Service Mesh, and some enterprises cooperating with Ant, such as China Citic Bank and Jiangxi Agricultural Credit, have also completed Mesh transformation based on MOSN.

The purpose of the Service Mesh architecture is to reduce the impact of infrastructure upgrades on the business, but how to smooth the transition from traditional microservices architecture to Service Mesh architecture is also a very challenging task. There are many details involved here. However, the most basic problem is that when we perform gray Mesh transformation, the Mesh nodes need to be able to maintain normal communication with the non-mesh nodes. Different companies choose different communication protocols, which directly leads to the selection of technology, the Sidecar must be able to support the protocol used. Some of the most widely used protocols may continue to be supported, and some may be customized by the company itself, so it is inevitable that Sidecar’s extended capabilities will be used for secondary development to support proprietary protocols.

Multi-protocol extension framework

Speaking of the Service Mesh Sidecar, one cannot fail to mention the Envoy, a widely used Service Mesh Sidecar proxy. Envoy’s extension framework enables developers to do secondary extensions, and many of the protocols that Envoy currently supports are based on its extension framework. In the Envoy extension framework, to extend a protocol, refer to the HTTP protocol process in Envoy, including the 4-layer Filter implementation codec part and the Connection Manager part. Implement 7 layers of filters on top of Connection Manager to support additional business scaling, routing capabilities, and Connection pooling capabilities of upstreams. It can be seen that the process of a protocol almost runs through various modules, and the cost of implementing a protocol extension is relatively high.

Take a look at the MOSN framework. MOSN can be divided into four layers for protocol processing. In addition to the network layer, which obtains data from network IO, MOSN can also be divided into protocol layer, Stream layer, and proxy layer. The Protocol layer is responsible for encoding and decoding related to protocol parsing, and is responsible for parsing data streams into protocol frames that MOSN can understand, or encoding protocol frames into binary streams. The stream layer is responsible for handling different request types, initializing the context of the request, association events, association between the response and the request, and upstream pool-related processing. The details of the processing are different for different protocols. The proxy layer is a protocol-independent proxy forwarding layer, which is responsible for routing and balancing of requests. It also has seven layers of expansion capabilities for different business implementations. According to this architecture, we can see that the core of protocol processing lies in the Protocol layer and stream layer. Compared to Envoy designs, routing, seven-layer extensions, etc., are capable of multi-protocol reuse. But at the same time can also see the stream layer involves more details, the implementation of the difficulty is relatively large, so MOSN on this basis and put forward a multi-protocol extension framework for simplifying the implementation of the protocol.

MOSN multi-protocol framework is mainly aimed at the stream layer multiplexing expansion ability, in the MOSN protocol processing layer design, network layer and proxy layer is protocol independent reusable design, if the Stream layer can also be multiplexed, Then the protocol implementation only needs to pay attention to the protocol layer decoding itself, the implementation difficulty will be greatly reduced. Does the Stream layer have the ability of reuse? We think it can be used for most protocols, especially RPC protocol. First, we abstract the protocol and define it as the XProtocol interface, representing any protocol. Each protocol implementation is to implement an XProtocol interface, which includes the basic codec interface, some special request response construction interface (such as heartbeat, exceptions), as well as protocol model (such as HTTP-like Pingpong model, common RPC multiplexing model, etc.), and protocol matching interface. All XProtocol implementations are related by the XProtocol Engine. In addition to specifying which protocol to use for processing through configuration, protocols that implement protocol-matching interfaces can be automatically identified based on request characteristics. Then we also perform uniform abstraction for the protocol frames parsed by XProtocol, including multiplexing related interfaces, protocol type judgment (request or response, or control frames like Goaway, etc.), Requests can be subdivided into heartbeat requests, oneway requests with no response, etc.), support for data modification of protocol frames (Header/Body modification), and unified status code management mapping, etc.

With the layered protocol processing mechanism of MOSN, and the multi-protocol extension framework with the abstract definition of XProtocol and XFrame as the core, we can completely process protocol based on interface in stream layer, and different protocol extension implementor only needs to focus on the protocol encoding and decoding itself. As well as the simple interface adaptation of the codec result, the access can be completed in the MOSN, thus obtaining the support of various general MOSN capabilities, such as traffic limiting extension, routing traffic diversion, etc. This is considerably simplified compared to the extension protocol implementation section in Envoy. Of course, MOSN, the multi-protocol framework, can not meet all protocol situations, but for most of the RPC protocols we see at present, on the basis of the proxy layer with the seven-layer Stream filter extension, can be well satisfied.

Practical cases

The following MOSN in community partners in Dubbo protocol landing case to understand the MOSN multi-protocol extension in detail. Much of the code here was also contributed by students in the MOSN community. In this case, in addition to requiring the protocol to support Dubbo, it also wants to use basic extension capabilities such as traffic limiting and routing capabilities such as blue-green grouping. The control plane selected is Istio, which is used for dynamic configuration delivery. How are these requirements implemented in MOSN?

The first part is protocol analysis, which adopts dubbo-Go framework based on open source to implement the protocol, and encapsulate XProtocol and XFrame model of MOSN based on Dubbo-Go. Current limiting, xDS and other capabilities directly reuse MOSN existing implementation, no additional implementation. However, the problem is that the route configuration dynamically delivered by Istio is RELATED to HTTP. The route configuration model of HTTP is still different from that of Dubbo, and the cost of modifying Istio is high, so another layer of extension is made here. StreamFilters based on MOSN layer 7 are customized to the Dubbo protocol before route matching to meet the HTTP route format. There are two main points, one is HTTP Host, using Dubbo Service corresponding to Host, the other is HTTP Path, this part directly add a default Path for wildcard. In the Filter extension, the Labels of the machine are also retrieved and added to the Header to match the route’s blue-green grouping function. Through the cooperation of MOSN Filter extension, we realize the standard Dubbo protocol support, and meet the function of using HTTP route configuration to meet the Dubbo route.

Plug-in extension

The introduction of the MOSN multi-protocol framework shows that if a scenario needs to access a custom protocol that the MOSN does not support, it needs to be extended and compiled with the MOSN framework code to obtain an MOSN that supports the custom protocol. Although already try to simplify the complexity of the protocol implementation, but still there are some problems, such as our commercial version of the code, different partners, corresponding to different scenarios, use different protocols, so with the development of the business, the agreement will be more and more, this part of the relevant code corresponding to the development and maintenance cost will be bigger. This is ok, but there are also some scenarios where the client may not want to submit the relevant implementation code to the MOSN repository for certain requirements, and the commercial version of the code, unlike open source, may not be able to directly hand over the source code to the client, so there will be problems with compiling. In order to solve this contradiction, but also to further simplify the protocol extension, MOSN also made the ability to extend the protocol based on the plug-in pattern.

The MOSN provides the unified plug-in extension framework capability. After the MOSN is independently compiled into binary, the plug-in mechanism is used to dynamically load different protocol extension plug-ins to obtain the corresponding protocol support capability. The implementation of the protocol extension can also be maintained independently as a plug-in, and it is even possible to support non-GO language extensions, such as protocol extensions that connect to other language implementations based on WASM extension capabilities. MOSN plug-in extension capability has two modes, one is based on Go Plugin mechanism extension, the other is based on WASM extension.

First, let’s look at the mechanics of the Go Plugin. This is a loading capability of SO provided by Go. We can compile the code written by Go into SO, which can then be loaded by other Go files. However, this mechanism has some major limitations. Firstly, the Go environment compiled by the main program and the extension plug-in must be consistent, including the VERSION of Go, GoPath and other environment variables. The other thing is that the main program and the extension must depend on the same library, which is Hash accurate, meaning that the two library interfaces are not compatible, but must be identical, which is a big limitation. Because we expect that the MOSN code and the protocol extension code are maintained independently of each other, the protocol extension code needs to rely on the MOSN framework. According to the mechanism of Go Plugin, every code change of MOSN framework requires the plug-in code to be updated synchronously and then recompile the plug-in, which is too troublesome.

To this end, we disintegrate the MOSN framework, split out some relatively stable interfaces and general capabilities, as the main MOSN program and protocol extension of the common dependence of the basis, such as XProtocol and XFrame related interface definition separately defined in the API library. Then, when the plug-in loads, you just need to register the corresponding interfaces with the MOSN framework. Because API definitions and tools change relatively little, protocol extensions rely on API definitions that change from MOSN framework to MOSN framework, minimizing the situation that plug-in code must be updated when MOSN framework code is updated. For the compilation environment this is easy to do, we provide a unified docker environment for compilation, just need to make MOSN and plug-in based on the same docker compilation can be. Plug-in extensions implemented through the Go Plugin have the same result as merging and compiling the code directly, except that the protocol extension code can be maintained independently.

Again, take a look at WASM’s extension mechanism. WASM is a developed, efficient, secure, and community-standard extension capability. WASM is theoretically language independent, and there is a widely accepted ABI specification for network proxy scenarios, because MOSN also supports WASM extension capabilities. It can also be used for protocol extension of MOSN.

WASM extensions for MOSN can be found in WebAssembly practices in MOSN – Basic Framework.

The protocol extension based on WASM is different from the protocol extension mentioned earlier. WASM plug-ins need to implement the proxy-WASM ABI standard, not the MOSN multi-protocol framework, which means that the WASM plug-in can theoretically be used for other applications that follow the proxy-WASM ABI specification. On the MOSN side, a glue layer named WASM Protocol is implemented. This glue layer encapsulates the MOSN multi-protocol framework and interacts with the WASM plug-in through the WASM ABI. From the perspective of the MOSN multi-protocol framework, you see an XProtocol implementation encapsulated by WASM Protocol, and the multi-protocol framework does not understand the WASM ABI’s interactions. Different from GoPlugin extension, WASM framework of MOSN is still in the initial stage, WASM protocol extension is only POC, and for stability, performance and other factors, has not been formally applied in the production environment, still need more optimization and testing.

Looking forward to

Finally talk about MOSN protocol support in the future some prospects and plans, mainly including more types of protocol support, such as streaming, duplex gRPC protocol, message type protocol such as Kafka, MQ, there is WASM protocol extension in the production environment implementation available.

Recommended Reading of the Week

  • MOSN sub-project Layotto: Open a new chapter of service grid + application runtime

  • Guide to developing Wasm protocol plug-ins

  • Protocol Extension Base On Wasm

  • WebAssembly practices in MOSN – Basic Framework

More articles please scan code to pay attention to “financial level distributed architecture” public number