Internal DDD external API – The design concept of external API

Author: Li Zheng Qunar net engineer

He joined Qunar in February 2017. The current focus is on domain service governance, standardization of domain capabilities based on API governance. Focus on addressing business complexity through domain, modeling, and awareness. Expect to use DDD drive, reduce system complexity, improve team efficiency.

preface

Internal DDD and external API are the business remodeling architecture design concepts that the business RESEARCH and development team of Qunar airline ticket destination business group focuses on launching in Q3 2020. In Q3 2020, Qunar has made some progress in the field of API standardization on the basis of the past. This article is mainly to share with you the experience in this regard.

What is internal DDD and external API? This is a practical principle of using DDD as a domain design, microservice design concept in our business development field, a simple way of saying that domains interact with APIS.

A brief history of qunar.com API development

Qunar actually has a number of mature tools in the API space, mostly wikis, YAPI and Swagger.

Wiki is the most traditional API hosting tool of Qunar. All the standard apis provided by the company’s early public platforms, such as payment center, are presented through wiki. The advantage of Wiki is relatively high degree of freedom. It is not restricted by various specifications and can be modified at will. It can be customized to meet the demands of some interface readers and has good natural security. The disadvantages are more, for example, because there is no certain interface specification for constraint, the definition of the interface is varied, and there are various very personalized conventions. The same conventions also have different presentation modes. Some tend to give nested presentation, while others tend to give modular presentation. The synchronization between interface definitions and interfaces is all dependent on specification and engineer awareness, and it’s easy to get wiki and code mismatches.

Because of this, we can say that wikis can provide readable apis from the point of view of providing apis, but wikis can’t provide trusted, reliable apis.

The second solution that supports Qunar is YAPI, which emerged in the context of the front-end separation architecture that began in 2016. The separation of the front and rear ends makes front-end engineering more independent. YAPI also became an open source project of Qunar in 2018. YAPI, as a front-end interface platform developed by students, remains one of the cornerstones of Qunar.com’s API solution. The purpose of YAPI is to “jointly maintain an interface definition and connect the front and back ends.”

The difference between YAPI and wiki can be illustrated in the following figure:

As you can see from the figure above, YAPI forces documentation to be consistent with the interface definition by supporting the pile-testing hook, by running the API interface and testing whether it gets the expected results. This implementation is very clever, but it also depends on the improvement of the test environment and the requirement of API testing, if these two cannot be guaranteed, then the result of such a clever pattern is still the classification and synchronization of the interface and interface documents. It’s not that different from using a wiki, but YAPI has a more RestFul API management platform, which is more constrained by the definition of an API than a wiki.

To prevent the interface definition from being out of sync with the interface (and the document from being out of sync with the code), the back-end students introduced Swagger for binding to code using annotations, and made some Maven-based extensions on top of swagger. Enables specifications written with Swagger to update interface changes to the YAPI platform through engineering state changes such as Maven commands and releases, addressing mismatches between interface documents and interface implementations.

Swagger -> YAPI works, but because there are fewer projects to write API documentation using Swagger and few people know about the tool, Swagger -> YAPI is not widely used within the company.

2. Why qunar promotes API standardization

In 2020, due to the epidemic, Qunar started a vigorous “internal performance training” action, which includes business DDD remodeling in the core business area, hardware cost saving, and API internal realization reconstruction, etc.

In doing these things, we faced some difficulties:

1. DDD refactoring needs to be invoked with interfaces outside the domain. How many interfaces is appropriate for a domain to provide with an external domain? 10? 30? 50? If you provide too many apis, does that mean that the API is not standard enough and not of high quality?

2. The hardware cost includes the cost of physical and virtual machine nodes, offline log costs and real-time log costs. How many physical and virtual machine nodes are appropriate? How much of the system’s offline log and real-time log output is appropriate? This part depends on the number of system apis and the availability of API access.

3. After the API internal functionality refactoring, which downstream accesses the API? After using tools like QueryDiff to regression the interface itself, which downstream systems do you need to regression test to be on the safe side? This relies heavily on the governance of the upstream and downstream relationships of the API, and the standardization of the API for automated tests based on governance results.

From the above examples, we can see that the re-emergence of API as an important technical improvement point is the overall need for internal DDD, external API system architecture concept, hardware cost management, and platform service refactoring.

3. Theoretical basis of API standardization

The theoretical basis for API standardization came from the concept of indirect communication of systems proposed by Jeff Bezos in 2002, which has since been gradually fleshed out into a concept called SOA Maturity.

Bezos expressed the idea of Amazon system interface like this:

In 2002, Bezos suddenly issued a company-wide directive.

– Starting today, all teams will provide data and functionality as a service interface.

– Teams must communicate with each other through interfaces.

– No other forms of interoperation are allowed: no direct links, no direct reading of other teams’ data, no shared memory, no back doors of any kind. The only permissible means of communication is to invoke the service over the network.

– Specific implementation technology is not specified, HTTP, Corba, PubSub, custom protocols are available.

– All service interfaces must be designed to be exposed from the start, with no exceptions. This means that the interface is designed with the default that it can be opened to outsiders and is not negotiable.

– If you don’t follow the rules, you’re fired.

The SOA maturity model developed from Jeff Bezos’ decision above has relatively clear requirements for API standardization, including:

• A detailed specification that defines interfaces, methods, parameters, types, and descriptions

• Define criteria,

• Development tool plug-in support: IDEA, Eclipse plug-in identification of specifications

• Publish system support for the API

From the above list of REQUIREMENTS for the API, we need the API to have the following characteristics:

1. Specification easy to understand

2. Component easy access

3. Easy to use grammar

4. Easy to manage

5. Platform easy to apply

There is also a general principle: all functions should be based on the existing Qunar infrastructure, without duplication of wheels.

Thus, the overall solution of QUNar API standardization is obtained:

Among them, THE API storage platform YAPI is ready-made, which can be modified slightly to support the demand of service API standardization. The application tree management platform is also ready-made. The existing system of Qunar.com is called Portal, and the application domain management platform Qtracer is also ready-made. In view of this requirement, the function has been extended. Gateway and open platform are also available. This time, the system integration of the two has been done. Qdoc-annotation and qDOC-maven-plugin are the only newly created plug-ins and tools.

4. Specific implementation process of API standardization

4.1 Specification Easy to understand

The first thing we need to do is to develop a set of API writing specifications. This writing specification mainly defines how we organize the annotations or annotations to the API specification, which is equivalent to the entity object relationship design of a business system Domain dimension. We designed the FOLLOWING elements for the API:

The following terms have been defined by the specification Development Committee:

Domain/Group

For the overall system boundary of a business, a domain may contain multiple applications (Appcodes). Interfaces exposed in a domain are exposed through Appcodes, which belong to the open part. Note: Domains can map to tier 3 / tier 4 nodes in the application tree.

Application (AppCode)

Application refers to the Appcode in Qunar system. An Appcode is an application. There is a special case where one set of code (a Git project) deploies multiple Appcodes, which we consider multiple applications.

Service (service)

One AppCode can provide multiple services. There are different representations for different service implementations (DUbbo/HTTP).

• For Dubbo, it makes sense that a service interface corresponds to a service.

• For HTTP, based on Spring MVC (restful) thinking, each service corresponds to a controller. This requires different controllers to distinguish a function group.

Interface

This is the most fine-grained functional dimension and is a true subset of services. There are different representations for different service implementations (DUbbo/HTTP).

• For Dubbo, this is a method provided under a service.

• For HTTP, this is a RequestMapping (method) provided under a controller.

Parameter

Parameters are an important part of an interface. There are different representations for different service implementations (DUbbo/HTTP).

• In the case of Dubbo, it’s easy to understand that the java-based method signature is just an input, including the contents of the RpcContext (which is equivalent to QTraceContext except in special cases).

• For HTTP, the parameter consists of four parts, the first part being passed by the parameter list in the URL, and the second part being put into the HTTP data in the POST. Usually these two parts are based on Spring MVC and can be defined via @requestParam. The third part is the cookie information (request header), and the fourth part is equivalent to QTraceContext.

(Type)

Both in and out arguments need to be identified by type. A type is a descriptive structure definition. There are different representations for different service implementations (DUbbo/HTTP).

• In the case of Dubbo, obviously, the Java type definition is type here. This is not allowed in type. You must identify the actual type (specific class definition, or JSON Schema definition) of the corresponding Object.

• In the case of HTTP, the input parameters are usually of basic or jSON-like types (including collections, etc., described in JSON), and the output parameters are described in JSON. HTTP is a weak type definition system and is unfriendly to documents, so we specify that the input and output parameter types are defined as JSON Schema.

After the specification of terms, we began to formulate the corresponding annotations and notes. In the first issue, we first standardized the corresponding annotations:

Domain (@qdomaindoc)

Application (@QAppCodeDoc) service (@QServiceDoc) interface (@qInterfaceDoc) which has parameter annotations and returns a value whether the parameter (@qparamdocs) parameter (@qparamDoc) can be deleted. Model attribute annotation (@qParamModelProperty) type (@qTypeDoc) is equivalent to @RequestParam. Currently not implemented, Description of the returned value status code (@qResponses) Description of the returned value status code (@qResponse) Description of the returned value Data (@qResponseDocs) Exception (@qExceptionDoc) extension (@qExtensionDoc)

For each annotation we’ll also give an annotation parameter description:

Service (@qServiceDoc) Describes the purpose of a service:

And as you can see, the specification terms and the annotations that QDoc gives out take into account the engineering context that the students of Qunar are using, for example Domain, AppCode, those are common words in the engineering context of Qunar, and when you look at those words, you can think about what they’re doing. These terms, which are close to the common vocabulary of engineers, make it easy for engineers to understand the QDoc specification without having to read a large section of the specification, and are cheap to learn and quick to use. This is a major difference between qunar’s specification and third-party API tools such as swagger2.0, swagger3.0, smart-doc, and so on. It is also an important reason why we adopted custom API terms in support of the OpenAPI3.0 specification.

After completing the annotation API normalization component definition, the next step is to develop the corresponding tools, plug-ins, and access the tools.

4.2 Easy Access of Components

In order to facilitate the access of business line engineers and prevent them from additional development in the process of accessing QDoc, the jar package and Maven plug-in are adopted. The access steps are very simple:

Service Access Procedure

1. To access qDOC service, we need to introduce maven plug-in into POM to complete the release of QDOC.

2. If you need to write an API document using annotations, you need the QDOC-Annotation package to complete the writing.

3. Specific dependency introduction :(Maven Plugin)

<plugin>
    <groupId>com.qunar.fd</groupId>
    <artifactId>qdoc-maven-plugin</artifactId>
    <version>${qdoc.maven.version}</version>
</plugin>
QDoc Annotation
 
<dependency>
    <groupId>com.qunar.fd</groupId>
    <artifactId>qdoc-annotation</artifactId>
    <version>${qdoc.annotation.version}</version>
    <scope>provided</scope>
</dependency>
Copy the code

4.3 Easy syntax

QDoc’s syntax is divided into two parts: the Git engineering side syntax and the code API side syntax, which together form qunar’s standardized API.

Git project syntax is as follows. Both QDomainDoc and QAppcodeDoc use this method.

Annotations other than QDomainDoc and QAppcodeDoc are used in combination with API code but are not intrusive:

@qinterfaceDoc (type = "dubbo", define = "send the verification code to the user,dubbo interface ", desc =" check whether the phone number is the user's, send the verification code to the user ", scene = "Send the verification code to the user ", Notice = "Intranet use ", since =" authors = "fanrong. Zhao ", Url = "dubbo_send") @qparamdocs ({@qparamdoc (name = "paramV1", value = "first parameter ", paramType = "form", DataType = "String", notice = "must be String", paramExample = "username"), @qparamdoc (name = "paramV2", value =" second parameter ", Required = false, paramType = "form", dataType = "Boolean", notice = "must be of type Boolean", }) @qResponses ({@qresponse (code = -1, message = "system exception "), @qResponse (code = 200, Message = "success ")}) @qresponseDocs ({@qresponseDoc (bindValueName = "data", description =" first argument ", BindValueType = "object", propertys = {@qProperty (type = "String", name = "name", desc = "description 1"), @qproperty (type = "int", name = "age", desc = "description 2"), @ QProperty (type = "com. Qunar. Fd. Qdoc. Qdocexample. Vo. ExampleResultVO" and name = "food", desc = "3")}). }) public ApiResponseV2<Food> example0(@RequestParam String paramV1,@RequestParam boolean paramV2) { return null; }Copy the code

According to the feedback from qunar engineers who are already in use, it takes only 5 minutes to write an API of moderate complexity through annotations.

4.4 Easy to Manage

Qunar QDoc supports two main methods for API synchronization: Maven command synchronization and release system synchronization. Maven command synchronization meets the requirements of Design2doc and is more flexible. It also inherits qunar’s swagger-YAPI heritage. Synchronization through the release system is a major part of QDoc’s work. This work solves all problems with the interface being out of sync with the Master version, including interface creation, update, rollback, etc.

We can see that:

Service Commissioning Procedure

1. Release system opening display

   In qunar’s application tree, there is a service list under the corresponding AppCode. In the service list, click “Open” to complete the integration and opening of the application tree.

2. CM release integration

   In the release system of Qunar.com, the service market shall be opened under the corresponding Appcode.

After the Portal is opened and published online, the document update operation is automatically triggered. In this case, our documents can be seen in the application tree.

In three simple steps, we integrated QDoc with the publishing system.

4.5 Platform Easy application

No matter how well the previous work is done, if there is no good interactive display platform to support it, then it is very painful for users, and it is difficult to popularize the system with high application cost. Our final choice is that YAPI is embedded in the App management platform and bound with Appcode management to give team managers a one-stop management experience.

YAPI can refer to the open source version hellosean1025. Making. IO/YAPI /

However, API management in the Appcode dimension can only facilitate one-stop API management for engineers, but cannot positively help API management in the Domain dimension, which is very important in DDD business remodeling. We are currently working on improving the Domain dimension API management platform:

Of course, once we have established the Domain dimension API management system, we can also use qunar’s mature gateway system to externalize our API:

With this introduction, Qunar’s entire process from API annotations in code to domain APIS on the open platform is introduced through the QDoc tool.

So what difficulties did we encounter in the project?

5. Difficulties in implementation of API standardization

1. Where do development resources come from

Ticket destination where the project is network business group of business development of TC project, there is no direct team resources to support, the resources needed to cross the CM, YAPI pilot project access platform, tool development, business development blocks, almost covers all company engineer team, the project with the method of the open source project management in the company, The project team is established, the project is independently established, and the resources of various teams are combined to complete the project. The project completion is also a big challenge for the project management personnel.

2, Design First or Code First

As the initial stage of the project, it is impossible to support both Design First and Code First. Through investigation, we found that Design First is more suitable for the formulation of non-domain dimension interfaces, such as a front and rear end interface. Such support interfaces are not universal. It is redeveloped as the page changes and is usually not reused. Code First is better suited to help maintain high quality interfaces supported by the Domain dimension for long-term maintenance. The original intention of our API standardization work this time is to help DDD business remodeling to do a good job in the standardized maintenance of Domain dimension interfaces. So we chose to support the interface provision of Code First First.

3, Annotation or Comments

Before we do QDoc, used in the company have certain degrees of API standard tools include swagger2.0, swagger3.0, smart – doc, on this point by investigation, we found that the There are significantly more engineers using Swagger related tools with annotations than using smart-Doc class annotations, and even though the annotation approach has the problem of having too many annotations piled up on top of the API code, which can lead to ugly code, However, since there are significantly more users using annotations, we decided that QDoc would prefer to support annotations. This does not mean that QDoc will not support comments in the future.

Support the full OpenAPI3.0 specification or a subset of OpenAPI3.0 specification

The question comes from the fact that we already have the mature YAPI platform which is open source in 2018, so is it enough that we meet YAPI’s interface requirements? Whether it is not necessary to fully meet the requirements of the OpenAPI3.0 specification, our answer is no. Platforms continue to evolve and YAPI will have its day. When YAPI does, we will choose between other platforms that support the OpenAPI3.0 specification. Knife4j is one API platform in our line of sight right now.

6. Project results

DDD and API finally meet. The core domain, support domain, and general domain apis have been connected.

1. Core domain API standardization effectively protects the domain from intrusion

2. Generic domain API is conducive to the realization of generic domain platform

3. Support domain APi-ization reduces development

There is no duplication of the wheel, everything is formed by reason:

Summary and follow-up plan of the project

From the previous introduction, we can see that the API construction of Qunar is hierarchical. The innermost layer is the API of AppCode dimension, the outer layer is the business interface dimension, and the outer layer is the domain dimension. The open interfaces are configured to the open platform through the gateway, forming the interface management and use of different dimensions.

The following things need to be done in AppCode dimension interface management:

• Complete IDEA interface compliance plug-in;

• Support annotation mode;

• Support client/server code generation;

• Investigate the domain interface dimension: if there is a better alternative than YAPI, Knife4j is always on the horizon;

• More importantly, through the development of API standardization, the idea of reverse promotion of DDD leads to a deeper understanding across all business areas.

END