Li zheng
Qunar engineer
I joined Qunar in February 2017. Currently, it focuses on domain service governance and standardization of domain capabilities based on API governance. Committed to addressing business complexity through domain, modeling, and perception. DDD driver is expected to reduce system complexity and improve team efficiency.
preface
Internal DDD and external API are business remolding architecture design concepts that qunar air ticket destination business group business RESEARCH and development team will focus on in Q3 2020. In Q3 2020, Qunar has made some progress in the field of API standardization on the basis of the past. This article is mainly to share the experience in this respect with you.
What is internal DDD and external API? This is the practice principle of DDD as the concept of domain design and micro-service design in our business research and development field. It is an easy to understand way to say that the domain uses API to interact with each other.
1. Brief history of qunar API development
Qunar has quite a few mature tools in the API field, including Wiki, YAPI and Swagger.
Wiki is the most traditional API bearer of Qunar, and all the standard APIS provided by the company’s early public platforms, such as payment center, are presented in the way of wiki. The advantage of Wiki is that it has relatively high degree of freedom, is not constrained by various norms, and can be modified at will. It can meet the demands of some interface readers with very personalized features. It has good natural security, and non-employees have no permission to access it. There are more disadvantages. For example, because there is no interface specification to constrain the interface, there are various definition ways of interface, and there are various personalized convention ways, and the same convention way, the presentation way is different, some tend to give nested presentation, some tend to give modular presentation. The synchronization between interface definitions and interfaces is entirely dependent on the specification and the engineer’s self-awareness, and wikis and code mismatches can easily occur.
Because of this, we can say that in terms of providing an API, wikis can provide a readable API but wikis cannot provide a credible, reliable API.
The second solution that underpins Qunar is YAPI, the background of YAPI’s emergence is the front and back end separation architecture that started in 2016. Separation of front and rear ends makes front-end engineering more independent. YAPI also became an open source project for Qunar in 2018. YAPI, as the interface platform developed by the front-end students, is still one of the cornerstones of Qunar API solutions until now. The purpose of YAPI is to “jointly maintain an interface definition and connect the front and back ends”.
The difference between YAPI and wiki can be illustrated as follows:
As you can see from the figure above, YAPI forces the documentation to conform to the definition of the interface by supporting piling to test the grip, and by running the API interface to test whether it gets the desired result. This is a very clever implementation, but it also depends on the perfection of the test environment and the rigid requirements of THE API testing. If these are not guaranteed, then the result of such a clever pattern is still the classification and synchronization of the interface and the interface document. Not much different from wiki, but YAPI has a more RestFul API specification interface management platform, which is a constraint compared to wiki definition of API.
In order to prevent the interface definition from being out of sync with the interface (documents and code are out of sync), the backend students introduced swagger which is used to bind the code with annotations, and made some Maven-based extensions based on Swagger. Enable the specification written by Swagger to update the interface changes to YAPI platform through maven commands and project state changes such as publishing, and solve the problem of interface documents and interface implementation mismatch.
Swagger -> YAPI has achieved some success, but because there are few projects using Swagger to document API, and few people know there is swagger -> YAPI, The swagger -> YAPI approach was not widely used within the company.
2. Why does Qunar promote the construction of API standardization
In 2020, due to the epidemic, Qunar started a vigorous “internal strength” action, including business DDD remodeling in core business areas, hardware cost saving, API internal implementation reconstruction, etc.
We faced some difficulties in doing these things:
1. DDD refactoring requires calls to external domains via interfaces. How many interfaces are appropriate between a domain and external domains? 10? 30? 50? If you provide too many apis, does that mean that the API is not standard or high quality?
2. The hardware cost includes the cost of physical machine and VIRTUAL machine node, offline log and real-time log, so how many physical machine and virtual machine nodes is appropriate? How much offline and real-time log output is appropriate for the system? Part of this depends on the number of system apis and access to them.
3. After the internal functions of the API are reconfigured, which downstream users access the API? After regression against the interface itself using tools such as QueryDiff, which downstream systems do you need to regression test for insurance? This aspect relies heavily on the governance of the upstream and downstream relationships of THE API, and the implementation of automated testing based on the results of governance depends on the standardization of the API.
As we can see from the above examples, API’s re-emergence as an important technical improvement point is the overall need for internal DDD and external API system architecture concept, hardware cost management, and platform service reconstruction.
3. Theoretical basis of API standardization
The theoretical basis of API standardization comes from the concept of indirect oral system proposed by Jeff Bezos in 2002, which has been gradually enriched into a concept called SOA maturity.
Here’s how Bezos describes the idea of Amazon’s system interface:
In 2002, Bezos suddenly issued a directive to the entire company.
– Starting today, all teams will provide data and functionality in the form of service interfaces.
– Teams must communicate through interfaces.
– No other form of interoperability is allowed: no direct linking, no direct reading of other teams’ data, no shared memory, no backdoors of any kind. The only permitted means of communication is to invoke the service over the network.
– The specific implementation technology is not specified, HTTP, Corba, PubSub, custom protocol can be.
– All service interfaces must be publicly available from the outset, without exception. That is, the interface was designed with the implicit assumption that it would be open to outsiders, with no room for bargaining.
– If you don’t follow the rules above, you’re fired.
The SOA maturity model developed from Jeff Bezos’ decision above has relatively clear requirements for API standardization, including:
- Defines detailed specifications for interfaces, methods, parameters, types, and descriptions
- Define evaluation criteria,
- Development tool plug-in support: IDEA, Eclipse plug-in recognition of specifications
- Publish system support for apis
To summarize the above list of API requirements, we need the API to have the following characteristics:
- Easy to understand specification
- Easy Component Access
- Syntax easy to use
- Easy management of execution
- Easy application of platform
There is also a general principle: all functions are based on existing Qunar infrastructure, not duplicate wheels.
Thus, the overall solution for Qunar API standardization is obtained:
Among them, the API storage platform YAPI is ready-made, which can be slightly modified to support the appeal of service API standardization. The application tree management platform is also ready-made. Qunar’s existing system is called Portal, and the application domain management platform Qtracer is also ready-made. Just for this demand to do the function of the expansion, gateway and open platform is ready-made, this time to do the system integration of both. It can be said that only qdoc-Annotation and qdoc-Maven-plugin are newly created plug-ins and tools.
4. Specific implementation process of API standardization
4.1 Specifications are easy to understand
The first thing we need to do is to develop a set of API writing specifications. This writing specification mainly defines how we organize annotations or annotations for API specifications, which is equivalent to the design of entity object relationships in the Domain dimension of a business system. We have roughly designed the following elements for the API:
The following terms have been regulated through the Specification Development Committee:
Domain/Group
According to the overall system boundary of a certain business, a domain may contain multiple appcodes. The exposed interfaces of a domain are exposed through appcode and belong to the open part. Note: Domains can be mapped to level 3 / Level 4 nodes of the application tree.
Application (AppCode)
Application refers to appcode in Qunar system, and an Appcode is an application. There is a special case where a single set of code (a Git project) deploys multiple AppCodes, which we consider multiple applications.
Service
Under the same Appcode, multiple services can be provided. There are different representations for different service implementations (Dubbo/HTTP).
- For Dubbo, it is easy to understand that a service interface corresponds to a service.
- For HTTP, based on Spring MVC (restful) thinking, each service corresponds to a Controller. (This requires different controllers to distinguish between a functional group).
Interface
This is the most fine-grained functional dimension and is a true subset of Services. There are different representations for different service implementations (Dubbo/HTTP).
- For dubbo, this is a method provided under a service.
- For HTTP, this is RequestMapping (Method) provided under a Controller.
Parameter
Parameters are an important part of an interface. There are different representations for different service implementations (Dubbo/HTTP).
- In the case of Dubbo, it’s easy to understand that java-based method signatures are just the input parameters, including the contents of the RpcContext (equivalent to QTraceContext except in special cases).
- For HTTP, the parameter consists of four parts, the first part is passed by the parameter list in the URL, and the second part is placed in the HTTP data in the POST. Usually these two parts are based on Spring MVC and can be defined via @requestParam. The third part is cookie information (Request header), and the fourth part is equivalent to QTraceContext.
Type (type)
Both input and output parameters need to be identified by type. A type is a descriptive structural definition. There are different representations for different service implementations (Dubbo/HTTP).
- For Dubbo, it’s obvious that the Java type definition is type here. At the same time, it is necessary to explain a special case Object. For Object, it represents any structure, which is not allowed in Type. The actual type of the corresponding Object (concrete class definition or JSON Schema definition) must be identified.
- For HTTP, incoming parameters are generally of primitive or JSON-like types (including collections, which are described in JSON), and outgoing parameters are described in JSON. HTTP is a weak type definition system that is unfriendly to documents, so we specify that the input and output parameter types are defined as JSON Schema.
After standardizing the terms, we started to make corresponding annotations and comments. In the first issue, we standardized the corresponding annotations:
Domain (@qDomaindoc)
Application (@qAppCodeDoc) service (@QServiceDoc) interface (@qInterfaceDoc) there is a parameter annotation and return value can delete parameter (@qParamdocs) parameter (@qparamDoc) The Model parameter (@qParamModel) annotation (@qParamModelProperty) type (@qTypeDoc) of the Model property is equivalent to @requestParam. Currently not implemented, Description of subsequent support for Returned value Status codes (@qresponses) Description of returned value Status codes (@qResponse) Description of returned value data (@qresponsedocs) Returned value data (@qresponsedoc) Exception (@qExceptionDoc) Extension (@qExtensionDoc)
We also give instructions for each annotation parameter:
Services (@qServiceDoc)
Describe the purpose of a service:
As you can see, the specification terms and annotations given by QDoc take full account of the engineering context commonly used by Qunar students, such as Domain and AppCode, which are commonly used in Qunar engineering context. Once you see these words, you can think of what they are used for. These terms, which are close to the common vocabulary of engineers, make it easy for engineers to understand the QDoc specification without having to read long paragraphs of instructions, and are cheap to learn and quick to use. This is the main difference between qunar specification and swagger2.0, swagger3.0, smart-doc and other third-party API tools. It is also an important reason why we adopt custom API terms under the premise of supporting OpenAPI3.0 specification.
After completing the annotation based API standardized component definition, the next step is the development of corresponding tools, plug-ins and tool access.
4.2 Easy Component Access
In order to facilitate the access of line of business engineers and minimize the extra development amount of line of business development engineers during the access of QDoc, THE jar package and Maven plug-in are adopted. The access steps are very simple:
Service Access Procedure
- To access the QDOC service, the corresponding Maven plug-in needs to be introduced into the POM to complete the qDOC publishing.
- If API documentation needs to be annotated, the QDOC-Annotation package is required to complete the writing.
- Specific dependency introduction :(Maven Plugin)
<plugin>
<groupId>com.qunar.fd</groupId>
<artifactId>qdoc-maven-plugin</artifactId>
<version>${qdoc.maven.version}</version>
</plugin>
QDoc Annotation
<dependency>
<groupId>com.qunar.fd</groupId>
<artifactId>qdoc-annotation</artifactId>
<version>${qdoc.annotation.version}</version>
<scope>provided</scope>
</dependency>
Copy the code4.3 Syntax easy to use
QDoc syntax is divided into two parts: one is git project-side syntax, the other is code API side syntax, the two parts together constitute qunar standardization API.
Git engineering syntax is as follows, QDomainDoc and QAppcodeDoc are both used in this way.
Other annotations besides QDomainDoc and QAppcodeDoc are used in combination with API code but not in an invasive manner:
@QInterfaceDoc(
type = "dubbo",
define = "Send SMS captcha to the user, Dubbo interface.",
desc = "Verify that the mobile phone number is owned by the user and send the SMS verification code to the user.",
scene = "Send SMS verification code to user",
notice = "Intranet Use",
since = "Send SMS verification code, product requirements introduced",
authors = "fanrong.zhao",
url = "dubbo_send"
)
@QParamDocs({
@QParamDoc(name = "paramV1", value = "First parameter", paramType = "form", dataType = "String", notice = "Must be a string.", paramExample = "username"),
@QParamDoc(name = "paramV2", value = "Second parameter", required = false, paramType = "form", dataType = "Boolean", notice = "Must be Boolean.", paramExample = "true")
})
@QResponses({
@QResponse(code = -1, message = "System exception"),
@QResponse(code = 200, message = "Success")
})
@QResponseDocs({
@QResponseDoc(bindValueName = "data", description = "First parameter", bindValueType = "object", propertys = {
@QProperty(type = "String", name = "name", desc = "1"),
@QProperty(type = "int", name = "age", desc = "2"),
@QProperty(type = "com.qunar.fd.qdoc.qdocexample.vo.ExampleResultVO",name = "food",desc = "3")
}),
})
public ApiResponseV2<Food> example0(@RequestParam String paramV1,@RequestParam boolean paramV2) {
return null;
}
Copy the codeAccording to the feedback from the engineers of Qunar who are already using it, it only takes 5 minutes to write the API through annotations for a medium complexity API.
4.4 Easy Management
Qunar QDoc tool mainly supports two methods of API synchronization: Maven command synchronization and publishing system synchronization. Maven command synchronization is designed to meet the needs of Design2doc and is also more flexible and inherits Qunar’s swagger-YAPI. Synchronization through the publishing system is the main work of QDoc. This work addresses all issues of interface not being synchronized with the Master version, including interface creation, update, rollback, and so on.
We can see:
Service Opening Procedure
-
Release system opening display
In qunar application tree, there is a service list under the corresponding Appcode. In the service list [QDoc], click Open to complete the integration and opening of the application tree.
-
CM Release integration
In qunar publishing system, it can be opened through service bazaar corresponding to AppCode.
Once enabled, when Portal publishes the document online, it will automatically trigger the document update operation, and then you can see our document in the application tree.
In three simple steps, we completed the integration of QDoc with the publishing system.
4.5 Easy application of the platform
No matter how well the previous work is completed, if there is no support for the display platform with good interaction, it will be very painful for users, and the system with high application cost is also difficult to popularize. Our final choice is that YAPI is embedded in the App management platform and bound with Appcode management, giving team managers a one-stop management experience.
YAPI can refer to the open source version hellosean1025. Making. IO/YAPI /
However, API management of appcode dimension only facilitates one-stop API management of the engineer team, but does not positively help API management of Domain dimension, which is very important in DDD service remodeling. Currently we are constantly optimizing the Domain dimension API management platform:
Of course, after we have established the Domain dimension API management system, what we can do incidentally is to export our API through qunar’s mature gateway system:
At this point, qunar through the QDoc tool, from the API annotation in the code to the open platform domain API process is introduced.
So what were the difficulties we encountered in the project?
5 implementation difficulties of API standardization
1. Where do development resources come from
Ticket destination where the project is network business group of business development of TC project, there is no direct team resources to support, the resources needed to cross the CM, YAPI pilot project access platform, tool development, business development blocks, almost covers all company engineer team, the project with the method of the open source project management in the company, To set up a project team and set up a project independently, and to complete the project by combining the resources of each team across the team, the completion of the project is not a small challenge for project managers.
2, Design First or Code First
At the initial stage of the project, it is impossible to support both Design First and Code First. Through investigation, we found that Design First is more suitable for the formulation of non-domain dimension interfaces, such as a front and back end interconnecting interface. Such support interfaces are not universal. It is redeveloped as the page changes and is usually not reused. Code First is better suited to help the long-term maintenance of the Domain dimension support interfaces to maintain high quality. The original intention of our API standardization work this time is to help DDD business reshape the standardization of Domain dimension interface maintenance work. So we chose to First support Code First interface provision.
3, Annotation or Comments
Before we do QDoc, used in the company have certain degrees of API standard tools include swagger2.0, swagger3.0, smart – doc, on this point by investigation, we found that the There are significantly more engineers who use swagger related tools in annotation mode than those who use smart-Doc class annotation mode. Although there are too many annotations above API code in annotation mode, which will cause the problem of ugly code. But since the number of users using annotation annotation is significantly higher, we decided that QDoc supports annotation first. This doesn’t mean that QDoc won’t support comments in the future.
4, support the full OpenAPI3.0 specification or support a subset of OpenAPI3.0 specification
The question comes from the fact that we have a mature YAPI platform that is open source in 2018. Is it enough for us to meet the interface requirements of YAPI? Is it necessary to fully meet the requirements of the OpenAPI3.0 specification? Our answer is no. Platforms are evolving and YAPI will get old, and when YAPI gets old we will have to choose between other platforms that support the OpenAPI3.0 specification, and knife4j is an API platform in our sights.
6. Project results
DDD and API finally meet. The core domain, support domain, and common domain apis have been connected.
- Core domain API standardization effectively protects the domain from invasion
- Api-ization of common domain is beneficial to realize the platformization of common domain
- The support domain API reduces the amount of development
There is no repetition of the wheel.
Summary and follow-up plan of the project
As we can see from the previous introduction, the API construction of Qunar is hierarchical. The innermost layer is the API of AppCode dimension, the outer layer is the business interface dimension, and the outer layer is the domain dimension. The open interface can be configured to the open platform through the gateway, forming interface management and interface use of different dimensions.
The following things should be done in the interface management of AppCode dimension:
- Complete IDEA interface compliance plug-in;
- Support annotation mode;
- Support client/server code generation;
- The domain interface dimension needs to be investigated: is there a better alternative to YAPI knife4J is always in view;
- One of the more important things is that the development of API standardization in reverse promotes the idea of DDD across the business.