I haven’t written much for a while. Today, I will write an article about my thoughts on API design. First of all, why write about this topic? First, I have benefited a lot from reading the article “Ali researcher Gu Pu: Thoughts on API Design Best Practice”. I reprinted this article two days ago, which also aroused the interest of the majority of readers. I think I should sort out my own thinking and share it with you. Second, I think I can handle this topic within half an hour and try to turn off the light and go to bed before 1 o ‘clock, haha.

Now, let’s take a look at API design. I’m going to throw out a couple of ideas that I’m welcome to explore.

First, the definition of the specification, has been more than half successful

Usually, the norm is the standard that everyone has agreed upon, and if everyone follows this standard, the communication cost is greatly reduced. For example, everyone wants to learn from Ali’s specification and define several domain models in their own business: VO, BO, DO, AND DTO. Among them, DO (Data Object) corresponds to the database table structure one by one, and transmits Data source objects up through DAO layer. Data Transfer Object (DTO) is a remote call Object, which is the domain model provided by RPC service. For the Business Object (BO), it is an Object that encapsulates Business logic at the Business logic layer. In general, it is a composite Object that aggregates multiple data sources. VO (View Object) is usually the Object transferred by the request processing layer, and it is usually a JSON Object after being transformed by the Spring framework.





In fact, if the domain models of DO, BO, DTO and VO are not clearly divided in such complex business as Ali, its internal code is easily confused. The manager layer is added to the internal RPC layer on the basis of the Service layer to realize the unification of internal specifications. However, if it’s just a single domain without too many external dependencies, then don’t design this complex at all, unless you expect it to get big and complicated. In this regard, it is particularly important to adjust measures to local conditions in the design process.

Another example of a specification is RESTful apis. In the REST architectural style, each URI represents a resource. Therefore, a URI is a unique resource locator for the address of each resource. A resource is actually an information entity, which can be a piece of text, a file, an image, a song, or a service on a server. RESTful apis allow users to perform operations on server resources in GET, POST, PUT, PATCH, and DELETE modes.

[GET] / users# update user info [PATCH] /users/1001 # update user info [PATCH] /users/1001 # update user info [PATCH] /users/1001 # update user info [PATCH] /users/1001 # 【DELETE】 /users/1001 # DELETE user informationCopy the code

In fact, there are four levels of RESTful API implementation. The first Level (Level 0) Web API services simply use HTTP as transport. Level 2 (Level 1) Web API services introduce the concept of resources. Each resource has a corresponding identifier and expression. The third Level (Level 2) Web API services use different HTTP methods for different operations and use HTTP status codes to represent different results. Level 4 (3) Web API services use HATEOAS. Link information is included in the representation of resources. The client can follow the link to discover actions that can be performed. Typically, Pseudo-restful apis are designed based on level 1 and level 2. For example, our Web API uses a variety of verbs, such as get_menu and save_menu, and a truly RESTful API needs to go beyond level 3. If we had followed the specification, we would probably have designed an API that was easy to understand.

Notice that we are more than halfway through the defined specification. If this set of specifications is the industry standard, then we can boldly practice, do not worry about others will not use, just throw the industry standard to him to learn. For example, Spring has become such a big part of the Java ecosystem that it’s hard to justify a newcomer not knowing Spring. However, due to the limitations of business and the technology of the company, we may use pseudo RESTful apis based on the design of level 1 and Level 2, but it is not necessarily backward or bad, as long as the team forms norms to reduce the learning cost of everyone. Many times, we try to change the team’s habits to learn a new specification, but the gain (input/output ratio) is too small to outweigh the loss.

To sum up, the goal of a defined specification is to reduce the cost of learning and make the API as easy to understand as possible. Of course, there are other ways to design an API that are easy to understand. For example, we define API names that are easy to understand and API implementations that are as generic as possible.

Second, explore the compatibility of API interfaces

Apis are constantly evolving. Therefore, we need to adapt to change to some extent. In RESTful apis, the API interface should be as compatible with previous versions as possible. However, in actual service development scenarios, as service requirements are constantly iterated, the existing API cannot support adaptation of the earlier version. In this case, forcibly upgrading the API of the server may cause old client functions to fail. In fact, the Web side is deployed on the server, so it can be easily upgraded to adapt to the new API interface of the server. However, other clients such as Android, IOS and PC are running on the user’s machine, so it is difficult for the current product to adapt to the new API of the server. In this case, the user must upgrade the product to the latest version before it can be used properly. To address this version incompatibility, a practical way to design RESTful apis is to use version numbers. In general, we keep the version number in the URL and work with multiple versions at the same time.

[GET] /v1/users/{user_id} // API used to query the user list of version v1 [GET] /v2/users/{user_id} // API used to query the user list of version v2Copy the code

Now, without changing the API of version V1, we can add the API of version V2 to meet the new service requirements. At this time, the new function of the client product will request the new API address of the server. Although the server is compatible with multiple versions at the same time, maintaining too many versions at the same time is a burden for the server because it has to maintain multiple sets of code. In this case, instead of maintaining all compatible versions, it is common to maintain only the latest compatible versions, such as the last three compatible versions. After a period of time, when the vast majority of users upgrade to a newer version, older versions of the API interface on some less-used servers are deprecated and users of very older versions of the product are required to force upgrades. Note that “the API for querying user lists without changing version V1” mainly means that it appears unchanged to the caller on the client side. In practice, if the business changes too much, developers on the server side need to adapt requests to the new API using adapter patterns from the old version.

Interestingly, GraphQL offers a different idea. In order to solve the problem of service API interface explosion and aggregate multiple HTTP requests into one request, GraphQL proposed to expose only a single service API interface and allow multiple queries in a single request. GraphQL defines API interfaces that we can call more flexibly on the front end, for example, we can select and load fields that need to be rendered according to different businesses. Therefore, the full range of fields provided by the server can be retrieved on demand by the front end. GraphQL can add new functionality by adding new types and new fields based on those types without causing compatibility problems.

In addition, we need to pay special attention to compatibility issues when using RPC API. Binary libraries should not rely on parent. In addition, local development can use SNAPSHOT, but online environment is prohibited to use it, to avoid changes and cause version incompatibility issues. We need to define a version number for each interface so that the version can be upgraded in case of subsequent incompatibilities. For example, Dubbo suggests that a third bit version number usually indicates a compatibility upgrade, and that only incompatible versions of the service need to be changed.

For example, we can look at K8S and Github, where K8S uses RESTful apis and Github uses GraphQL.

  • https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.10/

  • https://developer.github.com/v4/

Third, provide a clear thinking model

The so-called thinking model, as I understand it, aims at the abstract model of the problem domain, and has a unified cognition of the function of the domain model, builds a realistic mapping of a certain problem, and demarcates the boundary of the model. One of the values of the domain model is to unify ideas and clarify boundaries. Assuming that there is no clear mental model and no unified understanding of the API, the real problems in the picture below are likely to occur.

Mask business implementation in an abstract way

I believe that good APIS are abstract and therefore need to mask business implementations as much as possible. So, the question is, how do we understand abstractness? Consider the design of java.sql.Driver. Driver is a specification interface, and com.mysql.jdbc.driver is the mysql-connector-java-xxx.jar implementation interface to the specification. The cost of switching to Oracle is very low.

Normally, we provide services externally through apis. Here, the logic of the interface to which the API provides services is fixed; in other words, it is universal. But when we encounter scenarios with similar business logic, where the core backbone logic is the same, but the details are implemented slightly differently, where do we go from here? Many times, we choose to provide multiple apis for different business parties to use. In fact, we can do this more elegantly with SPI extension points. What is SPI? The full name of SPI is Serivce Provider Interface. It is a dynamic discovery mechanism that can dynamically discover the implementation classes of an extension point during program execution. Therefore, the specific implementation method of SPI is dynamically loaded and invoked when the API is called.

At this point, you might think of the template method pattern. The core idea of the template method pattern is to define the skeleton and transfer the implementation, in other words, it defines the framework of a process while deferring the concrete implementation of some steps to subclasses. In fact, in the implementation process of micro-services, this idea also provides us with a very good theoretical basis.

Now, let’s take a look at an example: an e-commerce business scenario where there is no shipping but only a refund. This situation is very common in e-commerce business. Users may apply for a refund after placing an order and paying for various reasons. At this point, since there is no return involved, the user only needs to apply for a refund and fill in the reason for the refund, and then let the seller review the refund. Well, since refund reasons may vary from platform to platform, we can consider SPI extension points to achieve this.

In addition, we often use the factory method + policy pattern to mask internal complexity. For example, if we expose an API getTask(int Operation), we can create instances through factory methods and define different implementations through policy methods. Operation is a specific instruction.

@Componentpublic class TaskManager {    private static final Logger logger = LoggerFactory.getLogger(TaskManager.class);    private static TaskManager instance;    public Map<Integer, ITask> taskMap = new HashMap<Integer, ITask>();    public static TaskManager getInstance() {        return instance;    }    public ITask getTask(int operation) {        returntaskMap.get(operation); } /** * init process */ @postconstruct private voidinit() {        logger.info("init task manager"); instance = new TaskManager(); Instance.taskmap. put(EventEnum. Chat_req.getvalue (), new ChatTask()); // GroupChatTask instance.taskmap. put(EventEnum. Group_chat_req.getvalue (), new GroupChatTask()); // Heartbeat task instance.taskmap. put(EventEnum. Heart_beat_req.getvalue (), new HeatBeatTask()); }}Copy the code

Another design to shield internal complexity is the facade interface, which encapsulates and integrates multiple service interfaces and provides a simple call interface for clients to use. The advantage of this design is that the client no longer needs to know so much about the interface of the service and just needs to invoke the facade interface. However, the disadvantages are also obvious, that is, the business complexity of the server side is increased, the interface performance is not high, and the reusability is not high. Therefore, local measures are taken to keep responsibilities as simple as possible, and lego-style assembly is done on the client side. If SEO optimized products need to be included by search engines like Baidu, HTML can be generated through server rendering when the first screen is displayed, so that the search engine can include it. If the first screen is not displayed, page rendering can be performed through client invoking server RESTful API interface.

In addition, with the popularity of microservices, we have more and more services, and many smaller services have more cross-service invocations. Therefore, microservice architectures make this problem more common. To solve this problem, consider introducing an “aggregation service,” which is a composite service that combines data from multiple microservices. The advantage of this design is that some information is consolidated through an “aggregation service” and then returned to the caller. Note that an “aggregated service” can also have its own cache and database. In fact, the idea of converged services is ubiquitous, such as the Serverless architecture. AWS Lambda, a function-as-a-Servcie (FaaS) computing service, can be used as the computing engine behind the Serverless service in our practical process. We write functions directly to run on the cloud. Well, this function can assemble existing capabilities for service aggregation.

Of course, there are many good designs, AND I will continue to supplement and discuss them in the public account.

Consider the performance behind it

We need to consider the various combinations of input parameter fields that lead to database performance problems. Sometimes, we may expose too many fields to external combinations, causing a full table scan without corresponding indexes in the database. In fact, this situation is particularly common in query scenarios. Therefore, we can provide only the combination of fields with indexes to external calls, or in the following case, require the caller to fill in taskId and caseId to ensure the rational use of indexes in our database and further guarantee the service performance of the service provider.

Result<Void> agree(Long taskId, Long caseId, Configger configger);Copy the code

At the same time, asynchronous capability should be considered for apis such as report operation, batch operation and cold data query.

GraphQL aggregates HTTP requests into a single request, but The Schema gets all the data recursively, layer by layer. For example, the total number of statistics for paging queries, which can be done once, has evolved into N + 1 queries to the database. In addition, if not written properly can lead to poor performance issues, so we need to pay special attention to the design process.

6. Exception response and error mechanism

There has been much debate in the industry about whether RPC apis throw exceptions or error codes. Alibaba Java Development Manual suggests that isSuccess() method, “error code” and “error brief message” should be preferred for cross-application RPC calls. 1) If the caller does not catch an exception, it will generate a runtime error. 2) If no stack information is added, only new custom exception is added and error message of its own understanding is added, it will not be too helpful for the caller to solve the problem. If stack information is added, performance loss in data serialization and transmission is also an issue in the case of frequent invocation errors. Of course, I also support the practical proponents of this argument.

public Result<XxxDTO> getXxx(String param) {    try {        // ...        return Result.create(xxxDTO);    } catch (BizException e) {        log.error("...", e);        return Result.createErrorResult(e.getErrorCode(), e.getErrorInfo(), true);    }}Copy the code

During the Web API design process, we use ControllerAdvice to wrap error messages uniformly. In complex chain calls to microservices, problems are harder to track and locate than in a single architecture. Therefore, in the design time, need special attention. A better solution is to use a global exception structure to respond to a non-2XX HTTP error code in a RESTful API interface. The code field is used to indicate the error code of a certain type of error. The prefix {bizName}/ should be added to the microservice to locate the service system where the error occurs. If an interface in the user center fails to obtain resources because it has no permission to do so, the service system can respond with “UC/AUTHDENIED” and obtain details of the error from the log system using the request_id field of the automatically generated UUID value.

HTTP/1.1 400 Bad RequestContent-Type: Application /json{"code": "INVALID_ARGUMENT"."message": "{error message}"."cause": "{cause message}"."request_id": "01234567-89ab-cdef-0123-456789abcdef"."host_id": "{server identity}"."server_time": "2014-01-01T12:00:00Z"}Copy the code

Consider the idempotent nature of apis

At its core, idempotent mechanisms ensure resource uniqueness, such as repeated client commits or multiple server retries resulting in only one result. Payment scenarios, refund scenarios, transactions involving money can not appear multiple deductions and other problems. In fact, the query interface is used to fetch resources, because it only queries data and does not affect resource changes, so no matter how many times the interface is called, the resource does not change, so it is idempotent. The new interface is non-idempotent, because the interface is called multiple times, resulting in resource changes. Therefore, we need idempotent processing when repeated commits occur. So how do you guarantee idempotent mechanisms? In fact, there are many implementations. One solution is to create a unique index, which is common. Creating unique indexes in the database for the resource fields that we want to constrain prevents the insertion of duplicate data. However, in the case of sub-database sub-table, the unique index is not so good. At this point, we can query the database first, and then determine whether there is duplication in the resource field constraint, and then insert the operation if there is no duplication. Note that to avoid concurrent scenarios, we can ensure data uniqueness through locking mechanisms such as pessimistic and optimistic locking. Distributed locking is a commonly used scenario here, and it is typically a pessimistic implementation of locking. However, pessimistic locking, optimistic locking, and distributed locking are often viewed as solutions to idempotent mechanisms, which is incorrect. In addition, we can also introduce state machines, which can be used for state constraint and state jump to ensure the process execution of the same business, so as to realize data idempotent. In fact, not all interfaces need to be idempotent. In other words, the need for an idempotent mechanism can be determined by the need to ensure resource uniqueness. For example, behavior logs can be idempotent without consideration. Of course, another design solution is that the interface does not consider idempotent mechanism, but is guaranteed by business level during business implementation, such as allowing multiple copies of data, but obtaining the latest version during business processing.

(To be continued, I plan to write several related articles, please look forward to it.)

About the author:

Liang Guizhao is an architect at Picotech, and co-author of high availability extendable Microservice Architecture: Based on Dubbo, Spring Cloud and Service Mesh.



We look forward to your joining us and growing up together

>>> Follow the official account for more technical sharing and recruitment information <<<