background

At the end of last year, the team took charge of more than 30 toB-oriented platform systems. There are many systems, and the demand is tight, and the manpower gap always exists. The main problems are as follows:

  1. Front-end and back-end deployment coupling: Front-end and back-end resources are deployed in the same Web container, and the front-end takes the front-end with it when it comes online.
  2. The r&d process is not standardized: the platforms lack unified specifications, such as branch specifications, deployment specifications, joint investigation specifications, etc.;
  3. R&d quality assurance: lack of basic quality assurance means, such as front-end unit testing, front-end and back-end interface change process, etc.

It has always been our goal to improve r&d efficiency through technological means. This year, through the relevant technical research of the industry and the company’s internal Aipanfan team, we decided to improve and standardize the existing R&D process, through CI/CD capability suggestions, improve r&d efficiency and reduce operation and maintenance costs (thanks to the CI/CD work of Aipanfan team!)

Today, THE best practices of CI/CD for ToB scenarios are mainly explained from the following aspects. In addition to explaining What and How, we hope to explain Why in terms of value benefits. (Note: ToC and ToB differ greatly in all aspects)

What is the CI/CD

Continuous Integration/Continuous Deployment

Deep understanding: Bridge the gap between development and operations teams by automating build, test, and deployment in the form of pipelines based on the emerging DevOps culture for fast and stable delivery.

Noun explanation

DevOps

DevOps, a combination of Development and Operations, is an idea, a set of best practices, and a culture that facilitates communication, collaboration, and integration between application Development, application Operations, and quality assurance (QA) departments. To break down the barriers and gaps between traditional development and operation.

CI

CI stands for Continuous Integration. Developers will frequently submit code to the trunk, which will be verified by compilation and automated test flows before it is finally merged into the trunk.

CD

Continuous Delivery CD Continuous Deployment CD Continuous Delivery

  • Continuous Delivery

    Continuous delivery automatically publishes validated code to the repository after completing the automated process of building and unit and integration tests in CI. To achieve an efficient continuous delivery process, it is important to ensure that CI is built into the development pipeline.

    The goal of continuous delivery is to have a code base ready to deploy to production.

  • The final stage for a mature CI/CD Pipeline is Continuous Deployment. As an extension of continuous delivery, which automatically releases production-ready builds to code repositories, continuous deployment can automatically release applications to production.

scope

Covers the entire software development life cycle, acting on all stages

The basic principle of

  • reliability

  • CI pipelining needs to be fast

  • Build and run in a standalone environment

  • A pre-release environment is equivalent to a production environment

The process specification

The basic prerequisite for CI/CD implementation is to formulate the r&d process specifications in line with the actual situation of the team according to the status quo of the team’s technical capabilities.

Technical panorama

Research and development process specification

  • Branch development, trunk testing, RB on-line
  • Historical branches are retained to ensure that they are ready to be released online
  • Untested code is not brought online
  • Periodically check the synchronization code mechanism to avoid conflicts
  • All inspection capacity CI on the ground, to ensure high quality delivery

Branch management specification

  • Code version branching is the foundation of an automated build that development teams must follow
  • Feature branches are not required, which can be used by each team according to the complexity of actual collaborative scenarios
  • The branch naming convention, which includes a live date, is only suitable for ToB scenarios

Pipeline specification

  • Compile: Compile using the company’s unified cluster
  • Safety scans: woodpeckers, owls
  • Static code scanning: Contains code specifications and potential bugs
  • End test: automation, stability

Cloud Deployment (BOS/CDN)

The deployment of design

Is the difference between plan 1 and Plan 2 that index.html is on BOS or RD server?

Preferred recommendation 1:

  • More completely decoupled from the back end, HTML updates can be controlled by the front end
  • Front-end version number management in HTML, no need to update version.js, deployment is easier

Some backends in Scheme 1 May not be able to configure forwarding;

Note: The optimal solution is that the BFE can configure forwarding rules to BOS without the RD cluster. However, BFE cannot directly forward the forwarding rules to BOS.

Release notes

Front-end static resource versions are generally managed in two ways:

  • Hash version: The Hash version number is based on the MD5 value of the file content. Each file has a different version number, which is suitable for incremental release to save storage space. For example: index.abcd12.js, manifest.123456

  • Timestamp version: The time folder is generated based on the current build point in time. The build structure is deployed in the time folder. Suitable for full release.

Currently, the timestamp version number is selected, considering the following:

  • Hash version, suitable for container deployment, such as Docker, Jarvis, including all resource files at a time; Bos If Hash is used, there are many resources with the same name in the folder.

Deploy the directory structure instance

performance
|---- 202101281200
|     |---- css
|     |---- js
|           |----index.js
|---- 202101291200
|     |---- css
|     |---- js
|           |----index.js
version.js
index.html
Copy the code

Index. HTML and version.js are responsible for maintaining the version number and updating it each time it goes live.

Unit testing

The status quo

Most of the front-end is untested. A foreign survey shows that 43% of developers have not done any front-end tests, for two main reasons:

  1. The front end is GUI programming with a lot of browsers, and it’s actually very expensive to do several rounds of testing on so many browsers;
  2. In some types of products, the UI often iterates faster than testing can keep up;

As a result of these two factors, front-end testing is not taken seriously, and many developers may work for years without writing tests.

Why do we do a single test

ToB business compared to ToC:

  1. The business is relatively stable: UI pages are stable and change less
  2. Stability requirements are high: frequent regression testing is required, and paying customers are served
  3. Continuous upgrade iteration: The iteration cycle is relatively long

Benefits from single test:

  • Bug reduction: Most of today’s projects are coordinated development of modules by multiple people, and problems can be found when each module is integrated. The cost of locating and communicating is very high. To ensure the correctness of each module through unit testing, problems can be found as soon as possible, and problems can be found during integration from time to time.
  • Rest assured reconstruction: For the project of continuous upgrade and iteration, the code is constantly changing and reconstructing. Through the unit test, we can develop the reconstructive code that can be safely modified, reduce the psychological burden during code modification and improve the success rate of reconstruction.
  • Improved design: The more good design code, and generally the more easy to write unit tests, the method of multiple small are usually larger than a single test method (hundreds of thousands of lines of code) of the single test code simple and stable, so in the process of writing single measurement, if it is found that a single test code is very difficult to write, usually shows that the tested code contains too much reliance on or function, need to reflect on the rationality of the code, Then promote the optimization of code design, forming a positive cycle;

Front-end automated test layering

  • E2E testing: A black box test that tests whether an application executes as designed from start to finish.

    The entire application has been tested in real-world scenarios, including testing communication between components, such as databases, networks, apis, etc., and executing code in various browsers to basically test everything. Setting up takes a lot of time and is the most expensive.

  • Integration testing: Tests interactions between programs, for example, communication between UI and API. The setup takes less time and is less expensive.

  • Unit testing: Is minimally granular testing because it involves testing isolated parts of code as units. These units can take the form of methods, properties, actions such as a UI element click, input, etc. For example, if I have a function, and I override different branches of logic like if and else with different input arguments, and then verify that the function returns the same value as we expect, that the code performs as we expect, then I need to write an assertion, which is the fastest and cheapest way to implement the first two types of tests.

As you can see, the higher up the pyramid we go, the more time and cost it takes to set up our tests (the more time it takes, the more ambiguous the failure message, and the more difficult it is to follow). This is why many projects tend to focus on unit tests because they can save time and effort by covering most scenarios to test our code.

Selection of test frame

Jest is strong, showing good performance in both satisfaction and usage dimensions:

  • Excellent performance: For concurrent testing of test cases, only the test corresponding to the changed file is executed, and the test speed is fast;
  • High integration: zero configuration, providing everything by default (Mock, Test Runner, Matcher, Test Coverage);
  • Daemon mode: focuses on the developer experience and helps us get quick feedback on test results while coding;

Front and rear end protocol

Status quo of the problem

As the business logic becomes more and more complex, there will inevitably be disputes over the division of labor and collaboration at the front and back ends, as shown in:

  • There is a certain responsibility ambiguity zone, such as data format assembly, format conversion and other work, that is, can be completed by the front end, can be completed by the back end, need many times of communication and discussion, communication cost is high;
  • In the mode of separation of the front and back ends, responsibilities of the front and back ends are divided, resulting in high cost of development and communication, poor customer experience (slow rendering speed, high power consumption) and other problems;

Therefore, this standard is designed to clarify the division of labor between front and back ends, reduce communication costs, reduce development and maintenance costs, and improve customer experience.

The basic principle of

  1. The back end splits the interface by business domain and avoids coupling the interface data structure to the UI presentation. The front end should pay attention to interactive rendering logic and perform data assembly and transformation according to THE needs of UI presentation;
  2. Browser processing performance is limited (JS single thread), mobile terminal battery life is limited, in order to ensure smooth and stable customer experience, batch calculation, sorting, paging and other large amount of data should be completed in the back end, the back end returns the final results, rather than intermediate results;
  3. If the front-end network environment is complex and unstable (2G/3G/4G/WIFI, elevator access, etc.), the number of network interactions between the front and rear ends should be minimized, and the number of network requests on a page should not be more than 3.
  4. When the front-end rendering speed is high and the number of back-end interfaces and data structure cannot meet the requirements, BFF layer (Backend For Frontend) can be used For interface adaptation or SSR technology can be used to implement interface adaptation.
  5. The interface follows the principle of minimalism, shielding business implementation details, avoiding invalid fields, exposing too much business logic, and avoiding redundant multi-level nested formats.

For other specific conventions, refer to the front and back end conventions section of Ali’s Java Development Manual.

Engineering capability map

value

Value of engineering capability maps:

  • Standardize the R&D and launch process: best practice guidance for the whole process of requirements, development, access, testing, deployment and operation;
  • Recommendation tool chain solution: Recommend the optimal tool that meets the company’s scenario in the whole process;

YAPI platform

value

Both front-end and back-end development suffers from interface documentation to some extent. The front end often complains that the back end provides interface documentation that is inconsistent with reality. The back end also feels that writing and maintaining interface documentation takes a lot of effort, and it is often too late to update. As time passes and versions are iterated, it is easy to keep up with the code.

Swagger provides a set of specifications to define interfaces and information related to interfaces. RD can automatically generate interface documents in various formats, generate client and server codes in multiple languages, and online interface debugging pages, etc.

YAPI has value for all roles:

  1. Back-end students to achieve “code is document”, simplify the interface document maintenance;
  2. Front-end students real-time, automatic, more realistic Mock data, complete automated deployment services, and automatic tuning tools;
  3. QA students implement based onSwaggerThe contract test of interface data, combined with ITP platform, realizes automatic contract test;

The value to the overall platform is mainly reflected in stability: after the back-end students modify the interface, the front-end uses Mock data to discover interface changes in time and solve the communication problem of interface changes (communication is always the highest cost).

F&Q

  • Q: Are there security issues with BOS/CDN deployment?
  • A: The front end sends all the code to the client, and there are generally no security issues with deployment. If you really want to consider, you can access the BOS resource through the platform domain name, at the gateway or Nginx.
  • Q: Are there any other issues with BOS deployment?
  • A: Based on our experience of over a year so far, there have been no problems with BOS deployments. If you have to consider the problem, you may want to pay attention to caching issues in case of accidental rollback. The deployment scenarios in the appendix are detailed in practice, measurement, and solution description.
  • Q: Is it appropriate to use a timestamp for version numbers?
  • A: The version number of the timestamp is suitable for the ToB scenario, and it comes online less frequently, unlike ToC, which comes online many times a day.