This article is written by dayao in the team.

The introduction

Mobile apps are frequently updated, which can be seen from the release frequency of major apps in the App market. High frequency releases mean rapid iteration and delivery, which puts higher demands on efficiency in requirements, development, testing, and operations. Then, in the rapidly changing Internet environment, how to improve the delivery speed of App on the premise of ensuring the quality? This is a common problem faced by the industry. DevOps provides an answer to this problem by advocating automating as many steps of the software build process as possible — that is, building an automated pipeline — to increase efficiency and shorten development cycles. In today’s IT world, the concept of continuous delivery advocated by DevOps has been widely accepted, and the implementation of DevOps practice has become the trend of software organization management. In recent years, as the Agile software model has deepened, we have been experimenting with agile and continuous integration practices to embrace DevOps in a positive manner.

It’s important to note that DevOps is not a fixed pattern or standard, in fact, there is no consensus on what DevOps is (and the definition of DevOps has evolved over time). It is a universal software engineering culture and practice, but every organization’s organizational culture is different, the DevOps foundation is not the same, the existing successful solutions are often not the perfect fit for the team, and there is no one universal example to copy. Therefore, should be based on their own actual situation, reference similar excellent cases and experience, to find the most suitable mode. On the one hand, we lack a dedicated tool department or team, which makes it impossible to build a large integrated continuous delivery platform in a short time. On the other hand, we urgently need to establish continuous delivery pipeline to improve the development and test process of agile products. Therefore, we take open source continuous integration tools as the core, and develop auxiliary tools appropriately to build a more practical continuous delivery pipeline of App products.

I. Case background

As the functions of App become more and more complex, more and more modules cooperate with the third party, and the difficulty of joint debugging in the development and testing stages also increases. At the same time, in order to rapidly respond to changes in market demand, App iterations are getting faster and faster, and the release cycle is getting shorter and shorter. For example, in order to complete the code on time, the development squeezed some of the time originally used for code review and unit test, resulting in the quality of the project code decreased; With the increase of production risk, product quality is difficult to guarantee; Development takes extra effort to pay off technical debt (bugs, non-standard code, etc.), which puts the next iteration at risk — a vicious cycle of “burnout.”

In order to solve the above problems, on the basis of referring to mainstream solutions such as DevOps, we also absorbed DevOps and agile concepts and carried out the practice of continuous delivery pipeline of App products. We improved the implementation process of continuous delivery of App products from the aspects of process improvement and organizational culture, and finally built a complete pipeline.

2. Case content

(1) Problem analysis

Unlike other products, App delivery is unique. For example, the H5 product needs to place the deployment package on the server and configure it accordingly, so that users can access the service by visiting the website. Android and IOS, on the other hand, are not deployed on the company’s own servers. Instead, they need to be “listed” in their respective App markets, where users can get their apps, install them on their phones, and complete the delivery process.

We must first make clear what affects the iteration efficiency of App. After analysis and summary, we found the following pain points in the implementation process of App:

1. Heavy development tasks and low code quality.

There are many reasons for this, such as improper requirements planning, failure of joint audit due to environmental issues, inadequate code review, incomplete test coverage, etc.

2. Version distribution is inefficient.

Due to the special nature of App products, users must install a new version on their phone to access its services. In a scenario where developers are frequently typing new versions, testers need to get the latest version of the App in a timely manner. But the reality is usually develop on their own computers to build (commonly known as “packaging”), then construct the App install package sent to the tester, so the construction of artificial distribution efficiency is low, build up the installation package is also difficult to guarantee “qualified” (that is, through the unit testing, code scanning and other necessary check link).

3, prone to different versions in different environments.

During the test phase, testers usually test the version in the test environment, while the production version corresponding to the test version is used when the product is released. The code of the two versions must be identical except for the different configuration of environment-related parameters. However, in the time between passing the test and manually switching to production parameters and building the production version, there is a risk that developers will change the code, resulting in an untested version of the App being released. Therefore, we need to find solutions to build multiple apps with different environmental parameters at the same time.

4, the test analysis is not in place, the same quality problems repeated.

Our goal is not just to fix the problem, but to make sure it doesn’t happen again in the future. In fact, the defects of one product can inspire other products. We need to learn from the experience and lessons of others and strengthen the communication between different products.

The above problems are actually caused by the lack of standardized processes and excessive manual intervention. When manual operations affect work efficiency, consider using tools to save manpower and improve efficiency.

(2) Method system

From the two aspects of process improvement and organizational culture, the implementation process of continuous delivery and the organizational structure matching it are gradually constructed.

1. From the perspective of process improvement, build the App continuous delivery line.

● Optimize and improve each step of DevOps, and combine it into a Continuous delivery pipeline based on Jenkins automation, covering every link from requirement to delivery, and standardize every link of App delivery from implementation process, such as optimizing requirement splitting, strengthening code review and other steps, to improve code quality.

● Independently develop small tools, such as App continuous delivery tool, baffle system, etc., to improve the above pipeline, in order to improve the development and testing efficiency.

2. Adjusted the organizational structure and carried out the integration practice of development and testing.

● Under the guidance of development and test integration thinking, development and test teams are planned in the same department, which eliminates departmental barriers, makes development and test become a whole in agile iteration, and strengthens the communication and connection between the two.

● Strengthen quality supervision, test team play a QA role. The test team is no longer only responsible for the traditional test work, but overall planning of test implementation, quality management and process improvement. In this mode, the test team is not only responsible for the traditional test implementation tasks, but also deep into the agile team, as a QA, overseeing automated testing, continuous integration, and other aspects of product quality control at a higher level.

● Carry out DevOps related communication activities, such as Open Day, series of training, etc., promote and communicate within and between departments, learn from each other’s experience, and cultivate employees’ continuous delivery concept and ability.

(iii) Build continuous delivery pipeline

With limited resources of all kinds, we must leverage existing tools to support the continuous delivery implementation process. As one of the most popular CI tools, Jenkins became our platform of choice. It supports various functions in the form of plug-ins (a rich library of plug-ins is available, and developers can develop custom plug-ins based on the official API). In addition, Jira, Sonarqube and other industry-recognized tools become part of the pipeline.

1. App is continuously delivered to the assembly line

As the saying goes, “if a worker wants to do a good job, he must sharpen his tools first”, the construction of automatic assembly line is bound to need the help of tools. The industry’s choice of continuous delivery pipeline tools can be broadly divided into two categories: one is to use generic CI and CD tools, and the other is to use an integrated DevOps platform. The former is flexible, versatile, doesn’t cost much, but lacks a business element. The latter provides a one-stop solution, but is not versatile enough.

Since we are not in a position to purchase or develop an integrated DevOps platform on our own anytime soon, we chose the former to implement the delivery pipeline. As shown in Figure 1, all links are connected in series like beads to form a complete assembly line.

1) Needs **/ knowledge: **

Requirements determine the amount of work, which in turn determines the workload of development and testing, and directly affects the quality of the code. Therefore, improvement should be carried out from the source of production line to avoid unreasonable demand planning affecting development efficiency.

Low in the past:

There are requirements accumulation, coarse granularity, frequent changes, development “crazy” situation; Too much reliance on manual kanban makes it difficult to count iterated burn out charts, story activity and other data.

Low now:

Reasonably plan requirements, reduce task granularity, reduce iteration workload, and prepare for small delivery; Use Jira electronic Kanban to archive user stories, questions, and more for better agile health control.

2) Source code:

Use SVN or Git to host code. The code base also has a chain of contempt, and it seems to most programmers that Git is the more “advanced” tool. In fact, both SVN and Git have quite complete functions. It is not important to choose a tool, but how to use it properly.

** Past: ** Code integration was complicated and labor-intensive due to multiple parallel tasks and confusing branching strategies.

** Now: ** gradually convert to serial tasks, adopt the trunk development strategy, reduce the situation of parallel development of products; Strengthen the code submission discipline, such as requiring that the development must pass local build success, unit test, specification check before submission, and pay attention to timely pull and submit the code. Strengthen the control and review of key logic codes and improve code review methods.

**● Past: ** manual check, often delayed, backlog a large number of code, resulting in a lot of time to check, the effect is difficult to ensure.

** Now: ** Implements code review sessions using related tools (such as Gerrit) instead of the original manual walkthroughs. This can not only detect problems in time, but also reduce the workload of developers.

**● Past: **CI discipline was a sham and often failed to be handled.

** Now: ** strengthen CI discipline, each Agile group is equipped with red and green dashboards to check the trend in real time. If the construction fails, repair it immediately. If the repair fails, the whole group does not work.

3) Code pull:

Jenkins pulls code from the code base and triggers subsequent pipeline /Job builds. This is usually the first step in Jenkins Job execution.

4) Automated testing:

It is mainly divided into automatic unit test, automatic interface test and automatic UI test. As a part of Job, automated unit test is executed with Job construction. The other two types of automated tests are implemented with the help of specialized automation tools (RobotFramework and Appium) and configured on Jenkins to trigger and build automated test jobs on a regular basis.

5) Static code scanning:

The quality of your code should not be limited to “visible” bugs. There are a lot of potential problems in the code, such as robustness issues, floating point comparison size, etc., which are not easy to spot, but like a time bomb, are a hidden threat to product quality. Therefore, we use static code scanning tools for code scanning. For example, the Sonarqube tool provides static code scanning support for all development languages, and users can customize their own rules. For all kinds of scanned problems, development is required to solve as soon as possible. Sonarqube also reads automated unit test reports and displays unit test results on its dashboard, making it easy to keep track of code quality.

6) Security scanning:

Add security scan to improve code security. Taking the Checkmarx security scan tool as an example, the Checkmarx plug-in is introduced into the pipeline to trigger the security scan. At the same time, for the problems in the scanning report, the development is required to solve the problem and zero in time.

7) **App reinforcement: ** reinforces App to prevent decompilation.

** past: ** uses the client for hardening.

** Now: ** To automate hardening and integrate it into the continuous delivery process, use shell scripts to invoke hardening commands instead of manual hardening.

8) Construction:

The previous sections are all preparation for this step. For App products, the final structure to build is to generate an installation package (IPA or APK). App development involves development, testing, gray scale and production environment. How to ensure the consistency of other codes except for environmental parameters is a problem worthy of our attention. In this regard, we propose a parallel construction mechanism with the help of Jenkins Pipeline Pipeline.

**● Past: ** development generally in the development, test environment to write code, after passing the test, manually modify the environment configuration in the code, and then build the corresponding production version, for subsequent production. If someone changes the functional code along the way, the built production package contains untested code, and the risk of going straight into production is obvious.

** Now: ** project code to set the configuration of multiple environments (development, production, testing and other environments), execute the build command can specify the environment, so as to build the App version of the corresponding environment; Using Pipeline parallel scripts, build multiple versions of environment parameters at the same time, these versions are completely consistent except for different environment configurations. When selecting the production version, the production version is forced to be selected. The production version and the test version are built at the same time, and there is no possibility of modifying the code, which can ensure the consistency between the App functional test version and the production version.

9) Products:

Artifacts are installation packages, or built versions, that will be pushed to the App continuous delivery tool for unified management.

2. Tool construction in the assembly line

Jenkins is not a panacea, it provides a pure pipelined engine with no business attributes. In other words, the pipeline on Jenkins could not be associated with the requirements, projects, tasks, products and other elements in the software management cycle, which could only be a set of parallel jobs without layers. Therefore, the above links cannot meet the actual needs of App construction. To this end, we independently developed several tools: App continuous delivery tool, QA dashboard, multi-protocol baffle system, error problem book, to further improve the delivery efficiency.

**App Continuous delivery tool: **App product version management platform, providing development, testing, delivery functions.

1) Jenkins products can be obtained by docking with Jenkins platform, and unified management can be conducted in stages according to corresponding environmental parameters to solve the problems of Jenkins Job disorder and lack of product management;

2) The App can be downloaded and installed by scanning the TWO-DIMENSIONAL code, replacing the original manual distribution mode and greatly improving the test efficiency.

3) Provide grayscale publishing function for App grayscale testing.

● Multi-protocol baffle system:

Provides a simulation interface for the scenario where the real interface cannot be invoked because the environment is not connected. Instead of writing fake data manually, developers call the interface simulated by the baffle as if it were a real interface to solve the problem of progress blocking caused by the environment.

1) In the actual development stage and the early stage of the test, there will often be problems with the interface, which will affect the development progress. Therefore, this baffle can be used to Mock the interface.

2) The existing interface does not have complex data and special scenarios, which makes it difficult to test abnormal situations. In this case, baffle can be used to simulate special data and carry out corresponding tests.

● QA dashboard:

Necessary quality supervision can improve the QA awareness of development and testing. QA dashboard is a platform to collect and display the quality data of the assembly line. Although we can see the corresponding data from SVN, Jenkins and Sonar, they are based on scattered jobs, have no product correlation information and lack statistical data. In order to better play the role of quality supervision, we developed QA dashboard to collect, process and display the quality data of each link.

1) Organize data based on products, and conduct statistics and display at different levels such as team and department.

2) Provide personalized functions such as email subscription to facilitate users to regularly monitor product quality data.

● Wrong questions:

As the name suggests, this is a platform for documenting quality issues. Issues injected at all stages (requirements, development, test analysis, test implementation, operations, etc.) can be incorporated into the platform and tracked in detail. The test manager will also regularly derive problems from the problem book and organize quality problem analysis meetings. By recording problems and analyzing problems in time, we can accumulate experience for future test analysis, avoid the repetition of similar quality problems and reduce the probability of “stepping on pits” in the future.

(IV) Development and test integration organization form

Development test convergence and agile implementation go hand in hand. In terms of organizational structure, the development and test teams are integrated into the same department, in which the test team undertakes the test implementation, quality control and process improvement (tool construction) of the department. The task implementation of the product is carried out by the Agile group as a unit, which includes PO, development and testing, and gradually establishes a full-function agile virtual team. Through this organizational change, the link between testing and development is strengthened, and the collective consciousness of agile teams is strengthened, which helps foster a DevOps culture.

Along with these organizational changes, the functions of the test team have changed. In the past, the main task of the test team was to implement the test task, and the test managers focused on the case execution, test process, etc. And along with the advancement of developing test fusion, we give the test manager more say: take product QA test manager role, from product demand to the end of the production stage, pay attention to the quality problem of the whole project cycle, such as a user story quality, unit testing and defect analysis, establish the whole QA process quality control system. From this point of view, test managers should constantly improve their own ability to develop a more comprehensive sense of quality.

3. Case effect

After the above series of promotion and improvement work, the delivery efficiency of App products has been significantly improved, and the quality awareness and CI awareness of agile teams have also been significantly improved, and the DevOps atmosphere has been formed.

1. The iteration rate of App products is significantly improved. Thanks to the effective implementation of App continuous delivery pipeline, the time from task to completion is significantly shortened. App continuous delivery tools, baffles and other tools have been effectively promoted, and tool empowerment has played a huge role in improving the efficiency of development and testing.

2. The quality of iteration has been significantly improved. Before 2018, the success rate of department iterations was only 70%, but in 2019, it increased to 93%. In the iterative process, the success rate of continuous delivery pipeline construction is also maintained at more than 80%.

3. Product quality has been significantly improved. The development team has significantly reduced the number of bugs by focusing more on their own code quality and on automated testing data under QA’s supervision. Automatic interface test has accumulated a large number of cases, and the coverage of automatic unit test has grown from scratch to high, and now the coverage of some products has reached more than 90%.

The practice cases presented in this article are still in their early stages, and there is still a lot of room for improvement in the App continuous delivery pipeline. In the future, there are plans to further optimize the continuous delivery pipeline, expand tool capabilities, and deepen DevOps practices.