A, leads

Compared with traditional application r&d processes, microservices architecture-based r&d teams need and rely more on the agile nature of the overall process. In order to help more projects that will or are based on micro-services and understand and solve many problems in the agile development process, we invite some core members of the product r&d team of Tencent Micro-services Platform (hereafter referred to as TSF) to introduce TSF’s own implementation of agile development, which will be sorted out and output by the author. It is hoped that it can play a certain reference role to the projects constructed by micro-service architecture.

2. Introduction to the author

Cui Kai

Tencent cloud CSIG micro service center product architect Distributed, high concurrency e-commerce system development, system architecture design experience, good at mainstream micro service architecture technology platform landing and implementation Current research focus on micro service architecture related middleware promotion and best practices of precipitation, is committed to helping enterprises through digital change

3. Introduction to TSF

Tencent Service Framework (TSF) is a PaaS technology platform centering on applications and micro-services, providing functions such as application life cycle management, digitalized operation, three-dimensional monitoring and Service governance. TSF embraces Spring Cloud and Service Mesh micro-service frameworks to help enterprise customers solve the difficulties of traditional centralized architecture transformation, build large-scale and highly available distributed system architecture, and realize the rapid implementation of services and products. TSF takes a number of mature distributed products from Tencent cloud middleware team as its core basic components, and provides distributed configuration services of second push, link tracking and other high availability stability components. In addition, TSF connects with Tencent Cloud API gateway and message queue, making it easy for enterprises to build large distributed systems.

In the microservice architecture system of TSF, the total number of services is not large, about 40+. In terms of quantity, the architecture scale of TSF itself belongs to the small and medium-sized microservice architecture. This is because, TSF product is essentially a set of business support technology platform, the micro service architecture surface appears very delicate, compared with the large system architecture in order to cope with high concurrency, mass storage complexity of factors such as architecture, architecture design tradeoffs and connotation of TSF thinking is mainly reflected in the product stability, cohesion, high availability, etc.

In terms of component distribution, the overall functions of TSF are mainly distributed in various sub-platforms such as control platform, registry, distributed task scheduling platform, distributed transaction platform and Mesh platform, and each sub-platform contains several services with different numbers.

TSF is essentially the product attributes – business support platform, determines it belongs to a relatively typical lightweight micro service architecture, its agile ground practice experience for the small and medium-sized service architecture program has a unique reference, you can quickly switch from different sides help r&d team in the agile development process under the micro service architecture. The following will briefly share with readers from the perspective of the overall R&D process, including projects, requirements, R&D, testing, construction and documentation.

Iv. Project management

After years of internal precipitation, Tencent has accumulated rich experience in agile RESEARCH and development and agile model tools. TSF’s iteration cycle and rhythm are gradually stable with the development of Tencent’s internal research and development environment. The management process of the whole project basically tends to be process-based and standardized, and there are almost no obvious blocking points. The following will illustrate the differences between TSF and other products in project management.

1. Overall process

Currently TSF team of research and development process in line with tencent development system has been in the practice of agile development concept (pictured), and the product overall style uphold the frontiers, open source, the innovation of technology, the general will be closed and open beta in public cloud to get products, will be the forefront of the latest technology to pratt to domestic enterprises and users, After users actively and continuously help the product to carry out functional iterations, the product will continue to be “privatized” through various channels and ways, so that it will continue to improve the compatibility and ease of use of customized environment, and maximize the commercial value for Tencent’s continuous output.

From the perspective of tools, we can see that the whole project management process is centered on TAPD. Source code management is carried out through worker bee, compilation, construction and packaging is carried out through Blue Shield, test cases, defect tracking and automation are managed through WeTest, and release, deployment and operation and maintenance management is carried out through Blue Whale. From the perspective of team roles, TSF R&D team is mainly composed of the following roles, each of which has its own main responsibilities. However, in the case of emergencies or areas with vague responsibility boundaries, the team mainly encourages the independent initiative of staff to solve the problems.

2. Schedule management

As a micro-service technology platform, TSF mainly serves enterprise users, and the time reserved for b-end clients (especially large clients) to realize their demands is generally very short, which requires the project manager to intervene in the coordination of resources, scheduling and other aspects, especially to play an important role in schedule management. For example, according to the function priority, customer importance, R & D and test resources and so on, the overall arrangement, at the same time whether the quality and quantity of customer delivery is not only a project manager, it is the r & D, test, product, delivery and other parties to think and work together to complete.

And large state-owned bank, for example, an important demand docking process, from the design finalized to deliver customer originally the normal cycle of at least 4 months, but the customers own technology platform modification and application of online business, deadline looming, demand for discussion, research and development design, functional testing, delivery, deployment time severely compressed to 2 months or so. Due to the timely intervention, continuous follow-up and coordination of the project manager in the early stage, the delivery and deployment of this demand iteration could be completed before the transformation of the customer’s technology platform, which won the trust of the customer and ensured that the team did not disrupt the future version development plan due to this urgent demand.

Demand management

Demand management is a very important part of the RESEARCH and development process. TSF demand management mainly includes demand summary, demand analysis and demand change. During the whole process of demand management, TSF team members will continue to think about what fundamental problems the current demand solves for customers, whether it removes the pain points of users, and whether it exceeds users’ expectations. Taking this as the core of its essence, TSF team members will continuously promote the iterative evolution of products to meet the essential needs of users.

1. Requirement summary

The sources of TSF product requirements mainly include: Product manager for friends of the analysis of the standard products (especially important feature), open source technology ability or tendency of tracking, the architect voice feedback of customers, also include some internal feedback, such as the experience from the testing personnel problems, research and development of non-functional requirements and associated products (such as container services and compatibility of virtual machine) or to demand, and so on. In view of the above demand sources, product managers will build and maintain the demand pool to summarize, classify and disassemble the functional modules they are responsible for.

The value of each source of demand have different points, is just to solve a class of have common problems, such as the important features can become big customers in the process of bidding evaluation the main point of competition products manufacturers, the future trend of the technology on the market about the product the development direction of the future, the architects deep excavated by the customer demand point also enriched product usage scenarios.

2. Requirements analysis

After completing the requirements summary, TSF’s product manager conducts an in-depth analysis of each requirement and several rounds of communication with developers. The main content of requirement communication includes the feasibility, universality, general implementation logic, task priority and so on. In the demand analysis stage, TSF will pay more attention to whether the product functions are more in line with the positioning of Paas technology platform, that is, to help users build business applications better and faster in the way of middleware technology platform, such as whether the scalability, availability and multi-tenancy of the product itself will be affected.

Among them, in the aspect of feasibility, developers will not only complete the most basic function realization, but also consider the stability and compatibility from the perspective of the current architecture system. For example, whether the technical selection of new functions is in line with the development direction of the product in the next 3-5 years, whether the current technical implementation scheme is simple, whether the impact on the current architecture is minimal, and what is the compatibility guarantee scheme for the existing version that customers are using?

In addition, the demand of generality and middleware products demand a more important factor in the analysis, compared to general situation general demand customization demand of lay particular stress on, for the masses of users, to produce more of the product value, the demand on the prioritization of also will more focus on the general demand.

Finally, different requirements (size refers to the overall cost of implementing the requirements) have slightly different communication styles between product managers and developers. As opposed to minor feature optimizations, the product manager communicates directly with the appropriate developer to determine which version the optimizations will be added to. If it is a major feature update, the product manager will engage the relevant personnel, including but not limited to product manager, r&d personnel, testers, project manager, product architect, etc., to discuss it several times. Each discussion may also focus on different topics, such as technical implementation scheme, schedule, etc.

3. Requirements change

There is a saying in Mencius: The cleverness of Li Lou and the cleverness of Gong Shu Zi can not be accomplished without rules and regulations. Similarly, no matter how skilled a team is, a lack of management control over a requirement change can lead to it spiraling out of control.

During the development of TSF, it is inevitable to encounter scenarios where requirements change, such as being forced to adjust requirements priorities in order to synchronize with the customer’s business launch date. The earlier a requirement change is proposed, the smaller the sunk cost to the entire r&d team and the less impact it has on the motivation of its members. TSF is not constrained by detailed process specifications for requirements change, but it does have a bottom line. First of all, the product manager will carefully measure the impact range of the change, and then estimate the time and labor cost of the change with relevant parties such as R&D and testing. Finally, the product manager will comprehensively discuss various factors and judge whether the change is worthwhile according to the cost and benefit, and whether there is a temporary transition plan if there is no change at present.

Vi. R&d management

Research and development management involves many aspects, such as performance management, cost-benefit management, team building, research and development standards and so on. Due to the limited space, this paper mainly introduces the TSF team’s actual development process, the practice of small team collaboration mode, and internal management experience and methods.

1. Development process

When the product manager and the corresponding r&d personnel reach a solution consensus on a requirement, the r&d personnel will start the following R&D process: outline design – detailed design – code writing – unit test – local self-test – joint test – test email.

In the actual RESEARCH and development process of TSF, each developer basically has a relatively wide technical horizon and very specialized points, so on the premise of high technical quality team, research and development task division, module responsibility boundary and other aspects, do not follow the traditional research and development process of very strict definition. Everyone has a functional module that they are primarily responsible for, but there is also a certain amount of turnover in terms of people. This approach also reflects the humanized team culture thinking of team managers, that is, encourage them to find what they are good at and interested in, and make comprehensive and substantial talent development and cultivation.

Due to the high maturity of TSF’s internal microservices and the comprehensive technical skills of its developers, a single requirement usually involves no more than three developers, and in most cases, 1-2 people can complete it. At the same time, each iteration will “condenses” the efforts of multiple developers, thus forming a joint force of the team, so that each iteration can bring users a high-quality experience. A streamlined and high-level technical team can effectively reduce the cost of communication and management coordination, and increase the efficiency of producing high-quality products. This team management philosophy and culture coincide with Steve Jobs’ concept of entrepreneurship team — “I try to keep a simple structure and hire smart people”.

In addition, as for the joint commissioning test, which is often criticized as inefficient in traditional business application development, TSF seldom seriously coupled other components in the joint commissioning test due to its high internal micro-service maturity and relatively reasonable separation, resulting in invalid waiting or involved in multi-party communication and confirmation. At the same time, developers ensure the quality of their own components by mocking peripheral interfaces in the environment, and this process is almost done independently by developers, with little cross-group and cross-department resource coordination. Finally, after completing all the contents of this version, the r&d will write a formal test email to inform the product manager and test manager. Generally, testers will start testing only after receiving the test email. Developers then move quickly to the next iteration of the requirements task, working through each release in “baby steps.”

2. Collaborative mode

Different teams have different collaborative modes. The internal collaboration of TSF r&d team mainly includes multi-location collaboration, internal collaboration between roles and collaboration between sibling products.

TSF team members are located in Beijing, Shanghai, Shenzhen, Hangzhou, etc. So how to ensure smooth and effective communication between r&d personnel in multiple regions? First of all, when the team members are all in the local area, face-to-face communication is generally adopted. Even for new employees from other places, they also go to the Headquarters in Shenzhen to get familiar with the transition mechanism with the team members for a period of time. Secondly, there are many ways and tools for daily communication. For example, remote desktop and real-time communication based on “Tencent Conference” can solve the situation where codes, documents and other information need to be displayed; “Enterprise wechat” provides quasi-real-time communication in the formal office environment and a series of related tools to solve the daily communication problems of team members; “Wechat”, as the largest IM social product in China, solves the need for emergency communication during non-office hours or communication between teams and external personnel. In addition, it also includes the traditional email approval process, special cases of telephone communication, etc.

Due to the fact that TSF’s R&D team is distributed in different regions, the cooperation between the roles in the team basically follows the principle of “mutual trust” and keeps the entrepreneurial team’s empty cup mentality all the time. On the premise of effectively improving the overall operation efficiency of the team, the initiative and value expression of the team members are aroused. Avoid unnecessary processes, norms and pretentious formalism.

As a Paas technology platform, TSF must have the management ability of resources and components and the compatibility ability of various environments, so it often involves the cooperation with other sibling products, such as container service TKE, VIRTUAL machine CVM, Serverless, message queue TDMQ, private cloud TCE and many other products. Team leader and core backbone play an important role in this process, such as communication of technical framework scheme, exploration of cooperation opportunities, design of external overall solution, docking ideas and process arrangement, etc., to avoid repeated communication or unclear communication problems in advance. 3. The quality of internal management code, in addition to the developers by local debugging technology in the form of self-test, unit testing and security, will select group of developers when submit code review code to each other, which prevents the “in” the good luck body is in the mountains may be produced by the error, is also a process of learning from each other. In addition, the developer is a gate where most of the functional problems converge, and the effectiveness of this gate directly affects the subsequent testing cost and r&d rhythm.

There is a preliminary PRD (product requirements Document) during the requirements review phase, but developers typically do not come up with a complete technical design until the requirements review is complete. During the requirement review, there will be continuous communication and discussion among the r&d staff. In general, if there is no fundamental change in the requirement content, there will be no big deviation in the direction of the overall technical solution. In the late stage of the requirement review, the developers are basically constantly improving the details of the technical solution.

In terms of code branch management, TSF currently adopts the method of continuously releasing master in a branch. The advantages of this branch management mode are faster iteration, more frequent production and release, and separate hotfix version can be baselined according to different versions of each customer. The disadvantages are that as the project gets bigger and bigger, the turn code cost is high and the development resources are in a serial structure. TSF r&d team is also constantly thinking and optimizing, trying to find a suitable branch management mode.

7. Test management

Software testing is the last and most important detection barrier before software release. Its importance lies in the use of test management system to systematically test whether the software is as described in the requirements document, which is different from the way of thinking of developers, and whether it has the conditions for delivery to the production environment. This paper mainly introduces TSF test process, use case management and quality assurance.

1. Test process

TSF’s general test process is divided into: smoke test, functional test, integration test, automation test, related compatibility and performance test, etc. The smoke test mainly focuses on the requirements and core functions of the product itself. If the smoke test fails, the team will be notified by email and call back for modification. The functional test will mainly focus on the boundary and new features, and the test cases mentioned above are mainly written for this part. System test mainly detects whether the modification affects other related modules; The automated test covers the scenarios with high labor costs and infrequent function changes as far as possible, such as clusters, deployment groups and namespaces. Meanwhile, the message linkage (automated test robot message) is carried out with the enterprise wechat. Compatibility and performance tests are tested separately based on the importance of the release after cost and solution assessment.

When the PRD and technical scheme are basically determined, the tester has started to design test cases. Before the R & D sends out the test email, the tester has basically prepared the smoke test and core function test. So that after the development of a complete test, the tester can quickly enter the test case execution link, find potential bugs.

2. Use case management

In the process of requirement review and technical solution review, corresponding testers are required to participate, which is convenient to ensure relatively complete use case coverage at the initial stage of test case design. After compiling the use cases, they will be reviewed jointly by the r&d personnel and product manager. The time point is generally arranged around the formal test of the R&D, because the development of the function is basically completed at that time, and the time is relatively abundant. In addition, during test case execution, testers will constantly supplement test cases due to omissions or requirements changes, or feedback some user experience issues to the product manager. TSF testers in the functional test tasks are relatively fixed, that some testers would have been responsible for some function modules of the test and use case writing, because the test task is different from development tasks, testers need to be very familiar with the basic principle of itself is responsible for the module, and test cases, the test results of mild abnormal needs a high sensitivity.

3. Quality assurance

In terms of automated testing, since most of the functions of each MODULE of TSF are exposed in the form of interfaces, it has great convenience in automated testing. However, as mentioned above, automation is mainly carried out for some long-term stable interfaces. For modules that change frequently and often add features, automation testing is generally not and not suitable, which will increase a lot of extra costs.

In the whole research and development process, measurement data will be collected and analyzed for the research and development process, such as the BUG rate of thousands of lines, smoke pass rate and so on. Through these metrics of the r&d process, the whole r&d team can be further guided to better improve the collaboration mechanism, improve the quality of their own code, and evolve to a more efficient way.

Build and release

1. Continuous integration

TSF continuous integration process: the local code push triggers the construction task – the release platform pulls the code of the code warehouse to the construction machine – carries out static inspection through unit testing, Sonar, automated testing, etc. – the material package is compiled into the material package – saves the material package in the product library – pulls the material package to the corresponding environment to complete the deployment.

Local developers use GIT as a repository of code, with hooks associated with the build platform that automate build tasks when code is pushed. Before you start building, you need to pull the code into the build environment and check it statically with platform tools such as Sonar, CheckStyle, FindBugs, etc. Maven automatically executes the unit test code within the code at compile time, and can then associate automated test tasks. Finally, the code is compiled, and the compiled bom is stored in the artifact repository, and the bom is pushed to the specified environment to complete the deployment.

2. Continuous delivery/deployment

TSF’s R & D environment is mainly divided into local development environment, joint development environment, functional testing environment, pre-release environment and production environment. In the process of production and release, grayscale release was adopted. First, a small city in China (such as Chongqing) was selected for pilot release, and then gradually transferred to a small region (such as Shanghai), and finally completed the release of a large region (such as Beijing, Guangzhou) and the whole country. In the rare case of an unexpected logic problem in a release, the developer immediately fixes it, and since the data structure is relatively stable, rereleasing the application solves most of the problems. In addition, the release of TSF may need to prepare resources in advance, so the release needs to go through the internal application and approval process.

Finally, in the whole TSF CI/CD process, TAPD is also the center, associated with the use of various tools, For example, code management is carried out through Gitlab and SVN, code compilation and construction is carried out through Blue Shield, code inspection is carried out through Sonar, package management is carried out through Nexus, unit testing and automatic testing is carried out through Junit and Jmeter, and deployment is carried out through Blue Whale as a publishing platform.

CICD scheme

1. Scheme introduction

The above chapter mainly introduces TSF’s own agile development practices, so how to quickly implement agile development for enterprises? Here TSF provides support solutions for three different scenarios based on its rich enterprise Agile development case experience.

  • Deploying applications using Python scripts For enterprises lacking in testing properties or operation and maintenance capabilities, you can use Python script deployment to make the whole process run and complete the evolution from “0” to “1”. You only need to prepare the Python running environment, Tencent Cloud access secret key and CVM resources to quickly complete the construction of the overall process. Click here for detailed steps.

  • For some enterprises that have used Jenkins as continuous integration solution, TSF supports docking compatibility after simple configuration, which basically does not affect the normal operation of the original R&D process. For details of the TSF+GitLab+Jenkins linkage example, please click here

  • TSF integrates various capabilities of Tencent Cloud CODING to form a complete set of agile development solutions. At the same time, CODING is fully compatible with Jenkins continuous integration service. Enterprises can implement fast and low-cost compatibility and migration based on their existing infrastructure. Click here to learn the detailed steps of TSF integrated CODING.

2. Introduction of CODING

Relying on industry-leading agile project management philosophy and DevOps system methodology, CODING integrates excellent ideas and tools into products, breaking down tool chain islands and collaboration barriers in the r&d process. In CODING platform can realize rapid iteration requirements, product code management, automated testing, continuous integration, building management, application of continuous deployment and a series of the development of closed-loop workflow, covering the whole life cycle of agile development, help the team improve r&d efficiency, fully embrace the industry leading IT concept and culture.

Ten,

Based on TSF team’s agile development practice, this paper briefly introduces the general process of TSF product agile development. As can be seen throughout, agile development practices do not have the most standard process, only a more appropriate process. The current TSF research and development process is not the most perfect, such as how to better associate material package version, material package configuration version, SQL statement version and other issues. TSF team will humbly accept your comments and suggestions, and will continue to explore the entrepreneurial mentality as always, to provide customers with better products and services.

reference

Navigating the Microservice DeathStar With DeployHub​

Tencent redefines agile

What is CI/CD? This article takes you through CI continuous integration and CD continuous delivery/deployment – Red Hat