DevOps is a comprehensive concept, and one perspective is at least partial. Especially in large organizations, different teams have different responsibilities, leading to different directions of focus.
The venue is a breakthrough attempt by DOIS organizers. The three lecturers come from different departments of Tencent, and their daily work is also their own. We shared our interpretation of DevOps from the first, middle and last three different perspectives of the development process through a morning of sharing.
This article is the sharing content of Tencent R&D management in DOIS Conference.
The author comes from the R&D management Department of Tencent Technology Engineering Business Group, which provides services supporting the whole Tencent business. This sharing focuses on agile R&D and configuration management in DevOps.
Many friends asked Tencent such a big company, so how to do internal research and development, research and development efficiency is how to ensure?
As we have mentioned in previous posts, Tencent has a wide range of product lines and a large number of products. Some of them are listed below, but many more are not.
Behind these products are Tencent’s 15,000 to 16,000 technical staff, which includes not only development, but also testing and operations. The product staff is around 6,000, including all product managers, operations, game planners and project managers. The total number is about 22,000.
In terms of r&d efficiency, the product capability provided by unit personnel reflects the r&d efficiency per capita of the whole group. Tencent maintains so many different products with a small number of employees. I boil down the key factors of R&D efficiency to four points:
1. Hire great R&D and product people
2. Bottom-up self-organizing agility
3. Automatic R & D tool system
4. Scientific and efficient quality control ability
Today’s sharing focuses on the second point, that is, what “agile” represents for large enterprises like Tencent Group.
The state of R&D
There are two main roles in r&d: product and R&D. The product manager arranges the requirements into the requirements pool and coordinates them with the R&D team. After several rounds of online and offline PK, the r&d team divides the priorities into iterations. After internal communication, the r&d team completes the architecture design, and then divides the requirements into different people to realize them, which is task allocation.
After that, development starts writing code, which requires all sorts of communication, such as dependencies or technical interfaces.
For example, product demand found some before did not think of the question back to blend. By the time the functionality is roughly developed and testing is involved, the product, development and testing sit side by side, checking the requirements of the testing, and tuning the development environment during the testing process.
Finally, problems found during testing should be identified and tracked together with product and r&d.
Going further, starting with iteration planning, the r&d team will first plan their own versions according to the product’s pace.
, of course, different forms of products, planning out the result is very different, if it is the Internet product, so planning is often associated with release rhythm, if it is delivered to the users of products, such as tencent cloud TCE and TSTACK, these products to different customers have different custom, version planning will certainly related to the customer delivery.
After version planning, the R&D leader starts to divide the work, and then the task sharing starts to be related to branch planning. Is it the main development? Or feature branching? Or do you want to develop a branch by one person? So what’s the plan?
At this point, the R&D Leader has the responsibility to regulate the r&d team’s actions. Should we change the branch development mode or what? What happens after submission, through merging, through automated detection of defect feedback, and finally to release delivery?
If you think about it, every team has to go through a lot of communication before they run through the process, and when they run into problems and solve problems. These communications can only be digested internally within the product team, the R&D team, the quality team, and the CMO (also known as the configuration manager).
If you think about the whole process, there are so many steps, if not in detail. We haven’t talked about how to write code, but we’ve wasted time on specification, communication, and communication.
Back in the early days of the Internet, the concepts of requirements, iteration, IPM, Scrum, defect feedback, etc., were not popular. For example, Tencent didn’t really know what IPM was when they wrote the first version of QQ when Pony and Tony were still writing code.
By analogy, research and development is often most efficient in the early stages of an industry’s development. For example, in AI Go in the last two years, the team could achieve breakthrough with only a few people. It can be said that the more early the industry is, the simpler and happier the r&d team will be.
But The Times have decided that we can not always go on so happy, because software engineering is more and more complex, more and more versions, more and more powerful, destined to research and development teams can not always be a small organization.
The larger the organization is, the greater the estrangement between individuals, which directly leads to the increase of communication costs.
Communication is the common essence of Agile and DevOps
Whether Agile, Extreme programming, or Scrum, these “Agile” practices are a communication specification or methodology. The problems they solve are essentially communication problems — you don’t understand me, I don’t understand you, and to understand you, I need to be close together.
It can be said that the optimization of communication efficiency is a common essence of the Agile and DevOps movements. What does DevOps say? R&d and operations are closely intertwined.
In agile methodology, communication between people is the norm. DevOps goes a step further, covering human-machine communication as well as machine-to-machine communication.
In order to improve efficiency at the group level, communication should be optimized in various scenarios. How to optimize communication? Tencent has done three things:
The first point is to reduce the job gap in the organization
This article does not cover business management, skip it
The second point is informatization
Here are some comparisons:
Although the above example is somewhat exaggerated with the picture, it is mainly to clarify the point for easy understanding: if you can write with TAPD why to write a note, after writing, also have to stick, stick and tear, it is a tiring thing. Technical documents, too, can be formatted in text in the code base, why doc?
Scrum has been around for more than 20 years since the birth of Agile. Over such a long period of time, Scrum has changed very little on a theoretical level.
But in those 20 years, the world has moved on, and in particular the way people communicate has changed dramatically. From word of mouth, to telephone, to PC, to Internet, to mobile network, to smart devices, to wechat and so on, communication tools have been upgrading.
Code library in public clouds outside the network although haven’t WeChat integration and enterprise, but the version in the tencent has realized the IM very early notice, such as a vacation at home, completely on mobile phones can understand the coding progress of our team, branch merge the news directly through the enterprise WeChat push to mobile phones, who all know to work overtime during the weekend.
Similarly, for TAPD, if there is any change in demand, you can view it directly through the wechat of the enterprise on the mobile phone.
In terms of informatization, the new concept is office without boundaries, including the PC on the desk, the PC at home, mobile terminals and intelligent devices, using a unified authentication and authentication scheme, to achieve the unbounded transmission of information in the process of research and development.
Third, identify a unified communication experience
Tencent is very principled in unifying the communication experience. We know that Tencent has many different forms of business, resulting in many different tools and systems on the R&D tool chain.
However, it ensures that the most basic communication tools are unified.
The R&D Management Department is responsible for the two r&d platforms. TAPD is Tencent’s agile R&D platform, carrying the product planning, iteration, test planning, task assignment, etc. Code is the source Code management platform of Tencent, now it is the worker bee, responsible for the whole Tencent Code base, resource file, branch, review, merge and release to internal open source development link.
So even if Tencent so pay attention to internal horse racing, but the unification of the most basic communication tools is still not lost.
Maintaining the unity of high-frequency communication applications has two major benefits. First, it reduces barriers for business cooperation within the group. When two departments with unrelated business activities cooperate with each other, they only need to add corresponding permissions because the system and account are connected.
Secondly, the unified communication application facilitates the movement of people within the group. Once people change positions, the unified communication experience within the group enables the transferees to quickly adapt to the work.
The bridging form of DevOps systems
Let’s take a look at how DevOps Bridges between systems. DevOps focuses on two processes, one development process and one quality process. Strictly speaking, the R&D loop can contain multiple quality process loops. There are several different tools involved in the closed loop of both processes.
In some areas in particular, it is difficult for a single tool to meet all needs. For example, Jenkins is not good for mobile projects, and games are not good for games. For minicomputers, it is difficult to find open source CI tools to compile. This leads to different tools for different business modalities.
In order to make the whole system flow, tools and tools call each other, for Tencent, the best way is through loose coupling.
Loose coupling is the concept of support but not dependence. Systems call each other using common interfaces and Hooks. Meanwhile, Tencent’s internal R&D system has gradually replaced traditional token authentication with OAuth to ensure multi-party access security.
Let’s start with an example of built-in integration, which is directly provided by platform products, in this case the repository of associated worker bees in TAPD.
Step 1, go to the TAPD project Settings page and click Associate Git project
The second step is to authenticate TAPD’s authorization to worker bees. Here is an OAuthV2 authorization box
Step 3: Select the Git project you want to bind to and click OK
When pushing, a — tapDID parameter in the comment belt binds the corresponding submission point to the requirement.
In just three steps, need to associate your code base.
But the ideal is full, the reality is backbone. Not all users will pay for features designed this way, and teams will often need to tailor the process to their own features. The following uses wechat Android team example to see how the development team does.
The whole Team deals with about 300 ~ 500 requirements per month, and each person makes 10 requirements on average. Collaborative model projects with Feature branches are adopted, with one iteration per month, and 3 to 4 Feature teams are developed in parallel.
This is the branch model used by wechat. It can be seen that it is a typical feature branch publishing model. Key branches are set according to the monthly release rhythm.
Let’s take a look at how they relate requirements to code. Instead of just tying requirements to submissions, they do a full range of things:
The default branch of the project represents the current iteration, with historical releases, current releases, and planned future releases corresponding to the monthly protection branch and release TAG. Each requirement corresponds to one or more Feature branches.
This is the demand page for an iteration of wechat project in TAPD
Here is the merge request page, where they asked developers to paste their own requirements into the merge request:
If the format is not correct, or the requirement does not exist, you will be prompted automatically to fill out the correct requirement form.
Also, if the product manager says the requirement will not be released this month, it can intercept the merge request directly with tapD’s logo.
Also integrated with the code base are automated scanning and testing processes, which, without going into detail, are also loosely coupled to automate calls. Automatically blocks merge requests when scan detection fails, intercepting joins of feature branches at the code level.
Requirements of Tencent enterprise development platform
A mature enterprise-level research and development system, taking code management as an example, should meet the basic requirements in at least four aspects:
Storage for Tencent, the total capacity must be infinite, single library capacity must be able to support such a large map tour level of the project.
At the management level, personnel, authority, and projects must be controlled by the enterprise, which is also a mandatory condition.
At the network level, Tencent has multiple offices, including overseas ones. In order to access projects quickly enough, it is necessary to deploy environment on adjacent nodes to save the bandwidth of dedicated lines. Therefore, it is also a necessary function for Tencent to access multiple places nearby.
In terms of security, it is even more mandatory to delete the database running data because an employee has a conflict with the leader today. At the same time, the business system should also meet various compliance and review requirements within the enterprise, such as checking the access records of a project team last month.
Tencent Worker Bee is a system tailored to Tencent under these requirements, and has now served countless Tencent businesses including wechat.
In the system design, also considered the industry’s advanced ideas and experience. The system directly integrates the capability of code review, supports branch management of a high order, supports conversational development, and displays the trend in the working process through dynamic mode. Meanwhile, the integration and customization capabilities are comprehensive, and every function that can be operated basically has its corresponding API or Hooks.
In the enterprise management aspect also made the multilayer enhancement
After the quiz
Q: Why did Tencent choose Git as the next generation code management platform?
Tencent realized the deficiency of the previous generation OF SCM in 2015-2016. During 2015-2017, Ali, Baidu, Microsoft and other companies have fully switched to Git. The new generation of Git system is not a pure Git, but a highly integrated and multi-capability performance platform.
The system under the new generation of thought design in the use process, imperceptibly increases the communication efficiency in the research and development process, and can settle down the research and development process.
Q: How do you see the difference in permission organization between Git and traditional SCM?
Git subtly cultivates the behavior of business teams to split architecture and components during team practice. It helps the project architecture to be more structured and unstructured, and this behavior of dividing large projects into fine granularity can help decentralize the management of permissions. This is the result of practice.
Q: Can you explain the branch authority in the PPT?
Because the daily management of large projects needs to be considered, many project teams have teams of at least dozens of people, and then divided into several smaller teams. In terms of branch strategy, different teams often have to manage different branches.
At this point, library-level permissions are no longer sufficient, and permissions must be added to the branch. This is also a specific function of worker bees, which will be further optimized in the future.
