Git offers a wealth of branching strategies and workflow approaches, and as we studied Git workflows in the industry, each one was well designed and seemed to apply to team practices. But be careful when introducing Git workflow specification development: Git workflows are just one part of the overall development process. The upstream project management/defect tracking system is in demand, the downstream CD (Continuous Delivery) is in demand, and the team size, product form, release mode, etc. Therefore, implementing Git workflow specifications in a team is not an easy decision.
The effective CR (Code Review) coverage rate of Bytedance Git repository is 70%, and there is still room for improvement. According to the research, GitHub Flow mode is the dominant mode in the team. As byte r&d efficiency has improved and GitHub Flow has been unable to fully utilize r&d facilities for efficiency and engineering quality, many teams have realized this and started building process specifications. This paper introduces Git workflow and research and development facilities in the industry, tries to analyze the warehouse form, deployment process and other aspects, and gives some suggestions on workflow specification.
Introduction to Git workflow in the industry
Git Flow
Novice Git developers, faced with the full map of branches and merge points, would like to tear author. Advanced Git developers are having a hard time putting this process into practice.
Git Flow has several advantages:
- Branches do their job and cover most development scenarios.
- Any COMMIT in the Master branch is expected to be deployable.
- Strictly follow the process, the occurrence of major accidents will be greatly reduced.
There are many disadvantages:
- It is too cumbersome to require all team members to follow this process rigorously.
- Violates the short-lived branch rule advocated by Git.
- The master branch history is not clean and can only be tagged to indicate which branches are actually deployed by the master.
- Unfriendly to continuous deployment and the Monorepo warehouse.
GitHub Flow
GitHub Flow is a lightweight branching based workflow. It highlights the importance of CR and helps us understand the development pattern of CR, but it does not answer the questions of deployment, environment, release, integration, etc.
GitLab Flow
Unlike Git Flow and GitHub Flow, GitLab Flow does not have obvious specifications. It is more a practice based on GitHub Flow, which comprehensively considers environment deployment, project management and other issues.
Based on the environment:
Based on the release plan:
Trunk-based Flow
Similar to the “release plan-based” GitLab Flow, there is a single branch that accepts all developer commits and provides a critical boost for subsequent CI/ CDS.
According to the official document: “You can choose to submit code directly to the trunk (for small teams) or use a pull-request approach, as long as the feature branch is non-permanent and the product is stand-alone.”, Trunk branch commit is optional (not necessarily deployable), but it does require a CR. You can use a fast-forward merge to make sure the trunk is one line, and at the right point in time, a checkout release-* branch is executed.
Once the release branch is found to have a need for hotfix, fix development is done on the trunk branch first. When the test is complete, cherry pick to the release-* branch to ensure that the fix code is online in the release-*. It can be included in the next release cycle.
Aone Flow
According to the alibaba Cloud developer community description: Aone Flow “The basic gameplay is to map each release branch to a specific environment, such as the release/test branch to the deployment test environment, release/ Prod branch to the online formal environment.” This release method ensures that every feature is tested. However, there is no guarantee that the feature passed by release/test CI can also pass by release/ PROd environment (the combination of feature pick is different).
“The gameplay of the advanced point is to link one release branch to multiple environments, such as a grayscale release and a formal release, with manual acceptance steps in between.” In essence, “release/test” and “release/prod” in the basic gameplay were changed to “release/combine-feature”, which fixed the combination of feature picks and ensured the consistency of features in various environments.
The pick mode of Aone Flow is suitable for large teams of complex warehouses to continuously go online, and avoids the problem of the introduction of unfinished feature of Trunk-based Flow. But it does not seem to fit the requirements of periodic release. Multiple features will be created in a release cycle, and the previous release cycle may leave some features. As time goes by, the number of features becomes more and more, and finally the developer goes crazy in the process of picking features.
The company practices
Bytedance’s Web services run on a private cloud CE (Compute Engine), while deployment artifacts are distributed through a unified code compilation, distribution and version management platform, and each build Artifact is managed by an AR (Artifact Repository).
Status of multi-environment deployment
Server perspective
Server microservices run on CE, and code compilation is completed by AR. CE and AR have a 1:N relationship. The operation of an application depends on multiple AR.
From the CE perspective, the company has 5 types of environments (taking domestic products as an example) :
Headers -h ‘x-env-tag:{env}’ is used to direct traffic to different environments to meet the testing requirements of “development test”, “QA test”, “pre-launch test”, “small traffic test”, and “full online”.
CE test environment Service example:
The front view
The front-end deployment is different from the server. A URL path usually produces resources from an AR, and the URL paths and AR are N:1, so the front-end deployment uses the AR version to distinguish the environment:
The front-end deployment can generate independent domain names for the test environment and product preview environment for testing, or set headers -h ‘x-env-tag:{env}’ for environment diversion.
Git workflow for the team practice
Combined with the environmental status of the front and back ends, three kinds of R&D processes can be sorted out:
-
Functional Test Process (Test environment)
-
QA Test process (test environment)
-
Online release process (test, pre-release, grayscale, online environment)
There are currently three Git workflows within the company that correspond to them:
- Trot: single trunk
- Periodic release: Dual trunk
- Periodical release: three trunks
Compare “double trunk” with “single trunk”,
Have a contact:
-
Iteration branch in MR state ≈≈ R&D trunk Dev
-
Single trunk Master ≈≈ Release trunk Master
There are also differences:
- Single trunk “R&D branch” does not have a fixed test environment (compared to double trunk DEV branch)
- When multiple features are deployed in the test environment simultaneously, they need to be combined into new branches, which is inconvenient to manage
- The CI pipeline of single trunk iteration branch is different in MR/non-MR state
Single trunk practice
Front-end microservice management platform
Byte front-end microservice platform belongs to mature business, only need to do a few feature/fix development, in the workflow of single trunk mode.
After the local test, the functional test environment is no longer tested. After a Merge Request is initiated, the CR passes the Merge code and is tested in the benchmark environment. If a problem is found, the CR is rolled back to the next CR.
Although minor modification has little impact risk, but the process lacks the process to enter the functional test environment, there are still hidden dangers, may affect the business side of the test environment to use, is not a good practice.
Single trunk is only suitable for small business scale, high maturity and no major changes in the project. However, no matter the scale of the business, the online publishing process must be verified in an offline environment first.
Dual trunk practice
Private Cloud Platform
Git workflows for cloud platforms are complicated to simple. Initially, a deployment branch was set up for each deployment environment. The commit point of the feature branch needs to merge with the deployment branch of the three environments.
After improvement, the standard trunk-based Flow is adopted, which can still meet the deployment requirements of online/ Sandbox/BOE environments.
There are hundreds of business parties participating in cloud platform (similar to Ali Cloud platform). Although only three environments are shown in the figure, in fact, the use of 5 environments is integrated into daily development.
A service center
Git workflow of a business center focuses on the problems encountered in the collaborative development of the same project by multiple people:
- Multiple features are independently tested, and there are many conflicts near the online code, which may lead to online bugs
- Before and during test promotion, if the master is updated, it may not be synchronized in time. Merging the master before going online may cause conflicts or bugs
- In terms of process design, master is the release branch, and release-* is the test branch, which combines the convenience of single trunk (Hotfix directly interacts with master) and dual-trunk feature management
- In contrast to trunk-based Flow, the main branch is the release branch and the test branch is short
- Another characteristic is that in the process of release testing, if a bug of a feature is found, it is directly checked out from the release branch to fix it, and then closed into the release again
Jupiter Workflow Specification
Jupiter is a byte Web development engine covering Web, component libraries, BFF, SSR and other front-end development areas. Jupiter recommends flows similar to Trunk-based flows and gives suggestions on How to do Git when developing new requirements, hotfixes, etc., from a CI/CD perspective.
- There is a consistent process and pattern between projects involving overlapping personnel to avoid increasing cognitive burden, avoid confusion and confusion when the same person switches between different projects, and focus on practice and improvement, sharing experience and infrastructure.
- Be flexible online. Take care of both early projects and large-scale projects, taking into account the frequency of rollout as projects pursue different milestones. Therefore, it is possible to go online once a week, once a day (suitable for projects with a large number of people and high stability requirements), or to go online several times within a day for multiple features. A fixed rhythm cannot be defined. Anyone can launch at any time.
- Support monorepo. Projects involving different people in different directions may share a common repository, making it easier to reuse code and infrastructure. Therefore, different projects in the warehouse may have different online rhythms and online requirements.
- Continuous integration (CI) is encouraged. Integration is not the same as deployment. MR integration code can be delivered without pressure and without being brought online without knowing it. Continuous deployment (CD) is also encouraged. Deployment does not mean release. Code that cannot be released has a chance to be closed before it goes live.
- The on-line process must be fixed, repeated, capable of unified improvement, and capable of gradually increasing automation. It is not allowed to re-plan or modify the on-line method temporarily every time when the on-line is launched, which will increase the burden and cost.
- CI is the premise of CD. You can’t enter CD without modifying CI.
- No changes can be put online without pre-staging (pre-release, as much as possible online).
- The history of CI and CD should be absolutely reliable and traceable and can only be added, not reduced or modified.
- Minimize manual operations and avoid online operations on specific personal machines.
Three trunk practice
Million level App
App distribution is by far the most complex branch management scenario. Once the client installation package is delivered to the channel and downloaded by users, if it cannot be repaired through hot update, App user retention will be severely affected.
App distribution has more standardized distribution rules, and Feature Toggle is essential. When we feel CR, CI/CD is troublesome, do we feel that it is much easier to do Web CI/CD than to develop and launch an App Feature that affects hundreds of millions of users?
conclusion
This article tries to explain how to use Git workflow from as many angles as possible. Git workflows are necessary to ensure that the tests are progressive and reliable. The combination of environment management and Git workflow has also formed a number of specifications within the Headlines, including “environment deployment”, “traffic scheduling”, “connectivity testing” and other usage specifications; Environmental constraints such as Limited scenario Permit, Temporary scenario Permit, and Limited process Permit. Combined with CI/CD, we can ensure the fast iteration and safe launch of the whole link.
The resources
-
Trunk-based Development vs. Git Flow
-
Please stop recommending Git Flow!
-
Understanding the GitHub flow
-
Introduction to GitLab Flow
-
cn.trunkbaseddevelopment.com
-
At Ali, how do we manage our code branches?
-
Google code management
Welcome to the Byte Front end
Resume delivery email address”[email protected]”