From my blog, ryougifujino.com.

Preface: It took me nearly 3 months to translate, polish, and correct this long article on and off. I had an Epiphany when I read this article by Martin Fowler (his website martinfowler.com/ is a treasure trove to me). To be honest, I’ve read a lot of articles about version control workflows, and I’ve always felt like I was skimping on some of them or avoiding them, but this article has covered the pros and cons of each model and answered some of the questions I had in mind. It’s a good idea to look at which patterns should be used in which scenarios (for commercial software, continuous integration is best).

---

By Martin Fowler

The original address

Modern source control systems provide powerful tools that make it easy to create branches in your source code. But eventually all the branches had to come together, and many teams spent a lot of time gluing the messy tangle of branches together. There are several patterns that allow teams to use branches effectively and to focus around integrating the work of multiple developers and organizing the path to production release. The overall theme is that branches should be integrated frequently, and efforts should be focused on healthy mainlines that can be released to production with minimal cost.

---

Source code is a vital asset to any software development team, and over the past few decades, a great deal of source code management software has been developed to keep code in order. These tools allow changes to be tracked, allowing us to rebuild previous versions of software and see how it has evolved over time. These tools are also coordinate centers with multiple teams of programmers, all working on a common code base. By recording the changes made by each developer, these systems can track multiple lines of work at once and help developers figure out how to combine those lines of work together.

This splitting and merging of lines of work is a core workflow for software development teams, and some patterns have evolved to help us deal with these activities. Like most software patterns, they are not golden rules that teams must follow. The workflow of software development is very dependent on the context, in particular the social structure of the team and other practices followed by the team.

In this article, my task is to discuss these patterns, and to do so in the context of a single article. In this article I describe the patterns, but intersperse the explanation of the patterns with narrative parts that better explain the interrelationships between the environment and the patterns. To make it easier to distinguish them, I’ve marked the parts of the pattern with the “✣” symbol.

The basic pattern

In thinking about these patterns, I find it useful to develop two main categories. One group looks at integration, how multiple developers combine their work into a coherent whole. The other group looks at the path to production, using branches to help manage the path from the integrated code base to the product running in production. There are models that support both of them, and I would consider these to be the basic models. The remaining models are neither basic nor fall into either of the two main groups — so I’ll save them for last.

Source branch behavior ✣

Create a copy and record all changes to that copy.

If several people work in the same code base, working on the same files can quickly become impossible. If I want to run the compilation while my colleague is typing an expression, the compilation will fail. We’ll have to yell at each other, “I’m compiling, don’t change things.” Even with two people, this is hard to sustain, let alone a larger team.

The simple answer to this question is that every developer keeps a copy of the code base. Now we can easily work on our own functionality, but a new problem arises: when we’re done, how do we merge our two copies back together?

Source control systems make this process easier. The key is that it records every change on every branch as a commit. Not only does this ensure that no one forgets the small changes they made to utils.java, but recording the changes makes merging easier, especially if several people have modified the same file.

This leads me to the branch definition I’ll use in this article. I define a branch as a specific commit sequence in the code base. The head, or tip, of the branch is the most recent commit in the sequence.

1. A branch is a series of commits
2. The header (or top) is the latest submission in this series

This is a noun, but there’s also a verb, “to branch.” By this I mean creating a new branch, or you can think of it as splitting the original branch in two. Branches merge when a commit from one branch is applied to another.

1. When two commits are created in parallel, a branch will split
2. One branch can be merged into another. There will be some work to deal with these parallel changes
3. If there have been no changes since the last merge, a change on one branch can simply be applied to the other

The definition I use of “branches” is consistent with the way MOST developers I’ve observed talk about them. But source control systems tend to use “branching” in a more specialized way. I can illustrate this point with a situation common to modern development teams that keep their source code in a shared Git repository. A developer named Scarlett needed to make some changes, so she cloned the Git repository and checked out the master branch. She made a few changes and submitted them to her master. Meanwhile, another developer, we’ll call her Violet, cloned the repository onto her desktop and checked out the master branch. So do Scarlett and Violet work on the same branch, or on different branches? They both work on a “master,” but their commits are independent of each other, and when they push their changes to a shared repository, they need to merge. What happens if Scarlett isn’t sure about the changes she made, so she tags the last commit and resets her master branch to Origin/Master (the last commit she cloned from the shared repository)?

According to the definition of branches I gave earlier, Scarlett and Violet work on separate branches, separate from each other and from the master branch of the shared warehouse. When Scarlett puts her work aside with a tag, it’s still a branch by my definition (and she probably thinks it is a branch), but in Git terms it’s a tagged codeline.

In a distributed version control system like Git, this means that every time we further clone a repository, we also get additional branches. If Scarlett cloned her local warehouse and put it on her laptop (in preparation for the train ride home), she created a third master branch. The same effect happens when forking on GitHub — each repository that is forked has its own set of additional branches.

This confusion in terminology is compounded when we come across different version control systems, each of which has its own definition of what a branch is. A Mercurial branch is completely different from a Git branch, which is closer to the Mercurial concept of a bookmark. Mercurial can also create branches with unnamed heads, and Mercurial users often create branches by cloning repositories.

All the confusion about the term has led some people to avoid it. A more general term that will be useful here is codeline. I define a codeline as a sequence of specific versions of a code base. It can end with a tag, a branch, or be lost in Git’s reflog. You’ll notice a strong similarity between my definition of a branch and my definition of a codeline. Codeline is in many ways the more useful term, and I do use it, but it’s not widely used in practice. So in this article, I’ll use the terms branch and codeline interchangeably unless I’m in the specific context of Git (or other tool) terminology.

The result of this definition is that no matter what version control system you use, every developer who makes local changes will have at least one personal codeline in the working copy on their machine. If I clone a git repository for a project, check out the master, and update some files — even before I commit anything, it’s a new codeline. Similarly, if I create a working copy of the trunk of my own Subversion repository, that working copy is its own codeline, even though there is no Subversion branch involved.

When to Use it

An old joke says that if you fall from a tall building, the fall doesn’t hurt you, but the landing does. So, for source code: easy to branch, harder to merge.

Source control systems that document every change on the commit do make the merge process easier, but they do not make the merge trivial. If Both Scarlett and Violet change a variable to a different name, the source code management system cannot resolve the conflict without human intervention. What’s even more embarrassing is that at least this kind of text conflict is something source control systems can detect and remind humans to take a look at. But often another kind of conflict occurs: the text merge is fine, but the system still doesn’t work. Imagine that Scarlett changed the name of a function, and Violet added some code to her branch to call the function under its old name. This is what I call semantic conflict. When these conflicts occur, the system may not be able to build, or it may be able to build but errors occur at runtime.

Jonny LeRoy likes to point out this flaw in how people (including me) draw branching diagrams

1. How do most people draw branching diagrams
2. How should we actually plot them to show the divergence over time

This problem will be familiar to anyone who has worked in concurrent or distributed computing. We have some shared state (code base) that developers can create updates in parallel. We need some way to combine these updates by serializing them into some consensus updates. Our task is further complicated by the fact that getting a system to execute and run correctly means that the shared state has very complex criteria for effectiveness. There is no way to create a definitive algorithm to find consensus. Humans need to find this consensus, and this consensus may involve mixing the selected parts of different updates. Often, a consensus can only be reached if the conflict is resolved through the original update.

Me: “What if there were no branches”. Everyone would edit the code in real time, half-baked changes would crash the system and people would step on each other. As a result, we give people the illusion of freezing time, that they are the only ones making changes to the system, and that those changes can wait until they are fully implemented before putting the system at risk. But this is a facade that will eventually pay the price. Who pays? When? How much is it? This is what these models talk about: an alternative to paying piper.

– Kent Beck

So for the rest of this article, I’ve listed patterns that support pleasant isolation and the high winds around your hair as you fall, but minimize the inevitable consequences of contact with the hard ground.

The mainline ✣

A single, shared branch as the current state of the product.

The mainline is a particular codeline that we consider to be the current state of the team’s code. Whenever I want to start a new job, I pull the code from the mainline into my local repository and start working again. Whenever I want to share my work with the rest of the team, I update my work to the mainline, preferably using the mainline integration pattern I’ll discuss shortly.

Different teams use different names for this particular branch, often dictated by the conventions of the version control system used. Git users usually refer to it as “master” and Subversion users as “Trunk.”

I must emphasize here that the mainline is a single, shared codeline. When people talk about “master” in Git, they can mean several different things, because each clone of the repository has its own local master. Often such teams have a central repository — a shared repository that serves as a single point of record for the project and is the source of most clones. Starting a new job from scratch means cloning the central warehouse. If I already have a clone, I start by pulling the master from the central repository so that it is consistent with the main line. In this case, the main line is the Master branch in the central repository.

When developing my functionality, I will have my own personal development branch, which may be my local master, or I may create a separate local branch. If I’ve been working on this for a while, I can keep up with the latest mainline changes by pulling the mainline changes every once in a while and incorporating them into my personal development branch.

Similarly, if I want to create a new version of a product for release, I can start with the current mainline. If I need to fix bugs to make the product stable enough to release, I can use a release branch to do that.

When to Use it

I remember going to talk to a client’s build engineer in the early 2000s. His job is to build the product that the team is working on. He would send an email to each member of the team, after which they would respond by sending a file of the code base they were preparing to integrate. He then copies these files into his integration tree and tries to compile the code base. It usually takes him a few weeks to create a compilable build and prepare it for some form of testing.

By contrast, with the mainline, anyone can quickly start a new product build from the top of the mainline. In addition, mainline not only makes it easier to see the state of the code base, but it also underlies many of the other patterns I’ll discuss shortly.

An alternative to the mainline is to release trains.

The healthy branch ✣

On each commit, an automated check is performed, typically by building and running tests to ensure that there are no defects on the branch.

Since the mainline has a shared and validated state, it is important to keep it stable. Again in the early 2000s, I remember talking to a team at another organization that was known for building each of their products on a daily basis. This was considered fairly advanced at the time, and the organization was praised for doing so. What is not mentioned in these articles is that these daily builds are not always successful. In fact, it’s easy to find teams whose daily builds haven’t been compiled for months.

To solve this problem, we can strive to keep the branch healthy — meaning that it builds successfully and that the software has few, if any, bugs at run time. To ensure this, I’ve found it crucial to write Self Testing Code. This development practice means that we write a full set of automated tests as well as production code, so that we can be sure that if those tests pass, the code will not contain any bugs. If we do this, then we can keep a branch healthy by running a build at each commit, which includes running the test suite. If the system fails to compile, or the tests fail, then our first priority is to fix them before we do anything else on that branch. This usually means that we “freeze” the branch — no commit will be allowed until the fix makes the branch healthy again.

There is a tension around the level of testing to provide enough healthy confidence. Many of the more thorough tests took a lot of time to run, delaying the submission of healthy feedback. The team addresses this problem by breaking testing into multiple phases on the Development Pipeline. The first phase of these tests should run quickly, usually no more than 10 minutes, but should still be comprehensive. I refer to such a suite as a commit suite (although it is often referred to as a “unit test” because commit suites are usually mostly unit tests).

Ideally, all tests should be run on each commit. However, if testing is slow, such as a performance test that requires soaking the server for several hours, this may not be realistic. Now, teams can typically build a commit suite that can run on each commit and run later stages of the deployment pipeline as often as possible.

The fact that the code runs bug-free is not enough to say that the code is good. To maintain a steady delivery rate, we need to maintain the internal quality of the code. One popular approach is to use pre-Integration Review, although there are other options, as we will see.

When to Use it

Each team should have clear criteria for the health of each branch of its development workflow. There is tremendous value in keeping the main line healthy. If the mainline is healthy, then developers can start a new work by pulling the current mainline without being bothered by defects that prevent them from doing their job. We often hear people spend days fixing or tweaking bugs in code they’ve dropped before starting a new project.

A healthy thread can also smooth the path to production. A new production candidate can be built from the mainline header at any time. The best teams find that they need to do very little to stabilize such a code base and are always able to release directly from mainline to production.

The key to having a healthy mainline is self-test code (whose commit suite can run in minutes). Building this capability can be a huge investment, but once we can make sure my commit doesn’t break anything in a few minutes, it can completely change our entire development process. We can make changes faster, confidently refactor our code to make it easier to work, and dramatically shorten the cycle time from the capabilities we need to the code running in production.

For personal development branches, it is wise to keep them healthy because this enables Diff Debugging. But this desire is at odds with frequent submissions to checkpoint the current state. If I’m prepared to try a different path, I might create a checkpoint if the compilation fails. My solution to this paradox is to squash any unhealthy commits once I’m done with the current work. This way, only healthy commits remain on my branch after a few hours.

If I keep my personal branch healthy, it will also make it easier for me to commit to the mainline — I will know that any errors in the mainline integration are caused by the integration, not by errors in my code base. This will make it quicker and easier for me to find and fix them.

Integration patterns

Branching behavior is about managing the interaction between isolation and integration. It doesn’t work to have everyone working in a single shared code base all the time, because if you’re in the middle of typing in a variable name, I can’t compile. So at least in part, we need the concept of a private workspace that we can work on for a while. Modern source control tools make it easy to manipulate branches and monitor changes on those branches. At some point, however, we need to integrate. Thinking about branching strategy is really about how and when we integrate.

Main line integration ✣

Developers integrate their work by pulling, merging, and — if healthy — pushing back from the mainline.

The mainline gives a clear definition of what the current state of the team’s software is. One of the biggest benefits of using mainlines is the ease of integration. Without the mainline, the collaboration between each member of my aforementioned team would be a complex task. However, with the mainline in place, each developer can integrate on their own.

I’ll show you an example of how this works. A developer, I’ll call her Scarlett, started working by cloning the mainline into her own warehouse. In Git, if she doesn’t already have a clone of the central repository, she clones it and checks out the master branch. If she already has a clone, she will pull the mainline to her local master. She can then work locally and submit to her local master.

While she was working, her colleague Violet pushed some changes to the mainline. Because she works within her own codelines, Scarlett can forget about these changes while she focuses on her own tasks.

At some point, she got to the point where she wanted to integrate. The first part of the integration is to fetch the mainline’s current state to her local master branch, which will pull in Violet’s modifications. Since she is working on the local master, the submission will appear as a separate codeline on Origin/Master.

Now she needed to combine her modifications with Violet’s. Some teams like to merge, others rebase. Generally, people use the word “merge” when talking about integrating branches together, whether they’re actually using git merge or rebase operations. I’ll follow this usage, so unless I’m really talking about the difference between merge and rebase, think of “merge” as a logical task that can be implemented either way.

There is a whole other discussion about whether to use native merge, to use or avoid fast-forward merges, or to use rebase. This is beyond the scope of this article, although if people send me enough Tripel Karmeliet, I might write an article about it. After all, swapping (quid-pro-Quos) is all the rage these days.

If Scarlett is lucky, merging Violet’s code will be a clean merge, if not, she’ll have some conflicts to deal with. These can be textual conflicts, most of which can be handled automatically by source control systems. But semantic conflicts are harder to deal with, which is where self-testing code comes in. (Since conflicts create a fair amount of work, and always introduce risk to a fair amount of work, I’ve highlighted them in bright yellow.)

At this point, Scarlett needs to verify that the merged code meets the health criteria for the mainline (assuming it is a health branch). This usually means building code and running any tests from the mainline commit suite. She needs to do this even with a clean merge, because even a clean merge hides semantic conflicts. Any failure in the commit suite should be purely due to merging, as both merge parents should be normal. Knowing this should help her figure out what the problem is because she can look for clues in diff.

Through this build and testing, she has successfully pulled the mainline into her codeline, but — this is both important and often overlooked — she has not yet completed the integration with the mainline. To complete the integration, she must push her changes onto the mainline. Unless she does so, everyone else on the team will be isolated from her changes — there is virtually no integration. Integration is both a pull and a push — only what Scarlett has pushed is integrated with the rest of the project.

Recently, many teams have required code reviews before adding a commit to the mainline — a pattern I call pre-integration reviews, which I’ll discuss later.

Occasionally, someone will integrate with the mainline before Scarlett pushes. In this case, she has to pull and merge again. Often this is just an occasional problem that can be solved without further coordination. I’ve seen teams with long build processes use the integration baton so that only the developer with the baton can integrate. But in recent years, as build times have improved, I hardly ever hear about it again.

When to Use it

As the name suggests, I can only use mainline integration if our product is using mainline integration.

One option to use mainline integration is to pull directly from the mainline and incorporate the changes into the personal development branch. This can be useful — at the very least, pulling can alert Scarlett to changes that others have integrated and uncover conflicts between her work and the mainline. But until Scarlett pushes, Violet won’t be able to see any conflict between her work and Scarlett’s changes.

When people use the word “integration,” they tend to overlook this important issue. It’s common to hear people say they’re integrating mainlines into their branches, and they’re just pulling. I’ve learned to be wary of such statements and explore further to see if they refer to mere pull or proper mainline integration. The results are very different, so it is important not to confuse these terms.

Alternatively, Scarlett is in the middle of some work that isn’t ready to be fully integrated with the rest of the team, but those work overlap with Violet and Scarlett wants to share them with her. In this case, they can choose to create a collaborative branch.

Branch mode ✣

Put all the work for a feature on its own branch and integrate it into the mainline when the feature is complete.

In using the feature branching pattern, the developer opens a branch at the start of the work, continues to work until the feature is complete, and then integrates into the mainline.

For example, let’s take a look with Scarlett. She developed the ability to add collections of local sales taxes to their website. She starts with a stable version of the current product, she pulls the mainline into her local repository, and then creates a new branch from the top of the current mainline. She developed the feature whenever needed, making a series of commits on the local branch.

She may push this branch to the project’s repo so that others can see her changes.

Other submissions continued to fall on the mainline as she worked. Therefore, she may pull from the main line from time to time, so she can see if there are changes that might affect her future.

Note that this is not the integration I mentioned above, as she does not push back to the mainline. She was the only one who could see her work. No one else could.

Some teams like to ensure that all code, whether integrated or not, is kept in a central repository. In this case, Scarlett pushes her feature branches to a central repository. It also allows other team members to see what she’s doing, even if it hasn’t been integrated into other people’s work.

When she is finished working on the feature, she will perform mainline integration to incorporate the feature into the product.

If Scarlett is developing more than one feature at a time, she opens a separate branch for each feature.

When to Use it

Functional branching mode is a popular mode in the industry today. To discuss when to use it, I need to introduce its main alternative, continuous integration. But first I need to talk about the integration frequency.

Integrated frequency

The frequency with which we integrate has a big impact on how a team operates. Research from the State Of Dev Ops Report shows that elite development teams integrate significantly more often than underperforming teams — an observation that is consistent with my experience and that Of many Of my industry peers. I’ll illustrate this with an example of two integrated frequencies led by Scarlett and Violet.

Low frequency integration

Let me start with the low frequencies. Here, our two heroes start a chapter of work: clone the main line to their branch, and then complete some local commits that they don’t want to push yet.

While they were working, others put some submissions on the main line. (I can’t think of another vivid name off the top of my head — maybe Grayham?)

The way the team works is to keep a healthy branch and pull from the main line after each commit. Scarlett couldn’t pull anything after the first two commits because the mainline hasn’t changed, but now she needs to pull down the M1.

I’ve highlighted the merge nodes in yellow. This node merges S1… 3 and M1. Soon, Violet will need to do the same.

At this point, both developers are updating to the latest mainline, but they are not integrated yet because they are isolated from each other. Scarlett was unaware of any changes Violet had made from V1 to V3.

Scarlett has completed a few more local commits and is ready for mainline integration. It was easy for her because she had pulled the M1 earlier.

Violet, however, has a more complicated assignment. When she does mainline integration, she must now convert S1… 5 with V1… 6 integration.

I scientifically calculated the size of the merger based on the number of submissions involved. But even if you ignored the tongue-like bulge on my cheek, you would agree that Violet’s amalgamation was the most likely to produce difficulties.

High frequency integrated

In the previous example, our two colorful developers integrated after a few local commits. Let’s see what happens if they do mainline integration after each local commit.

Violet’s first submitted changes were clear because she integrated them right away. Since the mainline hasn’t been modified, this is just a simple push.

Scarlett’s first commit also required mainline integration, but since Violet had done it, she needed a merge. But since she only needs to merge V1 and S1, the merging effort is minimal.

Scarlett’s next integration is a simple push, which means that Violet’s next commit also needs to be merged with Scarlett’s latest two commits. However, it’s still a fairly small merger: one from Violet, two from Scarlett.

When an external commit pushed to the mainline appears, it will be integrated by both Scarlett and Violet at their usual pace.

Although this is similar to what happened before, the integration is smaller. Scarlett only needs to integrate S3 with M1 this time because S1 and S2 are already on the mainline. This means that Grayham must integrate anything that is already on the mainline before pushing M1 (S1… 2, V1… 2).

The developers continue with the rest of their work and integrate with each commit.

Comparative integration frequency

Let’s look at these two general pictures again.

Low frequency

High frequency

There are two very clear differences. First, high-frequency integrations, as the name suggests, make more integrations — twice as many in this toy example alone. But what’s more important is that these integrations are much smaller than in the low frequency case. Smaller integration means less work, because fewer changes cause less conflict. But more importantly than less work, there is also less risk. The problem with big mergers is not the jobs they involve, but the uncertainty of those jobs. Most of the time, even big mergers go well, but occasionally things go really, really bad. In the end, the occasional pain is worse than the regular pain. Which one would I prefer if I spent 10 minutes more on each integration than if I had a 1 in 50 chance of taking 6 hours to fix an integration? If you just look at the amount of work, there’s a 1 in 50 chance it’s better because it takes six hours instead of eight hours and 20 minutes. But this uncertainty makes the 1 in 50 situation feel worse, and this uncertainty leads to the fear of integration.

Integration fears When teams have had several bad merge experiences, they tend to be wary of integration. This could easily become a positive feedback loop — and like many positive feedback loops, have very negative consequences. The most obvious consequence is that teams reduce the frequency of integration, which leads to more unfriendly merges, which leads to lower frequency of integration… And so on. A more subtle problem is that teams stop doing things they think will make integration more difficult. In particular, this will make them resistant to refactoring. However, reducing refactoring leads to an increasingly unhealthy code base that is difficult to understand and modify, thus dragging down the team’s functional delivery. Since the functionality takes longer to complete, this further reduces the frequency of integration and promotes a debilitating positive feedback loop. The counterintuitive answer to this question is as the slogan says – “If suffering…… Do it more often.”

Let’s look at the difference between these frequencies from another perspective. What happens if the initial commit between Scarlett and Violet is in conflict? When do they realize the conflict has already occurred? In the low frequency case, they don’t find out until Violet’s final merge, because that’s when S1 and V1 are first put together. But in the case of high frequency, they would have found out the first time Scarlett merged.

Low frequency

High frequency

Frequent integration increases the frequency of mergers but reduces their complexity and risk. Frequent integration also alerts teams to conflicts more quickly. Of course, the two things are related. Nasty mergers are often caused by conflicts that lurk within the team’s work and only surface when integration occurs.

Perhaps Violet is looking at a billing calculation and sees that it includes an assessment tax (the author hypothesizes a particular tax mechanism). Her function requires a different treatment of the tax, so the direct approach is to take the tax out of the billing calculation and make it a separate function later. Billing calculations are only called in a few places, so it’s easy to process them using Move Statements to Callers — and the results are more meaningful for future development of the program. However, Scarlett has no idea that Violet is doing this, and she writes her feature assuming that the billing feature has already addressed the tax issue.

Self-test code is our lifeline here. If we have a robust test suite and use it as part of the health branch to find conflicts, we can greatly reduce the chance of bugs entering production. But even with a strong test suite serving as the gatekeeper for the mainline, large integrations can make life harder. The more code we have to integrate, the harder it is to find bugs. There is also a higher chance of multiple bugs that interfere with each other, and these bugs are particularly difficult to understand. If the commits are small, not only do we have less to look at, but we can also use the Diff Debugging to help narrow down the changes that are causing the problem.

Many people don’t realize that a source control system is a communication tool. It allows Scarlett to see what the rest of the team is doing. With frequent integration, she not only gets immediate alerts when conflicts arise, but also has a better idea of what everyone is doing and how the code base is evolving. We are not like individuals struggling with difficulties alone, but more like a team working together.

One important reason for reducing the size of functionality is to increase the frequency of integration, but there are other benefits as well. The smaller the features, the faster they can be built, the faster they can get into production, and the faster they can start providing value. In addition, smaller features reduce feedback time, allowing teams to make better feature decisions while learning more about their customers.

Continuous integration ✣

As soon as developers have a healthy commit to share, mainline integration takes place, usually in less than a day’s work.

Once a team experiences high frequency integration as both more efficient and less stressful, the natural question to ask is “How often can we?” The branching feature pattern implies a lower limit on the size of a variable set – you can’t get smaller than a cohesive feature.

Continuous integration uses a different set of integration triggers — as long as you’ve made some progress on a feature and your branches are still healthy. We don’t expect the functionality to be complete, just that there are valuable changes that can be committed to the code base. The rule of thumb is “everyone commits to the mainline every day,” or more precisely: there should be no more than one day’s work in your local repository that is not integrated. In practice, most practitioners of continuous integration integrate many times a day and are happy to integrate for an hour or less.

For more details on how to do continuous integration effectively, see my detailed article. For more details, see this book by Paul Duvall, Steve Matyas, and Andrew Glover. Paul Hammant at trunkbaseddevelopment.com maintains a full of continuous integration technology website.

Developers using continuous integration need to get used to achieving integration points frequently, along with partially built functionality. They need to think about how to do this without exposing partially built functionality in the operating system. Often this is easy: if I’m implementing a discount algorithm that relies on a coupon code that is not yet in the valid list, my code will not be called even in production. Similarly, if I am adding a feature that asks an insurance claimant if he or she smokes, I can build and test the logic in the code behind it and ensure that the feature is not used in production by leaving the user interface for asking the question until the last day of implementation. It is often an effective technique to hide partially built functionality by connecting the Keystone Interface at the end.

If there is no easy way to hide parts of the functionality, we can use Feature Flags. In addition to hiding a partially built feature, this flag allows you to selectively reveal that feature to a subset of users — which is handy for gradually rolling out a new feature.

The way in which functionality is partially built is also of concern to those who worry about introducing bugs into the mainline. Therefore, people who use continuous integration also need to test their code, so they can be confident that even if partially built functionality appears on the mainline, it will not increase the chance of bugs. In this way, developers write tests for the partially built functional code as well as for it, and deliver the functional code along with the tests to the mainline (perhaps using test-driven development).

As far as local repositories are concerned, most people who use continuous integration don’t bother to work with a separate local branch. This is usually done directly to the local master, and then mainline integration is done. However, it’s also great to open a feature branch and work there if the developer prefers, as long as it integrates back to the local master and trunk at frequent intervals. The difference between the feature branching pattern and continuous integration is not whether there is a feature branching, but when the developer integrates with the mainline.

When to Use it

Continuous integration is an alternative to the functional branching pattern. The trade-offs between the two are enough to merit their own chapter in this article, and it’s time to address them.

When Thoughtworks started using continuous integration in 2000, we wrote CruiseControl, a daemon that automatically builds software products after each commit to the mainline. Since then, many such tools (such as Jenkins, Team City, Travis CI, Circle CI, Bamboo, and many others) have been developed. But most organizations that use these tools automatically build functional branches at commit time — which, while useful, means they don’t really practice continuous integration. (A better name for them might be continuous build tools). Because of this semantic proliferation, some people are beginning to use the term “mainstream-based development” instead of “continuous integration.” (Some people do make a subtle distinction between the two terms, but there isn’t a consistent use.) Although I’m generally a descriptivist when it comes to languages, I prefer to use “continuous integration”. Partly because I don’t think constantly coming up with new terms is a viable way to combat semantic proliferation. Perhaps the main reason, however, is that I think changing the terminology would crudely erase the contributions of early EXTREME Programming pioneers, particularly Kent Beck, who created and clearly defined the practice of continuous integration in the 1990s.

Compare the functional branching pattern to continuous integration

The functional branching pattern seems to be the most common branching strategy in the industry today, but there is a vocal group of practitioners who believe that continuous integration is generally a better approach. The key advantage of continuous integration is that it supports higher, often much higher, integration frequencies.

The frequency of integration varies depending on how small a team can make it. If a team’s functionality can all be completed in one day, then the feature branching pattern and continuous integration are both ways they can implement it. But most teams have longer features than that — and the greater the feature length, the greater the difference between the two modes.

As I’ve already pointed out, the more frequently you integrate, the less complex you will be, and the less fear you will have of integration. This is often a difficult thing to communicate. If you live in a world where integration happens only once every few weeks or months, integration is likely to be a very unsettling activity. It may be hard to believe that this is something that can be done many times a day. However, integration is something that is less difficult to use with frequency. It’s a counterintuitive concept — “If it hurts — do it more often.” But the smaller the integrations, the less likely they are to become epic mergers of pain and despair. The feature branching pattern advocates smaller features: days rather than weeks (months don’t exist anymore).

Continuous integration allows teams to reap the benefits of high frequency integration while decoupling feature length from integration frequency. If a team prefers a one-week or two-week feature length, continuous integration allows them to do so and still reap all the benefits that come from the highest integration frequency. With smaller merges, there is less work to handle. What’s more, as I explained above, doing mergers more frequently reduces the risk of unpleasant mergers, which reduces both the bad surprises and the overall fear of mergers. If there are conflicts in your code, high-frequency integration can quickly detect them before they cause those pesky integration problems. These benefits are powerful enough that some teams have functionality that lasts only a few days and still have continuous integration.

The obvious drawback of continuous integration is that it lacks closure for the climax integration to the mainline. This is not only a loss of celebration, but also a risk for a team that is not good at maintaining healthy branches. Putting all the feature submissions together can also delay a decision on whether to add a feature to an upcoming release. Although Feature Flags allows features to be turned on or off from the user’s perspective, the code for this feature is still in production. Concerns about this are often overblown — code is weightless, after all — but it does mean that teams that want to do continuous integration must develop a strong set of testing guidelines so that they can be sure that the mainline will remain healthy even if there are many integrations per day. Some teams find this skill unimaginable, but others find it both possible and liberating. This prerequisite does mean that the functional branching pattern is better suited to teams that do not insist on healthy branching and need a release branch for pre-release stable code.

Although the size and uncertainty of merging is the most obvious problem with the functional branching pattern, its biggest problem may be that it prevents refactoring. Refactoring is most effective when it is done regularly and with little friction. Refactoring introduces conflicts, and if those conflicts are not quickly discovered and resolved, merging becomes difficult. Therefore, refactoring works best in high-frequency integration, so it’s no surprise that it’s become popular as part of Extreme Programming (which also made continuous integration one of its original practices). The feature branching pattern also discourages developers from making changes to parts of the feature that are not being built, which undermines the ability to refactor steadily improving code bases.

When I come across scientific studies of software development practices, I am usually unconvinced because of the serious problems with their methods. One exception is the State Of Dev Ops Report, which developed metrics Of software delivery performance that they correlated to broader organizational performance measures, which in turn correlated to business metrics such as return on investment and profitability. In 2016, they first evaluated continuous integration and found that it helped improve software development performance, a finding that has been repeated in every survey since.

We found that a branch or fork has a very short lifecycle (less than a day) and a total of fewer than three active branches before being merged into the trunk, which are important aspects of continuous delivery and all contribute to improved performance. The same goes for merging code into Thunk or Master every day.

— State of DevOps Report 2016

Using continuous integration does not eliminate the other advantages of keeping functionality small. Frequent releases of small features provide a quick feedback cycle, which can go a long way toward improving the product. Many teams that use continuous integration also strive to build thin products and release new features as often as possible.

Function branch pattern ✅ all the code in a function can be used as a unit for quality evaluation ✅ function code only after completion of the function will be added to the product to the lower frequency ❌ merger of continuous integration ✅ support higher frequency than function length integration ✅ find less time conflict ✅ smaller ✅ encourages refactoring ❌ merger The commitment to maintaining a healthy branch (and therefore the need to self-test the code) ✅ scientific evidence shows that it helps improve software delivery performanceCopy the code

Functional branching patterns and open source

Many attribute the popularity of the feature branching pattern to GitHub and the pull-Request model from open source development. With this in mind, it is important to understand the huge differences between open source efforts and many commercial software developments. There are many different structures for open source projects, but a common structure is one in which a single person or small group acts as maintainer and does most of the programming. Maintainers work with a larger group of programmers as contributors. Maintainers are often unaware of contributors and have no expectations about the quality of the code they contribute. Maintainers also can’t be sure how much time contributors will actually put into their work, let alone how effective they will be at getting it done.

In this case, the functional branching pattern makes a lot of sense. If someone is going to add a feature, big or small, and I don’t know when (or if) it will be done, then it makes sense to wait for it to be done before integration. Moreover, it’s more important to be able to review the code to make sure it passes any quality thresholds I set for the code base.

But many commercial software teams work in a very different environment. They will have a full-time team of people who will focus on software. The project leader knows these people well (except when they first started) and can have a solid expectation of code quality and delivery. Because they are paid employees, leaders also have greater control over the time devoted to projects, as well as aspects such as coding standards and team habits.

I think of Pull Requests as a combination mechanism designed to support the branching pattern of functionality and pre-integration reviews. To decide whether and how to use pull requests, I first consider the role of those underlying patterns in the team workflow.

Given this very different environment, it should be clear that the branching strategies of commercial teams need not be the same as those that operate in the open source world. Continuous integration is almost impossible for temporary contributors to open source work, but it is a realistic option for commercial work. Teams should not assume that what works in an open source environment is automatically correct for their different environments.

Review before integration ✣

Every submission to the mainline is peer reviewed before it is accepted.

Code reviews have long been encouraged as a way to improve code quality, improve modularity, readability, and eliminate defects. However, business organizations often find it difficult to adapt to the workflow of software development. However, the open source world has widely adopted the idea that contributions to a project should be vetted before they are accepted into the project’s mainline. This approach has spread widely across development organizations in recent years, especially in Silicon Valley. Workflows like this are particularly well suited to GitHub’s pull-requests mechanism.

This workflow starts when Scarlett has completed a chunk of the work she wants to integrate. Once she builds successfully, she does mainline integration (assuming her team practices this), but before she pushes to the mainline, she sends out her submissions for review. Other members of the team, such as Violet, conduct a code review of the submission. If she had a problem with the submission, she’d post a few comments and go back and forth until Scarlett and Violet were happy. Only when they are done will the submissions be put on the main line.

Pre-integration reviews are gaining popularity in open source and fit well with the organization model of dedicated maintainers and temporary contributors. It enables maintainers to keep a close eye on any contribution. It also works well with the feature branching pattern, because a completed feature marks an explicit node for such code reviews. If you’re not sure a contributor is going to complete a feature, why review part of their work? It is best to wait until the feature is complete before reviewing it. The practice is also spreading among the larger Internet companies, with Google and Facebook both building special tools to help with such work.

It is important to develop the discipline to conduct timely pre-integration reviews. If a developer completes some work and then goes off to do something else for a few days, that work is no longer a priority when it comes time to review feedback. This can be frustrating for a completed feature, and even worse for a partially completed feature, as it may be difficult to make further progress until the review is finalized. In principle, it is possible, and in practice possible, to conduct pre-integration reviews alongside continuous integration — an approach adopted by Google. But while it is possible, it is difficult and relatively rare. The pre-integration review and function branching patterns are a more common combination.

When to Use it

Although pre-integration reviews have become a popular practice over the past decade, there are drawbacks and alternatives. Even when well done, pre-integration reviews introduce some delays in the integration process and encourage lower integration frequency. Pair programming provides a continuous code review process with a faster feedback cycle than waiting for a code review. (Like continuous integration and refactoring, it was one of extreme Programming’s original practices.)

Camille Fournier: Questioning the value of mandatory code reviews is definitely one of the most popular underground beliefs held by senior engineers I know. Ian Nowland: I think mandatory censorship, and practices like requiring annotation of the purpose of each function, are a manifestation of the open source envy and/or cargo worship of enterprise engineers. Camille Fournier: Mixing open source software with the needs of private software development teams is like the original sin of the current software development ritual.

Many teams that use pre-integration reviews don’t do it fast enough. The valuable feedback they can provide comes too late to be useful. In this case, there’s an awkward choice: do a lot of rework, or accept something that might work, but secretly breaks the quality of the code base.

Code reviews are not limited to code before it enters the mainline. Many technical leaders find it useful to review code after a commit, to follow up with developers when they find something noteworthy. This is when refactoring culture is valuable. When done well, this builds a community where everyone on the team regularly reviews the code base and fixes the problems they see.

The trade-offs around the pre-integration review are largely dependent on the social structure of the team. As I’ve already mentioned, open source projects typically have a structure composed of several trusted maintainers and many untrusted contributors. Business teams are usually full-time, but may have a similar structure. The project leader (just like a maintainer) trusts a small (possibly single) group of maintainers and is alert to the code contributed by the rest of the team. Team members may be assigned to multiple projects at the same time, making them more like open source contributors. If such a social structure exists, then the pre-integration review and functional branching patterns make a lot of sense. However, a team with more credibility will often find other mechanisms that don’t add friction to the integration process while maintaining high code quality.

So while pre-integration review can be a valuable practice, it is by no means a necessary path to a healthy code base, especially if you want to develop a balanced team that is not overly dependent on the original leader,

The integration of friction

One of the problems with pre-integration reviews is that they often make integration more cumbersome. This is an example of integration friction (integration friction is the activity that makes integration take time or effort to do). The more integration friction there is, the more developers tend to integrate less frequently. Imagine some (dysfunctional) organization that insists that all submissions to the mainline be filled out with a form that takes half an hour. Such a system would prevent people from integrating frequently. Regardless of your attitude to the feature branching pattern and continuous integration, it’s valuable to examine anything that adds to this friction. Any such friction should be removed unless it clearly adds value.

Pull requests increase overhead to address low trust situations, such as allowing people you don’t know to contribute to your project. Forcing pull requests on developers on your own team is like letting your family through airport security before entering your home. – Kief Morris

Manual operations are a common source of friction, especially when it comes to coordinating with different organizations. This friction can often be reduced by using automated processes, training developers to eliminate the need for such operations, and pushing steps to later steps in the deployment pipeline or QA in a production environment. You can find more ideas to eliminate this friction in continuous integration and continuous delivery of materials. This friction also occurs on the road to the production environment, with the same difficulties and solutions.

One of the reasons people are reluctant to consider continuous integration is that they may have only worked in environments with high integration friction. If each integration takes an hour, it’s obviously absurd to do it several times a day. Joining a team where integration is a killjoy, or where it can be done in minutes, feels like a whole different world. I suspect that most of the debate about the benefits of the functional branching pattern and continuous integration is vague because people don’t experience both worlds at the same time and therefore don’t fully understand either view.

Cultural factors influence integration friction — especially trust between team members. If I’m the leader of a team and I don’t trust my colleagues to do a good job, then I’ll probably want to prevent commits that break the code base. Naturally, this is also one of the drivers of the pre-integration review. However, if I were on a team where I trusted my colleagues’ judgment, I might be more comfortable with post-submission reviews, or cutting reviews altogether and relying on common refactoring to clean up any problems. In this environment, my gain was to eliminate the friction associated with pre-submission reviews, thus encouraging higher frequencies of integration. Often, the trust of the team is the most important factor in the functional branching pattern versus continuous integration debate.

Importance of modularity

Most people who care about software architecture stress the importance of modularity to a good system. If I’m faced with making a small change to a poorly modularized system, I have to know almost everything about it, because even a small change can ripple through many parts of the code base. However, with good modularity, I only need to understand the code of one or two modules, and the interfaces of a few others, and can ignore the rest. This ability to reduce the difficulty of my understanding is why it is worth investing so much effort in modularity as the system evolves.

Modularity also affects integration. If a system has good modules, then most of the time, Scarlett and Violet work in separate parts of the code base, and their changes don’t conflict. Good modularity also enhanced technologies such as Keystone Interface and Branch By Abstraction to avoid the need for isolation provided By branches. Often, teams are forced to use source branching behavior because the lack of modularity leaves them with no other option.

The function branching pattern is the poor man’s modular architecture, and instead of building systems that can easily turn functionality on and off at runtime/deployment time, they coupling themselves to source control, providing this mechanism through manual merging. – Dan Bodart

This support goes both ways. Despite many attempts, it is still very difficult to establish a good modular architecture before we start programming. To achieve modularity, we need to keep looking at the system as it grows and steer it in a more modular direction. Refactoring is key to achieving this goal, and refactoring requires high frequency integration. Therefore, modularity and rapid integration in a healthy code base support each other.

All of which is to say that modularity, while hard to achieve, is well worth the effort. This effort includes good development practices, learning design patterns, and learning from experience in the code base. Messy mergers should not be brushed aside because of an understandable desire to forget them, but rather to ask why they are messy. These answers are often important clues to how modularity can be improved to improve the health of the code base and hence the productivity of the team.

Personal thoughts on integration patterns

My purpose as a writer is not to persuade you to follow a particular path, but to tell you the factors you should consider when deciding which one to take. Nevertheless, I will mention here which of the models I mentioned earlier I prefer.

In general, I prefer to work on a team that uses continuous integration. I recognize that the environment is the key, and in many cases continuous integration is not the best option — but my response is to do the work to change that environment. I have this preference because I want an environment where it is easy for everyone to continually refactor the code base, make it more modular, and keep it healthy — all in order to enable us to respond quickly to changing business needs.

I’m more of a writer than a developer these days, but I still choose to work at Thoughtworks, where there are plenty of people who are in favor of this way of working. This is because I believe this extreme programming style is one of the most effective ways we develop software, and I want to focus on teams further developing this approach to improve the efficiency of our industry.

Path from mainline to production release

The mainline is an active branch, with regular releases of new and modified code. It’s important to keep it healthy so that when people start a new job, they start from a stable base. If it’s healthy enough, you can also release code directly from the mainline into production.

This philosophy of keeping the mainline always releasable is the core tenet of continuous delivery. To do this, there must be determination and skill to maintain the mainline as a healthy branch, often with Deployment Pipelines to support the intensive testing required.

Teams working in this way can usually keep track of their releases by using labels on each release version. But teams that do not use continuous delivery need another approach.

Release Branch ✣

A branch that accepts only these commits, that is, they are accepted in order to create a stable version of the product that is ready for release.

A typical release branch replicates from the current mainline, but does not allow any new functionality to be added to it. The main development team will continue to add functionality to the mainline, which will be put into future releases. The developers working on the release are focused solely on eliminating any defects that prevent the release from being production-ready. Any fixes to these defects are created on the release branch and merged into the mainline. Once there are no more defects to deal with, the branch is ready for production release.

While the scope of the fixes on the release branch is (hopefully) smaller than the new feature code, it becomes increasingly difficult to incorporate them into the mainline over time. Branches inevitably differ, so as more and more commit changes to the mainline, it becomes more and more difficult to merge the release branches into the mainline.

One of the problems with applying commits to the release branch in this way is that it’s too easy to forget to copy them to the mainline, especially since differences will become more difficult to make. The resulting regression is very awkward. Therefore, some people support creating commits on the mainline and cherry-pick them on the release branch once they work on the mainline.

Cherry-pick refers to copying a commit from one branch to another, but the two branches are not merged. That is, only one commit is copied, not all commits since the branching point. In this example, if I merge F1 into the release branch, then this will include M4 and M5. But Cherry-pick will only take F1. Cherry-pick may not be fully applicable to the release branch, as it may depend on the M4 and M5 modifications.

Many teams find it more difficult to write release fixes on the mainline, and it is frustrating to have to do the same on the release branch before a one-way fix on the mainline is released. This is especially true when there is schedule pressure on the release.

Teams with one release at a time in production need only a single release branch, but some products have many releases in production use. Software that runs on a customer’s suite will be upgraded only if the customer wants to be upgraded. Many customers are reluctant to upgrade unless there is a new feature that appeals to them because they have been hurt by failed upgrades. However, such customers still want bug fixes, especially when it comes to security. In this case, the development team keeps the release branch open for each version still in use and fixes it as needed.

Fixing older versions becomes more difficult as development goes on, but it is often a cost of doing business. This can only be mitigated by encouraging customers to upgrade to the latest version frequently. Maintaining product stability is critical to this, and customers who have been injured will have little incentive to upgrade unnecessarily.

(Other terms I’ve heard for release branches include. Release preparation branch, Stabilization branch, candidate branch, and Hardening branch. But the “release branch” seems to be the most common.)

When to Use it

The release branch is a valuable tool when a team is unable to keep its mainline healthy. It allows a portion of the team to focus on the necessary bug fixes to be ready for production. The tester can pull the most stable recent candidate from the top of this branch. Everyone can see what has been done to stabilize the product.

Despite the value of the release branch, most good teams don’t use this model for single-production products because they don’t need to. If the mainline remains healthy enough, any commit to the mainline can be published directly. In this case, the release artifact should be marked with the publicly visible version and build number.

You may have noticed that I used the clumsy adjective “single production” in the previous paragraph. This is because this pattern becomes critical when teams need to manage multiple releases in production.

The release branch can also come in handy when there is significant friction in the release process — such as having a release board that must approve all production releases. As Chris Oldwood says, “When the gears of the enterprise are turning slowly, the release branch acts more like a quarantine in these situations.” In general, this friction should be eliminated from the release process as much as possible, just as we need to eliminate integration friction. However, in some cases, such as mobile app stores, this may not be possible. In many cases a single tag is sufficient, and you only need to open a branch if there are some necessary changes to the source code.

The release branch can also be an environment branch, and the issues that need to be addressed fall under that pattern. There is also a variant of the long-term release branch, which I’ll talk about shortly.

Mature branch ✣

A branch whose header marks the latest version of the codebase maturity.

Teams often want to know what the latest version of the source code is, a fact that can be complicated in a codebase of varying maturity. A QA engineer might want to see the latest pre-release (staging) version of a product, and someone who is debugging a production glitch would like to see the latest production release.

Mature branches provide a way to do this tracing. The idea is that once a version of the code base reaches a certain level of readiness, it will be copied to a particular branch.

Consider a mature branch for production. When we are ready for a production release, we open a release branch to stabilize the product. Once it is ready, we copy it to a long-running production branch. I consider this a copy rather than a merge because we want the production code to be exactly the same as the code we tested on the upstream branch.

One of the beauties of a mature branch is that it clearly shows the version of each code that reaches that stage in the release workflow. So in the example above, we only want a commit that combines M1-3 and F1-2 on the production branch. Doing so requires a few supply chain management gimmicks, but in any case, it loses the connection to fine-grained submissions on the main line. These submissions should be recorded in the submission information to help people track them later.

Mature branches are usually named after the appropriate phase in the development flow. Hence the terms “production branch”, “pre-release branch”, and “QA branch”. Occasionally I hear someone refer to a production maturity branch as a “release branch.”

When to Use it

Source control systems support collaboration and tracking the history of the code base. Using mature branches allows people to capture small but important pieces of information through historical versions that appear at specific stages in the release workflow.

I can find the latest version, such as the production code currently running, by looking at the header of the relevant branch. If a bug occurs that I’m sure didn’t exist before, I can see what previous versions of the branch looked like, and then see the changes to the specific codebase in production.

Automation can be linked to changes in a particular branch — for example, an automated process can deploy a release into production as long as there are commits in the production branch.

An alternative to mature branching is to apply a tagging scheme. Once a release is ready for QA, it can be marked as QA — usually by including a build number. Therefore, version 762 can be marked “QA-762” when ready for QA, and “ProD-762” when ready for production. We can then retrieve the history by searching the code base for tags that match our tag scheme. Automated hooks can also be allocated based on tags.

Thus, a full-fledged branch can add some convenience to a workflow, and many organizations find that tags work very well. I see this as one of those models with no strong benefits or costs. Often, however, the need to use a source code management system for such tracking is a sign of the poor tooling of a team’s deployment pipeline.

Variant: Long release branch

I can think of this as a variant of the release branch pattern, combined with the mature branch of release candidate. When we want to publish, we copy the mainline to this release branch. As with every release branch, commits are made on the release branch only to improve stability. These fixes will also be incorporated into the mainline. We tag it when we release it, and when we want to do another release, we can copy the main line back in.

Commits can be copied or merged, just as they are in the more typical mature branches. If it is merged, we must be careful that the header of the publish branch matches the header of the main line exactly. One way to do this is to revert all the fixes applied to the mainthread prior to the merge. Some teams also squash commits after a merge to ensure that each commit represents a complete release candidate. (Those who find this troublesome have good reason to choose to cut a new branch for each release instead.)

This approach is only suitable for products that have a single release at a time.

One reason the team likes this approach is that it ensures that the head of the release branch always points to the next release candidate, rather than having to dig up the head of the latest release branch. However, at least in Git, you can achieve the same effect by setting the name of a “publish” branch. When the team cuts out a new publish branch, the name of the branch moves with a hard reset command, leaving a label on the old publish branch.

Environment Branch ✣

Configure a product to run in the new environment through source submission.

Software often needs to run in different environments, such as a developer’s workstation, a production server, and perhaps a variety of test and pre-release environments. Often running in these different environments requires some configuration changes, such as the URL used to access the database, the location of the messaging system, and the URLS of key resources.

An environment branch is a branch that contains the commits that are applied to the source code to reconfigure the product to run in different environments. We may have version 2.4 running on the mainline and now want to run it on our pre-release server. To do this, we cut a new branch from version 2.4, made the appropriate environment changes, rebuilt the product, and deployed it to the pre-release environment.

1. A branch for pre-release
2. Configuration changes for pre-release environments

These changes are usually made by hand, although if the person in charge is familiar with Git, they can cherry pick the changes from earlier branches.

The environmental branching pattern is often combined with mature branching. A long-term mature branch of QA may include configuration adjustments to the QA environment. Once merged into the environment branch, these configuration changes are removed. Similarly, a long-term release branch may include these configuration changes.

When to Use it

Environmental branching is an attractive approach. It allows us to adapt the application in any way to prepare it for a new environment. We can save these changes in a diff so that it can be cherry-picked into a future version of the product. However, it’s a classic example of an anti-pattern — something that looks attractive when you start out, but quickly leads to a world of pain, dragons and coronavirus.

The danger looms large if the behavior of the application changes as we move it from one environment to another. If we can’t debug a version running in production on a developer’s workstation, it becomes more difficult to fix the problem. We may introduce bugs that only occur in certain environments, the most dangerous being production. Because of this danger, we want to make sure that as much as possible the code that runs in production is the same as the code that runs elsewhere.

The problem with environmental branches is the super flexibility that makes them so appealing. Since we can modify any aspect of the code in those diff’s, we can easily introduce configuration patches that lead to different behavior and consequent bugs.

As a result, many organizations wisely adhere to an iron rule: once an executable is compiled, the same program should be run in all environments. If different configurations are required, they must be isolated by some mechanism, such as displayed configuration files or environment variables. This way, they can be minimized to simple constant Settings that do not change during execution, leaving less room for bugs to grow.

The simple dividing line between an executable and configuration can easily become blurred in software that directly executes its source code (for example, JavaScript, Python, Ruby), but the same principle applies. Keep any environment changes to a minimum, and don’t use source branching behavior on them. As a general rule of thumb, any version of the product you can check out should run in any environment, so anything that changes purely because of a different deployment environment should not be in source control. There is a debate over whether to store combinations of default parameters in source control, but each version of the application should be able to switch between these different configurations as needed, based on dynamic factors such as environment variables.

Environment branching is an example of a modular architecture that treats the behavior of source branching as poor. If an application needs to run in different environments, then the ability to switch between these environments needs to be a first consideration in its design. Environmental branching can be useful as a cutting mechanism for applications that lack such a design, but should then be removed with a sustainable alternative as a priority.

Hotfix branch ✣

A branch that captures work to fix urgent production defects.

If a serious bug occurs in the production environment, it needs to be fixed as soon as possible. Fixing this bug will take precedence over any other work the team is working on, and no other work should slow down the completion of this hot fix.

Hotfix work needs to be done in source control so that teams can properly document and collaborate on it. They can do this by opening a branch on the latest release and applying any changes for hotfixes to that branch.

Once the fix is applied to the production environment, everyone has a chance to get a good night’s sleep, and then the hotfix can be applied to the mainline to ensure that there will be no regression in the next release. If there is a release branch open for the next release, then hot fixes need to be done on that branch as well. If the interval between releases is too long, the hotfix is likely to build on already changed code, so it will be harder to fit in. Good tests that expose bugs in such cases can be very helpful.

If a team uses the release branch, the hotfix work can be done on the release branch, and a new release is done. In essence, this turns the old release branch into a hotfix branch.

As with the release branch, you can do hotfixes on the mainline and cherry pick them on the release branch. But this is not common, because hot fixes are usually done under tremendous time pressure.

If a team implements continuous delivery, it can release hot fixes directly to the mainline. They may still use a hotfix branch, but they will start with the latest commit instead of the last release commit.

I labeled the new version 2.2.1 because if a team worked this way, the M4 and M5 would probably not expose new features. If they are exposed, then the hotfix could be folded into version 2.3. This, of course, illustrates that in continuous delivery, hotfixes need not sidestep the normal release process. If a team has a responsive release process, hot fixes can be handled as normal — an important benefit of the continuous delivery idea.

A special treatment for continuous delivery teams is to not allow anything to be committed to the mainline until the hotfix is complete. This fits with the slogan “no one has a bigger job than fixing the mainline” — in fact, the same is true of any defect found in the mainline, even those that have not yet made it into production. (So I guess this isn’t really a special treatment.)

When to Use it

Hotfixes are usually done during times of considerable stress, and a team is most likely to make mistakes when it is most stressed. In this case, it is more valuable to use source control than usual, and it seems more reasonable to commit more frequently than usual. Having this work on one branch gives you an idea of what is being done to address the problem. The only exception is that simple fixes can be applied directly to the mainline.

The more interesting question here is to determine what are hot bugs that need fixing and what are things that can be left to the normal development workflow. The more often a team releases, the more it can leave production bug fixes to the regular pace of development. In most cases, this judgment depends on the business impact of the bug and how it fits in with the team’s release frequency.

Release the train ✣

Release at a fixed time interval, just like trains depart on a regular schedule. When the developers have completed their functionality, choose a train to ride.

Teams using release trains set a regular release pace, such as every two weeks or every six months. The date is based on the timing of each release that the team cuts out for release, like a train schedule. People decide which train their feature gets on, and by that train they establish their work, placing their submissions on the appropriate branch when the train is loading. Once the train leaves, the branch is the release branch and only accepts fixes.

A team using the monthly train will launch in February with a branch for March. They will add new features as the month progresses. On a set date, perhaps the third Wednesday of the month, the train would depart — putting the branch’s functionality on ice. They opened a new branch for April Trains and added new features to it. Meanwhile, some developers will stabilize March Train and release it into production when it’s ready. Any fixes applied to the March train will be cherry-picked to the April train.

Release trains are often used in conjunction with the function branch mode. When Scarlett senses when her features are ready, she decides which train to take. If she thinks she can do it in the March release, she will integrate on the March train, but if not, she will wait for the next one and integrate there.

Some teams use a soft freeze a few days before the train leaves. Once the release train is in soft freeze, developers should not push work onto that train unless they are confident that their functionality is stable and ready for release. Any buggy features added after soft freeze will be restored (pushing off the train) instead of being fixed on the train.

These days, when people hear “release train,” they often hear it from SAFeAgile Release trainThe concept of. SAFe’s Agile release train is a team organizational structure that refers to a large team within a team that shares a common release train schedule. Although it uses the mode of publishing trains, it is not the same as what I have described here.

When to Use it

One of the core concepts of the release train model is the regularity of the release process. If you know in advance when the release train will leave, you can plan what features to complete to get on that train. If you don’t think you can finish your function before the March train leaves, then you know you’ll catch the next one.

The release train is especially useful when there is obvious friction in the release process. An external test team takes several weeks to validate a release, or a release committee needs to agree on a new version of the product before it is released. If this is the case, it is usually wise to try to eliminate release friction and allow more frequent releases. Of course, in some cases this is almost impossible, such as the validation process used by app stores on mobile devices. Tweaking the release train to match such release friction may make the best of things.

Releasing a train mechanic helps to focus attention on what features should appear when, which in turn helps to predict when features will be completed.

One obvious drawback of this approach is that functions completed in the early days of the train will sit on it and read while waiting for departure. If these features are important, it means that the product will miss an important feature for weeks or months.

The release train can be a valuable stage in improving the team’s release process. If a team is having trouble delivering steady releases, jumping to continuous delivery can be a step too far. Picking the right release cycle, a difficult but plausible one, can be a good start. As teams acquire skills, they can increase the frequency of trains, eventually abandoning trains for continuous delivery as their capabilities grow.

Variant: Load trains of the future

The basic example of a functional train is when a new train arrives on the platform to pick up the function at the same time as the previous train leaves. But another way is for multiple trains to receive the function simultaneously. If Scarlett thinks her feature won’t be finished during March Train, she can push most of her finished feature to April Train and push further submissions to complete it before the train leaves.

Every once in a while, we pull from the March train to the April train. Some teams like to do this when the March train leaves, so they only have to merge once, but those of us who know that small mergers can be exponentially easier want to pull every March submission as soon as possible.

Loading the Future train allows developers who are working on the April feature to collaborate without affecting the work on the March train. The downside is that when people in April make changes that conflict with work done in March, march people don’t get feedback, making future merges more complicated.

Compare this to a pattern where regular releases fall into the mainline

One of the main benefits of release trains is the regular rhythm of release to production. But setting up multiple branches for new development adds complexity. If our goal is to release on a regular basis, we can do that with the mainline as well. Decide what the release schedule is, and then cut a release branch from the tip of the mainline based on that schedule.

If there is a publish-ready mainline, there is no need for a release branch. For regular releases like this, developers can still choose to shelve near-complete features for the next release by not pushing them to the mainline if they are before the scheduled release date. For continuous integration, if people want a feature to wait until the next regular release, then they can always do this by delaying placing Keystone or having a feature flag shut down.

Publish the ready mainline ✣

Keep the mainline healthy enough that the head of the mainline can always be put directly into the production environment.

When I started this section, I said that if you make the mainline a health branch and make the health check high enough, then you can publish directly from the mainline whenever you want, and label the version.

I’ve spent a lot of time describing alternatives to this simple mechanic, so I thought it was time to emphasize this mechanic, because it can be a great option if a team can practice it.

Every commit applied to the mainline is publishable, but that doesn’t mean it should be. This is the subtle difference between continuous delivery and continuous deployment. Teams using continuous deployment release each change as the mainline receives it, but in continuous delivery, although every change is release-able, the decision to release is business dependent. (Thus, continuous deployment is a subset of continuous delivery.) We can think of continuous delivery as giving us the choice to publish at any time, and our decision to implement that choice depends on the broader issue.

When to Use it

In addition to continuous integration as part of continuous delivery, the release ready mainline is a common feature of effective teams. Given this, and my well-known passion for continuous delivery, you might think I’d say that the release-ready mainline is always better than the alternatives I’ve described in this section.

However, patterns are situational. A pattern that works well in one situation can be a trap in another. The effectiveness of a release-ready mainline is determined by the team’s integration frequency. If a team uses the feature branching pattern and typically only integrates a new feature once a month, the team is likely in a bad position, and sticking with a release-ready mainline may actually hinder their progress. The bad part is that they can’t respond to changing product demands because the cycle time from idea to production is too long. They may also have complex merging and validation because each feature is large and can cause many conflicts. These things may show up at integration time, or are a constant drain on developers as they pull the mainline into their functional branches. This drag discourages refactoring, which in turn reduces modularity, exacerbating the problem.

The key to getting out of this trap is to increase the frequency of integration, but in many cases it can be difficult to do so while maintaining the release ready mainline. In such cases, it is often better to abandon the release-ready mainline, encourage more frequent integration, and use the release branch to stabilize the mainline for production. Of course, over time, we hope to eliminate the need for a release branch by improving the deployment pipeline.

In the context of high frequency integration, a release-ready mainline has the obvious advantage of simplicity. Don’t bother with the complexity of the various branches I’ve described. Even hotfixes can be applied to the mainline and then applied to production so that they are no longer special enough to have a name.

In addition, keeping the mainline ready for release encourages a valuable discipline. It keeps production preparation at the top of developers’ minds, ensuring that problems don’t creep into the system, either as bugs or as process issues that slow down the production cycle. The full discipline of continuous delivery — developers integrating mainlines multiple times a day without breaking them — seems daunting to many. However, once implemented and made a habit, the team found that it significantly reduced stress and was relatively easy to maintain. That’s why it’s a key factor in the Delivering Zone of the Agile Fluency® Model.

Other branching patterns

The purpose of this article is to discuss patterns that revolve around team integration and the path to production. But THERE are other models THAT I want to mention.

Experimental branch ✣

A collection of experimental work from a code base that is not expected to be directly incorporated into the product.

The experimental branch is where developers try out ideas but don’t want their changes to simply be integrated back into the main line. I may have found a new library and thought it would be a good replacement for one we were using. To help me decide whether to replace, I open a branch and try to use it to write or rewrite the relevant part of the system. The purpose of the job is not to contribute code to the code base, but to understand the applicability of a new tool in my particular context. I can do it alone or I can do it with some colleagues up there.

Likewise, I have a new feature to implement and can look at several ways to approach it. I spent a few days studying each method to help me decide which one to adopt.

The key point here is that we would expect the code on the experimental branch to be discarded and not merged into the main line. This is not absolute — if I happen to like the result and the code can be easily integrated, I won’t ignore the opportunity — but I don’t expect that to be the case. I might have followed some of my usual habits less, less testing, some random code duplication rather than trying to refactor it cleanly. Hopefully, if I like the experiment, I’ll apply the idea to production code from scratch, using the experiment branch as a reminder and guidance, but not using any of these commits.

Once I’ve finished working on an experimental branch, I usually add a tag in Git and then remove the branch. The tag keeps the codeline in case I revisit it later — I use a convention such as starting the tag name with “exp” to be clear about its nature.

When to Use it

The experimental branch is useful whenever I want to try something, but I’m not sure I’ll end up using it. That way, I can do whatever I like, no matter how wacky, and I can easily put it aside with the confidence.

Sometimes I thought I was doing a regular job, but eventually I realized that what I was doing was actually an experiment. If this happens, I can open a new experimental branch and reset my main work branch to the final stable commit.

Future branch ✣

A single branch that exists for changes that are too aggressive to be handled by any other means.

This is a rare pattern, but it appears occasionally when people use continuous integration. Sometimes a team needs to make a change that is very intrusive to the code base, and the usual techniques for integrating work-in-progress are not appropriate. In this case, the team does something that looks a lot like the functional branching pattern, where they cut out a future branch and pull only from the mainline until the end to do mainline integration.

The biggest difference between a future branch and a functional branch is that there is only one future branch. As a result, people working on the future branch will never stray too far from the main line, and there will be no other branching branches to deal with.

There may be some developers working on the future branch, in which case they are doing continuous integration on the future branch. When doing integration, they first pull the mainline into the future branch and then integrate their changes. This will slow down the integration process, but this is the price of using future branches.

When to Use it

I should stress that this is a rare pattern. I suspect that most teams that implement continuous integration never need to use it. I’ve seen it used to make particularly aggressive changes to the architecture in a system. Generally this is a last resort and only used when we can’t figure out how to do it using something like Branch By Abstraction.

Future branches should still be kept as short as possible because they create a partition within the team, and like partitions in any distributed system, we need to keep them to an absolute minimum.

Collaboration branch ✣

A branch for developers to share their work with the rest of the team without formal integration.

When a team uses the mainline, most collaboration takes place through the mainline. Only when mainline integration takes place does the rest of the team see what a particular developer is doing.

Sometimes a developer wants to share their work before integration. Opening a branch for collaboration allows them to do this on an AD hoc basis. This branch can be pushed to the team’s central repository, and collaborators can either pull and push directly from their personal repository, or set up a short-term repository to handle collaboration issues.

The collaboration branch is usually temporary and shuts down once the work is integrated into the mainline.

When to Use it

Collaboration branches become more useful as the frequency of integration decreases. Informal collaboration is often required for long-term functional branches if team members need to coordinate changes into areas of code that are important to some people. A team that uses continuous integration will probably never need to open a collaboration branch because their work is invisible to each other for such a short period of time. The main exception to this is the experimental branch, which by definition will never be integrated. If several people are doing an experiment together, they need to make the experiment branch a collaborative branch as well.

Team Integration Branch

Let a sub-team integrate with each other before integrating with the mainline.

Larger projects may have several teams working on a single logical code base. A team integration branch allows team members to integrate with each other without having to use the mainline to integrate with all members of the project.

In effect, teams treat team integration branches as internal mainlines and integrate with them, just as they would with the mainlines of the overall project. In addition to these integrations, the team will put in extra effort to integrate with the project mainline.

When to Use it

The most obvious motivation to use team integration branches are those code bases that are actively developed by many developers, and it makes sense to divide these developers into different teams. But we should be wary of such assumptions, as I have encountered many teams that seem too big to work together on a single thread, yet somehow manage to do so. (I have covered this 100-strong team of developers.)

A more important motivation for using team integration branches is the difference in expected integration frequency. If the project as a whole wants the team to use a few weeks long feature branch, but the sub-teams prefer continuous integration, the team can set up a team integration branch, use this branch for continuous integration, and integrate it with the main line once the feature they are working on is complete.

A similar effect occurs if there is a difference between the health criteria used by the overall project for the health branch and the health criteria used by the sub-teams. If the larger project cannot maintain a high enough level of stability for the mainline, then the subteam may choose to operate at a more rigorous level of health. Similarly, if the subteam is struggling to make its commit healthy enough to reach a well-controlled mainline, they may choose to use the team integration branch and use their own release branch to stabilize the code before moving into the mainline. This is not a situation I usually enjoy, but it may be necessary in some particularly difficult situations.

We can also think of team integration branches as a more structured form of collaboration branches, based on formal project organization rather than AD hoc collaboration.

Some branching strategies

Git-Flow

Git-flow has become one of the most common branching strategies I’ve come across. It was written by Vincent Driessen in 2010, around the time Git became popular. In the days before Git, branching patterns were generally considered an advanced topic. Git makes branching more attractive, partly because of tooling improvements (such as better file movement), but also because a cloned repository is essentially a branch, requiring similar thinking about merging when pushed back to the central repository.

Git-flow uses the main line (call it “Develop”) in a single “Origin” repository. It uses the feature branching pattern to coordinate multiple developers. Developers are encouraged to use their personal repository as a collaborative branch to collaborate with other developers doing similar work.

Traditionally, the core branch of Git is referred to as the “master,” and in Git-flow, the master is used as a full-fledged branch of production. Git-flow uses a release branch to pass work from “Develop” to “master.” Hotfixes are organized through hotfix branches.

Git-flow doesn’t discuss the length of the feature branches, so you don’t expect something about integration frequency. It also doesn’t say whether the main line should be a healthy branch, and if so, to what extent. The existence of a release branch means that it does not have a publish-ready mainline.

As Driessen pointed out in an appendix this year, Git-Flow is designed for projects that release multiple versions in production, such as software installed at customer sites. Of course, having multiple active releases is one of the main incentives to use a release branch. However, many users opt for Git-Flow in the context of webApp in a single production environment — such a branching structure could easily become more complex than needed.

Despite git-flow’s popularity, it feels like a lot of people say they use it, but it’s often found that people who say they use Git-flow are actually doing something completely different. Often, what they actually do is closer to the GitHub Flow.

GitHub Flow

While git-Flow is really popular, its branching structure is unnecessarily complex for Web applications and has given rise to many alternatives. With GitHub’s popularity, it’s no surprise that the branching strategy used by its developers, known as GitHub Flow, has become well-known. Scott Chacon put it best.

With a name like GitHub Flow, it’s no surprise that it’s intentionally based on and against Git-flow. The essential difference between the two is the variety of products available, which means different usage scenarios and therefore different patterns. Git-flow assumes that there are multiple versions of a product in production. GitHub Flow, on the other hand, assumes that there is a single version of the product in production and that it is frequently integrated into a release-ready mainline. In this case, the release branch is no longer needed. Production problems are fixed in the same way that regular functionality is developed, so there is no need for a hotfix branch, which usually means a deviation from the normal process. Removing these branches greatly simplifies the branching structure, leaving the main and functional branches.

GitHub Flow calls its main line “Master.” Developers work with the functional branching pattern. They periodically push feature branches to the central repository to provide visibility, but they are not integrated into the mainline until the feature branches are complete. Chacon points out that a feature branch can be a single line of code or it can run for weeks. The process works the same way in both cases. As GitHub, the pull-Request mechanism is part of the mainline integration, and pre-integration reviews are used.

Git-flow and GitHub Flow are often confused, so as ever, dig behind the names to understand what’s really going on. The general theme of both is the use of mainlines and functional branches.

Trunk-Based Development

As I’ve written before, most of the time I hear “trunk-driven development” I think of it as a synonym for continuous integration. But it is also reasonable to think of trunk-driven Development as an alternative branching strategy to Git-flow and GitHub Flow. Paul Hammant has written an in-depth website explaining this approach. Paul is a long time colleague of mine at Thoughtworks and has a solid track record of using his proven edge to break through rigid branch structures at clients.

Trunk-based Development focuses on doing all the work on the mainline (called “thunk,” a common synonym for “mainline”), thus avoiding any form of long-term branching. While smaller teams use mainline integration to commit directly to mainline, larger teams may use the short-term feature branching pattern, where “short” means no more than a few days — which in practice can be similar to continuous integration. Teams can use a release branch (called a “release branch”) or a release-ready mainline (” Release from the trunk “).

Final thoughts and recommendations

From the earliest programs, people found that if they wanted a program that was a little different from an existing one, they could easily do it by taking a copy of the source code and tweaking it to look the way they wanted. With all the source code, I have the ability to make any changes I want. But as this behavior progressed, it became harder for my copy to accept the new features and bug fixes in the original source. Eventually, it may become impossible, as many enterprises found in their early COBOL programs and suffer in today’s widespread custom ERP packages. Even if we don’t use that name, any time we copy source code and make changes to it, we’re doing source branching, even though version control is not involved.

As I said at the beginning of this long article: Creating a branch is easy, merging is harder. Branching is a powerful technique, but it reminds me of goto statements, global variables, and concurrent locking. Powerful, easy to use, but easy to overuse, they often become a trap for the unwary and inexperienced. Source control systems can help control branching behavior by carefully tracking changes, but ultimately they can only serve as a witness to the problem.

I’m not one to say that branching is evil. For everyday issues such as multiple developers contributing to a single code base, the judicious use of branches is essential. But we should always be wary of it, and remember Paracelsus’ observation that the difference between a good medicine and a poison is dose.

So my first piece of advice for branches is: whenever you’re considering using a branch, think about how you’re going to merge it. Any time you use any technology, you’re balancing alternatives. Without knowing all the costs of a technology, you can’t make an informed tradeoff decision, and in the branching act, when you merge, the piper will claim her fee.

So here’s the next guide: Make sure you understand alternatives to branching, which are usually better. With the law of Bodart in mind, is there a way to solve your problem by increasing modularity? Can you improve your deployment pipeline? Is one label enough? What changes could you make to your process that would make this branch unnecessary? There’s a good chance that branching is actually the best route right now — but it’s a smell that reminds you that there’s a deeper problem that should be solved in the next few months. Getting rid of the need for branches is usually a good thing.

Remember LeRoy’s illustration: Branches differ exponentially as work goes on without integration. So consider how often you integrate branches. Try to double your integration frequency. (There is obviously a limit here, but unless you are in the region of continuous integration, you will not approach it.) There are barriers to more frequent integration, but those barriers are often the very things that need to be given an excessive explosive to improve your development process.

Since merging is the hardest part of branching behavior, pay attention to what makes merging difficult. Sometimes it’s a process issue, sometimes it’s an architectural failure. Whatever it is, don’t give in to Stockholm syndrome. Any merger issue, especially one that leads to a crisis, is a signpost for improving team effectiveness. Remember, mistakes are only valuable if you learn from them.

The pattern I describe here Outlines common branch configurations that my colleagues and I encounter on our travels. By naming them, explaining them, and most importantly, explaining when they’re useful, I hope to help you evaluate when to use them. Remember that, like any pattern, they are rarely universally good or bad — their value to you depends on your circumstances. When you encounter branching strategies (whether well known such as Git-flow or Trunk-based Development, or something invented within a Development organization), I hope that understanding the patterns will help you decide whether they are appropriate for your situation, and which other patterns to incorporate will help.

thanks

Badri Janakiraman, Brad Appleton, Dave Farley, James Shore, Kent Beck, Kevin Yeung, Marcos Brizeno, Paul Hammant, Pete Hodgson and Tim Cochran read the draft of this article and gave me feedback on how to improve it.

Peter Becker reminded me that forks are also a form of branching. I got the name “Mainline” from Steve Berczuk’s software Configuration Management Patterns.

read

There’s a lot of branching material out there, and I can’t really investigate all of it. But I do want to highlight Steve Berczuk’s book, Software Configuration Management Patterns. The work of Steve and his contributor Brad Appleton has had a lasting impact on the way I think about source code management.

Major revision

January 4, 2021: Added sidebar on Pull Requests.

January 2, 2021: Change the name of Reviewed Submission to Pre-Integration Review, which I think is a clearer name.

May 28, 2020: The final chapter is published.

27 May 2020: “Strategies for Some Branches” is published.

May 21, 2020: Collaboration Branches and Team Integration Branches are published.

May 20, 2020: Draft final thinking.

May 19, 2020: The Future Branch is published.

May 18, 2020: Published in Experimental Branch.

May 14, 2020: Release-ready Mainline is released.

May 13, 2020: The chapter on branch strategy is drafted.

May 13, 2020: Release Train is published.

May 12, 2020: The Hot Fix Branch is published.

May 11, 2020: Drafted a Releas-ready Thread.

May 11, 2020: The Environmental Branch is published.

May 7, 2020: “Mature Branches” is published.

May 6, 2020: The Publishing Branch is published.

May 5, 2020: Published Integrated Friction, The Importance of Modularity, and My Personal Thoughts on Integration Patterns.

May 4, 2020: Reviewed Submissions are published.

April 30, 2020: Published continuous Integration vs. Functional Branching Patterns.

April 29, 2020: Continuous Integration is published.

April 28, 2020: Draft: Added a section on modularity.

April 28, 2020: Integrated Frequency is published.

April 27, 2020: Draft: Production Branch to Mature Branch in general.

April 27, 2020: Functional Branching Patterns are published.

April 23, 2020: Mainline Integration is published.

April 22, 2020: Branches of Health is published.

April 21, 2020: “Mainline” is published.

April 20, 2020: The first chapter, Source code Branching Behavior, is published.

5 April 2020: Fifth draft: Dealt with reviews of release modes, wrote release trains, revised source branch behavior.

March 30, 2020: Fourth draft: Addresses most of the review comments on the foundation and integration sections. Make source branching behavior a pattern.

March 12, 2020: Third draft: Rewrite the mode as a special chapter.

March 5, 2020: Second draft: Reorganize the text into integration patterns and paths to production. Added illustrations for the release branch and hotfix and rewrote the text to match them.

February 24, 2020: First draft: Share with reviewers.

January 28, 2020: Start writing.

August 6, 2021. Some of the content may evolve over time.