Git Branching Guidance for DevOps Teams

My customers that move to git from TFSVC or SVN tend to use GitFlow as their branching strategy. Not because it’s best suited for their needs – but because it’s the most popular strategy. But GitFlow introduces a lot of complexity that may not be necessary for your team. That’s why I wrote a little guidance that should help teams find the right strategy and companies to define a guidance that fits all their teams.

Distributed version control systems like git give individuals a wide flexibility in how they use version control to share and manage code. Your company should find a balance between this flexibility and the need to collaborate and share code in a consistent manner.

Teams should adopt a branching strategy, so that team members collaborate better and spend less time managing version control and more time developing code.

Your company should allow Teams the freedom to find a strategy that fits their needs. At the same time, it should enforce a minimum standard that allows developers to easily switch teams or to provide changes for other teams without the need to learn a new strategy.

The approach I prefer is to define a minimum branching strategy and a guidance, for what problems what approaches can be used. Each team can then pick their strategy – based on the minimum strategy – that bests fits to their way of work.

Simple minimal branching strategy

The basic branch strategy for your company should be as simple as possible and could look like this:

  • Keep a high-quality master branch that is locked by policies.
  • Use topic branches for all new features and bug fixes
  • Merge branches into master using a pull request

This strategy is called the GitHub-Flow and is widely used in the open source world. It is also called Trunk-Based-development (TBD) or Main-Line-Development (MLD).

Use topic branches for your work

Develop new features and fix bugs in topic branches based off your master branch. Feature branches isolate work in progress from the completed work in the master branch. Git branches are inexpensive to create and maintain, so even small fixes and changes should have their own feature branch.

featurebranching

Name your feature branches by convention

Use a consistent naming convention for your feature branches to identify the work done in the branch. You can also include other information in the branch name, such as who created the branch.

Some suggestions for naming your feature branches:

  • features/feature-name
  • features/feature-area/feature-name
  • bugfix/description
  • hotfix/description
  • users/username/description
  • users/username/workitem

To enforce the naming policy, the permission of the repository can be altered.

Block the creation of branches for all contributors:

tf git permission /deny:CreateBranch /group:[Project]\Contributors /collection:$coll /teamproject:$tp /repository:$repro

Grant users the right to create branches under features/ and /users

tf git permission /allow:CreateBranch /group:[Project]\Contributors /collection:$coll /teamproject:$tp /repository:$repro /branch:features

tf git permission /allow:CreateBranch /group:[Project]\Contributors /collection:$coll /teamproject:$tp /repository:$repro /branch:users

For more details see Require branch folders.

I created a small PowerShell script that you can use to quickly set the permissions according to your guidelines. See the readme in the repo on GitHub for more information.

Review and merge code with pull requests

The review that takes place in a pull request is critical for improving code quality. Only merge branches through pull requests that pass your review process. Don’t merge branches to the master branch without a pull request.

Reviews in pull requests take time to complete, so your team should agree on what’s expected from pull request creators and reviewers. Distribute reviewer responsibilities to share ideas across your team and spread out knowledge of your codebase.

Some suggestions for successful pull requests:

  • Two reviewers is an optimal number based on research.
  • If your team already has a code review process, bring pull requests into what you’re already doing.
  • Take care assigning the same reviewer(s) to a large number of pull requests. Pull requests work better when reviewer responsibilities are shared across the team.
  • Provide enough detail in the description to quickly bring reviewers up to speed with your changes.
  • Include a build or linked version of your changes running in a staged environment with your pull request so others can easily test the changes.

Enforce a clean master branch with branch policies

The code in your master branch should pass tests, build cleanly, and always be up to date. Your master branch needs these qualities so that feature branches created by your team start from a known good version of code.

To enforce this, a branch policy must be created for master.

You should enable the following policies:

Require minimal number of reviewers: You should require a minimal number of reviewers. This should at least be two – but to not block work in small teams you can allow that authors can approve their own pull request. This is then like the 4-eye-principle.

Check for linked work items: Enforce the each pull request is linked to a work item.

Build validation: Run the CI build and as many tests as possible.

You can optionally enable following policies:

Check for comment resolution: Comments from reviewers must be marked as resolved.

Enforce merge strategy: Require a stash or fast-forward-merge.

Automatically include reviewers: Add special reviewers for parts of your repo. The reviewers will only be added, if files in this area are modified. You can i.e. add an admin as a reviewer for the folder “deployment” that contains the deployment scripts and infrastructure as code (IaC).

Guidance

If you have defined a minimum strategy for all your teams, you can provide a guidance for the teams to adopt their own strategy.

Mono Repo vs. Multi-Repo

The first decision that has to be taken, is the number of repositories for your application. There are basically two approaches:

  1. Monolithic Repo: One big repository for the entire application
  2. Multiple Repos: The application is split into small parts that each resides in its own repo

The first approach is the classic one. You should consider a good folder structure that is expandable. For example:

  • Root
    • Module1
      • src
      • test
      • Readme.md
    • Module2
      • src
      • test
      • Readme.md
  • Readme.md

The advantage of this approach is, that you always have all dependencies in one place. The disadvantage is, that the size of the repo is bigger and it’s easier to cross boundaries between separated modules.

The second approach is especially for teams that embrace microservices. The problem here is not the folder structure inside the repo – the problem is the organization of a big amount of repositories. You should adopt a good naming strategy for the repos here. For example:

<team>-<application>-<interface/type>-module

team A prefix for all repos that keep them together in your local repo folder.
application The short name of your application
Optional: interface/type Depending on your architecture you may find it useful to group some repos together. For example by type (Packages like nugget or npm) or interface (REST service, message based etc.)
Module The name of the module

The advantage of this approach is, that it enforces a loosely coupled system and that the repos are completely independent. The disadvantage is, that is way harder to manage all the dependencies across many repositories.

Stabilizing releases

If your team does continuous delivery, fail fast and roll forward, and if you deploy multiple times a day to a single service you may don’t need release branches. But if you need to coordinate releases with other projects/products, or you have manual stabilizing phases you need an advanced branching strategy for this. The recommended way is this:

  • Create a release branch from master. Give this branch a clear name (i.e. releases/20). The branch is long lived and will never be merged back!
  • Create bugfix and hotfix branches and merge them back to the release branch using pull requests.
  • Deploy to your environments from the release branch

Port back your bugfixes to master

To port your changes back to master you have to create a new topic branch from master. Use cherry-picking instead of merging to bring over your changes to the topic branch. This gives you exact control over the changes you bring over to master. Merge the topic branch with the normal procedure (pull request).

releasebranching_release

Why not use tags for releases?

Other branching workflows use Git tags to mark a specific commit as a release. Tags are useful for marking points in your history as important, but tags introduce extra steps in your workflow that aren’t necessary if you are using branches for your releases.

Tags are maintained and pushed separately from your commits. Team members can easily miss tagging a commit and then have to go back through the history afterwards to fix the tag. You can also forget the extra step to push the tag, leaving the next developer working from an older version of the code when supporting the release.

The release branch strategy extends the basic feature branch workflow to handle releases. Your team doesn’t have to adopt any new version control process other than the cherry-pick to port changes.

Develop long-running or depended features

The simple branching strategy is best suited for small batched changes. Small changes reduce the risk of breaking something, when they are merged back.

Sometimes this is not possible. There are big, long-running features (like a new UI) that have a big impact, and there are features that are dependent on external system (i.e. a SAP release). In both cases it’s not possible to check the changes into master daily.

There are two different approaches to these challenges: feature flags or an integration branch.

Feature Flags

Feature Flags (aka Feature Toggles, Feature Switches) are a design pattern, that decuples the rollout of functionality from the deployment of code. This is done by changing the runtime behavior of an application. This allows even big changes to be checked into master before activating them. The pattern is basically this:

var featureFlag = GetFeatureFlagValue("my-feature-flag");

if (featureFlag)
{
    // new code
}
else
{
    // old code
}

The feature flag can be stored in a database, in a separate service, in LaunchDarkley or TFS. This sounds stupid – but if you think of it together with dependency injection this is very mighty.

var oracleAdapter = new OracleAdapter();
var sqlAdapter = new SqlAdapter();

var databaseTypeFlag = GetFeatureFlagValue("new-database");

if (databaseTypeFlag)
{
    dependencyContainer.Register(sqlAdapter);
}
else
{
    dependencyContainer.Register(oracleAdapter);
}

With this pattern you are able to check in your feature even if it is not ready to be used and deploy it to production. You can then turn the feature on to test it and when it is ready.

Feature Flags have a lot of advantages. You can do canary releases, perform A|B-Testing, allow Opt-In-Features for your users. You can also use the flags as Kill-Switches to turn off new features in production, if they do not behave as expected.

Integration branches

If you have long running feature branches it is very risky to merge them. Doing this on master would introduce a big risk. That’s why you have to introduce an integration branch (normally called develop). Topic branches are then not created out of master but out of the develop branch.

The develop branch is now your new “mainline”. All topics are created out of develop branch. To get the integrated features back into master, you use release branches (see 4.1). The release branches are tested and stabilized. If they are deployed to production they are merged into master.

GitFlow-Light

Hotfixes from the release branches should also be cherry picked into a new topic branch from develop to keep develop up to date.

I obviously prefer feature flags and TBD over the integration branch approach.

So this is basically it. I would not recommend the full GitFlow for DevOps teams. I’ve never seen agile teams that benefit from it – bit I’ve seen a lot that suffer from the complexity.

Summary

It’s very important to find a branching strategy that fits the way the team works. If branching is not done right, it introduces big burdens and slows the team down. It also can impact the quality of your product.

The best guidance is to start small and add complexity only if you need it. Often are other options available. The fastest cadence is achieved with the following setup:

  • Work in very small batches
  • Integrate all changes as often as possible

This guidance is based on the Git Branching Guidance from Microsoft. They are the customer with the largest git repos in the world (with Windows and Office) and use git now for all solutions. If you want to know, how Microsoft is using git, you can read it in this interesting post.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s