It’s been a few years since I’ve had any major mishaps with git and I was feeling pretty proud of myself. Yesterday, I had to reset my number of days without incident count to 0. Womp womp.
A bit of background: I use git flow at work. Git flow is a branching strategy that allows us to keep deployable code separate from new feature development. It is also a toolset with command line utilities to enforce the branching strategy. It works really well at protecting production code… when it’s used!
The flow is pretty simple: deployable code is on master, new feature code is on develop. New code is developed on feature branches off develop and merged back into develop. When you’re ready to release new code, you create a release branch off develop and merge it to both master and develop. A hotfix is used to patch production code and is like a release branch, except it stems from master and is merged back to both master and develop.
Yesterday I was under pressure to get out a few hotfixes for customer support, as well as review hotfixes from colleagues. Git flow enforces a rule that restricts you to working on one hotfix branch at a time. Since I’m responsible for code review, merging of the hotfix branch, pushing to GitHub, and deploying to production, it’s imperative that I follow the git flow process to ensure I don’t mess up along the way. However, git flow was not letting me work with multiple hotfix branches, so in a moment of impatience, I decided to create my hotfix branch manually from master. I knew how the process worked, what could go wrong?
I merged the hotfix branch, deployed to production and got on with my day. An hour later, I received a report from customer service that there were exceptions being thrown in our administrative app. The exception had suspiciously familiar sounding method names… a new feature had slipped out to production!
This has happened one or two times in the past, and I recognized the symptoms and immediately combed through our GitHub commit history to figure out how develop got merged to master. I was the only one who had pushed any merges to GitHub yesterday. In my haste I must have created a hotfix branch from develop instead of master, and merged it back to master — an early release!
Tracking down bad merges in GitHub commit history is not an easy task. After half an hour of frustration and clicking through commits, I enlisted the help of my colleague, Arian, to track down the commit and analyze our options.
There are some very good articles out there for dealing with this scenario. It appeared we had two choices:
- Rewrite git history
- Revert the merge
The first option is appealing because you can rewrite the history, making it look like nothing ever happened. You remove the bad commit by running a `git reset GOOD_COMMIT_HASH` and then force-push it to GitHub with a `git push –force origin master`. I tried this first, while alerting the team not to push or pull to GitHub to avoid overwriting my rewritten history with the original, shared history. While this approach it chopped off the bad merge, commits from develop were still in master’s history. Apparently those commits were now officially part of the master branch after the merge.
Number two was not appealing, but seemed to be our only option. Reverting does not remove the offending commit, but instead adds a new commit that “whites out” changes introduced by the reverted commit. The bad merge would remain in history, a tarnish on my record! Ego aside, another reason this is unappealing is that it would cause problems when we attempted to release. The release branch is created from develop and merged to master — the branch that we had previously merged and reverted commits from develop. Basically, that work we “whited out” would still be gone once we released. The only way to fix this would be to execute a revert of the revert prior to releasing (Inception, anyone?).
One upside to this approach is that reverting the merge is non-destructive and allows the team to continue development. This is because it simply adds a new commit for the fix. I reverted the merge and pushed to GitHub. We spot-checked the code base on master and after running automated tests, deployed to production. Customer support reported the problem was fixed!
An hour later, I pulled up develop to do some work and grimaced… something was awry. The develop branch was missing new code we’d worked on since the last release! Again, I went into investigative mode. After the revert was committed, it turned out I had merged a new hotfix from a colleague back to master and develop. Because this hotfix branch included the revert commit and that commit did not exist on develop, it was merged to develop as part of the hotfix essentially erasing our work on develop.
After some cursing and hair pulling, I realized that this could be rectified with a simple revert-of-the-revert commit added to develop. Better yet, this would have been something I’d needed to do to prepare for the release anyway. Two stones killed with one bird! I couldn’t pat myself on the back for this realization too much, however, because it turns out this is the right way to handle an accidental merge of develop into master on a shared repo: revert the merge commit on master and develop (to protect master), and then revert the revert on develop (to protect the release).
Lessons learned:
- When working on a shared repository, always revert a bad merge. When solo, feel free to re-write git history as much as you like.
- Develop and use a process, like git-flow, to protect yourself from mistakes.
- Don’t let the pressure get to you. Follow the process.
- If using git-flow and a revert on master is required, commit the inverse revert to develop to protect the future release merge.
- Seriously, follow the process.
What git mishaps have you encountered? How would you have handled this situation?