Hire Us

How to crash production from github admin page

When I get into github project page yet another time and switched view from master branch to development I came up with idea that we should change the default branch to development as we work with it more.

So, I navigate to admin tab and changed default branch with confidence that I did something useful.

On the other part of the project people found a typo in one of our front page and decided to hot fix it in production. They change 1 symbol and made the deployment. Checked that production is alive and went home. No body did precise checks as change was so minor.

After the midnight we discover that there are to many errors coming in the log. And the issue was that development branch was deployed to production.

At first sight it was complete magic, but after some investigation we found out the following.

By changing the github default branch, you don’t just change the default branch shown in the web interface. The HEAD ref is set to that branch.
Even that such behavior is non explicitly documented at help.github.com – clone function and stated at this blog post.

When no branch is set on capistrano deploy script – it will checkout HEAD, not master.
From Capistrano recipe for git:

 # When referencing "head", use the branch we want to deploy or, by # default, Git's reference of HEAD (the latest changeset in the default # branch, usually called "master"). def head variable(:branch) || 'HEAD' end 

The only way how the github default branch could be implemented – was changing the HEAD ref.
The structural way how capistrano without an explicit branch set would work – is using a HEAD ref, as master is just a convention (best practice).
When you think about it that way – it all makes sense. One thing led to another. No Magic.

So lessons learned:

  • never underestimate a delivery and do it only with a good QA coverage, even if it’s JUST 1 symbol patch;
  • use bookmarks to the what ever branch on the github page you like, and leave the default branch untouched;
  • when deciding to change it – make sure your deploy script is branch aware;
  • being used to git master branch and default patterns of using git – a good point is to re-read the documentation.