Steps to take to not break code production

Any help answering this question. I could not find an answer.

What steps would you take if you or a teammate (answer for both cases) merged code to production that broke everything?

I found this here but my question is more about something that already happened not how to prevent them.

git revert would probably be the first step. Or if you can see what caused the bug, then you can push a quick fix.

And it really doesn’t matter who in particular broke the prod, because you as a team are responsible for the code.

If it was something like this classic, then, no idea :man_shrugging:

1 Like

Thanks for referring me to the link. Great conv right there

git revert would have been my first answer, but I have worked for places that did not use git. In that case we had to rely on manual backups we uploaded via the fttp. Having learned that lesson a version control system is a much better option.

I would perform the following steps:

  1. Understand the current impact, and rollback the commit if possible (as mentioned above). This isn’t always possible. If the changes have made irreversible changes to the system. For example if the commit created breaking changes to the database, it means you might be better off “patching” things and moving forward.

  2. Identify how this change go into the system via a blameless postmortem This is not about accusing people, it’s about understanding what part of the overall software development process needs to be changed to prevent this from happening again.

  3. Take steps to prevent this from happening again.

Dealing with mistakes is another thing, blaming people for their mistakes is an even worse thing, but doing something to help prevent these mistakes in the future is a key step to stabilize things moving forward.

Mistakes happen, people are human, and over time systems get more complex so the last thing you want is to just “move on and be more careful” because at some point you’ll find your walking through a minefield of technical debt and if your process isn’t continually improved upon, you’ll find project speed will grind to a halt, or even start going backwards!

1 Like