"A DevOps is You!"
Log in
Don't be a Hero
July 9th, 2022 by will

We all want to jump in and save the day. While nobile, it can cause bigger problems. Software engineering is a discipline, and part of that discipline is evaluating choices. Sometimes it is appropriate to go rambo and get something done, and sometimes the better choice is to follow process or let things slide.

Focusing work

“Hey are you busy?” The words that plague us all, and usually followed by a request of some sort. We all want to be helpful, or at least most of us do. It’s tempting to want to address a request immediately, after all if you see a snake, why not just kill it? However, one of the biggest losses of efficiency is context switching. Sometimes it is appropriate to address the quick question, or restart a service if it is blocking developers, but keep in mind that it is impacting your work. Most requests can and should be funneled through the team’s process for accepting work. My rule of thumb, if it takes more than 5 minutes and I am not deep in thought, I do it, otherwise I open an issue and backlog it. Not surprisingly, many people are okay with that. The trick is to ensure it doesn’t sink to the bottom of the ocean floor of backlogged items. Even if it does, if they never bring it back up, it might not have been that important of a request in the first place, and you have just saved some time for more pressing issues, like meeting those pesky deadlines.

Working late or off hours

Deadlines loom and upper management gets comfortable, the project deadline is compromised. It is natural to want to work extended hours to get the project done. Sometimes this is a necessary evil, maybe there is a huge vulnerability, or a customer is threatening to walk, but deadlines for the sake of deadlines does not constitute an emergency. We all need our downtime, without it we get overwhelmed and burned out. Even if it feels like we can work 12 hours a day, we really can’t. There’s only so much capacity a person has. It is important to rest and take care of one’s self. Switching away from work also helps the brain process through the day. Usually I will end up coming up with solutions to problems well after I stop working on them. Hobbies are a great way to detach too. Just make sure to give work space, and keep work at work and home at home. Production outages excepted, of course.

Fixing production

Red lights fill up the dashboard and customer complaints are rolling in. Of course the quickest path is to just jump in and fix something, but that is not always the best solution. In fact, it usually is not. While it is tempting to run into the burning building, it is good to approach cautiously. Ensure that whatever changes that are happening go through automated and standard processes, if possible. Instead of hot patching code on servers, consider if the code can be released as a hotfix. If it can’t, then a post mortem item would be good for improving the deployment process. Ensure open communication when fixing things, and document them somewhere (such as a Jira issue). The biggest disservice anyone can do is to fix something with no paper trail of why or what was done. One of the biggest things that grinds my gears is seeing the dreaded “It’s fixed, try now.” What was done? Will it happen again? What if it happens again, how will we know what we did last time. The best approach is to setup an issue tracker for production issues that can correlate similar problems. It helps with future troubleshooting and to identify issues that can be permanently fixed. Nobody needs to be logging into a server and clearing up /var/log every week when logrotate can be fixed, but nobody will know to do that if the problem is fixed in the shadows.