Starting now, I’m experimenting with new post formats on my blog. Instead of just technical posts describing code, I’m going to begin posting some more free-form articles. Like this one, where I’m going to share a story with you that has some moral relating back to IT.

It was the start of December 2017 and I was in Toronto to attend MVP Community Connection day, which is an event exclusively for Microsoft MVPs where we get together, socialize and connect with each other and Microsoft employees, get a little soft skills training, and provide feedback on things we’d like to see from Microsoft in the upcoming months. MVPs from across Canada traveled to Toronto to enjoy this always enjoyable event.

Microsoft is kind enough to supply accommodations at a hotel I won’t name, and I had the great pleasure of sharing a room with fellow MVP and all-round good guy, Will Anderson. We had a great time except for one small inconvenience. From the moment we first used it, it was apparent that our toilet had issues. This thing wouldn’t even flush the water that it filled itself up with. I know what you’re thinking, but we treated it with respect.

[caption id=”attachment_669” align=”alignleft” width=”300”] Will and myself at MVP Summit 2016, preparing content for the 10th anniversary of PowerShell celebration.[/caption]

No big deal, we called the front desk and asked them to send someone to try and fix it. One more problem, though. It was after midnight by the time we got back to our room and that meant the normal maintenance people had gone home. So, the person they sent shows up empty handed, tries flushing the toilet again (like we didn’t try that), and said that was all she could do. Neither Will nor myself are plumbers, but we suggested maybe this person might try to use a plunger on the situation. She replied that she did not have the key to the plunger but could send someone at 7 AM with one. That’s right. This person responding to our potential plumbing emergency did not have the key to the plunger. Luckily, we didn’t have any urgent “needs” and nothing was in any risk of impending doom.

What’s the moral here? How does this relate to IT? Well, imagine you find yourself in an emergency situation - you’ve been hit by ransomware, your CFO’s Active Directory credentials have been compromised, a Hyper-V host just went down, or some other metaphorical “toilet” has become “clogged”. You have people there who are willing to help, just like Will and I had this maintenance person in the hotel, but are they able to actually do what they need to do in an emergency?

Say your IT staff need to respond to an emergency. Have they practiced normal emergency response scenarios? Does everyone know their role and responsibilities? Is anyone going to get stuck on managerial approval, missing a privilege, or following out dated instructions? Obviously safety measures and safeguards are important, but there’s a point of diminishing returns (can’t unclog a toilet because you don’t have the plunger key, can’t temporarily disable a compromised CFO’s account because you don’t have permission to disable executives’ accounts, etc.).

In addition to making sure you haven’t hit that point of diminishing returns with your IT safeguards, know that your disaster recovery and emergency reaction manuals are only as valid as the last time you tested them. You might be doing a great job of backing up your NAS every night, but how good are you at restoring it?

The bottom line is: make sure you stay out of your own way when it comes to IT emergencies, and test to make sure your steps to respond are actually valid. It could save you a lot of pain and embarrassment one day!