Sometimes when things go wrong at work they still go right.

Two Friday nights ago I was at the office till about 11 PM working on some infrastructure. Originally the scope was for 2 devices but that was expanded to cover 4 devices instead. After notifying different groups I started at 5 pm.

Naturally I did the 2 easier devices first and that was wrapped up by 7 PM. One of my co-workers helped me out and sanity-checked my work. That part went well without any issues. After those 2 I was on my own for the rest of the night.

The next 3 hours were an exercise in futility. This included almost 2 hours on hold with the vendor that was supposed to be providing support. While I was on hold I backed out the changes on that device and worked on the last one. No joy there either.

Except for emphatically expressing my displeasure to the vendor’s duty manager (who failed to get me the support I needed, thanks anyway) I was not fatigued, I was not frustrated, and I never lost myself in “where am I and what am I doing here???”

Here’s why: the entire work effort was all my plan and the team I worked with vetted and approved my steps.

My managers not only knew what I was doing but was available if I needed help or just to explain that I was stuck.

My back out plan called for leaving the old devices in place and untouched. Putting it back all back was a pretty straight forward process.

When I called my boss to report the status he said those magic words:

“It’s been 5 hours. Back it out and shoot off an email to the stake holders.”

I was prepared to go at the problem from a different angle but he’d pointed out that I’ve already crossed the line where I’ll start making mistakes due to being tired. I backed out the last 2 devices, performed my checkouts, and shot off that email.

I got home after 1 AM.

It’s all about the support at work

Anyone who works in IT has times when things just don’t work. When that happens you can spend hours on the phone and let people know the status is still “Yep, beaming Happy Thoughts™ at it hasn’t solved the problem”. Those support conference calls are the worst.

But what makes it all worthwhile is knowing that I have the support of my teammates and managers. It’s a small group that I work with but everyone has each other’s back.

I’m not the easiest person to manage (due to my sense of humor- WHAT??) but no matter how deep into the pool I get I will always have a lifeline. Also good planning and being able to roll back completely is a bonus too.

Here’s why two weeks ago was not a failure.

  1. I planned for the worst.
  2. I made sure my back out plan was solid and I checked my plan with the team.
  3. I communicated in planning, before the work, during the work, and after. Did I mention communicate?
  4. I set a threshold for when to declare a failure and roll back. Which I almost didn’t follow but my boss got me to do.

This next week was spent working out those problems I encountered and last Friday night I tried again. The job was completed successfully by 9 PM.

So what’s the point again?

Being prepared is great and can save you grief at work. Having the support of your managers and co-workers? That’s awesome and I’m glad it’s part of my work environment.