At work you can have good failures too

Sometimes when things go wrong at work they still go right.

Two Friday nights ago I was at the office till about 11 PM working on some infrastructure. Originally the scope was for 2 devices but that was expanded to cover 4 devices instead. After notifying different groups I started at 5 pm.

Naturally I did the 2 easier devices first and that was wrapped up by 7 PM. One of my co-workers helped me out and sanity-checked my work. That part went well without any issues. After those 2 I was on my own for the rest of the night.

The next 3 hours were an exercise in futility. This included almost 2 hours on hold with the vendor that was supposed to be providing support. While I was on hold I backed out the changes on that device and worked on the last one. No joy there either.

Except for emphatically expressing my displeasure to the vendor’s duty manager (who failed to get me the support I needed, thanks anyway) I was not fatigued, I was not frustrated, and I never lost myself in “where am I and what am I doing here???”

Here’s why: the entire work effort was all my plan and the team I worked with vetted and approved my steps.

My managers not only knew what I was doing but was available if I needed help or just to explain that I was stuck.

My back out plan called for leaving the old devices in place and untouched. Putting it back all back was a pretty straight forward process.

When I called my boss to report the status he said those magic words:

“It’s been 5 hours. Back it out and shoot off an email to the stake holders.”

I was prepared to go at the problem from a different angle but he’d pointed out that I’ve already crossed the line where I’ll start making mistakes due to being tired. I backed out the last 2 devices, performed my checkouts, and shot off that email.

I got home after 1 AM.

It’s all about the support at work

Anyone who works in IT has times when things just don’t work. When that happens you can spend hours on the phone and let people know the status is still “Yep, beaming Happy Thoughts™ at it hasn’t solved the problem”. Those support conference calls are the worst.

But what makes it all worthwhile is knowing that I have the support of my teammates and managers. It’s a small group that I work with but everyone has each other’s back.

I’m not the easiest person to manage (due to my sense of humor- WHAT??) but no matter how deep into the pool I get I will always have a lifeline. Also good planning and being able to roll back completely is a bonus too.

Here’s why two weeks ago was not a failure.

  1. I planned for the worst.
  2. I made sure my back out plan was solid and I checked my plan with the team.
  3. I communicated in planning, before the work, during the work, and after. Did I mention communicate?
  4. I set a threshold for when to declare a failure and roll back. Which I almost didn’t follow but my boss got me to do.

This next week was spent working out those problems I encountered and last Friday night I tried again. The job was completed successfully by 9 PM.

So what’s the point again?

Being prepared is great and can save you grief at work. Having the support of your managers and co-workers? That’s awesome and I’m glad it’s part of my work environment.

Camera 360

We are your typical American consumers

Sometimes you plan on buying one thing and end up getting something slightly different.

As part of my job responsibility, I provide support after hours. It’s not something that happens frequently because the first and second level support is really good. If a problem is escalated to me during my personal time, I don’t question it because I am treated as a last resort.

Last month I had a problem that started at Sunday 5 PM-ish and lasted till just after midnight. My home office line is a pair of cordless phones with old batteries and they kept dying on me. The rotten phones only had speakerphone on the handsets and not on the base station!

That was last month. This past week Lily had all day meetings and was working from home. The same thing happened to her so today we got a replacement at Costco.

Her: That model has a base speakerphone and a total of 4 cordless phones.

Me: Yes, that’s true, but we’re looking for just one cordless phone and speakerphone base station.

Her: It’s $65.

Me: Yes, but it’s got three more phones. What are we going to do with the other three?

Her: It’s $65. On sale. Best Buy’s would probably be more expensive and this one is normally $80.

So we got this unit. it’s a discontinued AT&T phone but it works well. I’ve put the three spares in the kids bed rooms and the guest room. I haven’t said a word to the kids, I wonder how long it will be before they notice?

It’s a neat phone. I kept getting “Messages available” even though there was none on the base station. It was from Vonage; the phone got the message notification that I had messages and was letting me know. Cool! The old ones never did that.

Ice cream anyone?

At work I had offered months ago to walk over to Carvel on 2nd Avenue between 24th and 23rd streets.  This was in return for the help I had gotten from the NOC.  It was turning into an on going joke and today when I was reminded about it I said “It’s not too cold, why not today?” and took orders.

I headed out for the ice cream run and this is what I found.

Photo

Nice! They went out of business. Probably just to spite me. I took the photo because no way the guys were going to believe this. My timing is still perfect.

What, you don't say hello anymore?

Sunday morning I had to be in New Jersey before 3 AM to work on powering up the data center there. The company tries to power down it’s data centers once a year to make sure that BCP and DR works.

I got there at 1:30 AM which was okay since we got the building back early and I started my tasks like a crazy person. Except for it being the Memorial Day weekend it was not bad at all. I might volunteer next year for the early shift when we do this again.

When you get to the front door they check that you are on the list and authorized to be there.  They also give you a badge signed by the coordinator that you have to wear. It has your name on it naturally.

After about 10 AM the data center started filling with server and application support staff. I work for Networks and until our equipment is up there is no point in those folks coming in. If they can’t communicate on layer 3 (look it up) then their apps can’t work.

Several times I was stopped and asked/told the following.

  • “Hey Stefan, how’s it going?”
  • “What, you are to good to say hello anymore?”
  • “I don’t understand. Why are you saying your name is Jan?”
  • “WOW. They messed up your first name on your name plate!”

You get the idea. I work in the same team as my identical twin brother and he’s been with the company for about 8 years. Lots of people know him. Even though they are aware that his identical twin brother works here, until it’s spelled out for them they just don’t connect that I’m not him.

Usually I walk back to the person, show them my name plate and explain it to them. I could have a lot of fun with it but by noon I was too tired to play games with people. My co-workers laughed it up though, so it’s all good.

One day limo driver

This morning Lily was going to head to LaGuardia airport and catch a plane to Detroit. I was going to head to the office at my regular time and put in a fully productive day at the office.

The person who was going to pick her up forgot.  I ended up getting dressed in record time and driving her to the airport.  She was running late and had to leave NOW.

She made the flight but it was close.  I headed back home and called my boss to apologize for the last minute change in my schedule.  If I headed to the office I’d have gotten into the city almost 2 hours late and I was not sure if I had any conference calls. I don’t like making changes to my work schedule all off a sudden. Do it too often and it’s abusive.

The day worked out but the high point was my being on a call with my 6 year old son stage whispering “Daddy, can I play with the Nintendo DS?” and me pantomiming the reply of “LEAVE NOW OR FACE THE AWESOME WRATH OF DAD”.  All this while replying to people on the call with “Sure, that change should work. Let’s do it Friday after hours”.

Sign language is a parenting tool.

With Lily not coming back until the evening, I had to bathe both kids.  I need to install a power washer; did you know that little girls need hair conditioner?  Boys are much easier. Ivory soap can double as shampoo and no one ever notices on a boy.

Later on it was her turn to play a game but she kept giving the game to her brother.  She also complained often that he’s not letting her play. This is a life lesson for my son: watch out for girls.  That she just gave him the game did not stop her from trying to get him in trouble.

I’m really glad Lily came back today.

DNS excitement! Panic at the office!

Well not really panic, just your usual vulnerability patching day at the office.

When I saw Dan Kaminsky demonstrate voice over DNS, I was convinced that he dreams in BIND source code.  It was a neat demonstration.

Now he has uncovered another vulnerability in BIND regarding UDP source port prediction. It’s causing some excitement in the work place as to what the impact could be and how soon our vendors can release patches.

I’ve had to do some explaining as what it means;  see Matasano’s blog for more information.  Thomas Ptacek sums it up really well here and states the impact more here.

You’ve got to love someone who can explain the seriousness using a movie quote from Jack Black.

Looks like no Amtrak strike

Penn Station as it used to be, Picture from Wikipedia

I heard on 1010 WINS this morning that a tentative agreement between 9 unions and Amtrak was going to be announced today. That’s a relief as both Lily and I both take the LIRR to Penn Station get to work. It would have been a pain for us to get to work, likely we’d have gone to Queens and taken the MTA.

We both enjoy working in NYC. When I started commuting again it was like I’d been banished to a third world country and had returned to civilization. My first week I was walking to work and trying not to look like a tourist.

That’s an exaggerated reaction and there are advantages to working close to home in Long Island. But I just prefer working in Manhattan and never mind the 90 minute commute.

Picture from of Wikipedia article on Penn Station.