Places you wished you had worked

[Jon Marshall]

Reading the telco stories got me thinking about some of the places i have been to while contracting.

I have worked mainly for large companies who have a lot of procedures and change controls in place so getting anything done is a major effort. In my last job one of my tasks was to connect up about 100 3rd party companies via VPN to our network. Some of these companies had inhouse expertise and some didn't so i spent a lot of time on the phone talking them through what they need to do.

A few of the companies didn't have a firewall so we supplied them with pix firewalls. Most managed to get them working, but one company just couldn't get it to work. I spent hours on the phone to them but still no joy so in the end i had to go to their site.  When i got there, they were some of the friendliest bunch of guys i have met. Straight away they told me they were clueless about networking so they would happily do anything i say.

They were in a converted 4 floor house so the computer room was on the 4th floor at the top of the stairs. The idea was to have their cable modem plugged into the pix on the outside and then the LAN on the inside of the pix. I thought this is what they had done but what they actually did was simply run ethernet cables from both pix interfaces to their LAN patch panel. This explained why with all the testing i had done over the phone i had never seen a single packet from them.

So they asked me if i knew what was wrong and i told them how it needed to be connected up. I remember i was kneeling down looking at the pix and the cable modem and i said we needed to disconnect the cable modem and reconnect it up to the pix. I was about to say "of course you will need to do this out of hours because it will take down the internet" when a cable appears over my left shoulder. At the same time, the other guy with me pops his head out of the door and shouts "internet will be down for a while" down the stairs.

Needless to say i connected things up pretty sharpish...

I really enjoy working for big companies because you get to use all the latest technologies but every now and then when i am submitting my 20th change control just to add a vlan on a switch i sometimes think it might be fun to work at a company like the one above !

Jon

[Rob Huffman]

Hey Jon,

Great story my friend!

I know what you're saying here.....we always joke that our

"change management" regime is so stringent that we feel like

we are working for NASA and launching the Shuttle...hahahahaha!

I think we understand the need for change control, but at times it seems

to have gone a little overboard.

I would love to yell down the stairs "hey folks, the phones will all be down for a while"

Sweeeeeeeeeeeet!!

Cheers!

Rob

PS: we'll have to come up with a new analogy now that the Shuttle program has ended

[Jon Marshall]

Hi Rob

I would hate to work for Change Management, they actually do a very valuable job but everyone else just thinks they are a pain in the backside !

I was so tempted at my last place after handing in my resignation to send a quick e-mail to CM telling them that i was going to reboot the core DC switches just because i could

Thankfully, my professionalism, such as it is, managed to win the day.

Jon

[Rob Huffman]

Hey Jon,

Too funny my friend

You are most correct that CM does a very valuable service. I have seem some

crazy a$$ed "hair brained" schemes over the years that would have been squashed

by the CM "gurus" in a heartbeat, but went forward with disastrous results

The problem is the fact that the paper work, project plans, CM requests etc.etc.

sometimes take longer than the job itself.

Our current CM window for any large jobs goes from Saturday @ 11:00PM to

Sunday @ 6:00AM other than that it's 36x7x365 (note that we try to squeeze in 36

hours per work day ...hahahahahahaha!)

Cheers!

Huff

[Dan Loring]

I once worked Network Operations at a bank.  Change controls were literally like Houston landing the shuttle with 6 large screen tv's monitoring the systems.  Now I work for lumber supply, ok it's 5pm everyone is off, systems are coming down!

[Jon Marshall]

Dan

If there are any network jobs going where you work let me know

Seriously though our CM was ridiculous. At times it seemed like just about anyone could say no to anything. I remember one specific incident where we had been waiting to do some work in the DC for about 3 months and we finally go the go-ahead to do the work on Saturday.

On Friday it was cancelled because some guy who literally works on his own in a hut somewhere in Scotland said it couldn't go ahead. 1 week later the firewalls, which we warned could fail at anytime, failed. And i ended up in the DC on a weekday night fixing them !

You just couldn't make it up !

Jon

[Giuseppe Larosa]

Hello Jon,

my dear in my last job it took six mounths to have a remote POP made of two PE nodes and two multilayer switches to reach the final setup ....

best practices would be good if all involved parties play fairly I have attended the ITIL service management foundation and most of it is just good sense but it has been used by consultancy firms as a key to get works about company re-organization(s)

this is a market segment.

what the course gave to me is some lingos to talk with managers

I remember in a primary bank a team made of three beautiful women going around to interview network operators and help desk suggesting how to deal with remedy tickets

I have been not interviewed because I had just to look at the critical datacenters and behind my desk there were the desks of the network monitoring manager (my boss there) and the head of network engineering.

At the beginning was a really crowed open space, and it was difficult to understand who where the bank employees and who were consultants of any kind

It was so noisy that it was difficult to make small changes

I left that job after 6 mounths with just some DLSW+ added, at the end I was a sort of companion for the network manager that had started to make me take part in meetings that were far from my duties there.

ok it is a bank paying a lot of money to have a senior network engineer to look at some alarms, feed management peoples with new visio diagrams every week, but it was really boring and at the same time there was high  pressure because  any incident could have a victim among the non employee network engineers ...

Nice to see how CSC people had similar experiences

edit: the bank job was in 2006 not my last job to be clear

Best Regards

Giuseppe

[Rob Huffman]

Hey Jon,

Isn't that just the way "Murphy's Law" always works

Our Data Centre folks are going through a huge re-design right now with new

equipment, design scope, etc. etc. etc. for big buck$ just to avoid a possible

2-3 minute outage for some APP's if something needs to be updated, but the upgrade for the re-design

will take things down for much longer???? I'm not saying the plan itself is flawed, it's actually

going to be great! But the CM rationale that forced the change is suspect to say the least.

I'm thinking of looking Dan up as well!

Cheers!

Rob

[Leo Laohoo]

Change Control.  Hmmm ... someone opened a can of worms here.

The last place where I worked was a stickler for CM.  If any part of your ITIL Change Request has a wrong spelling, misplaced punctuation mark or wrong grammatics, it was denied until the author can get their act together.

Right, so I fixed that one up.  Next step is to get it past the Change Control Board.  This is another painful process where you have to explain to the people who requested me configure or install the appliance WHY this was necessary.

Ok, so I got the CCB passed and the client finally approves my change and I implemented successfully.  Thought that was through, right?  Wrong.  I'm halfway finish.  Next I have to write up an engineering document about all the stuffs I have done, including the configuration.  Everything.  I find this process totally useless because I can guarantee you that in 3 months the configuration will change.

Lucky thing I quit that organization.  Nowadays, all I have to do is enter the command "reload", hit enter and I've just upgraded the IOS to the distribution switch of a site at 11am on a Monday.  Did I just hear someone scream?

[Jon Marshall]

Leo

I forgot about the CCB, thanks for bring back more painful memories

Trying to explain to non-technical people what excactly you are doing and what is the impact. You can never know if there will be problems but if you say there might be they reject the change so you end up guaranteeing it will work perfectly even though you know it could go completely wrong !

Jon

[Rob Huffman]

Yeah!!

Thanks Leo +5

To protect the innocent I won't name any names here ......but

someone I'm very close with was recently going through some

major Application upgrades and they sent all the Staff home @ noon

on Friday .....WOW!

Now that's a place I wished I'd worked. Sweet!

Cheers!

Huff

[Leo Laohoo]

Thanks for the ratings "Smooth" Rob.

Noon on a Friday?  What's wrong with that?  We're normally p1ssed-as-a-parrot before Friday lunch.

LOL

[Leo Laohoo]

To be frank with everyone reading this post:  I understand ITIL.  I understand WHY this process is an absolute requirement in a medium, large and very-large network environment.

However, what I dislike about this process is how can people, sitting behind a desk and have no technical knowledge, seems he11-bent in a well-oiled machine stop for the sake of "process" is beyond my sanity.  For example, you have have an incident and you are rolling your sleaves and start troubleshooting.  But, no.  You can't or NOT ALLOWED to roll up your sleaves until someone raises an incident record/trouble ticket.

Ok, so the incident record/trouble ticket has been raised and we nail where the culprit is.  Let's go fix it.  Hold it.  You can't even "think" of fixing it without someone raising a change request.  Let's say it's an emergency change request so we can get the ball rolling.  Oh no.  Even though it's an emergency change you still can't start until someone approves it.

So let's do the maths, shall we?  How long does it take for the incident to reach our attention and how long does it take to fix it?  Now throw the ITIL process in and how many hours does it take to accomplish something in half that time?

I remembered the first day of the ITIL training session.  The instructor knows that any PM work is looked upon with disdain and disgust by many tech-minded people because it promotes nothing but red-tape and a$$-covering.  SO he said, "Remember that when it comes to the ITIL process and common sense, common sense takes priority."  Unfortunately for him, the people behind those desks didn't get the message. 

[Rob Huffman]

Hi folks,

I really loved the recent additions to this thread from Giuseppe and Leo

+5 each for this nice insight!!

We all seem to agree here that the need for a good Change Management

process is a good thing (mostly ) but when the ability to do the job

you are mandated to do is put in jeopardy then CM loses it usefulness.

Part of our CM process are tickets called CSU's or Critical System Updates. So

whenever a major outage occurs you are supposed to write up and enter

a CSU that then notify's a large group of people about the problem...

OK....but isn't it more important to fix the problem!! I understand that

your CIO would like to know about issues in case he/she gets stopped in

the hallway by an irate user, but if you can't find one of your colleagues to

enter the CSU ....what to do.....fix the problem or do the paperwork??

I'll fix the problem every time, but this is often frowned upon by some,

but I just don't care anymore If you want to get rid of me for fixing

things then so be it!

It's this type of RED tape that doesn't make any sense and why engineers

hate the processes so much.

We recently added a very technical/common sense manager to the CM

approvals team and things are improving (slowly) but the whole cumbersome

set of tasks involved with CM just add to the pain and stress related to our

jobs!

Maybe I'm just jealous......I want to be the person who makes everyone jump

through hoops ....hahahahahahahaha!

Cheers!

Rob

[flashpointdevon]

I passed up a job at Intel for what seemed to be at the time a very lucrative startup. Management screwed the pooch and after that Intel wouldnt call me back. Missed out big time =(. I tend to enjoy small company settings, but really id trade that in for a company like Intel....