Managing IOS versions?

[John Blakley]

All,

How do you manage IOS updates? From that I mean, when you have a large enterprise, do you methodically do a site at a time, blast all images out at once and then schedule reloads? Do you update over the wire or do you copy the image to the site to update from the lan? Do you use standard tftp or do you use a Cisco product to manage these?

Currently, we don't have a true process in place to manage/update equipment. We update basically when we find a bug and the bug is affecting us. This can cause us to get a few versions apart on routers. For example, Router A may have version 12.4, but Router B could have 15.2, but Router B may have started at 12.4 as well. We're looking for a sound way to keep images across the enterprise standard, so I'm wondering what everyone else does.

Thanks!

John

[Leo Laohoo]

From that I mean, when you have a large enterprise, do you methodically do a site at a time, blast all images out at once and then schedule reloads?

Generally, yes.  I do several sites a night.  I schedule reloads between 3am and 4am.

Do you update over the wire or do you copy the image to the site to update from the lan?

90% of my sites are on dark fibre so I use my PC to host the file(s).  The rest, I choose one site and "plonk" the file there and all sites go to this site to get the IOS.

Do you use standard tftp or do you use a Cisco product to manage these?

I stopped using automated methods because I don't have the visibility to see error messages.  I use TFTP but I have used the faster HTTP method.

[John Blakley]

Thanks Leo! We're trying to come up with some standard. What that will be I don't know yet :)

Sent from Cisco Technical Support iPhone App

[Leo Laohoo]

So far, in my current job my supervisor encourages me to regularly upgrade the IOS.  Aside from stability, this gets IT Security off our back and makes us look good (compared to the other team).

Doesn't mean I've been getting good IOS versions.  I've had bad IOS and am not afraid to admit I've rolled-back a few IOS to previous versions (en masse) even after some weeks of testing.

Someone from management subscribed to Cisco's NOS (I know, what a rip-off!) and several times, the comment coming back was that our network has uniform IOS (and stable) versions.  They remarked that it must've taken months to get up to that level.  So they weren't prepared when we responded took me a full working day to push the IOS (across >900 routers and switches) and reboot the appliances.  And this was done just by using TFTP.

[John Blakley]

I haven't heard of NOS. What's that?

HTH,
John

*** Please rate all useful posts ***

[Edison Ortiz]

John,

http://www.cisco.com/web/services/portfolio/optimization/network-optimization-nos/index.html

[John Blakley]

Thanks Edison! Good to hear from you It's been a while!

HTH,
John

*** Please rate all useful posts ***

[Edison Ortiz]

Leo, the upgrade process isn't time consuming. What is time consuming is the politics involved for any upgrade in today's network. That's the reason the Network Optimization Team made that comment.

You are a lucky person to have that kind of liberty - upgrade the entire network without filing a ton of paperwork.

As for the use of NOS, they/we don't just provide IOS recommendations. There are more services such as: Design Assistance/Review, Software Recommendation, Field Notice/PSIRT advisories, Protocol/Technology Audits, HW End-Of-Life review/report and more.

Be sure to make use of them.

[Leo Laohoo]

As for the use of NOS, they/we don't just provide IOS recommendations. There are more services such as: Design Assistance/Review, Software Recommendation, Field Notice/PSIRT advisories, Protocol/Technology Audits, HW End-Of-Life review/report and more.

Hey Edison!

Glad to see you again.  Haven't heard of you for a long time!

No offense to you Edison, but but our experience with NOS, in this side of the world, is a not something I'd recommend to anyone.  They just send an email with an Excel attached to it.  Nothing more.   The Excel tells us what Cisco appliances have been found, what is the recommended IOS, what is our IOS, blah, blah, blah ...

As for the "Software Recommendation, Field Notice/PSIRT advisories" and "HW End-Of-Life review/report", let's just say that this forum is far superior than the NOS.

I remembered a NOS recommendation an IOS version which some of us know to be very buggy.  It took us some time to stop laughing.

[Edison Ortiz]

Leo,

I respect your opinion and I'm sorry you feel this way.

[Leo Laohoo]

Currently, we don't have a true process in place to manage/update equipment. We update basically when we find a bug and the bug is affecting us. This can cause us to get a few versions apart on routers. For example, Router A may have version 12.4, but Router B could have 15.2, but Router B may have started at 12.4 as well. We're looking for a sound way to keep images across the enterprise standard, so I'm wondering what everyone else does.

I've worked in an organization where there was a plethora of versions of a single switch model.  There were numerous reasons why this was the case:

• No one knows how to properly upgrade the IOS;
• No one cared; and
• Anyone and everyone was TERRIFIED to upgrade the IOS

In one organization I worked in, you were considered NOT a "team player" if you mention the phrase "IOS upgrade".  So I did it all quietly.  When they found out what I did, I threatened to roll-back the IOS to the old version with LESS features. 

[Jon Marshall]

John / Leo

In the last place i worked we used to try and standardise per model eg. 6500, 7200 router etc. Trying to standardise all routers for example to use the same IOS wasn't something we thought worthwhile for a number of reasons -

1) cost

2) functionality/features - we did upgrade to get these but then why pay for additonal licenses just to standardise on the rest of the devices where you don't need the new features

3) security - if you standardise on one IOS and there is a major bug every single device is vulnerable. Having different models running different IOS versions means you limit the vulnerability.

4) stability - there are core devices in your network (switches and routers) where you need a stable IOS and there are less important devices where you can be a bit more relaxed ie. you may need new features on these devices but not actually in the core.

So you can take a few more chances with the IOS on these type of devices that you would not want to take on the core devices.

I'm impressed with Leo's upgrade figures and don't doubt them but in the last place i worked we had sites across the entire UK and upgrades can go wrong so we could not always afford to take the risk that it wouldn;t work. Some devices simply don't like being rebooted never mind anything else.

For me it was always a balance between keeping the number of IOS versions under control and using supported IOS versions but not going to the extent of trying to run the same IOS version everywhere because i'm not sure it is necessarily the best choice.

But it does often come down to personal choice.

Jon

[John Blakley]

Jon,

I completely agree with number 3, and that also carries to number 4. I had an image on a 3825 that hung on me when I removed a policy map off of the interface. I found out that it was due to the version of ios that we were on, so I had to do an upgrade that night to a non-buggy ios. So, if that same image was rolled out to all of our routers, we would also need to upgrade all of our routers to fix the buggy ios because we now know that there was an issue with it.

I completely agree, and normally I wouldn't update the ios unless I run into a problem with the image itself. If it seems stable, I'd rather not mess with it. My boss has a different outlook, and I can see his point as well. If we have two different versions of IOS on, say for example, a 2960 and run into a weird issue, it's going to be hard to determine if it's switch or IOS related. The thought is that the switch that's having problems could have an earlier or later version of IOS as compared to the switch that doesn't have problems. Obviously, depending on the problem, it could be IOS related, but without finding the bug on bugtraq, it would be easier to say, "well, this other switch is running the same ios and it's not having the same problem."

[johnlloyd_13]

Hi John,

For me I do IOS update (downgrade acutally) via TFTP on my PC at our warehouse for all the 2911s before we roll it out.

This is because of the 'priority-list' command on the 15.0 code that we're after that's not present on 15.4.


Sent from Cisco Technical Support iPhone App

[Scott Pickles]

John -

I think that by posting here you did the right thing.  You now have evidence of the fact that there is no one way to do it, nor a way that is considered right or wrong.  You have to consider the size and scope of your environment, and where you place your priorities.  Any changes you make have a risk associated with that, and you need to manage your risk more than anything.  As others have pointed out, if keeping your devices on different versions of IOS manage that risk, then that's the way to go.  If new features come out, don't put them on anything in production at all if you can afford it.  If not, find a device in your network with the least impact should something go wrong, preferrably something you can get your hands on (i.e. not located miles away from you).  The best thing you can do at this point is to prepare a well crafted document of what you believe to be the positive and negative sides to maintaining your IOS versions as they pertain to your environment specifically.  Then perhaps you can speak with your supervisor about your concerns.  He or she may still mandate that you unify on common IOS releases.  In that case, you have at least brought up your concerns which shows a high level of understanding of your environment and your dedication to ensuring it runs as well as it can.  This isn't leading up to an 'I-told-you-so' when something goes wrong.  But when it does (and it will) your supervisor will know that you had done your due dilligence prior to minimize the risk and the pressure will be off you.  I personally do not upgrade IOS versions unless we find a bug or need new features.  Even then, the scope of the roll-out is one site at a time with a well defined plan of attack so that we know how to respond when things don't go well.  In the end, you'll always have the support of those of us that participate on the forums to help you bounce these ideas around before you hit the go button.

Regards,
Scott