Cisco IOS version so unstable we will no longer use cisco

[billy-kemp]

Recent Cisco IOS version for stack switching is so unstable we have pulled Cisco Inventory.   IOS bugs causing stacks to reboot, and all cisco can recommend is downgrading to a previous version, which also was so full of bugs we had to do yet another downgrade.  Cisco IOS developemnt and management has gone to crap.  Support only wants to refer problems to sales reps.  sales reps are no help.  No one at Senior Cisco Management teams seems to know or understand customers cannot continue to invoke downtime for criticle networks.  

We are now seeking replacement vendors for our 490 Cisco edge devices, 16 Cisco core backbone switches and routers.  We are done with Cisco not paying attention to developement bugs in IOS. 

[Leo Laohoo]

IOS bugs causing stacks to reboot, and all cisco can recommend is downgrading to a previous version, which also was so full of bugs we had to do yet another downgrade.

You've got to be more specific than this, Billy.  What is your IOS version & what is your hardware?

Just a wee bit of feedback, I've been actively using and testing the latest IOS for the 2960S and 3750-series family of switches.  They are the following

1. 12.2(55)SE (and later);
2. 12.2(58)SE (and later);
3. 15.0(1)SE (and later); and
4. 15.0(2)SE

Any IOS lower than the ones I've mentioned above, I would NOT recommend on using.  I've tested them, though not intensively.

3750

Of the ones I've mentioned above, 12.2(55)SE5 is the "most" stable I've seen.  12.2(55)SE6 was recently released and it looks to be like a very "fine lady".  Cr@@@@p, I'm gonna get into serious trouble ... 12.2(58)SE (all sub-versions) and 15.0(1)SE (SE, SE1) must be avoided like a plague.  I wouldn't touch it with a ten-foot-pole.

2960S

I'm currently running 15.0(1)SE2 and SE3 on my 2960S and I'm in the midst of upgrading to 15.0(2)SE.  So far, so good.

Crashes

You say your stacks crash.  Describe the "situation", please.  How many stacks do you have and how many will "crash" due to the bugs?

Do you have crashinfo file(s)/subdirectories created? 

Support only wants to refer problems to sales reps.  sales reps are no help.

I've never heard of treatment by Cisco TAC engineers like this before.  The only time TAC will encourage me to contact my Cisco AM is when I don't have a valid Service Contract.  Is this your issue?

We are done with Cisco not paying attention to developement bugs in IOS.

Firmwares and OS, like any MAN-MADE products, are not perfect.  NEVER.

Kindly don't take what I'm about to say next as a "dare", but, every manufacturer have issues too.  Lest it be HP, Juniper, Avaya. 

[tonyhayek]

Hello,

I have two 3750E stacked "Version 12.2 (50)", i would like to add another 3750E "Version 12.2 (52)" to the stack as a member not as master.

My boss does not want to upgrade to the latest version at all.

My question, is it ok to add a switch to the stack with higher IOS version and be just a member?

Any suggestion, ideas ?

Thanks

Sent from Cisco Technical Support iPad App

[Leo Laohoo]

Hi tony,

If you can't upgrade, then DOWNGRADE the new ones.

Take note, both IOS you are using are buggy. It's just a matter of time ...

Sent from Cisco Technical Support iPad App

[stubinski]

We are now seeking replacement vendors for our 490 Cisco edge devices, 16 Cisco core backbone switches and routers.

That is a whole lotta money, not to mention the training cost to support the new hardware.

 Support only wants to refer problems to sales reps.

Cisco has, IMHO, one of the best support services of any vender and it is much much better than Juniper or HP.

Just my 2 cents

[billy-kemp]

We work in a Hospital.  Uptime is not a 9 - 5 job.  Every downtime interferes with patient care, which is our #1 priority.  ANY downtime is a major event. 

We upgraded to the 2960S with IOS 15.0 with 4 switches stacked, and we are using VSS.   8 stacks in all.  Converting to these new switches, and doing this major upgrade in 4 hospitals was a major undertaking requiring hours of downtime.

Now the stackes are rebooting every day. 

Cisco says, hey, its a known bug in 15.0...downgrade the code.  A stock answer when they dont know the answer.

SURE,  reloading code and rebooting the switches, interrupting patient care, and these switches take 8 minutes apiece to reload, 8 x 4 - 32 minutes PER STACK.    Now they say, wait, you should never stack more than 2 switches.  Totally contrary to every doc on the Cisco web site, and not what our tech and sales reps recommended. 

Cisco is not doing due dilligence before releasing new code.  We see this same type of issue in WISM code and 6500 code.  We are spending alot of time reloading lower code versions.  

Cisco may claim great support, but recent IOS versions have been so buggy, we are investigating Nortel for replacements.  Our vendor is pricing them now.

Cisco does not understand 99.99 uptime is not just a number, its a requirement in hospitals.  Configuration Management is the I.T. field to assure software is compliant and bug free.   It seems someone is not paying attention at Cisco.

[Joseph W. Doherty]

Disclaimer

The   Author of this posting offers the information contained within this   posting without consideration and with the reader's understanding that   there's no implied or expressed suitability or fitness for any purpose.   Information provided is for informational purposes only and should not   be construed as rendering professional advice of any kind. Usage of  this  posting's information is solely at reader's own risk.

Liability Disclaimer

In   no event shall Author be liable for any damages whatsoever (including,   without limitation, damages for loss of use, data or profit) arising  out  of the use or inability to use the posting's information even if  Author  has been advised of the possibility of such damage.

Posting

Hmm, on the subject of "due diligence", what was your decision process selecting an IOS image to use?

I'm not a particular booster of Cisco, and have worked with other vendors too, but on the whole, Cisco does help provide what you need for selecting "stable" IOS code images often better than its competitors.

Many other vendors don't provide extensive release notes and/or document known bugs.

Many other vendors don't "tag" stable releases (i.e. old GD or new MD) or have special "safe harbor" testing with documented results.

Many other vendors won't provide their FEs to work with you on bug scrubs when reviewing IOS images for specific deployment usages.

When stability is of paramount importance, I would be a little leery of deploying recent technology, regardless of the vendor.  You mention VSS and 2960S stacks, but neither has had extensive time to be "wrung out".

Cisco isn't perfect, but I haven't found other vendors to be particularly better either.  I'm sure there's a good chance that Brand X will cost less, but you might want to reconsider continuing using the equipment you have with improved methodology in selecting IOS images to use.  If you do go with Brand X, you may still want to review your software selection methodology.

[billy-kemp]

thank you joe.   we were told to use 15.0 code >for VSS< by our Cisco tech rep.   Cisco support could not provide an answer as to why the stacks were rebooting, but they were able to recreate the issue.   Then they recommended turning off VSS and only stacking two switches at a time, and downgrading to a lower IOS version.  I know the risks of being on the bleeding edge of Cisco's IOS release's.  We faced the same issues with code for a large scale wifi install, as well and CWDM implementations.

In many of the Cisco class's I have taken, the instructors are quick to point out that IOS is written by programmers on different continents, Asia, Australia, US, ect..with each using different methologies and verbage.   Its left up to us to find the bugs.

Cisco offers no other advice or assitance except to go back and do it again. 

[paolo bevilacqua]

First let me say that I have multiple customers using that roughly same equipment, and everything works perfectly in a 24/7 enviroment.

I can give some advice based on previous experience, part of which spent working for Cisco.

• First and foremost ask your Cisco rep to be escalated to "Critical Account". That is onerous for both parties, but is the only way to solve exactly the kind of issues you're are having.
• Be open to accept design and configuration changes by senior TAC and Consulting System Engineers (CCIE level only).
• Exige that any answer you get from TAC is backed by engineering, and that your network topology is recreated by Cisco.
• Do not stop raising voice in this or any other appropriate venue.

Hope things will resolve eventually.

[billy-kemp]

thanks paolo.  We are always open to config and design input from Cisco Engineers...thats how we got here.  Unfortunately it almost always requires downtime, rebooting.  Ever reboot a network for 200+ doctors and nurses in a hospital?  In an ER, Operating Room, Surgery Center?  Everything comes to a stop.  Then have to go back and do it several times on advice from Cisco?  The interruption to patient care and revenue is horrendous.  Its not a pleasant experiance. We dont make donuts or sell mufflers, we save lives. 

Our configs for this install were fully vetted by Cisco Engineers prior to deployment.   Cisco does not get HealthCare.  After sending multiple sho tech files, logs and TAC calls, they see the issue, and they have no clue why. 

[darren.g]

billy-kemp wrote:
 thanks paolo.  We are always open to config and design input from Cisco Engineers...thats how we got here.  Unfortunately it almost always requires downtime, rebooting.  Ever reboot a network for 200+ doctors and nurses in a hospital?  In an ER, Operating Room, Surgery Center?  Everything comes to a stop.  Then have to go back and do it several times on advice from Cisco?  The interruption to patient care and revenue is horrendous.  Its not a pleasant experiance. We dont make donuts or sell mufflers, we save lives.  

Sorry, you've said this several times now, but if stuff in this environment is so critical, why do you have as a single point of failure one of the *cheapest* low-end switches in Cisco's enterprise range? The 2960 is not exactly a high-powered switch - if you want robust, reliable stacking, why not use the 3750 series?

I think the "revenue" part may be a more important part of your complaint than the patient care part, despite your indignation at not being able to save lives instead of making mufflers. If patient care is so d*mn important, then build your network on a mission-critical assumption, and have redundancies.

Your configs may have been fully vetted by Cisco engineers before deployment (something I doubt), but if they were I strongly suspect you didn't stress highly enough the reliability/uptime requirements, or they *certainly* wouldn't have recommended IOS 15. Joseph is correct - Cisco isn;t perfect - but neiter is Juniper, Extreme, Nortel, HP - any of the other major vendors. If you are going to go the expense of ripping out *all* your Cisco hardware in a fit of pique and replacing it with Notel, good on you, your mind is obviously made up despite the many valid counter arguments made by members of *much* better standing in this community than I am - but it would have cost a lot less to build the Cisco network properly in the first palce, and put in place sufficient redundancies to rsolve the issues you're having.

Cheers.

[Darren Ramsey]

Billy,

I work for a 1000 bed healthcare system and have used Cisco for over 10 years. We've had our share of issues with N5K, CUCM 5.x/6x, Wireless in the 3.x/4.x/5.x/6.x days, and have thought more than once about moving to Juniper/Aruba/Arista. TAC support is not what it used to be and here are articles from Greg Ferro that do a good job outlining my TAC experiences as well. http://etherealmind.com/cisco-culture-of-buggy-code-and-the-failure-of-the-tac/     http://etherealmind.com/selecting-cisco-ios-release-choice-which-how-choose-decide-bug-scrub/     Sometimes we get a good TAC engineer and sometimes we ask for the case to be reassigned. When we have exhausted all options we escalate to our account team and let them beat on TAC.

But in the end I agree with the other posters that you are still better off with Cisco. With 50-70% market share in most areas, there are many more users out there finding bugs than competing vendors, and Cisco makes documentation freely available and easy to locate in SRND and Cisco Live Virtual. Plus there is plenty of education and certified individuals to help.

Now with that being said, we ran 12.2(46)SE on our 3750 for 2-3 years without a crash or reboot, and now running 12.2(55)SE3 for over a year without a crash or reboot.  Our stacks are usually between 2-6 3750s deep. Cat6K and Cat4K both seem to be more solid than Cat3K, and that is why we are replacing all 400+ 3750 with Cat4K/Sup7E over the next several years. You can't put a price on ISSU in healthcare.

We have a lab environment that has more equipment than most smaller companies and have found many N5K and wireless issues from baselining and performance testing. Next we upgrade and test new code in our corporate offices for several months before we push it out to the clinical areas. I think if you get the right person in TAC to assist, get the right code, and go with a proven SRND design, you should be able to get 5 nines in healthcare. We've not had a planned or unplanned outage in our DWDM backbone since May 2005.

[dominic.caron]

It must have been a bad SE. Over the years, I've learned that if part of the network is realy critical, It's not a good idea to buy the cheapest switch or to stack. There is safety in having every peace of hardware operating in standalone mode. I've use hundreds of 3750-E-X switchs and I would never put them in critical segment of the network. Same for VSS, a bug with that could potentially take the pair of 6500 down.

Just like Joseph said, you need to vet the quality yourself.

And...I provide network services to 7 hospitals affiliated to my campus,

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Ah, if Cisco vetted your IOS image and platform selections, understandable why you're upset.  Perhaps you were a little "too trusting" of a vendor (recall I noted I'm not a particular Cisco booster, but their tech advice also is often too, I've found, slightly better than most of their competitors, but you still need to vet the quality yourself [similar to advice received on these forums]).

Paolo's post is excellent.

[Leo Laohoo]

I must say your vendor doesn't know what they are selling or the Cisco SE/AM ain't doing theirs job.

Hospital, did you say? I wouldn't stick a 2960/2960S in a hospital. The model doesn't have enough grunt.

Either 3750X or 4500R+E with dual supervisor is what I'd recommend of "hitless" IOS upgrade.

Sent from Cisco Technical Support iPad App