4510R crashed and SSO removed on new IOS

kirkster · ‎05-20-2005

Hi,

One of my customers with a 4510R with Sup 5 and 12.2.25EW experienced a software crash with a "low on memory" error. All the line cards had a red status LED and the console was unusable other than the "low on memory" error that kept being displayed. I had to switch the system off then back on (taking 1000 users with it) to recover the situation in a hospital network. If this had been a VOIP job it would have been a disaster for us. The 4510R system had SSO configured. I notice that the software has been deferred and that the replacement EW1 does NOT have SSO in the code. For that matter NONE of the latest 4500 releases have SSO. Do Cisco have problem with this feature? Bug navigator shows nothing. Do Cisco mysteriously remove features like this?

How could I have configured the system to capture info I could have sent to Cisco before the crash? I cannot find any coredump functionality on these switches.

This is the most serious situation I have encountered with Cisco kit. Any one got any other disaster stories?

Regards, Steve

Prashanth Krishnappa · ‎05-20-2005

12.2(25)EW has been deferred for unrelated issues. 12.2(25)EWA1 does have SSO support. There is no coredump capability on the 4500 switches. To find the root cause for the crash, attach outputs of

sh ver

sh mod

more slavecrashinfo:data

sh platform crashdump

sh log

If you have a syslog server which has the "low on memory" error messages, attach them as well.

kirkster · ‎05-20-2005

Hi,

Thanks for the reply. I could not configure SSO on the EW1 code - I will re investigate. Does the crashinfo on this platform get written to the flash or is it in memory?

I will get these details and raise a TAC case.

Thanks for your help.

Regards, Steve

Prashanth Krishnappa · ‎05-20-2005

Its in memory. Try 12.2(25)EWA1 rather than EW1

kirkster · ‎05-20-2005

Thanks again for the reply.

It's not much good being in memory when the box has no access to the console and I have no means of getting this info. I tried to regain control of the box for about ten minutes in front of the irate customer before I had to power cycle it - there was no way to get the console back and all the line cards and supervisors had red status LED's. All I had on the console was "low on memory" message every five seconds.

Coincidentaly, the night before when I confgured a few access ports into a vlan I noticed the console was very slow to return me to the command prompt after entering a command and occassionaly dropped my telnet session. Is this relevant do you think? I have noticed this slow command response on many other 4500 systems with Sup 5 redundancy.

I have already upgraded to the supported release. What can I do to get the crash info if this fault happens again?

A very worried Cisco 4500 integrator I now am.....

Steve

Prashanth Krishnappa · ‎05-20-2005

Its in NVRAM and is saved even after the reload/crash. So you should still be able to retrieve it. Post the commands I had asked for.

Josef Oduwo · ‎05-20-2005

Prashanth is right, crashdump info is saved (at least) on 4506s.

One of our core/distribution 4500s fell over, taking down senior managers in our department, ten days ago.

The switch Went into rommon mode - yes, I know, I know..."it wasn't me" did not help much (even though it was true...): Console extremely slow. Eventually I threw in a newer image and was able to bring back switching capability in the short term.

Josef.