Catalyst 6500 High CPU utiliatization IOS base

[rakuntal]

Hi,

Switch at our DC Cisco C6513 going into hang state with 'logs of not free Memory'. Upon checking CPU utilzatio found to be 100% with single process of IOS-base consuming almost 99%. Rt now we have shut down this switch as it is destabilizing the network.

%SYS-2-MALLOCFAIL: Memory allocation of 120 bytes failed from 0x7636F920, alignment 32
Pool: Shared Memory Free: 0 Cause: Not enough free memory
Alternate Pool: none Free: 0 Cause: No Alternate pool
: udp.proc : (PID=16429, TID=7) : -Traceback=(s72033_rp-adventerprisek9_wan-1-dso-p.so+0x27C3C) ([40:0]+0x2AEF0) ([30:-4]0-dso-n+0x19404) ([30:-4]57-dso-+0x13928) ([30:-9]70+0x7E54

Please help

Model cisco WS-C6513 (R7000) processor (revision 2.0)

Cisco IOS Software, s72033_rp Software (s72033_rp-ADVENTERPRISEK9_WAN-VM), Version 12.2(33)SXH5, RELEASE SOFTWARE (fc1)

[marce1000]

 - Look for and or upgrade to the latest advisory software release for this platform , check if that can help ,  

 M.

[Reza Sharifi]

Hi,

The version of IOS you are currently running on this switch is more than 15 years old. Depending on the sup module you have installed, there are still images from 2022 you can download and install.

HTH

[Joseph W. Doherty]

Memory allocation failures in all (?) IOS versions is a bad, bad thing.

There are various causes for running out of memory.  Some just from using features that consume lots of memory, but unexpectedly.  Some from just running with a low amount of free memory that gets very fragmented.  Some because of IOS bugs, especially the memory leak kind.

"Fixes" include "patient: doctor, doctor, it hurts when I do this - doctor: then don't do that".  Some from adding memory, (if possible).  Some from moving to an IOS release w/o the bug.

[Leo Laohoo]

6513 + Sup720

It is widely known and accepted that the most stable IOS (ever) are the SXI and SXJ trains.  

No other IOS/IOS-XE can match the performance and longevity of a 6500 on SXI and SXJ trains.  

[rakuntal]

After disconnecting all links the CPU utilization has reached to normal. tcp.proc  and ios-base showing high utilization. Is it some looping/multicasting or IOS problem? because hardware status is ok.

[Leo Laohoo]

---

@rakuntal wrote:
After disconnecting all links the CPU utilization has reached to normal. 

---

Plug the links in one-by-one and stop when the last link plugged in caused the CPU to go up again.

[rakuntal]

Hi, 

We reconnected all links to switch one by one. but now switch is neither hanging nor showing any high CPU utilization. Same configuration same connectivity, but currently switch is working perfectly normally without any issues. The switch was powered off  for 3-4 days. Now the problem is I can't give an explanation to managment that why switch was hanging because I have checked all logs and everthing but nothing found. This incident created havoc in our DC.

[Joseph W. Doherty]

Re: explanation

Since log did show memory allocation failures and a traceback, very possible this might be due to a bug.

As, I believe, both switch and software are no longer supported, don't believe Cisco would work to find cause, and if cause could be identified, nor would Cisco provide a fix.

You might explain such to management, i.e. the inherent risk of using hardware or software that's no longer supported.

Your options going forward include:

Doing nothing (beyond crossing your fingers and hoping it doesn't happen again).

Updating your software (such at to the latest release for what your running now, or, perhaps moving to the latest version and it's release, or perhaps moving to any version and it's release with almost the latest release date).  (BTW, upgrading your software was implied or explicitly mentioned by other posters.)

Migrate to a currently supported platform.