Cisco ISO issues

[q-le]

Hi,

We have 4 switches 3750, 2960 been running ios ver 12.2(55)SE6 for years !!! without any problem.

Suddenly yesterday all 4 xswitches crash same time and restart with the following error:

%PLATFORM-1-CRASHED: Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!

Any advices are much appreciated.

Thanks

Peter

[Leo Laohoo]

---

@q-le wrote:

been running ios ver 12.2(55)SE6 for years

---

Hi Peter,

If the switch or stack has been up for >2 years, then I can guarantee it'll crash due to memory leak.  

[q-le]

Hi Leo,

so how do stop memory leak ? reload/reboot it now and then ?

they restarted themselves yesterday and now I checked and the CPUs is quite high so they may 

reload again ???

P.

[Leo Laohoo]

1. Reload every year. Don't leave Cisco gear operating more than 2 years.
2. Post the complete output to the command "sh proc cpu sort | ex 0.00".

[q-le]

Please see output below:

Thanks

DATACENTRE#sh proc cpu sorted | ex 0.00
CPU utilization for five seconds: 19%/0%; one minute: 14%; five minutes: 13%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
4 928841 41403 22434 5.75% 1.49% 1.20% 0 Check heaps
75 3082514 499501 6171 3.83% 3.69% 3.72% 0 RedEarth Tx Mana
74 1364322 814471 1675 1.75% 1.65% 1.65% 0 RedEarth I2C dri
39 42252 1333 31696 0.63% 0.07% 0.01% 0 Per-minute Jobs
118 652210 77693 8394 0.47% 0.76% 0.78% 0 hpm counter proc
164 419 151 2774 0.47% 0.47% 0.12% 1 SSH Process
169 130398 15763 8272 0.15% 0.15% 0.15% 0 HQM Stack Proces
219 71188 516153 137 0.15% 0.09% 0.07% 0 Spanning Tree

[balaji.bandi · Accepted solution]

If you hit with memory leak, either you need to raise TAC case to replace.

or If you get a chance to upgrade to 12.2(55)SE12 - we have done some in 3 months back looks better and stable.

Some of them failed to come back due to memory leak, with the smartnet contract we have replaced.

Test before you roll out to all in production.

[q-le]

Thank you -much appreciated..

will upgrade to 12.2(55)SE12  as suggested.

[Leo Laohoo]

Peter,
The output shows very low CPU. I'm not concerned.

[q-le]

Hi Leo,

The fact is these 4 switches being running well for a year or more and suddenly all crashed and reboot

by themselves got us concern.

Maybe some changes in our network & infrastructure and we are still investigating.

For now , we will upgrade to 12.2(55)SE12 .

Thanks

[ysotocob]

For a complete analysis of this event we might need to get a show tech from the device and collect the crashfiles from the flash of each switch in the stack.

Upload the files and we can confirm if the problem is the memory leak or a software defect on the IOS in use.

Did you deploy any new service on the network or recent configuration on the stack ?