CPU load 100% on cisco 2960s (not 2960-x) CLI unresponsive

[Ditter]

Hi to all, 

i searched the forum but did not find any relevant info about the 100% cpu utilization on cisco2960s.

CPU utilization for five seconds: 99%/10%; one minute: 99%; five minutes: 99%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
286 1569407 24500 64057 31.68% 43.53% 44.43% 0 HULC SISF Proces
156 531119 32679 16252 18.04% 16.70% 16.65% 0 Hulc LED Process
225 113709 60273 1886 3.30% 3.37% 3.41% 0 IP Host Track Pr
208 106302 17488 6078 3.00% 3.19% 3.18% 0 Spanning Tree
116 80225 29466 2722 2.30% 2.58% 2.56% 0 HLFM address lea

I see that these two processes consume a lot of cpu resources , i also upgraded to 152-2.E9.bin which is the latest MD software for this specific switch but nothing changed.

This switch has dhcp snooping enabled as well as around 37 voip phones (it is a 48 port switch in a two stack configuration).

Any ideas of how to fix the 100% cpu load?

Thanks a lot,

Ditter

[balaji.bandi]

Looks like Bug :

https://bst.cisco.com/bugsearch/bug/CSCtg86211

there is no workaround - see any latest IOS Code to upgrade, (i believe you already top version i guess and think) -

other suggestion shutdown any ports not used shutdown.

[Ditter]

Thanks for your reply. Yes 152-2.E9 is the latest MD , so  i do not have any other option than shutting down unused ports as you suggested.

[Bryson F]

Hello @Ditter , i had a some what similar issue in the past i had a friend help fix but i do remember one of the key things he did to fix it was clearing the log files from the switch, i would recommend trying to do that if possible in your situation. perhaps running a   Switch# install remove inactive : it will prompt you with files not being used and ask if you want to remove them before doing so so you wont lose anything if you change your mind half way through. If at all possible could you stack a second switch with it to help?

any ways i attached a similar article with a solution that may be able to help you it involves adding an acl to only allow traffic that the switch needs i believe so DHCP snooping is not looking every i may be wrong but here it is.

https://community.cisco.com/t5/switching/high-cpu-on-2960-x-due-to-sisf-processes-causing-management-loss/td-p/3749290

[balaji.bandi]

is this stack? how big the switch connection to other network? check you can shuffle the ports in different stack members, schedule and reboot if time permits.

if the business has a budget, move to Cat 9K switches.

[Ditter]

Thanks for the reply, yes it is a two member stack.  Currently, all users are on the first stack member and the second stack member is empty (i installed it for future expansions). 

All ports are gigabit but limited to 100 mbps and the uplink is gigabit.

maybe some voip users should be moved from the first stack member to the second one but it is not so easy to be done , i will consider it to check if something changes. 

[Bryson F]

@Ditter just a few things out of curiosity, how long has this switch been up, also can you do a #session 2 (takes you to switch two cli) then #show proc cpu sort so we can see how the other switch is doing as well. if it is not in bad shape i believe sharing the load could be very helpful for you. atleast short term help. but i do like @balaji.bandi suggestion to upgrade if possible. any extended cpu usage over 80% is something to avoid.

[Ditter]

Thanks Bryson, the  show proc cpu at the second stack member (which is empty of users for the time being) shows a cpu utilization of 23% / 2% percentage which is typical and off course not 100% as it is the first stack member.

sh proc cpu sorted
CPU utilization for five seconds: 23%/2%; one minute: 14%; five minutes: 13%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
4 1234427 66162 18657 9.29% 1.41% 1.19% 0 Check heaps
142 2041058 1410808 1446 2.19% 2.10% 2.10% 0 HULC DOT1X Proce

[Bryson F]

I hope that could be a good solution for you to split the load during non production hours. Just me personally since they are voip i would just duplicate the interface configs on the second switch and move them over. you might even be able to get away with doing it during production time. cisco IP phones don't take too long to come back up maybe 5 minutes on average. If the person in charge is fine with that i would recommend that so you don't lose any sleep or off time. i cant remember if a 2960 is POE or not but if not you wont have to wait for a full boot and there should not be any need to do anything in your CM/CUCM.

[Ditter]

Yes you are right, i will program the transition of half the users to the second switch to see if the load of the first stack member drops.

Yes this specific switch is PoE.

Thanks.

[Leo Laohoo]

---

@Ditter wrote:
286 1569407 24500 64057 31.68% 43.53% 44.43% 0 HULC SISF Proces
156 531119 32679 16252 18.04% 16.70% 16.65% 0 Hulc LED Process

---

HULC is STP reconvergence. 

As a test, disable all the ports and watch if the CPU process drop.  

If it drops, then it is caused by one (or more) ports going down/up continuously.