Hello Everyone,
Over the last 2 weeks, I've had 3 separate occurrences where a 2960X stack at a remote site goes down, and requires a reboot to restore connectivity to the switch and anything behind it. I was able to console into it this time when it went down to try and determine the issue.
Checking the processor status, this was my result:
# sh processes cpu sorted
CPU utilization for five seconds: 99%/0%; one minute: 99%; five minutes: 99%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
405 2616288 46089 56765 73.84% 73.89% 73.77% 0 HULC archive dnl
176 57618088 9238590 6236 9.10% 9.23% 9.21% 0 Hulc LED Process
137 1335105 371081 3597 0.29% 0.42% 0.41% 0 hpm counter proc
152 9709 35503 273 0.23% 0.17% 0.21% 0 HULC archive dnl
434 110 233 472 0.17% 0.09% 0.02% 1 Virtual Exec
133 272194 4254003 63 0.17% 0.09% 0.09% 0 hpm main process
192 783399 74048 10579 0.17% 0.17% 0.17% 0 HQM Stack Proces
237 177738 2663762 66 0.11% 0.12% 0.09% 0 Spanning Tree
167 43106 1834402 23 0.11% 0.01% 0.00% 0 Hulc Storm Contr
227 433169 756703 572 0.11% 0.04% 0.05% 0 IP Input
90 139449 17044065 8 0.11% 0.11% 0.11% 0 RedEarth Tx Mana
The stack's logs look completely normal though, nothing indicating any issues. Trying to type commands into the switch even over a console connection was painfully slow, and after a few commands, the switch just stopped responding to console input. Rebooted the stack and it instantly comes back up as if nothing was wrong.
Switch Models/Firmware running on the switch:
Switch Ports Model SW Version SW Image
------ ----- ----- ---------- ----------
1 54 WS-C2960X-48FPS-L 15.2(6)E2 C2960X-UNIVERSALK9-M
* 2 30 WS-C2960X-24PS-L 15.2(6)E2 C2960X-UNIVERSALK9-M
Can anyone explain to me what that process is for? I've seen issues where the HULC LED status will spike CPU usage a bit, but never this process. As of right now I'm planinng to upgrade the firmware tonight to eliminate a firmware bug being the issue, but I saw no open caveats relating to this in the release notes for the 15.2(6)E2 firmware.
Any input is appreciated.
