01-06-2020 10:43 AM
Hello Everyone,
Over the last 2 weeks, I've had 3 separate occurrences where a 2960X stack at a remote site goes down, and requires a reboot to restore connectivity to the switch and anything behind it. I was able to console into it this time when it went down to try and determine the issue.
Checking the processor status, this was my result:
# sh processes cpu sorted
CPU utilization for five seconds: 99%/0%; one minute: 99%; five minutes: 99%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
405 2616288 46089 56765 73.84% 73.89% 73.77% 0 HULC archive dnl
176 57618088 9238590 6236 9.10% 9.23% 9.21% 0 Hulc LED Process
137 1335105 371081 3597 0.29% 0.42% 0.41% 0 hpm counter proc
152 9709 35503 273 0.23% 0.17% 0.21% 0 HULC archive dnl
434 110 233 472 0.17% 0.09% 0.02% 1 Virtual Exec
133 272194 4254003 63 0.17% 0.09% 0.09% 0 hpm main process
192 783399 74048 10579 0.17% 0.17% 0.17% 0 HQM Stack Proces
237 177738 2663762 66 0.11% 0.12% 0.09% 0 Spanning Tree
167 43106 1834402 23 0.11% 0.01% 0.00% 0 Hulc Storm Contr
227 433169 756703 572 0.11% 0.04% 0.05% 0 IP Input
90 139449 17044065 8 0.11% 0.11% 0.11% 0 RedEarth Tx Mana
The stack's logs look completely normal though, nothing indicating any issues. Trying to type commands into the switch even over a console connection was painfully slow, and after a few commands, the switch just stopped responding to console input. Rebooted the stack and it instantly comes back up as if nothing was wrong.
Switch Models/Firmware running on the switch:
Switch Ports Model SW Version SW Image
------ ----- ----- ---------- ----------
1 54 WS-C2960X-48FPS-L 15.2(6)E2 C2960X-UNIVERSALK9-M
* 2 30 WS-C2960X-24PS-L 15.2(6)E2 C2960X-UNIVERSALK9-M
Can anyone explain to me what that process is for? I've seen issues where the HULC LED status will spike CPU usage a bit, but never this process. As of right now I'm planinng to upgrade the firmware tonight to eliminate a firmware bug being the issue, but I saw no open caveats relating to this in the release notes for the 15.2(6)E2 firmware.
Any input is appreciated.
01-06-2020 11:44 AM
Hi,
Are you by any chance archiving files or logs to the flash?
Is the archive command running?
HTH
01-06-2020 11:47 AM
Hey Reza,
Thanks for taking the time to reply. I am not. I do have some devices in my environment archiving change logs , but not in this case. The switch is sending logs to an external logging server though.
01-06-2020 01:22 PM
01-06-2020 01:28 PM
That's the plan, getting it done tonight after production hours. I didn't see a bug related to this listed in the current firmware's notes so I figured I'd ask around to ensure it's not a different issue.
01-06-2020 05:58 PM
Got it. I could not find any specific bug ID for this platform but most likely and as Leo said, an upgrade should fix the issue.
Good Luck!
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide