Re: High CPU Utilization In Cisco Switch 2960X - Version 15.0(2)EX5

Lee Win Neng · ‎05-03-2018

Hi all,

I have some question would like to check on the issue with high CPU utilization and causing the switch being freeze and unable to operate as expected. The switch with freeze issue had being resolved after rebooting. I try to trace back the log after reboot but unable to find any caused in the log. Also, I had found that the CPU usage was "CPU utilization for five seconds: 31%/0%; one minute: 33%; five minutes: 34%" in the switch(stack).

According to my research, Cisco Bug Toolkit (CSCtg86211) had mentioned that the C2960X switch with 12.2(50)SE03 or later releases had hit the bugs and no workaround or fixed release provided. I would like to know that my version had hit the bugs mentioned and any recommended version to upgrade to resolve this issue. The current version running on the C2960X switch was 15.0(2)EX5.

Thank you.

Leo Laohoo · ‎05-03-2018

Post the complete output to the command "sh proc cpu sort | ex 0.00".

Lee Win Neng · ‎05-03-2018

Hi, kindly refer the attached file for "show process cpu".

Leo Laohoo · ‎05-03-2018

Hulc LED Process

Have a look in the logs for any interface that is flapping.

NOTE: There are some people who loves to suppress interface logging. This is not helpful in events like this.

Lee Win Neng · ‎05-04-2018

I had tried to collect the log after the reboot but unfortunately the log only shows "dot1x" authentication as shown below.

Apr 24 04:10:01.351: %AUTHMGR-5-START: Starting 'dot1x' for client (8887.177e.9558) on Interface Gi2/0/3 AuditSessionID C0A8FA4A0000011630F153FB
Apr 24 04:10:03.291: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/3, changed state to up
Apr 24 04:10:04.294: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/3, changed state to up
Apr 24 04:10:21.496: %AUTHMGR-5-START: Starting 'dot1x' for client (b05a.dab3.bf16) on Interface Gi1/0/7 AuditSessionID C0A8FA4A0000011730F1A2E2
Apr 24 04:10:31.566: %DOT1X-5-FAIL: Authentication failed for client (8887.177e.9558) on Interface Gi2/0/3 AuditSessionID C0A8FA4A0000011630F153FB
Apr 24 04:10:31.566: %AUTHMGR-7-RESULT: Authentication result 'no-response' from 'dot1x' for client (8887.177e.9558) on Interface Gi2/0/3 AuditSessionID C0A8FA4A0000011630F153FB
Apr 24 04:10:31.566: %AUTHMGR-7-FAILOVER: Failing over from 'dot1x' for client (8887.177e.9558) on Interface Gi2/0/3 AuditSessionID C0A8FA4A0000011630F153FB
Apr 24 04:10:31.566: %AUTHMGR-5-START: Starting 'mab' for client (8887.177e.9558) on Interface Gi2/0/3 AuditSessionID C0A8FA4A0000011630F153FB
Apr 24 04:10:31.692: %MAB-5-SUCCESS: Authentication successful for client (8887.177e.9558) on Interface Gi2/0/3 AuditSessionID C0A8FA4A0000011630F153FB
Apr 24 04:10:31.692: %AUTHMGR-7-RESULT: Authentication result 'success' from 'mab' for client (8887.177e.9558) on Interface Gi2/0/3 AuditSessionID C0A8FA4A0000011630F153FB
Apr 24 04:10:32.559: %AUTHMGR-5-SUCCESS: Authorization succeeded for client (8887.177e.9558) on Interface Gi2/0/3 AuditSessionID C0A8FA4A0000011630F153FB
Apr 24 04:10:37.298: %AUTHMGR-5-START: Starting 'dot1x' for client (b05a.dab3.bf16) on Interface Gi1/0/7 AuditSessionID C0A8FA4A0000011830F1E09C
Apr 24 04:11:02.420: %AUTHMGR-5-START: Starting 'dot1x' for client (b05a.dab3.bf16) on Interface Gi1/0/7 AuditSessionID C0A8FA4A0000011930F242BC
Apr 24 04:11:16.580: %AUTHMGR-5-START: Starting 'dot1x' for client (b05a.dab3.bf16) on Interface Gi1/0/7 AuditSessionID C0A8FA4A0000011A30F27A0C
Apr 24 04:11:31.373: %AUTHMGR-5-START: Starting 'dot1x' for client (b05a.dab3.bf16) on Interface Gi1/0/7 AuditSessionID C0A8FA4A0000011B30F2B3D0
Apr 24 04:11:56.107: %AUTHMGR-5-START: Starting 'dot1x' for client (b05a.dab3.bf16) on Interface Gi1/0/7 AuditSessionID C0A8FA4A0000011C30F31470
Apr 24 04:12:09.543: %AUTHMGR-5-START: Starting 'dot1x' for client (b05a.dab3.bf16) on Interface Gi1/0/7 AuditSessionID C0A8FA4A0000011D30F348EC
Apr 24 04:12:34.783: %AUTHMGR-5-START: Starting 'dot1x' for client (b05a.dab3.bf16) on Interface Gi1/0/7 AuditSessionID C0A8FA4A0000011E30F3AB78
Apr 24 04:12:47.814: %AUTHMGR-5-START: Starting 'dot1x' for client (b05a.dab3.bf16) on Interface Gi1/0/7 AuditSessionID C0A8FA4A0000011F30F3DE69
Apr 24 04:13:01.645: %AUTHMGR-5-START: Starting 'dot1x' for client (b05a.dab3.bf16) on Interface Gi1/0/7 AuditSessionID C0A8FA4A0000012030F41470
Apr 24 04:13:26.305: %AUTHMGR-5-START: Starting 'dot1x' for client (b05a.dab3.bf16) on Interface Gi1/0/7 AuditSessionID C0A8FA4A0000012230F474C3
Apr 24 04:13:40.143: %AUTHMGR-5-START: Starting 'dot1x' for client (b05a.dab3.bf16) on Interface Gi1/0/7 AuditSessionID C0A8FA4A0000012330F4AAD1
Apr 24 04:14:07.257: %AUTHMGR-5-START: Starting 'dot1x' for client (b05a.dab3.bf16) on Interface Gi1/0/7 AuditSessionID C0A8FA4A0000012430F5149D
Apr 24 04:14:25.191: %AUTHMGR-5-START: Starting 'dot1x' for client (b05a.dab3.bf16) on Interface Gi1/0/7 AuditSessionID C0A8FA4A0000012530F55AC7
Apr 24 04:14:37.827: %AUTHMGR-5-START: Starting 'dot1x' for client (b05a.dab3.bf16) on Interface Gi1/0/7 AuditSessionID C0A8FA4A0000012630F58C23
Apr 24 04:15:02.462: %AUTHMGR-5-START: Starting 'dot1x' for client (b05a.dab3.bf16) on Interface Gi1/0/7 AuditSessionID C0A8FA4A0000012730F5EC5A
Apr 24 04:15:17.468: %AUTHMGR-5-START: Starting 'dot1x' for client (b05a.dab3.bf16) on Interface Gi1/0/7 AuditSessionID C0A8FA4A0000012830F626FA
Apr 24 04:15:42.309: %AUTHMGR-5-START: Starting 'dot1x' for client (b05a.dab3.bf16) on Interface Gi1/0/7 AuditSessionID C0A8FA4A0000012930F687F9

Leo Laohoo · ‎05-04-2018

Multiple failed DOT1X authentication can cause the same issue as well.

Rafael Carvallo · ‎05-04-2018

Hi,

This doesn't look like a high-cpu issue to me, at least not by the output you sent, check here: https://bst.cloudapps.cisco.com/bugsearch/bug/CSCtg86211/?rfs=iqvred the process someone commented above is expected to use that many cycles. It does much more than just controlling leds.

I've also checked some 2960Xs I have around, they all tend to have the CPU usage between 22 - 36%, none are facing issues at all.

When you say it "freezes" what exactly do you mean?

Lee Win Neng · ‎05-06-2018

The freezes that I mentioned was the switch was totally "hang" and impact to network connectivity which will require to restart the switch. After restarted, switch "hang" issue had been resolved and I try to check the log and cpu process which does not show any issue related. During the switch "hang", I try to insert the command through telnet as shown below.

L28ESLSF01#
L28ESLSF01#show run
L28ESLSF01#

But it can't display the show run after I type the command through telnet.