cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
1395
Views
3
Helpful
18
Replies

High CPU Utilization caused by 802.1x

davidssemanda
Level 1
Level 1

Dear Cisco Community ,Kindly help , I'm seeing high CPU utilization and as per the stats it shows its caused by 802.1x .

Pls find attached logs and advice on what is the best solution to counter this

CPU utilization for five seconds: 97%/6%; one minute: 96%; five minutes: 97%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
196 709393542 14556454 48734 75.17% 74.90% 75.38% 0 802.1x switch
99 42220786 27085578 1558 4.15% 4.00% 4.04% 0 cpf_process_tpQ
407 24707546 14525299 1701 2.55% 2.62% 2.61% 0 SpanTree Helper
130 448734212 2279073278 0 2.55% 2.62% 2.62% 0 ACL deny punt se

Switch is running Below Version

Cisco IOS XE Software, Version 16.12.11
Cisco IOS Software [Gibraltar], Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 16.12.11, RELEASE SOFTWARE (fc2).

 

Thanks 

1 Accepted Solution

Accepted Solutions

Leo Laohoo
Hall of Fame
Hall of Fame

IOS-XE, specifically 3650/3850, does not like flapping ports.  Ports constantly having 802.1x failure is also another.  

If that port is constantly going to do that, the process will blow out.  It will not take long before that port (and two other adjacent ports) will stop forwarding traffic (including PoE).  

In our case, I usually default the port to VLAN 1 (where it is not enabled).  

Alternately, cold-reboot can snap the ports back.  

View solution in original post

18 Replies 18

DanielP211
VIP Alumni
VIP Alumni

Hello!

Which switch is it? Can you see a port going up/down and not finishing the dot1x process?

BR

****Kindly rate all useful posts*****

Switch Model : 3850
Log Buffer (4096 bytes):
2816.a804.56fd) with reason (No Response from Client) on Interface Gi2/0/10 AuditSessionID AC11184300004344AFA1889D
*Dec 10 10:27:42.835: %SESSION_MGR-5-FAIL: Switch 1 R0/0: sessmgrd: Authorization failed or unapplied for client (2816.a804.56fd) on Interface GigabitEthernet2/0/10 AuditSessionID AC11184300004344AFA1889D. Failure reason: Authc fail. Authc failure reason: Cred Fail.
*Dec 10 10:28:18.308: %SYS-5-CONFIG_P: Configured programmatically by process EPM IAL Process from console as console
*Dec 10 10:28:27.537: %DOT1X-5-FAIL: Switch 1 R0/0: sessmgrd: Authentication failed for client (0004.5fa2.48a0) with reason (No Response from Client) on Interface Gi3/0/16 AuditSessionID AC1118430000381FA96EEAFC
*Dec 10 10:28:27.605: %SESSION_MGR-5-FAIL: Switch 1 R0/0: sessmgrd: Authorization failed or unapplied for client (0004.5fa2.48a0) on Interface GigabitEthernet3/0/16 AuditSessionID AC1118430000381FA96EEAFC. Failure reason: Authc fail. Authc failure reason: Cred Fail.
*Dec 10 10:28:46.035: %DOT1X-5-FAIL: Switch 1 R0/0: sessmgrd: Authentication failed for client (54b2.0300.f3b5) with reason (No Response from Client) on Interface Gi1/0/1 AuditSessionID AC11184300003D3DACECBF78
*Dec 10 10:28:46.096: %SESSION_MGR-5-FAIL: Switch 1 R0/0: sessmgrd: Authorization failed or unapplied for client (54b2.0300.f3b5) on Interface GigabitEthernet1/0/1 AuditSessionID AC11184300003D3DACECBF78. Failure reason: Authc fail. Authc failure reason: Cred Fail.
*Dec 10 10:28:57.835: %DOT1X-5-FAIL: Switch 1 R0/0: sessmgrd: Authentication failed for client (2816.a804.56fd) with reason (No Response from Client) on Interface Gi2/0/10 AuditSessionID AC11184300004344AFA1889D
*Dec 10 10:28:57.861: %SESSION_MGR-5-FAIL: Switch 1 R0/0: sessmgrd: Authorization failed or unapplied for client (2816.a804.56fd) on Interface GigabitEthernet2/0/10 AuditSessionID AC11184300004344AFA1889D. Failure reason: Authc fail. Authc failure reason: Cred Fail.
*Dec 10 10:29:11.843: %SYS-5-CONFIG_P: Configured programmatically by process EPM IAL Process from console as console
*Dec 10 10:29:42.608: %DOT1X-5-FAIL: Switch 1 R0/0: sessmgrd: Authentication failed for client (0004.5fa2.48a0) with reason (No Response from Client) on Interface Gi3/0/16 AuditSessionID AC1118430000381FA96EEAFC
*Dec 10 10:29:42.682: %SESSION_MGR-5-FAIL: Switch 1 R0/0: sessmgrd: Authorization failed or unapplied for client (0004.5fa2.48a0) on Interface GigabitEthernet3/0/16 AuditSessionID AC1118430000381FA96EEAFC. Failure reason: Authc fail. Authc failure reason: Cred Fail.
*Dec 10 10:30:01.098: %DOT1X-5-FAIL: Switch 1 R0/0: sessmgrd: Authentication failed for client (54b2.0300.f3b5) with reason (No Response from Client) on Interface Gi1/0/1 AuditSessionID AC11184300003D3DACECBF78
*Dec 10 10:30:01.160: %SESSION_MGR-5-FAIL: Switch 1 R0/0: sessmgrd: Authorization failed or unapplied for client (54b2.0300.f3b5) on Interface GigabitEthernet1/0/1 AuditSessionID AC11184300003D3DACECBF78. Failure reason: Authc fail. Authc failure reason: Cred Fail.
*Dec 10 10:30:10.839: %SYS-5-CONFIG_P: Configured programmatically by process EPM IAL Process from console as console
*Dec 10 10:30:12.863: %DOT1X-5-FAIL: Switch 1 R0/0: sessmgrd: Authentication failed for client (2816.a804.56fd) with reason (No Response from Client) on Interface Gi2/0/10 AuditSessionID AC11184300004344AFA1889D
*Dec 10 10:30:12.889: %SESSION_MGR-5-FAIL: Switch 1 R0/0: sessmgrd: Authorization failed or unapplied for client (2816.a804.56fd) on Interface GigabitEthernet2/0/10 AuditSessionID AC11184300004344AFA1889D. Failure reason: Authc fail. Authc failure reason: Cred Fail.

davidssemanda
Level 1
Level 1

hi Daniel,

Switch Model is 3850.

logs as seen from the switch 

Log Buffer (4096 bytes):
2816.a804.56fd) with reason (No Response from Client) on Interface Gi2/0/10 AuditSessionID AC11184300004344AFA1889D
*Dec 10 10:27:42.835: %SESSION_MGR-5-FAIL: Switch 1 R0/0: sessmgrd: Authorization failed or unapplied for client (2816.a804.56fd) on Interface GigabitEthernet2/0/10 AuditSessionID AC11184300004344AFA1889D. Failure reason: Authc fail. Authc failure reason: Cred Fail.
*Dec 10 10:28:18.308: %SYS-5-CONFIG_P: Configured programmatically by process EPM IAL Process from console as console
*Dec 10 10:28:27.537: %DOT1X-5-FAIL: Switch 1 R0/0: sessmgrd: Authentication failed for client (0004.5fa2.48a0) with reason (No Response from Client) on Interface Gi3/0/16 AuditSessionID AC1118430000381FA96EEAFC
*Dec 10 10:28:27.605: %SESSION_MGR-5-FAIL: Switch 1 R0/0: sessmgrd: Authorization failed or unapplied for client (0004.5fa2.48a0) on Interface GigabitEthernet3/0/16 AuditSessionID AC1118430000381FA96EEAFC. Failure reason: Authc fail. Authc failure reason: Cred Fail.
*Dec 10 10:28:46.035: %DOT1X-5-FAIL: Switch 1 R0/0: sessmgrd: Authentication failed for client (54b2.0300.f3b5) with reason (No Response from Client) on Interface Gi1/0/1 AuditSessionID AC11184300003D3DACECBF78
*Dec 10 10:28:46.096: %SESSION_MGR-5-FAIL: Switch 1 R0/0: sessmgrd: Authorization failed or unapplied for client (54b2.0300.f3b5) on Interface GigabitEthernet1/0/1 AuditSessionID AC11184300003D3DACECBF78. Failure reason: Authc fail. Authc failure reason: Cred Fail.
*Dec 10 10:28:57.835: %DOT1X-5-FAIL: Switch 1 R0/0: sessmgrd: Authentication failed for client (2816.a804.56fd) with reason (No Response from Client) on Interface Gi2/0/10 AuditSessionID AC11184300004344AFA1889D
*Dec 10 10:28:57.861: %SESSION_MGR-5-FAIL: Switch 1 R0/0: sessmgrd: Authorization failed or unapplied for client (2816.a804.56fd) on Interface GigabitEthernet2/0/10 AuditSessionID AC11184300004344AFA1889D. Failure reason: Authc fail. Authc failure reason: Cred Fail.
*Dec 10 10:29:11.843: %SYS-5-CONFIG_P: Configured programmatically by process EPM IAL Process from console as console
*Dec 10 10:29:42.608: %DOT1X-5-FAIL: Switch 1 R0/0: sessmgrd: Authentication failed for client (0004.5fa2.48a0) with reason (No Response from Client) on Interface Gi3/0/16 AuditSessionID AC1118430000381FA96EEAFC
*Dec 10 10:29:42.682: %SESSION_MGR-5-FAIL: Switch 1 R0/0: sessmgrd: Authorization failed or unapplied for client (0004.5fa2.48a0) on Interface GigabitEthernet3/0/16 AuditSessionID AC1118430000381FA96EEAFC. Failure reason: Authc fail. Authc failure reason: Cred Fail.
*Dec 10 10:30:01.098: %DOT1X-5-FAIL: Switch 1 R0/0: sessmgrd: Authentication failed for client (54b2.0300.f3b5) with reason (No Response from Client) on Interface Gi1/0/1 AuditSessionID AC11184300003D3DACECBF78
*Dec 10 10:30:01.160: %SESSION_MGR-5-FAIL: Switch 1 R0/0: sessmgrd: Authorization failed or unapplied for client (54b2.0300.f3b5) on Interface GigabitEthernet1/0/1 AuditSessionID AC11184300003D3DACECBF78. Failure reason: Authc fail. Authc failure reason: Cred Fail.
*Dec 10 10:30:10.839: %SYS-5-CONFIG_P: Configured programmatically by process EPM IAL Process from console as console
*Dec 10 10:30:12.863: %DOT1X-5-FAIL: Switch 1 R0/0: sessmgrd: Authentication failed for client (2816.a804.56fd) with reason (No Response from Client) on Interface Gi2/0/10 AuditSessionID AC11184300004344AFA1889D
*Dec 10 10:30:12.889: %SESSION_MGR-5-FAIL: Switch 1 R0/0: sessmgrd: Authorization failed or unapplied for client (2816.a804.56fd) on Interface GigabitEthernet2/0/10 AuditSessionID AC11184300004344AFA1889D. Failure reason: Authc fail. Authc failure reason: Cred Fail.

the SW always try authc user and it failed!!

the SW must try three timers before disable port.

can I see interface config of g3/0/16

MHM

 

interface GigabitEthernet3/0/16
switchport access vlan 90
switchport mode access
switchport port-security maximum 3
switchport port-security violation restrict
switchport port-security aging type inactivity
device-tracking attach-policy iptracking
authentication control-direction in
authentication event fail action next-method
authentication event server dead action reinitialize vlan 90
authentication event server dead action authorize voice
authentication event server alive action reinitialize
authentication host-mode multi-auth
authentication order dot1x mab
authentication priority dot1x mab
authentication port-control auto
authentication periodic
authentication timer reauthenticate server
authentication violation restrict
mab
snmp trap mac-notification change added
snmp trap mac-notification change removed
dot1x pae authenticator
dot1x timeout tx-period 5
storm-control broadcast level 20.00
storm-control multicast level 20.00
spanning-tree portfast
spanning-tree bpdufilter enable
spanning-tree bpduguard enable
ip dhcp snooping limit rate 100
end

Can i see also 

Show authc session interface g3/0/16 detail 

MHM

?

sh auth sessions int gi3/0/16 de
Interface: GigabitEthernet3/0/16
IIF-ID: 0x1DDBA4E6
MAC Address: 0004.5fa2.48a0
IPv6 Address: Unknown
IPv4 Address: Unknown
User-Name: 00045fa248a0
Status: Unauthorized
Domain: UNKNOWN
Oper host mode: multi-auth
Oper control dir: in
Session timeout: N/A
Restart timeout: 60s, Remaining: 18s
Common Session ID: AC1118430000381FA96EEAFC
Acct Session ID: Unknown
Handle: 0xc80006b7
Current Policy: POLICY_Gi3/0/16


Server Policies:


Method status list:
Method State
dot1x Stopped
mab Stopped

User not authc by both 802.1x and mab' 

But SW still try authc it' reason I think is 

""authentication event fail action next-method""

So remove this command from g3/0/16 and check if there is any log for this port anymore' if there is no anymore log for 802.1x of g3/0/16 then remove this command from all othet port 

MHM

M02@rt37
VIP
VIP

Hello @davidssemanda 

Your logs (thanks) reveal that high CPU utilization is primarily caused by 802.1X authentication failures. The errors, such as "No Response from Client" and "Cred Fail," suggest that the connected devices are either unresponsive to authentication requests or incapable of completing the process. This can result in repeated retries, overwhelming the CPU as it attempts to handle numerous authentication sessions...

The "No Response from Client" issue typically indicates that the connected device lacks a configured supplicant or does not support 802.1X. Meanwhile, "Cred Fail" points to authentication failures, possibly due to incorrect or missing credentials. These failures are likely exacerbated by frequent authentication retries or aggressive timers configured on the switch.

So, you should first verify the capabilities of the connected devices. For non-802.1X-capable devices, consider implementing MAB, which allows such devices to be authenticated based on their MAC address instead of 802.1X credentials. This can be configured alongside 802.1X for fallback scenarios. For example, you can set up MAB and 802.1X in a priority order so that the switch first tries 802.1X and falls back to MAB if necessary. Additionally, you can place unauthenticated devices in a critical VLAN to ensure network access for essential services while they await remediation. For devices that do support 802.1X, ensure that their supplicants are correctly configured, with valid credentials or certificates matching the authentication server's requirements.

Another important step is to optimize the authentication settings on the switch to reduce CPU strain. By default, 802.1X settings may aggressively retry failed authentications, contributing to high CPU usage. Adjusting parameters like the transmission period and maximum request count can mitigate this. For instance, increasing the retry interval with dot1x timeout tx-period 30  and limiting retries with dot1x max-req 3 can significantly reduce the authentication load. Additionally, extending the reauthentication timeout with dot1x timeout reauth-period 3600  ensures that successful clients are not repeatedly authenticated within short intervals, further easing the switch's workload.

Best regards
.ı|ı.ı|ı. If This Helps, Please Rate .ı|ı.ı|ı.

i Will try this approach n revert back once i see any changes on the switch 

Could you please share the output for:

show dot1x statistics

 As mentioned before, you can fine tune the dot1x timmings to get a better CPU usage and also reduce the fallback time to MAB:

Intarface xxx
 authentication event fail action next-method
 authentication order dot1x mab
 authentication priority dot1x mab
 authentication port-control auto
 dot1x pae authenticator
 dot1x timeout tx-period 7 
 dot1x timeout supp-timeout 5 
 dot1x max-reauth-req 3 
 dot1x timeout auth-period 10
 mab

Dot1x Global Statistics for
--------------------------------------------
RxStart = 14148 RxLogoff = 361 RxResp = 375114 RxRespID = 25353
RxReq = 0 RxInvalid = 0 RxLenErr = 0
RxTotal = 427587

TxStart = 0 TxLogoff = 0 TxResp = 0
TxReq = 398848 ReTxReq = 150 ReTxReqFail = 211
TxReqID = 58832 ReTxReqID = 18466 ReTxReqIDFail = 8504
TxTotal = 468172

Leo Laohoo
Hall of Fame
Hall of Fame

IOS-XE, specifically 3650/3850, does not like flapping ports.  Ports constantly having 802.1x failure is also another.  

If that port is constantly going to do that, the process will blow out.  It will not take long before that port (and two other adjacent ports) will stop forwarding traffic (including PoE).  

In our case, I usually default the port to VLAN 1 (where it is not enabled).  

Alternately, cold-reboot can snap the ports back.  

update us if issue happened again. 
thanks 

MHM