Wired 802.1x not working on C9200L stack

[Simon Henley]

We have a strange issue with some new switches trying to authenticate wired users using 802.1x.

The new switches are 3 x C9200L-48PXG in a stack running IOS-XE 17.6.5

RADIUS is setup to use ISE 3.2, which is working with our existing switches.

The existing LAN switches are all standalone Catalyst 2960x or 2960c running IOS 15.2
Laptops are using Cisco Secure Client 5 for NAM, which works fine on the existing switches but not on the new ones.

The error in ISE shows as "5440 Endpoint abandoned EAP session and started new" so we've investigated many options on Cisco.com and Google, but so far nothing has helped.

We're using classic AAA configuration, but have also tried IBNS 2.0 with no luck.
When we use classic then the session gets authenticated but doesn't get authorised, using IBNS 2 the switch doesn't try sending the authc or authz to the radius it just seems to do this on the switch itself.

We've raised a ticket with our support provider, but still waiting for them to come up with an answer we haven't already tried!

Any suggestions appreciated.

[balaji.bandi]

Can you provide configuration on the Switch applied.

look at the even self explanatory :

Event 5440 Endpoint abandoned EAP session and started new

Failure Reason 5440 Endpoint abandoned EAP session and started new

Resolution Verify known NAD or supplicant issues and published bugs. Verify NAD and supplicant configuration.

Root cause Endpoint started new authentication while previous is still in progress. Most probable that supplicant on that endpoint stopped conducting the previous authentication and started the new one. Closing the previous authentication.

also worth looking this thread :

https://community.cisco.com/t5/network-access-control/dot1x-laptop-in-sh-auth-br-intermittent-showing-authorized-and/m-p/4095479

also check the troubleshooting tips :

https://www.ciscolive.com/c/dam/r/ciscolive/apjc/docs/2019/pdf/BRKSEC-3383.pdf

Also i would like to try different IOS XE version like 17.9.X see if that fixed (if all Cat 9200 running same version having issue, worth trying different version).

[Simon Henley]

Thanks Balaji

Unfortunately we've looked at those posts before and didn't help. 
Although we haven't tried 17.9.X, we did try 17.11.X which had the same issue.

The endpoints are connecting fine on the existing network, so assuming it is something to do with the new hardware/firmware that is causing our issue. Could be barking up the wrong tree of course.

Configuration of switch is as follows:

username switchradiuscheck password password

radius server DCPISESP102
address ipv4 10.1.11.105 auth-port 1812 acct-port 1813
automate-tester username switchradiuscheck ignore-acct-port probe-on
key passcode

radius server DCJISESP102
address ipv4 10.2.11.105 auth-port 1812 acct-port 1813
automate-tester username switchradiuscheck ignore-acct-port probe-on
key passcode

aaa group server radius ISE_RADIUS
server name DCPISESP102
server name DCJISESP102
ip radius source-interface vlan30

aaa new-model
aaa session-id common
aaa authentication dot1x default group ISE_RADIUS
aaa authorization network default group ISE_RADIUS
aaa accounting update newinfo periodic 1440
aaa accounting dot1x default start-stop group ISE_RADIUS

dot1x system-auth-control
dot1x auth-fail eapol
dot1x critical eapol
authentication mac-move permit

interface gi1/0/10
switchport access vlan 100
switchport mode access
switchport nonegotiate
authentication control-direction in
authentication event fail action next-method
authentication event server dead action authorize vlan 666
authentication event server alive action reinitialize
authentication order dot1x mab
authentication priority dot1x mab
authentication port-control auto
authentication periodic
authentication timer reauthenticate server
authentication violation restrict
mab
dot1x pae authenticator
dot1x timeout tx-period 10

radius-server attribute 6 on-for-login-auth
radius-server attribute 6 support-multiple
radius-server attribute 8 include-in-access-req
radius-server attribute 25 access-request include
radius-server attribute 31 mac format ietf upper-case
radius-server attribute 31 send nas-port-detail
radius-server dead-criteria time 5 tries 3
radius-server deadtime 10

[hslai]

@Simon Henley In case this is using EAP-TLS, it would worth trying password-based auth, such as PEAP MSCHAPv2. Maybe ISE is not getting the client certificates.

[Greg Gibbs]

If dot1x works with PEAP-MSCHAPv2 as suggested by @hslai, the issue is likely related to an MTU mismatch on the path between the switch and ISE and the fragmentation of the large packets needed for the certificate payload with EAP-TLS.

I suspect the Cat9200 switches use a default system MTU of 9000+, whereas the older switches likely use a system MTU of around 1500. You might try configuring the system MTU on the Cat9k to 1500 and see if that resolves the issue.

sw1(config)#system mtu 1500

If everything along the path between the switch and ISE PSN supports jumbo frames, ISE 3.2 also allows you to configure the interface for jumbo frames.

ise32-3/admin(config)#interface GigabitEthernet 0
ise32-3/admin(config-GigabitEthernet-0)#ip mtu ? 
Description: Choose MTU value to be set on IP interface
Possible completions:
<1280-9999> Recommended range VM:1280-9216;appliance:1280-9999

[Simon Henley]

Thanks for the suggestions.

PEAP did at least get denied by the default authorisation policy, but I didn't get enough time to setup the policy for a proper test. We're in the midst of moving office (where these switches are going) so had to do some work related to that!

We've looked at the path MTU previously and it's all good. I double checked it today and 1500 on the switch attack, all ports and links.

The machine certificate is  getting to ISE so not sure this is the issue unfortunately.

[Greg Gibbs]

Are you sure that all of the fragments are getting to the PSN? Confirming this typically involves comparing a packet capture on the client or switchport with a packet capture on the PSN to confirm all of the fragments are received.

You might also try using the Operations > Troubleshoot > Diagnostic Tools > Endpoint Debug tool to capture the details of the EAP-TLS session. If the PSN receives the full certificate and is able to reconstruct it, you will see the certificate as a separate file (*.cert extension) in that output.

I had a recent customer that had Palo Alto firewalls in the path that were dropping fragments and breaking the EAP-TLS handshake. Once it was confirmed that the PSN was not seeing some of the fragments, they had to do packet captures at every point in the path to find the culprit.

If all else fails, you may need to open a TAC case to investigate but they will likely need this same level of detail along the path.

[Simon Henley]

I tried using the endpoint debug but it didn't seem to work, showed zero packets for any client I tried. Help for that feature was hard to find but I read something about posture being needed for it, which we don't use.

I've collected packet capture on the client, access switch, and distribution switch. I also downloaded some logs off of ISE, although endpoint debug would have been better!

We have other switches which are working with our clients, and used 1 of these in the same distribution switch which worked fine.

We're currently waiting for tac , so appreciate all the suggestions from the community.

[Simon Henley · Accepted solution]

Afternoon All

The issue now appears to be solved after the switches were transferred to the new office.
No config was change between it not working before the move and working after being setup again.
Seems similar to this other thread, although we had already tried upgrading and rebooting:
https://community.cisco.com/t5/network-access-control/aaa-status-smd-platform-state-dead/td-p/4723871

Very strange and a bit disconcerting!