802.1x Certificate based failed authentications on AWS ISE Server

[Meenu Yadav]

Hello,

I have a distributed ISE deployment as below:

ISE 3.1 patch 6

PSN: 3 physical & 3 VM 

Admin: 1physical & 1VM

MnT: 1physical & 1VM

aaa group server radius ISERadius
server name ISE-VM
server name ISE-physical
ip radius source-interface xx

aaa server radius dynamic-author
client VM server-key 7 xxxx
client physical server-key 7 xxxx

When the radius server to pointed to physical ise server in radius server group, the cert based authentication works as expected but then pointed to aws ise server, it fails. The failure reason changes every time.

Failure reason: Authc fail. Authc failure reason: Cred Fail.

Authentication failed for client (abcd) with reason (No Response from Client) on Interface xx

Authentication failed for client (abcd) with reason (AAA Server Down) on Interface xx

The "Certificate Authentication Profile" is configured using option "Use Identity From" with Cert Attribute as "Subject-Common Name". As some machines are not part of AD so Identity Store option is not configured. I did use Cert Attribute as "Subject-Common Name - DNS" but no luck 

Both physical & VM ise servers have dedicated EAP certificate issued by digicert and their Root & Intermediate certs are present in "Trusted Certificates". The Intermediate cert is trusted for Infrastructure only. Since infrastructure trust is working on physical ise server, I am not sure if ISE VM needs "Trust for client authentication and Syslog" option as well.

I have ran packet captures where the ISE is sending CA chain but its never seen on the switch during radius challenge. The dot1x challenge runs for the configured timer & fails. Sometimes I don't see Authentication tab under ethernet properties & I restart the wired auto config just to manually trigger the authentication although the service is set to start automatically.

I really need to get this issue fixed at earliest and I would appreciate any assistance from experts here.

[MHM Cisco World]

Two ISE 

You need to add both ISE to SAN (multi SAN) or use wildcard cert.

MHM

[Meenu Yadav]

you mean add both admin nodes fqdn in PSN's SAN?

[Meenu Yadav]

you mean add both admin nodes fqdn in PSN's SAN?

[MHM Cisco World]

Both node fqdn in SAN of cert. 

Endpoint when try auth via any ISE it will validate cert. Since it fqdn is.list.

MHM

[Meenu Yadav]

endpoint with ise vm server supplicant >>> ise vm psn cert with Common Name = VM ise sever & SAN including physical & vm psn.

Will this still work after the physical server is shutdown and all the traffic failover to VM ise server only?

[Arne Bier]

EAP-TLS authentication can take the "subject" or "identity" being authenticated, either from the Subject CN, or the Subject Alternative Name. Contrary to https connections, the SAN is not mandatory or as important here. ISE will expect a non-empty Subject CN. And the ISE  Certificate Authentication Profile is the place where you define WHERE the identity should come from. For EAP-TLS to work, you must understand what your client certs look like - if the Subject CN contains a username/machine_name or whatever, that identifies the endpoint, then use the Subject CN - but it must be consistent across all clients.

And since Policy Set programming is the same for all PSNs, this logic will be applied to your on-prem PSNs, and your cloud PSNs.

As for the EAP System Certificate for each PSN, that should also be consistent - you don't need to purchase a separate Digicert cert for each PSN - you get one, and re-use that for all your PSNs (you can export a cert and its private key, and import that into all your PSN's that need an EAP System Cert)

In ISE Trusted Cert section, ensure that the entire CA cert chain that is involved in the ISE EAP System Cert (Digi Cert CA?) has these options set

The only other thing I can think of regarding cloud based PSN vs on-prem PSN is that the IP MTU might play a factor. With EAP-TLS, the RADIUS datagrams will mostly always exceed 1500 bytes during TLS negotiation, because the TLS client hello and server hello will contain the entire cert chain. ISE uses MTU 1500 bytes. If the PSN is connected to an L3 network that uses Jumbo frames then it will cause issue - not sure about AWS, but keep this in mind. Run a tcpdump on the cloud PSN and observe the RADIUS packets during a TLS negotiation. it's normal and expected to see UDP fragmentation to occur.  Some firewall in the past have also struggled with UDP fragmentation and re-assembly.

[Meenu Yadav]

The cloud psn is able to authenticate cert based wifi traffic but failing on all cert based LAN authentications. I will modify the usage Trust & test again

[Arne Bier]

One trick I remember doing once was to run ping tests with varying payload size. E.g. ping from the VLAN on which the switch management (RADIUS) runs, towards PSN using 1400, 1500, 1550 bytes. Etc and observe the results. But ensure that do not fragment is enabled in the ping command. If the ping succeeds beyond 1500 then it could be the problem. I have not seen this problem for a while, but I recall the issue I faced was resolved by setting the MTU on the L3 SVI to 1500 bytes (it was set to Jumbo Frames by mistake). But this topic is a bit nebulous in my own mind.

[Meenu Yadav]

ping host df-bit size 1400
Success rate is 100 percent (5/5), round-trip min/avg/max = 24/26/28 ms

ping host df-bit size 1500
Success rate is 0 percent (0/5)

[Meenu Yadav]

Hi Arne,

I resumed the troubleshooting today and have some snapshots captured. Please take a look at "on-prem" successful log and "euaws failed" log. It appears the switch is unable to reach ISE for 802.1x request on cloud psn.

I did try only MAB authentication on cloud psn and it works as expected.

[Arne Bier]

That's a great observation.

show aaa servers
show run | sec radius
show run | in aaa

Can you ping the VM ISE (if ICMP is permitted through your network)?

Another great testing tool is the command "test aaa" - specify your RADIUS group name, and then give it a dummy username and password - put the keyword 'new-code' at the end.  The response should be Auth Failed, but you can then check ISE Live Logs to see what the Access-Request looks like.

[Meenu Yadav]

I have enabled the trust for "authentication within ISE" & "client auth & syslog". MAC machines are now able to answer cert based auth on LAN but windows is still a challenge.

before enabling this trust:

    1]the on prem servers were authenticating endpoints based on cert and they continue to do after enabling the trust. 

    2] on aws servers, both  windows & mac machines were authenticating on wifi using certs ( they continue to work no as well)

    3] the aws ise servers were also authenticating user based authentication

the only challenge I had was on LAN which is now working on MAC but not on windows. I have verified the supplicant and ise logs still complain that supplicant stopped responding.

snapshot on aws server request on switch

I would appreciate any pointers to fix the windows issue

[Arne Bier]

If the MACOS is working then it seems it's not a switch or a network issue. Check the config on the Windows host - it seems like the Windows host does not trust the ISE EAP System Certificate - therefore, install the necessary CA Cert Chain into the PC's Trust Store (Enterprise Trust or Root CA Trust). The Windows host 802.1X supplicant most likely has the checkbox to "trust server cert" ticked - which is good - but in order for that to work, you need the CA cert chain that signed the ISE EAP System Cert. 

[MHM Cisco World]

When endpoint failed dis you check log detail in ISE 

Do you see error 5411?

Can yoh share the log detail 

MHM