06-17-2016 03:27 PM - edited 02-21-2020 10:01 PM
For an offline or printed copy of this document, simply choose ⋮ Options > Printer Friendly Page. You may then Print, Print to PDF or copy and paste to any other document format you like. |
Table of Contents
An ISE deployment relies on multiple components. When authentication fails in the AAA environment, it may be challenging to find out root cause of the issue because you may need to look at different components.
With recent enhancements, Cisco has put effort into providing a single point of view for troubleshooting by correlating switch syslog events to internal ISE events, as well as by providing interfaces on the ISE to poll for different authentication- related information on demand. Other enhancements on the ISE include a configuration validator, a TCP dump utility, and the ability to provide details about supplicant issues when the client is running Cisco AnyConnect® Network Access Manager with a certificate-based EAP type.
High-level Troubleshooting Flow
show authentication sessions interface Gig x/y/z
aaa authorization network radius
show vlan
debug dot1x
If the interface is configured with the settings for order and timers that are recommended for Cisco TrustSec 2.1, it will take 30 seconds before the switch will accept and use the traffic from the endpoint to send a MAB request. This is typically not an issue for chatty devices, such as Windows PC devices; however, some printers may take a while to go through the MAB. If you are experiencing long delays to successfully MAB a device, like a printer, consider running the interface-specific command authentication control-direction in to allow traffic from the network to the endpoint prior to authentication, which could accelerate the MAB process.
show authentication sessions interface <int_name>
show ip access-list interface <int_name>
show running-config interface <int_name>
show access-list <int_name>
test aaa group radius {test_user} {test_password} new-code
ipconfig /all
ifconfig
As discussed earlier, the ISE Policy Administration Node (PAN) should be the first stop when troubleshooting authentication failures. Some failures will require additional diagnostic work at the NAD level. In most cases, the logs and debugs from the ISE and the NAD should be enough to determine the root cause of the problem.
The diagnostic work that can be performed on a supplicant is largely dependent on the troubleshooting tools that a particular supplicant provides. The native Windows supplicant has almost no debugging tools. Cisco AnyConnect Network Access Manager has a diagnostic and reporting tool (DART) that can be deployed to clients and used to generate a detailed report file. However, the report file is primarily for use by Cisco support staff and not generally recommended for the end user.
Sniffer traces provide vital troubleshooting information, but they are also of limited use on the end client. In general, using the Cisco Switched Port Analyzer (SPAN) to sniff the traffic at the switch is a more reliable and effective way to gather EAP packet traces.
Some of the common supplicant failures arise in situations where the client sends an EAPoL Start request, but fails to respond to an Identity Request message from the switch. Usually this happens because the supplicant is unable to find valid credentials. When the client “goes silent,” there is no way for the switch or Cisco ISE to understand the failure.
Unlike Windows native supplicants or other supplicants available on other operating systems, Cisco AnyConnect Network Access Manager includes an enhanced feature for notifying the ISE of the failure reason. As an example, take the situation in which the client is misconfigured and does not trust the ISE certificate in an EAP Transport Layer Security (EAP-TLS) or Protected EAP (PEAP) authentication.
Failed Authentication Report: Native Supplicants
Above, the Windows native supplicant was used on the PC. There are no details on the event other than Failure Reason: 5411 No response received during 120 seconds on last EAP message sent to the client. At this point, the administrator will have to log in to the affected endpoint to troubleshoot the issue.
Below is an example in which the PC is running Cisco AnyConnect Network Access Manager. The Failure Reason clearly indicates what the issue is on the supplicant settings.
Most of the information needed to troubleshoot Cisco TrustSec authentication issues can be gathered from the ISE itself. In some situations, however, the ISE cannot provide sufficient information to troubleshoot a failed authentication. It is therefore necessary to examine the troubleshooting capabilities of the NAD.
One of the most useful show commands on the Cisco Catalyst switch is show authentication sessions interface. The command output shows the current authentication status of the specified port. Other useful commands include show dot1x interface and show running-config interface.
Switch# show authentication sessions interface fastEthernet 0/1
Interface: FastEthernet0/1
MAC Address: 0016.d42e.e8ba
IP Address: 192.168.1.78
User-Name: winxp.example.com
Status: Authz Success
Domain: DATA
Security Policy: Should Secure
Security Status: Unsecure
Oper host mode: multi-domain
Oper control dir: both
Authorized By: Authentication Server
Vlan Policy: 100
Session timeout: N/A
Idle timeout: N/A
Common Session ID: C0A8013C000006679C3F253D
Acct Session ID: 0x00000C51
Handle: 0x68000667
Runnable methods list:
Method State
dot1x Authc Success
mab Not run
Switch#
Switch#
Switch#show dot1x interface fastEthernet 0/1
Dot1x Info for FastEthernet0/1
-----------------------------------
PAE = AUTHENTICATOR
PortControl = AUTO
ControlDirection = Both
HostMode = MULTI_DOMAIN
QuietPeriod = 60
ServerTimeout = 0
SuppTimeout = 30
ReAuthMax = 2
MaxReq = 2
TxPeriod = 10
Switch#
Switch#
Switch#show running-config interface fastEthernet 0/1
Building configuration...
Current configuration : 599 bytes
!
interface FastEthernet0/1
description 802.1x Enabled
switchport access vlan 2
switchport mode access
switchport voice vlan 110
authentication event fail action next-method
authentication event no-response action authorize vlan 100
authentication event server alive action reinitialize
authentication host-mode multi-domain
authentication port-control auto
authentication periodic
authentication timer reauthenticate server
authentication timer inactivity server
authentication violation restrict
mab
dot1x pae authenticator
dot1x timeout tx-period 10
spanning-tree portfast
end
Switch#
One of the most useful tools for debugging 802.1X failures on the authenticator is the Switched Port Analyzer (SPAN). SPAN allows you to mirror all the EAP traffic sent and received on one port to a different port where it can be analyzed by a sniffer. By sniffing the actual EAP packets that are exchanged between the authenticator and the client, you can diagnose some failures that are not visible from the Cisco ISE.
To configure a Cisco Catalyst 3000 Series Switch to mirror all the traffic from one port (the source port) to another (the destination port), use the following Cisco IOS commands in configuration mode:
(config)# monitor session 1 source interface Gigabit 0/1
(config)# monitor session 1 destination interface Gigabit 0/2 encapsulation replicate
To configure a Cisco Catalyst 4500 Series Switch to mirror all the traffic from one port (the source port) to another (the destination port), use the following Cisco IOS commands in configuration mode:
(config)# monitor session 1 source interface Gigabit 1/1
(config)# monitor session 1 destination interface Gigabit 1/2
No special configuration options are required to use SPAN on Layer 2 frames on the Cisco Catalyst 4500 Series switch, since the Cisco Catalyst 4500 monitors all Layer 2 frames with the default SPAN configuration shown above.
There are three common reasons why the switch does not or cannot send RADIUS messages to the AAA server when a client attempts to authenticate:
To verify network connectivity, ping the AAA server from the switch. Here is an example ping command:
Switch#
Switch#ping 192.168.1.60
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.1.60, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
Switch#
If the ping is not successful, or some packets are dropped, use standard routing and switching debugging techniques to establish reliable connectivity between the switch and the AAA server.
If the ISE PSN is pingable, it can useful to use the test aaa diagnostic command in this situation. The following example illustrates this command:
Switch#test aaa group radius testuser cisco123 new-code
User successfully authenticated
Switch#
The test aaa command causes the switch to send an Access Request to the AAA server for a PAP (clear-text) authentication for (in this example) the user testuser with password cisco123. The switch will attempt to authenticate to the server configured in the radius-server host command. Optionally, if you are using AAA groups instead of the default RADIUS group, you can specify a specific RADIUS group to test a specific server configured as part of the group.
If the result of the test aaa command is User successfully authenticated, as shown in the preceding code snippet, it means that three things are true: the switch is properly configured to communicate with the AAA server (correct shared key); the switch has network connectivity to the AAA server; and the username and password specified in the test command are valid. The ISE Live Authentication event will show this authentication:
Switch#test aaa group radius testuser cisco123 new-code
User rejected
Switch#
If the result of the test aaa command is User authentication request was rejected by server, you know that the switch configuration is working and network connectivity is validated, but the username and/or password provided in the test command are not valid. This failed authentication will show up in the ISE Live Authentication event. Another possibility is that the switch is not able to authenticate to the AAA server. Either the shared key does not match or there is no network connectivity to the AAA server. This could be the reason that the AAA server receives no RADIUS messages. Revalidating the configuration and/or verifying network connectivity will allow the switch to communicate with the AAA server during 802.1X authentications.
If the ISE Live Authentications shows successful authentication for the endpoint, but the result of show authentication sessions interface Gigabit x/y/z indicates that the port unauthorized, there may be policy mismatch between the ISE policy and the switch. This means although the ISE was able to authenticate and authorize the session, the attribute value pair (AVP) sent from the ISE to the NAD was invalid. Common reasons for this are:
If the AAA server has attempted to assign a VLAN that is not defined on the switch, the switch will not be able to authorize the port. In the following example, the AAA server tries to assign the VLAN named EMPLOYEE. The switch returns the follow syslog message:
Switch#
May 28 07:06:11.156 UTC: %AUTHMGR-5-START: Starting 'dot1x' for client (0016.d42e.e8ba) on Interface Fa0/1 AuditSessionID C0A8013C0000066D9D16ABF7
May 28 07:06:11.592 UTC: %DOT1X-5-SUCCESS: Authentication successful for client (0016.d42e.e8ba) on Interface Fa0/1 AuditSessionID
May 28 07:06:11.592 UTC: %AUTHMGR-7-RESULT: Authentication result 'success' from 'dot1x' for client (0016.d42e.e8ba) on Interface Fa0/1 AuditSessionID C0A8013C0000066D9D16ABF7
May 28 07:06:11.592 UTC: %DOT1X_SWITCH-5-ERR_VLAN_NOT_FOUND: Attempt to assign non-existent or shutdown VLAN EMPLOYEE to 802.1x port FastEthernet0/1 AuditSessionID C0A8013C0000066D9D16ABF7
May 28 07:06:11.592 UTC: %AUTHMGR-5-FAIL: Authorization failed for client (0016.d42e.e8ba) on Interface Fa0/1 AuditSessionID C0A8013C0000066D9D16ABF7
May 28 07:06:11.592 UTC: %EPM-6-POLICY_REQ: IP 0.0.0.0| MAC 0016.d42e.e8ba| AuditSessionID C0A8013C0000066D9D16ABF7| AUTHTYPE DOT1X| EVENT APPLY
May 28 07:06:11.592 UTC: %EPM-6-IPEVENT: IP 0.0.0.0| MAC 0016.d42e.e8ba| AuditSessionID C0A8013C0000066D9D16ABF7| AUTHTYPE DOT1X| EVENT IP-WAIT
May 28 07:06:11.592 UTC: %EPM-6-POLICY_REQ: IP 0.0.0.0| MAC 0016.d42e.e8ba| AuditSessionID C0A8013C0000066D9D16ABF7| AUTHTYPE DOT1X| EVENT REMOVE
May 28 07:06:11.592 UTC: %DOT1X-5-RESULT_OVERRIDE: Authentication result overridden for client (0016.d42e.e8ba) on Interface Fa0/1 AuditSessionID C0A8013C0000066D9D16ABF7
Switch#
As you can see from the following output, the switch’s employee VLAN is named EMP, not EMPLOYEE:
Switch#sh vlan | i EMP
100 EMP active Fa0/15, Fa0/16, Fa0/17, Fa0/18, Fa0/19, Fa0/20, Fa0/21, Fa0/22,
Because the switch does not have an exact match for the VLAN name EMPLOYEE, it sends an EAP-Failure message to the endpoint. To remedy this problem, either rename the VLAN on the switch or define the correct name in the ISE authorization profile.
In the following example, the dACL uses a wrong syntax. ISE sent allow ip any any instead of permit ip any any.
Switch#
May 28 07:11:59.395 UTC: %AUTHMGR-5-START: Starting 'dot1x' for client (0016.d42e.e8ba) on Interface Fa0/1 AuditSessionID C0A8013C000006719D1BFAB1
May 28 07:11:59.815 UTC: %DOT1X-5-SUCCESS: Authentication successful for client (0016.d42e.e8ba) on Interface Fa0/1 AuditSessionID
May 28 07:11:59.815 UTC: %AUTHMGR-7-RESULT: Authentication result 'success' from 'dot1x' for client (0016.d42e.e8ba) on Interface Fa0/1 AuditSessionID C0A8013C000006719D1BFAB1
May 28 07:11:59.823 UTC: %EPM-6-POLICY_REQ: IP 0.0.0.0| MAC 0016.d42e.e8ba| AuditSessionID C0A8013C000006719D1BFAB1| AUTHTYPE DOT1X| EVENT APPLY
May 28 07:11:59.823 UTC: %EPM-6-AUTH_ACL: POLICY Auth-Default-ACL| EVENT Auth-Default-ACL Attached Successfully
May 28 07:11:59.823 UTC: %EPM-6-AAA: POLICY xACSACLx-IP-PERMIT_ALL_TRAFFIC-4fc368f7| EVENT DOWNLOAD-REQUEST
May 28 07:11:59.840 UTC: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan2, changed state to up
May 28 07:11:59.890 UTC: %EPM-6-AAA: POLICY xACSACLx-IP-PERMIT_ALL_TRAFFIC-4fc368f7| EVENT DOWNLOAD-FAIL
May 28 07:11:59.890 UTC: %EPM-4-POLICY_APP_FAILURE: IP 0.0.0.0| MAC 0016.d42e.e8ba| AuditSessionID C0A8013C000006719D1BFAB1| AUTHTYPE DOT1X| POLICY_TYPE dACL| POLICY_NAME xACSACLx-IP-PERMIT_ALL_TRAFFIC-4fc368f7| RESULT FAILURE| REASON AAA download failure
May 28 07:11:59.890 UTC: %EPM-6-IPEVENT: IP 0.0.0.0| MAC 0016.d42e.e8ba| AuditSessionID C0A8013C000006719D1BFAB1| AUTHTYPE DOT1X| EVENT IP-WAIT
May 28 07:11:59.890 UTC: %AUTHMGR-5-FAIL: Authorization failed for client (0016.d42e.e8ba) on Interface Fa0/1 AuditSessionID C0A8013C000006719D1BFAB1
May 28 07:11:59.890 UTC: %DOT1X-5-RESULT_OVERRIDE: Authentication result overridden for client (0016.d42e.e8ba) on Interface Fa0/1 AuditSessionID C0A8013C000006719D1BFAB1
May 28 07:11:59.890 UTC: %EPM-6-POLICY_REQ: IP 0.0.0.0| MAC 0016.d42e.e8ba| AuditSessionID C0A8013C000006719D1BFAB1| AUTHTYPE DOT1X| EVENT REMOVE
May 28 07:11:59.899 UTC: %EPM-6-AUTH_ACL: POLICY Auth-Default-ACL| EVENT DETACH-SUCCESS
May 28 07:11:59.899 UTC: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan2, changed state to down
May 28 07:12:00.846 UTC: %AUTHMGR-5-SUCCESS: Authorization succeeded for client (0016.d42e.e8ba) on Interface Fa0/1 AuditSessionID C0A8013C000006719D1BFAB1
Switch#
Because the switch was not able to process the dACL, it sends an EAP-Failure response to the endpoint. To remedy this problem, correct the syntax error on the dACL on ISE, as follows:
Switch# show authentication sessions interface FastEthernet 0/1
Interface: FastEthernet0/1
MAC Address: 0016.d42e.e8ba
IP Address: 192.168.2.100
User-Name: winxp.example.com
Status: Authz Failed
Domain: DATA
Security Policy: Should Secure
Security Status: Unsecure
Oper host mode: multi-domain
Oper control dir: both
Authorized By: Authentication Server
Vlan Group: N/A
Session timeout: N/A
Idle timeout: N/A
Common Session ID: C0A8013C000006719D1BFAB1
Acct Session ID: 0x00000C5D
Handle: 0xB2000671
Runnable methods list:
Method State
dot1x Authc Success
mab Not run
Switch#
Before looking at the symptoms and causes of specific failures, it is instructive to review what a successful authentication looks like from the ISE perspective. This section will also serve to review the tools that can be used to troubleshoot authentication failures.
The RADIUS Live Logs in ISE lists all the authentications that have reached ISE. If there is no entry for the user in this screen, the authentication request has not been received by ISE.
You can look at the RADIUS Live Logs by logging in to ISE primary PAN and going to Operations > RADIUS > Live Logs. Doing so will bring up a screen similar to the one shown in Figure 11.
Note: The Live Logs screen is provided by Primary MnT node. The same information is also available on backup MnT node. Access to the Live Logs is also available by logging in to the secondary PAN and also logging in directly to either MnT node.
The RADIUS Live Logs has several important pieces of information that are critical to determining who is on the network, when and where they connected, and how they were authenticated.
The RADIUS Live Logs viewer
Note: Some of the columns listed described here are visible only by using the Add/Remove Columns feature. To make these columns visible, right-click on the header row.
Column | Description |
Time | Shows the time that the log was received by the collection agent. This column is required and cannot be deselected. |
Status | Shows if the authentication was successful or failed. This column is required and cannot be deselected. |
Details | Brings up a report when you click the magnifying glass icon, allowing you to drill down to view more detailed information on the selected authentication scenario. This column is required and cannot be deselected. |
Repeat Count |
The number of times the same message was received. |
Identity | Shows the username that is associated with the authentication. |
Endpoint ID | Shows the unique identifier for an endpoint, usually a MAC or IP address. |
Endpoint Profile |
The matching endpoint profile for this endpoint. |
Authentication Policy |
Which Policy Set and Authentication Policy Rule was matched |
Authorization Policy |
Which Policy Set and Authorization Policy Rule was matched |
Authorization Profiles | Shows an authorization profile that was applied based on the Authorization Policy. |
IP Address | Shows the IP address of the endpoint device. |
Network Device | Shows the IP address of the network access device. |
Device Port | Shows the port number at which the endpoint is connected. |
Identity Group | Shows the identity group that is assigned to the user or endpoint, for which the log was generated. |
Posture Status | Shows the status of the posture validation and details on the authentication. |
Server | Indicates the policy service node (PSN) from which the log was generated. |
MDM Server Name |
Name of the MDM used, if any. |
Optional |
You may choose to show the following fields using the ⚙ dropdown option |
Event | Shows the event status. |
Failure Reason | Shows a detailed reason for failure, if the authentication failed. |
Auth Method | Shows the authentication method that is used by the RADIUS protocol, such as Microsoft Challenge Handshake Authentication Protocol version 2 (MSCHAPv2), IEE 802.1x, or dot1x, and so on. |
Authentication Protocol | Shows the authentication protocol used, such as Protected Extensible Authentication Protocol (PEAP), Extensible Authentication Protocol (EAP), and the like. |
Security Group | Shows the group that is identified by the authentication log. |
Session ID | Shows the session ID. |
ISE can show authentication details showing a successful authentication of a machine using EAP-TLS.
The Authentication Summary shows the information that was available when viewed in the RADIUS Live Logs page:
The Related Events come from the syslog for the NAD that is relevant to this session. This is automatically correlated and included in the detailed report when the NAD sends the event to ISE MnT node.
To configure the switch to send the syslog to ISE, enter the following:
(config)# logging host {Primary_MnT} transport udp port 20514
(config)# logging host {Backup_MnT} transport udp port 20514
In the figure below, the Authentication Details section shows other information produced during authentication:
The Steps section shows the detailed process that the session went through within ISE:
If the event happened more than 24 hours ago, it’s a historical event can be viewed by going to Operations Reports Catalog AAA Protocol RADIUS Authentication.
You can use the diagnostic tool to evaluate the configuration of a network device and identify any configuration problems. The Expert Troubleshooter compares the configuration of the device with the standard configuration. This shows the Evaluate Configuration Validator options:
The TCP Dump utility monitors the contents of packets on a network interface that match a given Boolean expression. You can use this utility to troubleshoot problems on your network. Cisco ISE troubleshooting diagnostic tools provide an intuitive user interface.
TrustSec authentications can fail for many reasons. These include an unknown user, bad credentials, expired credentials, missing certificates, misconfiguration, and so on. Many of these failures can be diagnosed using careful examination of the ISE logs. Common failures and their symptoms are explained below.
Applies to | All EAP types |
---|---|
Possible Causes |
NAD or supplicant: Timeout for EAP may be too aggressive. Supplicant: Configured with certificate base authentication and the supplicant either does not have valid credentials or does not trust ISE certificate. Supplicant and user: Configured with password-based authentication and the user did not provide valid credentials. |
Resolution | Verify that supplicant is configured properly to conduct a full EAP conversation with ISE. Verify that NAS is configured properly to transfer EAP messages to or from supplicant. Verify that supplicant or network access server (NAS) does not have a short timeout for EAP conversations. Check the network that connects the NAS to ISE. If the external ID store is used for the authentication, it may be not responding fast enough for current timeouts. |
Applies to | All EAP types and MAB |
---|---|
Possible Causes | NAD is not configured with change of authorization (CoA) from ISE PSN. |
Resolution | Check the connectivity between ISE and the NAD. Ensure that ISE is defined as the dynamic authorization client on NAD and that CoA is supported on device. |
Applies to | All EAP types and MAB |
---|---|
Possible Causes | NAD may not be in the network device list on ISE |
Resolution | Verify whether the network device or AAA client is configured in Administration > Network Resources > Network Devices. |
Applies to | All EAP types and MAB |
---|---|
Possible Causes | The shared RADIUS key does not match between ISE and NAD |
Resolution | Check whether the shared secrets on the AAA client and ISE server match. Ensure that the AAA client and the network device have no hardware problems or problems with RADIUS compatibility. Also ensure that the network that connects the device to the ISE has no hardware problems. |
Applies to | EAP-TLS (AnyConnect Network Access Manager) |
---|---|
Possible Causes | The supplicant does not trust the ISE PSN certificate. |
Resolution | Check whether the proper server certificate is installed and configured for EAP by going to the Local Certificates page (Administration > System > Certificates > Local Certificates ). Also ensure that the certificate authority that signed this server certificate is correctly installed in client's supplicant. Check the previous steps in the log for this EAP-TLS conversation for a message indicating why the handshake failed. Check OpenSSLErrorMessage and OpenSSLErrorStack for more information. |
Applies to | All EAP types |
---|---|
Possible Causes | The default AuthZ rule is to deny access, and there are no specific AuthZ rules for this session. |
Resolution | The authorization profile with the ACCESS_REJECT attribute was selected as a result of the matching authorization rule. Check the appropriate authorization policy rule-results. |
Applies to | Password-based EAP types |
---|---|
Possible Causes |
Check the password of the user in internal identity store. The shared RADIUS key does not match between ISE and NAD. |
Resolution | Check the user credentials and device shared secret in Administration > Network Resources > Network Devices. |
Applies to | EAP-TLS, PEAP-TLS |
---|---|
Possible Causes | ISE authentication policy is configured for password-based authentication, but the supplicant is sending certificate credentials. |
Resolution | Check the appropriate configuration in Policy > Authentication. This error happens when the identity source is configured for certificate-based authentication and received a password based authentication request. |
Applies to | EAP-FAST, PEAP-MSCHAPv2 |
---|---|
Possible Causes | ISE authentication policy is configured for certificate-based authentication, but the supplicant is sending password-based credentials. |
Resolution | Check the appropriate configuration in Policy > Authentication. This error happens when the identity source is configured for password-based authentication and received a certificate-based authentication request. |
Applies to | EAP-FAST, PEAP-MSCHAPv2, MAB |
---|---|
Possible Causes | User or device was not found in the configured identity store |
Resolution |
Check whether the subject is present in any one of the chosen identity stores. Note that some identity stores may have been skipped if they do not support the current authentication protocol. Make sure the authentication policy points to correct identity store. For authentication in a Microsoft Windows network with multiple domains, make sure that the supplicant is appending the domain suffix (For users: administrator@example.com, for machines: winxp.example.com). |
Applies to | EAP-FAST, PEAP-MSCHAPv2 |
---|---|
Possible Causes | User entered wrong password. |
Resolution | Check the user password credentials. If the RADIUS request is using PAP for authentication, also check the shared secret configured for the network device. |
Great!
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: