Cisco DNAC Client Health Trends - Co Client Connect Timeout (255)

[JordanJ]

Hi there,

I've got reports of users having devices 'drop off the network' and 'the network being very slow'. I've just been looking through the wireless health on DNAC. There seems to be a high number of clients failing 'AAA' for the reason 'Co Client Connect Timeout'. Any ideas what would be causing this?

We are running Cisco 1852 Wireless Access Points at this site with a 9800-CL Wireless Controller (17.3.5a) on a VM in our data centre. The APs are associating to the wireless controller over FlexConnect. Our radius server is Cisco ISE (3.0, Patch 5) which is also running on a VM. The live logs don't currently show many issues. There are are a few auth timeouts (I've filtered the logs to show only this affected site) but you can see that these clients then associate in a few moments later:

Obviously the user is reporting a very generic response and we are trying to get more detail from the site including a known affected MAC address but could this be something that is causing an issue?

Cheers,

Jordan

[Leo Laohoo]

Target one of the wireless clients reportedly having a problem and post the complete output to this command:  

sh wireless client mac <MAC ADDRESS> detail

[ammahend]

From Ise logs can you also post the detail of one of clients failing, this will tell you delays at different stages authenticating through ISE. 

[JordanJ]

Hey Leo,

Here is the DNAC 'Event Viewer' for this client MAC - one that DNAC shows as affected by the failure reason "Co Client Connect Timeout (305)". Hopefully I haven't blocked out any detail that you need.

Here is the output from the WLC

Controller#show wireless client mac 2078.<CLIENT MAC ADDRESS> detail

Client MAC Address : 2078.<CLIENT MAC ADDRESS>
Client MAC Type : Universally Administered Address
Client IPv4 Address : 10.221.145.149
Client IPv6 Addresses : fe80::14a8:c580:663a:fe4b
Client Username : PEKI-ISE3
AP MAC Address : 188b.<AP MAC ADDRESS>
AP Name: <AP NAME>
AP slot : 0
Client State : Associated
Policy Profile : <POLICY PROFILE>
Flex Profile : <FLEX POLICY PROFILE>
Wireless LAN Id: 500
WLAN Profile Name: <WLAN PROFILE NAME>
Wireless LAN Network Name (SSID): <SSID NAME>
BSSID : 188b.<AP MAC ADDRESS>
Connected For : 3286 seconds
Protocol : 802.11n - 2.4 GHz
Channel : 6
Client IIF-ID : 0xa0001717
Association Id : 13
Authentication Algorithm : Open System
Idle state timeout : N/A
Re-Authentication Timeout : 28000 sec (Remaining time: 24714 sec)
Session Warning Time : Timer not running
Input Policy Name : platinum
Input Policy State : Installed
Input Policy Source : QOS Internal Policy
Output Policy Name : voice-client-avc
Output Policy State : Installed
Output Policy Source : QOS Internal Policy
WMM Support : Enabled
U-APSD Support : Enabled
U-APSD value : 0
APSD ACs : BK, BE, VI, VO
Fastlane Support : Enabled
Client Active State : Active
Power Save : ON
Current Rate : m7
Supported Rates : 9.0,18.0,24.0,36.0,48.0,54.0
Mobility:
Move Count : 0
Mobility Role : Local
Mobility Roam Type : None
Mobility Complete Timestamp : 08/01/2022 11:26:13 UTC
Client Join Time:
Join Time Of Client : 08/01/2022 11:26:13 UTC
Client State Servers : None
Client ACLs : None
Policy Manager State: Run
Last Policy Manager State : IP Learn Complete
Client Entry Create Time : 3286 seconds
Policy Type : WPA2
Encryption Cipher : CCMP (AES)
Authentication Key Management : 802.1x
User Defined (Private) Network : Disabled
User Defined (Private) Network Drop Unicast : Disabled
Encrypted Traffic Analytics : No
Protected Management Frame - 802.11w : No
EAP Type : EAP-TLS
VLAN Override after Webauth : No
VLAN : 605
Multicast VLAN : 0
WiFi Direct Capabilities:
WiFi Direct Capable : No
Central NAT : DISABLED
Session Manager:
Point of Attachment : capwap_9040020c
IIF ID : 0x9040020C
Authorized : TRUE
Session timeout : 28000
Common Session ID: 3DD1A8C000053D8751746125
Acct Session ID : 0x00000000
Last Tried Aaa Server Details:
Server IP : 192.168.209.34
Auth Method Status List
Method : Dot1x
SM State : AUTHENTICATED
SM Bend State : IDLE
Local Policies:
Service Template : wlan_svc_<POLICY PROFILE> (priority 254)
VLAN : 605
Absolute-Timer : 28000
Server Policies:
Resultant Policies:
VLAN Name : VLAN0605
VLAN : 605
Absolute-Timer : 28000
DNS Snooped IPv4 Addresses : None
DNS Snooped IPv6 Addresses : None
Client Capabilities
CF Pollable : Not implemented
CF Poll Request : Not implemented
Short Preamble : Not implemented
PBCC : Not implemented
Channel Agility : Not implemented
Listen Interval : 0
Fast BSS Transition Details :
Reassociation Timeout : 0
11v BSS Transition : Implemented
11v DMS Capable : No
QoS Map Capable : No
FlexConnect Data Switching : Local
FlexConnect Dhcp Status : Local
FlexConnect Authentication : Central
FlexConnect Central Association : Yes
Client Statistics:
Number of Bytes Received : 103599
Number of Bytes Sent : 202776
Number of Packets Received : 608
Number of Packets Sent : 676
Number of Policy Errors : 0
Radio Signal Strength Indicator : -45 dBm
Signal to Noise Ratio : 48 dB
Fabric status : Disabled
Radio Measurement Enabled Capabilities
Capabilities: Link Measurement, Passive Beacon Measurement, Active Beacon Measurement, AP Channel Report
Client Scan Report Time : Timer not running
Client Scan Reports
Assisted Roaming Neighbor List
Nearby AP Statistics:
EoGRE : Pending Classification
Device Type : Apple-Device
Device Name : APPLE, INC.
Protocol Map : 0x000001 (OUI)
Max Client Protocol Capability: 802.11n
Cellular Capability : N/A

Here is the ISE logs for this device for the past 24hrs (no failures for this client which is odd)

 Here is the latest ISE log entry for the device:

 

[JordanJ]

I did grab this output on the 1/8/22 but looks like the Cisco Community didn't actually post my reply. Sorry for the delay. Here is the details of one particular client having the issue. It doesn't look like there is a corresponding ISE log failure at the same time as DNA is reporting a failure. Could this mean the delay is before the device even reaches ISE PSN?

[Leo Laohoo]

Something does not add up here. 

1. If the wireless client is an Apple iPad, why is it negotiating to 2.4 Ghz (Ch 6)?  
2. The output from the WLC shows this particular wireless client has been connected to the SSID for the past 3286 seconds.  I do not see that as a sign of anything wrong. 
3. The wireless client is configured for a Platinum QoS.  I do not think that is correct.

[JordanJ]

Ok thanks. I'll try and get a MAC address from the user and see if that client shows something different.

With the Platinum QOS, I assume you're taking about this section?

Input Policy Name : platinum
Input Policy State : Installed
Input Policy Source : QOS Internal Policy
Output Policy Name : voice-client-avc
Output Policy State : Installed
Output Policy Source : QOS Internal Policy

What is the effect of these Input/Output Policy Source? Is this a QOS policy set somewhere in the WLC?

[Leo Laohoo]

---

@JordanJ wrote:
What is the effect of these Input/Output Policy Source? Is this a QOS policy set somewhere in the WLC?

---

Platinum for RTP-related traffic.  

Traffic from smartphones or tablets are really not "RTP" regardless if they are on a Facetime, Viber, WhatsApp, etc call. 

Remember, if the traffic is manually classed as Platinum they go into this "pool" and that pool is not deep enough for everyone.  

[JordanJ]

This is another client having a similar issue. Keeps dropping the device apparently. A few auth failures. Here is the ISE log. Looks like it's going back and forth from ISE PSN to the device a few times. This user is connecting to the network using PEAP - can't see this device doing a lookup in AD though.

	11001	Received RADIUS Access-Request
	11017	RADIUS created a new session
	15049	Evaluating Policy Group
	15008	Evaluating Service Selection Policy
	15048	Queried PIP - Airespace.Airespace-Wlan-Id
	11507	Extracted EAP-Response/Identity
	12300	Prepared EAP-Request proposing PEAP with challenge
	12625	Valid EAP-Key-Name attribute received
	11006	Returned RADIUS Access-Challenge
	11001	Received RADIUS Access-Request
	11018	RADIUS is re-using an existing session
	12302	Extracted EAP-Response containing PEAP challenge-response and accepting PEAP as negotiated
	12319	Successfully negotiated PEAP version 1
	12800	Extracted first TLS record; TLS handshake started
	12805	Extracted TLS ClientHello message
	12806	Prepared TLS ServerHello message
	12807	Prepared TLS Certificate message
	12808	Prepared TLS ServerKeyExchange message
	12810	Prepared TLS ServerDone message
	12811	Extracted TLS Certificate message containing client certificate
	12305	Prepared EAP-Request with another PEAP challenge
	11006	Returned RADIUS Access-Challenge
	11001	Received RADIUS Access-Request
	11018	RADIUS is re-using an existing session
	12304	Extracted EAP-Response containing PEAP challenge-response
	12305	Prepared EAP-Request with another PEAP challenge
	11006	Returned RADIUS Access-Challenge
	11001	Received RADIUS Access-Request
	11018	RADIUS is re-using an existing session
	12304	Extracted EAP-Response containing PEAP challenge-response
	12305	Prepared EAP-Request with another PEAP challenge
	11006	Returned RADIUS Access-Challenge
	11001	Received RADIUS Access-Request
	11018	RADIUS is re-using an existing session
	12304	Extracted EAP-Response containing PEAP challenge-response
	12305	Prepared EAP-Request with another PEAP challenge
	11006	Returned RADIUS Access-Challenge
	11001	Received RADIUS Access-Request
	11018	RADIUS is re-using an existing session
	12304	Extracted EAP-Response containing PEAP challenge-response
	12305	Prepared EAP-Request with another PEAP challenge
	11006	Returned RADIUS Access-Challenge
	5440	Endpoint abandoned EAP session and started new (Step latency=51585 ms)

[Leo Laohoo]

Let's try a simple experiment:  Disable WMM.

[JordanJ]

Should I set this option to 'enabled' or 'disabled'? (currently set to 'required' as you can see)

[Leo Laohoo]

Turn WMM off/disable.

[Rich R]

Joke of the day - your previous post was classified as SPAM because "IPv4 & IPv6 Addresses are present in the discussion (Eg. 10.221.X.X, fe80::14a8:X:X:X etc.)" so they blocked the entire discussion.  They seem to have unblocked it after I complained and pointed out it wasn't SPAM but they've still deleted your post!  I kid you not - that quote is verbatim from the forum admins.  How very dare you mention IPv4 and IPv6 addresses on the same post on a networking forum ROFL

[ammahend]

For a recent failed client, can you also share ‘show log profile wir filter mac <MAC address> to-file output.txt’

also share eap-timer values configured

in one log I saw Tls another I saw peap, so the issue is with both right ?

[JordanJ]

I thought there may have been something going on with the contents of my post, that's why I attached it as an image! Makes sense now though LOL