Wireless users is suffering wireless disconnection randomly

[dawnccier]

WLC: Catalyst 9800-CL

OS Version: 16.12.4a

AP: C9115AXI-K

Symptom: Wireless users suffer wireless disconnection suddenly and randomly. Sometimes this kind of issue don't happen and works well. When this issue happened, the wifi icon at the windows bottom right corner is changed from 'connected WiFi' icon to 'Earth' icon even enable the 'Connect Automatically'. I have collected the debugging logs about this PC on the controller. There is an NAC server and the mac address of the PC has registered on NAC. The debugging logs file has been attached.

Does anyone know how to troubleshoot this issue? There is also a service impact to the customer.

[marce1000 · Accepted solution]

   - You will find the output for your DebugTrace from the wireless debug analyzer wright below , you can re-run that again , and select different flag (e.g.), especially show all can be useful too. Concerning the ip theft message check this guide : https://www.cisco.com/c/en/us/td/docs/wireless/controller/9800/16-12/config-guide/b_wl_16_12_cg/ip-theft.html

You may also have a sanity check of the controller configuration with : https://cway.cisco.com/tools/WirelessAnalyzer/

                           - Ref : https://cway.cisco.com/wireless-debug-analyzer/

TimeTaskTranslated

2021/10/12 00:08:37.780	client-orch-sm	Client made a new Association to an AP/BSSID: BSSID 34ed.1bdc.634d, old BSSID 0000.0000.0000, WLAN KT_ECSTA_SMP, Slot 1 AP 34ed.1bdc.6340, KT-HO16F-SAP09
2021/10/12 00:08:37.780	dot11	Association success for client, assigned AID is: 3
2021/10/12 00:08:37.795	client-keymgmt	Negotiated the following encryption mechanism: AKM:PSK Cipher:CCMP WPA2
2021/10/12 00:08:37.795	client-auth	Client successfully completed Pre-shared Key authentication. Assigned VLAN: 801
2021/10/12 00:08:37.795	client-orch-sm	Policy profile is configured for local switching
2021/10/12 00:08:37.795	client-orch-state	Starting Mobility Anchor discovery for client
2021/10/12 00:08:37.797	client-orch-state	Entering IP learn state
2021/10/12 00:10:34.555	client-orch-sm	Controller initiated client deletion with code: CO_CLIENT_DELETE_REASON_EXCLUDE_IP_THEFT. Code means: Client excluded due to IP theft

[dawnccier]

Hi, 

 Thanks for your reply to this issue. The IP theft feature is enabled on the customer controller and the wireless users get the IP address from the DHCP server. Is it possible that this issue happens due to the IP conflict ? But the IP assignment is controlled by the DHCP server and not possible assign the identical IP address to 2 different wireless useres.

I have tried to upload the file "show run".txt of controller to Wireless Config Analyzer Express. But there is no response coming out after uploading. Is there something wrong with that? 

[marce1000]

            > have tried to upload the file "show run".txt of controller to Wireless Config Analyzer Express. But there is no response coming out after uploading. Is there something wrong with that

  The facility for parsing configs from XE-based controllers is rather new. Make sure the provisioned output contains no 'more' prompts or try to save the configuration to an external repository with tftp , scp or ftp. Then upload that to the config-analyzer.

 M.

[dawnccier]

Hi,

Thanks for your advice. I have tried to collect logs of "show running-config" by an external TFTP server, but it is still no result after uploading the log file. So I tried another command 'show tech-support wireless' and the config analysis result came out after uploading. 

The "show running-config" is only for the AireOS-based controller. Is it recommended to disable IP Theft feature on the controller if there is an external DHCP server for the Wireless users?

[Jurgens L]

Do your sites by any chance have overlapping IP subnets with each other? In other words, you have remote sites with local internet breakout and they all use a static subnet since they don't get routed over your WAN.

[dawnccier]

Hi Jurgen,

Thanks for your reply. My customer has many global branches located world-wide connected by specialized tunnel. You mean there is an Ip subnet overlapping existing in customer network leading to disconnection according to IP theft.

Is it recommended to disable this IP theft on the controller？

[marce1000]

             >Is it recommended to disable this IP theft on the controller

 I would recommend to give it a try and see how the clients start behaving.

 M.

[Jurgens L]

If you have branches using the same ip subnet for end devices in remote branches, there is a limitation on 16.x. I've had a situation a year back where a client had a setup like this on AireOS WLC'S with no issues and when they were migrated to IOS-XE they ran into the IP theft issue.
Cisco advised me by disabling IP theft if wont fix the issue.

You have to upgrade to 17.3.3 or higher, where you will find in the flex connect profile a option to select for ip overlap.

[Arshad Safrulla · Accepted solution]

If you have overlapping IP's in different Flexconnect sites it is expected that the controller will identify this and mark as IP Theft. I wouldn't suggest disabling it as client exclusion provides a layer of security to WLC's in many ways.

My suggestion would be upgrade to 17.4.1 or higher, in this IOS-XE codes you can have overlapping client IP's across multiple sites without the client added to the exclusion blacklist. You need to enable this on the Flex profiles. 

https://www.cisco.com/c/en/us/td/docs/wireless/controller/9800/17-4/config-guide/b_wl_17_4_cg/m_vewlc_flex_connect.html#:~:text=Enabling%20Overlapping%20Client%20IP%20Address%20in%20Flex%20Deployment%20(GUI)

Remember this is supported 17.4.1 or higher codes only.

[dawnccier]

Hi Arshad,

Thanks for your suggestion. I will keep IP Theft feature enabled and have checked if there is IP subnet overlapping existing in customer service network. I found that there are many mac flapping logs display on the PoE switch as follwoing.

Example:

Sep 6 10:32:28 KST: %SW_MATM-4-MACFLAP_NOTIF: Host e0e6.2efb.281b in vlan 801 is flapping between port Gi1/0/1 and port Gi1/0/2

Sep 7 10:53:38 KST: %SW_MATM-4-MACFLAP_NOTIF: Host 9252.9724.04f3 in vlan 801 is flapping between port Gi1/0/30 and port Gi1/0/27
Sep 7 11:30:15 KST: %SW_MATM-4-MACFLAP_NOTIF: Host 6228.e1d7.f325 in vlan 801 is flapping between port Gi2/0/17 and port Gi1/0/18

<1> Is the flapping caused by the client roaming?

I also found that Host 9252.9724.04f3 is in the Excluded Clients list on the controller. The exclusion reason is IP Address Theft.

<2>Is it indicated that host 9252.9724.04f3 is excluded due to IP conflict? 

<3>What is the client preference on the controller? 

Wired > Wireless ?