WLC9800 Clients disconnect often and get wrong gateway and 169 IP

[mauricio2099]

Hello Community.

Users are suddenly disconnected from wireless. They stop reaching network services such as Internet and other network services. Windows wireless notification shows laptop is still connected to the SSID but "No Internet, secured". At the same time from ipconfig the Wi-Fi Adapter received and auto-assigend IP 169.254.X.X and a wrong default gateway (which actually exists but on a different WLAN).

From the radioactive traces I found these logs

2022/05/23 15:42:40.116 client-orch-sm Client made a new Association to an AP/BSSID: BSSID 2cf8.2cf8.2cf8, old BSSID 0000.0000.0000, WLAN smtcwireless, Slot 1 AP 2cf8.2cf8.2cf0, AP001
Association received. BSSID 2cf8.2cf8.2cf8, old BSSID 0000.0000.0000, WLAN smtcwireless, Slot 1 AP 2cf8.2cf8.2cf0, AP001
2022/05/23 15:42:40.116 client-orch-state __unknown__ Client state transition: S_CO_INIT -> S_CO_ASSOCIATING
2022/05/23 15:42:40.116 dot11 Association success for client, assigned AID is: 4 Association success. AID 4, Roaming = False, WGB = False, 11r = False, 11w = True
2022/05/23 15:42:40.117 client-orch-state __unknown__ Client state transition: S_CO_ASSOCIATING -> S_CO_L2_AUTH_IN_PROGRESS
2022/05/23 15:42:40.117 sanet-shim-translate __unknown__ 3868.5555.4444 :Requested session policy with MAC not present with EPM
2022/05/23 15:42:40.117 client-auth __unknown__ ADD MOBILE sent. Client state flags: 0x71 BSSID: MAC: 2cf8.2cf8.2cf8 capwap IFID: 0x9000016b
2022/05/23 15:42:40.121 client-auth __unknown__ L2 Authentication initiated. method DOT1X, Policy VLAN 0,AAA override = 0 , NAC = 0
2022/05/23 15:42:40.122 ewlc-infra-evq __unknown__ Authentication Success. Resolved Policy bitmap:11 for client 3868.5555.4444
2022/05/23 15:42:40.348 client-auth Starting EAPOL 4-Way Handshake L2 Authentication Key Exchange Start. Resolved VLAN: 80, Audit Session id: 79040C0A00000AE7F18C14D0
2022/05/23 15:42:40.416 client-keymgmt Negotiated the following encryption mechanism: AKM:DOT1X Cipher:CCMP WPA2
EAP Key management successful. AKM:DOT1X Cipher:CCMP WPA2
2022/05/23 15:42:40.416 client-orch-sm __unknown__ Mobility discovery triggered. Client mode: Local
2022/05/23 15:42:40.416 client-orch-state Starting Mobility Anchor discovery for client Client state transition: S_CO_L2_AUTH_IN_PROGRESS -> S_CO_MOBILITY_DISCOVERY_IN_PROGRESS
2022/05/23 15:42:40.417 mm-client __unknown__ Mobility Successful. Roam Type None, Sub Roam Type MM_SUB_ROAM_TYPE_NONE, Previous BSSID MAC: 0000.0000.0000 Client IFID: 0xa000001e, Client Role: Local PoA: 0x9000016b PoP: 0x0
2022/05/23 15:42:40.417 client-auth __unknown__ ADD MOBILE sent. Client state flags: 0x72 BSSID: MAC: 2cf8.2cf8.2cf8 capwap IFID: 0x9000016b
2022/05/23 15:42:40.417 client-orch-state __unknown__ Client state transition: S_CO_MOBILITY_DISCOVERY_IN_PROGRESS -> S_CO_DPATH_PLUMB_IN_PROGRESS
2022/05/23 15:42:40.417 dot11 __unknown__ Client datapath entry params - ssid:smtcwireless,slot_id:1 bssid ifid: 0x90000171, radio_ifid: 0x90000169, wlan_ifid: 0xf0400050
2022/05/23 15:42:40.418 dpath_svc __unknown__ Client datapath entry created for ifid 0xa000001e
2022/05/23 15:42:40.418 client-orch-state Entering IP learn state Client state transition: S_CO_DPATH_PLUMB_IN_PROGRESS -> S_CO_IP_LEARN_IN_PROGRESS
2022/05/23 15:42:43.315 client-iplearn Client got IP: 10.10.80.9, discovered through: DHCP Client IP learn successful. Method: DHCP IP: 10.10.80.9
2022/05/23 15:42:43.316 client-orch-state Client reached RUN state, connection completed. Client state transition: S_CO_IP_LEARN_IN_PROGRESS -> S_CO_RUN
2022/05/23 16:12:43.588 client-keymgmt Negotiated the following encryption mechanism: AKM:DOT1X Cipher:CCMP WPA2
EAP Key management successful. AKM:DOT1X Cipher:CCMP WPA2
2022/05/23 16:12:43.588 client-auth __unknown__ ADD MOBILE sent. Client state flags: 0x72 BSSID: MAC: 2cf8.2cf8.2cf8 capwap IFID: 0x9000016b
2022/05/23 16:12:43.588 client-orch-state __unknown__ Client state transition: S_CO_RUN -> S_CO_RUN
2022/05/23 16:42:44.679 client-keymgmt Negotiated the following encryption mechanism: AKM:DOT1X Cipher:CCMP WPA2
EAP Key management successful. AKM:DOT1X Cipher:CCMP WPA2
2022/05/23 16:42:44.679 client-auth __unknown__ ADD MOBILE sent. Client state flags: 0x72 BSSID: MAC: 2cf8.2cf8.2cf8 capwap IFID: 0x9000016b
2022/05/23 16:42:44.679 client-orch-state __unknown__ Client state transition: S_CO_RUN -> S_CO_RUN
2022/05/23 17:12:45.799 client-keymgmt Negotiated the following encryption mechanism: AKM:DOT1X Cipher:CCMP WPA2 EAP Key management successful. AKM:DOT1X Cipher:CCMP WPA2
2022/05/23 17:12:45.799 client-auth __unknown__ ADD MOBILE sent. Client state flags: 0x72 BSSID: MAC: 2cf8.2cf8.2cf8 capwap IFID: 0x9000016b
2022/05/23 17:12:45.799 client-orch-state __unknown__ Client state transition: S_CO_RUN -> S_CO_RUN
2022/05/23 17:35:05.870 avc-stats __unknown__ Received stats record for app 'myapp'(app-id: 0xd0004bb), client MAC: 3868.5555.4444 , SSID 'corpwireless', direction egress (1), WLAN ID <not provided>, #bytes 109, #packets 1
2022/05/23 17:35:06.135 auth-mgr __unknown__ [0000.0000.0000:unknown] Session info 0x55fffca49ac8 hdl 0x49000b18 client hdl 0 cur hdl 0xd7000b18 withclient name BM
2022/05/23 17:35:06.135 sisf-packet __unknown__ RX: ARP from interface capwap_9000016b on vlan 80 Source MAC: 3868.5555.4444 Dest MAC: ffff.ffff.ffff ARP REQUEST, ARP sender MAC: 3868.5555.4444 ARP target MAC: 0000.0000.0000 ARP sender IP: 10.10.80.9, ARP target IP: 192.168.1.1,
2022/05/23 17:35:06.136 sisf-packet __unknown__ TX: ARP from interface capwap_9000016b on vlan 80 Source MAC: 3868.5555.4444 Dest MAC: ffff.ffff.ffff ARP REQUEST, ARP sender MAC: 3868.5555.4444 ARP target MAC: 0000.0000.0000 ARP sender IP: 10.10.80.9, ARP target IP: 192.168.1.1,
2022/05/23 17:35:06.953 auth-mgr __unknown__ [0000.0000.0000:unknown] Session info 0x55fffca49ac8 hdl 0x49000b18 client hdl 0 cur hdl 0xd7000b18 withclient name BM
2022/05/23 17:35:06.953 sisf-packet __unknown__ RX: ARP from interface capwap_9000016b on vlan 80 Source MAC: 3868.5555.4444 Dest MAC: ffff.ffff.ffff ARP REQUEST, ARP sender MAC: 3868.5555.4444 ARP target MAC: 0000.0000.0000 ARP sender IP: 10.10.80.9, ARP target IP: 192.168.1.1,
2022/05/23 17:35:06.953 sisf-packet __unknown__ TX: ARP from interface capwap_9000016b on vlan 80 Source MAC: 3868.5555.4444 Dest MAC: ffff.ffff.ffff ARP REQUEST, ARP sender MAC: 3868.5555.4444 ARP target MAC: 0000.0000.0000 ARP sender IP: 10.10.80.9, ARP target IP: 192.168.1.1,
2022/05/23 17:35:07.871 avc-stats __unknown__ Received stats record for app 'app2'(app-id: 0xd0004b8), client MAC: 3868.5555.4444 , SSID 'corpwireless', direction egress (1), WLAN ID <not provided>, #bytes 87, #packets 1
2022/05/23 17:35:07.959 auth-mgr __unknown__ [0000.0000.0000:unknown] Session info 0x55fffca49ac8 hdl 0x49000b18 client hdl 0 cur hdl 0xd7000b18 withclient name BM
2022/05/23 17:35:07.959 sisf-packet __unknown__ RX: ARP from interface capwap_9000016b on vlan 80 Source MAC: 3868.5555.4444 Dest MAC: ffff.ffff.ffff ARP REQUEST, ARP sender MAC: 3868.5555.4444 ARP target MAC: 0000.0000.0000 ARP sender IP: 10.10.80.9, ARP target IP: 192.168.1.1,
2022/05/23 17:35:07.960 sisf-packet __unknown__ TX: ARP from interface capwap_9000016b on vlan 80 Source MAC: 3868.5555.4444 Dest MAC: ffff.ffff.ffff ARP REQUEST, ARP sender MAC: 3868.5555.4444 ARP target MAC: 0000.0000.0000 ARP sender IP: 10.10.80.9, ARP target IP: 192.168.1.1,

*****From here tons of avc-status logs and nothing else

And I haven't seen this before until today there are a lot, lot of avc-status messages: for example: Received stats record for app 'myapp'(app-id: 0xd0004bb), client MAC: 3868.5555.4444 , SSID 'corpwireless', direction egress (1), WLAN ID <not provided>, #bytes 109, #packets 1

Not sure if this is informative only or if had to do.

I restarted the appliance and the APs last Friday night, but still seeing the problem.

From what I undertsand is tht the state S_CO_RUN means that the clent is fully connected, then some minutes later the ARP logs came in.

[Flavio Miranda]

Hi

 Whatever is the WLC problem, does not see anything on the log.  If the symptoms is client loosing IP address, you need to start investigating from there.

[mauricio2099]

Hi Flavio,

Looks like clients roam and sometimes during the process we can see the target IP is a wrong gateway: 192.168.1.1

ARP sender IP: 10.10.80.9, ARP target IP: 192.168.1.1 

clients get a 169.254.X.X IP and sounds like this is the reason of the disconnection

192.168.1.1 exists but is the gateway of a different WLAN. 

DHCP Server was checked already, plenty of IPs available to deliver on vlan 10.10.80.0/24

[JPavonM]

Unless anything could be happening with AP forwarding DHCP packets to the client, this could be due to client's wNIC firmware as manufacturers are dealing with operability/performance defects on them all time.

Check latest firmware from vendor URL (if it's Intel look here) or from OS vendor (this seems to me like a Win10 machine so go here for Intel ones), but also try to keep WLC code at the latest recommended as Cisco has found and fixed multiple defects related to this.

[mauricio2099]

The issue doesn't happen to all clients and not at the same time, I think this starts with a roaming, from the logs the connection starts with client-orch-sm Client made a new Association to an AP/ (someties is the same AP)

I am on the understood that when a client roams all the process of authentication and receive IP starts over.

The "IP lookup" is wrong some times, ARP Request is sent to another Gateway:

ARP sender IP: 10.10.80.9, ARP target IP: 192.168.1.1 and should be 10.10.80.1 instead.  192.168.1.1 is the gateway for another WLAN. Of course no changes were done to the WLC this just started happening. 

We had the theory that this is happening only to laptops that had upgraded to Win10 latest 21H2, and wanted to find a way to check. Have you heard of wireless issues with win10 clients 21H2 to WLC?

Unfortunately we don't have active support contract and can't upgrade to latest version

[ammahend]

Try disabling fast ssid change

[marce1000]

 - You can also have radioactive trace(s) analyzed by : https://cway.cisco.com/wireless-debug-analyzer/ . also always useful is to have an overall in depth check of the 9800-configuration with (CLI/SSH) : show tech wireless , feed the output into https://cway.cisco.com/tools/WirelessAnalyzer/

 M.

[mauricio2099]

The issue just started, the configuration on the WLC didn't changed.
I used the wireless debug analyzer, thank you. amazing tool, but still hard for me being so limited in wireless knowledge.

We had the theory that this is happening only to laptops that had upgraded to Win10 latest 21H2, and wanted to find a way to check. Have you heard of wireless issues with win10 clients 21H2 to WLC?

[JPavonM]

Are those APs deployed in Flexconnect mode?

Is this a remote site with local forwarding and central DHCP?

If the above is true, are you using somekind of VPN tecnology to connect with the remote site?

[mauricio2099]

Hello, They are flex connect and local to the WLC.

[mauricio2099]

At this point 16.12.1 is not supported. Smartnet was renewed and a case was opened. Cisco Support suggested to perform upgrade to latest recommended version today it is Amsterdam 17.03.05a. 
Basically mentioned, the problem could be added by the software version 16.12.1, upgrade may solve the issue. Other case captures have to be taken from the WLC and from the AP. But upgrade has to be the first step.

I found an issue upgrading from 16.12.1 to 17.03.05a
When choosing HTTPS from the GUI to transfer the image, the WLC didn't like a file greater than 1GB. https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvp25150
Neither TFTP worked... the connection times out, TFTP can be reached (Trusted TFTP that works. and could be PINGed by the WLC).

I upgraded first to 16.12.7 from GUI. Image file is less than 1GB

After this I could upgrade to 17.3.5.a + 2 SMUs that are required. Remember to activate and commit them (this is manual).

At this point we are on a "monitoring phase" trying to see if this fix the problem