iOS devices lost network when switching from static IPs to DHCP - 9800-CL

[Mikolaj Moryto]

Hi,

Setup: 9800-CL + 2702i

I have been having issues in my home/lab setup with clients losing (only iOS apple devices) network after some time (usually less than 60 minutes). The only recent change I did was switching SSIDs to centralized DHCP and that's when it started.

Running Wireshark on the same VLAN where 9800's management interface connects to shows me that clients stop getting ARP responses. 9800-CL just stops passing them down the network. Bofore the issues starts happening I can see ARP request coming in encapsulated in CAPWAP and then being sent to the core switch without CAPWAP. Then I can see ARP reply as normal and being passed to the clients.

I have pulled Radioactive Trace for one of the clients and this is what I can see when everything is normal:

2020/02/09 12:38:55.023 {wncd_x_R0-0}{1}: [sisf-packet] [20681]: (debug): RX: ARP from interface capwap_90000003 on vlan 120 Source MAC: 4098.ad59.d36f  Dest MAC: 001b.90e5.a3c3  ARP REQUEST, ARP sender MAC: 4098.ad59.d36f  ARP target MAC: 0000.0000.0000  ARP sender IP: 192.168.120.5, ARP target IP: 192.168.120.254, 
2020/02/09 12:38:55.023 {wncd_x_R0-0}{1}: [sisf-packet] [20681]: (debug): TX: ARP from interface capwap_90000003 on vlan 120 Source MAC: 4098.ad59.d36f  Dest MAC: 001b.90e5.a3c3  ARP REQUEST, ARP sender MAC: 4098.ad59.d36f  ARP target MAC: 0000.0000.0000  ARP sender IP: 192.168.120.5, ARP target IP: 192.168.120.254, 
2020/02/09 12:38:55.025 {wncd_x_R0-0}{1}: [sisf-packet] [20681]: (debug): RX: ARP from interface Gi1 on vlan 120 Source MAC: 001b.90e5.a3c3  Dest MAC: 4098.ad59.d36f  ARP REPLY, ARP sender MAC: 001b.90e5.a3c3  ARP target MAC: 4098.ad59.d36f  ARP sender IP: 192.168.120.254, ARP target IP: 192.168.120.5, 
2020/02/09 12:38:55.025 {wncd_x_R0-0}{1}: [sisf-packet] [20681]: (debug): TX: ARP from interface Gi1 on vlan 120 Source MAC: 001b.90e5.a3c3  Dest MAC: 4098.ad59.d36f  ARP REPLY, ARP sender MAC: 001b.90e5.a3c3  ARP target MAC: 4098.ad59.d36f  ARP sender IP: 192.168.120.254, ARP target IP: 192.168.120.5,

Then what I see when it stops passing the traffic:

2020/02/09 12:40:23.353 {wncd_x_R0-0}{1}: [client-auth] [20681]: (info): MAC: 4098.ad59.d36f Client auth-interface state transition: S_AUTHIF_PSK_AUTH_DONE -> S_SANET_DELETED
2020/02/09 12:40:23.353 {wncd_x_R0-0}{1}: [client-orch-sm] [20681]: (info): MAC: 4098.ad59.d36f Deleting the client, reason: 24, CO_CLIENT_DELETE_REASON_SESSION_TIMEOUT, Client state S_CO_RUN
2020/02/09 12:40:23.353 {wncd_x_R0-0}{1}: [client-orch-sm] [20681]: (note): MAC: 4098.ad59.d36f Client delete initiated. Reason: CO_CLIENT_DELETE_REASON_SESSION_TIMEOUT, fsm-state transition 00|00|00|00|00|01|07|13|17|18|25|30|3f|40|42|44|49|4e|13|17|18|25|30|3f|40|42|44|48|53|55|64|7d|
2020/02/09 12:40:23.354 {wncd_x_R0-0}{1}: [client-orch-state] [20681]: (note): MAC: 4098.ad59.d36f Client state transition: S_CO_RUN -> S_CO_DELETE_IN_PROGRESS
2020/02/09 12:40:23.355 {wncd_x_R0-0}{1}: [client-orch-sm] [20681]: (ERR): MAC: 4098.ad59.d36f Failed to get resolved VLAN ID
2020/02/09 12:40:23.355 {wncd_x_R0-0}{1}: [multicast-main] [20681]: (info): MAC: 4098.ad59.d36f No Flex/Fabric main record exists for client
2020/02/09 12:40:23.355 {wncd_x_R0-0}{1}: [rog-proxy-capwap] [20681]: (debug): Managed client state notification: client left run state: 4098.ad59.d36f 
2020/02/09 12:40:23.370 {wncd_x_R0-0}{1}: [client-iplearn] [20681]: (info): MAC: 4098.ad59.d36f IP-learn state transition: S_IPLEARN_COMPLETE -> S_IPLEARN_DELETE_IN_PROGRESS
2020/02/09 12:40:23.371 {wncd_x_R0-0}{1}: [dpath_svc] [20681]: (note): MAC: 4098.ad59.d36f Client datapath entry deleted for ifid 0xa0000007
2020/02/09 12:40:23.371 {wncd_x_R0-0}{1}: [mm-transition] [20681]: (info): MAC: 4098.ad59.d36f MMIF FSM transition: S_MA_LOCAL -> S_MA_DELETE_PROCESSED_TR on E_MA_CO_DELETE_RCVD
2020/02/09 12:40:23.371 {wncd_x_R0-0}{1}: [mm-client] [20681]: (info): MAC: 4098.ad59.d36f Invalid transmitter ip in build client context
2020/02/09 12:40:23.372 {fman_fp_F0-0}{1}: [wireless-client] [22956]: UUID: 100000003bb69, ra: 5 (note): WLCLIENT 0xa0000007 AOM delete succeeded
2020/02/09 12:40:23.372 {mobilityd_R0-0}{1}: [mm-transition] [21997]: (info): MAC: 4098.ad59.d36f MMFSM transition: S_MC_RUN -> S_MC_HANDOFF_END_RCVD_TR on E_MC_HANDOFF_END_RCVD from WNCD[0]
2020/02/09 12:40:23.373 {wncd_x_R0-0}{1}: [client-auth] [20681]: (info): MAC: 4098.ad59.d36f Client auth-interface state transition: S_SANET_DELETED -> S_AUTHIF_DELETED
2020/02/09 12:40:23.373 {wncd_x_R0-0}{1}: [dot11] [20681]: (info): MAC: 4098.ad59.d36f Sent disassoc to client, disassoc reason: 252, CLIENT_DEAUTH_REASON_ADMIN_RESET delete reason: 24, CO_CLIENT_DELETE_REASON_SESSION_TIMEOUT.
2020/02/09 12:40:23.374 {wncd_x_R0-0}{1}: [dot11] [20681]: (info): MAC: 4098.ad59.d36f Sent deauth to client, deauth reason: 252, CLIENT_DEAUTH_REASON_ADMIN_RESET delete reason: 24, CO_CLIENT_DELETE_REASON_SESSION_TIMEOUT.
2020/02/09 12:40:23.374 {wncd_x_R0-0}{1}: [dot11] [20681]: (info): MAC: 4098.ad59.d36f DOT11 state transition: S_DOT11_ASSOCIATED -> S_DOT11_DELETED

and following that no ARP reply at all:

2020/02/09 12:41:24.542 {wncd_x_R0-0}{1}: [sisf-packet] [20681]: (debug): RX: ARP from interface capwap_90000003 on vlan 120 Source MAC: 4098.ad59.d36f Dest MAC: 001b.90e5.a3c3 ARP REQUEST, ARP sender MAC: 4098.ad59.d36f ARP target MAC: 001b.90e5.a3c3 ARP sender IP: 192.168.120.5, ARP target IP: 192.168.120.254, 
2020/02/09 12:41:25.105 {wncd_x_R0-0}{1}: [sisf-packet] [20681]: (debug): RX: ARP from interface capwap_90000003 on vlan 120 Source MAC: 4098.ad59.d36f Dest MAC: 001b.90e5.a3c3 ARP REQUEST, ARP sender MAC: 4098.ad59.d36f ARP target MAC: 001b.90e5.a3c3 ARP sender IP: 192.168.120.5, ARP target IP: 192.168.120.254, 
2020/02/09 12:41:27.117 {wncd_x_R0-0}{1}: [sisf-packet] [20681]: (debug): RX: ARP from interface capwap_90000003 on vlan 120 Source MAC: 4098.ad59.d36f Dest MAC: 001b.90e5.a3c3 ARP REQUEST, ARP sender MAC: 4098.ad59.d36f ARP target MAC: 001b.90e5.a3c3 ARP sender IP: 192.168.120.5, ARP target IP: 192.168.120.254,

I believe this is caused by some kind of a timeout hence clients de-associating and then 9800-CL having issues with re-associating them. I have been looking for an answer and based on what I found here I have increased the session timeout. Not sure if that is the right solution so any comments will be appreciated.

before:

session- timeout 1800

now:

session- timeout 3600

The interesting thing is this has been only happening on single SSID even despite the fact I enabled DHCP for all SSIDs the same way and wireless policy for all SSIDs looks identical.

Thanks

[Scott Fella]

Well it seems to me from process of elimination is that only one ssid is not functioning properly. Since the change was made to all others and all ssid as similar, my assumption from your statement, when not delete the ssid and create it again. If you can create the same policy maybe with a different name and it works, then something corrupted the other ssid. Create a new ssid and test, then delete the old and create a new.

[Mikolaj Moryto]

Hi Scott,

Thanks for the suggestion. I was thinking of that but it happened to be that after I sent it I also experienced the same problem on different SSID. So far I tried to modify the timeout to different values but the issue remains.

Any other ideas?

Thanks

[Scott Fella]

Well that changes everything. Now since all the SSID’s are having issues, it seems like a configuration setting causing it. I run a 9800-L at home with 9120/9130 AP’s and a bunch of iPads and iPhones along with Windows 10 Machines, ring devices, TV, streaming devices, IOT devices, etc and I have no issues at all. Make sure that you have your dhcp reservation defined and no other devices are using any of the address and make sure that there are no other static address devices using ip address that are used in dhcp.

[Mikolaj Moryto]

Thanks. I verified DHCP and I have a reservation for each single device. No other devices on the network without reservation that would use the same addresses. Also no static IPs in the range that DHCP assigns addresses from.

[Scott Fella]

Might be an issue with the 9800-CL. I have a 9800-L and EWC running at home with no issues with all my iOS, Windows and IOT devices all on dhcp. With the CL, wouldn’t you have local DHCP since typically that is used for a centralized cloud and your AP’s would have a FlexConnect group profile. Is that how you have it setup?

[Mikolaj Moryto]

No I don't use Flex. AP is in local mode and 9800-CL is at the same location. I have central switching and central DHCP on WLAN policies. I blame turning on the DHCP but trying to figure out why.

[Tony Rosolek · Accepted solution]

Hello Mikolaj,

as you suspected the "issue" is caused by a timeout. The value is called the "session timeout". It is the one, which you already increased from 1800 to 3600 seconds. To avoid that problem I recommend to set the value to 0. 

You can change that value via "Policy Profile" -> Edit -> Advanced -> Session Timout (sec) -> 0

You can see in the log which you posted, that the AP disassociates/deauthenticates your client and that's the reason you loose the session.

2020/02/09 12:40:23.373 {wncd_x_R0-0}{1}: [dot11] [20681]: (info): MAC: 4098.ad59.d36f Sent disassoc to client, disassoc reason: 252, CLIENT_DEAUTH_REASON_ADMIN_RESET delete reason: 24, CO_CLIENT_DELETE_REASON_SESSION_TIMEOUT.
2020/02/09 12:40:23.374 {wncd_x_R0-0}{1}: [dot11] [20681]: (info): MAC: 4098.ad59.d36f Sent deauth to client, deauth reason: 252, CLIENT_DEAUTH_REASON_ADMIN_RESET delete reason: 24, CO_CLIENT_DELETE_REASON_SESSION_TIMEOUT.

I'm really wondering why Cisco sets that as a default value, my expectation is that many clients struggle with that setting.

Best Regards

Tony

[Mikolaj Moryto]

Hi,

Thanks for the update.

I was actually thinking of disabling the timeout by setting the value to 0 when I was writing this thread but it seemed to be that you cannot set it to 0 from the GUI.

You got me thinking again and I managed to change it to 0 from the CLI which is something I didn't try before. No issues so far since I changed it.

Thanks!

[Tony Rosolek]

Which IOS do you have on your WLCs?

I saw in some documents, that the session timeout, needs to be at least 20 minutes. But in my lab with IOS 16.12.1 and 16.12.2 I can set the it to 0 via GUI.

[Scott Fella]

Session timeout can be 0, that’s what I usually have set in AireOS and IOS.