User unable to connect PSK base SSID in mobility controller setup

[Noovi]

Hello Guys,

We have Foreign and anchor setup where there is one SSID with PSK.

Foreign WLC is passing PSK authentication but eventually receives client deletion due to below error.

Controller initiated client deletion with code: CO_CLIENT_DELETE_REASON_REMOTE_MOBILITY_DELETE. Explanation: Mobility requested client to be deleted, could have different triggers, like tunnel down. Actions: Check RA traces

For anchor WLC, getting below debug logs.

*Dot1x_NW_MsgTask_7: Apr 03 03:27:00.531: 0c:23:69:9b:1f:1f apfCreateMobileStationEntryWrapper (apf_ms.c:4510) Changing state for mobile 0c:23:69:9b:1f:1f on AP 00:00:00:00:00:00 from Idle to Idle

*Dot1x_NW_MsgTask_7: Apr 03 03:27:00.531: 0c:23:69:9b:1f:1f Adding mobile on Remote AP 00:00:00:00:00:00(0)
*Dot1x_NW_MsgTask_7: Apr 03 03:27:00.531: 0c:23:69:9b:1f:1f Created Acct-Session-ID (67ee3844/0c:23:69:9b:1f:1f/807948) for the mobile
*Dot1x_NW_MsgTask_7: Apr 03 03:27:00.531: 0c:23:69:9b:1f:1f mmAnchorExportRcv:, Mobility role is Unassoc
.
*Dot1x_NW_MsgTask_7: Apr 03 03:27:00.531: 0c:23:69:9b:1f:1f mmAnchorExportRcv Ssid=OTI-Scan-provision useProfileName=0 profileNameToUse=0Security Policy=0x40046040

*Dot1x_NW_MsgTask_7: Apr 03 03:27:00.531: 0c:23:69:9b:1f:1f Scheduling deletion of Mobile Station: reasonCode 1 (callerId: 69) in 1 seconds
*osapiBsnTimer: Apr 03 03:27:01.491: 0c:23:69:9b:1f:1f apfMsExpireCallback (apf_ms.c:688) Expiring Mobile!
*apfReceiveTask: Apr 03 03:27:01.492: 0c:23:69:9b:1f:1f pemApfDeleteMobileStation2: APF_MS_PEM_WAIT_L2_AUTH_COMPLETE = 0.
*apfReceiveTask: Apr 03 03:27:01.492: 0c:23:69:9b:1f:1f the value of url acl preserve flag is 0 for mobile 0c:23:69:9b:1f:1f (caller pem_api.c:5292)
*apfReceiveTask: Apr 03 03:27:01.492: 0c:23:69:9b:1f:1f 0.0.0.0 START (0) Deleted mobile LWAPP rule on AP [00:00:00:00:00:00]
*apfReceiveTask: Apr 03 03:27:01.492: 0c:23:69:9b:1f:1f Delete PMK cache along with client. Reason code: 1, Mobility Role: Unassociated
*apfReceiveTask: Apr 03 03:27:01.492: 0c:23:69:9b:1f:1f 2 PMK-remove groupcast messages sent
*apfReceiveTask: Apr 03 03:27:01.492: 0c:23:69:9b:1f:1f Deleted global PMK cache and MSCB PMKID/PMK cache entry for the client
*apfReceiveTask: Apr 03 03:27:01.492: 0c:23:69:9b:1f:1f Deleting mobile on AP 00:00:00:00:00:00(0)
*apfReceiveTask: Apr 03 03:27:01.492: 0c:23:69:9b:1f:1f apf_ms.c:5636 Clearing the SGT 0 of mobile
*Dot1x_NW_MsgTask_7: Apr 03 03:27:01.531: 0c:23:69:9b:1f:1f Received Handoff End request for client from 10.230.74.135

*Dot1x_NW_MsgTask_7: Apr 03 03:27:02.531: 0c:23:69:9b:1f:1f Received Handoff End request for client from 10.230.74.135

*Dot1x_NW_MsgTask_7: Apr 03 03:27:02.808: 0c:23:69:9b:1f:1f Mobile Announce recvd from 10.230.74.135 Vlan List payload not found, ignoring ...

*Dot1x_NW_MsgTask_7: Apr 03 03:27:02.808: 0c:23:69:9b:1f:1f Mobile Announce recvd from 10.230.74.135 VNID payload not found...

*Dot1x_NW_MsgTask_7: Apr 03 03:27:03.809: 0c:23:69:9b:1f:1f Mobile Announce recvd from 10.230.74.135 Vlan List payload not found, ignoring ...

*Dot1x_NW_MsgTask_7: Apr 03 03:27:03.809: 0c:23:69:9b:1f:1f Mobile Announce recvd from 10.230.74.135 VNID payload not found...

*Dot1x_NW_MsgTask_7: Apr 03 03:27:04.809: 0c:23:69:9b:1f:1f Mobile Announce recvd from 10.230.74.135 Vlan List payload not found, ignoring ...

*Dot1x_NW_MsgTask_7: Apr 03 03:27:04.809: 0c:23:69:9b:1f:1f Mobile Announce recvd from 10.230.74.135 VNID payload not found...

*Dot1x_NW_MsgTask_7: Apr 03 03:27:05.811: 0c:23:69:9b:1f:1f Anchor Export Request Recvd for mobile 0c:23:69:9b:1f:1f from 10.230.74.135 type : 16 subtype : 0 seq no : 0 xid : -1637662208
*Dot1x_NW_MsgTask_7: Apr 03 03:27:05.811: 0c:23:69:9b:1f:1f mmAnchorExportRcv: Extracting mmPayloadExportForeignLradMac
*Dot1x_NW_MsgTask_7: Apr 03 03:27:05.811: 0c:23:69:9b:1f:1f IPv6 ACl Name is none

Any suggestions?

[Scott Fella]

Are you having issues or just curious about the logs?  Are you able to troubleshoot onsite and can this be logs due to roaming?  Really need more information about the equipment you have, versions, is the anchoring working or not, et.

[Scott Fella]

If this is a new setup, make sure the SSID's you are anchoring is configured similar and also make sure that the SSID on the foreign is anchored to the mobility controller and the anchor controller SSID is anchored to itself.

Verify your setup with this guide: 
https://www.cisco.com/c/en/us/support/docs/wireless/catalyst-9800-series-wireless-controllers/213912-configure-mobility-anchor-on-catalyst-98.html#toc-hId-1694215469

Please provide more information also.

[Noovi]

Hi,

Lets say, on foreign i have selected WPA+WPA2 (WPA2 tick) and on anchor i have selected WPA2+WPA3 (WPA2 tick)

Will it work?

[Scott Fella]

The SSID needs to be identical, with the exception of the interface.  Test with an open SSID to confirm things are working and then work your way up.  It's easier to troubleshoot that way as it eliminates issues with mobility if you can get an open SSID working as you intend.

[Saikat Nandy]

Rule of thumb in mobility (from 9800 perspective).. WLAN profile has to be same, that includes your all the AKM config. Policy Profile config also has to be same. However you can define the WMI VLAN in the Foreign Policy Profile, whereas the actual VLAN needs to be present in the anchor WLC. Irrespective of the type of auth L2/L3, IP address assignment will always be done by the Anchor WLC side. 

[Rich R]

I suspect from @Noovi debugs that this is on AireOS and I suspect from the description that the foreign and anchor are running very different version of code (possibly 8.5.x and 8.10.x for example).

It would be nice if @Noovi had provided full details of the WLC models and software versions being used, which is a basic requirement for any post here, and for which @Scott Fella has already asked the details in a previous reply!  But as they haven't provided those details we can only give generic advice...

Make sure the setup meets the requirements for IRCM compatibility:
https://www.cisco.com/c/en/us/td/docs/wireless/controller/technotes/8-8/b_c9800_wireless_controller-aireos_ircm_dg.html
https://www.cisco.com/c/en/us/td/docs/wireless/compatibility/matrix/compatibility-matrix.html#ircm-platforms
If one of the WLCs is running 8.5 then make sure it's on 8.5.182.109 (IRCM, 8.5.182.111 for 3504) and any supporting 8.10 code should be on 8.10.196.0.

Take a look at @Rasika Nayanajith example at  https://mrncciew.com/2013/03/22/auto-anchor-mobility/

[Noovi]

Hello Rich,
Please find WLC details below.

Foreign WLC - 9800-40 WLC with 17.9.4a
Anchor WLC -  5520 WLC with 8.10.190.0
Intersting thing is we have other SSID with 802.1X on mobiliity which are working fine. So, its not about any mobility tunnel issue
Only one SSID with PSK is not working on mobility. 

Let me know if you need any more infor

[Saikat Nandy]

Looks like your authentication part is getting completed. So probably IP learn is where it might be failing. If the understanding is correct, please check the VLAN/SVI/DHCP config on the Anchor side from where your users are supposed to get IP address.