Errors after IOS upgrade to 17.6.3 from 17.3.4c

[frederick.mercado]

Some clients show IP learn state and errors below, any idea? I assume this has to do with FT and 802.1r but is this a bug? Some clients work just fine.

2022/04/07 14:09:09.687947 {wncd_x_R0-0}{1}: [apmgr-bssid] [17104]: (ERR): 1006.ed3f.1d60 BSSID validate wlan radio policy root is NULL

2022/04/07 14:11:07.683367 {wncd_x_R0-0}{1}: [client-orch-sm] [17104]: (ERR): MAC: 347d.f69e.7ee7 Triggering notification for IP learn timeout

2022/04/07 14:11:07.683387 {wncd_x_R0-0}{1}: [ewlc-infra-evq] [17104]: (ERR): 347d.f69e.7ee7CLIENT_STAGE_TIMEOUT State = IP_LEARNING, WLAN profile = DATA2, Policy profile = DATA, AP name = LSL_Main_2

2022/04/07 14:11:07.685183 {wncd_x_R0-0}{1}: [client-auth] [17104]: (ERR): MAC: 347d.f69e.7ee7 Failed to build flex client cache payload for FT-PSK. Couldn't get client AKM.

2022/04/07 14:11:09.569345 {wncd_x_R0-0}{1}: [dot11-validate] [17104]: (ERR): MAC: 347d.f69e.7ee7 Failed to validate dot11r pmkid. PMK cache not found

2022/04/07 14:11:09.569411 {wncd_x_R0-0}{1}: [apmgr-db] [17104]: (ERR): Failed to get opt roam statusInvalid (null) rf common record

2022/04/07 14:11:09.569412 {wncd_x_R0-0}{1}: [dot11k] [17104]: (ERR): MAC: 347d.f69e.7ee7 Rssi check failed, Unable to get the smart roam status for rf profile default_rf_5gh

[Flavio Miranda]

Hummm...this Bug afect your version and it seems that the log match.

2022/04/07 14:11:07.685183 {wncd_x_R0-0}{1}: [client-auth] [17104]: (ERR): MAC: 347d.f69e.7ee7 Failed to build flex client cache payload for FT-PSK. Couldn't get client AKM.

CSCwa67566 Controller rejects clients with wrong PMKID when client moves from FT-AKM to dot1x-AKM.

But it is a long shot.

You can check all the bugs for this version here:

https://www.cisco.com/c/en/us/td/docs/wireless/controller/9800/17-6/release-notes/rn-17-6-9800.html#Cisco_Concept.dita_e7a84ef7-e77e-42c4-933b-94c72ca735c4 

[Leo Laohoo]

Upgrade to 17.3.5a and see if the problem persist.

[frederick.mercado]

TAC case on this is 693381256

Issues since upgrading to 17.06.03 from 17.03.04c

License portal down? Not loading just spins.
2017-02-03 19:53:22.771 UTC SAEVT_INIT_CRYPTO success="False" error="Crypto Initialization has not been completed"

Unexplained timeouts
Apr 7 19:52:50.123: %DOT1X-5-FAIL: Chassis 1 R0/0: wncd: Authentication failed for client (dc1b.a1cb.9cec) with reason (Timeout) on Interface capwap_9000001b AuditSessionID C81F4A0A0000015F058E7E0F Username: LAMW2K\lslpej0

AP DTLS error on 1 AP
Apr 7 22:17:46.524: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: AP Name: LSL_Main_1 Mac: 1006.ed3f.1d60 Session-IP: 10.74.31.1[5268] 10.74.31.200[5246] CAPWAP DTLS session closed for AP, cause: DTLS server session shutdown

AAA Method List GUEST setup - disappears....
Cannot add GUEST login local to AAA Method List

FT issue?
CSCvv17251
2022/04/08 09:16:15.231747 {wncd_x_R0-0}{1}: [client-orch-sm] [17778]: (note): MAC: 3cf0.11b2.f06b Client delete initiated. Reason: CO_CLIENT_DELETE_REASON_ADMIN_RESET, details: , fsm-state transition 00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|01|07|13|17|18|28|33|42|44|46|48|a1|

Radio policy issue?
2022/04/08 09:15:47.449981 {wncd_x_R0-0}{1}: [apmgr-bssid] [17778]: (ERR): 1006.ed3f.df80 BSSID validate wlan radio policy root is NULL
2022/04/08 09:15:47.449985 {wncd_x_R0-0}{1}: [apmgr-bssid] [17778]: (ERR): 1006.ed3e.fca0 BSSID validate wlan radio policy root is NULL
2022/04/08 09:15:47.449987 {wncd_x_R0-0}{1}: [apmgr-bssid] [17778]: (ERR): 1006.ed3c.02a0 BSSID validate wlan radio policy root is NULL
2022/04/08 09:15:47.449990 {wncd_x_R0-0}{1}: [apmgr-bssid] [17778]: (ERR): 1006.ed3f.df80 BSSID validate wlan radio policy root is NULL
2022/04/08 09:15:47.449992 {wncd_x_R0-0}{1}: [apmgr-bssid] [17778]: (ERR): 1006.ed3e.fca0 BSSID validate wlan radio policy root is NULL
2022/04/08 09:15:47.449994 {wncd_x_R0-0}{1}: [apmgr-bssid] [17778]: (ERR): 1006.ed3c.02a0 BSSID validate wlan radio policy root is NULL
2022/04/08 09:16:15.231747 {wncd_x_R0-0}{1}: [client-orch-sm] [17778]: (note): MAC: 3cf0.11b2.f06b Client delete initiated. Reason: CO_CLIENT_DELETE_REASON_ADMIN_RESET, details: , fsm-state transition 00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|01|07|13|17|18|28|33|42|44|46|48|a1|
2022/04/08 09:16:15.231871 {wncd_x_R0-0}{1}: [client-orch-sm] [17778]: (note): MAC: 3cf0.11b2.f06b Delete mobile payload sent forbssid: 1006.ed3f.1d6e WTP mac: 1006.ed3f.1d60 slot id: 1

The BIGGEST issue is clients sitting in IP LEARN...any help here would be appreciated. I was supposed to get a call in 45 min to 1 hr but they had to reassign case. I am hesitant to downgrade, unless you all believe I can do so safely without more issues. 

[Rich R]

In that case I think best to try 17.3.5a with SMU (if you might be affected by that bug) as per

https://www.cisco.com/c/en/us/support/docs/wireless/catalyst-9800-series-wireless-controllers/214749-tac-recommended-ios-xe-builds-for-wirele.html#anc11

Otherwise you'll just have to work through the problems with TAC.

[Rich R]

And just to be clear 17.6.3 -> 17.3.4c is a DOWNGRADE not an upgrade ...

Agreed with @Leo Laohoo try 17.3.5a

17.6 is the next extended support release and will likely become the recommended version soon so if you did not have the problem with that then maybe go back to 17.6.3.  We're currently on 17.6.2 (no problems noticed so far) and testing 17.6.3.

[Arshad Safrulla · Accepted solution]

Downgrades in 9800 WLC's are not officially supported by TAC. As I was made to understand 17.3.5a will not be starred because of the AP join bug. You must understand that 17.6.X has major behavioral changes on certain features on 9800's. So my action plan will be 

1. If you had any previous known config on 17.3.X you can restore it in the WLC. 

2. If not Reconfigure new policies, profiles & tags and assign it to the AP's.

3. If you have a test bed move some AP's there for any testing. You can always fire up 9800-CL in any supported hypervisor for testing.

4. Enable FT only after testing extensively in your environment, there are lot of dependencies like client OS support, WLAN driver support etc. Do not use FT in adaptive mode.

Try to rebuild your configuration from the scratch if possible. If you want to fix what you have I think it's going to consume more time. But try take TAC feedback on this.

[frederick.mercado]

After factory resetting, I am now back up on 17.3.4c. Thanks.

Lastly, I still seem to have the bug for creating the AAA method list…for local GUEST access. Can I get resolve on this?

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvv34025

Is the closest description of what is happening. We cannot use another method list name, as our global wireless config multinational company uses “GUEST” for this connection.

[Arshad Safrulla]

As per the workaround you simply have to use a name which doesn’t contain the keyword “guest”

You may consider below names;
Visitor
Visitor_list etc.

[frederick.mercado]

Do we have to use the same name as the SSID however or is this independent of this?

[Rich R · Accepted solution]

The method list can literally be called anything you want as it only has local significance so can't see any reason why you would have to call it GUEST?  The method list name does not have to be the same as the SSID which is presumably what your "global wireless config multinational company uses" GUEST for.