2802i not joining 9800-CL with Invalid MTU error

[stephen.surbey]

I have a 2802i that was previously connected to a vWLC. I have reset it to factory using the button and I'm testing it against a 9800-CL version 17.7.1.11 and it fails to join. I have a new 9120AXI which does join the 9800. The 2802 seems to get all the way through to the controller offering to join the AP and then it fails with Invalid MTU. Below is the Radioactive trace of the join. Thanks for any help!

2022/02/23 19:36:22.833791821 {wncmgrd_R0-0}{1}: [capwapac-discovery] [17765]: (note): MAC: 28ac.9e82.4620 IP:10.1.239.17[5272], Discovery Request received

2022/02/23 19:36:22.833824013 {wncmgrd_R0-0}{1}: [capwapac-discovery] [17765]: (note): MAC: 28ac.9e82.4620 EWLC TAGS payload is not present in discovery request

2022/02/23 19:36:22.834096611 {wncmgrd_R0-0}{1}: [ewlc-infra-evq] [17765]: (debug): instance :0 port:19245MAC: 3900.0100.0b46

2022/02/23 19:36:22.834259961 {wncmgrd_R0-0}{1}: [capwapac-discovery] [17765]: (note): MAC: 28ac.9e82.4620 Public IP learnt is FALSE, public IP discovery is FALSE, private IP discovery is TRUE.

2022/02/23 19:36:22.834373289 {wncmgrd_R0-0}{1}: [capwapac-discovery] [17765]: (note): MAC: 28ac.9e82.4620 IP:10.1.239.17[5272], Discovery Response sent

2022/02/23 19:36:23.059992666 {wncmgrd_R0-0}{1}: [capwapac-discovery] [17765]: (note): MAC: 28ac.9e82.4620 IP:10.1.239.17[5272], Discovery Request received

2022/02/23 19:36:23.060069991 {wncmgrd_R0-0}{1}: [capwapac-discovery] [17765]: (note): MAC: 28ac.9e82.4620 EWLC TAGS payload is not present in discovery request

2022/02/23 19:36:23.060155567 {wncmgrd_R0-0}{1}: [ewlc-infra-evq] [17765]: (debug): instance :0 port:19245MAC: 3900.0100.0b46

2022/02/23 19:36:23.060275487 {wncmgrd_R0-0}{1}: [capwapac-discovery] [17765]: (note): MAC: 28ac.9e82.4620 Public IP learnt is FALSE, public IP discovery is FALSE, private IP discovery is TRUE.

2022/02/23 19:36:23.060392814 {wncmgrd_R0-0}{1}: [capwapac-discovery] [17765]: (note): MAC: 28ac.9e82.4620 IP:10.1.239.17[5272], Discovery Response sent

2022/02/23 19:36:32.418773089 {wncd_x_R0-0}{1}: [ewlc-infra-evq] [18463]: (debug): instance :0 port:38932MAC: 28ac.9e04.c7dc

2022/02/23 19:36:32.419225902 {wncd_x_R0-0}{1}: [capwapac-smgr-srvr] [18463]: (note): MAC: 28ac.9e82.4620 DTLS session create callback received.

2022/02/23 19:36:32.419554326 {wncd_x_R0-0}{1}: [capwapac-smgr-sess] [18463]: (info): Mac: Session-IP: 10.1.239.17[5272] PMTU 576, effective capwap size 416

2022/02/23 19:36:32.419570646 {wncd_x_R0-0}{1}: [ewlc-infra-evq] [18463]: (debug): DTLS record type: 22, handshake

2022/02/23 19:36:32.419570988 {wncd_x_R0-0}{1}: [ewlc-infra-evq] [18463]: (info): DTLS client hello

2022/02/23 19:36:32.560494481 {wncd_x_R0-0}{1}: [ewlc-dtls-sessmgr] [18463]: (info): Remote Host: 10.1.239.17[5272] MAC: 28ac.9e82.4620 Completed cert verification, status:CERT_VALIDATE_SUCCESS

2022/02/23 19:36:37.928210052 {wncd_x_R0-0}{1}: [capwapac-smgr-sess] [18463]: (note): MAC: 28ac.9e82.4620 Received CAPWAP join request

2022/02/23 19:36:37.929156676 {wncd_x_R0-0}{1}: [apmgr-db] [18463]: (ERR): 28ac.9e82.4620 Failed to get radio oper record. No Radio record for given slot=1

2022/02/23 19:36:37.929157252 {wncd_x_R0-0}{1}: [apmgr-db] [18463]: (ERR): Failed to set external antenna gain. Band record retrieval error:22

2022/02/23 19:36:37.929158432 {wncd_x_R0-0}{1}: [apmgr-db] [18463]: (ERR): 28ac.9e82.4620 Ap join rf-tag validation is failed

2022/02/23 19:36:37.929695496 {wncd_x_R0-0}{1}: [apmgr-db] [18463]: (info): 802.11b Radio summary Total 0 Up 0 Down 0 Client serving Total 1 Up 1 Down 0

2022/02/23 19:36:37.929696084 {wncd_x_R0-0}{1}: [apmgr-db] [18463]: (info): 802.11 6ghz Radio summary Total 0 Up 0 Down 0 Client serving Total 0 Up 0 Down 0

2022/02/23 19:36:37.929696299 {wncd_x_R0-0}{1}: [apmgr-db] [18463]: (info): 802.11a Radio Total 2 Up 2 Down 0 Client Serving Total 2 Up 2 Down 0

2022/02/23 19:36:37.929696517 {wncd_x_R0-0}{1}: [apmgr-db] [18463]: (info): XOR Radio summary Total 1 Up 1 Down 0 RX-only Radio summary Total 0 Up 0 Down 0

2022/02/23 19:36:37.929696919 {wncd_x_R0-0}{1}: [apmgr-db] [18463]: (info): Monitor dual-band Radio summary Total 0 Up 0 Down 0 Monitor 80211bg Radio summary Total 0 Up 0 Down 0 Monitor 80211a Radio summary Total 0 Up 0 Down 0 Monitor 80211 6ghz Radio summary Total 0 Up 0 Down 0

2022/02/23 19:36:37.929697172 {wncd_x_R0-0}{1}: [apmgr-db] [18463]: (info): Sensor Radio summary Total 0 Up 0 Down 0

2022/02/23 19:36:37.929697382 {wncd_x_R0-0}{1}: [apmgr-db] [18463]: (info): All Radio summary Total 3 Up 3 Down 0

2022/02/23 19:36:37.930306029 {wncd_x_R0-0}{1}: [rrm-client] [18463]: (ERR): 28ac.9e82.4620 Failed to override default values inradio oper for slot 1, reg domain chk status failed

2022/02/23 19:36:37.930683083 {wncd_x_R0-0}{1}: [apmgr-db] [18463]: (info): 802.11b Radio summary Total 0 Up 0 Down 0 Client serving Total 2 Up 2 Down 0

2022/02/23 19:36:37.930683335 {wncd_x_R0-0}{1}: [apmgr-db] [18463]: (info): 802.11 6ghz Radio summary Total 0 Up 0 Down 0 Client serving Total 0 Up 0 Down 0

2022/02/23 19:36:37.930683582 {wncd_x_R0-0}{1}: [apmgr-db] [18463]: (info): 802.11a Radio Total 2 Up 2 Down 0 Client Serving Total 2 Up 2 Down 0

2022/02/23 19:36:37.930683815 {wncd_x_R0-0}{1}: [apmgr-db] [18463]: (info): XOR Radio summary Total 2 Up 2 Down 0 RX-only Radio summary Total 0 Up 0 Down 0

2022/02/23 19:36:37.930684068 {wncd_x_R0-0}{1}: [apmgr-db] [18463]: (info): Monitor dual-band Radio summary Total 0 Up 0 Down 0 Monitor 80211bg Radio summary Total 0 Up 0 Down 0 Monitor 80211a Radio summary Total 0 Up 0 Down 0 Monitor 80211 6ghz Radio summary Total 0 Up 0 Down 0

2022/02/23 19:36:37.930684341 {wncd_x_R0-0}{1}: [apmgr-db] [18463]: (info): Sensor Radio summary Total 0 Up 0 Down 0

2022/02/23 19:36:37.930684563 {wncd_x_R0-0}{1}: [apmgr-db] [18463]: (info): All Radio summary Total 4 Up 4 Down 0

2022/02/23 19:36:37.931100069 {wncd_x_R0-0}{1}: [rrm-client] [18463]: (ERR): 28ac.9e82.4620 Failed to override default values inradio oper for slot 0, reg domain chk status failed

2022/02/23 19:36:37.931265183 {wncd_x_R0-0}{1}: [apmgr-capwap-join] [18463]: (info): 28ac.9e82.4620 Retrieved AP SW version: 17.7.1.11, for AP model: AIR-AP2802I-A-K9, AP image type: ap3g3, site-tag: default-site-tag

2022/02/23 19:36:37.931281288 {wncd_x_R0-0}{1}: [apmgr-db] [18463]: (info): 28ac.9e82.4620 Operation state of AP changed to: Registered

2022/02/23 19:36:37.931932318 {wncd_x_R0-0}{1}: [apmgr-capwap-join] [18463]: (note): MAC: 28ac.9e82.4620 Successfully processed Join request. AP name: AP28AC.9E04.C7DC, Model: AIR-AP2802I-A-K9, radio slots: 2, rlan slots: 0, site tag name: default-site-tag, policy tag name: default-policy-tag, rf tag name: default-rf-tag

2022/02/23 19:36:37.931947540 {wncd_x_R0-0}{1}: [capwapac-smgr-srvr] [18463]: (ERR): MAC: 28ac.9e82.4620 Invalid MTU value 0 in join request

2022/02/23 19:36:37.931963739 {wncmgrd_R0-0}{1}: [ewlc-infra-evq] [17765]: (note): Msg type :mesg->msgtype Priority:1

2022/02/23 19:36:37.931981407 {wncd_x_R0-0}{1}: [ap-join-info-db] [18463]: (note): MAC: 28ac.9e82.4620 AP disconnect initiated. Reason: Invalid path MTU, Phase: Join

2022/02/23 19:36:37.931996949 {wncd_x_R0-0}{1}: [capwapac-smgr-srvr] [18463]: (ERR): MAC: 28ac.9e82.4620 Failed to get wtp session record : Get ap join fail info

2022/02/23 19:36:37.932000607 {wncd_x_R0-0}{1}: [capwapac-smgr-sess-fsm] [18463]: (ERR): Mac: 28ac.9e82.4620 Session-IP: 10.1.239.17[5272] 10.1.244.201[5246] Unmapped previous state in transition S_APMGR_JOIN_RSP_PROCESS to S_END

2022/02/23 19:36:37.932012219 {wncd_x_R0-0}{1}: [capwapac-smgr-sess-fsm] [18463]: (note): Mac: 28ac.9e82.4620 Session-IP: 10.1.239.17[5272] 10.1.244.201[5246] Last Control Packet received 0 seconds ago.

2022/02/23 19:36:37.932012504 {wncd_x_R0-0}{1}: [capwapac-smgr-sess-fsm] [18463]: (note): Mac: 28ac.9e82.4620 Session-IP: 10.1.239.17[5272] 10.1.244.201[5246] Last Data Keep Alive Packet information not available. Data session was not established

2022/02/23 19:36:37.932149205 {wncd_x_R0-0}{1}: [errmsg] [18463]: (note): %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: R0/0: wncd: AP Event: AP Name: AP28AC.9E04.C7DC Mac: 28ac.9e82.4620 Session-IP: 10.1.239.17[5272] 10.1.244.201[5246] Disjoined Invalid path MTU

2022/02/23 19:36:37.932374637 {wncd_x_R0-0}{1}: [ewlc-dtls-sessmgr] [18463]: (info): Remote Host: 10.1.239.17[5272] MAC: 28ac.9e82.4620 Sending DTLS alert message, closing session..

2022/02/23 19:36:37.932376116 {wncd_x_R0-0}{1}: [ewlc-dtls-sessmgr] [18463]: (info): Remote Host: 10.1.239.17[5272] MAC: 28ac.9e82.4620 alert type:warning, description:close notify

2022/02/23 19:36:37.932580436 {wncd_x_R0-0}{1}: [ewlc-dtls-sess] [18463]: (note): Remote Host: 10.1.239.17[5272] MAC: 28ac.9e82.4620 DTLS session destroy

[Scott Fella]

I run the 9800-CL in my home lab on the same code versions with 9130's, 9120's, 3802's and 4802's with no issues.  My suggestion is to factory reset the AP and then look at the console output logs to see if you see anything that points to the issue.  Since you have other ap's currently joined, there should be no issue with your initial setup.

[stephen.surbey]

On the AP I'm getting the following and haven't been able to find any clues on the internet as to what might be important here. Thanks

[*02/24/2022 02:17:42.8731] CAPWAP State: Discovery
[*02/24/2022 02:17:42.8787] Discovery Request sent to 10.1.244.201, discovery type STATIC_CONFIG(1)
[*02/24/2022 02:17:42.8788] Got WLC address 10.1.244.201 from DHCP.
[*02/24/2022 02:17:42.8788] IP DNS query for CISCO-CAPWAP-CONTROLLER
[*02/24/2022 02:17:43.0476] Discovery Request sent to 10.1.244.201, discovery type STATIC_CONFIG(1)
[*02/24/2022 02:17:43.0487] Discovery Request sent to 255.255.255.255, discovery type UNKNOWN(0)
[*02/24/2022 02:17:43.0487] Discovery Response from 10.1.244.201
[*02/24/2022 02:17:43.1101] Discovery Response from 10.1.244.201
[*02/24/2022 02:17:52.0000]
[*02/24/2022 02:17:52.0000] CAPWAP State: DTLS Setup
[*02/24/2022 02:17:52.8792] First connect to vWLC, accept vWLC by default
[*02/24/2022 02:17:52.8792]
[*02/24/2022 02:17:52.8837]
[*02/24/2022 02:17:52.8837] CAPWAP State: Join
[*02/24/2022 02:17:52.8854] Sending Join request to 10.1.244.201 through port 5272
[*02/24/2022 02:18:13.2932] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Join(5).
[*02/24/2022 02:18:13.8660] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Join(5).
[*02/24/2022 02:18:49.0171]
[*02/24/2022 02:18:49.0171] CAPWAP State: DTLS Teardown
[

[Scott Fella]

Take a look at this post... you might be able to find more.  This is what alerts me, but the log is very short.

[*02/24/2022 02:17:52.8792] First connect to vWLC, accept vWLC by default

AP 1852 does not join vWLC virtual controllers - Cisco Community

[Scott Fella]

Did you factory reset the AP?

[stephen.surbey]

Yep, with the reset button. In the morning I'll add more debugging on the AP.

[Arshad Safrulla · Accepted solution]

Can you post a show version from the AP? It is noticed that Wave 2 AP's running certain AIreOS codes cannot join XE controllers without an upgrade performed in the AP manually.

I believe that you are running your 9800-CL as a test bed. Do you have the minimum recommended specs assigned to the VM? Hyperthreading is not supported and also resources assigned must be reserved for the VM.

[stephen.surbey]

Just shows that you need to trust your instincts. Thought of this several times over the last week and didn't do it. Manually upgraded AP from 8.5.161 to 17.7.1.11 and it joined up right away.

Thanks everyone for your help!

[Rich R]

I think this is related to https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvx98176 - essentially latest IOS-XE CAPWAP is not compatible with older code.  If the AP was upgraded to 8.10.162.0 code or 8.5.182.0 code I expect it would have worked.  The bug refers to the other way round but I think the same applies to your case.

This is documented at 

https://www.cisco.com/c/en/us/support/docs/wireless/wireless-lan-controller-software/200046-tac-recommended-aireos.html#anc11

https://www.cisco.com/c/en/us/support/docs/wireless/catalyst-9800-series-wireless-controllers/214749-tac-recommended-ios-xe-builds-for-wirele.html#anc35

It's well worth making sure you keep an eye on those pages.  They're updated regularly and tend to warn about those things that are most likely to catch you.

[JPavonM]

This error also appeared when you try to move to latest 17.3.5a for some APs, and this is due to a known bug CSCwb13784 so recommendation is not to upgrade to those versions affected, 17.7.1 between them.