Perhaps just a fluke. - 17.15.1 - 9166i cannot join correctly anymore.

[Thomas Obbekaer Thomsen]

So behind the "strange" title, is a something strange that happened when I upgraded my lab to 17.15.1.

All APs joined and works except a single 9166i (that was originally an -MR, do not know if it is relevant to the "case").

The AP pre-downloaded the software, and everything seems fine, but after reloading everything the 9166i cannot join the WLC.

It gets an IP, CDP says its running the new software, I can ping the AP (until it reloads because it cannot join the WLC).

Disconnect reason when looking at WLC is : "DTLS  close alert from peer".

From what I can tell from the radioactive trace i did, the AP joins, and then it seems like the DTLS phase completes ?!?! (key DTLS Sess: 300000000000003 Inserted successfully).

Then a few ms later : (note): MAC: 10a8.2931.0ee0 AP disconnect initiated. Reason: DTLS close alert from peer, Phase: Join

I found out that the above errors is because the AP boots into the old 17.12.3 software (backup image), after a few times rebooting on the 17.15.1 software where it does not even start its discovery process it seems, then tries to join the WLC that runs 17.15.1. It then closes the DTLS session and reboots into 17.15.1 (primary image) because it already has this software already. - See thread for more interesting feedback and logs from the AP(s).

The more appropriate headline for this should, at the moment, be: "Perhaps just a fluke. - 17.15.1 - 9166 model(s) do not send / do Discovery Request."

Since my other APs ( 9164 normal CAPWAP from "birth" ) are joined and works just fine, I have a suspicion that it might be because this AP originally was an MR ? But that is of course just speculation.

I have attached the radioactive trace if anyone wants to have a look.

When I can physically get to the AP i might know more.

/Thomas

[marce1000]

   - 17.15.1 is very recent ; for starters have a  checkup of the  9800 controller's configuration with the CLI command
       show tech wireless and feed the output from that into Wireless Config Analyzer
      use the full command as denoted in green , do not use a simple show tech as input for this procedure
                  If this was done then it is useful to do again considering the upgrade

  Below , you will find the output for your attached radio active trace when  processed with Wireless Debug Analyzer
 Check All flag was checked!  If you don't have business dependencies of 17.15.1 's features then consider to fall
 back to advisory 17.12.3.

    Note in that output I see >... meraki connect: AP supports meraki monitoring
    I am not sure if that means if the AP is in full cisco 9800 client image mode (capable!)

2024/08/23 08:40:06.545	capwapac-discovery	__unknown__	EWLC TAGS payload is not present in discovery request
2024/08/23 08:40:06.545	loadbalance-notify	__unknown__	ite tag:default-site-tag populated from lb-info
2024/08/23 08:40:06.545	capwapac-discovery	__unknown__	Public IP learnt is FALSE, public IP discovery is FALSE, private IP discovery is TRUE.
2024/08/23 08:40:06.545	capwapac-discovery	__unknown__	IP:10.24.11.25[5256], Discovery Response sent
2024/08/23 08:40:06.580	capwapac-smgr-srvr	__unknown__	DTLS session create callback received.
2024/08/23 08:40:13.462	capwapac-smgr-sess	__unknown__	Received CAPWAP join request
2024/08/23 08:40:13.462	apmgr-db	__unknown__	Failed to get radio oper record. No Radio record for given slot=1
2024/08/23 08:40:13.464	apmgr-db	__unknown__	Failed to search validation db record during band id change
2024/08/23 08:40:13.464	apmgr-db	__unknown__	Failed to set current active band in 11kv data 2
2024/08/23 08:40:13.464	apmgr-db	__unknown__	Failed to search validation db record during band id change
2024/08/23 08:40:13.464	apmgr-db	__unknown__	Failed to set current active band in 11kv data 1
2024/08/23 08:40:13.464	apmgr-db	__unknown__	Failed to search validation db record during band id change
2024/08/23 08:40:13.464	apmgr-db	__unknown__	Failed to set current active band in 11kv data 0
2024/08/23 08:40:13.464	apmgr-capwap-join	__unknown__	meraki connect: AP supports meraki monitoring
2024/08/23 08:40:13.464	apmgr-capwap-join	__unknown__	Process join request: AP_DOMAIN payload count is 0
2024/08/23 08:40:13.465	apmgr-capwap-join	__unknown__	Successfully processed Join request. AP name: wap03, Model: CW9166I-MR, radio slots: 3, rlan slots: 0, site tag name: default-site-tag, policy tag name: default-policy-tag, rf tag name: default-rf-tag
2024/08/23 08:40:13.465	capwapac-smgr-srvr	__unknown__	Join processing complete. AP in joined state
2024/08/23 08:40:30.767	ap-join-info-db	__unknown__	AP disconnect initiated. Reason: DTLS close alert from peer, Phase: Join
2024/08/23 08:40:30.768	capwapac-smgr-sess	__unknown__	CAPWAP-IDB-Delete: IFID is 0. Returning
2024/08/23 08:40:30.768	apmgr-db	__unknown__	Failed to search 11kv validation db record
2024/08/23 08:40:30.768	apmgr-ap-global	__unknown__	AP SM Purge. AP deregister complete
2024/08/23 08:40:30.768	mm-aplist	__unknown__	an't find such AP MAC in the DB
2024/08/23 08:40:30.768	mm-aplist	__unknown__	annot remove AP: 2
2024/08/23 08:40:30.768	mm-aplist	__unknown__	emove non-existing AP
2024/08/23 08:40:30.768	apmgr-db	__unknown__	Delete ap jointime map record from the apmgr failed: 2
2024/08/23 08:40:30.768	wncloudm-geolocation	__unknown__	afc_loc_data delete record: Null ap_afc_loc_data record
2024/08/23 08:40:30.768	wncloudm-geolocation	__unknown__	geo_loc_data delete record: Null ap_geo_loc_data record
2024/08/23 08:40:30.768	wncloudm-geolocation	__unknown__	gnss_loc_data delete record: Null ap_gnss_loc_data record
2024/08/23 08:40:30.768	wncloudm-geolocation	__unknown__	Null ap_geo_loc_info record
2024/08/23 08:40:30.769	apmgr-db	__unknown__	Failed to delete ap band info entries. Get band record entry for slot 2 and band 0 error:22.
2024/08/23 08:40:30.769	awips	__unknown__	AWIPS:Record does not exist. Return
2024/08/23 09:02:11.293	capwapac-discovery	__unknown__	IP:10.24.11.25[5264], Discovery Request received
2024/08/23 09:02:11.293	capwapac-discovery	__unknown__	EWLC TAGS payload is not present in discovery request
2024/08/23 09:02:11.294	capwapac-discovery	__unknown__	Public IP learnt is FALSE, public IP discovery is FALSE, private IP discovery is TRUE.
2024/08/23 09:02:11.294	capwapac-discovery	__unknown__	IP:10.24.11.25[5264], Discovery Response sent

[eglinsky2012]

Try SSHing to the AP itself and doing a "show log," and share the output. SSH may be intermittently unavailable since the AP will be resetting its wired interface, rebooting, etc. in its the current state.

[Thomas Obbekaer Thomsen]

It does not look good.

I just had a console connected to the AP and it seems like it never starts its CAPWAP deamon.

Perhaps because of this:

[*08/11/2024 06:15:35.9371] 2024-08-11 06:15:35 TAMD: [ERR] There are 11 Apps in ACT2 device. TAm Service supports no more than 10 Apps.
[*08/11/2024 06:15:35.9433] 2024-08-11 06:15:35 TAMS: [ERR] Failed to register device ap-tam rc = TAM_ERROR_DEVICE_UPGRADE_TOO_MANY_APPS cleaning up
[*08/11/2024 06:15:35.9467] 2024-08-11 06:15:35 TAMD: [ERR] HW Heartbeat Failure TAM_ERROR_NO_EXT_CB_REGISTERED
[*08/11/2024 06:15:35.9490] 2024-08-11 06:15:35 TAMS: [ERR] Original request with trans id 0x3 not found

I will do a factory reset of the AP and post the result.

[Thomas Obbekaer Thomsen]

So the good news is when i hard reset the 9166-MR model that was failing it reverted back to Meraki Persona, and I can recover it that way.

The bad news is that the 9166D1-E I unfortunatly had connected to the same WLC seems to be having the above really bad time.

As mentioned all other APs seems fine on 17.15.1 ... just not 9166 models.

[marce1000]

 - You should then use the RadioActive trace again and have it analyzed as I showed in my first reply.

 M.

[Thomas Obbekaer Thomsen]

But the AP never does any capwap, so nothing will show up in the radioactive trace.

[Leo Laohoo]

Check if you are hitting this:  CSCwm07499

[Thomas Obbekaer Thomsen]

Perhaps, but the AP has downloaded the 17.15.1 and seemingly installed the 17.15.1 image, because it boots on that image it seems.

[Thomas Obbekaer Thomsen]

Hmmm didnt work anyway on the 9166-MR , after a VERY short while being connected to the dashboard it rebooted and reverted to "Catalyst" persona, with the same error as above.

When I try to do any CAPWAP command on the AP it returns : IPC socket server not ready for capwapd. Try after few moments, Errno: 2 Msg_id: 2

Even commands like capwap ap erase all

Very strange

[marce1000]

 - Yet the IPC socket server errpr seems like local problem on the AP and not 17.15.1 ; can ii (still) be used on older XE versions ?

 M.

[Rasika Nayanajith]

I had a similar issue when upgrading from 17.14.x to 17.15.x. The 9166 struggled to upgrade to 17.15 cleanly, possibly due to a corrupted image. I used two different WLCs and forced the AP to get 17.12.x first (to override corrupted image), then moved it to 17.15.x, which worked.

HTH
Rasika
*** Pls rate all useful responses ***

[Rich R]

Good point @Rasika Nayanajith 
@Thomas Obbekaer Thomsen what version did you upgrade from?
There are mandatory intermediate versions for upgrade (or at least there were in the beta) but I don't see those in the release notes.  For beta:

I'd recommend opening a TAC case because it sounds like a bug and also ask them to clarify the supported upgrade path.

ps: I've submitted feedback on the release notes and asked a question on beta to clarify the required upgrade path.

[Thomas Obbekaer Thomsen]

I upgraded from 17.12.3 so it should not have been a problem (PS: actually the official release notes just says from 17.12.x you can upgrade directly, but I did see that upgrade path mention in the beta).

As mentioned all other APs seems fine (9130, 9136 9164 and some ... ehhh .. "new ones" I have .. ).

I actually think I have another 9166D1 that failed, come to think of it, from another WLC I upgraded to 17.15.1 ( I will investigate ).

PPS:

The only thing all 9166(D) has in common is that at some point they have been in "Meraki Persona" mode.

[Rich R]

Ok cool - will still be good to clarify the upgrade path for others.

Definitely sounds buggy in that case.