topic Re: Image upgrade issues in Wireless

Image upgrade issues

Philip Brown — Mon, 05 Jul 2021 18:49:30 GMT

I have a pair of 3504's running in SSO mode on AIROS v 8.5.140. All AP's are model 2802i's. The ap image predownload primary all command worked but only for a few of the AP's. The Primry image on the WLC's is 8.5.151 by the way. No matter how many times the predownload is started, not all of the AP's download the image.

I now have 8.5.161 as the primary image on the WLC's and have set up Flexconnect image upgrade for the three locations concerned. Makes no difference where I start the predownload process, the master AP's don't predownload the image.

Any help would be most appreciated.

Re: Image upgrade issues

Mark Elsen — Fri, 06 Mar 2020 09:42:18 GMT

- Check the involved ap's logs write after the pre-imnage download is attempted, look for error messages , if any.

Re: Image upgrade issues

Philip Brown — Fri, 06 Mar 2020 12:36:17 GMT

I checked the log on one of the master AP's, seems the CAPWAP tunnel gets restarted for some reason or another.
Mar 6 12:17:18 kernel: [*03/06/2020 12:17:18.4366] UBIFS: reserved for root: 3682397 bytes (3596 KiB)
Mar 6 12:17:18 kernel: [*03/06/2020 12:17:18.4366] UBIFS: media format: w4/r0 (latest is w4/r0), UUID D36509B8-8343-4919-8651-B8C56A82B3A6, small LPT model
Mar 6 12:17:18 kernel: [*03/06/2020 12:17:18.4931] UBIFS: un-mount UBI device 0, volume 0
Mar 6 12:17:18 kernel: [*03/06/2020 12:17:18.4931] UBIFS: background thread "ubifs_bgt0_0" stops
Mar 6 12:17:18 kernel: [*03/06/2020 12:17:18.5006] Image Data Request sent to 10.x.x.x
Mar 6 12:17:18 kernel: [*03/06/2020 12:17:18.5588] CAPWAP HW tunnel params changed, DELETING the existing
Mar 6 12:17:19 kernel: [*03/06/2020 12:17:19.6149] HW CAPWAP tunnel is ADDED
Mar 6 12:17:58 kernel: [*03/06/2020 12:17:22.2454] ........
Mar 6 12:18:15 kernel: [*03/06/2020 12:18:03.8860] ...Re-Tx Count=1, Max Re-Tx Value=5, SendSeqNum=100, NumofPendingMsgs=3
Mar 6 12:18:15 kernel: [*03/06/2020 12:18:15.5474]
Mar 6 12:18:18 kernel: [*03/06/2020 12:18:18.3983] Re-Tx Count=2, Max Re-Tx Value=5, SendSeqNum=100, NumofPendingMsgs=3
Mar 6 12:18:18 kernel: [*03/06/2020 12:18:18.3983]
Mar 6 12:18:21 kernel: [*03/06/2020 12:18:19.4311] .Re-Tx Count=3, Max Re-Tx Value=5, SendSeqNum=100, NumofPendingMsgs=3
Mar 6 12:18:21 kernel: [*03/06/2020 12:18:21.2492]
Mar 6 12:18:24 kernel: [*03/06/2020 12:18:24.1000] Re-Tx Count=4, Max Re-Tx Value=5, SendSeqNum=100, NumofPendingMsgs=3
Mar 6 12:18:24 kernel: [*03/06/2020 12:18:24.1000]
Mar 6 12:18:26 kernel: [*03/06/2020 12:18:24.8193] .Re-Tx Count=5, Max Re-Tx Value=5, SendSeqNum=104, NumofPendingMsgs=7
Mar 6 12:18:26 kernel: [*03/06/2020 12:18:26.9510]
Mar 6 12:18:29 kernel: [*03/06/2020 12:18:29.8018] Re-Tx Count=6, Max Re-Tx Value=5, SendSeqNum=104, NumofPendingMsgs=7
Mar 6 12:18:29 kernel: [*03/06/2020 12:18:29.8018]
Mar 6 12:18:29 kernel: [*03/06/2020 12:18:29.8018] Max retransmission count exceeded, going back to DISCOVER mode.
Mar 6 12:18:29 kernel: [*03/06/2020 12:18:29.8025]
Mar 6 12:18:29 kernel: [*03/06/2020 12:18:29.8025] CAPWAP State: DTLS Teardown
Mar 6 12:18:29 kernel: [*03/06/2020 12:18:29.9155] DOT11_DRV[0]: vendor_send_mgmt_frame VAP 2 is not up as expected
Mar 6 12:18:29 kernel: [*03/06/2020 12:18:29.9156] DOT11_DRV[0]: vendor_send_mgmt_frame VAP 2 is not up as expected
Mar 6 12:18:29 kernel: [*03/06/2020 12:18:29.9157] DOT11_DRV[0]: vendor_send_mgmt_frame VAP 2 is not up as expected
Mar 6 12:18:29 kernel: [*03/06/2020 12:18:29.9158] DOT11_DRV[0]: vendor_send_mgmt_frame VAP 2 is not up as expected
Mar 6 12:18:29 kernel: [*03/06/2020 12:18:29.9164] DOT11_DRV[0]: vendor_send_mgmt_frame VAP 2 is not up as expected
Mar 6 12:18:29 kernel: [*03/06/2020 12:18:29.9165] DOT11_DRV[0]: vendor_send_mgmt_frame VAP 2 is not up as expected
Mar 6 12:18:30 NCI: CLEANAIR: Slot 0 CAPWAP down
Mar 6 12:18:30 NCI: I0: shutdownNci
Mar 6 12:18:30 NCI: CLEANAIR: Slot 1 CAPWAP down
Mar 6 12:18:30 NCI: I1: shutdownNci
Mar 6 12:18:30 kernel: [*03/06/2020 12:18:30.9242] Dropping dtls packet since session is not established. Peer 10.x.x.x-5246, Local 10.x.x.x-5256, conn (nil)
Mar 6 12:18:30 kernel: [*03/06/2020 12:18:30.9243] Dropping dtls packet since session is not established. Peer 10.x.x.x-5246, Local 10.x.x.x-5256, conn (nil)

Re: Image upgrade issues

Mark Elsen — Fri, 06 Mar 2020 13:44:06 GMT

- It could be some timeout while the AP is 'thinking' about the change to flexconnect ; I found a bug which seems indicative to the same :

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvg34947/?rfs=iqvred

Re: Image upgrade issues

Philip Brown — Fri, 06 Mar 2020 14:05:17 GMT

Would have been nice if that was it but all of the AP's are running in Flexconnect mode, we are not switching them from local to Flexconnect.
The predownload process just seems to cause the CAPWAP tunnel to be torn down and reinitiated. I saw something somewhere concerning a similar problem where LAG was causing the problems.
To disable LAG we would have to disable redundancy first. Might be best to reset all the kit to factory defaults and start over. The customer would be well pleased with that solution.

Re: Image upgrade issues

Scott Fella — Fri, 06 Mar 2020 14:34:17 GMT

Check to make sure the AP’s have enough memory. You could have extra images and or crash files that are consuming the memory and thus the image doesn’t have enough space. I have done upgrades in FlexConnect with AP’s in the US and I don’t even bother with the ore-download, I just upgrade the controller and reboot it. It comes down to your comfort level and experience to know what you need to do later. If you decide to reboot the controller and everything upgrades, that is great. However, if you upgrade and you notice that those AP’s are not upgrading, you can always reboot your controller to the old image and reboot. This way the aps will join again.

Re: Image upgrade issues

Philip Brown — Sun, 08 Mar 2020 13:53:44 GMT

Hi Scott,
Ok, the AP's concerned have not been online for that long. I checked two which are in a Flexconnect goup now and they both have more than enough Flash space available.
I disabled Flexconnect image upgrade for these two AP's and tried a normal predownload primary to them, didn't work either which is really weird because one of them, defined as master for the flexconnect group when enabled, predownloaded 8.5.151 and when decided to get 8.5.140 back on the WLC's and AP's, predownloaded that too.
I have now got 8,5.161 on the WLC's as the primary image but can't get one single AP to predownload it. The reason for now having 8.5.161 on the wlc's was to predownload it to those AP's which successfully predownloaded 8.5.151 and replace it with 8.5.161 so as to be able to use the Flexconnect image upgrade option to get 8.5.151 on all of the AP's.
The reason for predowloading are twofold, limit downtime to a min and locate any AP's with problems upfront. I was hoping not to have any AP's with problems as they are all the same model and spec but that was just wishful thinking.

Re: Image upgrade issues

Scott Fella — Sun, 08 Mar 2020 16:50:44 GMT

I would suggest you open a TAC case. If the pre-download works on some but not others, there can be other issues. You have already checked for free space, so that is good.