9800-CL virtual WLC - upgrade to 17.3.6 - C9115 in constant U-BOOT

[kenneth.gregersen]

Hi

So this morning we upgraded one of our Cisco 9800-CL clusters from 17.3.5a to 17.3.6.

All C9115AXI-B APs (14) on a certain site did not join after the upgrade.
The pre-download went fine, but when I finally got a console to one of the missing APs it was in a constant reload mode (u-boot).
I tried to force the AP to boot from the other partition

- setenv BOOT <other-partition>

It then boots up on the previous version (17.3.5a). Finds the WLC, figures out it got the wrong version, and reboots to load the new version (from the other partition (17.3.6)) - and we are back to endless reloads in u-boot.

I checked field notice: FN - 72278 to see if my APs were affected by this bad batch, but they were not.
At least that is what the Serial Number Validation Tool tells me.

The strange thing is that other C9115AXI-B APs, at a different site (but at the same WLC), works just fine with the new version.

I have tried the full factory reset, and also have the AP join a different WLC in our network, but to no avail.

I'm thinking that I might have to downgrade the WLC back to 17.3.5a to see if I can "unbrick" the APs.
Or, maybe follow the repair guide - even though my APs are not affected.

I know the 17.3.6 version is fairly new, but I'm wondering if anybody else have seen this?
And if you have any good suggestion on how to remedy the situation?

Thanks!

[Leo Laohoo]

@lawrence.allhands, how is your TAC Case progressing?

I was testing 17.3.4, 17.3.5 and 17.3.5b and we found several bugs before we stopped.  I will be visiting 17.3.X in the next few weeks because we still have a large quantity of AP3700 in our fleet.  

[lawrence.allhands]

I just opened it and pushed it out to our site rep as well to get things rolling quickly. I was also able to replicate this behavior with a different WLC-9800 and a different 9120 and also a 2702 (To a lesser extent, but the latency is still there). The behavior was exactly the same, and rolling back a version fixed it.

The reason we are worried is we have the new 9120 v7s that require this software upgrade to deploy (We have about 100 of these in and cannot deploy them now. They have different hardware than previous versions of 9120s and will not join 17.3.5b or prior. We also have about 5K 2702s deployed on my network and are waiting for the 9136s/9166s to get here when we plan to roll out 17.9.2 site wide, but need to orphan our 5K 2702s on a pair of WLC 9800 80s running 17.3.6 during the process.

So between bad code, new hardware requiring it, and delayed shipment of new Wi-Fi 6E APs, Cisco is really doing a number on our scheduled upgrade.

[Leo Laohoo]

Our worst-case-scenario is going to be:  If 17.3.X is not going to be ready for the party, we may have to leave all our 3700 with our 8540 and all other APs go straight to 17.6.4.  

Currently having major fragmentation issue with 8.10.181.0.  In some scenario, 8.10.181.0 is not suitable if 2800/3800 is being used.  

Keep us posted with the status of your TAC Case.