9800-CL virtual WLC - upgrade to 17.3.6 - C9115 in constant U-BOOT

[kenneth.gregersen]

Hi

So this morning we upgraded one of our Cisco 9800-CL clusters from 17.3.5a to 17.3.6.

All C9115AXI-B APs (14) on a certain site did not join after the upgrade.
The pre-download went fine, but when I finally got a console to one of the missing APs it was in a constant reload mode (u-boot).
I tried to force the AP to boot from the other partition

- setenv BOOT <other-partition>

It then boots up on the previous version (17.3.5a). Finds the WLC, figures out it got the wrong version, and reboots to load the new version (from the other partition (17.3.6)) - and we are back to endless reloads in u-boot.

I checked field notice: FN - 72278 to see if my APs were affected by this bad batch, but they were not.
At least that is what the Serial Number Validation Tool tells me.

The strange thing is that other C9115AXI-B APs, at a different site (but at the same WLC), works just fine with the new version.

I have tried the full factory reset, and also have the AP join a different WLC in our network, but to no avail.

I'm thinking that I might have to downgrade the WLC back to 17.3.5a to see if I can "unbrick" the APs.
Or, maybe follow the repair guide - even though my APs are not affected.

I know the 17.3.6 version is fairly new, but I'm wondering if anybody else have seen this?
And if you have any good suggestion on how to remedy the situation?

Thanks!

[marce1000]

                    - Could you post the full boot sequence of the problematic access point ?

 M.

[kenneth.gregersen]

Sorry for late reply. I have attached a very long putty log file that shows the boot process.

It's more or less the same over and over again.

[Leo Laohoo]

Could be a bug.  Raise a TAC Case with TAC EMEA or NAM/LATAM.  

[kenneth.gregersen]

Unfortunately we only have software and regular warranty on these APs, so when I talke with TAC I was told that my only option was to RMA the APs.
Not a very good solution for us since we have many more vWLCs (and APs) in the pipeline for upgrade.

[marce1000]

 - What about trying  https://www.cisco.com/c/en/us/support/docs/wireless/catalyst-9120axi-access-point/217537-repairing-c9120-c9115-access-points-from.html , even if at first glance the APs are not affected ?

 M.

[Leo Laohoo]

---

@kenneth.gregersen wrote:

[*01/01/1970 00:00:22.8660] verify signature failed for /bootpart/part2/ramfs_data_cisco.cpio.lzma

---

This bug has been re-introduced to 17.3.6.  

Which TAC did you engage, were they with TAC EMEA or NAM/LATAM?  

RMA-ing the APs when the issue could, potentially, be an firmware bug does not make any logical sense.

[kenneth.gregersen]

Hi

I just used the TAC on the web-page, the chat function.
So I'm not sure where they were located.

I totally agree that it doesn't make sense to RMA the AP when it's clearly a bug in the 17.3.6 code.

Frustrating "exercise". Now we have to connect to each and every AP with console cable to make it boot back on the backup partition, and re-connect back to our rolled back WLC (17.3.5a).

[Leo Laohoo]

If you are still chatting with TAC right now, then I strongly recommend you get it re-queued to EMEA or NAM/LATAM desk.  

This is a not a case about a bad AP, rather, a case of bad firmware and TAC EMEA or NAM/LATAM needs to investigate thoroughly and professionally.

[kenneth.gregersen]

Started a new chat with TAC today. Explained the situation and pointed to this thread.
Told that our company probably were beyond help at this point, but that they should investigate this problem thoroughly.

TAC Answer: You have no contract....You can do RMA...bla-bla....
And the request to bring the problem to the right team in Cisco were outside of this person scope, and blatantly ignored.

[Leo Laohoo]

Roll back to 17.3.5a.

[kenneth.gregersen · Accepted solution]

Yes, we have already done that.

We have also configured a new WLC on 17.3.5b, and we will test to migrate a C9115AXI AP to this test WLC - before we upgrade all our main WLCs.

[Surendra BG]

Hi Kenneth,

do you have an SR opened for this? we suspect a bug, please share SR# with me and attach the file what you have shared here to SR. this will help us to RCA and provide next steps

Regards,Surendra BG

[kenneth.gregersen]

Hi Surendra

I don't have a SR opened for this.
Cisco TAC showed no interest in helping out when I reached out to them via the Cisco TAC chat function.

There is a text file in this community post that shows the failing u-boot process from one of the APs.

We have rolled back all the WLCs to 17.3.5a/b, so we can't re-create the problem.

/Kenneth

[lawrence.allhands]

We upgraded a lab and pilot WLC 9800 to 17.3.6 and have noticed some major client performance issues on ac/ax clients using MS TEAMS while connected to a 9120. Dropped audio and video on regular intervals (Every few minutes). We also noted a lot of latency in he wireless connections (No dropped packets, just latency), almost as if the AP is caching. We saw nothing out of the ordinary in AP CPU or memory load, and it only had 4 5GHz clients (ax/ac/n) and a few 2.4 GHz clients. We rolled back the WLC software to 17.3.5b and all of these issues went away. I am going to be opening a TAC case today after a little more investigation, but be aware of this if you deploy 17.3.6.