Perhaps just a fluke. - 17.15.1 - 9166i cannot join correctly anymore.

[Thomas Obbekaer Thomsen]

So behind the "strange" title, is a something strange that happened when I upgraded my lab to 17.15.1.

All APs joined and works except a single 9166i (that was originally an -MR, do not know if it is relevant to the "case").

The AP pre-downloaded the software, and everything seems fine, but after reloading everything the 9166i cannot join the WLC.

It gets an IP, CDP says its running the new software, I can ping the AP (until it reloads because it cannot join the WLC).

Disconnect reason when looking at WLC is : "DTLS  close alert from peer".

From what I can tell from the radioactive trace i did, the AP joins, and then it seems like the DTLS phase completes ?!?! (key DTLS Sess: 300000000000003 Inserted successfully).

Then a few ms later : (note): MAC: 10a8.2931.0ee0 AP disconnect initiated. Reason: DTLS close alert from peer, Phase: Join

I found out that the above errors is because the AP boots into the old 17.12.3 software (backup image), after a few times rebooting on the 17.15.1 software where it does not even start its discovery process it seems, then tries to join the WLC that runs 17.15.1. It then closes the DTLS session and reboots into 17.15.1 (primary image) because it already has this software already. - See thread for more interesting feedback and logs from the AP(s).

The more appropriate headline for this should, at the moment, be: "Perhaps just a fluke. - 17.15.1 - 9166 model(s) do not send / do Discovery Request."

Since my other APs ( 9164 normal CAPWAP from "birth" ) are joined and works just fine, I have a suspicion that it might be because this AP originally was an MR ? But that is of course just speculation.

I have attached the radioactive trace if anyone wants to have a look.

When I can physically get to the AP i might know more.

/Thomas

[marce1000]

  @Rich R  - I fully endorse you  on those items : but if we start doing like testing the AP on a virtual controller
                   at home , then things get totally  cloaked from what the real issue could be , because using such a 'test bed' (....)
                   which  has insufficient business networking support to get to real conclusions , and we probably create additional issues

 M.

[Rich R]

On the contrary - if the issue is reproducible on a bare bones setup without any features or complications that is useful for isolating the problem.  Home lab or work lab - not much difference except for scale and complexity.  I use the same approach to testing.  For some things I can do it at home - for others where I need a more complex/extensive integrated environment then I'll do it in the work lab, but functionally it's no different.  And sometimes home is preferable precisely because I want and need it to be isolated from the more complex lab environment.

[marce1000]

 @Rich R                              >On the contrary ......
                    It's called 'standard emo-flexing based response syndrome',  as on my behalf : I have now
                   muted this thread ,

    Nevertheless -> Best Regards , 

   Marc.
                  

[Thomas Obbekaer Thomsen]

Through backchannels (because I hate creating TAC cases, a Cisco product with "Center" in the name has made me lothe TAC cases ), I have now confirmed that it is a known internal bug for 9166, and from the notes its exactly what I have experienced.

I cant share the bugID with you at this moment, but Im sure it will show up in the bugtoolkit at some point.

The good thing is that overwringing the 17.15.1 image on the AP (that it got from the 9800-L-C controller it was attached to) and then having it download the same software, but this time of my local 9800-CL it now works as intended.

I do not know what would have happened if I had just overwritten the software, and then had it connect to the same 9800-L-C WLC. Perhaps I should try for fun ?

Thanks to everyone for all your good and insightful input.

[Leo Laohoo]

---

@Thomas Obbekaer Thomsen wrote:
I have now confirmed that it is a known internal bug for 9166, and from the notes its exactly what I have experienced.

---

So WNBU &/or the developers knew about it and, rather than withdraw 17.15.1 and fix it, they went on and published the firmware with a buggy AP Device Pack.

[Rich R]

Exactly! Just beggars belief! Releasing software knowing that it will brick customer APs without any warning!
At the very least it should have been in the release notes. 
Probably justifies a Field Notice.

[Rich R]

This info is public because anybody can run this command on a 17.15.1 WLC now that it has been released:
9800#sh ap image file summary
AP Image Active List
============================
Install File Name: base_image.bin
-------------------------------
AP Image Type Capwap Version Size (KB) Supported APs
------------- -------------- --------- ----------------------------------------------------------------------------------------------------------------------------------------------------- --------
ap1g4 17.15.1.6 54600 AP1852E, AP1852I, AP1832I, AP1830I
ap1g5 17.15.1.6 52440 AP1815W, AP1815T, OEAP1815, AP1815I, AP1800I, AP1800S, AP1815M, 1542D, AP1542I, AP1100AC, AP1101AC, P-WIFI-AC2, AP1840I
ap1g6 17.15.1.6 72120 AP2900I, C9117AXI
ap1g6a 17.15.1.6 87760 C9130AXI, C9130AXE, C9124AXI, C9124AXD, C9124AXE, C9136AXI, C9136I, IW9167EH, CW9164I, CW9166I, CW9166D1, IW9167IH
ap1g6b 17.15.1.6 83700 CW9162I, IW9165E, IW9165DH, CW9163E
ap1g7 17.15.1.6 76730 AP1900I, C9115AXI, AP1900E, C9115AXE, C9120AXE, C9120AXP, C9120AXI, C1115AXI
ap1g8 17.15.1.6 68860 C9105AXI, C9105AXW, C1105AXI, WP-WIFI6, ISR-AP1101AX
ap3g2 17.15.1.6 15460 NA
ap3g3 17.15.1.6 57170 AP3802E, AP3802I, AP3802P, AP4800, AP2802E, AP2802I, AP2802H, AP3800, AP1562E, AP1562I, AP1562D, AP1562PS, APVIRTUAL, IW-6300H-DC, IW-6300H-AC, IW-6 300H-DCW, ESW-6300
ap3g4 17.15.1.6 79130 CW9178I, CW9176I, CW9176D1
c1570 17.15.1.6 13050 AP1572E, AP1572I
c3700 17.15.1.6 14500 IW3702

[Leo Laohoo]

---

@Rich R wrote:

ap3g4 17.15.1.6 79130 CW9178I, CW9176I, CW9176D1

---

`nuff said.

[eglinsky2012]

---

@Leo Laohoo wrote:

---

@Rich R wrote:

ap3g4 17.15.1.6 79130 CW9178I, CW9176I, CW9176D1

---

`nuff said.

---

Indeed, sounds like those will be the successor to the 9136, 9166i, and 9166d respectively. Finally, they settle on a model naming scheme that makes sense! I never understood why the 9136 wasn't named the 9168....

In addition, I'm more fascinated by these lines:

ap1g6 17.15.1.6 72120 AP2900I, C9117AXI

ap1g7 17.15.1.6 76730 AP1900I, C9115AXI, AP1900E, C9115AXE, C9120AXE, C9120AXP, C9120AXI, C1115AXI

Aahh, the mysterious 1900 series that never saw the light of day... or did it? Was that what they branded pre-release versions of the 9117 and 9115 models?

Sorry to derail the topic at hand.

[thefranmanatt]

I have a 9164 that started life as a Meraki AP.  I was on 17.14.1 and upgraded to 17.15.1.  This AP is just boot looping with "TAM_ERROR_DEVICE_UPGRADE_TOO_MANY_APPS" bad cert.

[Rich R]

Did you try Thomas's workaround?
Overwrite the software:
https://software.cisco.com/download/home/286329934/type/286288051/release/15.3.3-JPT

[thefranmanatt]

No, I emailed my Cisco team and then installed a Ruckus Wi-Fi 7 AP until they fix it.   

[Leo Laohoo]

---

@thefranmanatt wrote:
installed a Ruckus Wi-Fi 7 AP until they fix it.   

---

Good workaround.