Issues with 8.10.181.0 and 3802s

[Wes Schochet]

Hi All-

I just upgraded my 5520s to 8.10.181.0 in anticipation of deploying some new APs.  8.10.181.0 appears to be the only version that supports the 9120.  It did no go well!

We have pairs of controllers.  I upgraded one to 8.10.181.0 and left one at 8.10.171.0

Tuesday night, I upgraded the controller at one of my campuses.  365 APs 3702i and 3802i all in local mode.  Moved AP's from one controller to the other one at a time.  No problems, no complaints.

Wednesday night, I did another controller.  This one supports my smaller sites.  Again, 3702s and 3802s.  We had many tickets with connectivity issues for Windows EAP-TLS laptops at several sites.  Moving these sites back to the 8.10.171 controller resolved the problems.  All of the problem sites are ones that run 3802s.  They are all back on 8.10.171.0  The other sites on 8.10.181 that have 3702s report no problems.

I did open a TAC case on this and am waiting for them to go through logs etc.  I'm curious to hear if anyone else is running this code and has seen any similar issues?

Thanks,
Wes

[Leo Laohoo · Accepted solution]

Ask TAC to verify if this is CSCwd37092.

We have rolled back our controllers from 8.10.181.0 to 8.10.171.0 because setting the TCP MSS to 1250 is not a workable fix 100% of the time.  Example, we had to set some site's TCP MSS value to 896 or 600.  

Our TAC Engineer stated a re-build is beginning to take shape and we are not certain if DE and BU will be releasing a different release or a re-build.

[Wes Schochet]

Yea - that's the story. Apparently, they swapped out a part to ease manufacturing delays - but that meas new code.  

Thanks!

[Rich R · Accepted solution]

It's definitely CSCwd37092.
TCP MSS is a workaround for slow throughput but there's no workaround for the EAP TLS problem.  Your only option for now is to rollback and wait for the fixed release which they are apparently working on ASAP.  The bug notes show they already have an engineering build 8.10.181.3 with the fix so TAC might be able to get you an escalation image if you don't want to roll back.

[Wes Schochet]

Yea - odd that only some of our devices had this issue.  If I had more time, I'd look at chipsets etc but...

Thanks!

[Rich R]

Yes you need 8.10.181.0 to support the new C9100 hardware revisions but the CSCwd37092 regression affects 2800/3800/4800/1562/6300 only.  Other AP models are not affected.  So that means if you could keep new C9100 APs on a separate WLC you could upgrade that to 8.10.181.0 to support the new APs while keeping the older affected APs on 8.10.171.0 until the fixed version is released.

[pannick]

Experienced this over the weekend. What a mess.

[Rich R]

Yep it's always worth keeping a watchful eye on https://www.cisco.com/c/en/us/support/docs/wireless/wireless-lan-controller-software/200046-tac-recommended-aireos.html

[Leo Laohoo]

I was lucky to get a no-nonsense TAC Engineer.  He asked a several questions and he knew what we hit.  He gave us this newly-created Bug ID not long after.

He also mentioned that there is background talk about removing 8.10MR8 but, as of 01 November 2022, the link is still present.  

As of yesterday, he said the re-build (I suspect to be 8.10.181.103) will be out in the next 48-hours.

[Leo Laohoo]

Re-build version 8.10.181.3 is now out.  Special download link provided by TAC.

AireOS 8.10MR8 Escalation Special