WLAN clients can't communicate since update from 17.3.3 to 17.3.4c

[Bernd Nies]

Hello,

We have Catalyst 9800-40 WLAN controllers and 90 Catalyst 9120AXI access points. Three weeks ago we upgraded from 17.3.3 to 17.3.4c. But instead of fixing a bug, it got exchanged with a new one: Since then all the WLAN clients can't communicate with each other. The WLAN is configured for central switching. Now it behaves like peer-blocking is enabled, but we had it disabled on purpose. It's an office wlan with 802.1x authentication and not a public wlan. Connectivity between wireless and wired LAN works fine.

Traffic analysis shows that the access point does not reveal the MAC addresses of other WLAN clients when one wants to reach them. Even two clients connected to the same access point can't communicate to each other.

Anyone else having this issue or are we the first one to upgrade to 17.3.4c?

Regards,

Bernd

[ammahend]

Haven’t run into this issue but certainly not immune to bugs on this, one of them was crashing WLC every 3-4 days, we eventually ended up getting special engineering image to fix it, since then it’s been stable, I understand the pain. 
This one seems buggy, try different options, may be try p2p forward upstream as temporary fix, let us know if it works, eventually you will need to work with TAC I think. 

[Bernd Nies]

Actually, Cisco TAC is on it for three weeks but with no result.

[marce1000]

   - Although this  is a new problem which appeared  after the upgrade 

, always useful is to have a configuration  check with (CLI) show tech wireless , have the output analyzed with :

• Cloud Version

look for warnings and errors  (anyway) , sometimes tweaking or  correcting them. may help.

 M.

[patoberli]

By now 17.3.5a is out, you might want to try this. Please note, it requires a hotfix to be added after the installation. 

Otherwise, what happens if you enable p2p blocking and afterwards disable it again?

[Bernd Nies]

I upgraded our backup WLAN controller to C9800-40-universalk9_wlc.17.03.05a.SPA.bin plus hotfix C9800-universalk9_wlc.17.03.05a.CSCwb13784.SPA.smu.bin and connected an access point. There it works again. Seems 17.3.4c is a bad release, although it is starred in the download portal.

Update: It works only partially. I joined two AP to the updated WLC.

On AP1

• Ping between two notebooks on same AP does not work.

• Ping from notebook to any other WLAN device connected to other APs from primary controller does work.

On AP2

• Ping from notebook to any other WLAN device does not work

[Rich R]

Have you tried?

ipv4 arp-proxy

https://www.cisco.com/c/en/us/td/docs/wireless/controller/9800/17-3/config-guide/b_wl_17_3_cg/m_arp_proxy.html

Also note: https://www.cisco.com/c/en/us/td/docs/wireless/controller/9800/17-3/release-notes/rn-17-3-9800.html#Cisco_Concept.dita_9e01664a-322e-4753-b915-a8f875a0d73d

Behavior Change

• From Cisco IOS XE Amsterdam 17.3.5a onwards, rate limiting is performed for ARP packets for each client to prevent a denial-of-service attack. If a client sends an ARP storm, then the client is excluded. To configure rate limiting, use the ip arp-limit rate command at the policy profile level.

[Bernd Nies]

Thanks for the hint. The ARP proxy setting probably didn't made it yet to the web UI. Also it was introduced in 17.3.1 and we were at 17.3.3. I tried it, no improvement. ARP resolution from a WLAN client for non-WLAN clients in same VLAN works. 

[Bernd Nies]

Cisco confirmed this to be a bug and is working on a solution. I wonder if we are the only ones that ever have upgraded beyond 17.3.3 or are all the others not using intra WLAN client communication?

[patoberli]

Thanks for the update. I guess most people indeed don't (anymore) use client to client communication and solely client to server.

[Bernd Nies]

Yes. We're probably the only ones who had to add all HP printers to WLAN because we were not allowed to install LAN cables through the office's false floor.

[Rich R]

Thanks for the update @Bernd Nies 

Did Cisco give you the bug id?

If not, can you ask them for it and share with us please?

It will be useful to see what other releases might be affected.

[Bernd Nies]

Got no bug ID. Support could reproduce it in lab. Issue exists on 17.3.4c and 17.3.5a with Catalyst 9800-40 WLC and 9120 APs. Don't know if the hardware platform is relevant.

[Rich R]

Keep pushing them for the bug id.

I've seen this happen lately - even after TAC reproduce the problem they won't open a bug for it until the BU give permission to do so and that can take weeks (or even months if you don't maintain pressure).  I don't know why they're doing this but I suspect it's about managing developers bonus targets because BU don't have to start working on the problem until after the bug is opened and by delaying the opening of the bug they can appear to respond more quickly when it's opened (even if it has taken weeks or months).

[Bernd Nies]

Finally, 69 days after opening the ticket on 24 March 2022: Cisco TAC opened BEMS and is waiting on BU engineer to accept and respond.