WLC 9800-80 has intermittent delays sending multicast to AP Multicast

[joshua Slaney]

Hello,

I have a 9800-80 (HA Pair) running 17.9.4a code.  We've observed intermittent delays with vocera broadcasts.  Media-Stream multicast direct is disabled on all the WLANs. "show ap multicast mom" command shows the AP as "up" for the multicast communication from the AP to the WLC Multicast IP.  I can observe the groups on the WLC with the command "show ip igmp snooping groups" when the button is pressed for broadcast. I can also see with packet captures from the WLC and the AP switchport the clients that need to receive the broadcasts send a IGMPv2 Membership to the AP, the AP forwards that on to the WLC.  The packet captures on a "successful" broadcast show the WLC getting the traffic and immediately sending the RTP streams to the multicast IP address configured for the AP to use for the clients to receive the multicast traffic.  On a "delayed" broadcast, I see the controller receives 3-4 IGMPv2 membership packets from the AP/wireless client, before I see the controller management IP start sending to the AP multicast stream.  We did not have these issues with AireOS which me migrated off of in late January.   The cpu usage for the WLC doesn't seem high.  I've observed this on sites that are on different CPU from a Site tag perspective.

Pid PPid 5Sec 1Min 5Min Status Size Name
--------------------------------------------------------------------------------
18242 18234 39% 38% 40% R 2417912 wncd_4
17897 17889 38% 38% 37% S 1895356 wncd_1
17782 17774 19% 19% 19% S 1419496 wncd_0
18361 18353 16% 17% 17% S 1339508 wncd_5
18127 18119 14% 11% 11% S 1121224 wncd_3
18012 18004 13% 15% 16% S 1282176 wncd_2
18475 18468 7% 7% 7% S 1003204 wncd_6
18589 18583 0% 11% 11% S 1147836 wncd_7

Has anyone else experienced this issue?  The delays seem to be coming from the WLC actually sending the stream to the AP multicast IP address.  Any recommended commands I can run to troubleshoot this further?  

[No Cables on Me]

Are you using Vocera? we had a complete outage with our multicast, and after  about 5 days of TAC, we rebooted the WLC and everything been working since, that was in Dec/2023.. 
We would see the UP/UP in show multicast mom on the AP, and then they would go down.. We  pcap at ever hop between AP, WLC and RP.

[joshua Slaney]

Yeah we are using Vocera.  What code are you running on your WLC?  I have also seen multicast not work at times and turned off the "igmp snooping" and "igmp snooping querier" on the WLC and turned it back on, only to have everything work again.  With Vocera badges, we've also seen issues with the badges trying to use IGMPv3 and IGMPv2.  The IGMPv2 seemed to work, whereas the IGMPv3 did not.  In the last couple of firmware releases, vocera switched the default for the badges to only use IGMPv2 due to the # of calls they had been getting. In this case, we have up-to-date firmware on the badges, they are only using IGMPv2, and we are still seeing delays on broadcasts.  From the captures I have, it's pretty clear the delay is due to the WLC sending to the AP Multicast group IP.  I have a packet capture I've sent to TAC that shows a 10+ second delay from when the WLC receives the IGMPv2 membership from the client, until it starts sending to the AP multicast group.  I suspect this is either a processing issue on the WLC or an issue where the controller disregards the client membership IGMP packets.  I can't seem to find any bugs in this release related to the multicast.

[No Cables on Me]

We are running Version 17.9.3.. at this exact moment, I was drafting and email to Vocera support to see what I would need to enable to get some good logs for what going with the badges during a multicast session. 

We get reports of great days and sometimes really bad ones. 

[joshua Slaney]

17.9.3 definitely has IGMP bugs thats fixed in 17.9.4 and 17.9.5 higher  https://www.cisco.com/c/en/us/td/docs/wireless/controller/9800/17-9/release-notes/rn-17-9-9800.html 
-Access points intermittently stop sending Internet Group Management Protocol (IGMP) membership report. CSCwd41463 (fixed in 17.9.4
-Controller running Cisco IOS-XE 17.9.3 sends IGMP queries without controller IP and MAC addresses. CSCwh63050 (Fixed in 17.9.5)
What model of Vocera badges are you running?  For the B3000n badges, they set the default IGMPv2 starting in this release: https://pubs.vocera.com/rnotes/ReleaseNotes_b3000n_4.3.4.html. There has since been a newer release issued as well. 

What's New in 4.3.4.12

• The B3000n was validated by the CMVP to comply with the FIPS 140-2 standard for cryptographic security and was granted certificate # 4047 in October 2021.
• B3000n now uses IGMP version 2 by default instead of version 3

The behavior we were seeing before we did this firmware upgrade was that a number of badges would receive the broadcasts, while others wouldn't.  We were able to get captures and confirm the ones that weren't were attempting to use IGMPv3.  So we upgraded our firmware.  The same story goes with the v5000 badges.