Re: Urequested multicast video traffic

cjonescbs · ‎09-18-2017

I manage a small multicast video network. We have remote cameras and encoders that create multicast mpeg-ts over UDP streams and decoders that decode them. During a busier than normal night I noticed something strange, three ports on my ws-c3750x-48 with nearly 100Mbs of egress. After looking at logs I notice they have been like this for quite some time. These ports are hooked to decoders and would be expected to consume 12-30Mbs depending on what they were decoding. After using SPAN to get the egress data into wireshark I see that these ports are getting multicast traffic they aren't requesting.

For example 239.255.1.11 is a camera that is not being decoded by decoder 1, but I see 12Mbs of egress on the decoder 1 port, and with wireshark it is all 239.255.1.11 UDP traffic. I can open terminal and with command "show ip igmp snooping groups"see that 239.255.1.11 isn't on any ports on this switch. If I configure decoder 1 to decode 239.255.1.11 it will do that without a problem, but the port traffic doubles. I am 99.9% positive it's not hardware, I have 5 of the same model decoders identical to the firmware level and only two have the port problem. The remote devices behave normally across the network. Multiple power cycles of the decoders, I've up/down the ports in question, but I still get unrequested traffic. IGMP v2 appears to be functioning normally, I can see ports appear and dissappear from multicast groups as expected when I decode different streams.

Apologies, the extent of my experience is managing this network. It's extremely likely this is a multicast "gotcha" I am not aware of, or some configuration I've buggered but I can't find it. Any help would be appreciated. I've tried some searches but can't find the keywords to string together to find anyone else with a similar issue.

chrihussey · ‎09-19-2017

Interesting issue. Could be a bug. Aside from that, could you post the switch config and the output of "sh ip igmp snooping detail". Also, is your multicast environment sparse of dense mode?

Thanks

cjonescbs · ‎09-19-2017

Thanks for taking the time, terminal log of the command you requested and config output attached. I set this up quite a few years ago but for dense vs sparse I don't think it's either as I think those only come into play if multicast routing is enabled. PIM-DVMRP is listed on all the ports, the 3750 is the igmp querier, it's pretty much a layer 2 network. I have all the multicast transmitters and receivers isolated to their own vlan on port fast enabled static access ports, they should (and almost all have been) behaving like good igmp v2 devices and entering and leaving as they requested, keeping unwanted high BW traffic to a minimum.

chrihussey · ‎09-20-2017

I don't see anything unusual in what you are doing. The output of "sh ip igmp snooping querier detail" may shed some light on things. A few thoughts:

Is there more than one querier configured on the network? My understanding is that there should only be one.

From what you describe, 2 or 3 of your five decoders are experiencing this issue. You could flip ports between one having the issue and one not to see if it follows or stays with the port. Also, it would be a good idea to sniff the decoder port before the unwanted stream starts just to see what type of IGMP packets / communications are occuring.

Finally, one can't rule out the possibility something is "stuck". A reboot of the 3750 may solve things for you.

cjonescbs · ‎09-20-2017

The only querier is the 3750. My hazy recollection is that there should be only one as well.

Busy with other things so only managed a couple other troubleshooting tests yesterday, I added some new multicast sources to the network. Any new multicast source I added instantly showed up on the three problem ports. Traffic egress continues even with no device hooked to them. I am going to swap ports, but I suspect you're correct, these ports are stuck (they behave like an unmanaged switch in regards to multicast video, treating it like broadcast) and the 3750 needs a reboot.

The port swap is a good idea, should have been the first test, I'll let you know.

cjonescbs · ‎09-20-2017

After moving devices around the problem stays with the port.

The thought of rebooting is a bit scary. As this is all layer 2, is their some command that would delete all mac addresses in the forwarding table for a particular port? I tried clear mac address-table dynamic interface but the problem persists. It's almost like the switch has decided to forward traffic to the cisco mac address of the switchport itself.

Traffic did stop egress with no device connected, not sure what I was looking at yesterday, probably my SPAN traffic. I may take the cowards way out and just move off the problem ports.

chrihussey · ‎09-20-2017

You could also try removing the igmp querier config and then put it back to see if that has an affect.

You also could try "clear interface XXXX" which I believe is supposed to clear the hardware logic of the interface. Admittedly that's a long shot.

Other than that, short of the reload, moving to other ports if available may be the way to go.

cjonescbs · ‎09-21-2017

I was optimistic about the clear interface command, unfortunately it did not resolve the issue. Until I can find some time to reboot the switch (likely fix) I have just moved off the three problem ports.

Thanks for your help.

chrihussey · ‎09-21-2017

Thanks for the update and the helpful votes.
Regards

Joseph W. Doherty · ‎09-20-2017

Assuming you've also are running IGMP snooping, possible failure of it for those ports? As Chris already wondered, possible bug.(?) BTW, what IOS version are you running on your 3750X?

cjonescbs · ‎09-21-2017

Version is 12.55(SE3)-UNIVERSALK9. Sorry I missed your post earlier.

Joseph W. Doherty · ‎09-21-2017

Ah, 55 is a solid chain, but SE3 is an early version. If you have a support contract, upgrade to the latest 55 release.

cjonescbs · ‎01-12-2018

This 3750 switch recently began having more serious issues. The processor load went up for no apparent reason, random reboots, loss of snmp connectivity. I copied the config over to a new 2960XR and brought those problem ports back online and have no issues with unrequested multicast traffic on the new switch. Not sure what to chalk this up as, hardware problem or IOS issue, but now I don't have to worry about it. I guess it wasn't a config issue because that would have made it to the new switch.