Nexus not receiving some IGMP Membership reports on the same VLAN

[g748437]

Hello,
I have been dealing with this Multicast issue for over a week now and I can't seem to understand why it's happening.
We have 2 Core Switches

Nexus 5548 with vPC (N5K-C5548UP-SUP, 7.3(5)N1(1)) and a 3750X (C3750X-48PF-S, 15.2(4)E9) stack with 5 SWs

They are connected via Port-channel

4094 (on the NX side) and Po10 (on the 3750 side), Po4096 is the vPC Peer-link.

These are the config for

VLAN interface 880

CORE-1# show run int vlan 880
interface Vlan880
no shutdown
ip address 172.30.80.2/24
ip pim sparse-mode
ip igmp version 3
hsrp version 2
hsrp 880
preempt
priority 80
ip 172.30.80.1




CORE-2(config)# show run int vlan 880
interface Vlan880
no shutdown
ip address 172.30.80.3/24
ip pim sparse-mode
ip igmp version 3
hsrp version 2
hsrp 880
preempt
priority 80
ip 172.30.80.1




CORE-2(config)# show ip igmp snooping mrouter vlan 880
Type: S - Static, D - Dynamic, V - vPC Peer Link
I - Internal, F - Fabricpath core port
C - Co-learned, U - User Configured
P - learnt by Peer
Vlan Router-port Type Uptime Expires
880 Po4096 SVD 24w6d 00:03:31
880 Vlan880 I 1w2d never




CORE-1# show ip igmp snooping mrouter vlan 880
Type: S - Static, D - Dynamic, V - vPC Peer Link
I - Internal, F - Fabricpath core port
C - Co-learned, U - User Configured
P - learnt by Peer
Vlan Router-port Type Uptime Expires
880 Po4096 SV 24w6d never
880 Vlan880 ID 1w2d 00:04:59




CORE-1# show ip igmp snooping querier vlan 880
Vlan IP Address Version Expires Port
880 172.30.80.2 v3 00:03:38 Vlan880 (internal)

CORE-2(config)# show ip igmp snooping querier vlan 880
Vlan IP Address Version Expires Port
880 172.30.80.2 v3 00:03:18 port-channel4096




The core switch is the mrouter for VLAN 880 and the 3750 stack shows the mrouter for VLAN 880 is the nexus:

IDF-3750X#show ip igmp snooping mrouter vlan 880
Vlan ports
---- -----
880 Po10(dynamic)




IDF-3750X#show ip igmp snooping querier vlan 880
IP address : 172.30.80.2
IGMP version : v3
Port : Po10
Max response time : 10s
Query interval : 125s
Robustness variable : 2




IDF-3750X#show ip igmp snooping querier
Vlan IP Address IGMP Version Port
-------------------------------------------------------------
880 172.30.80.2 v3 Po10


The problem appear to be that not all the groups learned on the 3750 are reported to the Nexus.
On the 3750 I have 224.0.1.129 and 239.255.254.253. Only 239.255.254.253 shows on the Nexus via Po4094.




IDF-3750X#show ip igmp snooping groups vlan 880
Vlan Group Type Version Port List
-----------------------------------------------------------------------
880 224.0.1.129 igmp v2 Gi4/0/11, Gi5/0/38, Po10
880 239.255.254.253 igmp v2 Gi4/0/11, Po10


CORE-2# show ip igmp snooping groups vlan 880
Type: S - Static, D - Dynamic, R - Router port, F - Fabricpath core port

Vlan Group Address Ver Type Port list
880 */* - R Vlan880 Po4096
880 224.0.1.129 v3 D Eth1/9

880 239.255.254.253 v2 D Eth1/9 Po4094

880 239.255.255.76 v3 D Eth1/9

880 239.255.255.250 v2 D Po101 Eth1/9 Po300
Po4091 Po4093

880 239.255.255.255 v3 D Eth1/9

CORE-1# show ip igmp snooping groups vlan 880
Type: S - Static, D - Dynamic, R - Router port, F - Fabricpath core port

Vlan Group Address Ver Type Port list
880 */* - R Vlan880 Po4096
880 224.0.1.129 v3 D Po4096

880 239.255.254.253 v2 D Po4096 Po4094

880 239.255.255.76 v3 D Po4096

880 239.255.255.250 v2 D Po101 Po4093 Po4096
Po300 Po4091

880 239.255.255.255 v3 D Po4096




I did a debug on the 3750 for this group and it shows is forwarding to the router ports but the Nexus never receives it or is silently discarding them.
I have not found a way to do a similar debug on the nexus but I did a capture using an SPAN port of Po4094 and no reports from group 224.0.1.129 are showing. See the screenshot attached.










IDF-3750X(config)#debug ip igmp snooping 224.0.1.129
Feb 4 21:43:32.652: IGMPSN: Received IGMPv2 Report for group 224.0.1.129 received on Vlan 880, port Gi5/0/38
Feb 4 21:43:32.652: IGMPSN: group: Received IGMPv2 report for group 224.0.1.129 from Client 172.30.80.52 received on Vlan 880, port Gi5/0/38
Feb 4 21:43:32.652: IGMPSN: Add v2 group 224.0.1.129 member port Gi5/0/38, on Vlan 880
Feb 4 21:43:32.652: IGMPSN: group: Added port Gi5/0/38 to group 224.0.1.129
Feb 4 21:43:32.652: IGMPSN: group: Forwarding 224.0.1.129 report to router ports
Feb 4 21:45:32.228: IGMPSN: Received IGMPv2 Report for group 224.0.1.129 received on Vlan 880, port Gi5/0/38
Feb 4 21:45:32.228: IGMPSN: group: Received IGMPv2 report for group 224.0.1.129 from Client 172.30.80.52 received on Vlan 880, port Gi5/0/38
Feb 4 21:45:32.228: IGMPSN: Add v2 group 224.0.1.129 member port Gi5/0/38, on Vlan 880
Feb 4 21:45:32.228: IGMPSN: group: Added port Gi5/0/38 to group 224.0.1.129
Feb 4 21:45:32.228: IGMPSN: group: Forwarding 224.0.1.129 report to router ports

Hope you can provide some advice to resolve this.

Thanks

[Georg Pauwen]

Hello,

looking at the debug output:

--> 

IGMPSN

group: Received

IGMPv2

it looks like the Catalyst is sending

v2

reports. Although

v3 and v2

should be backwards compatible, it might be worth trying to run

v2

on all devices, rather than

v3

Not sure if that is feasible in your topology.

[g748437]

I was using

v2

initially and it was the same issue. I changed it to

v3

to see if it could help somehow but it didn't.

[Georg Pauwen]

Hello,

so apparently the version is not the issue. I don't know in how far this is an environment where you can randomly change things, to test, but the IGMP snooping itself might be the problem. Can you turn that off, just to see if it makes a difference ?

[g748437]

I could turn it off to test but we need IGMP snooping enabled to avoid flooding traffic on that VLAN.

If I disable it, there are 2 possible outcomes:

1- the Group Membership is broadcasted and the Nexus should receive it. 

2- the Group Membership is broadcasted but for some reason the Nexus still doesn't receive it. 

The first one is expected to happen while the second one is unlikely. 

If option 1 ends happening and confirms IGMP is not working properly then what? Just leave it disabled? That is not an option since that VLAN spans many more stacks throughout the campus.

[Georg Pauwen]

Hello,

turning IGMP snooping off permanently is definitely a bad idea. The test would only serve to prove that IGMP snooping is actually causing the issue. Let's assume the outcome will be option 1.

IGMP snooping has several parameters, one of them is 'report suppression' which is enabled by default. You could try and enable that:

Nexus5548(config-vlan-config)#ip igmp snooping report-flood all

[g748437]

My understanding of report suppression is that the switch will send only 1 report to the mrouter per group G, instead of sending all the reports from each device connected to the switch that wants to join group G. 

Why applying this on the nexus mrouter? if this is the one that should receive the reports of others that want to join group

224.0.1.129

and currently is not receiving any. 

Should I try on the 3750 instead? 

[g748437]

Also the command you proposed is not available on this Nexus

CORE-1(config-vlan-config)# ip igmp snooping ?
access-group Configures filter policy for groups mentioned in route-map
explicit-tracking Configures Explicit Host tracking for VLAN/BD
fast-leave Configures Fast leave for the VLAN/BD
last-member-query-interval Configures interval between group-specific Query transmissions
limit Limits number of groups that could be joined per interface to <max-grps>
link-local-groups-suppression Configures VLAN/BD link-local groups suppression
minimum-verison Doesn't process IGMP packets of versions below <min-ver>
mrouter Configures static multicast router interface
querier Enables snooping querier
querier-timeout Configures querier timeout for IGMPv2
query-interval Configures interval between query transmission
query-max-response-time Configures MRT for query messages
report-suppression Configures IGMPv1/IGMPv2 Report Suppression for the VLAN/BD
robustness-variable Configures RFC defined Robustness Variable
startup-query-count Configures number of queries sent at startup
startup-query-interval Configures query interval at startup
static-group Configures static group membership
v3-report-suppression Configures IGMPv3 Report Suppression and Proxy Reporting for the VLAN/BD
version Configures IGMP version number for VLAN/BD

CORE-1(config-vlan-config)# ip igmp snooping report-suppression ?
<CR>

[Georg Pauwen]

Hello,

I have just been thinking: could you create a new Vlan, and turn off IGMP snooping for just that one Vlan, in order to test ? That way, the impact on your network is minimal.

The report suppression is just one of several options. There must be some sort of incompatibility between Nexus and IOS. I don't have your topology available for testing, so I cannot go the trial and error route...

[g748437]

Will do to test and let you know.

Thanks 

[g748437]

I created new

VLAN 888

and disabled IGMP snooping on that VLAN across all the Switches I am testing this on:

Diagram goes like this:

multicast_device(172.30.88.150)--IDF-3750X-Stack(Po10)---(Po4094)[MDF-CORE-1=VPC=MDF-CORE-2](Po4095)---(Po3095)[IDF-NEXUS-1=VPC=IDF-NEXUS-2]--3750X---My_Desk_2960---Multicast_device(172.30.88.51)

IDF-NEXUS-1 and IDF-NEXUS-2

are used as Layer2 and they are the same model as the Core ones. When I disabled

IGMP on IDF-NEXUS-1|2 for VLAN 888

it started flooding

224.0.1.129 to the uplink Po3095

Before that, this

Nexus

will also discard|silently drop the group membership report destined to the

mrouter MDF-CORE-1 or MDF-CORE-2

without any logs saying why.

All group memberships are learned fine on all the Catalyst until it gets to the

Nexus

From there the

224.0.1.129

simply disappears.

This particular group

224.0.1.129

is used for PTP traffic and the devices using it should all receive it so they elect a primary clock among themselves. I did not mention the PTP part before since I do think something specific is needed for those packets to be forwarded like any other multicast. But now after all this time without a solution I am not sure. 

I have an SPAN port on the

IDF-Nexus-1

and captured on

Po3095 TX and RX

direction separately. Screenshots attached.

Even without Snooping on the Core SWs I am not able to see

224.0.1.129 from the 172.30.88.150 device

IDF-3750X#show ip igmp snooping vlan 888

Vlan 888:
--------
IGMP snooping : Disabled
IGMPv2 immediate leave : Disabled
Multicast router learning mode : pim-dvmrp
Robustness variable : 2
Last member query count : 2
Last member query interval : 1000







MDF-CORE-1(config)# show ip igmp snooping vlan 888

IGMP Snooping information for vlan 888
IGMP snooping disabled
Lookup mode: IP
Optimised Multicast Flood (OMF) disabled
IGMP querier none
Switch-querier disabled
IGMPv3 Explicit tracking enabled
IGMPv2 Fast leave disabled
IGMPv1/v2 Report suppression enabled
IGMPv3 Report suppression disabled
Link Local Groups suppression enabled
Router port detection using PIM Hellos, IGMP Queries
Number of router-ports: 0
Number of groups: 0




MDF-CORE-2(config)# show ip igmp snooping vlan 888

IGMP Snooping information for vlan 888
IGMP snooping disabled
Lookup mode: IP
Optimised Multicast Flood (OMF) disabled
IGMP querier none
Switch-querier disabled
IGMPv3 Explicit tracking enabled
IGMPv2 Fast leave disabled
IGMPv1/v2 Report suppression enabled
IGMPv3 Report suppression disabled
Link Local Groups suppression enabled
Router port detection using PIM Hellos, IGMP Queries
Number of router-ports: 0
Number of groups: 0




IDF-NEXUS-1(config)# show ip igmp snooping vlan 888

IGMP Snooping information for vlan 888
IGMP snooping disabled
Lookup mode: IP
Optimised Multicast Flood (OMF) disabled
IGMP querier none
Switch-querier disabled
IGMPv3 Explicit tracking enabled
IGMPv2 Fast leave disabled
IGMPv1/v2 Report suppression enabled
IGMPv3 Report suppression disabled
Link Local Groups suppression enabled
Router port detection using PIM Hellos, IGMP Queries
Number of router-ports: 0
Number of groups: 0




IDF-NEXUS-2# show ip igmp snooping vlan 888

IGMP Snooping information for vlan 888
IGMP snooping disabled
Lookup mode: IP
Optimised Multicast Flood (OMF) disabled
IGMP querier none
Switch-querier disabled
IGMPv3 Explicit tracking enabled
IGMPv2 Fast leave disabled
IGMPv1/v2 Report suppression enabled
IGMPv3 Report suppression disabled
Link Local Groups suppression enabled
Router port detection using PIM Hellos, IGMP Queries
Number of router-ports: 0
Number of groups: 0
VLAN vPC function enabled

  

[g748437]

I also found another stack of

3750

where on a different

VLAN 818

the multicast from

224.0.1.129

is learned fine. This VLAN is using Dense mode and the mrouter is not the

Nexus

Diagram is similar :

GW-3750(Po1)--(Po300)[MDF-CORE-1=VPC=MDF-CORE-2](Po4095)---(Po3095)[IDF-NEXUS-1=VPC=IDF-NEXUS-2]--(1/30)Copper-3750X

MDF-CORE-1# show run int vlan 818

interface Vlan818
no shutdown
ip address 169.254.6.235/16
ip igmp version 3
ip igmp query-interval 30




MDF-CORE-1# show ip igmp snooping groups vlan 818
Type: S - Static, D - Dynamic, R - Router port, F - Fabricpath core port

Vlan Group Address Ver Type Port list
818 */* - R Po4095 Po4096
818 224.0.1.129 v3 D Po300 Po4093 Po4096 Po4091 Po101

This debug was captured on the

Core Nexus

2022 Feb 18 15:58:38.673734 igmp: SN: <818> Process a valid IGMP packet, pkttype:v3report(34), iif:Po300
2022 Feb 18 15:58:38.673821 igmp: SN: <818> Processing v3 report with 6 group records, packet-size 56 from 169.254.90.177 on Po300
2022 Feb 18 15:58:38.673944 igmp: SN: <818> Record type: "mode-is-exclude" for group 224.0.1.129, sources count: 0
2022 Feb 18 15:58:38.674002 igmp: SN: <818> Received v3 report: group 224.0.1.129 from 169.254.90.177 on Po300
2022 Feb 18 15:58:38.674050 igmp: SN: <818> Updated oif Po300 for (*, 224.0.1.129) entry
2022 Feb 18 15:58:38.674156 igmp: SN: <818> Processing reportfrom_cfs: 0, on_internal_mcec: 0, im_is_iod_valid: 1im_is_if_up: 1 im_id_ifindex_veth = 0, rc = 0, ifindex = Po300
2022 Feb 18 15:58:38.674263 igmp: SN: <818> Processing reportfrom_cfs: 0, on_internal_mcec: 0, im_is_iod_valid: 1im_is_if_up: 1 im_id_ifindex_veth = 0, rc = 0, ifindex = Po300
2022 Feb 18 15:58:38.674359 igmp: SN: <818> Processing reportfrom_cfs: 0, on_internal_mcec: 0, im_is_iod_valid: 1im_is_if_up: 1 im_id_ifindex_veth = 0, rc = 0, ifindex = Po300
2022 Feb 18 15:58:38.674453 igmp: SN: <818> Processing reportfrom_cfs: 0, on_internal_mcec: 0, im_is_iod_valid: 1im_is_if_up: 1 im_id_ifindex_veth = 0, rc = 0, ifindex = Po300
2022 Feb 18 15:58:38.674516 igmp: SN: <818> Forwarding the packet to router-ports

Not sure why this IGMP packets are seen as valid while the others on

VLAN 888 or 880

(that was the initial issue) are not showing. 

I found a bug on

N7k

that seems similar :

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvk01435?rfs=iqvred

Could the Nexus been seeing the packets as invalid and discarding them? is there a way to check that?

[paul driver]

Hello

Looks like you are running SSM?

Do you have

igmpv3

compatiable applications?
Have you specified

igmpv3/ssm

and the

mc

groups on all interfaces and devcies inpath between FHR/LHR?

sh ip mroute ssm
sh ip mroute
sh ip mroute count
sh ip pim ssm default
sh run | in ssm

[g748437]

Thanks Paul,

There are no routers involved on the path the Multicast packets for 

224.0.1.129 (PTP) 

should take. There are just L2 switches and one L3 Switch (the

nexus MDF-Core-1

that have PIM enabled and it is the mrouter and querier). I am not trying route multicast outside of the VLAN (on neither the one I created for testing

VLAN888

or the one where the problem started

VLAN880

).

The problem is only seen with

224.0.1.129

once it gets to the

Nexus

It doesn't learn the group even when it receives the membership reports (verified by SPAN capture) other multicast group these devices use to communicate are learned fine

(Ex. 239.255.254.253)

  

[paul driver]

Hello
Apologies some confusion on my part, If you have multicast within a single

vlan

then you don't need to enable

pim

are you trying to implement PTP?

Regards IGMP, if you have no mrouter then try apply  snooping queirer or specifying an mrouter port on the lan access switches.

ip igmp snooping querier 
or
ip igmp snooping vlan xx mrouter interface xx