Multicast not working from Cisco ACI to Cat9500

[Nilay Patel]

Problem multicast feed is not coming out from Data Center(server) to Outside network(client) but other direction multicast is working. Multicast tree(routing) are building end to end but only data is not transferring. ((( observation: traffic start initially for 1-2 seconds & stops almost looking like c9500(L3 core) blocking for some Safety. We have seen flag: T comes for few seconds on 9500 and then go away. Also notice RPF failure/ MCAST data drop at QoS/CPU level. but not sure what's casing this issue

Setup: (C9500-48Y4C(Version 17.3.4) (2x L3 Core) connected to 2x Border leaf (Cisco ACI - 4.2(7u) ). 4 links L3 connected connected to ACI 2xBL from cat9500)

Outside of Data Center Cisco ACI multicasting working perfectly fine.

9500-COR-SW-01-A #sh ip mroute | sec 239.192.198.200

(*, 239.192.198.200), 00:01:06/00:03:22, RP 192.168.254.101, flags: S

  Incoming interface: Null, RPF nbr 0.0.0.0

  Outgoing interface list:

    TwentyFiveGigE1/0/27, Forward/Sparse, 00:01:06/00:03:22

(192.168.102.247, 239.192.198.200), 00:00:56/00:02:37, flags:       (Missing flags after few seconds)

  Incoming interface: TwentyFiveGigE1/0/14, RPF nbr 192.168.253.238

  Outgoing interface list:

    TwentyFiveGigE1/0/27, Forward/Sparse, 00:00:56/00:03:22

##########

Plus traffic getting P - Pruned

9500-COR-SW-01-A # sh ip mroute 239.11.2.41
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,
L - Local, P - Pruned, R - RP-bit set, F - Register flag,
T - SPT-bit set, J - Join SPT, M - MSDP created entry, E - Extranet,
X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement,
U - URD, I - Received Source Specific Host Report,
Z - Multicast Tunnel, z - MDT-data group sender,
Y - Joined MDT-data group, y - Sending to MDT-data group,
G - Received BGP C-Mroute, g - Sent BGP C-Mroute,
N - Received BGP Shared-Tree Prune, n - BGP C-Mroute suppressed,
Q - Received BGP S-A Route, q - Sent BGP S-A Route,
V - RD & Vector, v - Vector, p - PIM Joins on route,
x - VxLAN group, c - PFP-SA cache created entry,
* - determined by Assert, # - iif-starg configured on rpf intf,
e - encap-helper tunnel flag, l - LISP Decap Refcnt Contributor
Outgoing interface flags: H - Hardware switched, A - Assert winner, p - PIM Join
Timers: Uptime/Expires
Interface state: Interface, Next-Hop or VCD, State/Mode

(*, 239.11.2.41), 10w3d/stopped, RP 192.168.254.101, flags: SP
Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list: Null

(192.168.220.22, 239.11.2.41), 10w3d/00:02:16, flags: P
Incoming interface: TwentyFiveGigE1/0/14, RPF nbr 192.168.253.238
Outgoing interface list: Null

###############

also in debug(Cat 9500) we are seeing continuous msgs (2x cat 9500 connects to 2x Boarder Leafs Cisco ACI)

 014626:JAN 1 17:36:33.016 EDT: (Data-header) for 192.168.102.247, group 239.192.195.196
014627:JAN 1 17:36:33.016 EDT: PIM(0): Adding register decap tunnel (Tunnel0) as accepting interface of (192.168.102.247, 239.192.195.196)
014628:JAN 1 17:36:33.016 EDT: PIM(0): Send v2 Register-Stop to 192.168.102.1 for 192.168.102.247, group 239.192.195.196
014629:JAN 1 17:36:33.017 EDT: PIM(0): Insert (192.168.102.247,239.192.195.196) join in nbr 192.168.253.238's queue
014630:JAN 1 17:36:33.017 EDT: PIM(0): Building Join/Prune packet for nbr 192.168.253.238
014631:JAN 1 17:36:33.017 EDT: PIM(0): Adding v2 (192.168.102.247/32, 239.192.195.196), S-bit Join
014632:JAN 1 17:36:33.017 EDT: PIM(0): Send v2 join/prune to 192.168.253.238 (TwentyFiveGigE1/0/14)
014633:JAN 1 17:37:22.754 EDT: PIM(0): Received v2 Join/Prune on TwentyFiveGigE1/0/27 from 192.168.253.114, to us
014634:JAN 1 17:37:22.754 EDT: PIM(0): Join-list: (*, 239.192.195.196), RPT-bit set, WC-bit set, S-bit set
014635:JAN 1 17:37:22.754 EDT: PIM(0): Update TwentyFiveGigE1/0/27/192.168.253.114 to (*, 239.192.195.196), Forward state, by PIM *G Join
014636:JAN 1 17:37:22.754 EDT: PIM(0): Update TwentyFiveGigE1/0/27/192.168.253.114 to (192.168.102.247, 239.192.195.196), Forward state, by PIM *G Join
014637:JAN 1 17:37:23.225 EDT: PIM(0): Building Periodic (*,G) Join / (S,G,RP-bit) Prune message for 239.192.195.196
014638:JAN 1 17:37:23.225 EDT: PIM: rp our address
014639:JAN 1 17:37:31.325 EDT: PIM(0): Insert (192.168.102.247,239.192.195.196) join in nbr 192.168.253.238's queue
014640:JAN 1 17:37:31.325 EDT: PIM(0): Building Join/Prune packet for nbr 192.168.253.238
014641:JAN 1 17:37:31.325 EDT: PIM(0): Adding v2 (192.168.102.247/32, 239.192.195.196), S-bit Join
014642:JAN 1 17:37:31.325 EDT: PIM(0): Send v2 join/prune to 192.168.253.238 (TwentyFiveGigE1/0/14)
014643:JAN 1 17:37:34.976 EDT: PIM(0): Received v2 Register on TwentyFiveGigE1/0/6 from 192.168.102.1
014626:JAN 1 17:36:33.016 EDT: (Data-header) for 192.168.102.247, group 239.192.195.196

——-

****also noticed 50-70 routes for SSDP traffic multicast address 239.255.255.250: not sure this is causing issue 

9500-A-L3-SW-01#sh ip mroute 239.255.255.250 count 

Use "show ip mfib count" to get better response time for a large number of mroutes.

IP Multicast Statistics

409 routes using 655080 bytes of memory

121 groups, 2.38 average sources per group

Forwarding Counts: Pkt Count/Pkts per second/Avg Pkt Size/Kilobits per second

Other counts: Total/RPF failed/Other drops(OIF-null, rate-limit etc)




Group: 239.255.255.250, Source count: 207, Packets forwarded: 83320143, Packets received: 83320149

[marce1000]

 - Use this document for troubleshooting : https://www.cisco.com/c/en/us/support/docs/ip/ip-multicast/16450-mcastguide0.html

 M.

[Georg Pauwen]

Hello,

can you post the running config of one of the 'problem' 9500s ? Do you have any sort of storm control configured ?

Other than that, if possible, try and upgrade to one of the recommended IOS versions, 17.3.5 or 17.6.4...

[Nilay Patel]

-

[Nilay Patel]

Any one using multicast between cisco cat 9500 & cisco aci? what's version & general config idea.  - any assistance would be very much appreciated.

[Nilay Patel · Accepted solution]

Thanks to Cisco TAC (we have it IP Data-Plane Learning Disabled)

L3 Multicast destined external from ACI dropped when IP Data-Plane Learning Disabled
CSCvw03177  

Conditions: This issue occurs when IP data plane learning is disabled under the VRF instance.

"

[+] when the src is behind the epg, we depend on the endpoint manager to learn the multicast source, when dp is disable we don’t learn the entry by any traffic we depend only on arp. Hence no ep learning on the src host no forwarding from the first hop router:

<ServerLeaf> - <Spine> - <BorderLeaf> - <External ACI Devices>

"

[paul driver · Accepted solution]

Hello
If you have RPF failure then, this means the mc traffic on its received (upstream) interface has failed the rtrs global route table check as such the mc source/rp is showing in the rtrs rib a alternative route/path other than the interface the mc traffic arrive on, 

So yoiu need to enable pim on this other interface or add a static mroute for RPF to succeed

 As  for those SSDP packets they relate to devices advertising their availability for UPnP , I had to look this one up myself the other day -here

So you may be able to negate these by a simple access-list applied to the routed interfaces in path of your mc tree.

deny SSDP - 
ip access-list extended DENy_SSDP
remark Block SSDP
deny ip any host 239.255.255.250 
permit ip any any


interface x
ip access-group DENY_SSDP in 
end

[Nilay Patel]

Yes SSDP part fixed. I tested. Thanks a lot. 

I still need to test "IP Data-Plane Learning"