Multicast Dilemmas. Help!

Andrew WEISS · ‎11-06-2014

Hi all, we've been experiencing some multicast problems and I'm hopeful that the experts out there can give me a hand figuring them out before I decide to throw in the towel and put in place GRE tunneling.

Some facts about our network:

Each of our PE has at least 3 uplinks to at least 2 different Pops.
Each PE receives about 90k BGP routes from our 4 route-reflectors with a default route (0/0) for any missing specific routes.
Each PE receives about 3k multicast routes from our 4 route-reflectors with a default route (0/0) for any missing specific routes.
ISISI is our IGP (L1 only, multitopology -- IPv6 unicast/multicast is running as well on all PE with the full ipv6 table (18k))
RFP check is running on all backbone line in loose mode (ip verify unicast source reachable-via any)
PIM sparse mode is active on all backbone links.
There are multiple paths with equal cost metrics on all backbone links and inbound/outbound traffic is not symmetrical (...yes I know)
There are 2 anycast RP (one in the north and the second in the south). IGP metrics determine which is used.

The problem...

One of our clients it trying to join a group but he's having problems doing so. The configuration of his interface is correct.

From the PE he is connected to as well as within our lab, I can join the group that he is interested in on the router using igmp:

---------------

LAB-NOC-7200-GATEWAY(config-subif)#ip igmp join-group 233.29.151.44

LAB-NOC-7200-GATEWAY#sh ip igmp groups

IGMP Connected Group Membership

Group Address Interface Uptime Expires Last Reporter Group Accounted

233.29.151.44 GigabitEthernet0/2.431 00:00:10 00:02:49 193.51.177.0

224.0.1.40 GigabitEthernet0/2.431 1w1d 00:02:58 193.51.177.0

--------------------------------------------------------

The MROUTE results are coherent:

-------------------------------------------------------

LAB-NOC-7200-GATEWAY#sh ip mroute

IP Multicast Routing Table

Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,

L - Local, P - Pruned, R - RP-bit set, F - Register flag,

T - SPT-bit set, J - Join SPT, M - MSDP created entry,

X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement,

U - URD, I - Received Source Specific Host Report,

Z - Multicast Tunnel, z - MDT-data group sender,

Y - Joined MDT-data group, y - Sending to MDT-data group,

V - RD & Vector, v - Vector

Outgoing interface flags: H - Hardware switched, A - Assert winner

Timers: Uptime/Expires

Interface state: Interface, Next-Hop or VCD, State/Mode

(*, 233.29.151.44), 00:02:48/stopped, RP 193.51.178.254, flags: SJPL

Incoming interface: Tunnel650580, RPF nbr 193.51.177.6

Outgoing interface list: Null

(202.158.196.68, 233.29.151.44), 00:01:39/00:02:57, flags: PLT

Incoming interface: Tunnel650580, RPF nbr 193.51.177.6, Mbgp

Outgoing interface list: Null

(202.158.195.196, 233.29.151.44), 00:01:41/00:02:58, flags: PLT

Incoming interface: Tunnel650580, RPF nbr 193.51.177.6, Mbgp

Outgoing interface list: Null

(129.78.221.17, 233.29.151.44), 00:01:43/00:02:59, flags: PLT

Incoming interface: Tunnel650580, RPF nbr 193.51.177.6, Mbgp

Outgoing interface list: Null

(202.6.112.4, 233.29.151.44), 00:01:46/00:02:57, flags: PLT

Incoming interface: Tunnel650580, RPF nbr 193.51.177.6, Mbgp

Outgoing interface list: Null

(202.158.196.134, 233.29.151.44), 00:01:46/00:02:57, flags: PLT

Incoming interface: Tunnel650580, RPF nbr 193.51.177.6, Mbgp

Outgoing interface list: Null

(202.158.195.132, 233.29.151.44), 00:01:48/00:02:55, flags: PLT

Incoming interface: Tunnel650580, RPF nbr 193.51.177.6, Mbgp

Outgoing interface list: Null

(202.158.196.4, 233.29.151.44), 00:01:48/00:02:58, flags: PLT

Incoming interface: Tunnel650580, RPF nbr 193.51.177.6, Mbgp

Outgoing interface list: Null

(202.158.195.68, 233.29.151.44), 00:01:48/00:02:53, flags: PLT

Incoming interface: Tunnel650580, RPF nbr 193.51.177.6, Mbgp

Outgoing interface list: Null

(138.77.139.71, 233.29.151.44), 00:01:48/00:02:53, flags: PLT

Incoming interface: Tunnel650580, RPF nbr 193.51.177.6, Mbgp

Outgoing interface list: Null

(138.77.155.72, 233.29.151.44), 00:01:48/00:02:58, flags: PLT

Incoming interface: Tunnel650580, RPF nbr 193.51.177.6, Mbgp

Outgoing interface list: Null

(131.172.135.210, 233.29.151.44), 00:01:49/00:02:58, flags: PLT

Incoming interface: Tunnel650580, RPF nbr 193.51.177.6, Mbgp

Outgoing interface list: Null

(203.27.221.247, 233.29.151.44), 00:01:49/00:02:59, flags: PLT

Incoming interface: Tunnel650580, RPF nbr 193.51.177.6, Mbgp

Outgoing interface list: Null

(131.172.8.214, 233.29.151.44), 00:01:49/00:02:56, flags: PLT

Incoming interface: Tunnel650580, RPF nbr 193.51.177.6, Mbgp

Outgoing interface list: Null

(202.158.195.4, 233.29.151.44), 00:01:51/00:02:58, flags: PLT

Incoming interface: Tunnel650580, RPF nbr 193.51.177.6, Mbgp

Outgoing interface list: Null

(195.220.53.138, 233.29.151.44), 00:01:51/00:02:59, flags: PLT

Incoming interface: Tunnel650580, RPF nbr 193.51.177.6, Mbgp

Outgoing interface list: Null

(134.129.95.122, 233.29.151.44), 00:01:51/00:02:58, flags: PLT

Incoming interface: Tunnel650580, RPF nbr 193.51.177.6, Mbgp

Outgoing interface list: Null

(138.77.147.71, 233.29.151.44), 00:01:51/00:02:59, flags: PLT

Incoming interface: Tunnel650580, RPF nbr 193.51.177.6, Mbgp

Outgoing interface list: Null

(*, 224.0.1.40), 1w1d/00:02:17, RP 0.0.0.0, flags: DPL

Incoming interface: Null, RPF nbr 0.0.0.0

Outgoing interface list: Null

------------------------------------------------

The mulitcast packets arrive on our lab 7200 and packets received increase.

------------------------------------------------------

LAB-NOC-7200-GATEWAY#sh ip mroute count

IP Multicast Statistics

19 routes using 6536 bytes of memory

2 groups, 8.50 average sources per group

Forwarding Counts: Pkt Count/Pkts(neg(-) = Drops) per second/Avg Pkt Size/Kilobits per second

Other counts: Total/RPF failed/Other drops(OIF-null, rate-limit etc)

Group: 233.29.151.44, Source count: 17, Packets forwarded: 0, Packets received: 8248

RP-tree: Forwarding: 0/0/0/0, Other: 0/0/0

Source: 202.158.195.132/32, Forwarding: 0/-1/0/0, Other: 398/0/398

Source: 202.158.195.4/32, Forwarding: 0/-2/0/0, Other: 499/0/499

Source: 138.77.147.71/32, Forwarding: 0/-1/0/0, Other: 630/0/630

Source: 129.78.221.17/32, Forwarding: 0/-1/0/0, Other: 450/0/450

Source: 202.6.112.4/32, Forwarding: 0/-2/0/0, Other: 492/0/492

Source: 202.158.195.196/32, Forwarding: 0/-3/0/0, Other: 468/0/468

Source: 202.158.195.68/32, Forwarding: 0/-1/0/0, Other: 397/0/397

Source: 195.220.53.138/32, Forwarding: 0/-4/0/0, Other: 539/0/539

Source: 131.172.135.210/32, Forwarding: 0/-2/0/0, Other: 458/0/458

Source: 203.27.221.247/32, Forwarding: 0/-1/0/0, Other: 398/0/398

Source: 138.77.139.71/32, Forwarding: 0/-1/0/0, Other: 398/0/398

Source: 131.172.8.214/32, Forwarding: 0/-1/0/0, Other: 537/0/537

Source: 134.129.95.122/32, Forwarding: 0/-2/0/0, Other: 748/0/748

Source: 138.77.155.72/32, Forwarding: 0/-2/0/0, Other: 443/0/443

Source: 202.158.196.134/32, Forwarding: 0/-1/0/0, Other: 473/0/473

Source: 202.158.196.4/32, Forwarding: 0/-1/0/0, Other: 458/0/458

Source: 202.158.196.68/32, Forwarding: 0/-2/0/0, Other: 462/0/462

Group: 224.0.1.40, Source count: 0, Packets forwarded: 0, Packets received: 0

------------------------------------------------------DEBUG-----------------------------------

LAB-NOC-7200-GATEWAY#debug ip mpacket

IP multicast packets debugging is on

LAB-NOC-7200-GATEWAY#

LAB-NOC-7200-GATEWAY#term

LAB-NOC-7200-GATEWAY#terminal moni

LAB-NOC-7200-GATEWAY#terminal monitor

LAB-NOC-7200-GATEWAY# (Tunnel650580) d=233.29.151.44 id=0, ttl=113, prot=17, len=40(40), mroute olist null

Nov 6 11:15:13: IP(0): s=203.27.221.247 (Tunnel650580) d=233.29.151.44 id=0, ttl=109, prot=17, len=547(547), mroute olist null

Nov 6 11:15:13: IP(0): s=202.158.196.134 (Tunnel650580) d=233.29.151.44 id=0, ttl=113, prot=17, len=40(40), mroute olist null

Nov 6 11:15:13: IP(0): s=202.158.196.68 (Tunnel650580) d=233.29.151.44 id=0, ttl=111, prot=17, len=170(170), mroute olist null

Nov 6 11:15:13: IP(0): s=195.220.53.138 (Tunnel650580) d=233.29.151.44 id=1682, ttl=116, prot=17, len=81(81), mroute olist null

Nov 6 11:15:13: IP(0): s=131.172.135.210 (Tunnel650580) d=233.29.151.44 id=2118, ttl=109, prot=17, len=550(550), mroute olist null

Nov 6 11:15:13: IP(0): s=134.129.95.122 (Tunnel650580) d=233.29.151.44 id=43998, ttl=112, prot=17, len=40(40), mroute olist null

Nov 6 11:15:13: IP(0): s=129.78.221.17 (Tunnel650580) d=233.29.151.44 id=0, ttl=106, prot=17, len=97(97), mroute olist null

Nov 6 11:15:13: IP(0): s=134.129.95.122 (Tunnel650580) d=233.29.151.44 id=110, ttl=112, prot=17, len=40(40), mroute olist null

Nov 6 11:15:13: IP(0): s=134.129.95.122 (Tunnel650580) d=233.29.151.44 id=19326, ttl=112, prot=17, len=40(40), mroute olist null

Nov 6 11:15:13: IP(0): s=134.129.95.122 (Tunnel650580) d=233.29.151.44 id=16680, ttl=112, prot=17, len=62(62), mroute olist null

---------------------------------------------------------------END DEBUG-------------------------------------------------------------------

However, if I mtrace the destination I get stuck on our paris1 router.

------------------------------------------------------MTRACE-------------------------------------

LAB-NOC-7200-GATEWAY#mtrace 202.6.112.4 193.51.178.198 233.29.151.44

Type escape sequence to abort.

Mtrace from 202.6.112.4 to 193.51.178.198 via group 233.29.151.44

From source (?) to destination (?)

Querying full reverse path... * switching to hop-by-hop:

0 193.51.178.198

-1 193.51.178.198 PIM/MBGP [202.6.112.0/24]

-2 tu650580-paris1-rtr-032.noc.renater.fr (193.51.177.6) PIM [202.6.112.0/24]

-3 * * * gi8-3-paris1-rtr-021.noc.renater.fr (193.51.177.154)

----------------------------------------------

On paris1 the routing is fine, or so it would seem

-----------------------------------------------Paris1 routing-----------------------------------------------------------------------

paris1-rtr-021#sh ip mroute 233.29.151.44

IP Multicast Routing Table

Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,

L - Local, P - Pruned, R - RP-bit set, F - Register flag,

T - SPT-bit set, J - Join SPT, M - MSDP created entry, E - Extranet,

X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement,

U - URD, I - Received Source Specific Host Report,

Z - Multicast Tunnel, z - MDT-data group sender,

Y - Joined MDT-data group, y - Sending to MDT-data group,

V - RD & Vector, v - Vector

Outgoing interface flags: H - Hardware switched, A - Assert winner

Timers: Uptime/Expires

Interface state: Interface, Next-Hop or VCD, State/Mode

(*, 233.29.151.44), 00:08:37/00:02:43, RP 193.51.178.254, flags: S

Incoming interface: GigabitEthernet8/8, RPF nbr 193.51.177.159

Outgoing interface list:

GigabitEthernet8/3, Forward/Sparse, 00:08:37/00:02:43

(202.158.206.38, 233.29.151.44), 00:00:44/00:02:45, flags: