Re: Problem getting IPV4 Multicast working over a GRE Tunnel

richard-artevea · ‎06-10-2012

I am currently stuck in troubleshooting a problem of multicast data not working over a GRE tunnel joining two networks; I have re-created the problem in a very simple lab network having just two Cisco routers.

I suspect that I have overlooked something in the configuration of the C1921 - ideas/suggestions welcome :-)

Lab network details

-------------------
Host A <---> Router A <------ IP Cloud ------> Router B <---> Host B

Hosts A and B are PCs running Windows XP Pro
Router A is an old C3620 I had lying around in the lab
Router B is a C1921 from the network in the field I'm having the troube with.

The IP Cloud is a routed network that is not multicast-capable; in the lab this is simulated by an old Linksys cable router having its two Ethernet ports in different subnets. In the field it is provided by a third party telecoms provider (C1800 series routers with Ethernet ports); I do not have any administrative access to their network, but if I needed to make a configuration change and knew exactly what to ask for I could have it done by them...

The problem experienced in the field and in the lab is that:
1. The GRE Tunnel I have created appears to work perfectly for unicast traffic
2. Depending upon where I attempt to put the PIM rendezvous point, multicast traffic does not flow at all, or flows only in one direction.

I tried three tests on the lab system:
In each test, Host A (192.168.200.10) and Host B (192.168.210.50) ran the old Microsoft tool 'MPING.EXE' that sent out packets on the group 225.100.1.1 (and if it received a packet from another sender, would echo it back)

Test #1 - No RP address defined; everything running in dense-mode
-----------------------------------------------------------------
Result: One-way communication - Host A received only the first packet from Host B, Host B received all packets from Host A

While the test was running, the output of 'show ip mroute 225.100.1.1' was:

At Router A
-----------
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,
       L - Local, P - Pruned, R - RP-bit set, F - Register flag,
       T - SPT-bit set, J - Join SPT, M - MSDP created entry,
       X - Proxy Join Timer Running, A - Advertised via MSDP, U - URD,
       I - Received Source Specific Host Report
Outgoing interface flags: H - Hardware switched
Timers: Uptime/Expires
Interface state: Interface, Next-Hop or VCD, State/Mode

(*, 225.100.1.1), 00:00:49/00:02:59, RP 0.0.0.0, flags: DJC
Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list:
FastEthernet0/0, Forward/Sparse-Dense, 00:00:38/00:00:00
Tunnel0, Forward/Sparse-Dense, 00:00:49/00:00:00

(192.168.200.50, 225.100.1.1), 00:00:37/00:02:59, flags: CTA
Incoming interface: FastEthernet0/0, RPF nbr 0.0.0.0
Outgoing interface list:
Tunnel0, Forward/Sparse-Dense, 00:00:37/00:00:00

(192.168.210.10, 225.100.1.1), 00:00:49/00:02:59, flags: CTA
Incoming interface: Tunnel0, RPF nbr 172.16.0.2
Outgoing interface list:
FastEthernet0/0, Forward/Sparse-Dense, 00:00:39/00:00:00

At Router B
-----------
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,
       L - Local, P - Pruned, R - RP-bit set, F - Register flag,
       T - SPT-bit set, J - Join SPT, M - MSDP created entry, E - Extranet,
       X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement,
       U - URD, I - Received Source Specific Host Report,
       Z - Multicast Tunnel, z - MDT-data group sender,
       Y - Joined MDT-data group, y - Sending to MDT-data group,
       V - RD & Vector, v - Vector
Outgoing interface flags: H - Hardware switched, A - Assert winner
Timers: Uptime/Expires
Interface state: Interface, Next-Hop or VCD, State/Mode

(*, 225.100.1.1), 00:02:07/stopped, RP 0.0.0.0, flags: DC
Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list:
GigabitEthernet0/0, Forward/Sparse-Dense, 00:02:07/stopped
Tunnel10, Forward/Sparse-Dense, 00:02:07/stopped

(192.168.200.50, 225.100.1.1), 00:01:54/00:01:05, flags: T
Incoming interface: Tunnel10, RPF nbr 172.16.0.1
Outgoing interface list:
GigabitEthernet0/0, Forward/Sparse-Dense, 00:01:54/stopped

(192.168.210.10, 225.100.1.1), 00:02:06/00:00:53, flags: T
Incoming interface: GigabitEthernet0/0, RPF nbr 0.0.0.0
Outgoing interface list:
Tunnel10, Forward/Sparse-Dense, 00:01:55/stopped

Test #2 - Sparse-mode, with the RP created in Router A
------------------------------------------------------
Interface Loopback1 (10.10.10.10) created in Router A; ip pim rp-address commands entered in both routers poiting at it.

Result: No communication in either direction. Output of 'show ip mroute' command at Router B suggests a problem of registering with the RP.

While the test was running, the output of 'show ip mroute 225.100.1.1' was:

At Router A
-----------
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,
       L - Local, P - Pruned, R - RP-bit set, F - Register flag,
       T - SPT-bit set, J - Join SPT, M - MSDP created entry,
       X - Proxy Join Timer Running, A - Advertised via MSDP, U - URD,
       I - Received Source Specific Host Report
Outgoing interface flags: H - Hardware switched
Timers: Uptime/Expires
Interface state: Interface, Next-Hop or VCD, State/Mode

(*, 225.100.1.1), 00:00:35/00:02:59, RP 10.10.10.10, flags: SJC
Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list:
Tunnel0, Forward/Sparse-Dense, 00:00:31/00:02:58
FastEthernet0/0, Forward/Sparse-Dense, 00:00:35/00:02:25

(192.168.200.50, 225.100.1.1), 00:00:34/00:02:59, flags: CTA
Incoming interface: FastEthernet0/0, RPF nbr 0.0.0.0
Outgoing interface list:
Tunnel0, Forward/Sparse-Dense, 00:00:31/00:02:28

At Router B
-----------
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,
       L - Local, P - Pruned, R - RP-bit set, F - Register flag,
       T - SPT-bit set, J - Join SPT, M - MSDP created entry, E - Extranet,
       X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement,
       U - URD, I - Received Source Specific Host Report,
       Z - Multicast Tunnel, z - MDT-data group sender,
       Y - Joined MDT-data group, y - Sending to MDT-data group,
       V - RD & Vector, v - Vector
Outgoing interface flags: H - Hardware switched, A - Assert winner
Timers: Uptime/Expires
Interface state: Interface, Next-Hop or VCD, State/Mode

(*, 225.100.1.1), 00:01:50/stopped, RP 10.10.10.10, flags: SJCF
Incoming interface: Tunnel10, RPF nbr 172.16.0.1
Outgoing interface list:
GigabitEthernet0/0, Forward/Sparse-Dense, 00:01:50/00:02:48

(192.168.210.10, 225.100.1.1), 00:01:49/00:01:10, flags: PFT
Incoming interface: GigabitEthernet0/0, RPF nbr 0.0.0.0, Registering
Outgoing interface list: Null

Test #3 - Sparse-mode, with the RP created in Router B
------------------------------------------------------
Interface Loopback1 (10.10.10.10) deleted from Router A and then created in Router B; ip pim rp-address commands left defined in both routers poiting at it.

Result: One-way communication; host A received all packets from Host B and echoed them back; Host B received nothing.

While the test was running, the output of 'show ip mroute 225.100.1.1' was:

At Router A
-----------
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,
       L - Local, P - Pruned, R - RP-bit set, F - Register flag,
       T - SPT-bit set, J - Join SPT, M - MSDP created entry,
       X - Proxy Join Timer Running, A - Advertised via MSDP, U - URD,
       I - Received Source Specific Host Report
Outgoing interface flags: H - Hardware switched
Timers: Uptime/Expires
Interface state: Interface, Next-Hop or VCD, State/Mode

(*, 225.100.1.1), 00:00:31/00:02:59, RP 10.10.10.10, flags: SJCF
Incoming interface: Tunnel0, RPF nbr 172.16.0.2
Outgoing interface list:
FastEthernet0/0, Forward/Sparse-Dense, 00:00:31/00:02:29

(192.168.200.50, 225.100.1.1), 00:00:30/00:03:29, flags: CFT
Incoming interface: FastEthernet0/0, RPF nbr 0.0.0.0
Outgoing interface list:
Tunnel0, Forward/Sparse-Dense, 00:00:30/00:03:02

(192.168.210.10, 225.100.1.1), 00:00:26/00:02:59, flags: CJT
Incoming interface: Tunnel0, RPF nbr 172.16.0.2
Outgoing interface list:
FastEthernet0/0, Forward/Sparse-Dense, 00:00:26/00:02:33

At Router B
-----------
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,
       L - Local, P - Pruned, R - RP-bit set, F - Register flag,
       T - SPT-bit set, J - Join SPT, M - MSDP created entry, E - Extranet,
       X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement,
       U - URD, I - Received Source Specific Host Report,
       Z - Multicast Tunnel, z - MDT-data group sender,
       Y - Joined MDT-data group, y - Sending to MDT-data group,
       V - RD & Vector, v - Vector
Outgoing interface flags: H - Hardware switched, A - Assert winner
Timers: Uptime/Expires
Interface state: Interface, Next-Hop or VCD, State/Mode

(*, 225.100.1.1), 00:01:24/00:03:05, RP 10.10.10.10, flags: SJC
Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list:
GigabitEthernet0/0, Forward/Sparse-Dense, 00:01:20/00:02:41
Tunnel10, Forward/Sparse-Dense, 00:01:24/00:03:05

(192.168.210.10, 225.100.1.1), 00:01:19/00:01:40, flags: T
Incoming interface: GigabitEthernet0/0, RPF nbr 0.0.0.0
Outgoing interface list:
Tunnel10, Forward/Sparse-Dense, 00:01:19/00:03:05

(192.168.200.50, 225.100.1.1), 00:01:23/00:01:38, flags: T
Incoming interface: Tunnel10, RPF nbr 172.16.0.1
Outgoing interface list:
GigabitEthernet0/0, Forward/Sparse-Dense, 00:01:20/00:02:41

Extracts from the router running-configs showing how I created the GRE tunnel:

Router A (C3620)
----------------
interface Loopback0
ip address 192.168.200.224 255.255.255.255

interface Tunnel0
ip address 172.16.0.1 255.255.255.252
ip pim sparse-dense-mode
ip ospf network point-to-point
tunnel source FastEthernet0/1
tunnel destination 192.168.250.2

interface FastEthernet0/0
ip address 192.168.200.1 255.255.255.128
ip pim sparse-dense-mode

interface FastEthernet0/1
ip address 192.168.251.2 255.255.255.0

router ospf 10
redistribute static subnets
network 0.0.0.0 255.255.255.255 area 0

ip route 192.168.200.128 255.255.255.248 192.168.200.66
ip route 192.168.250.0 255.255.255.0 192.168.251.1

Router B (C1921)
----------------
interface Loopback0
ip address 192.168.210.224 255.255.255.255

interface Tunnel10
ip address 172.16.0.2 255.255.255.252
ip pim sparse-dense-mode
ip ospf network point-to-point
tunnel source GigabitEthernet0/1
tunnel destination 192.168.251.2

interface GigabitEthernet0/0
ip address 192.168.210.1 255.255.255.128
ip pim sparse-dense-mode

interface GigabitEthernet0/1
ip address 192.168.250.2 255.255.255.0

router ospf 10
redistribute static subnets
network 0.0.0.0 255.255.255.255 area 0

ip route 192.168.210.128 255.255.255.240 192.168.210.66
ip route 192.168.251.0 255.255.255.0 192.168.250.1

Thanks,

Richard Culpan - Artevea Digital Ltd.

John Blakley · ‎06-10-2012

Richard,

It looks like you could be having some rpf failures. To me, the traffic looks like it's going over the tunnel, but the path back to the source is not going over the tunnel:

Router A:

(192.168.200.50, 225.100.1.1), 00:00:37/00:02:59, flags: CTA
Incoming interface: FastEthernet0/0, RPF nbr 0.0.0.0
Outgoing interface list:
Tunnel0, Forward/Sparse-Dense, 00:00:37/00:00:00

(192.168.210.10, 225.100.1.1), 00:00:49/00:02:59, flags: CTA
Incoming interface: Tunnel0, RPF nbr 172.16.0.2
Outgoing interface list:
FastEthernet0/0, Forward/Sparse-Dense, 00:00:39/00:00:00

The quick way to fix this, if this is truly the cause, is to add a static multicast route on both routers pointing back to their real ip address:

For example, if the source at A is address 192.168.200.10, then on RouterB your route would be:

ip mroute 192.168.200.10 255.255.255.255 tu0

And do the same on the other side.

HTH,

John

HTH, John *** Please rate all useful posts ***

richard-artevea · ‎06-10-2012

Thanks for the quick reply...

It didn't seem to be an RPF failure as the path back to the source was known via OSPF; 'show ip rpf ' was showing the expected path back to that sender.

But in a few hours of tinkering I did eventually come across the real culprit! It was multicast fast-switching. The old C3620 and 2611 routers don't seem to like fast-switching ('ip mroute-cache') on the tunnel interface - turning it off ('no ip mroute-cache' ) fixed the problem. FWIW I noticed that this is also true of legacy serial interfaces (e.g. X.21) - there was a 'no ip mroute-cache' put into the running configuration by default, but not for the tunnel interface.

Network in the field now running with a single static RP (equivalent to 'Router A' in my lab system) and multicast now goes both ways through the tunnel :-)

I'll revisit it another day and try to improve things (e.g. add some resilience by having an "anycast" RP or running separate RPs each side of the tunnel and MSDP to link them together)

Regards,

Richard Culpan - Artevea Digital Ltd.