PIM-bidir not forwarding upstream traffic

[joseph.y.chen1982]

Good evening all,
This is my first time posting, and was just wondering if anyone has run into an issue where PIM-bidir doesn't forward traffic in one direction.  I'm working on a deployment where all multicast sources are also receivers; bidir seemed like the best choice in this application.  I'm seeing some strange behavior where multicast traffic is passed downstream, but not upstream.  There are four devices in this simple setup, BD1<-->BE1<-->L2 SWITCH<-->JC1.  The problematic upstream traffic is generated from BD1 (10.92.13.51), passes up to BE1, but then gets squashed with an mroute olist null.  Group in question is 239.0.8.101:

*Mar  2 21:29:33.205: IP(0): s=10.92.13.51 (Vlan3334) d=239.0.8.101 id=21251, ttl=8, prot=17, len=98(84), mroute olist null
*Mar  2 21:29:33.230: IP(0): s=10.92.13.51 (Vlan3334) d=239.0.8.101 id=21252, ttl=8, prot=17, len=214(200), mroute olist null
*Mar  2 21:29:33.255: IP(0): s=10.92.13.51 (Vlan3334) d=239.0.8.101 id=21253, ttl=8, prot=17, len=214(200), mroute olist null
*Mar  2 21:29:33.272: IP(0): s=10.92.13.51 (Vlan3334) d=239.0.8.101 id=21254, ttl=8, prot=17, len=214(200), mroute olist null
*Mar  2 21:29:33.297: IP(0): s=10.92.13.51 (Vlan3334) d=239.0.8.101 id=21255, ttl=8, prot=17, len=214(200), mroute olist null

show ip mroute 239.0.8.101 output from BE1:
(*, 239.0.8.101), 00:34:17/00:03:25, RP 10.92.99.6, flags: B
  Bidir-Upstream: Null, RPF nbr 0.0.0.0
  Outgoing interface list:
    GigabitEthernet0/1, Forward/Sparse-Dense, 00:33:14/00:02:34
    Vlan3334, Forward/Sparse-Dense, 00:01:03/00:03:25

show ip mroute 239.0.8.101 output from BD1:
(*, 239.0.8.101), 00:02:19/00:02:27, RP 10.92.99.6, flags: BC
  Bidir-Upstream: GigabitEthernet0/0, RPF nbr 192.168.100.73
  Outgoing interface list:
    GigabitEthernet1/0.130, Forward/Sparse-Dense, 00:02:19/00:02:27
    GigabitEthernet0/0, Bidir-Upstream/Sparse-Dense, 00:02:19/stopped
 

show ip mroute 239.0.8.101 output from JC1:
(*, 239.0.8.101), 00:35:16/00:02:11, RP 10.92.99.6, flags: BC
  Bidir-Upstream: GigabitEthernet0/1, RPF nbr 10.92.100.6
  Outgoing interface list:
    Vlan80, Forward/Sparse-Dense, 00:35:16/00:02:11
    GigabitEthernet0/1, Bidir-Upstream/Sparse-Dense, 00:35:16/stopped

PIM-SM works fine with the same setup; so I'm fairly certain that the auto-rp config  and group mappings are good.  Any insight would be greatly appreciated!  Thanks in advance!

[joseph.y.chen1982]

Sorry, forgot to mention that BD1 and JC1 are 2851 routers on Version 15.1(4)M7 and BE1 is a 3560 on 15.0(2)SE1.  The L2 switch is just a 2950, one vlan for all ports, nothing crazy there.

[Peter Paluch]

Joe,

First of all, the RP for this Bidir group is 10.92.99.6. What router is this? Based on the output of the show ip mroute, it would seem that BE1 is this RP, based on the "Bidir-Upstream: Null, RPF nbr 0.0.0.0" output in its Bidir-Upstream interface. Is that right? If not then BE1 is missing the route to this IP address in its routing table and that is definitely one problem requiring correction.

In addition, the packet is sourced from 10.92.13.51 and according to the debug outputs you have posted, it arrived over the interface Vlan3334. Does the show ip route 10.92.13.51 and show ip rpf 10.92.13.51 agree that the proper RPF interface back to this source is the Vlan3334?

Yet another thing: On interface Vlan3334, is BE1 elected as the Designated Forwarder? Please check the output of the show ip pim interface df on BE1 to verify.

Looking forward to hearing from you.

Best regards,
Peter

[joseph.y.chen1982]

Hi Peter,

Thank you for taking the time to look into this, much appreciated.  In response to your inquiries:
-Yes, that's right, the RP for the the bidir group 239.0.8.101 is BE1, 10.92.99.6 (sorry, should have posted that with the original)
-Yes, the show ip route 10.92.13.51 and show ip rpf 10.92.13.51 issued from BE1agree:
BE1#sh ip route 10.92.13.51
Routing entry for 10.92.13.0/24
  Known via "ospf 1", distance 110, metric 26, type intra area
  Last update from 192.168.100.74 on Vlan3334, 2d09h ago
  Routing Descriptor Blocks:
  * 192.168.100.74, from 10.92.13.1, 2d09h ago, via Vlan3334
      Route metric is 26, traffic share count is 1
BE1#sh ip rpf 10.92.13.51
RPF information for ? (10.92.13.51)
  RPF interface: Vlan3334
  RPF neighbor: ? (192.168.100.74)
  RPF route/mask: 10.92.13.0/24
  RPF type: unicast (ospf 1)
  Doing distance-preferred lookups across tables
  RPF topology: ipv4 multicast base, originated from ipv4 unicast base

-Yes, BE1 has been elected the DF on interface Vlan3334:
Berakas#sh ip pim int df
* implies this system is the DF
Interface                RP               DF Winner        Metric     Uptime
Loopback0                10.92.99.2       *10.92.99.6       11         11:57:49
                                 10.92.99.6       *10.92.99.6       0          11:58:17
GigabitEthernet0/1    10.92.99.2        10.92.100.2      0          11:57:14
                                 10.92.99.6       *10.92.100.6      0          11:57:14
Vlan3334                 10.92.99.2       *192.168.100.73   11         11:56:56
                                 10.92.99.6       *192.168.100.73   0          11:56:56

I'm at a bit of a loss as to why it's only operating unidirectional, unless I've totally missed something; the config seems all right though.  In this test setup, there aren't any redundant paths to destinations and aforementioned in the original post, as soon as the bidir is disabled throughout, the traffic is forwarded bidirectionally. 

Thanks again!

Joe

[Peter Paluch]

Hi Joe,

Hmmm, this is strange indeed. If you don't mind, post please the configuration of BE1 after removing sensitive information. However, we're truly looking for something rather discrete.

A blind shot: Would you mind replacing the ip pim sparse-dense-mode with ip pim sparse-mode on all interfaces? The presence of the dense mode fallback doesn't sound right to me.

Best regards,
Peter

[joseph.y.chen1982]

Hi Peter,

Thanks for the expedient response.  As requested:

BE1#sh run
Building configuration...

Current configuration : 8225 bytes
!
version 15.0
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname BE1
!
boot-start-marker
boot-end-marker
!
!
aaa new-model
!
!
aaa authentication login default local
aaa authorization exec default local
!
!
!
!
!
!
aaa session-id common
clock timezone SST 8 0
system mtu routing 1500
vtp domain F8
vtp mode off
!
shutdown vlan 6
ip routing
no ip domain-lookup
!
!
!
ip multicast-routing distributed
!
!
!
!
!
!
!
!
!
spanning-tree mode rapid-pvst
spanning-tree extend system-id
no spanning-tree vlan 1,60,605,607,609,611-612,3334
!
vlan internal allocation policy ascending
!
vlan 6
 name PARKING_VLAN-SHUT
 shutdown
!
vlan 60
!
!
vlan 3334
 name BD1
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
interface Loopback0
 ip address 10.92.99.6 255.255.255.255
 ip pim sparse-dense-mode
 ip ospf network point-to-point
 ip ospf 1 area 10.92.6.1
!
interface GigabitEthernet0/1
 description UPLINK TO FIBER
 no switchport
 ip address 10.92.100.6 255.255.255.240
 ip pim dr-priority 150
 ip pim sparse-dense-mode
 no ip mroute-cache
 ip ospf priority 100
 ip ospf 1 area 0.0.0.0
!
interface GigabitEthernet0/2
 switchport access vlan 60
 switchport mode access
 spanning-tree portfast
!
<------output omitted------>
!
interface GigabitEthernet0/24
 switchport access vlan 3334
 spanning-tree portfast
!
<------output omitted------>
!
interface Vlan1
 no ip address
 shutdown
!
interface Vlan60
 ip address 10.92.6.1 255.255.255.0
 ip pim sparse-dense-mode
 ip ospf priority 100
 ip ospf 1 area 10.92.6.1
!
!
!
interface Vlan3334
 description PTP link to BD1thru_HCLOS
 bandwidth 40000
 ip address 192.168.100.73 255.255.255.252
 ip pim sparse-dense-mode
 no ip mroute-cache
 ip ospf network point-to-point
 ip ospf 1 area 10.92.34.1
!
router ospf 1
 router-id 10.92.6.1
 auto-cost reference-bandwidth 1000
 area 10.92.6.1 nssa default-information-originate no-summary
 area 10.92.34.1 nssa default-information-originate no-summary
 area 10.92.34.1 filter-list prefix deny_PTP_transit_nets out
!
no ip http server
no ip http secure-server
!
!
ip pim bidir-enable
ip pim send-rp-announce Loopback0 scope 16 group-list 5 bidir
ip pim send-rp-discovery scope 16
!
!
ip prefix-list deny_PTP_transit_nets seq 5 deny 192.168.100.0/24 le 32
ip prefix-list deny_PTP_transit_nets seq 10 permit 0.0.0.0/0 le 32
!
access-list 1 permit 239.0.4.0 0.0.0.255
access-list 1 permit 239.0.5.0 0.0.0.255
access-list 1 permit 239.0.6.0 0.0.0.255
access-list 1 permit 239.0.7.0 0.0.0.255
access-list 1 permit 239.0.8.0 0.0.0.255
access-list 1 permit 239.0.9.0 0.0.0.255
access-list 1 permit 239.0.11.0 0.0.0.255
access-list 1 permit 239.0.12.0 0.0.0.255
access-list 5 permit 239.0.2.0 0.0.0.255
access-list 5 permit 239.0.4.0 0.0.0.255
access-list 5 permit 239.0.5.0 0.0.0.255
access-list 5 permit 239.0.6.0 0.0.0.255
access-list 5 permit 239.0.7.0 0.0.0.255
access-list 5 permit 239.0.8.0 0.0.0.255
access-list 5 permit 239.0.9.0 0.0.0.255
access-list 5 permit 239.0.11.0 0.0.0.255
access-list 5 permit 239.0.12.0 0.0.0.255
!
!
!
!
!
line con 0
line vty 0 4
 exec-timeout 0 0
line vty 5 15
 exec-timeout 0 0
!
end

BE1#

I was under the impression that auto-rp required sparse-dense mode in the interfaces to flood 224.0.1.39 and 224.0.1.40?  I did end up configuring sparse mode with ip pim autorp listener, and the same behavior is exhibited.

Not sure if is related, but found this (albeit for the default multicast range only):

https://tools.cisco.com/quickview/bug/CSCts59004

Thanks again!

Joe

[Peter Paluch]

Hi Joe,

Thanks for responding! The configuration does not seem to be out of ordinary in any way.

I was under the impression that auto-rp required sparse-dense mode in the interfaces to flood 224.0.1.39 and 224.0.1.40?

Yes, you are completely right - I forgot you are running AutoRP to advertise RPs and group-to-RP mappings. I am sorry. I usually go with BSR, and in smaller networks, with a static configuration. AutoRP is not exactly my favourite mechanism.

I have had a look at the bug report. I do not think it applies to you. It says that the upstream traffic is affected whereas in your case, it is the downstream traffic that is not forwarded. Also, it mentions that the prerequisite for this bug is to configure the entire D-scope (224.0.0.0/4) as a Bidir-eligible scope which is clearly not your case - you are using the ACL 5 to limit the list of Bidir-enabled groups.

At this point, I am thoroughly confused. Just wondering: We have troubles getting the multicast traffic across from BD1 to JC1, right? What if the situation was reversed, and the traffic went from JC1 to BD1, that is, if JC1 (or someone behind it) was a multicast source for the same group?

Best regards,
Peter

[joseph.y.chen1982]

Hi Peter,
Thanks for the response!  Haha, that makes two of us.  Yep, we are having troubles getting multicast traffic from BD1 to JC1.  Even weirder is that multicast traffic from JC1 to BD1 on the same group passes just fine. 

The application is part of an an audio dispatching system; there's a software piece that gets installed on a computer and sends audio streams over multicast (or unicast, depending on the system configuration).  We have one end at BD1 and another at JC1, both are sources and receivers for the same group.

[Peter Paluch]

Joe,

I am wondering: What exact platforms are your BD1, BE1, and JC1, and what IOS versions are they running? Would you mind doing an experiment in which you would configure the BiDir RP to be either at BD1 or JC1 and see if it changed anything?

Best regards,
Peter

[joseph.y.chen1982]

Hi Peter,

Thanks for the response.  The platforms are:

BE1: WS-C3560G-24TS     15.0(2)SE1            C3560-IPSERVICESK9-M

BD1: 2851 (C2800NM-ADVENTERPRISEK9-M), Version 15.1(4)M7, RELEASE SOFTWARE (fc2)

JC1: 2851 (C2800NM-ADVENTERPRISEK9-M), Version 15.1(4)M7, RELEASE SOFTWARE (fc2)

I've done the experiment, and have some interesting results.  Results follow:

RP	BD1->JC1	JC1->BD1	notes
BD1	yes	no
JC1	no	yes
BE1	depends	depends	depends on which endpoint registers first

Where the RP has been configured at either 'endpoint' router (BD1 or JC1), traffic forwards unidirectionally from where the RP is configured to the destination.  Traffic from the distant end to the RP endpoint router does not forward, regardless of which endpoint registers first.  However, interestingly, when the RP is configured to be BE1, it appears traffic will flow from the first registered endpoint to the other, but not vice versa.

I suppose the next experiment to run is to replace BE1 with a routing platform?

Thanks in advance,

Joe

[Peter Paluch]

Hi Joe,

I suppose the next experiment to run is to replace BE1 with a routing platform?

Yes, that would be perfect.

I have a couple of 3560 and 3560V2 in our lab with similar IOSes. I can try to reproduce this problem in my lab tomorrow or in two days. I am curious if there is any glitch in the IOS - which is quite possible. I was definitely able to concoct a very simple test in my Dynamips running 2691 12.4T IOSes and the Bidir worked nicely so I really don't think there is any issue with your configuration.

Best regards,
Peter

[joseph.y.chen1982]

Hi Peter,

Thanks again for looking into this for me, much appreciated.  I'm sure you're busy with other stuff, so whenever you're able and if you're so inclined, a lab reproduction would be great.  I'm wondering if I'm starting to go crazy or something.

I did replace the BE1 switch with a router, and also with interesting results.  Same specs as the other 2851s and with a NM-ESW-16.  When connected through the router's L3 built-in interfaces (no VLAN3334 SVI, config directly applied to the interface), PIM-bidir works as expected.  Ifthe VLAN3334 SVI is used in conjucntion with a port on the ESW, the failure occurs again.  However, when the port on the ESW is converted to a routed port with the VLAN3334 config, PIM-bidir again works as expected.

Also interesting, is that if I forgo the VLAN3334 SVI on the 3560G and used a routed port instead, the failure still occurs.

I have the current topology labbed-up, but in the production environment where the issue was first seen, the switches in question are actually IE3010s running 15.1 (I believe).

Thanks again,

Joe

[Peter Paluch]

Hi Joe,

So I've recreated the topology in our lab, using 1841 running 15.3(3)XB12 Advanced IP services as BD/JC, and a Catalyst 3560 running 12.2(55)SE6 IP Services as BE1 and RP. Unsurprisingly, I have had the same disappointing results as you had, and even worse: Not only the multicast traffic was forwarded in a single direction only, but also the 3560 appeared to be losing the packets and delivering just a tiny fraction of them.

When I googled around the internet trying to find some more information on what could be wrong, I stumbled across this somewhat outdated but still relevant PDF:

http://www.a-trac.com/documents/Cisco/Enterprise/brochures/Cisco_BROCHURE_Switch%20Guide_2.14.12.pdf

Check out the page 42/43 in the PDF (it's around Page 24 of the PDF): It states that none of the switches listed on that page, including Catalyst 3560, has hardware support for Bidirectional PIM. According to that document, the closest platform that supports Bidir PIM in hardware is Catalyst 4900 (page 24/25).

In addition, after further googling, I've found out that according to the Configuration Guide for the latest IOS version, all Bidir PIM-related commands are unsupported, see:

http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3560/software/release/15-0_2_se/configuration/guide/scg3560/swuncli.html#pgfId-1093588

And then it all clicked together. These switches simply do not support Bidir PIM, at least not with current software (and perhaps this is a hardware limitation that cannot be circumvented by an updated firmware/microcode). Whatever we configure in IOS is not translated into the hardware, or is translated only in a partial way. The results are, as you have seen yourself, dismal.

It appears that the new Catalyst platforms, such as 3650, support Bidir-PIM. I haven't found an explicit statement saying that "3650 and 3850 support Bidirectional PIM", however, the Configuration Guide for 3650 routinely discusses Bidir-PIM so I would suppose that the support is already there.

http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3650/software/release/37e/multicast/configuration_guide/b_mc_37e_3650_cg.html

The resume, alas, is that as long as you are going to use your current multilayer switches lower than 4900 series in your network (4500 could be an exception), Bidir-PIM is sadly out of question.

Thanks for raising this issue here on CSC. I would not have even dreamt of such a ... stupid limitation wreaking such a havoc.

Best regards,
Peter

[joseph.y.chen1982]

Hi Peter,

Thank you so much for all your help!  Now that I think about it, I did remember going through the configuration guides for the 3010/3560 platforms some time ago, and as you mentioned, there are no references to PIM-bidir, at least for IPv4 (IPv6 looks okay though: http://www.cisco.com/c/en/us/td/docs/switches/lan/cisco_ie3010/software/release/15-0_2_se/configuration/guide/scgie3010/swipv6m.html). ; I guess it didn't really click in my mind that this might not be a supported option, especially since the devices allow you to configure the commands.

Thanks again so much for all the help, much obliged!

Joe

[Peter Paluch]

Joe,

You are very much welcome! :)

Best regards,
Peter