Static route being ignored when traffic comes from MPLS neighbor

[7tsommer]

Given this setup:

R1 and R2 are ASR9Ks running IOS-XR, and R3 is an ASR920 running IOS-XE

We have MPLS with MP-BGP running between R2 and R3 (with OSPF as the underlay in the default VRFs on those routers). Routing between R1 and R2 on the production circuit is strictly OSPF, with the protocol running in VRF3 for R2. We built a bypass link for rerouting traffic around the OSPF link with static routes for testing some things. We have no routing protocols running across the bypass link, but we do have "redistribute connected" configured in OSPF. Both of the links (the OSPF production link and bypass link) from R1 to R2 exist in the default VRF on R1 and in VRF3 on R2. There are also ports on R3 that exist in VRF3. There is no MPLS between R1 and R2.

Under normal circumstances, R1 advertises a default route to R2, which propagates it to R3 via MP-BGP and the rest of the network in VRF3. We want to be able to bypass the OSPF link between R1 and R2 for testing by using static /32 routes on both ends. On R2, the static route is in VRF3. 

When we install a static route and look at the VRF3 routing table on R2, it indeed shows only one path to the destination - the /32 static we configured. When we run traces on R2 in that VRF towards the destination in the static route (westbound, beyond R1), it shows R1's side of the bypass link in the traceroutes as expected.

However, when we run traces from VRF3 on R3 to the same destination, R2 still forwards those packets across the OSPF link between R1 and R2. Since we're not redistributing the static bypass route, R3 still uses the default route to send the traffic to R2. But why, if R2 itself does have that static route and does show it to be the only path to the destination, is it sending R3's traffic through the default route path via the OSPF link? 

From what I can tell, R3 is popping the label off when sending traffic towards the default route to R2. So shouldn't R2 treat it as just another packet on VRF3 and use the static route to send it across the bypass link? What is making it ignore that when receiving traffic from R3 over the MPLS link? I looked at the CEF table, and the only entry for the static destination is the static route over the bypass link, but traces from R3 to that same destination still show the OSPF link as the next-hop. What is causing R2 to forward it using the VRF3 OSPF default route instead of the static?

We did notice that redistributing that static into BGP caused R2 to forward R3's VRF3 traffic through the bypass link, but we really want to avoid redistributing those statics at all. Redistributing into BGP caused routing loops because it was being redistributed into OSPF elsewhere and propagated to R1, so R1 was just sending back to R2 and thus looping. We're trying to avoid having to do a bunch of other configuration work on R2 and other adjacent OSPF routers to prevent these loops. Seems like there should be something we can do on R2 to get it to treat R3's VRF3 traffic like any other port on R2 in that VRF, thus causing it to use the static route to send the traffic across the bypass link.

[Harold Ritter · Accepted solution]

Hi @7tsommer ,

This is normal behaviour, as the default label allocation mode is per prefix. So when R2 receives labelled traffic from R3, R2 lookups the received service label in the LFIB and forward the traffic directly to the interface towards R1 without doing a VRF3 RIB lookup.

As you mentioned, redistributing the static routes solves the issue, as it installs another entry in the LFIB with the bypass link as the next hop. The other way to solve the issue would be to change the label allocation mode from

per prefix to per vrf

This would force R2 to perform a VRF3 RIB lookup after performing the LFIB lookup.

Regards,

[7tsommer]

Shouldn't penultimate hop popping cause R3 to send the packet to R2 unlabeled?

[Harold Ritter]

HI @7tsommer ,

Bear in mind that L3VPN uses a label stack. The PHP will pop the IGP label (top most label). Packets will get to R2 with only the service label. The service label (bottom label) is required for the egress PE to make the proper forwarding decision.

Regards,

[MHM Cisco World]

hard to answer we need to see 

show ip route vrf 3 <<- 

we must totally sure that the static route is from

VRF-VRF not from GRT-VRF

share the route here if you can 

[7tsommer]

The static route is configured in the

VRF and is VRF-VRF

R2#show route vrf 3 x.x.x.x/32
Fri Oct 14 10:19:21.629 CDT

Routing entry for x.x.x.x/32
Known via "static", distance 1, metric 0
Installed Oct 13 16:35:54.121 for 17:43:27
Routing Descriptor Blocks
y.y.y.y, via GigabitEthernet0/0/0/0
Route metric is 0, Wt is 1
No advertising protos.

R2#show ip vrf 3 int bri | incl 0/0/0/0

Fri Oct 14 10:23:02.129 CDT
GigabitEthernet0/0/0/0 y.y.y.z Up Up

[Harold Ritter]

Hi @7tsommer ,

As I mentioned in my other post, only the LFIB is looked by default. The VRF3 RIB won't be looked up unless you change the label allocation mode to

per vrf. The default is per prefix

Regards, 

[7tsommer]

Thanks Harold. I think I understand now given how PHP affects the IGP label but not the service label. Sounds like we're just going to have to redistribute the statics and put looping prevention measures in place since reconfiguring how we do MPLS isn't an option. 

[Harold Ritter]

You are welcome @7tsommer . I agree that redistributing is a safer approach, but changing the label allocation mode is also something you might consider in the future. It is a simple configuration change. 

Regards,

[7tsommer]

<removed redundant reply>

[MHM Cisco World]

this my small lab,  there is MPLS run between R2-R5-R3 and there is OSPF run between R2-R1
I make second link between R1(CE) and R2(PE) and config static route 

and you can see traceroute is OK there is no issue at all 

so are you redistribute the static route into MP-BGP ?
do you check the prefix in BGP VPNv4 in both R2 and R3 ?

[Harold Ritter]

Hi @MHM Cisco World ,

The behaviour described by the OP is due to the fact that they advertise the default route to the remote PE, but did not want to redistribute the static route (/32) in BGP.  

The assumption was that traffic would follow the default route from R3 to R2, but that it would follow the more specific route (/32 static route) once traffic got to R2. This assumed behaviour could only be obtained if the label allocation was set to "per vrf" allocation, in which case a RIB lookup would be performed after the LFIB lookup is.

Regards,

[MHM Cisco World]

Yes, but can I mention here something 
the redistribute static or ospf into MP-BGP (iBGP)
the MP-BGP will select either static or OSPF not advertise both, am I right ?
I think he redistribute the static route but the not connect (which is next-hop of static route)

may be I am wrong.

[Harold Ritter]

Hi @MHM Cisco World ,

Since the /32 is a static route, they will need to configure a

redistribute static

Regards,

[MHM Cisco World]

but also we must redistribute the connect (next-hop) of static route ? am I right ?