NCS540: Cannot ping over MPLS VPN to non-adjacent router (BGP Confed)

[Ronit Bhattacharjee]

We are in the process of designing a medium sized MPLS network using NCS540 routers. A requirement was not to have a full-mesh of BGP neighbourships or use RR or to change the next-hop at every hop.

So we decided to use BGP confed. Maybe our design assumption is wrong to begin with.

Anyways, we setup a small lab with relevant design below. The status is

1. OSPF is up everywhere
2. All Loopback0 can ping all Loopback0
3. BGP neighbourship is up
4. All Loopback10 (Inside vrf Management) can only ping Loopback10 of routers they have direct BGP neighbourships with
5. In the below diagram, Loopback 10 of R5-CRT1 cannot ping Loopback 10 on R6-CRT2, although BGP has installed the prefix in the routing table
6. If we add "next-hop-self" on the middle router, they can ping each other
7. If we change to regular eBGP neighbourship without confederation, they can ping each other

What can we be doing wrong?

[MHM Cisco World]

MHM

[Ronit Bhattacharjee]

Hello. Thank you for the quick response. However, I do not see where the next hop 1.1.6.5 is unknown? I see it is Loop0 of R6-CRT2?

RP/0/RP0/CPU0:R5-CRT1#show ip route vrf Management 10.8.6.5
Mon Aug 5 17:17:06.698 +07

Routing entry for 10.8.6.5/32
Known via "bgp 105", distance 200, metric 0
Tag 106, type internal
Installed Aug 5 15:49:04.977 for 01:28:01
Routing Descriptor Blocks
1.1.6.5, from 1.1.6.4
Nexthop in Vrf: "default", Table: "default", IPv4 Unicast, Table Id: 0xe0000000
Route metric is 0
No advertising protos.

RP/0/RP0/CPU0:R5-CRT1#sh ip route vrf Management
Mon Aug 5 18:24:46.766 +07

Codes: C - connected, S - static, R - RIP, B - BGP, (>) - Diversion path
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP
i - ISIS, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, su - IS-IS summary null, * - candidate default
U - per-user static route, o - ODR, L - local, G - DAGR, l - LISP
A - access/subscriber, a - Application route
M - mobile route, r - RPL, t - Traffic Engineering, (!) - FRR Backup path

Gateway of last resort is not set

C 10.7.0.0/24 is directly connected, 04:43:33, GigabitEthernet0/0/0/17
L 10.7.0.171/32 is directly connected, 04:43:33, GigabitEthernet0/0/0/17
L 10.8.5.4/32 is directly connected, 3d09h, Loopback10
B 10.8.5.5/32 [200/0] via 1.1.5.5 (nexthop in vrf default), 02:35:15
B 10.8.6.4/32 [200/0] via 1.1.6.4 (nexthop in vrf default), 02:35:41
B 10.8.6.5/32 [200/0] via 1.1.6.5 (nexthop in vrf default), 02:35:41
B 10.8.7.4/32 [200/0] via 1.1.7.4 (nexthop in vrf default), 02:35:41
B 10.8.7.5/32 [200/0] via 1.1.7.5 (nexthop in vrf default), 02:35:41
B 10.8.7.101/32 [200/0] via 1.1.7.101 (nexthop in vrf default), 02:35:41
B 10.8.8.4/32 [200/0] via 1.1.8.4 (nexthop in vrf default), 02:35:41
B 10.8.8.5/32 [200/0] via 1.1.8.5 (nexthop in vrf default), 02:35:41
B 10.8.8.101/32 [200/0] via 1.1.8.101 (nexthop in vrf default), 02:35:41

[MHM Cisco World]

1.1.6.5 <<- this next hop 

In router can you ping it ?

MHM

[Ronit Bhattacharjee]

Yes, can ping it. I understand it is recursive, but OSPF has full convergence in the control plane, so we can ping all Loop0 IPs

RP/0/RP0/CPU0:R5-CRT1#ping 1.1.6.5
Mon Aug 5 18:33:44.110 +07
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 1.1.6.5 timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/3/4 ms
RP/0/RP0/CPU0:R5-CRT1#

RP/0/RP0/CPU0:R5-CRT1#sh ip route 1.1.6.5
Mon Aug 5 18:34:35.275 +07

Routing entry for 1.1.6.5/32
Known via "ospf 1", distance 110, metric 3, type intra area
Installed Aug 5 11:25:18.652 for 07:09:16
Routing Descriptor Blocks
192.168.5.2, from 1.1.6.5, via TenGigE0/0/0/23
Route metric is 3
192.168.56.2, from 1.1.6.5, via GigabitEthernet0/0/0/16
Route metric is 3
No advertising protos.
RP/0/RP0/CPU0:R5-CRT1#

[MHM Cisco World]

MHM

[Ronit Bhattacharjee]

But we are using MPLS. So OSPF is running on the control plane "global". We don't need to run OSPF on the data plane for MPLS, yes?

[Ronit Bhattacharjee]

Here's a capture of cef entries of one reachable and one unreachable address. Both appear as label switched with next-hop in global

RP/0/RP0/CPU0:R5-CRT1#sh ip cef vrf Management 10.8.6.5
Mon Aug 5 18:50:54.280 +07
10.8.6.5/32, version 178, internal 0x5000001 0x30 (ptr 0x8b4d86e8) [1], 0x0 (0x0), 0xa08 (0x8ed1b6f0)
Updated Aug 5 15:49:04.983
Prefix Len 32, traffic index 0, precedence n/a, priority 3
gateway array (0x8b304440) reference count 1, flags 0x2038, source rib (7), 0 backups
[1 type 1 flags 0x48441 (0x8ed65d08) ext 0x0 (0x0)]
LW-LDI[type=0, refc=0, ptr=0x0, sh-ldi=0x0]
gateway array update type-time 1 Aug 5 13:53:25.301
LDI Update time Aug 5 13:53:25.301
via 1.1.6.5/32, 3 dependencies, recursive [flags 0x6000]
path-idx 0 NHID 0x0 [0x8ee19498 0x0]
recursion-via-/32
next hop VRF - 'default', table - 0xe0000000
next hop 1.1.6.5/32 via 24010/0/21
next hop 192.168.56.2/32 Gi0/0/0/16 labels imposed {ImplNull 24009}
next hop 192.168.5.2/32 Te0/0/0/23 labels imposed {ImplNull 24009}

Load distribution: 0 (refcount 1)

Hash OK Interface Address
0 Y recursive 24010/0
RP/0/RP0/CPU0:R5-CRT1#sh ip cef vrf Management 10.8.5.5
Mon Aug 5 18:50:56.957 +07
10.8.5.5/32, version 191, internal 0x5000001 0x30 (ptr 0x8b4dc158) [1], 0x0 (0x0), 0xa08 (0x8ed1b418)
Updated Aug 5 15:49:31.417
Prefix Len 32, traffic index 0, precedence n/a, priority 3
gateway array (0x8b305660) reference count 1, flags 0x2038, source rib (7), 0 backups
[1 type 1 flags 0x48441 (0x8ed670a0) ext 0x0 (0x0)]
LW-LDI[type=0, refc=0, ptr=0x0, sh-ldi=0x0]
gateway array update type-time 1 Aug 5 15:49:31.417
LDI Update time Aug 5 15:49:31.417
via 1.1.5.5/32, 8 dependencies, recursive [flags 0x6000]
path-idx 0 NHID 0x0 [0x8ee15e68 0x0]
recursion-via-/32
next hop VRF - 'default', table - 0xe0000000
next hop 1.1.5.5/32 via 24000/0/21
next hop 192.168.5.2/32 Te0/0/0/23 labels imposed {ImplNull 24024}

Load distribution: 0 (refcount 1)

Hash OK Interface Address
0 Y recursive 24000/0
RP/0/RP0/CPU0:R5-CRT1#ping vrf Management 10.8.6.5
Mon Aug 5 18:51:06.773 +07
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.8.6.5 timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
RP/0/RP0/CPU0:R5-CRT1#ping vrf Management 10.8.5.5
Mon Aug 5 18:51:20.592 +07
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.8.5.5 timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms

[MHM Cisco World]

that why we need next-hop self 
use it and see how next-hop change to IP reachable from mgmt vrf. 
MHM

[Ronit Bhattacharjee]

The next-hop is now the next-router instead of the end router, still in the global VRF. I think it is normal for MPLS VPN routes to have next-hop in the global VRF. 

P/0/RP0/CPU0:R5-CRT1#show ip route vrf Management 10.8.6.5
Mon Aug 5 19:37:01.627 +07

Routing entry for 10.8.6.5/32
Known via "bgp 105", distance 200, metric 0
Tag 106, type internal
Installed Aug 5 19:11:04.977 for 01:28:01
Routing Descriptor Blocks
1.1.5.5, from 1.1.5.5
Nexthop in Vrf: "default", Table: "default", IPv4 Unicast, Table Id: 0xe0000000
Route metric is 0
No advertising protos.

RP/0/RP0/CPU0:R5-CRT1#ping vrf Management 10.8.6.5
Mon Aug 5 19:37:06.773 +07
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.8.6.5 timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5)

[MHM Cisco World]

MPLS VPN routes to have next-hop in the global VRF. 

That correct' l3vpn mp-bgp by default ibgp peer run next-hop-self.

So indeed you need next-hop-self to make bgp mpls work.

MHM

[Harold Ritter · Accepted solution]

Hi @Ronit Bhattacharjee ,

The issue you are seeing is due to the fact that you are using the "ebgp-multihop" command on R6-CRT2 without specifying the mpls keyword. This causes an implicit null label to be used for prefixes received from that neighbor. This causes the label stack for R6-CRT2 to R5-CRT1 to be broken. Try configuring the following on R6-CRT2:

router bgp 106
neighbor 1.1.5.5
ebgp-multihop 2 mpls

This should fix the issue.

Regards,

[Ronit Bhattacharjee]

This is the correct answer and it solved our problem.

I never implemented MPLS with eBGP before, so were not aware of this behaviour.

Thank you so much for the quick support.

[Harold Ritter]

You are very welcome @Ronit Bhattacharjee and thanks for the feedback

[MHM Cisco World]

Hi Friend
I make deep review and make some notes can please check it
thanks a lot

MHM

RP/0/RP0/CPU0:R5-CRT1#sh ip cef vrf Management 10.8.6.5 <<- failed ping from vrf
Mon Aug 5 18:50:54.280 +07
10.8.6.5/32, version 178, internal 0x5000001 0x30 (ptr 0x8b4d86e8) [1], 0x0 (0x0), 0xa08 (0x8ed1b6f0)
Updated Aug 5 15:49:04.983
Prefix Len 32, traffic index 0, precedence n/a, priority 3
gateway array (0x8b304440) reference count 1, flags 0x2038, source rib (7), 0 backups
[1 type 1 flags 0x48441 (0x8ed65d08) ext 0x0 (0x0)]
LW-LDI[type=0, refc=0, ptr=0x0, sh-ldi=0x0]
gateway array update type-time 1 Aug 5 13:53:25.301
LDI Update time Aug 5 13:53:25.301
via 1.1.6.5/32, 3 dependencies, recursive [flags 0x6000]
path-idx 0 NHID 0x0 [0x8ee19498 0x0]
recursion-via-/32
next hop VRF - 'default', table - 0xe0000000
next hop 1.1.6.5/32 via 24010/0/21
next hop 192.168.56.2/32 Gi0/0/0/16 labels imposed {ImplNull 24009}
next hop 192.168.5.2/32 Te0/0/0/23 labels imposed {ImplNull 24009}<<- why there is two path for this prefix ? can you check 1.1.6.5 in global RIB are it have two paht ? your topology is chain but here I can see two interface use for label ??

Load distribution: 0 (refcount 1)

Hash OK Interface Address
0 Y recursive 24010/0
RP/0/RP0/CPU0:R5-CRT1#sh ip cef vrf Management 10.8.5.5 <<- success ping from vrf 
Mon Aug 5 18:50:56.957 +07
10.8.5.5/32, version 191, internal 0x5000001 0x30 (ptr 0x8b4dc158) [1], 0x0 (0x0), 0xa08 (0x8ed1b418)
Updated Aug 5 15:49:31.417
Prefix Len 32, traffic index 0, precedence n/a, priority 3
gateway array (0x8b305660) reference count 1, flags 0x2038, source rib (7), 0 backups
[1 type 1 flags 0x48441 (0x8ed670a0) ext 0x0 (0x0)]
LW-LDI[type=0, refc=0, ptr=0x0, sh-ldi=0x0]
gateway array update type-time 1 Aug 5 15:49:31.417
LDI Update time Aug 5 15:49:31.417
via 1.1.5.5/32, 8 dependencies, recursive [flags 0x6000]
path-idx 0 NHID 0x0 [0x8ee15e68 0x0]
recursion-via-/32
next hop VRF - 'default', table - 0xe0000000
next hop 1.1.5.5/32 via 24000/0/21
next hop 192.168.5.2/32 Te0/0/0/23 labels imposed {ImplNull 24024}