VPNV4-Option-C label issues

[feroz syed]

Good Morning,

i have issue with below Topology, where R16 and R1 have each other routes but not reachable, the packet get drop from directly connected PE,

Details :- XR4 (11.11.11.11/32) and R3 (8.8.8.8/32) both advertise their local PE loopback address using BGP labled protocol,

• XR4 reddtribute learned routes (8.8.8.8/32) to OSPF 1.
• R3 Redistribute learned routes (11.11.11.11/32) to OSPF 321.
• XR1 and R8 form vpnv4 neighbour from Redistributed routes by XR4 & R3(Ebgp-lulti-hop).
• XR4 has static routes towards R3 using /32 (for label support).
• R8 redistribute vpnv4 learned routes to vrf X (address-family ipv4 vrf X
  redistribute eigrp 100) (router eigrp 100 address-family ipv4 vrf X autonomous system 100 redistribute bgp 321 metric 1000 10 255 1 1500).
• XR1 has Ibgp neighbor to CE router, also vrf export and Import done according to isp-2 R8 router.

Verification:-
RP/0/0/CPU0:XR-1#
RP/0/0/CPU0:XR-1#
RP/0/0/CPU0:XR-1#sh mpls forwarding
Thu Nov 10 07:57:34.532 UTC
Local Outgoing Prefix Outgoing Next Hop Bytes
Label Label or ID Interface Switched
------ ----------- ------------------ ------------ --------------- ------------
111000 44444 22.22.22.22/32 Gi0/0/0/0 192.168.13.3 0
111001 Pop 33.33.33.33/32 Gi0/0/0/0 192.168.13.3 15334
111002 44445 44.44.44.44/32 Gi0/0/0/0 192.168.13.3 520
111003 44447 55.55.55.55/32 Gi0/0/0/0 192.168.13.3 22352
111004 Pop 8.8.8.8/32 Gi0/0/0/0 192.168.13.3 18899
111005 Pop 192.168.23.0/24 Gi0/0/0/0 192.168.13.3 0
111006 Pop 192.168.34.0/24 Gi0/0/0/0 192.168.13.3 0
111007 Pop 192.168.35.0/24 Gi0/0/0/0 192.168.13.3 0
111008 Unlabelled 1.1.1.1/32[V] Gi0/0/0/1 172.16.16.1 1876
RP/0/0/CPU0:XR-1#sh mpls forwarding vrf A
Thu Nov 10 07:57:41.281 UTC
Local Outgoing Prefix Outgoing Next Hop Bytes
Label Label or ID Interface Switched
------ ----------- ------------------ ------------ --------------- ------------
111008 Unlabelled 1.1.1.1/32[V] Gi0/0/0/1 172.16.16.1 1876

RP/0/0/CPU0:XR-1#sh bgp vpnv4 un sum
Thu Nov 10 07:58:37.488 UTC
BGP router identifier 11.11.11.11, local AS number 12345
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0x0 RD version: 0
BGP main routing table version 8
BGP NSR Initial initsync version 7 (Reached)
BGP NSR/ISSU Sync-Group versions 0/0
BGP scan interval 60 secs

BGP is operating in STANDALONE mode.

Process RcvTblVer bRIB/RIB LabelVer ImportVer SendTblVer StandbyVer
Speaker 8 8 8 8 8 0

Neighbor Spk AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down St/PfxRcd
8.8.8.8 0 321 148 134 8 0 0 02:10:46 2

RP/0/0/CPU0:XR-1#sh bgp vpnv4 un
Thu Nov 10 07:58:43.677 UTC
BGP router identifier 11.11.11.11, local AS number 12345
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0x0 RD version: 0
BGP main routing table version 8
BGP NSR Initial initsync version 7 (Reached)
BGP NSR/ISSU Sync-Group versions 0/0
BGP scan interval 60 secs

Status codes: s suppressed, d damped, h history, * valid, > best
i - internal, r RIB-failure, S stale, N Nexthop-discard
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1:2 (default for vrf A)
*> 1.1.1.1/32 172.16.16.1 0 0 100 i
*> 16.16.16.16/32 8.8.8.8 156160 0 321 ?
*> 192.168.86.0/24 8.8.8.8 0 0 321 ?
Route Distinguisher: 8:16
*> 16.16.16.16/32 8.8.8.8 156160 0 321 ?
*> 192.168.86.0/24 8.8.8.8 0 0 321 ?

------------------------------------------------------------------------------------
------------------------------------------------------------------------------------
------------------------------------------------------------------------------------
------------------------------------------------------------------------------------
------------------------------------------------------------------------------------
R8#sh mpls forwarding-table
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
800 Pop Label 7.7.7.7/32 0 Fa0/0 192.168.78.7
801 700 6.6.6.6/32 0 Fa0/0 192.168.78.7
802 702 5.5.5.5/32 0 Fa0/0 192.168.78.7
803 701 3.3.3.3/32 0 Fa0/0 192.168.78.7
804 Pop Label 192.168.37.0/24 0 Fa0/0 192.168.78.7
805 Pop Label 192.168.57.0/24 0 Fa0/0 192.168.78.7
806 Pop Label 192.168.67.0/24 0 Fa0/0 192.168.78.7
807 No Label 16.16.16.16/32[V] \
0 Fa0/1 192.168.86.16
808 No Label 192.168.86.0/24[V] \
0 aggregate/X
809 704 11.11.11.11/32 0 Fa0/0 192.168.78.7
R8#
R8#sh mpls forwarding-table vr
R8#sh mpls forwarding-table vrf X
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
807 No Label 16.16.16.16/32[V] \
0 Fa0/1 192.168.86.16
808 No Label 192.168.86.0/24[V] \
0 aggregate/X

R8#sh ip route vrf X

Routing Table: X
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
+ - replicated route, % - next hop override

Gateway of last resort is not set

1.0.0.0/32 is subnetted, 1 subnets
B 1.1.1.1 [20/0] via 11.11.11.11, 02:09:16
16.0.0.0/32 is subnetted, 1 subnets
D 16.16.16.16 [90/156160] via 192.168.86.16, 02:17:34, FastEthernet0/1
192.168.86.0/24 is variably subnetted, 2 subnets, 2 masks
C 192.168.86.0/24 is directly connected, FastEthernet0/1
L 192.168.86.8/32 is directly connected, FastEthernet0/1
R8#

---------------------------------------------------------------------------
---------------------------------------------------------------------------

R16#sh ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
Gateway of last resort is not set

1.0.0.0/32 is subnetted, 1 subnets
D EX 1.1.1.1 [170/2565120] via 192.168.86.8, 02:33:31, FastEthernet0/0
16.0.0.0/32 is subnetted, 1 subnets
C 16.16.16.16 is directly connected, Loopback0
192.168.86.0/24 is variably subnetted, 2 subnets, 2 masks
C 192.168.86.0/24 is directly connected, FastEthernet0/0
L 192.168.86.16/32 is directly connected, FastEthernet0/0

---------------------------------------------------------------------------
---------------------------------------------------------------------------

R1#sh ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
+ - replicated route, % - next hop override

Gateway of last resort is not set

1.0.0.0/32 is subnetted, 1 subnets
C 1.1.1.1 is directly connected, Loopback0
16.0.0.0/32 is subnetted, 1 subnets
B 16.16.16.16 [20/0] via 172.16.16.11, 01:37:46
172.16.0.0/16 is variably subnetted, 2 subnets, 2 masks
C 172.16.16.0/24 is directly connected, FastEthernet0/0
L 172.16.16.1/32 is directly connected, FastEthernet0/0
B 192.168.86.0/24 [20/0] via 172.16.16.11, 01:37:47

[Harold Ritter]

Hi @feroz syed ,

Looking at the "show mpls for" you provided for XR-1, there seems to be something wrong with the LSP to 8.8.8.8, as seen below:

RP/0/0/CPU0:XR-1#sh mpls forwarding

111004 Pop 8.8.8.8/32 Gi0/0/0/0 192.168.13.3 18899

There seems to be no issue in the other direction (R8 to XR-1)

R8#sh mpls forwarding-table
809 704 11.11.11.11/32 0 Fa0/0 192.168.78.7

I know there is some issues sometimes with XRv, given the level of code used is kind of old.

I would suggest one thing if you don't mind. Can you extend the BGP LU from XR-4 to XR-1, instead of redistributing BGP-LU in OSPF on XR-4 and see if it solves the issue. You also need to remove the network statement 11.11.11.11/32 from XR-4 and move it to XR-1 instead.

Regards,

[feroz syed]

"""would suggest one thing if you don't mind. Can you extend the BGP LU from XR-4 to XR-1, instead of redistributing BGP-LU in OSPF on XR-4 and see if it solves the issue. You also need to remove the network statement 11.11.11.11/32 from XR-4 and move it to XR-1 instead."""

sorry am confused, XR1 and R8 both vpvn4 neighbor using loop back address, from redistributed address.

[Harold Ritter]

Hi @feroz syed ,

It looks like the redisbution of the BGP LU routes into OSPF on the first AS doesn't work, as shown by the output of the "show mpls forwarding", where we see pop label for 8.8.8.8. This is wrong and that is why you don't have end to end connectivity. 

What I am proposing is to have an iBGP LU session between XR1 and XR-4. This way XR-1 would learn 8.8.8.8 and label via iBGP LU instead of learning it via OSPF/LDP. It is a common deployment practice for interAS option C by the way. Here's what you need to do.

- configure iBGP LU between XR-1 XR-4.

- remove redistribution from BGP to OSPF on XR-4

- remove network 11.11.11.11/32 under address-family ipv4 unicast on XR-4

- add network 11.11.11.11/32 under address-family ipv4 unicast on XR-1

- add allocate-label all under address-family ipv4 unicast on XR-1

Regards,

[feroz syed]

RP/0/0/CPU0:XR-1# sh mpl forwarding
Thu Nov 10 19:10:09.192 UTC
Local Outgoing Prefix Outgoing Next Hop Bytes
Label Label or ID Interface Switched
------ ----------- ------------------ ------------ --------------- ------------
111000 Pop 33.33.33.33/32 Gi0/0/0/0 192.168.13.3 12142
111001 44444 44.44.44.44/32 Gi0/0/0/0 192.168.13.3 5539
111002 Pop 8.8.8.8/32 Gi0/0/0/0 192.168.13.3 26159
111003 Pop 192.168.34.0/24 Gi0/0/0/0 192.168.13.3 1040
111004 Pop 192.168.23.0/24 Gi0/0/0/0 192.168.13.3 0
111005 Pop 192.168.35.0/24 Gi0/0/0/0 192.168.13.3 0
111006 Unlabelled 1.1.1.1/32[V] Gi0/0/0/1 172.16.16.1 0

-------------------------------------------------------------------

R3#sh mpls forwarding-table
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
300 Pop Label 7.7.7.7/32 0 Fa0/1 192.168.37.7
301 Pop Label 192.168.78.0/24 0 Fa0/1 192.168.37.7
302 Pop Label 192.168.67.0/24 0 Fa0/1 192.168.37.7
303 Pop Label 192.168.57.0/24 0 Fa0/1 192.168.37.7
304 701 8.8.8.8/32 7640 Fa0/1 192.168.37.7
305 Pop Label 172.168.168.4/32 0 Fa0/0 172.168.168.4
306 434347 11.11.11.11/32 10674 Fa0/0 172.168.168.4

[feroz syed]

both side the vpnv4 neighbor went down the loopback address not reachable at both end.

RP/0/0/CPU0:XR-1#sh run router bg
Thu Nov 10 19:27:40.250 UTC
router bgp 12345
address-family ipv4 unicast
network 11.11.11.11/32
allocate-label all
!
address-family vpnv4 unicast

neighbor 44.44.44.44
remote-as 12345
update-source Loopback0
address-family ipv4 labeled-unicast
next-hop-self (removed)

[Harold Ritter]

Hi @feroz syed ,

Did you configure next-hop-self on XR-4 for the iBGP LU session to XR-1? That is the one thing I forgot to tell you.

router bgp 12345

neighbor 11.11.11.11
address-family ipv4 labeled-unicast
next-hop-self

Regards,

[feroz syed]

its already configured,

RP/0/0/CPU0:XR-4#sh run router bgp
Thu Nov 10 20:32:22.445 UTC
router bgp 12345
address-family ipv4 unicast
allocate-label all
!
neighbor 11.11.11.11
remote-as 12345
update-source Loopback0
address-family ipv4 labeled-unicast
next-hop-self
!
!
neighbor 172.168.168.3
remote-as 321
address-family ipv4 labeled-unicast
route-policy PASS in
route-policy PASS out

[feroz syed]

RP/0/0/CPU0:XR-4#sh ip route
Thu Nov 10 20:35:36.081 UTC

Codes: C - connected, S - static, R - RIP, B - BGP, (>) - Diversion path
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP
i - ISIS, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, su - IS-IS summary null, * - candidate default
U - per-user static route, o - ODR, L - local, G - DAGR, l - LISP
A - access/subscriber, a - Application route
M - mobile route, r - RPL, (!) - FRR Backup path

Gateway of last resort is not set

B 8.8.8.8/32 [20/3] via 172.168.168.3, 03:05:31
O 11.11.11.11/32 [110/3] via 192.168.34.3, 03:08:19, GigabitEthernet0/0/0/0
O 33.33.33.33/32 [110/2] via 192.168.34.3, 03:08:33, GigabitEthernet0/0/0/0
L 44.44.44.44/32 is directly connected, 03:08:41, Loopback0
C 172.168.168.0/24 is directly connected, 03:08:39, GigabitEthernet0/0/0/1
S 172.168.168.3/32 is directly connected, 03:08:39, GigabitEthernet0/0/0/1
L 172.168.168.4/32 is directly connected, 03:08:39, GigabitEthernet0/0/0/1
O 192.168.13.0/24 [110/2] via 192.168.34.3, 03:08:33, GigabitEthernet0/0/0/0
O 192.168.23.0/24 [110/2] via 192.168.34.3, 03:08:33, GigabitEthernet0/0/0/0
C 192.168.34.0/24 is directly connected, 03:08:39, GigabitEthernet0/0/0/0
L 192.168.34.4/32 is directly connected, 03:08:39, GigabitEthernet0/0/0/0
O 192.168.35.0/24 [110/2] via 192.168.34.3, 03:08:33, GigabitEthernet0/0/0/0
RP/0/0/CPU0:XR-4#
RP/0/0/CPU0:XR-4#sh bgp ipv4 labeled-unicast
Thu Nov 10 20:36:26.148 UTC
BGP router identifier 44.44.44.44, local AS number 12345
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0xe0000000 RD version: 8
BGP main routing table version 8
BGP NSR Initial initsync version 2 (Reached)
BGP NSR/ISSU Sync-Group versions 0/0
BGP scan interval 60 secs

Status codes: s suppressed, d damped, h history, * valid, > best
i - internal, r RIB-failure, S stale, N Nexthop-discard
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
*> 8.8.8.8/32 172.168.168.3 3 0 321 i
*>i11.11.11.11/32 11.11.11.11 0 100 0 i

Processed 2 prefixes, 2 paths
RP/0/0/CPU0:XR-4#sh bgp ipv4 labeled-unicast summ
Thu Nov 10 20:36:31.568 UTC
BGP router identifier 44.44.44.44, local AS number 12345
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0xe0000000 RD version: 8
BGP main routing table version 8
BGP NSR Initial initsync version 2 (Reached)
BGP NSR/ISSU Sync-Group versions 0/0
BGP scan interval 60 secs

BGP is operating in STANDALONE mode.

Process RcvTblVer bRIB/RIB LabelVer ImportVer SendTblVer StandbyVer
Speaker 8 8 8 8 8 0

Neighbor Spk AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down St/PfxRcd
11.11.11.11 0 12345 135 136 8 0 0 02:10:01 1
172.168.168.3 0 321 181 194 8 0 0 03:09:28 1

RP/0/0/CPU0:XR-4#sh ip route
Thu Nov 10 20:36:45.437 UTC

Codes: C - connected, S - static, R - RIP, B - BGP, (>) - Diversion path
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP
i - ISIS, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, su - IS-IS summary null, * - candidate default
U - per-user static route, o - ODR, L - local, G - DAGR, l - LISP
A - access/subscriber, a - Application route
M - mobile route, r - RPL, (!) - FRR Backup path

Gateway of last resort is not set

B 8.8.8.8/32 [20/3] via 172.168.168.3, 03:06:40
O 11.11.11.11/32 [110/3] via 192.168.34.3, 03:09:29, GigabitEthernet0/0/0/0
O 33.33.33.33/32 [110/2] via 192.168.34.3, 03:09:42, GigabitEthernet0/0/0/0
L 44.44.44.44/32 is directly connected, 03:09:51, Loopback0
C 172.168.168.0/24 is directly connected, 03:09:48, GigabitEthernet0/0/0/1
S 172.168.168.3/32 is directly connected, 03:09:48, GigabitEthernet0/0/0/1
L 172.168.168.4/32 is directly connected, 03:09:48, GigabitEthernet0/0/0/1
O 192.168.13.0/24 [110/2] via 192.168.34.3, 03:09:42, GigabitEthernet0/0/0/0
O 192.168.23.0/24 [110/2] via 192.168.34.3, 03:09:42, GigabitEthernet0/0/0/0
C 192.168.34.0/24 is directly connected, 03:09:48, GigabitEthernet0/0/0/0
L 192.168.34.4/32 is directly connected, 03:09:48, GigabitEthernet0/0/0/0
O 192.168.35.0/24 [110/2] via 192.168.34.3, 03:09:42, GigabitEthernet0/0/0/0
RP/0/0/CPU0:XR-4#sh mpl ld for
Thu Nov 10 20:36:59.216 UTC

Codes:
- = GR label recovering, (!) = LFA FRR pure backup path
{} = Label stack with multi-line output for a routing path
G = GR, S = Stale, R = Remote LFA FRR backup

Prefix Label Label(s) Outgoing Next Hop Flags
In Out Interface G S R
--------------- ------- -------------- ------------ ------------------- -----
11.11.11.11/32 434347 44446 Gi0/0/0/0 192.168.34.3
33.33.33.33/32 434343 ImpNull Gi0/0/0/0 192.168.34.3
172.168.168.3/32 434348 Unlabelled Gi0/0/0/1 point2point
192.168.13.0/24 434344 ImpNull Gi0/0/0/0 192.168.34.3
192.168.23.0/24 434345 ImpNull Gi0/0/0/0 192.168.34.3
192.168.35.0/24 434346 ImpNull Gi0/0/0/0 192.168.34.3

[Harold Ritter]

Hi @feroz syed ,

I ran a quick recreate and hit the same issues you did. For some reason, it looks like XR-1 will not install the LDP received label in the CEF entry. It looks like the solution is not to run the VPNv4 session directly between XR-1 and R8, but rather have a eBGP VPNv4 unicast between the two route reflectors, as described in the following document and which is more likely to happen in a production network anyhow:

https://www.cisco.com/c/en/us/support/docs/multiprotocol-label-switching-mpls/mpls/200523-Configuration-and-Verification-of-Layer.html

So in your scenario, you could run the eBGP VPNv4 unicast session between XR-3 and R7 and reflect these routes to XR1 and R8.

It worked for me. Please give it a try and let me know if you have any questions.

Regards,

[feroz syed]

hello mr. @Harold Ritter thanks for your reply, am already working on CML same topology to see it work or not, however if it's same means, i will follow your suggestion, again thanks for your help.

[Harold Ritter · Accepted solution]

Hi @feroz syed ,

I found the reason why CEF didn't install the LDP learned label for 8.8.8.8/32 on XR-1. It is because you need to use the following command for the eBGP multi hop session to 8.8.8.8 (R8).

router bgp 12345
neighbor 8.8.8.8
ebgp-multihop mpls

By default, XR forces ebgp multi hop sessions via a non label switched path (non MPLS path) by installing an implicit null label for the specific neighbor, which breaks the end connectivity in the case of vpnv4 traffic. The "ebgp-multihop mpls" disables that default behaviour.

With the addition of this command the CEF entry for 8.8.8.8/32 will be properly installed on XR-1 and you will be able to get your original design should work as is.

Regards,

[feroz syed]

if you don't mind, could you please describe how did you troubleshoot the issue, how did you discover t.

[Harold Ritter]

Hi @feroz syed ,

I was really puzzled by the fact that XR-1 did not install the LDP received label. At first I thought it could be a XRv 6.3.1 limitation. I had a flash last night. I had already seen a similar situation here on the Cisco Support Community and remembered that it was related to the "ebgp-multihop" command. I looked it up and found the information I provided.

This is really a corner case, as very few people will have an eBGP VPNv4 multi hop session from one PE in one AS to a PE in the other AS. The most common solution is to have the eBGP VPNv4 session between RRs in each AS.

The reason why it works without the "ebgp-multihop mpls" when the eBGP VPNv4 multi hop session is between the RRs is that the RR is only control plane and therefore doesn't mind to learn updates from the other AS via a non label switched path.

On the other hand, if the session is from PE to PE, the control plane will work just as it does when it is from RR to RR and VPNv4 routes are learned correctly, but since the routes are learned via a non label switched path, which is the default behaviour for "ebgp-multihop" in XR, the data plane breaks because it uses the same non label switched path as the control plane and this is not acceptable for the VPNv4 data plane.

So in conclusion, "ebgp-multihop mpls" is required if both the eBGP VPNv4 control plane and data plane share the same path.

Regards,

[Rustam Safarov]

Thanks a lot for your help and your time.