eBGP in MPLS network

[James Rune Hansen]

Hi

I have an MPLS network using Cisco routers.
PE routers mostly IOS-XE
2 RR IOS-XR running full table from upstream in a VRF.

The RR also run my eBGP peerings to my upstreams.
The Internet routing table is in a VRF.
The RR send a default route to all PE's
And i have a lot of customers running on that platform.

I have a costumer that want full table, so i have set him up with a peering directly to the RR's

So he is connected to a PE router with a /28 he uses to establish a eBGP peering to both my RR's.
The RR's are using a loopback in the internet VRF to do the peering.
Peering is up.
Right now he advertise 22.33.44.0/24 to me and gets the full table in return.
BUT traffic to his /24 (22.33.44.0/24) does not jump out to his CE from the PE

In the PE
PE44#sh ip route vrf INET 22.33.44.0

Routing Table: INET
Routing entry for 22.33.44.0/24
Known via "bgp 2222", distance 200, metric 0
Tag 4444, type internal
Last update from 10.10.10.106 00:00:08 ago
Routing Descriptor Blocks:
* 10.10.10.106 (default), from 10.10.10.103, 00:00:08 ago
Route metric is 0, traffic share count is 1
AS Hops 1
Route tag 4444
MPLS label: 16200
MPLS Flags: MPLS Required
So the PE router routes the traffic back to 10.10.10.106 (The RR)

Traceroute from the RR to the IP in the net he tries to announce.
RR1#traceroute vrf INET 22.33.44.1

1 11.11.11.194 [MPLS: Labels 91/29 Exp 0] 3 msec 3 msec 3 msec
2 11.11.11.186 [MPLS: Labels 81/29 Exp 0] 2 msec 2 msec 2 msec
3 11.11.11.106 [MPLS: Labels 57/29 Exp 0] 2 msec 2 msec 2 msec
4 44.44.44.121 2 msec 2 msec 2 msec
5 11.11.11.81 [MPLS: Labels 71/16200 Exp 0] 3 msec 3 msec 2 msec
6 11.11.11.105 [MPLS: Labels 100/16200 Exp 0] 3 msec 3 msec 3 msec
7 11.11.11.185 [MPLS: Labels 49/16200 Exp 0] 3 msec 3 msec 3 msec
8 11.11.11.193 [MPLS: Label 16200 Exp 0] 3 msec 2 msec 2 msec
9 11.11.11.194 [MPLS: Labels 91/29 Exp 0] 3 msec 3 msec 2 msec
10 11.11.11.186 [MPLS: Labels 81/29 Exp 0] 3 msec 3 msec 2 msec
11 11.11.11.106 [MPLS: Labels 57/29 Exp 0] 8 msec 3 msec 3 msec
12 44.44.44.121 3 msec 3 msec 3 msec
13 11.11.11.81 [MPLS: Labels 71/16200 Exp 0] 4 msec 4 msec 5 msec
14 11.11.11.105 [MPLS: Labels 100/16200 Exp 0] 3 msec 3 msec 3 msec
15 11.11.11.185 [MPLS: Labels 49/16200 Exp 0] 4 msec 3 msec 3 msec
16 11.11.11.193 [MPLS: Label 16200 Exp 0] 3 msec 3 msec 3 msec
17 11.11.11.194 [MPLS: Labels 91/29 Exp 0] 3 msec 3 msec 3 msec
18 11.11.11.186 [MPLS: Labels 81/29 Exp 0] 3 msec 3 msec 3 msec
19 11.11.11.106 [MPLS: Labels 57/29 Exp 0] 3 msec 3 msec 3 msec
20 44.44.44.121 3 msec 3 msec 3 msec
21 11.11.11.81 [MPLS: Labels 71/16200 Exp 0] 4 msec 4 msec 4 msec
22 11.11.11.105 [MPLS: Labels 100/16200 Exp 0] 4 msec 4 msec 4 msec
23 11.11.11.185 [MPLS: Labels 49/16200 Exp 0] 4 msec 4 msec 4 msec
24 11.11.11.193 [MPLS: Label 16200 Exp 0] 4 msec 4 msec 4 msec
25 11.11.11.194 [MPLS: Labels 91/29 Exp 0] 4 msec 4 msec 4 msec
26 11.11.11.186 [MPLS: Labels 81/29 Exp 0] 4 msec 4 msec 4 msec
27 11.11.11.106 [MPLS: Labels 57/29 Exp 0] 4 msec 4 msec 4 msec
28 44.44.44.121 4 msec 4 msec 4 msec
29 11.11.11.81 [MPLS: Labels 71/16200 Exp 0] 5 msec 4 msec 4 msec
30 11.11.11.105 [MPLS: Labels 100/16200 Exp 0] 4 msec 5 msec 5 msec

44.44.44.121 is actually an ip on the PE router in the INET vrf, but it belongs to an other costumers /30 and is an BDI interface on the PE.
A side note - When i shut that BDI (44.44.44.121) line 4,12,20 and 28 in the traceroute, the ip 44.44.44.121 is replaced by the right costumers BDI interface IP.

I have tried to add "next-hop-unchanged" to the neighbor on the RR, but the command is not allowed when in VRF.

What am i doing wrong.

[James Rune Hansen]

In all the text i forgot to write that the CE router that peers with the RR are using ebgp multihop  

[MHM Cisco World]

let get something here, 
RR is VRF to other Peer and get full internet table 
you have CE connect to PE and you want to eBGP between PE and RR in such that the CE get full table ?
if that right, can you double check the route-target export in RR and import in PE.

[James Rune Hansen]

RR is RR for many VRF's incl. the INET where all the public adresses are.
I wan't eBGP from CE to RR. But CE is connected to PE to gain access to the INET VRF. But Peering is from CE (Not MPLS aware) to RR.

Route target on RR:

vrf INET
address-family ipv4 unicast
import route-target
2222:200
!
export route-target
2222:200

Route target on PE

ip vrf INET
description Internet
rd 2222:200
route-target export 2222:200
route-target import 2222:200

[MHM Cisco World]

OK, I think but the issue is in PE,
CE is connect to PE in VRF X, in order to make CE direct connect to RR we need to make CE IP reachable by RR, 
that mean we need leak VRF to global routing table of MPLS Core, and I think that pad option. 
So
how we can override the MPLS Core? I think and I do lab, 
in small Lab I build GRE tunnel between PE and RR 
this tunnel use to forward traffic between CE and RR over MPLS Core,
the tunnel is in same VRF of CE-PE and config only static route toward tunnel interface for each side RR and PE. 
and this give me direct link between CE and RR without need to add any IP to global table of MPLS Core. 

hope this idea help you. 
good luck 

[Harold Ritter]

Hi @James Rune Hansen ,

Here's a brief explanation of what you are seeing.

- RR learns 22.33.44.0/24 with a next hop set to the ip address that belongs to the /28 directly connected to PE44.

- You redistribute connected on PE44, which will create an aggregate label for the /28

- Packets sent from the RR towards 22.33.44.0/24 will contain the aggregate label and an IGP label. The latter will be popped on the  penultimate router (last router before getting to PE44). Packets towards 22.33.44.0/24 will reach PE44 with only the aggregate label.

- The aggregate label behaviour is that the label is popped and a lookup is performed against the VRF routing table.

- Given that 22.33.44.0/24 is received from the RR and installed in the VRF INET routing table, a routing loop occurs, sending packets back to the RR.

I would recommend you establish the eBGP session on the CE router behind PE44 using a loopback address on that CE. You need to add a static route in VRF INET on PE44 to reach that loopback address and  do "redistribute static" under "address-family ipv4 vrf INET". The label that will be created and installed in the LFIB for the CE loopback address will be a "per-prefix" label, which will cause PE44 to forward the packets destined to 22.33.44.0/24 directly to the CE without an IP lookup. 

This should fix the issue.

Regards, 

[James Rune Hansen]

Hi Herold

Makes sense.. I will try it out, as soon as i have a window. 

But it sounds 100% like you nailed it.
I will give a feedback when i have done the test.

Thanks a lot 

/James