MPLSL3VPN in a two sub-as confederation and route reflectors (IOS XR).

[kod34]

I have this mplsl3vpn architecture that doesn't seem to work.

I made two sets of configurations, in the first one I configured a neighbors relationships between PEs of the same client and the ping works in inter sub-AS let's say for example from CE1 to CE8 (I linked the configuration files).

In the second one I removed neighbor relationships from the PEs and gave it to the RRs and the ping this time worked between CEs of the same sub-AS let's say CE1 to CE4, but not inter sub-AS from CE1 to CE8. 

I used ospf in the PE-CE links and is-is, mpls and MP-BGP in the core.

[MHM Cisco World]

Can you share config as text 

[kod34]

## CE1
hostname CE1
interface Loopback0
ip address 1.1.1.1 255.255.255.255
ip ospf network point-to-point
ip ospf 1 area 0
!
interface GigabitEthernet0/0
ip address 111.111.111.1 255.255.255.0
ip ospf network point-to-point
ip ospf 1 area 0
no shut
!
interface GigabitEthernet0/2
ip address 117.117.117.1 255.255.255.0
ip ospf network point-to-point
ip ospf 1 area 0
no shut

router ospf 1
router-id 1.1.1.1

## PE1

hostname PE1
vrf C1
address-family ipv4 unicast
import route-target
63000:100
!
export route-target
63000:100
!
!
!
vrf C3
address-family ipv4 unicast
import route-target
63000:300
!
export route-target
63000:300
!
!
!
interface Loopback0
ipv4 address 11.11.11.11 255.255.255.255
!
interface GigabitEthernet0/0/0/0
vrf C1
ipv4 address 111.111.111.11 255.255.255.0
no shut
!
interface GigabitEthernet0/0/0/1
vrf C3
ipv4 address 112.112.112.11 255.255.255.0
no shut
!
interface GigabitEthernet0/0/0/2
ipv4 address 48.17.17.11 255.255.255.0
no shut
!
interface GigabitEthernet0/0/0/3
ipv4 address 42.11.11.11 255.255.255.0
no shut
!
router isis core
net 49.a000.0000.0011.00
address-family ipv4 unicast
metric-style wide
!
interface Loopback0
passive
address-family ipv4 unicast
!
!
interface GigabitEthernet0/0/0/2
circuit-type level-2-only
address-family ipv4 unicast
!
!
interface GigabitEthernet0/0/0/3
circuit-type level-2-only
address-family ipv4 unicast
!
!
!
router ospf 100
router-id 11.11.11.11
vrf C1
redistribute bgp 64512
address-family ipv4 unicast
area 0
interface GigabitEthernet0/0/0/0
network point-to-point
!
!
!
vrf C3
redistribute bgp 64512
address-family ipv4 unicast
area 0
interface GigabitEthernet0/0/0/1
network point-to-point
!
!
!
!
router bgp 64512
bgp confederation identifier 63000
bgp router-id 11.11.11.11
address-family ipv4 unicast
!
address-family vpnv4 unicast
!
neighbor-group Ps
remote-as 64512
update-source Loopback0
address-family ipv4 unicast
!
address-family vpnv4 unicast
!
!
neighbor-group RRs
remote-as 64512
update-source Loopback0
address-family ipv4 unicast
!
address-family vpnv4 unicast
!
!
neighbor 71.71.81.81
use neighbor-group RRs
!
neighbor 72.72.82.82
use neighbor-group RRs
!
neighbor 31.31.31.31
use neighbor-group Ps
!
neighbor 37.37.37.37
use neighbor-group Ps
!
!
vrf C1
rd 63000:1
address-family ipv4 unicast
redistribute ospf 100
!
!
vrf C3
rd 63000:3
address-family ipv4 unicast
redistribute ospf 100
!
!
!
mpls ldp
router-id 11.11.11.11
interface GigabitEthernet0/0/0/2
interface GigabitEthernet0/0/0/3
!
!
mpls label range table 0 16000 16200
end

## P1 

hostname P1
interface Loopback0
ipv4 address 31.31.31.31 255.255.255.255
!
interface GigabitEthernet0/0/0/0
ipv4 address 102.11.11.31 255.255.255.0
no shut
!
interface GigabitEthernet0/0/0/1
ipv4 address 48.17.17.31 255.255.255.0
no shut
!
interface GigabitEthernet0/0/0/2
ipv4 address 103.12.12.31 255.255.255.0
no shut
!
interface GigabitEthernet0/0/0/5
ipv4 address 42.11.11.31 255.255.255.0
no shut
!
router isis core
is-type level-2-only
net 49.1000.0000.0031.00
address-family ipv4 unicast
metric-style wide
!
interface Loopback0
passive
address-family ipv4 unicast
!
!
interface GigabitEthernet0/0/0/0
circuit-type level-2-only
address-family ipv4 unicast
!
!
interface GigabitEthernet0/0/0/1
circuit-type level-2-only
address-family ipv4 unicast
!
!
interface GigabitEthernet0/0/0/2
circuit-type level-2-only
address-family ipv4 unicast
!
!
interface GigabitEthernet0/0/0/5
circuit-type level-2-only
address-family ipv4 unicast
!
!
!
router bgp 64512
bgp confederation identifier 63000
bgp router-id 31.31.31.31
address-family ipv4 unicast
!
address-family vpnv4 unicast
!
neighbor-group allofem
remote-as 64512
update-source Loopback0
address-family ipv4 unicast
!
address-family vpnv4 unicast
!
!
neighbor 17.17.17.17
use neighbor-group allofem
!
neighbor 11.11.11.11
use neighbor-group allofem
!
neighbor 71.71.81.81
use neighbor-group allofem
!
neighbor 72.72.82.82
use neighbor-group allofem
!
!
mpls ldp
router-id 31.31.31.31
interface GigabitEthernet0/0/0/0
!
interface GigabitEthernet0/0/0/1
!
interface GigabitEthernet0/0/0/2
!
interface GigabitEthernet0/0/0/5
!
!
mpls label range table 0 18201 18400
end

## RR1

hostname RR1
interface Loopback0
ipv4 address 71.71.81.81 255.255.255.255
!
interface MgmtEth0/0/CPU0/0
shutdown
!
interface GigabitEthernet0/0/0/0
ipv4 address 102.11.11.81 255.255.255.0
!
interface GigabitEthernet0/0/0/1
ipv4 address 104.13.13.81 255.255.255.0
!
interface GigabitEthernet0/0/0/2
ipv4 address 143.12.12.81 255.255.255.0
!
interface GigabitEthernet0/0/0/3
ipv4 address 105.14.14.81 255.255.255.0
!
interface GigabitEthernet0/0/0/4
ipv4 address 144.13.13.81 255.255.255.0
!
interface GigabitEthernet0/0/0/5
ipv4 address 108.17.17.81 255.255.255.0
!
interface GigabitEthernet0/0/0/6
ipv4 address 111.101.101.81 255.255.255.0
!
interface GigabitEthernet0/0/0/7
ipv4 address 110.19.19.81 255.255.255.0
!
interface GigabitEthernet0/0/0/8
ipv4 address 199.1.1.1 255.255.255.0
!
route-policy ebgp
pass
end-policy
!
router isis core
is-type level-2-only
net 49.b000.0000.0081.00
address-family ipv4 unicast
metric-style wide
!
interface Loopback0
passive
address-family ipv4 unicast
!
!
interface GigabitEthernet0/0/0/0
circuit-type level-2-only
address-family ipv4 unicast
!
!
interface GigabitEthernet0/0/0/1
circuit-type level-2-only
address-family ipv4 unicast
!
!
interface GigabitEthernet0/0/0/2
circuit-type level-2-only
address-family ipv4 unicast
!
!
interface GigabitEthernet0/0/0/3
circuit-type level-2-only
address-family ipv4 unicast
!
!
interface GigabitEthernet0/0/0/4
circuit-type level-2-only
address-family ipv4 unicast
!
!
interface GigabitEthernet0/0/0/5
circuit-type level-2-only
address-family ipv4 unicast
!
!
interface GigabitEthernet0/0/0/6
circuit-type level-2-only
address-family ipv4 unicast
!
!
interface GigabitEthernet0/0/0/7
circuit-type level-2-only
address-family ipv4 unicast
!
!
interface GigabitEthernet0/0/0/8
circuit-type level-2-only
address-family ipv4 unicast
!
!
!
router bgp 64512
bgp confederation peers
65535
!
bgp confederation identifier 63000
bgp router-id 71.71.81.81
address-family ipv4 unicast
!
address-family vpnv4 unicast
!
neighbor-group Ps
remote-as 64512
update-source Loopback0
address-family ipv4 unicast
route-reflector-client
!
address-family vpnv4 unicast
route-reflector-client
!
!
neighbor-group PEs
remote-as 64512
update-source Loopback0
address-family ipv4 unicast
route-reflector-client
!
address-family vpnv4 unicast
route-reflector-client
!
!
neighbor-group RRs_in_mySub
remote-as 64512
update-source Loopback0
address-family ipv4 unicast
!
address-family vpnv4 unicast
!
!
neighbor-group RRs_out_mySub
remote-as 65535
ebgp-multihop 255
update-source Loopback0
address-family ipv4 unicast
route-policy ebgp in
route-policy ebgp out
!
address-family vpnv4 unicast
route-policy ebgp in
route-policy ebgp out
!
!
neighbor 11.11.11.11
use neighbor-group PEs
!
neighbor 12.12.12.12
use neighbor-group PEs
!
neighbor 13.13.13.13
use neighbor-group PEs
!
neighbor 17.17.17.17
use neighbor-group PEs
!
neighbor 18.18.18.18
use neighbor-group PEs
!
neighbor 19.19.19.19
use neighbor-group PEs
!
neighbor 31.31.31.31
use neighbor-group Ps
!
neighbor 33.33.33.33
use neighbor-group Ps
!
neighbor 34.34.34.34
use neighbor-group Ps
!
neighbor 37.37.37.37
use neighbor-group Ps
!
neighbor 39.39.39.39
use neighbor-group Ps
!
neighbor 40.40.40.40
use neighbor-group Ps
!
neighbor 72.72.82.82
use neighbor-group RRs_in_mySub
!
neighbor 73.73.83.83
use neighbor-group RRs_out_mySub
!
neighbor 74.74.84.84
use neighbor-group RRs_out_mySub
!
!
mpls ldp
router-id 71.71.81.81
interface GigabitEthernet0/0/0/0
!
interface GigabitEthernet0/0/0/1
!
interface GigabitEthernet0/0/0/2
!
interface GigabitEthernet0/0/0/3
!
interface GigabitEthernet0/0/0/4
!
interface GigabitEthernet0/0/0/5
!
interface GigabitEthernet0/0/0/6
!
interface GigabitEthernet0/0/0/7
!
interface GigabitEthernet0/0/0/8
!
!
mpls label range table 0 20201 20400
end

The other routers follow roughly a similar configuration, when this config is applied intra sub-AS pings works but not inter sub-AS pings between loopbacks of CEs.

[Giuseppe Larosa]

Hello @kod34 ,

I did a similar lab setup many years ago using IOS routers

The key point is that to build MP e BGP sessions between RR servers belonging to different sub-ASes and sourced by your loopback0 you need neighbor xxx  ebgp-multihop 3 or more and you need to activate the neighbors and to send community both under each address family specially vpnv4.

send community both is a way to send standard BGP communities and extended BGP communities i.e. route targets

neighbor-group RRs_out_mySub
remote-as 65535
>> ebgp-multihop 255
update-source Loopback0
address-family ipv4 unicast
route-policy ebgp in
route-policy ebgp out

address-family vpnv4 unicast
route-policy ebgp in
route-policy ebgp out
!

>>> end of your config

address-family vpnv4

neighbor x.x.x.x activate

neighobr x.x.x.x send-community both

You need to apply different route policies under address family vpnv4 unicast

You are applying the same policies to eBGP ipv4 and to  AF vpnv4 unicast

in IPv4 eBGP no extended communities are involved

in VPNv4 unicast MP BGP you need to be able to propagate the route targets extcommunties to the other sub ASes and you need to configure the devices to be able to accept them.

Hope to help

Giuseppe

[kod34]

Hello, thanks for your comment, I tried what you said but unfortunately it didn't work, in IOS XR send-community-ebgp and send-extended-community-ebgp fall under address-family ipv4 unicast config mode but not under address-family vpvn4 unicast mode.

I'm also not sure if there should be a neighbor relationship between PEs with the same VRF of different sub-ASs or not.

[Giuseppe Larosa]

Hello @kod34 .

you need to import the route targets coming from the other sub Ases or you need to add at RRS s lòcally defined route target that will be imported by RRS client PE nodes iin same sub AS.

remember thst sub ASes are only a solution to scalabilty issues caused by iBGP split horizon rule:

or you use RRS or you use BGP conferations or both and the last is your case.

It is a single provider for outer world you are not in an inter -AS or CSC contect.

Find the way to accept the appropriate route targets and then you will be able to achieve connectivity

Hope to help

Giuseppe

[kod34]

Thanks for your insight, I ended up making the Ps of the first sub in full mesh with the Ps of the second sub for the data plane and the RRs for the control plane, the RRs have a neighborship with the PEs while there is no neighborship between the Ps and the PEs nor the Ps and the RRs. Not sure if that's the optimal way to do it but it works.