Re: MPLS L3VPN: Hosts not reachable to each other

tonitones10 · ‎05-15-2023

Hello everyone,

This will be my first discussion here. Just like the title says, host1 of CE1 and CE2 cannot reach each other. I'm not well versed with MPLS, nor an expert with BGP, OSPF, and the likes. This will be my first time configuring MPLS L3VPN.

Further details are provided below.

This is the current topology:

P1(AS65001) ---- P2(AS65001)
    |                |
PE1(AS65001)       PE2(AS65001)
    |                |
CE1(AS65004)       CE2(AS65004)
    |                |
HostA(PC)           HostB(PC)

A little bit background of the topology: P1, P2, PE1, and PE2 are existing routers in production. BGP, OSPF, P2P links are also existing in production (just replicated in simulator) except for the MPLS configs and CE1 & CE2 routers. P1 is the RR of P2. P2 is the RR of PE2. PE1 is linked to P1 through iBGP. CE1 and CE2 are linked to their respective PE's through eBGP. What I'm trying to accomplish is to make HostA and HostB reachable to each other with the existing configs + the MPLS configs.

Please see the configs below for the routers included in the topology. Not all of the configs are pasted, only the configs that are relevant (I think) to the current problem.

CE1 and CE2 configs:

CE1

interface Loopback0
 description CE1-Lo
 ip address 1.1.1.1 255.255.255.255
!
interface Port-channel1.303
 description PE1 LINK
 encapsulation dot1Q 303
 ip address 100.65.0.10 255.255.255.248
!
router bgp 65004
 bgp router-id 1.1.1.1
 bgp log-neighbor-changes
 neighbor 100.65.0.9 remote-as 65001
 !
 address-family ipv4
  network 192.168.101.0
  neighbor 100.65.0.9 activate
 exit-address-family
!
ip route 192.168.101.0 255.255.255.0 Null0 200

CE2

interface Loopback0
 description CE2-Lo
 ip address 2.2.2.2 255.255.255.255
!
interface Port-channel1.301
 description PE2 LINK
 encapsulation dot1Q 301
 ip address 100.65.0.2 255.255.255.252
!
router bgp 65004
 bgp router-id 2.2.2.2
 bgp log-neighbor-changes
 neighbor 100.65.0.1 remote-as 65001
 neighbor 100.65.0.1 description PE2
 !
 address-family ipv4
  network 192.168.99.0
  neighbor 100.65.0.1 activate
 exit-address-family
!
ip route 192.168.99.0 255.255.255.0 Null0 200

PE1 and PE2 Configs:

PE1

vrf definition CustA_CE1
 rd 65004:10
 route-target export 65004:10
 route-target import 65004:20
 !
 address-family ipv4
 exit-address-family
!
!
mpls label range 500 599
no mpls ip propagate-ttl forwarded
!
interface Loopback0
 description PE1-Lo0
 ip address 172.16.35.0 255.255.255.255
!
interface Port-channel1.100
 description P1 iBGP LINK
 encapsulation dot1Q 100
 ip address 100.75.0.2 255.255.255.252
 mpls ip
 mpls label protocol ldp
!
router bgp 65001
 bgp router-id 172.16.35.0
 bgp log-neighbor-changes
 neighbor 100.75.0.1 remote-as 65001
 neighbor 100.75.0.1 description P1
 !
 address-family ipv4
  neighbor 100.75.0.1 activate
 exit-address-family
 !
 address-family vpnv4
  neighbor 100.75.0.1 activate
  neighbor 100.75.0.1 send-community both
 exit-address-family
 !
 address-family ipv4 vrf CustA_CE1
  neighbor 100.65.0.10 remote-as 65004
  neighbor 100.65.0.10 description CustA_CE1
  neighbor 100.65.0.10 activate
  neighbor 100.65.0.10 as-override
 exit-address-family
!
ip route 0.0.0.0 0.0.0.0 100.75.0.1
!
mpls ldp router-id Loopback0 force

PE2

vrf definition CustA_CE1
 rd 65004:10
 route-target export 65004:10
 route-target import 65004:20
 !
 address-family ipv4
 exit-address-family
!
mpls label range 100 199
no mpls ip propagate-ttl forwarded
!
interface Loopback0
 description PE2-Lo
 ip address 172.16.16.0 255.255.255.255
!
interface Port-channel1.110
 description P1 OSPF LINK
 encapsulation dot1Q 110
 ip address 100.75.0.10 255.255.255.252
 ip ospf network point-to-point
 ip ospf hello-interval 2
 ip ospf cost 1
 mpls ip
 mpls label protocol ldp
!
interface Port-channel1.301
 description CE2 LINK
 encapsulation dot1Q 301
 vrf forwarding CustA_CE2
 ip address 100.65.0.1 255.255.255.252
!
router ospf 100
 router-id 172.16.16.0
 passive-interface default
 no passive-interface Port-channel1.110
 network 100.75.0.8 0.0.0.3 area 0
 network 172.16.16.0 0.0.0.0 area 16
!
router bgp 65001
 bgp router-id 172.16.16.0
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor 172.16.0.0 remote-as 65001
 neighbor 172.16.0.0 description P2
 neighbor 172.16.0.0 update-source Loopback0
 !
 address-family ipv4
  neighbor 172.16.0.0 activate
 exit-address-family
 !
 address-family vpnv4
  neighbor 172.16.0.0 activate
  neighbor 172.16.0.0 send-community both
 exit-address-family
 !
 address-family ipv4 vrf CustA_CE2
  neighbor 100.65.0.2 remote-as 65004
  neighbor 100.65.0.2 activate
  neighbor 100.65.0.2 as-override
 exit-address-family
 !

P1 and P2 Configs:

P1

mpls label range 400 499
!
interface Loopback0
 description P1-Lo0
 ip address 172.16.0.249 255.255.255.255
!
interface Port-channel1.100
 description PE1 iBGP LINK
 encapsulation dot1Q 100
 ip address 100.75.0.1 255.255.255.252
 mpls ip
 mpls label protocol ldp
!
interface Port-channel1.200
 description P2 OSPF LINK
 encapsulation dot1Q 200
 ip address 100.75.0.5 255.255.255.252
 ip ospf network point-to-point
 ip ospf hello-interval 2
 ip ospf cost 1
 mpls ip
 mpls label protocol ldp
!
router ospf 100
 router-id 172.16.0.249
 passive-interface default
 no passive-interface Port-channel1.200
 network 100.75.0.4 0.0.0.3 area 0
 network 172.16.0.249 0.0.0.0 area 0
!
router bgp 65001
 bgp router-id 172.16.0.249
 bgp log-neighbor-changes
 neighbor 100.75.0.2 remote-as 65001
 neighbor 100.75.0.2 description PE1
 neighbor 172.16.0.0 remote-as 65001
 neighbor 172.16.0.0 description P2
 neighbor 172.16.0.0 update-source Loopback0
 !
 address-family ipv4
  network 100.75.0.0 mask 255.255.255.252
  neighbor 100.75.0.2 activate
  neighbor 100.75.0.2 next-hop-self
  neighbor 172.16.0.0 activate
  neighbor 172.16.0.0 route-reflector-client
  neighbor 172.16.0.0 next-hop-self
 exit-address-family
 !
 address-family vpnv4
  neighbor 100.75.0.2 activate
  neighbor 100.75.0.2 send-community both
  neighbor 172.16.0.0 activate
  neighbor 172.16.0.0 send-community both
  neighbor 172.16.0.0 route-reflector-client
 exit-address-family
!

P2

mpls label range 100 199
no mpls ip propagate-ttl forwarded
!
interface Loopback0
 description P2-Lo0
 ip address 172.16.0.0 255.255.255.255
!
interface Port-channel1.110
 description PE2 LINK OSPF
 encapsulation dot1Q 110
 ip address 100.75.0.9 255.255.255.252
 ip ospf network point-to-point
 ip ospf hello-interval 2
 ip ospf cost 1
 mpls ip
 mpls label protocol ldp
!
interface Port-channel1.200
 description P1 LINK OSPF
 encapsulation dot1Q 200
 ip address 100.75.0.6 255.255.255.252
 ip ospf network point-to-point
 ip ospf hello-interval 2
 ip ospf cost 1
 mpls ip
 mpls label protocol ldp
!
router ospf 100
 router-id 172.16.0.0
 passive-interface default
 no passive-interface Port-channel1.110
 no passive-interface Port-channel1.200
 network 100.75.0.4 0.0.0.3 area 0
 network 100.75.0.8 0.0.0.3 area 0
 network 172.16.0.0 0.0.0.0 area 0
!
router bgp 65001
 bgp router-id 172.16.0.0
 bgp log-neighbor-changes
 neighbor 172.16.0.249 remote-as 65001
 neighbor 172.16.0.249 description P1
 neighbor 172.16.0.249 update-source Loopback0
 neighbor 172.16.16.0 remote-as 65001
 neighbor 172.16.16.0 description PE2
 neighbor 172.16.16.0 update-source Loopback0
 !
 address-family ipv4
  neighbor 172.16.0.249 activate
  neighbor 172.16.16.0 activate
  neighbor 172.16.16.0 route-reflector-client
 exit-address-family
 !
 address-family vpnv4
  neighbor 172.16.0.249 activate
  neighbor 172.16.0.249 send-community both
  neighbor 172.16.16.0 activate
  neighbor 172.16.16.0 send-community both
  neighbor 172.16.16.0 route-reflector-client
 exit-address-family
!

Additional details that might help:

PE1

PE1# show bgp vpnv4 unicast all
BGP table version is 14, local router ID is 172.16.35.0
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
              x best-external, a additional-path, c RIB-compressed,
              t secondary path,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 65004:10
 *>i  192.168.99.0     172.16.16.0              0    100      0 65004 i
Route Distinguisher: 65004:30 (default for vrf CustA_CE1)
 *>i  192.168.99.0     172.16.16.0              0    100      0 65004 i
 *>   192.168.101.0    100.65.0.10              0             0 65004 i

PE2

PE2# show bgp vpnv4 unicast all
BGP table version is 23, local router ID is 172.16.16.0
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
              x best-external, a additional-path, c RIB-compressed,
              t secondary path,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 65004:10 (default for vrf CustA_CE2)
 *>   192.168.99.0     100.65.0.2               0             0 65004 i
 *>i  192.168.101.0    100.75.0.2               0    100      0 65004 i
Route Distinguisher: 65004:20
 *>i  192.168.101.0    100.75.0.2               0    100      0 65004 i

I can see the subnets for each other but the hosts cannot reach each other. I may be missing something here, hoping that you guys can help.

-tonitones

M02@rt37 · ‎05-15-2023

Hello @tonitones10

On your PE-2, on your BGP instance, you have vrf called "CustA_CE2" but not declared on your configuration:

address-family ipv4 vrf CustA_CE2
neighbor 100.65.0.2 remote-as 65004
neighbor 100.65.0.2 activate
neighbor 100.65.0.2 as-override
exit-address-family

After modifications, please poste the output:

PE1: sh bgp vrf CustA_CE1 neighbor 100.65.0.10 routes

PE2: sh bgp vrf CustA_CE1 neighbor 100.65.0.2 routes

Best regards
.ı|ı.ı|ı. If This Helps, Please Rate .ı|ı.ı|ı.

tonitones10 · ‎05-15-2023

Hi!

Apologies, its a copy-paste error on my end. I double checked my config on PE2, it checks out vrf CustA_CE2. Did not modify anything.

But here's the output:

PE1

PE1#sh bgp vrf CustA_CE1 neighbor 100.65.0.10 routes
BGP table version is 14, local router ID is 172.16.35.0
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
              x best-external, a additional-path, c RIB-compressed,
              t secondary path,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 65004:10 (default for vrf CustA_CE1)
 *>   192.168.101.0    100.65.0.10              0             0 65004 i

PE2

PE2#sh bgp vrf CustA_CE2 neighbor 100.65.0.2 routes
BGP table version is 23, local router ID is 172.16.16.0
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
              x best-external, a additional-path, c RIB-compressed,
              t secondary path,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 65004:10 (default for vrf CustA_CE2)
 *>   192.168.99.0     100.65.0.2               0             0 65004 i

Additional details that might help:

Traceroute from HostA to HostB gateway.

trace to 192.168.99.1, 8 hops max, press Ctrl+C to stop
 1   192.168.101.1   1.212 ms  0.648 ms  0.554 ms
 2   100.65.0.9   3.228 ms  2.175 ms  3.916 ms
 3     *  *  *
 4     *  *  *
 5     *  *  *
 6     *  *  *
 7     *  *  *
 8     *  *  *

Traceroute from HostB to HostA gateway

trace to 192.168.101.1, 8 hops max, press Ctrl+C to stop
 1   192.168.99.1   1.978 ms  0.794 ms  0.715 ms
 2   100.65.0.1   3.437 ms  1.713 ms  2.354 ms
 3     *  *  *
 4     *  *  *
 5     *  *  *
 6     *  *  *
 7     *  *  *
 8     *  *  *

-tonitones

Harold Ritter · ‎05-15-2023

Hi @tonitones10 ,

You only advertise one network for each CE1 (192.168.101.0/24) and CE2 (192.168.99.0/24). You need to use an address from that network as the source when you ping or traceroute from CE1 to CE2 and vice versa.

Otherwise, it will use the CE egress interface (CE-PE interface) as the source and this address is not know on the opposite CE, unless you configure "redistribute connected" under the VRF section of the BGP configuration.

Regards,

Regards,
Harold Ritter, CCIE #4168 (EI, SP)

tonitones10 · ‎05-15-2023

Hi @Harold Ritter ,

Im not sure if I followed you correctly, but I tried what you suggested to add redistribute connected under the VRF section of the BGP configuration. Tried adding it to both PE1 and PE2, but did not work. The same thing with adding to only PE1, and to only PE2.

-tonitones

Harold Ritter · ‎05-15-2023

Hi @tonitones10 ,

My suggestion was for if you ping from the CE, you should specify the source address. But I see that you are actually ping and traceroute from a PC behind the CE, so you do not need this.

Regards,

Regards,
Harold Ritter, CCIE #4168 (EI, SP)

M02@rt37 · ‎05-15-2023

Ok @tonitones10,

Thanks a lot for the output.

Please add another output:

from PE-1:

#sh bgp vpnv4 unicast vrf CustA_CE1

from PE-2:

#sh bgp vpnv4 unicast vrf CustA_CE2

Also as @Harold Ritter said, do you use source for your ping/traceroute ?

---- To be more effective/reliable during debug/troubleshoot is a good practice to have the same VRF name on each PE and same RT id in export/import ----

For example for your use case, because LAN 192.168.99.0/24 and 192.168.101.0/24 are LAN for the same Customer:

vrf definition Customer_A
rd 65004:10
route-target export 65004:10
route-target import 65004:10
!
address-family ipv4
exit-address-family

Best regards
.ı|ı.ı|ı. If This Helps, Please Rate .ı|ı.ı|ı.

tonitones10 · ‎05-15-2023

Hello M02@rt37 ,

As requested,

PE1

PE1#sh bgp vpnv4 unicast vrf CustA_CE1
BGP table version is 42, local router ID is 172.16.35.0
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
              x best-external, a additional-path, c RIB-compressed,
              t secondary path,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 65004:30 (default for vrf CustA_PGD)
 *>i  100.65.0.0/30    172.16.16.0              0    100      0 ?
 *>i  192.168.99.0     172.16.16.0              0    100      0 65004 i
 *>   192.168.101.0    100.65.0.10              0             0 65004 i

PE2

PE2#sh bgp vpnv4 unicast vrf CustA_CE2
BGP table version is 41, local router ID is 172.16.16.0
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
              x best-external, a additional-path, c RIB-compressed,
              t secondary path,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 65004:10 (default for vrf CustA_DVO)
 *>   100.65.0.0/30    0.0.0.0                  0         32768 ?
 *>   192.168.99.0     100.65.0.2               0             0 65004 i
 *>i  192.168.101.0    100.75.0.2               0    100      0 65004 i

As for this part

"For example for your use case, because LAN 192.168.99.0/24 and 192.168.101.0/24 are LAN for the same Customer:"

I did what was suggested: already have the same vrf name on each PE and the same RT id export/import. Outputs displayed below with sh bgp vpnv4 unicast vrf Cust_A.

PE1

PE1#sh bgp vpnv4 unicast vrf Cust_A
     Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 65004:10 (default for vrf Cust_A)
 *>i  100.65.0.0/30    172.16.16.0              0    100      0 ?
 *>i  192.168.99.0     172.16.16.0              0    100      0 65004 i
 *>   192.168.101.0    100.65.0.10              0             0 65004 i

PE2

PE2#sh bgp vpnv4 unicast vrf Cust_A
     Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 65004:10 (default for vrf Cust_A)
 *>   192.168.99.0     100.65.0.2               0             0 65004 i
 *>i  192.168.101.0    100.75.0.2               0    100      0 65004 i

-tonitones

MHM Cisco World · ‎05-15-2023

interface Loopback0
 description PE2-Lo
 ip address 172.16.16.0 255.255.255.255

interface Loopback0
 description PE1-Lo0
 ip address 172.16.35.0 255.255.255.255

how the LO IP end with 0 <<- that not correct

Note:- correct the VRF name

tonitones10 · ‎05-15-2023

Hi,

Most of the Lo we have in production end with 0 and we never had issues with it.

The VRF name was a copy-paste error on my end. I checked my config and it is correct. Apologies for the error.

Thank you.

-tonitones

MHM Cisco World · ‎05-15-2023

Yes, it can use but with some careful,
can I see the
show mpls forwarding in both PE1 and PE2 ?

tonitones10 · ‎05-15-2023

Yes, please refer to the output below.

PE1

PE1# show mpls forwarding-table
Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop
Label      Label      or Tunnel Id     Switched      interface
500        No Label   192.168.101.0/24[V]   \
                                       0             Po1.303    100.65.0.10

PE2

PE2#show mpls forwarding-table
Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop
Label      Label      or Tunnel Id     Switched      interface
100        100        172.16.0.249/32  0             Po1.110    100.75.0.9
101        Pop Label  172.16.0.0/32    0             Po1.110    100.75.0.9
103        102        172.16.20.0/32   0             Po1.110    100.75.0.9
104        No Label   192.168.99.0/24[V]   \
                                       0             Po1.301    100.65.0.2

-tonitones

MHM Cisco World · ‎05-15-2023

you can see in PE1 the MPLS label is missing,

mpls ldp router-id Loopback0 force

this I think is making issue,
MPLS with ID that is not /32 is making routing not learn Label.

Harold Ritter · ‎05-15-2023

Hi @MHM Cisco World ,

There is nothing wrong with the "No Label" as the outgoing label for the L3VPN prefixes. This is not a missing label. It is just the normal behavior.

Regards,

Regards,
Harold Ritter, CCIE #4168 (EI, SP)

MHM Cisco World · ‎05-15-2023

the PE1 have only label of V (tunnel) i.e. it VPNv4 is work but the other labels is missing meaning that the LDP between PE1 and it neighbor is missing something prevent PE1 to exchange the label.
I am not talking about no-label