cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
2459
Views
0
Helpful
5
Replies

VRF - Unreachable NH Issue

chanjohn01
Level 1
Level 1

Hello,

 

I'm practicing VRFs in GNS3, I have four routers: PE-1, PE-2, CustA-HQ and CustA-Remote. I'm unable to ping CustA-HQ from PE-2, but can ping the interface to which it connects on PE-1. When running debug ip packet detail on both CustA-HQ and PE-2, I noticed the following error:

 

****CustA-HQ*****

Mar 10 22:08:26.791: FIBfwd-proc: CustA_HQ:0.0.0.0/0 not enough info to forward via fib (none none)

*Mar 10 22:08:26.791: FIBipv4-packet-proc: packet routing failed

*Mar 10 22:08:26.791: IP: s=10.1.1.1 (local), d=10.2.2.20, len 100, unroutable

*Mar 10 22:08:26.791:     ICMP type=0, code=0

 

****PE-2****

PE-2#ping vrf CustA_Remote ip 10.1.1.1 repeat 10

*Mar 10 23:07:44.868: FIBipv4-packet-proc: route packet from (local) src 10.2.2.20 dst 10.1.1.1

*Mar 10 23:07:44.868: FIBfwd-proc: CustA_Remote:10.1.1.0/24 process level forwarding

*Mar 10 23:07:44.869: FIBfwd-proc: depth 0 first_idx 0 paths 1 long 0(0)

*Mar 10 23:07:44.869: FIBfwd-proc: try path 0 (of 1) v4-rcrsv-1.1.1.10 first short ext DCBB9B4(0)

*Mar 10 23:07:44.870: FIBfwd-proc: v4-rcrsv-1.1.1.10 valid short mbl

*Mar 10 23:07:44.870: FIBfwd-proc: label[0] 20 connid 0 link ILLEGAL

*Mar 10 23:07:44.870: FIBfwd-proc: ip_pak_table 2 ip_nh_table 0 if none nh 1.1.1.10 deag 0 chg_if 0 via fib FC09370 path type recursive

*Mar 10 23:07:44.870: FIBfwd-proc: depth 1 first_idx 0 paths 1 long 0(0)

*Mar 10 23:07:44.871: FIBfwd-proc: try path 0 (of 1) v4-anh-1.1.1.1-Gi0/0 first short ext DCBBA14(0)

*Mar 10 23:07:44.871: FIBfwd-proc: v4-anh-1.1.1.1-Gi0/0 valid short

*Mar 10 23:07:44.871: FIBfwd-proc: label[1] 3 connid 0 link TAG

*Mar 10 23:07:44.872: FIBfwd-proc: ip_pak_table 2 ip_nh_table 65535 if GigabitEthernet0/0 nh 1.1.1.1 deag 0 chg_if 0 via fib 0 path type attached nexthop

*Mar 10 23:07:44.872: FIBfwd-proc: packet routed to GigabitEthernet0/0 1.1.1.1(2) with label 20

*Mar 10 23:07:44.873: FIBipv4-packet-proc: packet routing succeeded

Success rate is 0 percent (0/10)

PE-2#Screen Shot 2018-03-10 at 4.39.14 PM.png 

********************************************************

* PE-1 Config * 

Building configuration... 

Current configuration : 4421 bytes

!

! Last configuration change at 23:28:10 UTC Sat Mar 10 2018

!

version 15.5

service timestamps debug datetime msec

service timestamps log datetime msec

service password-encryption

!

hostname PE-1

!

boot-start-marker

boot-end-marker

!

vrf definition CustA_HQ

rd 100:10

route-target export 100:10

route-target import 100:10

route-target import 100:20

!

address-family ipv4

exit-address-family

!

vrf definition CustA_Remote

rd 100:20

route-target export 100:20

route-target import 100:20

!

address-family ipv4

exit-address-family

!

no aaa new-model

ethernet lmi ce

! 

mmi polling-interval 60

no mmi auto-configure

no mmi pvc

mmi snmp-timeout 180

!

no ip icmp rate-limit unreachable

!

no ip domain lookup

ip cef

no ipv6 cef

!

multilink bundle-name authenticated

mpls label protocol ldp

mpls ldp password required

mpls ldp neighbor 2.2.2.20 password 7 09415E050A

mpls ldp session protection

no mpls ip propagate-ttl forwarded

!      

redundancy

!

no cdp log mismatch duplex

no cdp run

!

ip tcp synwait-time 5

!

interface Loopback0

description /// MPLS MP-BGP LOOPBACK ///

ip address 1.1.1.10 255.255.255.255

!         

interface Loopback10

description /// VRF CustA_HQ IGP LOOPBACK ///

ip address 10.10.10.10 255.255.255.255

!

interface GigabitEthernet0/0

ip address 1.1.1.1 255.255.255.252

ip ospf 100 area 0

duplex auto

speed auto

media-type rj45

mpls ip

!

interface GigabitEthernet0/1

vrf forwarding CustA_HQ

ip address 10.1.1.10 255.255.255.0

duplex full

speed 1000

!

 

router ospf 10 vrf CustA_HQ

router-id 10.10.10.10

log-adjacency-changes detail

network 10.1.1.0 0.0.0.255 area 0

network 10.10.10.10 0.0.0.0 area 0

!

router ospf 100

router-id 1.1.1.10

log-adjacency-changes detail

network 1.1.1.10 0.0.0.0 area 0

!

router bgp 100

bgp log-neighbor-changes

neighbor 2.2.2.20 remote-as 100

neighbor 2.2.2.20 update-source Loopback0

!

address-family vpnv4

  neighbor 2.2.2.20 activate

  neighbor 2.2.2.20 send-community extended

exit-address-family

!        

address-family ipv4 vrf CustA_HQ

  network 10.1.1.0 mask 255.255.255.0

  redistribute ospf 10 metric 3

exit-address-family

!

ip forward-protocol nd

!

no ip http server

no ip http secure-server

!

mpls ldp router-id Loopback0 force

!

control-plane

!

********************************************************

* CustA-HQ Config *

Building configuration...

Current configuration : 1893 bytes

!

! Last configuration change at 23:23:25 UTC Sat Mar 10 2018

!

version 15.2

service timestamps debug datetime msec

service timestamps log datetime msec

!

hostname CustA-HQ

!

boot-start-marker

boot-end-marker

!

!

vrf definition CustA_HQ

rd 100:10

route-target export 100:10

route-target import 100:10

route-target import 100:20

!

address-family ipv4

exit-address-family

!         

vrf definition CustA_Remote

rd 100:20

route-target export 100:20

route-target import 100:20

!

address-family ipv4

exit-address-family

!

no aaa new-model

no ip icmp rate-limit unreachable

!

no ip domain lookup

ip cef

no ipv6 cef

!

multilink bundle-name authenticated

!

ip tcp synwait-time 5

!

interface Loopback10

description /// VRF CustA_HQ IGP LOOPBACK ///

ip address 10.10.10.1 255.255.255.255

! 

interface GigabitEthernet0/0

vrf forwarding CustA_HQ

ip address 10.1.1.1 255.255.255.0

media-type gbic

speed 1000

duplex full

negotiation auto

!

interface Ethernet1/0

vrf forwarding CustA_HQ

ip address 1.0.0.1 255.0.0.0

duplex full

!

router ospf 10 vrf CustA_HQ

router-id 10.10.10.1

log-adjacency-changes detail

capability vrf-lite

redistribute bgp 100 metric 5 subnets

network 1.0.0.0 0.255.255.255 area 0

network 10.1.1.0 0.0.0.255 area 0

network 10.10.10.1 0.0.0.0 area 0

!         

ip forward-protocol nd

!

no ip http server

no ip http secure-server

!

control-plane

!

end

********************************************************

* PE-2 Config *

Building configuration...

Current configuration : 4329 bytes

!

! Last configuration change at 22:31:58 UTC Sat Mar 10 2018

!

version 15.5

service timestamps debug datetime msec

service timestamps log datetime msec

no service password-encryption

!

hostname PE-2

!

boot-start-marker

boot-end-marker

!

!

vrf definition CustA_HQ

rd 100:10

route-target export 100:10

route-target import 100:10

!

address-family ipv4

exit-address-family

!         

vrf definition CustA_Remote

rd 100:20

route-target export 100:20

route-target import 100:20

route-target import 100:10

!

address-family ipv4

exit-address-family

!

no aaa new-model

ethernet lmi ce

!

mmi polling-interval 60

no mmi auto-configure

no mmi pvc

mmi snmp-timeout 180

!

no ip icmp rate-limit unreachable

!

no ip domain lookup

ip cef

no ipv6 cef

!

multilink bundle-name authenticated

mpls label protocol ldp

mpls ldp password required

mpls ldp neighbor 1.1.1.10 password mpls

mpls ldp session protection

no mpls ip propagate-ttl forwarded

! 

redundancy

!

no cdp log mismatch duplex

!

ip tcp synwait-time 5

!

interface Loopback0

description /// MPLS MP-BGP LOOPBACK ///

ip address 2.2.2.20 255.255.255.255

!

interface Loopback20

description /// VRF CustA_Remote IGP LOOPBACK ///

ip address 20.20.20.20 255.255.255.255

!

interface GigabitEthernet0/0

ip address 1.1.1.2 255.255.255.252

ip ospf 100 area 0

duplex auto

speed auto

media-type rj45

mpls ip

!

interface GigabitEthernet0/1

vrf forwarding CustA_Remote

ip address 10.2.2.20 255.255.255.0

duplex full

speed 1000

!

 

router ospf 100

router-id 2.2.2.20

log-adjacency-changes detail

passive-interface default

no passive-interface GigabitEthernet0/0

network 2.2.2.20 0.0.0.0 area 0

!

router bgp 100

bgp log-neighbor-changes

neighbor 1.1.1.10 remote-as 100

neighbor 1.1.1.10 update-source Loopback0

!

address-family vpnv4

  neighbor 1.1.1.10 activate

  neighbor 1.1.1.10 send-community extended

exit-address-family

!

address-family ipv4 vrf CustA_Remote

  network 10.2.2.0 mask 255.255.255.0

  redistribute ospf 20 metric 3

exit-address-family

!

ip forward-protocol nd

!

no ip http server

no ip http secure-server

!

mpls ldp router-id Loopback0 force

!

control-plane

**REMAINING OUTPUT OMITTED**

********************************************************

* CustA-Remote Config *

Building configuration...

Current configuration : 1839 bytes

!

! Last configuration change at 23:45:30 UTC Sat Mar 10 2018

!

version 15.2

service timestamps debug datetime msec

service timestamps log datetime msec

!

hostname CustA_Remote

!

boot-start-marker

boot-end-marker

!

vrf definition CustA_HQ

rd 100:10

route-target export 100:10

route-target import 100:10

!

address-family ipv4

exit-address-family

!

vrf definition CustA_Remote

rd 100:20

route-target export 100:20

route-target import 100:20

route-target import 100:10

!

address-family ipv4

exit-address-family

!

no aaa new-model

no ip icmp rate-limit unreachable

!

no ip domain lookup

ip cef

no ipv6 cef

!

multilink bundle-name authenticated

!

ip tcp synwait-time 5

!

interface Loopback20

description /// VRF CustA_Remote IGP LOOPBACK ///

ip address 20.20.20.1 255.255.255.255

!

interface GigabitEthernet0/0

vrf forwarding CustA_Remote

ip address 10.2.2.1 255.255.255.0

media-type gbic

speed 1000

duplex full

negotiation auto

!

interface Ethernet1/0

ip address 2.0.0.1 255.0.0.0

duplex full

!

router ospf 20 vrf CustA_Remote

router-id 20.20.20.1

log-adjacency-changes detail

redistribute bgp 100 metric 5 subnets

network 2.0.0.0 0.255.255.255 area 0

network 10.2.2.0 0.0.0.255 area 0

network 20.20.20.1 0.0.0.0 area 0

!

ip forward-protocol nd

!

!         

no ip http server

no ip http secure-server

!

control-plane

!

!   

end

********************************************************

PE-1#sh ip bgp vpnv4 all

BGP table version is 35, local router ID is 10.10.10.10

Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,

Origin codes: i - IGP, e - EGP, ? - incomplete

RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path

Route Distinguisher: 100:10 (default for vrf CustA_HQ)

*>  1.0.0.0             10.1.1.1                 3                 32768 ?

*>  10.1.1.0/24       0.0.0.0                  0                 32768 i

*>i 10.2.2.0/24      2.2.2.20                 0    100               0 i

Route Distinguisher: 100:20 (default for vrf CustA_Remote)

*>i 10.2.2.0/24      2.2.2.20                 0    100              0 i

 

PE-1#sh ip route vrf CustA_HQ

Routing Table: CustA_HQ

Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP

       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area

Gateway of last resort is not set

O     1.0.0.0/8 [110/11] via 10.1.1.1, 01:34:01, GigabitEthernet0/1

      10.0.0.0/8 is variably subnetted, 3 subnets, 2 masks

C        10.1.1.0/24 is directly connected, GigabitEthernet0/1

L        10.1.1.10/32 is directly connected, GigabitEthernet0/1

B        10.2.2.0/24 [200/0] via 2.2.2.20, 00:00:37

PE-1#

********************************************************

CustA-HQ#sh ip route vrf CustA_HQ      

Routing Table: CustA_HQ

Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP 

Gateway of last resort is not set

      1.0.0.0/8 is variably subnetted, 2 subnets, 2 masks

C        1.0.0.0/8 is directly connected, Ethernet1/0

L        1.0.0.1/32 is directly connected, Ethernet1/0

      10.0.0.0/8 is variably subnetted, 2 subnets, 2 masks

C        10.1.1.0/24 is directly connected, GigabitEthernet0/0

L        10.1.1.1/32 is directly connected, GigabitEthernet0/0

CustA-HQ#

CustA-HQ#sh ip route

Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP

Gateway of last resort is not set 

      10.0.0.0/32 is subnetted, 1 subnets

C        10.10.10.1 is directly connected, Loopback10

CustA-HQ#

********************************************************

PE-2#sh ip bgp vpnv4 all

BGP table version is 52, local router ID is 20.20.20.20

Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,

              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,

              x best-external, a additional-path, c RIB-compressed,

Origin codes: i - IGP, e - EGP, ? - incomplete

RPKI validation codes: V valid, I invalid, N Not found

 

     Network          Next Hop            Metric LocPrf Weight Path

Route Distinguisher: 100:10 (default for vrf CustA_HQ)

*>i 1.0.0.0             1.1.1.10                 3    100      0 ?

*>i 10.1.1.0/24      1.1.1.10                 0    100      0 i

Route Distinguisher: 100:20 (default for vrf CustA_Remote)

*>i 1.0.0.0             1.1.1.10                 3    100      0 ?

*>i 10.1.1.0/24      1.1.1.10                 0    100      0 i

*>  10.2.2.0/24      0.0.0.0                  0         32768 i

PE-2#sh ip route vrf CustA_Remote

 

Routing Table: CustA_Remote

Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP

       D - EIGRP, EX - EIGRP external, O - OSPF

Gateway of last resort is not set

 

B     1.0.0.0/8 [200/3] via 1.1.1.10, 00:05:40

      10.0.0.0/8 is variably subnetted, 3 subnets, 2 masks

B        10.1.1.0/24 [200/0] via 1.1.1.10, 00:05:40

C        10.2.2.0/24 is directly connected, GigabitEthernet0/1

L        10.2.2.20/32 is directly connected, GigabitEthernet0/1

PE-2#

PE-2#ping vrf CustA_Remote ip 10.1.1.10 (Gi0/0 on PE-1 - connects to CustA-HQ)

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 10.1.1.10, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 3/4/8 ms

********************************************************

CustA_Remote# sh ip route vrf CustA_Remote

Routing Table: CustA_Remote

Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP

 

Gateway of last resort is not set

 

      10.0.0.0/8 is variably subnetted, 2 subnets, 2 masks

C        10.2.2.0/24 is directly connected, GigabitEthernet0/0

L        10.2.2.1/32 is directly connected, GigabitEthernet0/0

CustA_Remote#

 

CustA_Remote#sh ip route

Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP

 

Gateway of last resort is not set

 

      2.0.0.0/8 is variably subnetted, 2 subnets, 2 masks

C        2.0.0.0/8 is directly connected, Ethernet1/0

L        2.0.0.1/32 is directly connected, Ethernet1/0

      20.0.0.0/32 is subnetted, 1 subnets

C        20.20.20.1 is directly connected, Loopback20

CustA_Remote#

 

CustA_Remote#sh ip ospf neighbor

 

Neighbor ID     Pri   State           Dead Time   Address         Interface

20.20.20.20       1   FULL/BDR        00:00:33    10.2.2.20       GigabitEthernet0/0

CustA_Remote#

 

Any assistance greatly appreciated

 

 

1 Accepted Solution

Accepted Solutions

Harold Ritter
Level 12
Level 12

> IP: s=10.1.1.1 (local), d=10.2.2.20, len 100, unroutable

 

From the output above, it looks like CustA-HQ does not have route back to 10.2.2.0/24. You advertise it via BGP, but never redistrbute it from BGP to OSPF, as per your OSPF config:

 

>router ospf 10 vrf CustA_HQ

>router-id 10.10.10.10

>log-adjacency-changes detail

>network 10.1.1.0 0.0.0.255 area 0

>network 10.10.10.10 0.0.0.0 area 0

 

You have the same problem in the othet direction. 10.1.1.0/24 is advertized via BGP, but not redistribute from BGP to OSPF.

 

Regards,

Harold Ritter
Sr Technical Leader
CCIE 4168 (R&S, SP)
harold@cisco.com
México móvil: +52 1 55 8312 4915
Cisco México
Paseo de la Reforma 222
Piso 19
Cuauhtémoc, Juárez
Ciudad de México, 06600
México

View solution in original post

5 Replies 5

Harold Ritter
Level 12
Level 12

> IP: s=10.1.1.1 (local), d=10.2.2.20, len 100, unroutable

 

From the output above, it looks like CustA-HQ does not have route back to 10.2.2.0/24. You advertise it via BGP, but never redistrbute it from BGP to OSPF, as per your OSPF config:

 

>router ospf 10 vrf CustA_HQ

>router-id 10.10.10.10

>log-adjacency-changes detail

>network 10.1.1.0 0.0.0.255 area 0

>network 10.10.10.10 0.0.0.0 area 0

 

You have the same problem in the othet direction. 10.1.1.0/24 is advertized via BGP, but not redistribute from BGP to OSPF.

 

Regards,

Harold Ritter
Sr Technical Leader
CCIE 4168 (R&S, SP)
harold@cisco.com
México móvil: +52 1 55 8312 4915
Cisco México
Paseo de la Reforma 222
Piso 19
Cuauhtémoc, Juárez
Ciudad de México, 06600
México

Thanks so much for replying!

I must be redistributing incorrectly. I tried the following with no success on either end:

 

HQ

router ospf vrf CustA_HQ

redistribute bgp 100 metric 5

 

Remote

router ospf vrf CustA_Remote

redistribute bgp 100 metric 5

 

Am I redistributing in the wrong place? 

I've updated my configs, unfortunately still no luck. I completely started over, focusing on one half of the config at a time. It doesn't really make sense for me to work on PE-2 side, if one-way distribution isn't working yet :/

 

 

Capture.PNG

 

!
router ospf 10 vrf CustA_HQ
 router-id 1.1.1.10
 log-adjacency-changes detail
 redistribute bgp 100 metric 100 subnets route-map BGP_TO_OSPF
 network 1.1.1.10 0.0.0.0 area 0
 network 10.1.1.0 0.0.0.255 area 0
!

!
router ospf 100
 router-id 1.1.1.1
 log-adjacency-changes detail
 network 1.1.1.1 0.0.0.0 area 0
!

router bgp 100
 bgp log-neighbor-changes
 neighbor 2.2.2.2 remote-as 100
 neighbor 2.2.2.2 update-source Loopback0
 !
 address-family ipv4
  bgp redistribute-internal
  neighbor 2.2.2.2 activate
  neighbor 2.2.2.2 send-community extended
 exit-address-family
 !
 address-family vpnv4
  neighbor 2.2.2.2 activate
  neighbor 2.2.2.2 send-community extended
 exit-address-family
 !
 address-family ipv4 vrf CustA_HQ
  network 10.1.1.0 mask 255.255.255.0
  redistribute ospf 10 metric 5
 exit-address-family
!
ip prefix-list BGP_INTERNAL seq 5 permit 1.1.1.110/31
!
route-map BGP_TO_OSPF permit 10
 match ip address prefix-list BGP_INTERNAL

%%%%%%%%%%PE-1 PINGS%%%%%%%%%%%%%
PE-1#ping vrf CustA_HQ 10.1.1.10
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.1.1.10, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/8 ms
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
PE-1#ping vrf CustA_HQ 10.1.1.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.1.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 16/28/48 ms
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
PE-1#ping vrf CustA_HQ 1.0.0.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 1.0.0.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 16/28/40 ms
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
PE-1#ping vrf CustA_HQ 1.0.0.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 1.0.0.2, timeout is 2 seconds:
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 24/299/1084 ms

%%%%PINGS FROM PC CONNECTED TO CustA-HQ%%%%%%
PC1> ping 1.0.0.1
84 bytes from 1.0.0.1 icmp_seq=1 ttl=255 time=9.051 ms
84 bytes from 1.0.0.1 icmp_seq=2 ttl=255 time=9.007 ms

PC1> ping 10.1.1.1
84 bytes from 10.1.1.1 icmp_seq=1 ttl=255 time=15.626 ms
84 bytes from 10.1.1.1 icmp_seq=2 ttl=255 time=15.626 ms

PC1> ping 10.1.1.10
84 bytes from 10.1.1.10 icmp_seq=1 ttl=254 time=46.878 ms
84 bytes from 10.1.1.10 icmp_seq=2 ttl=254 time=31.246 ms

PC1> ping 1.1.1.110
*1.0.0.1 icmp_seq=1 ttl=255 time=15.626 ms (ICMP type:3, code:1, Destination host unreachable)

 

>PC1> ping 1.1.1.110
>*1.0.0.1 icmp_seq=1 ttl=255 time=15.626 ms (ICMP type:3, code:1, Destination host unreachable)

 

Why are you trying to ping the core link (PE1 to PE2) from PC1? This will not work and it is the expected resut. You should try pinging something in the VRF on the other side.

 

Regards,

Harold Ritter
Sr Technical Leader
CCIE 4168 (R&S, SP)
harold@cisco.com
México móvil: +52 1 55 8312 4915
Cisco México
Paseo de la Reforma 222
Piso 19
Cuauhtémoc, Juárez
Ciudad de México, 06600
México

were a couple of issues I realized. I was using the vrf forwarding statements incorrectly, in that I had the statement on both my "LAN" subnets and my point-to-point. that's wrong. So, I configured OSPF on the Provider routers and configured MP-BGP and OSPF between the PE's and did my redistribution from the PE's. Now I can ping between all my LAN subnets. The Provider should not see my LAN routes only its connection to the PEs. Thanks for your help! Corrected config below for anyone that may run into the same issue down the road:

 

******P1*******
version 15.2
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname P1
!
boot-start-marker
boot-end-marker
!
no aaa new-model
!
ip cef
no ipv6 cef
!
!
mpls label protocol ldp
mpls ldp router-id Loopback0
mpls ldp password required
mpls ldp neighbor 123.0.0.2 password 7 0009030A17
mpls ldp neighbor 223.0.0.1 password 7 020B145718
mpls ldp session protection
multilink bundle-name authenticated
!
interface Loopback0
 ip address 123.0.0.1 255.255.255.255
!
interface GigabitEthernet0/0
 ip address 10.1.1.1 255.255.255.252
 media-type gbic
 speed 1000
 duplex full
 negotiation auto
 mpls ip
!
interface GigabitEthernet1/0
 ip address 111.0.0.1 255.255.255.252
 negotiation auto
 mpls ip
!
router ospf 100
 router-id 123.0.0.1
 log-adjacency-changes detail
 passive-interface default
 no passive-interface GigabitEthernet0/0
 no passive-interface GigabitEthernet1/0
 network 10.1.1.0 0.0.0.3 area 0
 network 111.0.0.0 0.0.0.3 area 0
 network 123.0.0.1 0.0.0.0 area 0
 default-information originate always
!
ip forward-protocol nd
!
no ip http server
no ip http secure-server
!
control-plane
!
line con 0
 exec-timeout 120 0
 stopbits 1
!
end
************************************************************************
*****PE-1*****
version 15.2
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname PE-1
!
boot-start-marker
boot-end-marker
!
vrf definition CustA_HQ
 rd 100:10
 route-target export 100:10
 route-target import 100:10
 route-target import 100:20
 !
 address-family ipv4
 exit-address-family
!
vrf definition CustA_Remote
 rd 100:20
 route-target export 100:20
 route-target import 100:20
 !
 address-family ipv4
 exit-address-family
!
no aaa new-model
!
no ip domain lookup
ip cef
no ipv6 cef
!
mpls label protocol ldp
mpls ldp router-id Loopback0
no mpls ip propagate-ttl forwarded
mpls ldp password required
mpls ldp neighbor 123.0.0.1 password 7 151F1B0017
mpls ldp session protection
multilink bundle-name authenticated
!
ip tcp path-mtu-discovery age-timer 10
!
interface Loopback0
 ip address 123.0.0.2 255.255.255.255
!
interface GigabitEthernet0/0
 ip address 10.1.1.2 255.255.255.252
 media-type gbic
 speed 1000
 duplex full
 negotiation auto
 mpls ip
!
interface FastEthernet1/0
 vrf forwarding CustA_HQ
 ip address 1.0.0.1 255.255.255.248
 duplex full
!
router ospf 10 vrf CustA_HQ
 log-adjacency-changes detail
 redistribute bgp 100 metric 5 subnets
 passive-interface default
 no passive-interface FastEthernet1/0
 network 1.0.0.1 0.0.0.0 area 0
!
router ospf 100
 router-id 123.0.0.2
 log-adjacency-changes detail
 passive-interface default
 no passive-interface GigabitEthernet0/0
 network 10.1.1.2 0.0.0.0 area 0
 network 123.0.0.2 0.0.0.0 area 0
!
router bgp 100
 template peer-policy IBGP-P
  next-hop-self
  maximum-prefix 1000
  send-community extended
 exit-peer-policy
 !
 template peer-session IBGP-S
  remote-as 100
  description IBGP/PE Sessions
  password 7 1043191516
  update-source Loopback0
  version 4
  timers 5 20
 exit-peer-session
 !
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor 223.0.0.2 inherit peer-session IBGP-S
 neighbor 223.0.0.2 description PE-2
 !
 address-family ipv4
 exit-address-family
 !
 address-family vpnv4
  neighbor 223.0.0.2 activate
  neighbor 223.0.0.2 send-community extended
  neighbor 223.0.0.2 inherit peer-policy IBGP-P
 exit-address-family
 !
 address-family ipv4 vrf CustA_HQ
  network 1.0.0.0 mask 255.255.255.248
  redistribute ospf 10 metric 20 match internal external 1 external 2
 exit-address-family
!
ip forward-protocol nd
!
no ip http server
no ip http secure-server
!
control-plane
!
line con 0
 exec-timeout 120 0
 stopbits 1
line aux 0
 stopbits 1
line vty 0 4
 login
!
end
************************************************************************
****P2*****
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname P2
!
boot-start-marker
boot-end-marker
no aaa new-model
no ip domain lookup
ip cef
no ipv6 cef
!
mpls label protocol ldp
mpls ldp router-id Loopback0
mpls ldp password required
mpls ldp neighbor 223.0.0.2 password 7 141A02071F
mpls ldp neighbor 123.0.0.1 password 7 151F1B0017
mpls ldp session protection
multilink bundle-name authenticated
!
interface Loopback0
 ip address 223.0.0.1 255.255.255.255
!
interface GigabitEthernet0/0
 ip address 10.2.2.1 255.255.255.252
 media-type gbic
 speed 1000
 duplex full
 negotiation auto
 mpls ip
!
interface GigabitEthernet1/0
 ip address 111.0.0.2 255.255.255.252
 negotiation auto
 mpls ip
!
router ospf 100
 router-id 223.0.0.1
 log-adjacency-changes detail
 passive-interface default
 no passive-interface GigabitEthernet0/0
 no passive-interface GigabitEthernet1/0
 network 10.2.2.0 0.0.0.3 area 0
 network 111.0.0.0 0.0.0.3 area 0
 network 223.0.0.1 0.0.0.0 area 0
 default-information originate always
!
ip forward-protocol nd
!
no ip http server
no ip http secure-server
!
control-plane
!
line con 0
 exec-timeout 120 0
 stopbits 1
!
!
end
**********PE-2**************
PE-2#sh run
Building configuration...

Current configuration : 2752 bytes
!
! Last configuration change at 04:59:03 UTC Fri Mar 16 2018
!
version 15.2
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname PE-2
!
boot-start-marker
boot-end-marker
!
!
vrf definition CustA_HQ
 rd 100:10
 route-target export 100:10
 route-target import 100:10
 !
 address-family ipv4
 exit-address-family
!
vrf definition CustA_Remote
 rd 100:20
 route-target export 100:20
 route-target import 100:20
 route-target import 100:10
 !
 address-family ipv4
 exit-address-family
!
!
no aaa new-model
!
no ip domain lookup
ip cef
no ipv6 cef
!
!
mpls label protocol ldp
mpls ldp router-id Loopback0
no mpls ip propagate-ttl forwarded
mpls ldp password required
mpls ldp neighbor 223.0.0.1 password 7 1104090904
mpls ldp session protection
multilink bundle-name authenticated
!
!
ip tcp path-mtu-discovery age-timer 10
!
!
interface Loopback0
 ip address 223.0.0.2 255.255.255.255
!
interface GigabitEthernet0/0
 ip address 10.2.2.2 255.255.255.252
 media-type gbic
 speed 1000
 duplex full
 negotiation auto
 mpls ip
!
interface FastEthernet1/0
 vrf forwarding CustA_Remote
 ip address 2.0.0.1 255.255.255.248
 duplex full
!
router ospf 20 vrf CustA_Remote
 log-adjacency-changes detail
 redistribute bgp 100 metric 5 subnets
 passive-interface default
 no passive-interface FastEthernet1/0
 network 2.0.0.1 0.0.0.0 area 0
!
router ospf 100
 router-id 223.0.0.2
 log-adjacency-changes detail
 passive-interface default
 no passive-interface GigabitEthernet0/0
 network 10.2.2.2 0.0.0.0 area 0
 network 223.0.0.2 0.0.0.0 area 0
!
router bgp 100
 template peer-policy IBGP-P
  next-hop-self
  maximum-prefix 1000
  send-community extended
 exit-peer-policy
 !
 template peer-session IBGP-S
  remote-as 100
  description IBGP/PE Sessions
  password 7 060B1F2D5F
  update-source Loopback0
  version 4
  timers 5 20
 exit-peer-session
 !
 bgp router-id 223.0.0.2
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor 123.0.0.2 inherit peer-session IBGP-S
 neighbor 123.0.0.2 description PE-1
 !
 address-family ipv4
 exit-address-family
 !
 address-family vpnv4
  neighbor 123.0.0.2 activate
  neighbor 123.0.0.2 send-community extended
  neighbor 123.0.0.2 inherit peer-policy IBGP-P
 exit-address-family
 !
 address-family ipv4 vrf CustA_Remote
  network 2.0.0.0 mask 255.255.255.248
  redistribute ospf 20 metric 20 match internal external 1 external 2
 exit-address-family
!
ip forward-protocol nd
!
no ip http server
no ip http secure-server
!
!
control-plane
!
line con 0
 exec-timeout 120 0
 stopbits 1
!
end

 

Review Cisco Networking for a $25 gift card