MPLS BGP-LU CE VRF network routes not being advertised

[davehouser1]

Hi,

System:
(2)CE: Cisco IOS running eBGP advertising two loopback networks (To represent customer subnets) advertises loopback10 and loopback11, using 192.168 subnets.
(2)PE: Cisco vXR running eBGP to CE routers, connects to P / RR network and advertises MPLS labels via BGP-LU off RR.

CE01(AS65011)                       CE02 (AS65011)
   |                                     |
   |                                     |
  PE01 (AS65000)                  PE02 (AS 65000)
      \                                 /
       \  --  P network (AS 65001) -- /
                        |
              RR network (AS 65000)

Here is bgp config of two CE routers BGP configurations
CustA-CE-1:

CustA-CE-1#show run | s router
router bgp 65011
 bgp router-id interface Loopback0
 bgp log-neighbor-changes
 network 192.168.111.0 mask 255.255.255.128
 network 192.168.111.128 mask 255.255.255.128
 neighbor 10.11.1.1 remote-as 65000

CustA-CE-2:

CustA-CE-2#show run | s router
router bgp 65011
 bgp router-id interface Loopback0
 bgp log-neighbor-changes
 network 192.168.112.0 mask 255.255.255.128
 network 192.168.112.128 mask 255.255.255.128
 neighbor 10.11.2.1 remote-as 65000

There are two separate PE routers they connect to. Here are their configurations: 

RP/0/RP0/CPU0:PE01#show run vrf
Fri Aug 12 19:30:02.564 UTC
vrf CustA-CE-1
 vpn id 65011:11
 address-family ipv4 unicast
  import route-target
   65011:11
  !
  export route-target
   65011:11
  !
 !
!
vrf CustB-CE-2
 vpn id 65013:13
 address-family ipv4 unicast
  import route-target
   65013:13
  !
  export route-target
   65013:13
  !
 !
!
RP/0/RP0/CPU0:PE01#show run router bgp
router bgp 65000
 timers bgp 30 90
 bgp router-id 7.7.7.7
 mpls activate
  interface GigabitEthernet0/0/0/3
  interface GigabitEthernet0/0/0/5
 !
 bgp graceful-restart
 bgp bestpath compare-routerid
 ibgp policy out enforce-modifications
 address-family ipv4 unicast
  maximum-paths ebgp 6 selective
  maximum-paths ibgp 2 selective
  redistribute connected route-policy SET_REDIST_CONNECTED_V4
  allocate-label route-policy SET_BGP_LABEL_ALLOCATION_V4
 !
 address-family vpnv4 unicast
 !
 address-family ipv4 flowspec
 !
 neighbor-group CustA
  remote-as 65011
  address-family ipv4 unicast
   route-policy CE-PASS in
   route-policy CE-PASS out
  !
 !
 neighbor-group CustB
  remote-as 65013
  address-family ipv4 unicast
   route-policy CE-PASS in
   route-policy CE-PASS out
  !
 vrf CustA-CE-1
  rd 65011:11
  address-family ipv4 unicast
   redistribute connected
  !
  neighbor 10.11.1.2
   use neighbor-group CustA
  !
 !
 vrf CustB-CE-2
  rd 65013:13
  address-family ipv4 unicast
   redistribute connected
  !
  neighbor 10.13.2.2
   use neighbor-group CustB
  !
 !
!

And PE02

RP/0/RP0/CPU0:PE02#show run vrf
Fri Aug 12 19:34:23.804 UTC
vrf CustA-CE-2
 vpn id 65011:11
 address-family ipv4 unicast
  import route-target
   65011:11
  !
  export route-target
   65011:11
  !
 !
!
vrf CustB-CE-1
 vpn id 65013:13
 address-family ipv4 unicast
  import route-target
   65013:13
  !
  export route-target
   65013:13
  !
 !
!
RP/0/RP0/CPU0:PE02#show run router bgp
Fri Aug 12 19:34:56.802 UTC
router bgp 65000
 timers bgp 30 90
 bgp router-id 9.9.9.9
 mpls activate
  interface GigabitEthernet0/0/0/3
  interface GigabitEthernet0/0/0/5
 !
 bgp graceful-restart
 bgp bestpath compare-routerid
 ibgp policy out enforce-modifications
 address-family ipv4 unicast
  maximum-paths ebgp 6 selective
  maximum-paths ibgp 2 selective
  redistribute connected route-policy SET_REDIST_CONNECTED_V4
  allocate-label route-policy SET_BGP_LABEL_ALLOCATION_V4
 !
 address-family vpnv4 unicast
 !
 address-family ipv4 flowspec
 !
 neighbor-group CustA
  remote-as 65011
  address-family ipv4 unicast
   route-policy CE-PASS in
   route-policy CE-PASS out
  !
 !
 neighbor-group CustB
  remote-as 65013
  address-family ipv4 unicast
   route-policy CE-PASS in
   route-policy CE-PASS out
  !
 !
 vrf CustA-CE-2
  rd 65011:11
  address-family ipv4 unicast
   redistribute connected
  !
  neighbor 10.11.2.2
   use neighbor-group CustA
  !
 !
 vrf CustB-CE-1
  rd 65013:13
  address-family ipv4 unicast
   redistribute connected
  !
  neighbor 10.13.1.2
   use neighbor-group CustB
  !
 !
!

NOTE: There are much more configurations in these systems but I just wanted to share the PE <-> CE legs. There is also a VRF for CustB networks but I did not share that here as its the same problem. the PE router config is slimmed down, there are many routing-policies, and neighbors to the P and RR systems, I just did not include them. 

Here is the interesting part. If I check the vrf routes I can see the subnets populating that the CE networks are being advertised. This is what shows up on PE01:

RP/0/RP0/CPU0:PE01#show bgp vrf all
Fri Aug 12 19:38:10.372 UTC

VRF: CustA-CE-1
---------------
BGP VRF CustA-CE-1, state: Active
BGP Route Distinguisher: 65011:11
VRF ID: 0x60000001
BGP router identifier 7.7.7.7, local AS number 65000
Non-stop routing is enabled
BGP table state: Active
Table ID: 0xe0000001   RD version: 66
BGP main routing table version 69
BGP NSR Initial initsync version 7 (Reached)
BGP NSR/ISSU Sync-Group versions 0/0

Status codes: s suppressed, d damped, h history, * valid, > best
              i - internal, r RIB-failure, S stale, N Nexthop-discard
Origin codes: i - IGP, e - EGP, ? - incomplete
   Network            Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 65011:11 (default for vrf CustA-CE-1)
*> 10.11.1.0/30       0.0.0.0                  0         32768 ?
*>i10.11.2.0/30       9.9.9.109                0    100      0 ?
*> 192.168.111.0/25   10.11.1.2                0             0 65011 i
*> 192.168.111.128/25 10.11.1.2                0             0 65011 i
*>i192.168.112.0/25   9.9.9.109                0    100      0 65011 i
*>i192.168.112.128/25 9.9.9.109                0    100      0 65011 i

Processed 6 prefixes, 6 paths

VRF: CustB-CE-2
---------------
BGP VRF CustB-CE-2, state: Active
BGP Route Distinguisher: 65013:13
VRF ID: 0x60000002
BGP router identifier 7.7.7.7, local AS number 65000
Non-stop routing is enabled
BGP table state: Active
Table ID: 0xe0000002   RD version: 69
BGP main routing table version 69
BGP NSR Initial initsync version 7 (Reached)
BGP NSR/ISSU Sync-Group versions 0/0

Status codes: s suppressed, d damped, h history, * valid, > best
              i - internal, r RIB-failure, S stale, N Nexthop-discard
Origin codes: i - IGP, e - EGP, ? - incomplete
   Network            Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 65013:13 (default for vrf CustB-CE-2)
*>i10.13.1.0/30       9.9.9.109                0    100      0 ?
*> 10.13.2.0/30       0.0.0.0                  0         32768 ?
*>i192.168.131.0/25   9.9.9.109                0    100      0 65013 i
*>i192.168.131.128/25 9.9.9.109                0    100      0 65013 i
*> 192.168.132.0/25   10.13.2.2                0             0 65013 i
*> 192.168.132.128/25 10.13.2.2                0             0 65013 i

And on PE02

RP/0/RP0/CPU0:PE02#show bgp vrf all
Fri Aug 12 19:38:48.061 UTC

VRF: CustA-CE-2
---------------
BGP VRF CustA-CE-2, state: Active
BGP Route Distinguisher: 65011:11
VRF ID: 0x60000001
BGP router identifier 9.9.9.9, local AS number 65000
Non-stop routing is enabled
BGP table state: Active
Table ID: 0xe0000001   RD version: 60
BGP main routing table version 63
BGP NSR Initial initsync version 19 (Reached)
BGP NSR/ISSU Sync-Group versions 0/0

Status codes: s suppressed, d damped, h history, * valid, > best
              i - internal, r RIB-failure, S stale, N Nexthop-discard
Origin codes: i - IGP, e - EGP, ? - incomplete
   Network            Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 65011:11 (default for vrf CustA-CE-2)
*>i10.11.1.0/30       7.7.7.107                0    100      0 ?
*> 10.11.2.0/30       0.0.0.0                  0         32768 ?
*>i192.168.111.0/25   7.7.7.107                0    100      0 65011 i
*>i192.168.111.128/25 7.7.7.107                0    100      0 65011 i
*> 192.168.112.0/25   10.11.2.2                0             0 65011 i
*> 192.168.112.128/25 10.11.2.2                0             0 65011 i

Processed 6 prefixes, 6 paths

VRF: CustB-CE-1
---------------
BGP VRF CustB-CE-1, state: Active
BGP Route Distinguisher: 65013:13
VRF ID: 0x60000002
BGP router identifier 9.9.9.9, local AS number 65000
Non-stop routing is enabled
BGP table state: Active
Table ID: 0xe0000002   RD version: 63
BGP main routing table version 63
BGP NSR Initial initsync version 19 (Reached)
BGP NSR/ISSU Sync-Group versions 0/0

Status codes: s suppressed, d damped, h history, * valid, > best
              i - internal, r RIB-failure, S stale, N Nexthop-discard
Origin codes: i - IGP, e - EGP, ? - incomplete
   Network            Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 65013:13 (default for vrf CustB-CE-1)
*> 10.13.1.0/30       0.0.0.0                  0         32768 ?
*>i10.13.2.0/30       7.7.7.107                0    100      0 ?
*> 192.168.131.0/25   10.13.1.2                0             0 65013 i
*> 192.168.131.128/25 10.13.1.2                0             0 65013 i
*>i192.168.132.0/25   7.7.7.107                0    100      0 65013 i
*>i192.168.132.128/25 7.7.7.107                0    100      0 65013 i

Processed 6 prefixes, 6 paths

So I know MPLS is forwarding correctly ( I think ). 
But when I check the routes on the CE, only the transit route appears for the other CE. 
CustA-ce-01:

CustA-CE-1(config-router)#do show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       a - application route
       + - replicated route, % - next hop override, p - overrides from PfR

Gateway of last resort is not set

      10.0.0.0/8 is variably subnetted, 3 subnets, 2 masks
C        10.11.1.0/30 is directly connected, Ethernet1/0
L        10.11.1.2/32 is directly connected, Ethernet1/0
B        10.11.2.0/30 [20/0] via 10.11.1.1, 00:57:15
      11.0.0.0/32 is subnetted, 1 subnets
C        11.11.11.11 is directly connected, Loopback0
      192.168.0.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.0.0/24 is directly connected, Ethernet0/0
L        192.168.0.131/32 is directly connected, Ethernet0/0
      192.168.111.0/24 is variably subnetted, 4 subnets, 2 masks
C        192.168.111.0/25 is directly connected, Loopback10
L        192.168.111.1/32 is directly connected, Loopback10
C        192.168.111.128/25 is directly connected, Loopback11
L        192.168.111.129/32 is directly connected, Loopback11

And CustA-ce-02

CustA-CE-2#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       a - application route
       + - replicated route, % - next hop override, p - overrides from PfR

Gateway of last resort is not set

      10.0.0.0/8 is variably subnetted, 3 subnets, 2 masks
B        10.11.1.0/30 [20/0] via 10.11.2.1, 00:49:49
C        10.11.2.0/30 is directly connected, Ethernet1/0
L        10.11.2.2/32 is directly connected, Ethernet1/0
      12.0.0.0/32 is subnetted, 1 subnets
C        12.12.12.12 is directly connected, Loopback0
      192.168.0.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.0.0/24 is directly connected, Ethernet0/0
L        192.168.0.132/32 is directly connected, Ethernet0/0
      192.168.112.0/24 is variably subnetted, 4 subnets, 2 masks
C        192.168.112.0/25 is directly connected, Loopback10
L        192.168.112.1/32 is directly connected, Loopback10
C        192.168.112.128/25 is directly connected, Loopback11
L        192.168.112.129/32 is directly connected, Loopback11

So If I check the PE advertised routes, thats exactly what I see, only the transit route is being advertised to CE's

PE01

RP/0/RP0/CPU0:PE01#show bgp vrf CustA-CE-1 advertised
Fri Aug 12 19:41:28.266 UTC
Route Distinguisher: 65011:11 (default for vrf CustA-CE-1)
10.11.1.0/30 is advertised to 10.11.1.2
  Path info:
    neighbor: Local           neighbor router id: 7.7.7.7
    valid  redistributed  best  import-candidate
Received Path ID 0, Local Path ID 1, version 26
  Attributes after inbound policy was applied:
    next hop: 0.0.0.0
    MET ORG AS EXTCOMM
    origin: incomplete  metric: 0
    aspath:
    extended community: RT:65011:11
  Attributes after outbound policy was applied:
    next hop: 10.11.1.1
    MET ORG AS
    origin: incomplete  metric: 0
    aspath: 65000

Route Distinguisher: 65011:11 (default for vrf CustA-CE-1)
10.11.2.0/30 is advertised to 10.11.1.2
  Path info:
    neighbor: 5.5.5.5         neighbor router id: 9.9.9.9
    valid  internal  best  import-candidate  imported
Received Path ID 0, Local Path ID 1, version 64
  Attributes after inbound policy was applied:
    next hop: 9.9.9.109
    MET ORG AS LOCAL EXTCOMM
    origin: incomplete  metric: 0  local pref: 100
    aspath:
    extended community: RT:65011:11
    originator: 9.9.9.9    cluster list: 5.5.5.5
  Attributes after outbound policy was applied:
    next hop: 10.11.1.1
    ORG AS LOCAL
    origin: incomplete  local pref: 100
    aspath: 65000

PE02:

RP/0/RP0/CPU0:PE02#show bgp vrf CustA-CE-2 advertised
Fri Aug 12 19:42:09.464 UTC
Route Distinguisher: 65011:11 (default for vrf CustA-CE-2)
10.11.1.0/30 is advertised to 10.11.2.2
  Path info:
    neighbor: 5.5.5.5         neighbor router id: 7.7.7.7
    valid  internal  best  import-candidate  imported
Received Path ID 0, Local Path ID 1, version 58
  Attributes after inbound policy was applied:
    next hop: 7.7.7.107
    MET ORG AS LOCAL EXTCOMM
    origin: incomplete  metric: 0  local pref: 100
    aspath:
    extended community: RT:65011:11
    originator: 7.7.7.7    cluster list: 5.5.5.5
  Attributes after outbound policy was applied:
    next hop: 10.11.2.1
    ORG AS LOCAL
    origin: incomplete  local pref: 100
    aspath: 65000

Route Distinguisher: 65011:11 (default for vrf CustA-CE-2)
10.11.2.0/30 is advertised to 10.11.2.2
  Path info:
    neighbor: Local           neighbor router id: 9.9.9.9
    valid  redistributed  best  import-candidate
Received Path ID 0, Local Path ID 1, version 14
  Attributes after inbound policy was applied:
    next hop: 0.0.0.0
    MET ORG AS EXTCOMM
    origin: incomplete  metric: 0
    aspath:
    extended community: RT:65011:11
  Attributes after outbound policy was applied:
    next hop: 10.11.2.1
    MET ORG AS
    origin: incomplete  metric: 0
    aspath: 65000

Why are none of advertised CE network routes being forwarded to the other CE's in the vrf?
I am being conservative with my configurations as there are many, also the P and RR are Juniper vMX systems. If you need to know anything about them let me know, however I have a feeling this is a CE <-> PE configuration issue. 

[Harold Ritter]

Hi @davehouser1 ,

I do not see any BGP neighbor sessions with address-family vpnv4 unicast in your configuration. Can you confirm that you configured VPNv4? Also, can you please confirm what you are trying to achieve. If I understand you correctly, you want to run L3VPN without using LDP, right?

Regards, 

[davehouser1]

@Harold Ritter 
That is correct, I am using BGP-LU to distribute labels, and using a RR with vpnv4 to reflect those routes.

Here is the vpnv4 unicast neighbor connection to the vpnv4 enabled RR. I am using iBGP to connect to it via the PE's Loopback0 address. 

Note:
5.5.5.5 = vpnv4 RR
6.6.6.6 = a separate unicast RR

PE01:

RP/0/RP0/CPU0:PE01#show bgp vpnv4 unicast  neighbors brief
Neighbor        Spk    AS Description                          Up/Down  NBRState
5.5.5.5           0 65000                                      02:17:07 Established
RP/0/RP0/CPU0:PE01#show bgp summary
Fri Aug 12 21:08:01.298 UTC
BGP router identifier 7.7.7.7, local AS number 65000
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0xe0000000   RD version: 480
BGP main routing table version 480
BGP NSR Initial initsync version 11 (Reached)
BGP NSR/ISSU Sync-Group versions 0/0
BGP scan interval 60 secs

BGP is operating in STANDALONE mode.


Process       RcvTblVer   bRIB/RIB   LabelVer  ImportVer  SendTblVer  StandbyVer
Speaker             480        480        480        480         480           0

Neighbor        Spk    AS MsgRcvd MsgSent   TblVer  InQ OutQ  Up/Down  St/PfxRcd
6.6.6.6           0 65000    2509    2809      480    0    0 02:17:48          3
10.17.101.0       0 65001    5193    5522      480    0    0 01:58:01          3
10.17.201.0       0 65001    6037    5548      480    0    0 01:58:01          3

PE02:

RP/0/RP0/CPU0:PE01#show bgp vpnv4 unicast  neighbors brief
Fri Aug 12 21:07:52.881 UTC

Neighbor        Spk    AS Description                          Up/Down  NBRState
5.5.5.5           0 65000                                      03:29:09 Established
RP/0/RP0/CPU0:PE02#show bgp summary
Fri Aug 12 21:07:56.005 UTC
BGP router identifier 9.9.9.9, local AS number 65000
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0xe0000000   RD version: 170
BGP main routing table version 170
BGP NSR Initial initsync version 7 (Reached)
BGP NSR/ISSU Sync-Group versions 0/0
BGP scan interval 60 secs

BGP is operating in STANDALONE mode.


Process       RcvTblVer   bRIB/RIB   LabelVer  ImportVer  SendTblVer  StandbyVer
Speaker             170        170        170        170         170           0

Neighbor        Spk    AS MsgRcvd MsgSent   TblVer  InQ OutQ  Up/Down  St/PfxRcd
6.6.6.6           0 65000    1155    1291      170    0    0 03:29:19          3
10.17.102.0       0 65001    2250    2394      170    0    0 01:41:09          3
10.17.202.0       0 65001    2616    2407      170    0    0 01:41:24          3

I should have mentioned but I am new to all of this. I am trying to teach myself RR + BGP-LU + MPLS + VRF RD / RI / RE, SO I am sorry if I maybe asking questions that are obvious. I appreciate your patience and help through my learning!

[MHM Cisco World]

*>i192.168.112.0/25   9.9.9.109                0    100      0 65011 i
*>i192.168.112.128/25 9.9.9.109                0    100      0 65011 i

I know little about the IOS XR but the next-hop  for prefix is not change and hence the the BGP NOT validate prefix and not install in RIB, 
the next-hop must not change it must be the CE-PE connection IP not the router-ID of PE.

sorry for my little info.

[Harold Ritter · Accepted solution]

HI @davehouser1 ,

Since both CEs are configured with the same ASN (65011), you need to configure the following on the PE01 and PE02 to allow the updates to be accepted on the CEs.

neighbor-group CustA
address-family ipv4 unicast
as-override

 Regards,

[davehouser1]

@Harold Ritter That worked! I can now see the advertised networks per VRF routes in the CE's. I am getting closer TBH your answer is the solution to the problem of this post.
However, pinging from CustA-ce-01 to CustA-ce-02 on the CustA vrf does not work (Same for CustB), it seems the icmp packet stops at the PE and goes no further. I may need to open a new ticket on this, not sure if you recommend anything I can try to figure out why they cannot ping? 

@MHM Cisco World I am not clear on what you mean when saying this:

the next-hop must not change it must be the CE-PE connection IP not the router-ID of PE

---

Based on your statement, what should the IP be for the next hop be? Example, currently, you are right, the next hop listed on PE01 is 9.9.9.109, which is secondary Loopback address on PE02. But it is not the primary ID of that router. Performing a show route on the PE01 I see the following:

RP/0/RP0/CPU0:PE01#show route 9.9.9.109
Sat Aug 13 14:11:01.103 UTC

Routing entry for 9.9.9.109/32
  Known via "bgp 65000", distance 20, metric 10, [ei]-bgp, labeled unicast (3107) (AIGP metric)
  Tag 65001, type external
  Installed Aug 13 13:47:05.770 for 00:23:55
  Routing Descriptor Blocks
    10.18.101.0, from 10.18.101.0, BGP external
      Route metric is 10
  No advertising protos.

However when I try to ping 9.9.9.109 from PE01, I get no response. I did a capture on the line to 10.18.101.0/31, but no ICMP pings show to 9.9.9.109.

Feel like I am missing more here. Not sure if simple fix or some other issue possibly in my RR.

[Harold Ritter]

Hi @davehouser1 ,

Great, it sound like the control plane is now working.

I presume that you only advertise the secondary loopback IP addresses via BGP-LU. So when you ping 9.9.9.109 from PE01, you should specify the source as the secondary loopback. If it still doesn't work it means you have and issue with your data plane. You could also try the following command to verify you MPLS data plane between PE01 and PE02:

ping mpls ipv4 9.9.9.109/32 source <secondary loopback interface IP address>

Please provide the following:

"show mpls int" on all routers from PE01 to PE02 and make sure all interfaces are MPLS enabled

"show cef 9.9.9.109/32" from PE01?

"show run router bgp"  from PE01

"show runs router static" from PE01

Regards,

[davehouser1]

@Harold Ritter thanks for the tips. Here is the output from the mpls ping (All "Q"s)

RP/0/RP0/CPU0:PE01#show ip int br | i Loopback
Sat Aug 13 15:10:32.790 UTC
Loopback0                      7.7.7.7         Up              Up       default
Loopback10                     7.7.7.107       Up              Up       default

RP/0/RP0/CPU0:PE01#ping mpls ipv4 9.9.9.109/32 source 7.7.7.107
Sat Aug 13 14:59:16.246 UTC

Sending 5, 100-byte MPLS Echos to 9.9.9.109/32,
      timeout is 2 seconds, send interval is 0 msec:

Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
  'L' - labeled output interface, 'B' - unlabeled output interface,
  'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch,
  'M' - malformed request, 'm' - unsupported tlvs, 'N' - no rx label,
  'P' - no rx intf label prot, 'p' - premature termination of LSP,
  'R' - transit router, 'I' - unknown upstream index,
  'X' - unknown return code, 'x' - return code 0

Type escape sequence to abort.

QQQQQ
Success rate is 0 percent (0/5)

I read another post where you commented that "Q"s are caused because the entry is not in the the RIB,FIB,LIB, or LFIB
Here are the entries in each, however I do see 9.9.9.109 in each. 

RP/0/RP0/CPU0:PE01#show rib tables
Sat Aug 13 15:05:21.006 UTC

Codes: N - Prefix Limit Notified, F - Forward Referenced
       D - Table Deleted, C - Table Reached Convergence

VRF/Table              SAFI  Table ID     PrfxLmt   PrfxCnt TblVersion  N F D C
default/default        uni   0xe0000000  10000000        17        439  N N N Y
CustA-CE-1/default     uni   0xe0000001  10000000         7        165  N N N Y
CustB-CE-2/default     uni   0xe0000002  10000000         7        165  N N N Y
**iid/default          uni   0xe0003fff  10000000         0          0  N N N Y
default/default        multi 0xe0100000  10000000         0          0  N N N Y

RP/0/RP0/CPU0:PE01#show ip route
Sat Aug 13 15:16:31.569 UTC

Codes: C - connected, S - static, R - RIP, B - BGP, (>) - Diversion path
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP
       i - ISIS, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, su - IS-IS summary null, * - candidate default
       U - per-user static route, o - ODR, L - local, G  - DAGR, l - LISP
       A - access/subscriber, a - Application route
       M - mobile route, r - RPL, t - Traffic Engineering, (!) - FRR Backup path

Gateway of last resort is not set

B    5.5.5.5/32 [20/10] via 10.17.101.0, 00:12:23
B    6.6.6.6/32 [20/10] via 10.17.101.0, 00:12:23
L    7.7.7.7/32 is directly connected, 06:33:31, Loopback0
L    7.7.7.107/32 is directly connected, 06:33:31, Loopback10
B    9.9.9.9/32 [20/10] via 10.17.101.0, 00:12:23
B    9.9.9.109/32 [20/10] via 10.18.101.0, 00:12:51
C    10.17.101.0/31 is directly connected, 00:12:53, GigabitEthernet0/0/0/2
L    10.17.101.1/32 is directly connected, 00:12:53, GigabitEthernet0/0/0/2
C    10.17.201.0/31 is directly connected, 00:12:53, GigabitEthernet0/0/0/4
L    10.17.201.1/32 is directly connected, 00:12:53, GigabitEthernet0/0/0/4
C    10.18.101.0/31 is directly connected, 00:12:53, GigabitEthernet0/0/0/3
L    10.18.101.1/32 is directly connected, 00:12:53, GigabitEthernet0/0/0/3
C    10.18.201.0/31 is directly connected, 00:12:53, GigabitEthernet0/0/0/5
L    10.18.201.1/32 is directly connected, 00:12:53, GigabitEthernet0/0/0/5
L    127.0.0.0/8 [0/0] via 0.0.0.0, 06:33:33
C    192.168.0.0/24 is directly connected, 06:33:30, MgmtEth0/RP0/CPU0/0
L    192.168.0.127/32 is directly connected, 06:33:30, MgmtEth0/RP0/CPU0/0



RP/0/RP0/CPU0:PE01#show ip cef
Sat Aug 13 15:05:54.317 UTC

Prefix              Next Hop            Interface
------------------- ------------------- ------------------
0.0.0.0/0           drop                default handler
0.0.0.0/32          broadcast
5.5.5.5/32          10.17.101.0/32      <recursive>
6.6.6.6/32          10.17.101.0/32      <recursive>
7.7.7.7/32          receive             Loopback0
7.7.7.107/32        receive             Loopback10
9.9.9.9/32          10.17.101.0/32      <recursive>
9.9.9.109/32        10.18.101.0/32 (?)  <recursive>
10.17.101.0/31      attached            GigabitEthernet0/0/0/2
10.17.101.0/32      10.17.101.0/32      GigabitEthernet0/0/0/2
10.17.101.1/32      receive             GigabitEthernet0/0/0/2
10.17.201.0/31      attached            GigabitEthernet0/0/0/4
10.17.201.1/32      receive             GigabitEthernet0/0/0/4
10.18.101.0/31      attached            GigabitEthernet0/0/0/3
10.18.101.0/32      10.18.101.0/32      GigabitEthernet0/0/0/3
10.18.101.1/32      receive             GigabitEthernet0/0/0/3
10.18.201.0/31      attached            GigabitEthernet0/0/0/5
10.18.201.1/32      receive             GigabitEthernet0/0/0/5
127.0.0.0/8         receive
192.168.0.0/24      attached            MgmtEth0/RP0/CPU0/0
192.168.0.0/32      broadcast           MgmtEth0/RP0/CPU0/0
192.168.0.121/32    192.168.0.121/32    MgmtEth0/RP0/CPU0/0
192.168.0.123/32    192.168.0.123/32    MgmtEth0/RP0/CPU0/0
192.168.0.125/32    192.168.0.125/32    MgmtEth0/RP0/CPU0/0
192.168.0.126/32    192.168.0.126/32    MgmtEth0/RP0/CPU0/0
192.168.0.127/32    receive             MgmtEth0/RP0/CPU0/0
192.168.0.129/32    192.168.0.129/32    MgmtEth0/RP0/CPU0/0
192.168.0.254/32    192.168.0.254/32    MgmtEth0/RP0/CPU0/0
192.168.0.255/32    broadcast           MgmtEth0/RP0/CPU0/0
224.0.0.0/4         0.0.0.0/32
224.0.0.0/24        receive
255.255.255.255/32  broadcast

RP/0/RP0/CPU0:PE01#show mpls forwarding
Sat Aug 13 15:07:42.933 UTC
Local  Outgoing    Prefix             Outgoing     Next Hop        Bytes
Label  Label       or ID              Interface                    Switched
------ ----------- ------------------ ------------ --------------- ------------
24000  Aggregate   CustA-CE-1: Per-VRF Aggr[V]   \
                                      CustA-CE-1                   0
24001  Aggregate   CustB-CE-2: Per-VRF Aggr[V]   \
                                      CustB-CE-2                   0
24002  Unlabelled  192.168.132.0/25[V]   \
                                      Gi0/0/0/1    10.13.2.2       0
24003  Unlabelled  192.168.132.128/25[V]   \
                                      Gi0/0/0/1    10.13.2.2       0
24004  306400      9.9.9.109/32                    10.18.101.0     0
24005  Unlabelled  192.168.111.0/25[V]   \
                                      Gi0/0/0/0    10.11.1.2       0
24006  Unlabelled  192.168.111.128/25[V]   \
                                      Gi0/0/0/0    10.11.1.2       0

RP/0/RP0/CPU0:PE01#show mpls forwarding detail
Sat Aug 13 15:07:51.232 UTC
Local  Outgoing    Prefix             Outgoing     Next Hop        Bytes
Label  Label       or ID              Interface                    Switched
------ ----------- ------------------ ------------ --------------- ------------
24000  Aggregate   CustA-CE-1: Per-VRF Aggr[V]   \
                                      CustA-CE-1                   0
     Updated: Aug 13 09:00:24.988
     Label Stack (Top -> Bottom): { }
     MAC/Encaps: 0/0, MTU: 0
     Packets Switched: 0
24001  Aggregate   CustB-CE-2: Per-VRF Aggr[V]   \
                                      CustB-CE-2                   0
     Updated: Aug 13 09:00:24.992
     Label Stack (Top -> Bottom): { }
     MAC/Encaps: 0/0, MTU: 0
     Packets Switched: 0
24002  Unlabelled  192.168.132.0/25[V]   \
                                      Gi0/0/0/1    10.13.2.2       0
     Updated: Aug 13 15:03:44.788
     Path Flags: 0x6020 [  EXT ]
     Version: 163, Priority: 3
     Label Stack (Top -> Bottom): { Unlabelled }
     NHID: 0x0, Encap-ID: N/A, Path idx: 0, Backup path idx: 0, Weight: 0
     MAC/Encaps: 4/4, MTU: 1500
     Outgoing Interface: GigabitEthernet0/0/0/1 (ifhandle 0x010000a0)
     Packets Switched: 0

24003  Unlabelled  192.168.132.128/25[V]   \
                                      Gi0/0/0/1    10.13.2.2       0
     Updated: Aug 13 15:03:44.788
     Path Flags: 0x6020 [  EXT ]
     Version: 165, Priority: 3
     Label Stack (Top -> Bottom): { Unlabelled }
     NHID: 0x0, Encap-ID: N/A, Path idx: 0, Backup path idx: 0, Weight: 0
     MAC/Encaps: 4/4, MTU: 1500
     Outgoing Interface: GigabitEthernet0/0/0/1 (ifhandle 0x010000a0)
     Packets Switched: 0

24004  306400      9.9.9.109/32                    10.18.101.0     0
     Updated: Aug 13 15:03:39.764
     Path Flags: 0x6020 [  EXT ]
     Version: 423, Priority: 4
     Label Stack (Top -> Bottom): { 306400 }
     NHID: 0x0, Encap-ID: N/A, Path idx: 0, Backup path idx: 0, Weight: 0
     MAC/Encaps: 0/4, MTU: 0
     Packets Switched: 0

24005  Unlabelled  192.168.111.0/25[V]   \
                                      Gi0/0/0/0    10.11.1.2       0
     Updated: Aug 13 15:03:42.599
     Path Flags: 0x6020 [  EXT ]
     Version: 163, Priority: 3
     Label Stack (Top -> Bottom): { Unlabelled }
     NHID: 0x0, Encap-ID: N/A, Path idx: 0, Backup path idx: 0, Weight: 0
     MAC/Encaps: 4/4, MTU: 1500
     Outgoing Interface: GigabitEthernet0/0/0/0 (ifhandle 0x01000078)
     Packets Switched: 0

24006  Unlabelled  192.168.111.128/25[V]   \
                                      Gi0/0/0/0    10.11.1.2       0
     Updated: Aug 13 15:03:42.599
     Path Flags: 0x6020 [  EXT ]
     Version: 165, Priority: 3
     Label Stack (Top -> Bottom): { Unlabelled }
     NHID: 0x0, Encap-ID: N/A, Path idx: 0, Backup path idx: 0, Weight: 0
     MAC/Encaps: 4/4, MTU: 1500
     Outgoing Interface: GigabitEthernet0/0/0/0 (ifhandle 0x01000078)
     Packets Switched: 0

 Red flags that are raised for me is the fact that the cef table has a "(?)" next to the 9.9.9.109 entry, also there is not actual interface for the entry in the MPLS forwarding table for 9.9.9.109.
Not sure if there are other commands I can use to check RIB,FIB,LIB, and LFIB, are there? 

[Harold Ritter]

Hi @davehouser1 ,

9.9.9.109/32        10.18.101.0/32 (?)  <recursive>

 This is definitely the issue.

There is a requirement in XR to have a static route towards the BGP peer used for BGP-LU.

router static

address-family ipv4 unicast

<ip address used for BGP-LU/32> <interface to the neighbor>

You need this on all the routers propagating the BGP-LU routes. This will solve the CEF issue.

Regards,

[davehouser1]

I am not clear on what I should be doing

static route towards the BGP peer used for BGP-LU

router static

address-family ipv4 unicast

<ip address used for BGP-LU/32> <interface to the neighbor>

---

<ip address used for BGP-LU/32> : Is this the RR that is serving vpnv4? In this case the bgp neighbor address is its loopback 5.5.5.5/32
<interface to the neighbor> : I am not sure what this should be. Each PE has two physical connections to each P router, which then has a single connection to each RR (One RR for unicast, and One for vpnv4). PE<->P connection1 = BGP-unicast. PE<-> connection2 = BGP-LU. I think I should be using my local loopback address of 9.9.9.109/32 but I am not sure. Right now the cef table shows that 5.5.5.5/32 is trying to use the physical unicast link which does not seem to be right. 

Either way I tried the following

This entry was try to use the PE's BGP-LU physical link

router static address-family ipv4 unicast 5.5.5.5/32 gig0/0/0/3

This did not seem to work I cant ping 5.5.5.5/32 still.

This entry was to try and use the PE's Looback10 interface 

router static address-family ipv4 unicast 5.5.5.5/32 Loopback10

This did not work either, I cant ping 5.5.5.5/32.
I tried adding these routes in each PE01 and PE02, did not seem to change anything from the CE side (pinging a BGP advertised route in the VRF) or the PE (pinging 5.5.5.5). 

Is there something else I am missing? 

[Harold Ritter]

Hi @davehouser1 ,

This issue has nothing to do with the BGP VPNv4 configuration, but rather with the BGP-LU configuration. If you want to use BGP-LU to exchange labels instead of LDP, you need to setup BGP-LU on a hop by hop basis between routers in the path between PE1 and PE2. Is that what you have configured?

The static route that I am referring to should be configured to the BGP-LU peer on the directly connected router. 

Can you please share the BGP-LU configuration from PE1?

Regards,

[MHM Cisco World]

as I mention before sorry for little info. but 
I do my best to help other
the BGP run between CE-PE is different than any other protocol, 
for example run ospf between CE-PE, the PE will advertise all prefix receive from CE (R5) and ALSO the link between CE-PE (which is next-hop for ALL prefix learn from CE) 

in BGP this need more config 
BGP advertise the prefix receive from CE but link between CE-PE not advertise by default, 
this make other CE (R4) can not reach the prefix in CE (R5)
even so the prefix in both PE (R2 & R3)
so advertise the the Link solve the issue make CE (R4) know the next-hop for prefix advertise by CE(R5)

and return to your issue 
there is primary and secondary IP in link between CE and PE, 
are you advertise correctly?
I prefer advertise the prefix as /32 Why because the LDP is have some issue with not /32 prefix. 

hope this help you