Route Reflector is not only getting 2/3 of the Full internet table

[DanielGutierrez615]

I'm trying to figure out why my router reflector is only getting 2/3 of the full Internet routing table.

the first output here is from the route reflector if you notice it's only accepting "664019 accepted prefixes, 664019 are bestpaths"

If you look at the second output is accepting over 924094 routers from the Service provider, but it's only injecting about 664105. 

any ideas of why???

P/0/RP0/CPU0:LBV-RTR-PE1-01#show bgp vpnv4 unicast summary
Neighbor Spk AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down St/PfxRcd
10.0.24.5 0 11749 6217030 2678082 90750965 0 0 00:38:58 664105
10.0.24.6 0 11749 6565847 2773507 90750965 0 0 00:46:09 664105 ##################################################

show bgp vpnv4 unicast neighbors 10.0.24.4 detail

For Address Family: VPNv4 Unicast
BGP neighbor version 1947899
Update group: 0.2 Filter-group: 0.3 No Refresh request being processed
Route-Reflector Client
Inbound soft reconfiguration allowed (override route-refresh)
Advertise Translation Enabled
AF-dependent capabilities:
Graceful Restart capability received
Remote Restart time is 120 seconds
Neighbor did not preserve the forwarding state during latest restart
Additional-paths Send: advertised and received
Additional-paths Receive: advertised and received
Extended Nexthop Encoding: advertised and received
Route refresh request: received 0, sent 0
664019 accepted prefixes, 664019 are bestpaths
Exact no. of prefixes denied : 0.
Cumulative no. of prefixes denied: 0.
Prefix advertised 1308264, suppressed 0, withdrawn 6631
AIGP is enabled
An EoR was received during read-only mode
Last ack version 1947898, Last synced ack version 0
Outstanding version objects: current 1, max 9, refresh 0
Additional-paths operation: Send and Receive
Send Multicast Attributes
Slow peer flags: 18

##############################################################

RP/0/RP0/CPU0:LBV-RTR-PE1-01#show bgp vrf Internet summary
Mon Mar 4 13:46:55.133 EST
BGP VRF Internet, state: Active
BGP Route Distinguisher: 10.0.24.2:0
VRF ID: 0x60000001
BGP router identifier 10.0.24.2, local AS number 11749
Non-stop routing is enabled
BGP table state: Active
Table ID: 0xe0000001 RD version: 90740253
BGP main routing table version 90740253
BGP NSR Initial initsync version 7 (Reached)
BGP NSR/ISSU Sync-Group versions 0/0

BGP is operating in STANDALONE mode.

Process RcvTblVer bRIB/RIB LabelVer ImportVer SendTblVer StandbyVer
Speaker 90740253 90740253 90740253 90740253 90740253 0

Neighbor Spk AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down St/PfxRcd
38.142.102.97 0 174 2355844 5851 90740253 0 0 1d19h 924094

[DanielGutierrez615]

I do see many of these in the debugs , but I do not have the address-family ipv4 rt-filter  on the peering nodes . Only on the access nodes or PE devices

router bgp 11749
bgp router-id 10.0.24.5
bgp log neighbor changes detail
ibgp policy out enforce-modifications
address-family ipv4 unicast
additional-paths receive
additional-paths send
!
address-family vpnv4 unicast
additional-paths receive
additional-paths send
retain route-target all
!
address-family l2vpn vpls-vpws
retain route-target all
!
address-family ipv4 rt-filter
!
address-family l2vpn evpn
additional-paths receive
additional-paths send
!
neighbor-group core_nodes
remote-as 11749
update-source Loopback0
address-family vpnv4 unicast
route-reflector-client
advertise vpnv4 unicast
soft-reconfiguration inbound always
!
address-family ipv4 rt-filter
route-reflector-client
soft-reconfiguration inbound always
!
address-family l2vpn evpn
route-reflector-client
soft-reconfiguration inbound always
advertise l2vpn evpn
!
!
neighbor-group peer_nodes
remote-as 11749
update-source Loopback0
address-family ipv4 unicast
route-reflector-client
soft-reconfiguration inbound always
!
address-family vpnv4 unicast
route-reflector-client
advertise vpnv4 unicast
soft-reconfiguration inbound always
!
address-family l2vpn evpn
route-reflector-client
soft-reconfiguration inbound always
advertise l2vpn evpn
!
!
neighbor-group access_nodes
remote-as 11749
update-source Loopback0
address-family vpnv4 unicast
route-reflector-client
advertise vpnv4 unicast
soft-reconfiguration inbound always
!
address-family l2vpn vpls-vpws
route-reflector-client
soft-reconfiguration inbound always
Signalling ldp disable
!
address-family ipv4 rt-filter
route-reflector-client
soft-reconfiguration inbound always
!
address-family l2vpn evpn
route-reflector-client
soft-reconfiguration inbound always
advertise l2vpn evpn
!
!

0x00001022, sendlab=1: net=v4Addr:10.0.24.2:0:194.36.223.0/24, nver=2063903: PELEM=0x7f1f5d85e298 (lpathid=1, ver=2063903, fl=0x00001001): PATH=0x7f1f483492a8 (10.0.24.2/32,10.0.24.2,1, 0x2000000025068205)::: allowbe=0, isbe=0, allowspurwdr=0, pelem-send=1, pelem-wdr=0
RP/0/RP0/CPU0:Mar 4 20:23:19.058 UTC: bgp[1078]: [default-upd] (vpn4u): <UPDLABEL>: tbl=TBL:default (1/128), afi=4: ug=0.3, sg=0.2, ugfl=0x0010418b: net=v4Addr:10.0.24.2:0:194.36.223.0/24, PELEM=0x7f1f5d85e298(lpathid=1, fl=0x00001001), PATH=0x7f1f483492a8(10.0.24.2/32,10.0.24.2,1, 0x2000000025068205), reflected=1, bmsgfl=0x00000000, wdr=0::: netlab=0/0, pathlab=24017, updlab=24017
RP/0/RP0/CPU0:Mar 4 20:23:19.058 UTC: bgp[1078]: [default-upd] (vpn4u): Deny UPDATE to filter-group 0.28 (Regular, pelem Regular) for v4Addr:10.0.24.2:0:194.36.223.0/24 (changedfl=0x0/0x1000), path 174 3356 3214 reason Dropped by RT Filter
RP/0/RP0/CPU0:Mar 4 20:23:19.058 UTC: bgp[1078]: [default-upd] (vpn4u): No unreachable (Dropped by RT Filter) sent to sub-group 0.2 (Regular) with v4Addr:10.0.24.2:0:194.36.223.0/24 - already withdrawn
RP/0/RP0/CPU0:Mar 4 20:23:19.058 UTC: bgp[1078]: [default-upd] (vpn4u): Permit UPDATE to filter-group 0.4 (Regular, pelem Regular) for v4Addr:10.0.24.2:0:194.36.223.0/24 (changedfl=0x0/0x1000), path 174 3356 3214
RP/0/RP0/CPU0:Mar 4 20:23:19.058 UTC: bgp[1078]: [default-upd] (vpn4u): process UPDATE for: tbl=TBL:default (1/128), afi=4: ug=0.3, (Regular), pelem (Regular), sg=0.2, ugfl=0x0010418b: bgpctxfl=0x23, tblctxfl=0x00001022, ltblctxfl=0x00001022, sendlab=1: net=v4Addr:10.0.24.2:0:149.50.74.0/24, nver=2063914: PELEM=0x7f1f4c911188 (lpathid=1, ver=2063914, fl=0x00001001): PATH=0x7f1f483482e8 (10.0.24.2/32,10.0.24.2,1, 0x2000000025068205)::: allowbe=0, isbe=0, allowspurwdr=0, pelem-send=1, pelem-wdr=0
RP/0/RP0/CPU0:Mar 4 20:23:19.058 UTC: bgp[1078]: [default-upd] (vpn4u): <UPDLABEL>: tbl=TBL:default (1/128), afi=4: ug=0.3, sg=0.2, ugfl=0x0010418b: net=v4Addr:10.0.24.2:0:149.50.74.0/24, PELEM=0x7f1f4c911188(lpathid=1, fl=0x00001001), PATH=0x7f1f483482e8(10.0.24.2/32,10.0.24.2,1, 0x2000000025068205), reflected=1, bmsgfl=0x00000000, wdr=0::: netlab=0/0, pathlab=24017, updlab=24017
RP/0/RP0/CPU0:Mar 4 20:23:19.059 UTC: bgp[1078]: [default-upd] (vpn4u): Deny UPDATE to filter-group 0.28 (Regular, pelem Regular) for v4Addr:10.0.24.2:0:149.50.74.0/24 (changedfl=0x0/0x1000), path 174 3356 3214 55158 reason Dropped by RT Filter
RP/0/RP0/CPU0:Mar 4 20:23:19.059 UTC: bgp[1078]: [default-upd] (vpn4u): No unreachable (Dropped by RT Filter) sent to sub-group 0.2 (Regular) with v4Addr:10.0.24.2:0:149.50.74.0/24 - already withdrawn
RP/0/RP0/CPU0:Mar 4 20:23:19.059 UTC: bgp[1078]: [default-upd] (vpn4u): Permit UPDATE to filter-group 0.4 (Regular, pelem Regular) for v4Addr:10.0.24.2:0:149.50.74.0/24 (changedfl=0x0/0x1000), path 174 3356 3214 55158
RP/0/RP0/CPU0:Mar 4 20:23:19.059 UTC: bgp[1078]: [default-upd] (vpn4u): Sending UPDATE message(0x0x7f2021930da8) to sub-group 0.2 (Regular, pelem Regular) for v4Addr:10.0.24.2:0:149.50.74.0/24 (changedfl=0x0/0x1000) - creating new message with bmsgflags=0x00002000, attributes: nexthop 0.0.0.0, originator 10.0.24.2
RP/0/RP0/CPU0:Mar 4 20:23:19.059 UTC: bgp[1078]: [default-upd] (vpn4u): origin i, path 174 3356 3214 55158, metric 119040, localpref 100, community 174:21000 174:22013, extended community RT:100:242 , clusterlist 10.0.24.5, originator 10.0.24.2
RP/0/RP0/CPU0:Mar 4 20:23:19.059 UTC: bgp[1078]: [default-upd] (vpn4u): Created msg elem 0x0x7f2021789bf8 (pointing to message 0x0x7f2021930da8), for filtergroup 0.4
RP/0/RP0/CPU0:Mar 4 20:23:19.059 UTC: bgp[1078]: [default-upd] (vpn4u): process UPDATE for: tbl=TBL:default (1/128), afi=4: ug=0.3, (Regular), pelem (Regular), sg=0.2, ugfl=0x0010418b: bgpctxfl=0x23, tblctxfl=0x00001022, ltblctxfl=0x00001022, sendlab=1: net=v4Addr:10.0.24.2:0:149.50.81.0/24, nver=2063913: PELEM=0x7f1f4c911038 (lpathid=1, ver=2063913, fl=0x00001001): PATH=0x7f1f48347f88 (10.0.24.2/32,10.0.24.2,1, 0x2000000025068205)::: allowbe=0, isbe=0, allowspurwdr=0, pelem-send=1, pelem-wdr=0
RP/0/RP0/CPU0:Mar 4 20:23:19.059 UTC: bgp[1078]: [default-upd] (vpn4u): <UPDLABEL>: tbl=TBL:default (1/128), afi=4: ug=0.3, sg=0.2, ugfl=0x0010418b: net=v4Addr:10.0.24.2:0:149.50.81.0/24, PELEM=0x7f1f4c911038(lpathid=1, fl=0x00001001), PATH=0x7f1f48347f88(10.0.24.2/32,10.0.24.2,1, 0x2000000025068205), reflected=1, bmsgfl=0x00000000, wdr=0::: netlab=0/0, pathlab=24017, updlab=24017
RP/0/RP0/CPU0:Mar 4 20:23:19.059 UTC: bgp[1078]: [default-upd] (vpn4u): Deny UPDATE to filter-group 0.28 (Regular, pelem Regular) for v4Addr:10.0.24.2:0:149.50.81.0/24 (changedfl=0x0/0x1000), path 174 3356 3214 55158 reason Dropped by RT Filter
RP/0/RP0/CPU0:Mar 4 20:23:19.059 UTC: bgp[1078]: [default-upd] (vpn4u): No unreachable (Dropped by RT Filter) sent to sub-group 0.2 (Regular) with v4Addr:10.0.24.2:0:149.50.81.0/24 - already withdrawn
RP/0/RP0/CPU0:Mar 4 20:23:19.059 UTC: bgp[1078]: [default-upd] (vpn4u): Permit UPDATE to filter-group 0.4 (Regular, pelem Regular) for v4Addr:10.0.24.2:0:149.50.81.0/24 (changedfl=0x0/0x1000), path 174 3356 3214 55158
RP/0/RP0/CPU0:Mar 4 20:23:19.059 UTC: bgp[1078]: [default-upd] (vpn4u): process UPDATE for: tbl=TBL:default (1/128), afi=4: ug=0.3, (Regular), pelem (Regular), sg=0.2, ugfl=0x0010418b: bgpctxfl=0x23, tblctxfl=0x00001022, ltblctxfl=0x00001022, sendlab=1: net=v4Addr:10.0.24.2:0:149.50.88.0/24, nver=2063912: PELEM=0x7f1f4c910e08 (lpathid=1, ver=2063912, fl=0x00001001): PATH=0x7f1f483544c8 (10.0.24.2/32,10.0.24.2,1, 0x2000000025068205)::: allowbe=0, isbe=0, allowspurwdr=0, pelem-send=1, pelem-wdr=0
RP/0/RP0/CPU0:Mar 4 20:23:19.059 UTC: bgp[1078]: [default-upd] (vpn4u): <UPDLABEL>: tbl=TBL:default (1/128), afi=4: ug=0.3, sg=0.2, ugfl=0x0010418b: net=v4Addr:10.0.24.2:0:149.50.88.0/24, PELEM=0x7f1f4c910e08(lpathid=1, fl=0x00001001), PATH=0x7f1f483544c8(10.0.24.2/32,10.0.24.2,1, 0x2000000025068205), reflected=1, bmsgfl=0x00000000, wdr=0::: netlab=0/0, pathlab=24017, updlab=24017
RP/0/RP0/CPU0:Mar 4 20:23:19.059 UTC: bgp[1078]: [default-upd] (vpn4u): Deny UPDATE to filter-group 0.28 (Regular, pelem Regular) for v4Addr:10.0.24.2:0:149.50.88.0/24 (changedfl=0x0/0x1000), path 174 3356 3214 55158 reason Dropped by RT Filter

[MHM Cisco World]

retain route-target all <<- this apply under VPNv4 only not under l2vpn 
this I think make prefix not learn from RR

MHM

[Harold Ritter]

Hi @MHM Cisco World ,

This statement applies to any address-family using RT when you want to retain the updates even though the RT is not locally configured. 

Regards,

[MHM Cisco World]

Thanks just to confirm 
for any address family meaning both VPNv4 and L2VPN ?
MHM

[Harold Ritter]

L2VPN AF uses RTs as well. so yes it applies to this AF too.

Regards,

[Harold Ritter]

Hi @DanielGutierrez615 ,

The default label allocation in IOS-XR is per prefix. This means you will allocate an MPLS label for each and every Internet prefix you advertised via VPNv4. You might be running out of labels on PE1. Check for any related error messages in PE1 logs.

The better approach would be to configure per CE or per VRF label allocation mode on PE1. This would mean that a single label would be allocated for each CE (Internet peering) or one label for the entire Internet VRF.

router bgp 11749

vrf Internet

  address-family ipv4 unicast

   label mode per-ce (allocates 1 label per CE/peering router)

router bgp 11749

vrf Internet

  address-family ipv4 unicast

   label mode per-vrf (allocates 1 label for the entire VRF and forces an extra lookup on the egress PE)

Regards,

[DanielGutierrez615]

Thank you for your response.  I was thinking about that.I do have it as label-allocation-mode per-ce on the peering router  connections to service provider , but no the Access PE.  Access PEs are only getting the default route.  I just tried applying the same command on the router reflector, but same result. I didn't think it needed since the only thing it's doing is reflecting routes.

vrf Internet
rd auto
label-allocation-mode per-ce
address-family ipv4 unicast
network 0.0.0.0/0
network XXX.XXX.XXX.X/22
redistribute connected
redistribute static

[Harold Ritter]

Hi @DanielGutierrez615 ,

The change of alloc mode is only required on the Internet facing PE injecting the Internet routes in the MPLS VPN network. 

Regards,

[DanielGutierrez615]

As I'm troubleshooting I found that if I shut down one of my peering routers (meaning a router that is getting full routes from a service provider) The route reflector will start receiving all Internet routers. but if I no shut the interface facing toward the Ebgp peer the route reflector is back to 626k routes. 

[Harold Ritter]

Hi @DanielGutierrez615 ,

Can you check if you have any error messages in the route reflector logs?

Regards,

[DanielGutierrez615]

yes,  I posted earlier, the second post message. If you see the logs, the only thing I can see is that the networks are being discarded because of the rt address family, but I only use this address family to stop advertisement from the router reflectors to the PE nodes

RP/0/RP0/CPU0:Mar 4 20:23:19.058 UTC: bgp[1078]: [default-upd] (vpn4u): No unreachable (Dropped by RT Filter) sent to sub-group 0.2 (Regular) with v4Addr:10.0.24.2:0:147.78.179.0/24 - already withdrawn
RP/0/RP0/CPU0:Mar 4 20:23:19.058 UTC: bgp[1078]: [default-upd] (vpn4u): Permit UPDATE to filter-group 0.4 (Regular, pelem Regular) for v4Addr:10.0.24.2:0:147.78.179.0/24 (changedfl=0x0/0x1000), path 174 3356 3214

[Harold Ritter]

Hi @DanielGutierrez615 ,

Since the peering routers are being used as VPNv4 PEs, I think it would be better to configure the AF ipv4 rt-filter between the RR and the peering routers as well.

Regards,

[MHM Cisco World]

these logs not help us so much
debug ip bgp vpnv4 unicast import update 10 <<- 10 is ACL match any prefix missing from RIB table of router 
above command is for IOS XE and I think it same for IOS XR

MHM

[DanielGutierrez615]

I think the issue has to be on  importing and exporting the route targets on the edge router peering with the ISP. Right now I get a little bit more 900k from each ISP.  All PEs are peering IBGP with router reflectors 1 and 2.   All Edge peering nodes are IBGP peering with both router reflectors.  IF I look at the routes received at the RR1 and RR2 I'm only getting a little bit more than 600k. BUT !! when I shut down the peer of one of the peering routers, the RR starts to get the full 900k internet routers. Same happens when I shut down the other peering router.  If I have both peering sessions UP. the both RR get back to 600k. So I say the problem has to be on the importing and exporting route-target of the internet vrf.   Now everything works there is not issue. I have not packet loss and traffic gets rerouted to the ISP that is available when I shutdown the other ISP or ebgp session to the ISP, but my understanding is that both RRs should have the full routing table from Edge peering routers for better functionality. I'm attaching a diagram with the Internet VRF with the route targets and policies to you can take a look. I'm still researching on this to find an answer, but any hints you can provide will help a lot. thank you!!!!

Same output on both RR1 AND RR2

Neighbor Spk AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down St/PfxRcd
10.0.24.2 0 11749 311045 771459 3898203 0 0 16:25:44 622374
10.0.24.3 0 11749 994 771459 3898203 0 0 16:25:46 3  (this is the future peering router that does not have an ISP connection yet)
10.0.24.4 0 11749 555300 771459 3898203 0 0 16:25:46 663023