Redistribute static under router bgp zzzzz vrf xxxxx does not work

[DanielGutierrez615]

I was working on a maintenance window.  moving some customers to our  new MPLS network  PE router NCS540 SOFTWARE 7.6.2.

When I move 192.168.70.0/24 to the new mpls network   I was able to ping it from everywhere. however a static routed network I was not able to reach even though I could see the route advertized everywhere in the network

int bvi 100

vrf CORE_14288

ipv4 address 192.168.70.1 255.255.255.0

!

router bgp 11749
vrf CORE_14288
rd auto
address-family ipv4 unicast
redistribute connected
redistribute static
!

!

router static
vrf CORE_14288
address-family ipv4 unicast

192.168.10.0/27 192.168.70.2

After doing some digging I found that the redistribute static was not  allowing the traffic to pass thru even though I could see it advertized from every PE in the network.

If I hardcode it like shown below, it works. 

router bgp xxxx

vrf CORE_14288
address-family ipv4 unicast
192.168.10.0/27 192.168.70.2
!
!

I opened a ticket with cisco just to make sure this is not a bug, but I'm curious to see if anybody has experience the same behaviour. 

[Harold Ritter]

Hi @DanielGutierrez615 ,

It looks like the label learned via BGP VPNv4 on MTD-RTR-PE1-01 is 26065 when it works and 26085 when it does not work. The label installed on the egress PE (SCTLAB-NCS540-02) is always 26065.

It looks like the wrong label is installed when the redistribute static is configured on SCTLAB-NCS540-02. 

Can you please look for any error message in the log on SCTLAB-NCS540-02.

Regards,

[DanielGutierrez615]

You right Harold.  I do see an errors  look

RP/0/RP0/CPU0:SCTLAB-NCS540-02(config-bgp-vrf-af)#RP/0/RP0/CPU0:Jun 12 15:21:20.781 EDT: fib_mgr[456]: %ROUTING-FIB-4-RETRYDB_NONEMPTY : 1 FIB object(s) have been in IPv4 retry queue for at least 15360 seconds

RP/0/RP0/CPU0:SCTLAB-NCS540-02(config-bgp-vrf-af)#
RP/0/RP0/CPU0:SCTLAB-NCS540-02(config-bgp-vrf-af)#
RP/0/RP0/CPU0:SCTLAB-NCS540-02(config-bgp-vrf-af)#
RP/0/RP0/CPU0:SCTLAB-NCS540-02(config-bgp-vrf-af)#end
RP/0/RP0/CPU0:Jun 12 15:21:42.961 EDT: config[69558]: %MGBL-SYS-5-CONFIG_I : Configured from console by dxg5emergency on vty1 (10.190.0.3)

RP/0/RP0/CPU0:SCTLAB-NCS540-02#show mpls forwarding vrf CORE_14288
Wed Jun 12 15:21:52.365 EDT
RP/0/RP0/CPU0:Jun 12 15:21:52.365 EDT: locald_DLRSC[442]: %SECURITY-LOCALD-6-LOCAL_CMD_ACCT : CLI CMD: "show mpls forwarding vrf CORE_14288" by dxg5emergency from TTY /dev/pts/8 10.190.0.3
Local Outgoing Prefix Outgoing Next Hop Bytes
Label Label or ID Interface Switched
------ ----------- ------------------ ------------ --------------- ------------
26065 Aggregate CORE_14288: Per-VRF Aggr[V] \
CORE_14288 2337313
26085 Unlabelled 216.97.207.0/27[V] BV100 216.97.200.66 0

[MHM Cisco World]

I have idea about issue but still not so clear 

Anyway what make it correct is this

26065 Aggregate CORE_14288: Per-VRF Aggr[V] \
CORE_14288 2337313
26085 Unlabelled 216.97.207.0/27[V] BV100 216.97.200.66 0

 PE send two prefix agg and static route.

So the solution is tune your aggregate prefix to include static route

MHM

[DanielGutierrez615]

Yeah I was looking into that

I was trying this 

router static
vrf CORE_14288
address-family ipv4 unicast
216.97.207.0/27 216.97.200.66 vrflabel 26065
!
!
!

clear mpls label discrepancy 26085

but nothing good

[Harold Ritter · Accepted solution]

Hi @DanielGutierrez615 ,

The first label (26065) is a per vrf label and is allocated by the network statement. The second one (26085) is a per prefix/per CE label and is generated by the redistribute static statement.

It looks like there might be an issue when the per prefix/per CE is used, possibly because of the BVI interface. You could validate that by changing the label mode under the VRF and verify that it works with the redistribute static as well.

router bgp xxx

vrf CORE_14288

address-family ipv4 unicast

label mode per-vrf

This will cause the label to be allocated per-vrf regardless if you use the network statement or redistribute static statement.

Regards,

[DanielGutierrez615]

yes that was the problem., but will you recommend having this setting on  the peering routers as well.  I have 5.

On an article I read it was recommended to use label-allocation-mode per-ce, But should I change it to per-vrf  so they all match . This 5 routers have the full routing table 900k plus routes

[Harold Ritter]

Hi @DanielGutierrez615 ,

You will must probably have the exact same problem with per CE label allocation mode.

Per CE allocation is normally preferred over per prefix, especially when you have the full Internet routing table. But if you run into the same issue on these other routers, you might need to change the allocation mode to per vrf as well.

Regards,

[MHM Cisco World]

First glad we help you here and it in end not bug 

Second I will check tomorrow I am tired today'

But the case is one PE send agg prefix with it label' what make issue exactly I dont know.

Using per-vrf for 900K I think not all device can handle it.

Anyway @Harold Ritter can sure answer you about per-vrf with 900K prefix 

Until tomorrow 

Thanks 

MHM

[Harold Ritter]

Hi @MHM Cisco World ,

> First glad we help you here and it in end not bug 

Not sure whether it is a bug or a restriction, but I don't think it should be this way.

> But the case is one PE send agg prefix with it label' what make issue exactly I dont know.

I think it is a mix of per PE/per CE label and BVI interface adjacency.

> Using per-vrf for 900K I think not all device can handle it.

The per vrf label allocation mode is the most scalable one, as it allocates only one label for all received prefixes. So 900k received routes will generate one label. This obviously comes with a drawback. The egress PE will have to perform an extra lookup (LFIB lookup + FIB lookup) when it receives traffic from remote PE.

Per CE label allocation mode allocates on label per CE (per next hop) and per prefix label allocation mode allocates one label per prefix. So the worst mode for scalability is obviously per prefix. 

Regards,

[MHM Cisco World]

I have a simple view but I need to be sure

Per prefix label is simple label for each prefix

Per vrf label is also simple label for each vrf have rd

Per CE (only ios xr) here we stop what makings PE know that this prefix is for this CE ? That lead me to read more it seem that per ce is use only for ce-pe bgp where the next hop is know (ce) and hence PE know that all prefix will be source from same CE and assign single label.

So per ce not work for static or igp it only for bgp. 

That what I get until now

Thanks 

MHM

[Harold Ritter]

Hi @MHM Cisco World ,

Per CE is not only if you run BGP with the CE. It could be seen as per next hop. So any routing protocol (including static) with a different next hop will assign an additional label. So if you have 100 CE connected to the PE, you will allocate 100 labels. Per CE is the default mode on many platforms as it is the best choice in most cases, as described in the document that @DanielGutierrez615 included.

NCS5500 Routing in VRF Cisco NCS5500 @xrdocs

Per CE is scalable, flexible and does not require an extra lookup.

Regards,

[DanielGutierrez615]

I have to assume that this does not affect EVPN or EVISs right? .  EVIs is a form of vrf but for layer 2 

[Harold Ritter]

Hi @DanielGutierrez615 ,

These label allocation modes are used in the context of L3VPN only.

Regards,

[DanielGutierrez615]

I just want to select something that is not going stop us from using features in the future. that are the disavantages of using per vrf. what features will I loose if I change the config on all access routers to label allocation mode per vrf as supposed to per ce.  On the cisco NCS540 Labe allocation mode per ce is the default option

[Harold Ritter]

Hi @DanielGutierrez615 ,

You need to continue to work with TAC then to find out why the per CE label allocation mode does not work in your configuration. As I mentioned before, I have a feeling that it might be related to the BVI adjacency, but  I could be wrong about that. TAC might also recommend a code upgrade if it is bug related. Please keep us posted on what they say, as this can help other members of the community in the future.

Regards,