VXLAN EVPN Advertise PIP

[j.a.m.e.s]

Dear All,

The EVPN fabric I have built has a per-VRF iBGP peering between the leaf switches (which are VPC peers). This is working well, but I would like to eliminate the overhead of using an iBGP peering every time I create a new VRF.

When I built the fabric, the advertise-pip feature was not available on the N9K-93180YC-EX, but since moving to 7.0.3I7(4), it's possible. Could anyone tell me:

1. Will advertise-pip make the per-VRF peering obsolete? I assume it will because any type 5 routes (such as SVIs or routes learned downstream of the leaf) will be advertised from the NVE's primary IP. Only EVPN host routes connected to the vpc-pair (and therefore reachable via layer 2 from either peer) would still be advertised via the secondary IP (VIP).

2. How disruptive would migrating to advertise-pip be? Presumably it would be fairly seemless if I removed the per-VRF L3 peering at the end of the change?

Thanks in advance for any advice.

Regards

James.

[brdewal2 · Accepted solution]

Hi James,

Your understanding is correct. Any unique routes on either vPC peer will be advertised only with the PIP of the NVE loopback.

For example I have a leaf pair with unique loopbacks 10.0.0.145 & .146 with a shared secondary IP of .204.

On the .145 leaf I created an SVI with 10.150.3.254/24 and I have attached to both of them a host 10.150.1.36/24.

I turned advertise-pip on and you can see the routes here from a remote leaf:

10.150.3.0/24, ubest/mbest: 1/0
*via 10.0.0.145%default, [200/0], 00:00:16, bgp-65502, internal, tag 65502 (evpn) segid: 10500 tunnelid: 0xa000091 encap: VXLAN

10.150.1.36/32, ubest/mbest: 1/0
*via 10.0.0.204%default, [200/0], 00:00:34, bgp-65502, internal, tag 65502 (evpn) segid: 10500 tunnelid: 0xa0000cc encap: VXLAN

As for impact, I believe you shouldn't see any traffic loss if you remove the iBGP peerings after turning on advertise-ip as you suggested.

[j.a.m.e.s]

Just rolled out advertise-pip and had some nasty surprises. There was widespread packet loss after configuring 'advertise-pip' and 'virtual-router rmac'.

After spending time with tac we could see pings ariving on the endhost and the replies being sent back up towards the spine, but they think we hit a bug.

Rollback was also not much fun because I had to amend the system nve infra-vlans on the 93108TC switches which needed a reboot (I was using an SVI for the per-vlan leaf crosslinks).

I'll post again if we can find a solution.

Regards

James

[brdewal2]

Hi James,

I was actually just looking over your case when I got notified you replied to this.

I'm interested myself to see what cause is as this obviously shouldn't have caused any issues.

Hopefully the rest of your weekend goes a little better :)

[j.a.m.e.s]

For anyone reading this thread I thought I would follow up to explain why we had a catastrophe with the advertise-pip. I'm really grateful to the TAC engineers who helped find this as it's been a mystery for a while.

1. As a reminder, switching on advertise-pip means that type-5 routes (such as 0.0.0.0/0 from a border leaf) will be announced using the vpc VTEP's PIP as the next hop, rather than the VIP.
2. Most spine-leaf builds will have an underlay peering between vpc nodes (in my case this is iBGP, but I suspect most use OSPF). As a consequence of this peering, both VPC nodes learn each other's PIP across the peering and they then advertise it towards the spine. The spine therefore has ECMP to the PIPs and will deliver the traffic destined for the PIP to either vpc node.
3. To be clear, if a leaf wanted to send traffic to BorderLeaf A's PIP, the spine might hash it to BorderLeaf B, who would try to deliver it across the MCT to A
4. When traffic followed the path mentioned in step 3, it was CPU switched on node A and with hundreds of servers in the fabric the CoPP policy was overwhelmed. This explains why we saw around 50% packet loss.

The solution was to ensure that VPC node A does not advertise node B's pip to the spine and conversely node B must not advertise node A's to the spine. Fail to do this and the N9k platform will CPU-switch 50% of your VXLAN traffic (tested on a C9332PQ and C9336FX2). The solution is very easy with a route map and BGP as your underlay but I would be less confident about it with an OSPF underlay.

According to TAC all this is by design which is why I'm surprised the advertise-pip documentation doesn't mention it. Hopefully this will help others avoid a similar catastrophe!

Regards

James.

[j.a.m.e.s · Accepted solution]

In case it helps others, we eventually heard about bug CSCvk59796. I understand a fix has been applied from 7.0.3I7(5) release. Following this release, I believe it would be acceptable to advertise the PIP from both vpc nodes.