BUM issues on a VPC-enabled spine layer + anycast-rp

[ss1]

Hello,

Our Datacenter deploys a topology which is almost alike a Spine and Leaf VXLAN-enabled design. The Spines are N3K-C3164Q and leafs are N9K-C9396PX. Routerports connect the 9396 to the VXLAN fabric. 

There is an anycast-rp configured on the C3164Q in order to keep the network redundant etc.

So far, so good. The issue is that we still have some switchports on the spine layer due to some hosts which are not migrated yet to the leaf switches and consequently we still have VPC and VTEPs running on the spines themselves. I have detected the following scenario, a VPC-enabled spine together with anycast-rp and VTEPs on both switches of the spine results in BUM traffic being duplicated and received by the hosts twice. This morning I removed the anycast-rp settings and only left one switch to act as a RP. As a result all BUM issues I was seeing till now immediately disappeared, I had some hosts missing arps in point to multipoint scenarios, or some other hosts duplicating etc.

Of course this would mean I have to clean up the network design by removing hosts from the spine but you know it's not always that easy and fast to migrate a switched network to a routed network due to all necessary maintenances, customers etc. However I'm curious if somebody ever deployed a scenario like mine - an anycast-rp + VTEPs on a VPC-enabled spine layer => leading to BUM issues like duplicated BUM traffic, or blackholed BUM traffic or something alike?

Most of our switches are using NX-OS 7.0.3.I7.9 and NX-OS 7.0.3.I7.7.

Thank you.

[ss1]

Hello,

It seems we progressed with this issue by procuring another set of switches to deploy as a real spine layer - our new 9364C are in place for that purpose since yesterday. Let me show you a diagram. 

 

The anycast-rp is moved to the 9364C and the other 3164Q and 9396PX are leafs on routing perspective. All gray lines are routerports between the nodes - each port is a different point to point /30 in one and same OSPF area. 

The blue ellipse drawings represent different VPC domains where applicable, due to presence of switched ports towards hosts and other VXLAN-incapable switches as illustrated. 

Note the 3164 in the middle. These switches are both VTEPs and routers towards the 9396 switches below them. So we have a VTEP for one and same VLAN configured on both pair of switches. The problem that left to resolve is such that the traffic flow is not reliable between all three hosts in a point to multi-point scenario of the same vn-segment. For some reason we are blackholing BUM traffic between the hosts.

In order to mitigate this we need to do a point to point topology of that same vn-segment. I mean, all issues are gone as soon as we rework the configuration so that the 3164 in the middle start switching the traffic rather than routing towards the 9396 below them. And the opposite - if we start switching from the same 9396 backwards to the 3164, the issues are also gone. 

It seems we are having some huge BUM blackholing issues if a VPC-enabled node is acting as a spine and a router at the same time. Any ideas will be appreciated.

NX-OS version is 7.0.3.I7.9 on the 9396PX and 3164. For 9364C we use 9.3.6. 

Thank you.

[melchib]

Have you considered using ingress replication for BUM traffic to see if it resolves your issue?  This can be done on a per VNI basis - so you could test with just a single VNI at first.

[ss1]

Hi

I never tried. So far only PIM ASM. 

Thank you.

[ss1]

By the way I think it's good to mention we are enabling BFD on all routerports over the backbone and also we are disabling ip redirects. So the configuration of a port looks like:

ip address x.x.x.x/30
no ip redirects
no ipv6 redirects
ip router ospf 1 area 0.0.0.0
ip ospf bfd
ip pim sparse-mode
mtu 9216
no shutdown

Do you think I can expect some issues because of disabled ip redirects and enabled BFD?