Hello,
I recently deployed quite a lot of traffic over VXLAN bridged spine and leaf topologies. As a next step, multiple challenges to load-balance this traffic arose especially in cases of Nx10G uplinks.
Long story short, I have 2 switches on an international topology:
Route A is direct routerport between both switches: A <---- 100G wavelength----> B
Route B is an indirect SVI-enabled underlay due to the fact the route is travelling through multiple VXLAN-incapable switches with standard switchport trunks. Switch A (vlan interface) <----- (Multiple switches with Nx10G switchports in port-channel) -------> B (vlan interface)
A challenge did occur in the second case due to the fact we are trying to balance the underlay traffic which is one to one IP etc. So I had the underlay traffic only passing through one of the ports in the port-channels. This was actually a logical end result so I enabled multiple VLANs and VLAN interfaces over the switched network in a hope to have each different VLAN balanced over different port in the port channels. I even generated random MAC addresses for each VLAN interface in order to make it random enough.
Second challenge is when we have to balance multicast traffic. We all know that the BUM traffic is routed through the respective multicast groups as set in the nve interface, however I have a circuit which is carrying approx. 10G of multicast traffic - hence, absolutely no balance over all port-channels due to inability to split one single multicast groups across multiple ports in the spine layer on the way from one leaf to the other.
Third challenge is actually something I raised here recently and it sounds like a bug but I don't have any TAC access in order to have it investigated. If we have a traffic which is ingressing from a VXLAN-enabled VLAN and then has to egress through switchports in a port-channel, we have quite uneven load-balance on egress towards the switchports. I suspect that the switches don't balance it after decapsulation but somehow save all hashes as they were before decapsulation (one source IP of the neighbor nve peer etc).
I read actually about an option to enable ECMP on host routes but not sure if this would be a perfect practice. This is usually disabled by default and I suspect Cisco done so due to not recommending its usage, but hopefully I would be wrong.
Any opinions would be appreciated.
Thank you.
