BGP VXLAN EVPN: Maximum underlay multicast groups for N9K-C9396PX/ N3K-C3164Q

[ss1]

Hello,

I'm currently reading the verified scalability guide of N9K-C9396PX and I see a maximum number of Underlay multicast groups for an VXLAN BGP EVPN topology: 128 

I would appreciate if anybody could help me understand this limit better. My topology is a spine and leaf architecture. 12 pieces Nexus 9396 are used as leafs, they are all uplinked to two redundant N3K-C3164Q. The 3164Q are used as a spine layer, so they have routerports towards every Nexus 9396 switch. 
In this design the C3164Q have no vn-segments configured and also no member vni's on interface nve1. However they have multicast routing enabled in order to route multicast groups from leaf X to leaf Y so that switching is done. In this case the C3164Q does not decap/encap the BUM at the multicast groups, however they are seen on Layer 3 / PIM level in order to reach from leaf X to leaf Y.

My question is, does this limit only apply when multicast groups are defined as a group for a vni member at interface nve1, or does it also apply in case they are only routed from leaf X to leaf Y ?

Also, I have to admit that I just powered on our laboratory switch N9K-C9396PX and tried to populate the nve1 with approx. 160 vni's each having a different multicast group. Also created the respective vn-segments. The switch never complained that I reached some limit, neither syslogged some error nor refused to accept the configuration. My version is 7.0.3.I7.8. Not sure if I misunderstood something regarding this scalability entry?

Thank you.

[Sergiu.Daniluk]

Hi @ss1 

What is written in the scalability guide is the tested and validated scalability of that single feature (unidimensional scalability), the limit where the system still functions as expected running that single feature. Most of the times this is a soft limit (meaning you can configure more and system will not complain through an error or warning message), BUT you expose yourself to hardware or software becoming unstable or not being capable to respond to changes. What's important to know is if the hardware is capable of a higher scale, future software releases might increase this verified maximum limit. 

Now coming back to your question, the "Underlay multicast groups for an VXLAN BGP EVPN topology" most likely refer to the underlay switches, but it can include as well the VTEPs (if all your VNIs are configured on one VTEP).

I would suggest to lower the number of mcast groups and duplicate the group mapping to VN-segments which have common VTEPs. This way to prevent any potential issues. Remember, the maximum scalability values might be lower with multiple features enabled.

Stay safe,

Sergiu

[ss1]

Hi

Thanks for your reply.
Initially I thought about configuring each VLAN into a separate multicast group. However, it's obvious that I need to rework the concept in order to stay within the recommended verified scalability value. 

We have 12 devices which have one nve1 interface each - Nexus does not support multiple nve interfaces. What do you mean by "the underlay switches" ? Sorry please help me to understand better.
A VXLAN may travel through: host -> Leaf (1) -> Spine -> Leaf (2) -> host
In this case I have two VTEPs for the VXLAN on Leaf (1) and Leaf (2) but switch "Spine" is only routing the respective multicast group between each other's routerports. I do not understand if the number of "underlay multicast groups" are those configured on the nve1 interface or all those who are routed on the spine, even without having a VTEP configured over there for the respective VLAN. 

Thank you.

[Sergiu.Daniluk]

Hello,

By underlay switches, I was referring to the spine switches. But if your VTEPs are configured with all the existing VN-segments from your VXLAN fabric, then the number of multicast groups running through spines and and the ones configured on VTEPs will be the same.

Anyway, I think the scalability guide refers to the number of multicast groups configured on a VTEP/NVE. 

Regards,

Sergiu