Switching half a terabit over 600x10G = What's the best switch?

[ss1]

Hello,

I am currently trying to figure out what could be the best switching solution for a service provider network. That's basically a density briefly described as ~600 10gig customer ports, ~160 1gig ports. MAC address count is less than 16000. Must support approx. 10 000 IGMP entries or more.

Currently we are performing all this via a couple of Nexus N3K-C3064-10GE, N3K-C3048-1GE. They are connected with each other directly via the QSFP ports. I was thinking about using two N3K-C3164-40GE in order to deploy a better aggregation of all that traffic in something like a star topology, however I came across the main limitation of N3K and possibly other Nexus series. I must use PVRST in order to control all the vlans over the network, MST is no option. It seems that neither N3K-C3164, nor N3K-C3064 can satisfy the need to run more than 507 VLANs over a single switch. That's a bottleneck preventing me to use N3K-C3164 for distribution between many N3K-C3064 linked to it via QSFP LACPs.

I would be happy to look at your suggestions regarding what's the right product to use for my distribution layer. It must have reasonable density of 40G and/or 100G with a possible breakout down to 4x10G, wire-rate switching performance without any oversubscribed interfaces. I can also look at a recommendation to migrate my whole network of C3064 to a new solution (chassis/FEX) in order to enable 300+ 10Gig ports and a couple of 40G/100G in a single platform. However, any possibility to run PVST with 4000 vlans over C3164 is the best possible case since I already have them.  

Looking forward to any suggestions.

Thank you.

[Rich Uline · Accepted solution]

Ss1,

As I understand it, you are looking at switches (possibly restricting your search to the Nexus line) that will switch 6Tbps at line rate. The host-facing ports are a combination of 1Gbps (approximately 160) and 10Gbps (approximately 600) You also have fewer than 16k MAC addresses and10k+ IGMP entries. Finally, your main limitation with the noted offerings is that you require more than 507 RPVST instances. You would also like to utilize the existing hardware as much as possible.

Addressing the main issue - the next minimal step up from the 3164 is the 3232, however, that switch only supports 512 RPVST instances. The next meaningful step up is the 9200/9300 series which supports 3967 RPVST instances. Following the spine and leaf topology, it may be possible to replace your spine with a few 9336PQ or similar. You could have up to 3967 RPVST instances in the spine while limiting the per-leaf RPVST instances to 507 or less. If you did this, then you could utilize the existing 3164 switches.

[ss1]

In this case, I can use either Nexus 9236C or Nexus 9272Q in place of N3K-3164. Do these two support up to 3967 RPVST instances? I will connect existing 3064 to them and just plan the topology better, in order not to have so many vlans over a single 3064. If true, I can successfully use a 9272Q as an aggregation of the whole network over it.

On the other hand, do you know a solution in which I can completely disable the STP in a C3164, so that I can enable all vlans in it? The network would work perfect if I do so. STP must not be filtered through the uplink port, so bpdu filter wouldn't do the job. If a 3064 can see other 3064 bpdu packets it will be more than enough. That way the 3164 does have real chances to be a traffic distribution solution in my case.

[Rich Uline]

Ss1,

As far as I can tell, the whole 9200/9300 series supports 3967 RPVST instances.

Regarding disabling STP; you shouldn't unless you remove all layer 2 loops. The only two ways I know to accomplish that is to flatten the network or to convert all dot1q trunks to layer 3 interfaces. It's possible to retain redundancy in a flattened network by leveraging technology such as VSS with MEC, but those features are not available in the Nexus line to my knowledge. The Nexus line does support routing, though, if your device has an Enterprise Services license.