C9K Layer3 Switch Performance

[whistleblower14]

Hi,

I‘m planning to use a C9300 as a layer 3 switch... probably someone can tell me which Parameters (switching bandwidth, forwarding rate, etc.) I should pay attention to and understand why, so that I don‘t get any performance problems?

thank you for your help in advance!

[Joseph W. Doherty]

With regard to switching bandwidth, forwarding rate, etc., I believe 9300 is an all ports, concurrently, wire speed switch.

Where you might run into performance issues, is with things like buffer resources, stack performance or switching latency (if ultra low latency is needed).

Also depending on your needs, the switch might not have some features you desire.

[whistleblower14]

---

@Joseph W. Doherty wrote:

Where you might run into performance issues, is with things like buffer resources, stack performance or switching latency (if ultra low latency is needed).

Also depending on your needs, the switch might not have some features you desire.

---

... and that is exactly one of the main fears I`ve! Because I´m currently concerned with Interface output queue drops on the TenGig`s which seem to be caused by microbursts! How far the stacking performance could be impaired is not yet clear to me?!

Regarding the features which`re probably not useable - which functions could that be?

You said in a later post on this topic - "I've used earlier Catalyst 3K switches in both distribution and core roles, but where the load on the switch wasn't very demanding."

What criteria do you use to define how high the load is that the switch role is expected to have? I mean, is the number of VLANs relevant, SVI functions, the connection to potential servers or other Layer 2 switches using load balancing (Etherchannel),... what "matters the most" or how exactly can I come up with the right design?

[Joseph W. Doherty]

"Because I´m currently concerned with Interface output queue drops on the TenGig`s which seem to be caused by microbursts!"

As already noted by other posters, configuration changes, for buffer allocations, can mitigate the impact of microbursts.  However, there are limits to what configuration changes can accomplish if the actual problem is the hardware has inadequate resources.

"How far the stacking performance could be impaired is not yet clear to me?!"

Unlike a switch fabric, which should provide direct port-to-port channels, a stack stack is basically a high bandwidth daisy chain of switches, so it can add additional latency, for inter-stack-member traffic, and might congest (although the "better" stackable switches provide a good bit of stack bandwidth).

"Regarding the features which`re probably not useable - which functions could that be?"

Possible things like (hardware) support of GRE tunnels or even (hardware) support for IPv6 GRE tunnels.  Or "advanced" QoS features, like the 4500 series support for DBL and/or FRED, not supported but any (to my knowledge) stackable switches.  Or, lack of support for some routing protocols.  For example, at my former employer we didn't upgrade 3750s to 3850s because we had a requirement for IPv6 IS-IS, which the 3850s didn't (at least initially) support.

"What criteria do you use to define how high the load is that the switch role is expected to have? . . . what "matters the most" or how exactly can I come up with the right design?"

Volume of traffic expected to transit switch on particular ports.

[Javier Acuña]

The 9300 is a Swicth that is used mostly at the access layer, that is, direct connection to the hosts is a good device but it is not used much at the layer 3 level.

In the same way, the Document of the data Sheest of the device is sent

https://www.cisco.com/c/en/us/products/collateral/switches/catalyst-9300-series-switches/nb-06-cat9300-ser-data-sheet-cte-en.html

Remember to rate your answer

[AdamF1]

That’s not true at all,  many people use the 9300 models for the L3 aspect. Now whether it’s used as a wan router can be debated. 

I use them as a L3 edge devices and combination of edge and wan for remote p2p sites. 

[balaji.bandi]

adding to others, it all depends where you want to use - in most cases if this is an access switch, you get better performance and value paid off. it got a new ASIC of cisco so it manages well. there are many reports shared by the users and independent tests - you can google instead of looking cisco's point of view...look other test results.

[Joseph W. Doherty]

". . . in most cases if this is an access switch . . ."

I just want to also add, the switch's L3 performance is also, generally, like its L2 performance.  It's the issue of using this model of switch, in a more "demanding" role, which often entails routing, where you're likely to more often encounter bursts of traffic that queue up on an egress port.  This is where buffer resources come into play, and where lack of them, cause issues.

I've used earlier Catalyst 3K switches in both distribution and core roles, but where the load on the switch wasn't very demanding.  In such roles, or for the access/edge roles (the latter often not needing L3 routing, but the stacking is nice) they generally work fine.

For more demanding usage, often you'll want to use the chassis based L3 switches, which often have additional hardware resources and/or features not found within the "smaller" Catalysts.

". . . and independent tests . . ."

Also note, such tests often show (accurately) all ports running concurrently, often at full rate, but a) that's very, very rarely seen in the real world and b) multiple ports and/or a higher bandwidth port sending to a single egress port is more often the norm, and again, this is where buffer resources may be critical.

[Scott Hodgdon]

Just throwing in my 2 cents on this one ...

It has been quite some time since L3 switching has had any kind of performance impact on Catalyst switching products. We have been using hardware-based CEF switching in all of our Catalyst products since I joined Cisco 20 years ago (disclaimer: most of that time I worked on Cat 6K, but I have also worked on Cat 9K since before they were launched in 2017). Moving the RIB into the FIB is what took us form CPU-based performance to ASIC-based performance.

As has been noted, there are other factors to consider like buffering, but these are independent of L2 vs L3. I have seen just as many burstiness  / buffer issues in L2 environments as L3 environments.

Also consider that Cisco's SD-Access enterprise fabric solution is an L3 transport from Access layer to core within a fabric site, and we would not have created such a solution if L3 forwarding had any kind of impact on performance.

Hope that sheds some light.

Cheers,
Scott Hodgdon

Senior Technical Marketing Engineer

Enterprise Networking and Cloud Group

[whistleblower14]

---

@Scott Hodgdon wrote:

As has been noted, there are other factors to consider like buffering, but these are independent of L2 vs L3. I have seen just as many burstiness  / buffer issues in L2 environments as L3 environments.

---

Hi Scott,

maybe my question is a little bit off-topic but maybe you can help me understand how the buffer-issues can be fixed? As I mentioned earlier, I´m currently dealing with problems which look like are due to microbursts...  I don`t understand how that Bursts and Qutput-Queue-Drops can occur when I´m using on each side of the link a TenGig Interface (2x Switch Interfaces are bundled in an L2 LACP Channel...)? I always thought that an interface has to buffer when on the other side an interface with lower bandwidth is connected and an oversubscription is the case?!

dfsdf

[balaji.bandi]

this will help you to troubleshoot for the buffers, that can be tweaked based on the drops.

https://www.cisco.com/c/en/us/support/docs/switches/catalyst-9300-switch/216236-troubleshoot-output-drops-on-catalyst-90.html

[Scott Hodgdon]

@whistleblower14 ,

As Balaji stated, you can adjust the depths of the queues that make up the buffer. Each buffer is broken into queues (priority, high, low, scavenger, etc), and each queue has a certain size (depth). As you can imagine , the queues will be serviced based on their priority, weighting, etc. 

I would highly recommend having a look at the Cisco Live session DGTL-BRKCRS-2501 : Campus QoS Design Simplified for a good understanding of QoS and specifically QoS capabilities on various platforms. If you do not have a ciscolive.com account, you can crete one and view the session videos and download the slides in PDF format. 

Cheers,
Scott Hodgdon

Senior Technical Marketing Engineer

Enterprise Networking and Cloud Group

[Joseph W. Doherty]

" I don`t understand how that Bursts and Qutput-Queue-Drops can occur when I´m using on each side of the link a TenGig Interface (2x Switch Interfaces are bundled in an L2 LACP Channel...)? I always thought that an interface has to buffer when on the other side an interface with lower bandwidth is connected and an oversubscription is the case?!"

Simply, there's a finite amount of buffers.  If you exceed it, there no place to store frames/packets, and so, they are dropped.

Fortunately, often buffers limits are "logically" exceeded, and when that happens, changing those limits, mitigates the drops.  However, if actual "physical" buffer limits are exceeded, you're going to have drops.

If I remember correctly, the 2960 series (at least the original models) only had 2K of buffers for the whole switch.  The 3750 series had 4K of buffers for each bank of 24 copper ports and 4K for the uplink ports.  (BTW, because of the latter, it could be advantageous to have a really busy server connected to an "uplink" port.)  The point here, though, this buffer resource difference was why, in some cases, a more expensive 3750 was "better" than a 2960 if either was only using 48 copper ports, strictly as a L2 switch.