cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
2763
Views
15
Helpful
15
Replies

Catalyst 9300 L3 VLAN Overloaded

CMack6
Level 1
Level 1

Hello everyone!  Thank you in advance for your assistance in this matter.

I have a Catalyst 9300 running as my CORE switch in a CCTV environment.  I have been seeing an unusual large amount of TCP re-transmission errors in Wireshark captures recently.  So much so that my video is starting to be effected by it.  When I'm looking at the captures, there is 1 commonality with all the errors, the VLAN which has my servers and clients on it (VLAN 2).  Devices from VLAN 2 are either in the Source or Destination columns.  When I look at the L3 interface on the device, VLAN 2 has a txload of 255/255 and shows an available bandwidth of just under 1Gbps.  

I've looked over various posts on this website and some say that since the VLAN is a virtual interface that it is not limited to the 1Gbps bandwidth and others say that it is and that one needs to increase the allowable bandwidth.  Can anyone offer any guidance on this?  I've calculated the txloads from all the interfaces on VLAN 2 and I am at a steady ~2Gbps passing through the CORE (most of it being the storage arrays).  

I hope this all makes some sense.  Thank you!

1 Accepted Solution

Accepted Solutions

balaji.bandi
Hall of Fame
Hall of Fame

VLAN interface is just virtual you can set the bandwidth if you like example :

config-if)#bandwidth ?
<1-200000000> Bandwidth in kilobits
inherit Specify how bandwidth is inherited
qos-reference Reference bandwidth for QOS

Look at some QoS features you may consider here :

https://www.cisco.com/c/en/us/products/collateral/switches/catalyst-9000/white-paper-c11-742388.html

how about when  you look at off peak time is the CCTV video ok ? (i mean when the Server traffic is Low )

Can you post some interface utilisation and interface config.

 

 

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

View solution in original post

15 Replies 15

balaji.bandi
Hall of Fame
Hall of Fame

VLAN interface is just virtual you can set the bandwidth if you like example :

config-if)#bandwidth ?
<1-200000000> Bandwidth in kilobits
inherit Specify how bandwidth is inherited
qos-reference Reference bandwidth for QOS

Look at some QoS features you may consider here :

https://www.cisco.com/c/en/us/products/collateral/switches/catalyst-9000/white-paper-c11-742388.html

how about when  you look at off peak time is the CCTV video ok ? (i mean when the Server traffic is Low )

Can you post some interface utilisation and interface config.

 

 

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

I pulled a new Wireshark capture prior to increasing the bandwidth to 10Gbps on the SVI (VLAN 2) and it was filled with TCP errors.  Once I made the change, I pulled another capture and there was 0 TCP errors in a 15m pull.  I'm going to monitor to ensure there isn't another issue, but at the moment I'm confident in the solution.  

Thank you everyone for the assistance!

Interesting!!!

Changing the bandwidth value on the SVI should not have impacted actual data transmission (e.g. impacting TCP retransmissions).

Interface bandwidth settings are used for "informing" other "things" of expected bandwidth.  For example, when you mention loading showed 255/255, that would be expected if bandwidth is set below actual available bandwidth and actual throughput meets or exceeds the bandwidth configuration.

So, if you could/would, let us know in about a week or so, all is still good.  Or, if not, don't be embarrassed to let us know changing the SVI's bandwidth setting did not actually cure your problem.

Absolutely I will!  I'm still relatively new to this field compared to many others on this forum so I'm not too embarrassed to admit when I have it wrong or don't understand something.  I just feel bad for all the smart folks that have to put up with my learning style   I'm all self taught and OJT as the solo network guy so there is plenty I'm still learning!

That's what I was rather confused about from reading other posts about similar issues, which is why I posted this topic.  Once I changed the bandwidth, the interface showed a bandwidth of 10Gbps and the txload dropped to 52/255.  

 

Nothing to be embarrassed about.  Most of us don't start out as "experts".    Learning is also often a lifetime activity.

"Once I changed the bandwidth, the interface showed a bandwidth of 10Gbps and the txload dropped to 52/255."

Yea, that too isn't unexpected.  Again, it's seeing TCP re-transmits happening before the bandwidth setting change, and not happening after the change, that's "curious".

BTW, I'm almost, but not quite, a full/true expert, assuming you go by the definition that "an expert learns more and more about less and less until they know everything about nothing".  ; ) 

Alright, so here is my update.  I'm still seeing TCP errors in Wireshark, although they are considerably less than before.  Originally I was seeing TCP errors locally coming from cameras destined for the storage and between different servers.  Now I'm not seeing any of that anymore (so not sure why that stopped).  All I am currently seeing is TCP errors on devices that are transiting the uplinks to other buildings.  My uplinks are very small - though I do have "larger" ones coming.  On the uplink's port interface, I have a considerable amount of Total output drops.  My initial thought, after this post and some research is that I have not changed the bandwidth on the uplink interface.  For example, I have a 300Mbps uplink from the ISP provider running on a 1Gbps port.  I'm wondering if I am trying to transmit at that 1Gbps speed but dropping all the packets on the interface because there is congestion since the pipe can only take 300Mbps.  So, in theory, if I were to adjust the bandwidth on the interface to 300Mbps the routing protocol would calculate appropriately and only transmit up to 300Mbps on that interface?  Am I on the right track?

Me and @Joseph W. Doherty know that and wait until point that TCP issue appear again. 
from my view check the MTU and TCP MSS config, this can lead to tcp retrasnmission. 

(replying on phone, more details later)

Re: output drops, ah, that's more what I would expect for your TCP re-transmits.

Re: gig port with 300 Mbps CIR, likely more of a problem than port drops "show".

If an interface show drops, especially if they are, more-or-less rapidly, increasing, you have enough oversubscription congestion that's often adverse to application performance.

Further, in a case like yours, if you have upstream bandwidth (300 Mbps?) less than port bandwidth (gig?), drops can be happening there (in path), which are not directly visible to you (as they wouldn't show on your interface stats).  (In fact, often you can have drops within the path while your ports show no drops at all.)

Normally, I recommend (if equipment supports) shaping your port for the path's bandwidth.  This should preclude your path from dropping traffic, but generally increases your drops at your port.  (By moving where the drops are happening, from path to your port, you get visibility into total extent of the drops, and can, sometimes, mitigate the congestion drops at the port.)

What's the device and IOS version hosting the uplink with the drops?

Regarding changing the bandwidth statement on your port, most of the time, that has no actual impact on traffic rate.

Thanks everyone again for the help!

 

MHM Cisco World - I just reviewed a recent Show Tech that I have from the C9300-48T and the MTU is the same on all physical interfaces.  The only ones that are different are loopback0, which is configured as a static PIM RP and Tunnels0 & 1 which are showing PIM.  Regarding the TCP MSS config, I’m not seeing that in my Show Tech so I will need to do a quick look up to determine how to identify that.

 

Joseph – That makes sense to me!  The C9400 that I connect to for the uplink is provided by the corporate IT department so that is all out of my control.  Currently I have just a VLAN on their network which routes the security network to other buildings.  The new uplinks will be VPNs which I’m looking forward to!  Am I able to drop a Gig port down to 300Mbps?  I’m under the assumption that it’s 10/100/1000 for Gig ports?  I’ve never delt with a speed like that before.  I’m operating a C9300-48T on IOS 17.6.4

 

balaji.bandi – I am going to the building now to clear the interface errors and get a updated Show Tech. 

@CMack6 you may be luck.  From reading a C9300's QoS documentation, shaping appears to be supported.

The Catalyst 9Ks have interesting QoS support (supporting documentation, also better than it was years ago).

What I'm going to suggest, provides better visibility into your uplink's 300 Mbps, and perhaps provides more buffers for bursts.  Note, without my working with your device, interactively, it would take a long, long (long) time to try to find fully effective QoS mitigation, if even possible.  (I.e. forums don't allow fast turn around on analysis of what your device is doing, and improvement, if any, with QoS changes.)

Anyway, studying your device's QoS features, the following may help:

qos share-buffer
qos queue-softmax-multiplier 1200

!NB: unsure where the shaped packets will be queued, at parent or child or both.
!What I hope the following does is define only one active queue for your
!interface (well possibly two, parent and child), and maximize buffer allocations
!for that/those queue/queues, using both "hard" and "soft" buffers.
!If you issues is sustained congestion, buffer tuning alone, won't make
!much of a difference.

policy-map ParentShaper
class class-default
shape average 300000000 !may need to set up to about 15% less if shaper doesn't also count L2 overhead
priority level 2
queue-buffers ratio 100
service-policy ChildQoS

policy-map ChildQoS
class class-default
priority level 2
queue-buffers ratio 100

interface gig # !the "uplink"
service-policy output ParentShaper

Can you post the output here, also clear the interface errors and start monitoring how quick they increasing.

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

It this very interesting to see your outcome. by the way is the problem resolved? i belive you did this 2 days back...the results still stable and optimal  ?

 

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

are all SVI have same MTU ? 
are all VLAN L2 port have same MTU ?