Hi,
This is likely a microburst thing (as others have mentioned), you say you don't see over 28 mbps but this measure is done over what timeframe? 60 seconds? Microbursts are short lived and quite difficult to detect unless you measure transference stats over small periods (ie. 1 second).
What shows if you issue this:
show interface IF_NAME counters errors
If you have a lot of xmit errors this could prove it's a microburst problem or the ports being overrun by speeds conversions
Cisco's documentation https://www.cisco.com/c/en/us/support/docs/switches/catalyst-6500-series-switches/12027-53.html
"""
errors. This is an indication that the internal send (Tx) buffer is full.
Common Causes: A common cause of Xmit-Err can be traffic from a high bandwidth link that is switched to a lower bandwidth link, or traffic from multiple inbound links that are switched to a single outbound link. For example, if a large amount of bursty traffic comes in on a gigabit interface and is switched out to a 100Mbps interface, this can cause Xmit-Err to increment on the 100Mbps interface. This is because the output buffer of the interface is overwhelmed by the excess traffic due to the speed mismatch between the inbound and outbound bandwidths.
"""
It's for the 6500 but I believe this is the same over all the catalyst line.
Also I'd start checking closely the switch for output drops by first reseting the counters:
clear counter IF_NAME
And then checking every second if this increases, if you see something like:
1.- No drops at all
2.- Sudden drops (a lot)
3.- No drops at all
4.- No drops at all
5.- Sudden drops (a lot)
Would also point to a microburst thing. If this is, you either increase the capacity of that link or increase the buffer size.
I had to troubleshoot an issue like this over a 3850 where the peak rate over a single 1GE interface never went over 300 Mbps, but a lot of drops were seen, turns out this interface was used as the uplink for 4 other devices all using 1 GE port upon further inspection we saw it was due to microbursting.
We ended using this command to increase the buffer allocation and the drops ended:
qos queue-softmax-multiplier 1200
here's the documentation for this: https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3850/software/release/3e/qos/command_reference/b_qos_3e_3850_cr/b_qos_3e_3850_cr_chapter_010.html#wp4043987550
Although I'm not sure whether this applies to the 3650 as well
Now if the issue isn't microburst related then this could probably be a hardware fault in which case you should open a TAC case.
HTH
Please remember to rate useful posts
