PortChannel Output Drops

[CGS Service Desk]

Hi everyone,

I'm having some trouble with output drops that maybe someone would be kind enough to check over for me.

The map for this section is here https://imgur.com/a/baPYoxb

I have 2x 2960 connected to 2x Nexus 3500 via 4x Fibre, as I make adjustments the port dealing with the majority of traffic on the 2960S will always be the one dropping output (changes between 1/0/25 and 26 depending on which link is up). As you'll see on the interfaces these never seem to hit high traffic but I would expect 4 connections to be plenty for it to handle.

I will attach the configuration and interface information in a text file.

Thank you in advance for your puzzle solving.

[Leo Laohoo]

Post the complete output to the following commands: 

sh controll e G1/0/25
sh controll e G2/0/25

I am concerned that both physical link have CRC errors.

[CGS Service Desk]

Hi Leo,

Thank you for the response, I'll add those commands below. I originally thought it was L1 after checking configuration which is why the port channel now has 4 interfaces. I doubled up to remove connections but found interfaces that had no drops would start dropping traffic when I removed the faulty interface.

I have a ticket with support but it has been over a month now and hasn't made any progress, the technician suggested there was too much traffic and has today responded asking to debug the buffers. I'm about to check this but realistically there shouldn't be too much traffic for these to handle and the drops continued while the network was under very low use.

SW12176#sh controllers e gi1/0/25

Transmit GigabitEthernet1/0/25 Receive
524637963 Bytes 3092007329 Bytes
2688943677 Unicast frames 2698055409 Unicast frames
6006409 Multicast frames 3531733005 Multicast frames
97816390 Broadcast frames 327640618 Broadcast frames
0 Too old frames 2091406202 Unicast bytes
0 Deferred frames 432937073 Multicast bytes
0 MTU exceeded frames 131332607 Broadcast bytes
0 1 collision frames 0 Alignment errors
0 2 collision frames 0 FCS errors
0 3 collision frames 26896 Oversize frames
0 4 collision frames 0 Undersize frames
0 5 collision frames 0 Collision fragments
0 6 collision frames
0 7 collision frames 7196184 Minimum size frames
0 8 collision frames 1377148303 65 to 127 byte frames
0 9 collision frames 3229501382 128 to 255 byte frames
0 10 collision frames 3122848533 256 to 511 byte frames
0 11 collision frames 3952976280 512 to 1023 byte frames
0 12 collision frames 2948299868 1024 to 1518 byte frames
0 13 collision frames 0 Overrun frames
0 14 collision frames 0 Pause frames
0 15 collision frames
0 Excessive collisions 0 Symbol error frames
0 Late collisions 26896 Invalid frames, too large
0 VLAN discard frames 509393075 Valid frames, too large
0 Excess defer frames 0 Invalid frames, too small
538026 64 byte frames 0 Valid frames, too small
3729873618 127 byte frames
4051526399 255 byte frames 0 Too old frames
373628062 511 byte frames 1 Valid oversize frames
4123981324 1023 byte frames 0 System FCS error frames
2737975798 1518 byte frames 0 RxPortFifoFull drop frame
660145137 Too large frames
0 Good (1 coll) frames
0 Good (>1 coll) frames

SW12176#sh controllers e gi1/0/26

Transmit GigabitEthernet1/0/26 Receive
3087450615 Bytes 2389446268 Bytes
3230833665 Unicast frames 3555161550 Unicast frames
316464 Multicast frames 51520611 Multicast frames
5350306 Broadcast frames 21768400 Broadcast frames
0 Too old frames 1787957914 Unicast bytes
0 Deferred frames 1645895290 Multicast bytes
0 MTU exceeded frames 1623680842 Broadcast bytes
0 1 collision frames 0 Alignment errors
0 2 collision frames 594 FCS errors
0 3 collision frames 0 Oversize frames
0 4 collision frames 0 Undersize frames
0 5 collision frames 3 Collision fragments
0 6 collision frames
0 7 collision frames 265432 Minimum size frames
0 8 collision frames 1166558655 65 to 127 byte frames
0 9 collision frames 1352137600 128 to 255 byte frames
0 10 collision frames 177538705 256 to 511 byte frames
0 11 collision frames 100807445 512 to 1023 byte frames
0 12 collision frames 820285985 1024 to 1518 byte frames
0 13 collision frames 0 Overrun frames
0 14 collision frames 0 Pause frames
0 15 collision frames
0 Excessive collisions 0 Symbol error frames
0 Late collisions 6 Invalid frames, too large
0 VLAN discard frames 10857327 Valid frames, too large
0 Excess defer frames 3 Invalid frames, too small
74780 64 byte frames 0 Valid frames, too small
1521594336 127 byte frames
2080737342 255 byte frames 0 Too old frames
952610399 511 byte frames 0 Valid oversize frames
150033531 1023 byte frames 0 System FCS error frames
1279797097 1518 byte frames 0 RxPortFifoFull drop frame
1546620246 Too large frames
0 Good (1 coll) frames
0 Good (>1 coll) frames

SW12176#sh controllers e gi2/0/25

Transmit GigabitEthernet2/0/25 Receive
1188497238 Bytes 3906528765 Bytes
3959669663 Unicast frames 1289786590 Unicast frames
11751273 Multicast frames 788104439 Multicast frames
86538302 Broadcast frames 246472331 Broadcast frames
0 Too old frames 2714623627 Unicast bytes
0 Deferred frames 4126883854 Multicast bytes
0 MTU exceeded frames 674541886 Broadcast bytes
0 1 collision frames 0 Alignment errors
0 2 collision frames 137923 FCS errors
0 3 collision frames 3449 Oversize frames
0 4 collision frames 0 Undersize frames
0 5 collision frames 0 Collision fragments
0 6 collision frames
0 7 collision frames 2850111 Minimum size frames
0 8 collision frames 2890102436 65 to 127 byte frames
0 9 collision frames 205722233 128 to 255 byte frames
0 10 collision frames 1415284562 256 to 511 byte frames
0 11 collision frames 2004705051 512 to 1023 byte frames
0 12 collision frames 1397311 1024 to 1518 byte frames
0 13 collision frames 0 Overrun frames
0 14 collision frames 0 Pause frames
0 15 collision frames
0 Excessive collisions 9 Symbol error frames
0 Late collisions 3848 Invalid frames, too large
0 VLAN discard frames 99406482 Valid frames, too large
0 Excess defer frames 3 Invalid frames, too small
538106 64 byte frames 0 Valid frames, too small
1947823973 127 byte frames
3368016605 255 byte frames 0 Too old frames
2801366110 511 byte frames 0 Valid oversize frames
892122861 1023 byte frames 0 System FCS error frames
3334822385 1518 byte frames 0 RxPortFifoFull drop frame
303203790 Too large frames
0 Good (1 coll) frames
0 Good (>1 coll) frames

SW12176#sh controllers e gi2/0/26

Transmit GigabitEthernet2/0/26 Receive
1879759818 Bytes 2403316990 Bytes
1360889452 Unicast frames 17872712 Unicast frames
467867 Multicast frames 397018877 Multicast frames
2364571 Broadcast frames 20005182 Broadcast frames
0 Too old frames 1012935132 Unicast bytes
0 Deferred frames 1946515598 Multicast bytes
0 MTU exceeded frames 2006668660 Broadcast bytes
0 1 collision frames 0 Alignment errors
0 2 collision frames 0 FCS errors
0 3 collision frames 327 Oversize frames
0 4 collision frames 0 Undersize frames
0 5 collision frames 1 Collision fragments
0 6 collision frames
0 7 collision frames 914734 Minimum size frames
0 8 collision frames 1521923299 65 to 127 byte frames
0 9 collision frames 752733548 128 to 255 byte frames
0 10 collision frames 176340785 256 to 511 byte frames
0 11 collision frames 138487836 512 to 1023 byte frames
0 12 collision frames 2100014554 1024 to 1518 byte frames
0 13 collision frames 0 Overrun frames
0 14 collision frames 0 Pause frames
0 15 collision frames
0 Excessive collisions 0 Symbol error frames
0 Late collisions 327 Invalid frames, too large
0 VLAN discard frames 39449312 Valid frames, too large
0 Excess defer frames 1 Invalid frames, too small
60495 64 byte frames 0 Valid frames, too small
628914697 127 byte frames
427071225 255 byte frames 0 Too old frames
85328014 511 byte frames 0 Valid oversize frames
23570441 1023 byte frames 0 System FCS error frames
177959116 1518 byte frames 0 RxPortFifoFull drop frame
20817902 Too large frames
0 Good (1 coll) frames
0 Good (>1 coll) frames

[Leo Laohoo]

Is QoS configured on the 2960?

[CGS Service Desk]

Hi Leo, 

No QOS on the 2960, the configuration is very standard for a L2 device. I believe the Nexus is also fairly out of the box for QOS but would have to dig into what the default configuration normally is as these are the first Nexus devices I've looked after.

[CGS Service Desk · Accepted solution]

I have adjusted the load balancing to;

port-channel load-balance src-dst-ip

The drops have lowered drastically but still climbs (on all 4 ports instead of 1 now). It's good enough at this stage as I'm looking to lower the traffic through that switch.

[Joseph W. Doherty]

That load balancing algorithm is often a good or even best choice (for what the platform offers).

To decrease port drops further would depend on why you're getting drops.  For transient congestion, sometimes drops can be decreased by just increasing queue size (which on something like a 2960 may require QoS activation for advanced egress queue tuning).

For long term congestion, you may need to add bandwidth or do "things" (if possible) to manage source transmission rates.