3850 packet drops

[Amafsha1]

Sorry, I know this is a subject that people hardly have info on.  I feel like the documentation online is very very limited to explaining these problems and how to fix them.

So I have this IPS system that recently  has been dropping a lot of traffic.  When we look under the swport that it's connected, we see the tot ouput drops increasing pretty rapidly incrementing every few min.    Would it matter to move this interface to a 10Gig link...will that give it more buffer for bursty traffic?  Or is there a way to increase buffer on that swport?  Thank you in advance

Switch31#sh int g1/0/2
GigabitEthernet1/0/21 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 1d3s.5f32.3a92 
Description: IPS devices
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 13/255, rxload 12/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output never, output hang never
Last clearing of "show interface" counters 5d16h
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 136590177
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 49864000 bits/sec, 7538 packets/sec
5 minute output rate 52462000 bits/sec, 8704 packets/sec
2376977535 packets input, 1563001136366 bytes, 0 no buffer
Received 2402879 broadcasts (245385 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 245385 multicast, 0 pause input
0 input packets with dribble condition detected
3848369562 packets output, 4074227663066 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

[Leo Laohoo]

---

@Amafsha1 wrote:

Last clearing of "show interface" counters 5d16h

---

Clearing of the counters does not help.  

Please post the complete output to the command "sh interface Gi1/0/1 CONTROLL". 

I am looking at the amount of packets in and out and in the time of "5d16h" it has transferred that much, I'd say that switch's very-shallow buffer got overwhelmed.  

If the output I've requested is really "clean", I'd recommend putting QoS on this interface.

[Amafsha1]

Thanks for the reply. We had a complete power outage and we don't have any generators atm.  Trust me, I'm not a fan of clearing interfaces either.

Here is the output. 

When you say qos, will auto qos be good?  any specifics

GigabitEthernet1/0/21 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 
Description: IPS
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 13/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output never, output hang never
Last clearing of "show interface" counters 5d19h
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 137198905
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 6177000 bits/sec, 2426 packets/sec
5 minute output rate 54060000 bits/sec, 5411 packets/sec
2425089904 packets input, 1599224067341 bytes, 0 no buffer
Received 2453273 broadcasts (250479 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 250479 multicast, 0 pause input
0 input packets with dribble condition detected
3913137512 packets output, 4132503620381 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
Transmit GigabitEthernet1/0/21 Receive
135128953139510 Total bytes 134460796198630 Total bytes
237172984290 Unicast frames 194455531867 Unicast frames
135128951665208 Unicast bytes 134441381628332 Unicast bytes
124 Multicast frames 34415255 Multicast frames
7936 Multicast bytes 2202576954 Multicast bytes
22909 Broadcast frames 268937396 Broadcast frames
1466366 Broadcast bytes 17211993344 Broadcast bytes
0 System FCS error frames 0 IpgViolation frames
0 MacUnderrun frames 0 MacOverrun frames
0 Pause frames 0 Pause frames
0 Cos 0 Pause frames 0 Cos 0 Pause frames
0 Cos 1 Pause frames 0 Cos 1 Pause frames
0 Cos 2 Pause frames 0 Cos 2 Pause frames
0 Cos 3 Pause frames 0 Cos 3 Pause frames
0 Cos 4 Pause frames 0 Cos 4 Pause frames
0 Cos 5 Pause frames 0 Cos 5 Pause frames
0 Cos 6 Pause frames 0 Cos 6 Pause frames
0 Cos 7 Pause frames 0 Cos 7 Pause frames
0 Oam frames 0 OamProcessed frames
0 Oam frames 0 OamDropped frames
61120981013 Minimum size frames 45162159674 Minimum size frames
26461057535 65 to 127 byte frames 39502464469 65 to 127 byte frames
53262003965 128 to 255 byte frames 14279223078 128 to 255 byte frames
9588031950 256 to 511 byte frames 6222330805 256 to 511 byte frames
9064633103 512 to 1023 byte frames 6090067633 512 to 1023 byte frames
77676299757 1024 to 1518 byte frames 83502638859 1024 to 1518 byte frames
0 1519 to 2047 byte frames 0 1519 to 2047 byte frames
0 2048 to 4095 byte frames 0 2048 to 4095 byte frames
0 4096 to 8191 byte frames 0 4096 to 8191 byte frames
0 8192 to 16383 byte frames 0 8192 to 16383 byte frames
0 16384 to 32767 byte frame 0 16384 to 32767 byte frame
0 > 32768 byte frames 0 > 32768 byte frames
0 Late collision frames 0 SymbolErr frames
0 Excess Defer frames 0 Collision fragments
0 Good (1 coll) frames 0 ValidUnderSize frames
0 Good (>1 coll) frames 0 InvalidOverSize frames
0 Deferred frames 0 ValidOverSize frames
0 Gold frames dropped 0 FcsErr frames
0 Gold frames truncated
0 Gold frames successful
0 1 collision frames
0 2 collision frames
0 3 collision frames
0 4 collision frames
0 5 collision frames
0 6 collision frames
0 7 collision frames
0 8 collision frames
0 9 collision frames
0 10 collision frames
0 11 collision frames
0 12 collision frames
0 13 collision frames
0 14 collision frames
0 15 collision frames
0 Excess collision frames

LAST UPDATE 4890 msecs AGO

[Leo Laohoo]

---

@Amafsha1 wrote:

will auto qos be good?  

---

Line is clean.  I don't see any line errors.  This issue can definitely be rectified if QoS is enabled to shape the traffic.  

Auto QoS won't be of any good because this is a very bursty traffic.  The forum's QoS "guru" is Joseph and if he's lurking around he should be able to help.  

Otherwise, raise a TAC Case and get them to knock up a QoS configuration.

[Amafsha1]

Thanks, appreciate it.  I'll do some digging unless I get lucky :)

[Amafsha1]

Random Idea.  What if I moved the interface to a 10gb interface with a copper sfp and configured it as speed 1000.  Since 10GB ports have more buffers, you think that could potentially resolve the issue? 

[mohammed01701]

Before you move on 10g can you run the command "show queueing interface x/x/x". Are you running QoS?

Regards

Mohammed

[Amafsha1]

We just configured qos today and I did not move it to a 10Gig interface, moved it to a 1 gig interface on the uplink module of the 3850. I don't think that even matters anyways:

Switch#sh queueing interface g1/1/2
Interface GigabitEthernet1/1/2 queueing strategy: fair queueing

[Joseph W. Doherty]

Yup, I'm lurking - laugh.

Unfortunately, I don't have any hands-on with 3650/3850 series. However, I think (?), like the earlier 3650/3750 series, they support buffer allocation tuning when QoS is enabled. If so, it may be possible to allocate more buffers to the port suffering a high percentage of drops, which could help if your drops are due to transient bursts. (NB: when you provide more buffers for port to use, you also chance taking them from other ports. I.e. port drops can move from one port to another.)

(NB: if the 3650/3850 do support buffer tuning, hope it's not as confusing as the 3560/3750 series.)

You asked about moving to a 10g port. Again, I'm not familiar with the 3650/3850 architecture, and so I don't know if 10g ports get any additional buffer space. They might not, unless they are "uplink" ports, then they might. (Again, that's how the earlier 3560/3750 architecture works.) That aside though, all else being equal, a "faster" port should drain faster, so often it won't queue packets as would a "slower" port. I.e. a 10g port might see less drops.

[Amafsha1]

Thanks for responding.  I will try moving it the uplink module port of the 3850 since I don't have much to lose.  From what I see, QOS configs are pretty complicated and since I only need it just for this 1 issue.  

[Joseph W. Doherty]

Let us know how it goes.

[Amafsha1]

Hello All, the issue has been resolved.  We put qos on the interface and increased the buffer size with the "qos queue-softmax-multiplier 1200" command.   This fixed the "slowness" issue people were complaining about.  speed tests now show a lot better speeds.   

[Joseph W. Doherty]

Ah, good to read buffer tuning did mitigate the issue.

I think (?) that command is the 3560/3750 equivalent of increasing the logical limit.