Isolating Cause Of Packets Drops

[jeremyt03]

Hi All,

I am trying to isolate issues noticed with packet drops on our core layer 3 switch 3560.  To give you guys the layout...we have 3 access 48 port Cisco 2960 switches running off the core for client access.  2 additional Dell switches are also coming off this core for iscsi connections (we have 4 virtual hosts running 35 servers)  Additional equipment would be the ASA, and we use an Ironport web appliance as well as an an Ironport email appliance. C170/S170 respectively.

Spanning tree portfast is enabled however show interfaces shows multiple interfaces with packet drops.  Here's a sample of the worst.

GigabitEthernet0/6 is up, line protocol is down (monitoring)
  Hardware is Gigabit Ethernet, address is 5475.d042.4786 (bia 5475.d042.4786)
  Description: $IRONPORT_L4_TRAFFIC_MONITOR$
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 92/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 5w5d, output hang never
  Last clearing of "show interface" counters 6d02h
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 867035621
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 362533000 bits/sec, 50410 packets/sec
     0 packets input, 0 bytes, 0 no buffer
     Received 0 broadcasts (0 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     22887211441 packets output, 23531458740687 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

GigabitEthernet0/31 is up, line protocol is up (connected)
  Hardware is Gigabit Ethernet, address is 5475.d042.479f (bia 5475.d042.479f)
  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 100Mb/s, media type is 10/100/1000BaseTX
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 00:00:00, output hang never
  Last clearing of "show interface" counters 6d02h
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 1322
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 16000 bits/sec, 8 packets/sec
  5 minute output rate 522000 bits/sec, 70 packets/sec
     4219280 packets input, 697034051 bytes, 0 no buffer
     Received 6638 broadcasts (5415 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 5415 multicast, 0 pause input
     0 input packets with dribble condition detected
     12772949 packets output, 9078282929 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out
 

GigabitEthernet0/36 is up, line protocol is up (connected)
  Hardware is Gigabit Ethernet, address is 5475.d042.47a4 (bia 5475.d042.47a4)
  Description: $UPLINK_TO_iSCSI_NETWORK$
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 8/255, rxload 28/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:28, output 00:00:01, output hang never
  Last clearing of "show interface" counters 6d02h
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 1860601
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 113351000 bits/sec, 10772 packets/sec
  5 minute output rate 32075000 bits/sec, 6250 packets/sec
     6604363493 packets input, 8707945514559 bytes, 0 no buffer
     Received 117873 broadcasts (26216 multicasts)
     0 runts, 11985 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 26216 multicast, 932524 pause input
     0 input packets with dribble condition detected
     3673675691 packets output, 2651124584467 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

If you notice I cleared the stats on these interfaces less than a week ago so these drops have accumulated in the last 6 days!!!!

Here's the output from these same worst offenders using the "show platform port-asic stats" command

Interface Gi0/6 TxQueue Drop Statistics
    Queue 0
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 1
      Weight 0 Frames 0
      Weight 1 Frames 1
      Weight 2 Frames 0
    Queue 2
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 3
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 1399651172
    Queue 4
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 5
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 6
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 7
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0

Interface Gi0/31 TxQueue Drop Statistics
    Queue 0
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 1
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 2
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 3
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 1941042
    Queue 4
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 5
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 6
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 7
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0

  Interface Gi0/36 TxQueue Drop Statistics
    Queue 0
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 1
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 2
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 3
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 36305016
    Queue 4
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 5
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 6
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 7
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0

Any input in next steps to help isolate the issue or narrowing it down would be greatly appreciated.

Thanks,

Jeremy

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

The 2960/3560/3750 are a bit infamous for having packet drops when dealing with bursty traffic, especially with QoS enabled with default settings.

What IOS version are you running?  Reason I ask, I believe some internal buffer management improvements were made in later IOS versions.  I.e. depending on your IOS version, upgrading might help a bit.

From your stats, I presume QoS is not enabled.  If so, enabling it, with manual buffer tuning, might mitigate some of the drops.

What also might mitigate some of these drops, is moving these ports to any available uplink (i.e. SFP) ports.  (The reason for this, the 3560/3750 provides 2 MB of buffers to each bank of 24 copper ports and to the SFP ports.  So, moving to a SFP port can allow for more buffering, per port.)

[jeremyt03]

Joseph,

You are correct that QoS is not enabled/configured.

IOS

Cisco IOS Software, C3560E Software (C3560E-UNIVERSALK9-M), Version 12.2(53)SE2, RELEASE SOFTWARE (fc3)
 

I will check configure QoS and see if IOS revisions are available and post results tomorrow.  Thanks for the speedy response.

I am including the config in this response for anybody that has the time to take a peak

Thanks guys,

Jeremy

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

(53)SE2, is probably recent enough, I suspect there may not be internal QoS improvements.

BTW, if you do upgrade, (55)SE9 (now SE10?) is considered a very stable release.

Drop mitigation, if possible, will likely require one of the approaches I've already described.

PS:

BTW, in theory, full-duplex flow control might also mitigate egress drops.  However, using it should be approached with much care.