Congestion throughout entire switch

[Chris Knipe]

Hi I am working on an issue that is giving me some trouble and was hoping someone here could help out.  I have a 3750 that I am looking at running 12.2.(40)SE and had some reports of a database application that was running slow today.  The customer mentioned internet is a little bit slower but other than that it is just this application.  I am able to reach everything fine but as I was looking at port counters I started noticing every single port I check on the switch is showing Output discards. Now that I have an idea that there is some congestion on the LAN, whats the best way to pinpoint the source?  When I used to work onsite, in emergencies, we would just pull out each blade on the switch until the problem subsided but in this case I dont have that luxury. I cleared the counters and the errors did come back right away on the interface.

show int fa1/0/1

FastEthernet1/0/1 is up, line protocol is up (connected)

  Hardware is Fast Ethernet, address is 0024.97b5.0003 (bia 0024.97b5.0003)

  Description: 802.1X Access Port with PEAP Machine Authentication

  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA, loopback not set

  Keepalive set (10 sec)

  Full-duplex, 100Mb/s, media type is 10/100BaseTX

  input flow-control is off, output flow-control is unsupported

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input 07:13:43, output 00:00:04, output hang never

  Last clearing of "show interface" counters never

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 119157

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  5 minute input rate 0 bits/sec, 0 packets/sec

  5 minute output rate 0 bits/sec, 0 packets/sec

[paul driver]

Hi Chris,

You dont mention your topology for this site, just this switch?

I notice you haven’t cleared the counters on this switch, I would suggest to do this, as it will give you a reference on how often the output drops occur.

what is the icmp responses to and from your hosts - and drops? latency etc...

Hardcode all the host interfaces if possible especially the DB server, and check the cabling.

res

Paul

[Chris Knipe]

Its mainly just latency.  I have experienced a loop before where someone double connected a hub or an ip phone and does not seem to be this severe.  From what I can gather there is just 1 user on this switch right now.  I cleared the counters right after I posted this last time but the errors still are returning. I am having trouble locating this on my diagram but this appears to be just an  access switch from what I can tell.  I am able to ping the user without any latency or drops as well. I did a show int | include|\/ and the only ports with traffic are the user and the port channel.

FastEthernet1/0/1 is up, line protocol is up (connected)

  5 minute input rate 6000 bits/sec, 2 packets/sec

  5 minute output rate 95000 bits/sec, 2 packets/sec

Port-channel31 is up, line protocol is up (connected)

  5 minute input rate 2814000 bits/sec, 374 packets/sec

  5 minute output rate 599000 bits/sec, 316 packets/sec

[Leo Laohoo]

Total output drops: 119157

My guess is your switch is pushing as much data as it can but the data is being dropped because the remote end can't take it any more.

Can you please post the command of "sh controller ethernet  FastEthernet1/0/1"?

[gerald.suiza]

hmm...interface fa1/0/1 is the user port isn't it?...where is the database server connected to? is it on the same switch?

I assume that the server has nic teaming since you showed the output of a portchannel. would you also be able to tell if th database is a virtual machine or a physical standalone machine?

[burleyman]

Chris,

Check this link. It explains what you may be seeing.

http://www.cisco.com/en/US/products/hw/routers/ps133/products_tech_note09186a0080094791.shtml

I helped this when I saw it on our switch by increasing the number of ports in the ether-channel, and in our case, moving them to different ASICS on our 4506 switch.

If you are not seeing dropped packets or extreme latency it probably is not causing any problems.

Mike

[Chris Knipe]

gerald

port 1 is the user and the other port is the ether channel.  ports 1-10 are all users and exhibit the same behaivor with interface stats and latency.  I checked the link from burleyman and this stuck out :

Note:

However, there is one exception. If the output queue is constantly full and if no packets are sent out of the interface at all, you must check for a hardware failure on the interface.

I found one of the interfaces that is not being used. I cleared the counters and watched the interface stats jump back and forth from 0 to 4294836759 drops over and over. I am starting to think this could be related to ios.  I remember not to long ago running across an issue like this that was found to be a known issue in some versions.  However the latency was not experienced in that situation. I will try to give the stack a reboot tonight and see if that does anything for the issue.

[paolo bevilacqua]

Update IOS before rebooting. You are running very old software. Ghost drop counts are due to bugs.