Re: Slow network performance on 3750's

asaykao73 · ‎08-10-2011

Hi All,

Just trying to narrow down what might be causing slow network issues on the LAN for a group of end users on one level. They are complaining that "everything" is slow from the PC booting up, getting a DHCP lease, logging onto Novell, browsing internal and external web sites.

End users switch ports are configured as access ports and port fast is enabled.

We're using a set of 4 x 3750's configured in a stack.

Switch Ports Model SW Version SW Image

------ ----- ----- ---------- ----------

* 1 54 WS-C3750X-48P 12.2(53)SE2 C3750E-UNIVERSALK9-M

2 54 WS-C3750X-48P 12.2(53)SE2 C3750E-UNIVERSALK9-M

3 54 WS-C3750X-48P 12.2(53)SE2 C3750E-UNIVERSALK9-M

4 54 WS-C3750X-48P 12.2(53)SE2 C3750E-UNIVERSALK9-M

The only thing I can track down is that there are heaps of output drops on these access ports.

Is this a red herring or could it be causing the slow network performance issues???

The OQD for most of these access ports are very high as seen below.

8NS-3750X-1#show interfaces summary

*: interface is up

IHQ: pkts in input hold queue IQD: pkts dropped from input queue

OHQ: pkts in output hold queue OQD: pkts dropped from output queue

RXBS: rx rate (bits/sec) RXPS: rx rate (pkts/sec)

TXBS: tx rate (bits/sec) TXPS: tx rate (pkts/sec)

TRTL: throttle count

Interface IHQ IQD OHQ OQD RXBS RXPS TXBS TXPS TRTL

-------------------------------------------------------------------------

Vlan1 0 0 0 0 0 0 0 0 0

* Vlan900 0 0 0 0 2000 4 2000 3 0

FastEthernet0 0 0 0 0 0 0 0 0 0

GigabitEthernet1/0/1 0 0 0 0 0 0 0 0 0

* GigabitEthernet1/0/2 0 0 0 1171 10000 9 128000 24 0

GigabitEthernet1/0/3 0 0 0 0 0 0 0 0 0

* GigabitEthernet1/0/4 0 0 0 44 2000 2 34000 5 0

* GigabitEthernet1/0/5 0 0 0 1277 0 0 3000 3 0

* GigabitEthernet1/0/6 0 0 0 676 0 0 3000 4 0

GigabitEthernet1/0/7 0 0 0 0 0 0 0 0 0

GigabitEthernet1/0/8 0 0 0 0 0 0 0 0 0

GigabitEthernet1/0/9 0 0 0 0 0 0 0 0 0

GigabitEthernet1/0/10 0 0 0 38 0 0 0 0 0

* GigabitEthernet1/0/11 0 0 0 26 7000 5 56000 11 0

* GigabitEthernet1/0/12 0 0 0 0 0 0 3000 4 0

GigabitEthernet1/0/13 0 0 0 0 0 0 0 0 0

* GigabitEthernet1/0/14 0 0 0 3944 1000 1 14000 4 0

* GigabitEthernet1/0/15 0 0 0 1004 0 0 4000 3 0

* GigabitEthernet1/0/16 0 0 0 991 3000 1 5000 4 0

* GigabitEthernet1/0/17 0 0 0 2347 8000 7 24000 12 0

* GigabitEthernet1/0/18 0 0 0 0 0 0 3000 4 0

* GigabitEthernet1/0/19 0 0 0 2692 9000 8 21000 13 0

* GigabitEthernet1/0/20 0 0 0 0 3000 2 5000 5 0

* GigabitEthernet1/0/21 0 0 0 2767 0 0 3000 3 0

* GigabitEthernet1/0/22 0 0 0 5674 0 0 3000 3 0

* GigabitEthernet1/0/23 0 0 0 857 0 0 3000 3 0

GigabitEthernet1/0/24 0 0 0 1629 0 0 0 0 0

* GigabitEthernet1/0/25 0 0 0 369 3000 4 6000 7 0

* GigabitEthernet1/0/26 0 0 0 43 0 0 3000 4 0

* GigabitEthernet1/0/27 0 0 0 648 1000 1 3000 4 0

* GigabitEthernet1/0/28 0 0 0 4327 1000 2 12000 6 0

Thanks.

Andy

lgijssel · ‎08-10-2011

Have you configured qos?

From past experience I have learned that using a minimal qos config can lead to this behavior.

Try to deactivate it (no mls qos) or configure full auto-qos.

regards,

Leo

Latchum Naidu · ‎08-11-2011

Hi,

I would requst you to enable CEF as it is always a good idea to have running, but if you are getting output queue drops you are probably over-utilizing the links.

Please rate the helpfull posts.
Regards,
Naidu.

Leo Laohoo · ‎08-11-2011

Spanning-tree enabled on access ports?

Maybe you have an STP loop happening.

smogra · ‎08-11-2011

Check for oversubscription.

Cheeers

Sweta

asaykao73 · ‎08-11-2011

No QOS running.

#sh mls qos

QoS is disabled

QoS ip packet dscp rewrite is enabled

CEF is not enabled - should it be??

#sh ip cef

%IPv4 CEF not running

STP is disabled on the access ports like so with portfast enabled.

interface GigabitEthernet1/0/30

switchport access vlan 101

switchport mode access

switchport port-security maximum 3

switchport port-security

switchport port-security aging time 1

switchport port-security violation restrict

load-interval 30

storm-control broadcast level 50.00

storm-control multicast level 50.00

spanning-tree portfast

spanning-tree bpduguard enable

I don't really see any STP loop for Vlan 101.

One uplink is forwarding and the other uplink is blocking.

#sh spanning-tree vlan 101

Gi1/1/1 Root FWD 4 128.49 P2p Peer(STP)

Gi4/1/1 Altn BLK 4 128.211 P2p Peer(STP)

Doesn't seem like an oversubscription issue.

3750's have a 1G uplink to the core switch which isn't saturated or flat lining.

Not sure what else I can look at on the switch.

Thanks all.

Andy

Somasundaram Jayaraman · ‎08-12-2011

HI,

Have you configured any port-channels?

What type of port-channel load balance you are using?

show port-channel loadbalance

Have you tried increasing the hold queue limit?

Please share the below outputs.

sho platfo port-asic stats drop

show proc cpu sorted | e 0.0

Hope this helps.

Cheers

Somu

andtoth · ‎08-13-2011

Hi Andy,

Check the interfaces with 'sh interface' and look for input and output errors. Also check if there are any collisions, which indicates a duplexity mismatch. Make sure that speed and duplex are set to Auto on both sides.

Do you see anything suspicious in the logs?

Additionally, make sure you have CEF enabled with the 'ip cef distributed' global configuration command.

Best regards,

Andras