Narrowing down performance bottleneck

yantiscompany · ‎11-22-2011

Over the past year, I have been working on fixing my companies network, and have been struggling with what I believe is much worse performance than we should be getting. I've been blaming this on our iSCSI SAN, an EMC AX4-5i, but after working with EMC to try to find a problem and coming up with nothing, I think its time to start looking at our network. I have some basic knowledge of IOS, but its a lot easier for me to work on things from scratch, and our 3 stack Cisco WS-C3750G-48PS-S was setup long before I arrived. If someone could take a look at our config I'd really appreciate it. I don't think we are pushing the switches, as cpu and memory utilization never seems to go above 60% in the CNA health view, bandwidth utilization is usually less than 3% or so, and packet errors are at 0. If the config looks good, what are some favorite tools for testing the perfomance of the network? One of the most constant things I notice on the network side is that no bandwidth graph I ever look at is straight, and always seems to be continuous huge peaks and drops. That probably is more of an issue with the way hard drives and networks work, but I thought I'd mention it since speed comparisons I've seen from others have looked much more linear.

Still, I might just be imagining things and our network could be fine. I just think that transferring a 4GB file from one 2008r2 VM (VMware) on a 6 disk raid 5 pool to another 2008r2 VM with a 16 disk Raid 10 pool should be faster than 45MB/s?

Oh the joys of network administration. Too many moving parts. Any help is greatly appreciated.

added some desriptions and just included a few sample port configs.

company-core-sw1#sh run

Building configuration...

Current configuration : 53931 bytes

!

version 12.2

no service pad

service timestamps debug uptime

service timestamps log datetime localtime

service password-encryption

service sequence-numbers

!

hostname company-core-sw1

!

boot-start-marker

boot-end-marker

!

logging buffered 128000

no logging console

enable secret 5 passwordhash

!

username ausername privilege 15 secret 5 passwordhash

!

aaa new-model

!

aaa authentication login default local

aaa authorization exec default local

!

aaa session-id common

clock timezone CST -6

clock summer-time CDT recurring

switch 1 provision ws-c3750g-48ps

switch 2 provision ws-c3750g-48ps

switch 3 provision ws-c3750g-48ps

system mtu routing 1500

ip routing

no ip domain-lookup

ip domain-name company.local

!

ip dhcp snooping vlan 2-3

no ip dhcp snooping information option

ip dhcp snooping database flash:dhcp.log

ip arp inspection vlan 2-3

!

mls qos map policed-dscp 24 26 46 to 0

mls qos map cos-dscp 0 8 16 24 32 46 48 56

mls qos srr-queue input bandwidth 90 10

mls qos srr-queue input threshold 1 8 16

mls qos srr-queue input threshold 2 34 66

mls qos srr-queue input buffers 67 33

mls qos srr-queue input cos-map queue 1 threshold 2 1

mls qos srr-queue input cos-map queue 1 threshold 3 0

mls qos srr-queue input cos-map queue 2 threshold 1 2

mls qos srr-queue input cos-map queue 2 threshold 2 4 6 7

mls qos srr-queue input cos-map queue 2 threshold 3 3 5

mls qos srr-queue input dscp-map queue 1 threshold 2 9 10 11 12 13 14 15

mls qos srr-queue input dscp-map queue 1 threshold 3 0 1 2 3 4 5 6 7

mls qos srr-queue input dscp-map queue 1 threshold 3 32

mls qos srr-queue input dscp-map queue 2 threshold 1 16 17 18 19 20 21 22 23

mls qos srr-queue input dscp-map queue 2 threshold 2 33 34 35 36 37 38 39 48

mls qos srr-queue input dscp-map queue 2 threshold 2 49 50 51 52 53 54 55 56

mls qos srr-queue input dscp-map queue 2 threshold 2 57 58 59 60 61 62 63

mls qos srr-queue input dscp-map queue 2 threshold 3 24 25 26 27 28 29 30 31

mls qos srr-queue input dscp-map queue 2 threshold 3 40 41 42 43 44 45 46 47

mls qos srr-queue output cos-map queue 1 threshold 3 5

mls qos srr-queue output cos-map queue 2 threshold 3 3 6 7

mls qos srr-queue output cos-map queue 3 threshold 3 2 4

mls qos srr-queue output cos-map queue 4 threshold 2 1

mls qos srr-queue output cos-map queue 4 threshold 3 0

mls qos srr-queue output dscp-map queue 1 threshold 3 40 41 42 43 44 45 46 47

mls qos srr-queue output dscp-map queue 2 threshold 3 24 25 26 27 28 29 30 31

mls qos srr-queue output dscp-map queue 2 threshold 3 48 49 50 51 52 53 54 55

mls qos srr-queue output dscp-map queue 2 threshold 3 56 57 58 59 60 61 62 63

mls qos srr-queue output dscp-map queue 3 threshold 3 16 17 18 19 20 21 22 23

mls qos srr-queue output dscp-map queue 3 threshold 3 32 33 34 35 36 37 38 39

mls qos srr-queue output dscp-map queue 4 threshold 1 8

mls qos srr-queue output dscp-map queue 4 threshold 2 9 10 11 12 13 14 15

mls qos srr-queue output dscp-map queue 4 threshold 3 0 1 2 3 4 5 6 7

mls qos queue-set output 1 threshold 1 138 138 92 138

mls qos queue-set output 1 threshold 2 138 138 92 400

mls qos queue-set output 1 threshold 3 36 77 100 318

mls qos queue-set output 1 threshold 4 20 50 67 400

mls qos queue-set output 2 threshold 1 149 149 100 149

mls qos queue-set output 2 threshold 2 118 118 100 235

mls qos queue-set output 2 threshold 3 41 68 100 272

mls qos queue-set output 2 threshold 4 42 72 100 242

mls qos queue-set output 1 buffers 10 10 26 54

mls qos queue-set output 2 buffers 16 6 17 61

mls qos

!

crypto pki trustpoint TP-self-signed-3861525248

enrollment selfsigned

subject-name cn=IOS-Self-Signed-Certificate-3861525248

revocation-check none

rsakeypair TP-self-signed-3861525248

!

crypto pki certificate chain TP-self-signed-3861525248

certificate self-signed 01

30820254 308201BD A0030201 02020101 300D0609 2A864886 F70D0101 04050030

31312F30 2D060355 04031326 494F532D 53656C66 2D536967 6E65642D 43657274

69666963 6174652D 33383631 35323532 3438301E 170D3933 30333031 30303033

32355A17 0D323030 31303130 30303030 305A3031 312F302D 06035504 03132649

4F532D53 656C662D 5369676E 65642D43 65727469 66696361 74652D33 38363135

32353234 3830819F 300D0609 2A864886 F70D0101 01050003 818D0030 81890281

8100C59C 65C74B06 40CCA1C4 8066EFC8 783F80A9 7AFB47DC DD55C69C 11A03585

DFD6EC81 9B060F84 2F6A7D68 8DA9A4CC 9F79BE0F 52032A55 01917B2D E1E74D32

68AB728E 169890CE 6CDAEBA8 8F932E9C DD0D0DA3 CEE37C6B 665FC146 A15F237A

F37B27ED 313ED639 30D3A3CC D1EE72C2 C6B1982B 4ACEE55E 0FFE0872 D7BC3AA0

C6C50203 010001A3 7C307A30 0F060355 1D130101 FF040530 030101FF 30270603

551D1104 20301E82 1C79616E 7469732D 636F7265 2D737731 2E79616E 7469732E

6C6F6361 6C301F06 03551D23 04183016 8014BDCB F3748A5E 91B1E853 5DDC8799

E1F64D54 79D0301D 0603551D 0E041604 14BDCBF3 748A5E91 B1E8535D DC8799E1

F64D5479 D0300D06 092A8648 86F70D01 01040500 03818100 55FF5648 8701110B

798A22AB 11C4F68E 0BFD5D5A BBD7ABDE B96B0D0C BE446A08 7FBFBC76 2BD06F55

8B7519A1 B7A3B141 29E1A73F AB6A5E27 DB3CF406 33B9E7E7 695989E9 EB52D0D5

853D9235 96B081D7 89B83A78 A8469701 6A6E8A07 21AE8F6F F8A6E3A3 21657F20

5FDE5AC2 748D4378 883E64A8 8955391A DA9364F6 B0C5CA0A

quit

!

errdisable recovery cause udld

errdisable recovery cause bpduguard

errdisable recovery cause security-violation

errdisable recovery cause channel-misconfig (STP)

errdisable recovery cause pagp-flap

errdisable recovery cause dtp-flap

errdisable recovery cause link-flap

errdisable recovery cause gbic-invalid

errdisable recovery cause psecure-violation

errdisable recovery cause dhcp-rate-limit

errdisable recovery cause vmps

errdisable recovery cause storm-control

errdisable recovery cause arp-inspection

errdisable recovery cause loopback

errdisable recovery interval 120

spanning-tree mode pvst

no spanning-tree optimize bpdu transmission

spanning-tree extend system-id

!

vlan internal allocation policy ascending

!

class-map match-all AutoQoS-VoIP-RTP-Trust

match ip dscp ef

class-map match-all AutoQoS-VoIP-Control-Trust

match ip dscp cs3 af31

!

policy-map AutoQoS-Police-CiscoPhone

class AutoQoS-VoIP-RTP-Trust

set dscp ef

police 320000 8000 exceed-action policed-dscp-transmit

class AutoQoS-VoIP-Control-Trust

set dscp cs3

police 32000 8000 exceed-action policed-dscp-transmit

!

interface Loopback0

ip address 192.168.0.2 255.255.255.255

!

interface GigabitEthernet1/0/1

description ASA Inside interface

switchport access vlan 16

ip arp inspection trust

spanning-tree portfast

ip verify source port-security

ip dhcp snooping trust

!

interface GigabitEthernet1/0/9

description Server Ethernet Connection

switchport access vlan 2

ip arp inspection trust

spanning-tree portfast

ip verify source port-security

ip dhcp snooping trust

!

interface GigabitEthernet1/0/10

description Server Ethernet connection

switchport access vlan 2

ip arp inspection trust

spanning-tree portfast

ip verify source port-security

ip dhcp snooping trust

!

interface GigabitEthernet1/0/11

description Storage NIC on VMware server

switchport access vlan 6

ip arp inspection trust

load-interval 30

speed 1000

duplex full

spanning-tree portfast

ip verify source port-security

ip dhcp snooping trust

!

interface GigabitEthernet1/0/12

description Storage NIC on VMware Server

switchport access vlan 2

ip arp inspection trust

speed 1000

duplex full

spanning-tree portfast

ip verify source port-security

ip dhcp snooping trust

!

interface GigabitEthernet1/0/28

description IP Phone and Workstation

switchport access vlan 3

switchport voice vlan 4

ip arp inspection trust

srr-queue bandwidth share 10 10 60 20

srr-queue bandwidth shape 10 0 0 0

queue-set 2

priority-queue out

mls qos trust device cisco-phone

mls qos trust cos

auto qos voip cisco-phone

spanning-tree portfast

ip verify source port-security

ip dhcp snooping trust

!

interface GigabitEthernet2/0/46

description EMC Storage SAN port

switchport access vlan 6

ip arp inspection trust

speed 1000

duplex full

spanning-tree portfast

!

interface GigabitEthernet3/0/43

description EMC Storage SAN port

switchport access vlan 7

ip arp inspection trust

speed 1000

duplex full

spanning-tree portfast

!

interface GigabitEthernet3/0/44

description Storage NIC on Win server

switchport access vlan 6

ip arp inspection trust

spanning-tree portfast

ip dhcp snooping trust

!

interface GigabitEthernet3/0/45

description Storage NIC on Win server

switchport access vlan 6

ip arp inspection trust

spanning-tree portfast

!

interface GigabitEthernet3/0/46

description EMC Storage SAN port

switchport access vlan 6

ip arp inspection trust

speed 1000

duplex full

spanning-tree portfast

!

interface GigabitEthernet3/0/47

description Voice Gateway Spare

switchport access vlan 4

switchport mode access

ip arp inspection trust

spanning-tree portfast

ip dhcp snooping trust

!

interface GigabitEthernet3/0/48

description Unified Communications Business Edition Eth 1 Spare

switchport access vlan 4

ip arp inspection trust

spanning-tree portfast

ip dhcp snooping trust

!

interface GigabitEthernet3/0/49

!

interface Vlan1

no ip address

no ip proxy-arp

shutdown

!

interface Vlan2 - servers

ip address 192.168.1.1 255.255.255.0

ip helper-address 192.168.1.15

no ip proxy-arp

!

interface Vlan3 - Office workstations

ip address 192.168.2.1 255.255.255.0

ip helper-address 192.168.1.15

no ip proxy-arp

!

interface Vlan4 - Office Phones

ip address 192.168.3.1 255.255.255.0

ip helper-address 192.168.1.15

no ip proxy-arp

!

interface Vlan5 - Office WLAN

ip address 192.168.4.1 255.255.255.0

ip helper-address 192.168.1.15

no ip proxy-arp

!

interface Vlan14 - Satellite office workstations

ip address 192.168.14.1 255.255.255.0

ip helper-address 192.168.1.15

!

interface Vlan15 - Satellite office phones

ip address 192.168.15.1 255.255.255.0

!

interface Vlan16 - not used

ip address 172.16.0.2 255.255.255.0

!

interface Vlan17 - not used

no ip address

!

interface Vlan18 - not used

ip address 172.18.0.1 255.255.255.0

!

interface Vlan22 - other satellite office workstations

ip address 192.168.22.1 255.255.255.0

!

interface Vlan23 - other satellite office phones

ip address 192.168.23.1 255.255.255.0

!

interface Vlan30 - not used

ip address 172.30.0.1 255.255.255.0

ip helper-address 192.168.1.15

no ip proxy-arp

!

ip classless

ip route 0.0.0.0 0.0.0.0 172.16.0.1

ip route 192.168.0.1 255.255.255.255 192.168.3.16

ip route 192.168.12.0 255.255.255.0 192.168.254.2

ip route 192.168.13.0 255.255.255.0 192.168.254.2

ip route 192.168.14.0 255.255.255.0 192.168.254.6

ip route 192.168.15.0 255.255.255.0 192.168.254.6

ip http server

ip http secure-server

!

ip sla enable reaction-alerts

access-list 7 permit 0.0.0.0 255.255.255.0

snmp-server community snmppass RO

snmp-server community snmppass RO 7

snmp-server enable traps tty

snmp-server enable traps license

snmp-server enable traps stpx root-inconsistency loop-inconsistency

snmp ifmib ifindex persist

!

line con 0

line vty 0 4

exec-timeout 0 0

password 7 hash

line vty 5 15

password 7 hash

!

monitor session 1 source interface Gi2/0/2

monitor session 1 destination interface Gi3/0/12

ntp clock-period 36028630

ntp server 132.163.4.101 prefer

end

Leo Laohoo · ‎11-22-2011

Kindly post the output for the following commands:

sh mls qos

sh inventory

sh version

sh switch details

sh int

sh controller e

Have you tried removing the following lines:

ip arp inspection trust

ip verify source port-security

yantiscompany · ‎11-22-2011

Here ya go

inspection trust and port security were already on there and I think the CNA likes to through those on when you configure through the GUI. I think I took them off once, and was having problems with connectivity. That problem was actually probably due to a terrible cable I was using before, so I may have to revisit removing those. Would these cause a big performance hit? If I remember right, they probably aren't needed on any of our secure server stuff, but should be used on the workstation connections. Is this correct?

PS. Is there any way to post config outputs in their own scrollable window? Seems kinda silly to post these huge outputs, and make a super long thread to scroll.

company-core-sw1#sh mls qos

QoS is enabled

QoS ip packet dscp rewrite is enabled

company-core-sw1#sh inventory

NAME: "1", DESCR: "WS-C3750G-48PS"

PID: WS-C3750G-48PS-S , VID: V05 , SN: FOC1140Y0xx

NAME: "2", DESCR: "WS-C3750G-48PS"

PID: WS-C3750G-48PS-S , VID: V05 , SN: FOC1140Y0xx

NAME: "3", DESCR: "WS-C3750G-48PS"

PID: WS-C3750G-48PS-S , VID: V05 , SN: FOC1140Y0xx

company-core-sw1#sh version

Cisco IOS Software, C3750 Software (C3750-IPBASEK9-M), Version 12.2(55)SE1, RELE

ASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Compiled Thu 02-Dec-10 07:46 by prod_rel_team

Image text-base: 0x01000000, data-base: 0x02D00000

ROM: Bootstrap program is C3750 boot loader

BOOTLDR: C3750 Boot Loader (C3750-HBOOT-M) Version 12.2(44)SE5, RELEASE SOFTWARE

(fc1)

company-core-sw1 uptime is 17 weeks, 3 days, 1 hour, 18 minutes

System returned to ROM by power-on

System image file is "flash:c3750-ipbasek9-mz.122-55.SE1/c3750-ipbasek9-mz.122-5

5.SE1.bin"

This product contains cryptographic features and is subject to United

States and local country laws governing import, export, transfer and

use. Delivery of Cisco cryptographic products does not imply

third-party authority to import, export, distribute or use encryption.

Importers, exporters, distributors and users are responsible for

compliance with U.S. and local country laws. By using this product you

agree to comply with applicable laws and regulations. If you are unable

to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:

http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to

export@cisco.com.

cisco WS-C3750G-48PS (PowerPC405) processor (revision F0) with 131072K bytes of

memory.

Processor board ID FOC1140Y0FC

Last reset from power-on

13 Virtual Ethernet interfaces

156 Gigabit Ethernet interfaces

The password-recovery mechanism is enabled.

512K bytes of flash-simulated non-volatile configuration memory.

Base ethernet MAC Address : 00:1D:E6:2A:33:00

Motherboard assembly number : 73-10216-08

Power supply part number : 341-0108-03

Motherboard serial number : FOC11393CQ6

Power supply serial number : DCA1136A097

Model revision number : F0

Motherboard revision number : C0

Model number : WS-C3750G-48PS-S

System serial number : FOC1140Y0xx

Top Assembly Part Number : 800-26853-01

Top Assembly Revision Number : C0

Version ID : V05

CLEI Code Number : CNMWN00ARC

Hardware Board Revision Number : 0x09

Switch Ports Model SW Version SW Image

------ ----- ----- ---------- ----------

* 1 52 WS-C3750G-48PS 12.2(55)SE1 C3750-IPBASEK9-M

2 52 WS-C3750G-48PS 12.2(55)SE1 C3750-IPBASEK9-M

3 52 WS-C3750G-48PS 12.2(55)SE1 C3750-IPBASEK9-M

Switch 02

---------

Switch Uptime : 17 weeks, 3 days, 1 hour, 17 minutes

Base ethernet MAC Address : 00:1D:E6:94:9C:00

Motherboard assembly number : 73-10216-08

Power supply part number : 341-0108-03

Motherboard serial number : FOC11393CKF

Power supply serial number : DCA1138A1EN

Model revision number : F0

Motherboard revision number : C0

Model number : WS-C3750G-48PS-S

System serial number : FOC1140Y0xx

Top assembly part number : 800-26853-01

Top assembly revision number : C0

Version ID : V05

CLEI Code Number : CNMWN00ARC

Switch 03

---------

Switch Uptime : 17 weeks, 3 days, 1 hour, 17 minutes

Base ethernet MAC Address : 00:1D:E6:94:9D:80

Motherboard assembly number : 73-10216-08

Power supply part number : 341-0108-03

Motherboard serial number : FOC11393CJV

Power supply serial number : DCA1138A0B6

Model revision number : F0

Motherboard revision number : C0

Model number : WS-C3750G-48PS-S

System serial number : FOC1140Y0xx

Top assembly part number : 800-26853-01

Top assembly revision number : C0

Version ID : V05

CLEI Code Number : CNMWN00ARC

Configuration register is 0xF

company-core-sw1#

company-core-sw1#sh switch de

Switch/Stack Mac Address : 001d.e62a.3300

H/W Current

Switch# Role Mac Address Priority Version State

----------------------------------------------------------

*1 Master 001d.e62a.3300 1 0 Ready

2 Member 001d.e694.9c00 1 0 Ready

3 Member 001d.e694.9d80 1 0 Ready

Stack Port Status Neighbors

Switch# Port 1 Port 2 Port 1 Port 2

--------------------------------------------------------

1 Ok Ok 2 3

2 Ok Ok 3 1

3 Ok Ok 1 2

company-core-sw1#sh int gig1/0/11 (The storage iSCSI Storage NIC on VMware Host)

GigabitEthernet1/0/11 is up, line protocol is up (connected)

Hardware is Gigabit Ethernet, address is 001d.e62a.330b (bia 001d.e62a.330b)

Description: Storage NIC on VMware Server

MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX

input flow-control is off, output flow-control is unsupported

ARP type: ARPA, ARP Timeout 04:00:00

Last input never, output 00:00:01, output hang never

Last clearing of "show interface" counters never

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 12721

Queueing strategy: fifo

Output queue: 0/40 (size/max)

30 second input rate 719000 bits/sec, 160 packets/sec

30 second output rate 1880000 bits/sec, 206 packets/sec

6126320351 packets input, 5310396369184 bytes, 0 no buffer

Received 5273817 broadcasts (8 multicasts)

0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 watchdog, 8 multicast, 0 pause input

0 input packets with dribble condition detected

6501737456 packets output, 7042556624904 bytes, 0 underruns

0 output errors, 0 collisions, 1 interface resets

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier, 0 PAUSE output

0 output buffer failures, 0 output buffers swapped out

company-core-sw1#sh controller e gig1/0/11

Transmit GigabitEthernet1/0/11 Receive

2929158919 Bytes 1748576307 Bytes

2196853825 Unicast frames 1826080377 Unicast frames

5125955 Multicast frames 8 Multicast frames

2656872 Broadcast frames 4419862 Broadcast frames

0 Too old frames 1390594214 Unicast bytes

0 Deferred frames 760 Multicast bytes

0 MTU exceeded frames 357981333 Broadcast bytes

0 1 collision frames 0 Alignment errors

0 2 collision frames 0 FCS errors

0 3 collision frames 0 Oversize frames

0 4 collision frames 0 Undersize frames

0 5 collision frames 0 Collision fragments

0 6 collision frames

0 7 collision frames 24508 Minimum size frames

0 8 collision frames 2669518967 65 to 127 byte frames

0 9 collision frames 609557 128 to 255 byte frames

0 10 collision frames 1175415 256 to 511 byte frames

0 11 collision frames 90022308 512 to 1023 byte frames

0 12 collision frames 3364116788 1024 to 1518 byte frames

0 13 collision frames 0 Overrun frames

0 14 collision frames 0 Pause frames

0 15 collision frames

0 Excessive collisions 0 Symbol error frames

0 Late collisions 0 Invalid frames, too large

0 VLAN discard frames 0 Valid frames, too large

0 Excess defer frames 0 Invalid frames, too small

5910565 64 byte frames 0 Valid frames, too small

1878593132 127 byte frames

1442262 255 byte frames 0 Too old frames

1996092 511 byte frames 0 Valid oversize frames

63480450 1023 byte frames 0 System FCS error frames

253214151 1518 byte frames 0 RxPortFifoFull drop frame

0 Too large frames

0 Good (1 coll) frames

0 Good (>1 coll) frames

company-core-sw1#sh int gig2/0/46 (The iSCSI connection port on EMC SAN)

GigabitEthernet2/0/46 is up, line protocol is up (connected)

Hardware is Gigabit Ethernet, address is 001d.e694.9c2e (bia 001d.e694.9c2e)

Description: StoreageSPB0

MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,

reliability 255/255, txload 28/255, rxload 1/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX

input flow-control is off, output flow-control is unsupported

ARP type: ARPA, ARP Timeout 04:00:00

Last input never, output 00:00:57, output hang never

Last clearing of "show interface" counters never

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 447743

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 4386000 bits/sec, 5317 packets/sec

5 minute output rate 111006000 bits/sec, 9876 packets/sec

70264947933 packets input, 95020372996692 bytes, 0 no buffer

Received 23637 broadcasts (22 multicasts)

0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 watchdog, 22 multicast, 0 pause input

0 input packets with dribble condition detected

26206687475 packets output, 18875893941277 bytes, 0 underruns

0 output errors, 0 collisions, 1 interface resets

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier, 0 PAUSE output

0 output buffer failures, 0 output buffers swapped out

company-core-sw1#sh controller e gig2/0/46

Transmit GigabitEthernet2/0/46 Receive

1886654464 Bytes 1271819202 Bytes

442461703 Unicast frames 3104585442 Unicast frames

5126050 Multicast frames 22 Multicast frames

7055601 Broadcast frames 21243 Broadcast frames

0 Too old frames 1268210828 Unicast bytes

0 Deferred frames 2680 Multicast bytes

0 MTU exceeded frames 3605694 Broadcast bytes

0 1 collision frames 0 Alignment errors

0 2 collision frames 0 FCS errors

0 3 collision frames 0 Oversize frames

0 4 collision frames 0 Undersize frames

0 5 collision frames 0 Collision fragments

0 6 collision frames

0 7 collision frames 178941 Minimum size frames

0 8 collision frames 1214289839 65 to 127 byte frames

0 9 collision frames 145131091 128 to 255 byte frames

0 10 collision frames 721345907 256 to 511 byte frames

0 11 collision frames 328006981 512 to 1023 byte frames

0 12 collision frames 695653948 1024 to 1518 byte frames

0 13 collision frames 0 Overrun frames

0 14 collision frames 0 Pause frames

0 15 collision frames

0 Excessive collisions 0 Symbol error frames

0 Late collisions 0 Invalid frames, too large

0 VLAN discard frames 0 Valid frames, too large

0 Excess defer frames 0 Invalid frames, too small

21584477 64 byte frames 0 Valid frames, too small

3581374897 127 byte frames

85735182 255 byte frames 0 Too old frames

78806669 511 byte frames 0 Valid oversize frames

279429758 1023 byte frames 0 System FCS error frames

702679667 1518 byte frames 0 RxPortFifoFull drop frame

0 Too large frames

0 Good (1 coll) frames

0 Good (>1 coll) frames

company-core-sw1#

Leo Laohoo · ‎11-22-2011

Hmmm ... In the output of the "sh interface " I am seeing "ouput drops". The ports have not had their counters cleared for the last 17 weeks or so.

Output drops is not normal. You only see this when the switch is shoving data out the port (and into the client connected) and the client is saying "stop, I can't process enough, I'm gonna puke". The "puke" is the counters incrementing.

One way of testing whether or not the servers can handle the amount of inbound data from the switch is to force the connection down to 100 Mbps and see if the transfer improves AND whether or not "output drop" counters are incrementing.

As to your cable suspicion, you can run a simple TDR test. Here's how you do it:

1. Command: test cable tdr interface ;

2. Wait for about 5 to 7 seconds for the test to run;

3. Command: sh cable tdr interface ;

4. Post the output.

Read the link I've posted. Just remember that in some cases, TDR is disruptive.

nkarpysh · ‎11-22-2011

Need to add something here,

I see flowcontrol disable so we don't even here from server to drop in output direction. Usually output discards happening when multiple ingress ports sending traffic throughout single interface. Thus that can't send on wire speed and need to buffer. If buffer is full eventually - outdiscards happen. Those also can happen due to bursty traffic - thus not always you need to have traffic of wire speed to cause outdiscards - just exausted buffers.

The other thing is that only particular queue can be affected and that can be tuned with QoS setting to get more buffers.

So first of all - clear counters to make sure we have current drops and that is not historical data and collect following logs to see there the drops are:

- clear counters

- show queueing int gix/y

- 'show mls qos statistics'

- show mls qos int gix/y stat

- show int gix/y switching

Nik

HTH,
Niko

yantiscompany · ‎11-23-2011

Reset the counters, and it looks like the output drops returned in the connection between the switch and the SAN. Ports on the SAN are Gigabit, and all cables are Cat6. There are 3 Gigabit servers ports trying to connect for every 1 gigabit port on the SAN, so that would certainly be the multiple ingress trying to go out the one interface? Most of the articles I read said to disable flowcontrol, but maybe in this case it would help? I certainly don't mind giving these interfaces more buffers, as they are quite important.

Appreciate the help guys!

Here are the outputs you asked for.

company-core-sw1#clear counters

Clear "show interface" counters on all interfaces [confirm]y

company-core-sw1#show queueing int gig1/0/11

Interface GigabitEthernet1/0/11 queueing strategy: none

company-core-sw1#show queueing int gig2/0/46

Interface GigabitEthernet2/0/46 queueing strategy: none

company-core-sw1#sh mls qos int gig1/0/11 sta

GigabitEthernet1/0/11 (All statistics are in packets)

dscp: incoming

-------------------------------

0 - 4 : 1832729103 0 0 0 0

5 - 9 : 0 0 0 0 0

10 - 14 : 0 0 0 0 0

15 - 19 : 0 0 0 0 0

20 - 24 : 0 0 0 0 0

25 - 29 : 0 0 0 0 0

30 - 34 : 0 0 0 0 0

35 - 39 : 0 0 0 0 0

40 - 44 : 0 0 0 0 0

45 - 49 : 0 0 0 0 0

50 - 54 : 0 0 0 0 0

55 - 59 : 0 0 0 0 0

60 - 64 : 0 0 0 0

dscp: outgoing

-------------------------------

0 - 4 : 2203785476 0 0 0 0

5 - 9 : 0 0 0 0 0

10 - 14 : 0 0 0 0 0

15 - 19 : 0 0 0 0 0

20 - 24 : 0 0 0 0 0

25 - 29 : 0 0 0 0 0

30 - 34 : 0 0 0 0 0

35 - 39 : 0 0 0 0 0

40 - 44 : 0 0 0 0 0

45 - 49 : 0 0 0 0 0

50 - 54 : 0 0 0 0 0

55 - 59 : 0 0 0 0 0

60 - 64 : 0 0 0 0

cos: incoming

-------------------------------

0 - 4 : 1838043806 0 0 0 0

5 - 7 : 0 0 0

cos: outgoing

-------------------------------

0 - 4 : 2207001782 0 0 0 0

5 - 7 : 0 0 0

output queues enqueued:

queue: threshold1 threshold2 threshold3

-----------------------------------------------

queue 0: 2 0 0

queue 1: 77 21411100 7201577

queue 2: 0 0 0

queue 3: 0 0 2211926973

output queues dropped:

queue: threshold1 threshold2 threshold3

-----------------------------------------------

queue 0: 0 0 0

queue 1: 0 0 0

queue 2: 0 0 0

queue 3: 0 0 12721

Policer: Inprofile: 0 OutofProfile: 0

company-core-sw1#sh mls qos int gig2/0/46 sta

GigabitEthernet2/0/46 (All statistics are in packets)

dscp: incoming

-------------------------------

0 - 4 : 1856553051 0 0 0 0

5 - 9 : 0 0 0 0 0

10 - 14 : 0 0 0 0 0

15 - 19 : 0 0 0 0 0

20 - 24 : 0 0 0 0 0

25 - 29 : 0 0 0 0 0

30 - 34 : 0 0 0 0 0

35 - 39 : 0 0 0 0 0

40 - 44 : 0 0 0 0 0

45 - 49 : 0 0 0 0 0

50 - 54 : 0 0 0 0 0

55 - 59 : 0 0 0 0 0

60 - 64 : 0 0 0 0

dscp: outgoing

-------------------------------

0 - 4 : 625107487 0 0 0 0

5 - 9 : 0 0 0 0 0

10 - 14 : 0 0 0 0 0

15 - 19 : 0 0 0 0 0

20 - 24 : 0 0 0 0 0

25 - 29 : 0 0 0 0 0

30 - 34 : 0 0 0 0 0

35 - 39 : 0 0 0 0 0

40 - 44 : 0 0 0 0 0

45 - 49 : 0 0 0 0 0

50 - 54 : 0 0 0 0 0

55 - 59 : 0 0 0 0 0

60 - 64 : 0 0 0 0

cos: incoming

-------------------------------

0 - 4 : 1856698502 0 0 0 0

5 - 7 : 0 0 0

cos: outgoing

-------------------------------

0 - 4 : 633627219 0 0 0 0

5 - 7 : 0 0 0

output queues enqueued:

queue: threshold1 threshold2 threshold3

-----------------------------------------------

queue 0: 2 0 0

queue 1: 77 21420991 7201804

queue 2: 0 0 0

queue 3: 0 0 633263809

output queues dropped:

queue: threshold1 threshold2 threshold3

-----------------------------------------------

queue 0: 0 0 0

queue 1: 0 0 0

queue 2: 0 0 0

queue 3: 0 0 448323

Policer: Inprofile: 0 OutofProfile: 0

company-core-sw1#sh int gig1/0/11 switching

GigabitEthernet1/0/11 Storage NIC port on VMware Server

Throttle count 0

Drops RP 0 SP 0

SPD Flushes Fast 0 SSE 0

SPD Aggress Fast 0

SPD Priority Inputs 0 Drops 0

Protocol Path Pkts In Chars In Pkts Out Chars Out

Other Process 0 0 1759771 116187330

Cache misses 0

Fast 0 0 0 0

Auton/SSE 0 0 0 0

Spanning Tree Process 0 0 5265131 315907860

Cache misses 0

Fast 0 0 0 0

Auton/SSE 0 0 0 0

CDP Process 0 0 176846 85416618

Cache misses 0

Fast 0 0 0 0

Auton/SSE 0 0 0 0

company-core-sw1#sh int gig2/0/46 switching

GigabitEthernet2/0/46 iSCSI Storage port on SAN

Throttle count 0

Drops RP 0 SP 0

SPD Flushes Fast 0 SSE 0

SPD Aggress Fast 0

SPD Priority Inputs 0 Drops 0

Protocol Path Pkts In Chars In Pkts Out Chars Out

CDP Process 0 0 176707 85349481

Cache misses 0

Fast 0 0 0 0

Auton/SSE 0 0 0 0

company-core-sw1#

company-core-sw1#sh int gig1/0/11

GigabitEthernet1/0/11 is up, line protocol is up (connected)

Hardware is Gigabit Ethernet, address is 001d.e62a.330b (bia 001d.e62a.330b)

Description: Storage NIC on VMware Server

MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX

input flow-control is off, output flow-control is unsupported

ARP type: ARPA, ARP Timeout 04:00:00

Last input never, output 00:00:00, output hang never

Last clearing of "show interface" counters 00:09:36

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: fifo

Output queue: 0/40 (size/max)

30 second input rate 621000 bits/sec, 101 packets/sec

30 second output rate 900000 bits/sec, 104 packets/sec

55658 packets input, 39422805 bytes, 0 no buffer

Received 287 broadcasts (0 multicasts)

0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 watchdog, 0 multicast, 0 pause input

0 input packets with dribble condition detected

61129 packets output, 62961347 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier, 0 PAUSE output

0 output buffer failures, 0 output buffers swapped out

company-core-sw1#sh int gig2/0/46

GigabitEthernet2/0/46 is up, line protocol is up (connected)

Hardware is Gigabit Ethernet, address is 001d.e694.9c2e (bia 001d.e694.9c2e)

Description: iSCSI Storage port on SAN

MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,

reliability 255/255, txload 48/255, rxload 40/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX

input flow-control is off, output flow-control is unsupported

ARP type: ARPA, ARP Timeout 04:00:00

Last input never, output 00:00:55, output hang never

Last clearing of "show interface" counters 00:10:03

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 73

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 157272000 bits/sec, 21320 packets/sec

5 minute output rate 190044000 bits/sec, 18984 packets/sec

12888577 packets input, 11856220194 bytes, 0 no buffer

Received 0 broadcasts (0 multicasts)

0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 watchdog, 0 multicast, 0 pause input

0 input packets with dribble condition detected

11634444 packets output, 14447713815 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier, 0 PAUSE output

0 output buffer failures, 0 output buffers swapped out

company-core-sw1#

nkarpysh · ‎11-23-2011

Hello,

I see most drops happening in queue4. That is bigest queue you have but possibly still you can find some rom to increase the buffers for it reducing those for other queues.

mls qos queue-set output 1 buffers 10 10 26 54

mls qos queue-set output 2 buffers 16 6 17 61

These are the commands responsible for buffers in your two queue-sets. For now you have 2 queue-sets configured but none applied to interface. Thus by default queue-set 1 is used. You can apply queue-set 2 to affected interfaces as it has more buffers for queue4 to see how it works.

Just one pre-cation - you have packets enqueued in queue 2 and 4:

queue: threshold1 threshold2 threshold3

-----------------------------------------------

queue 0: 2 0 0

queue 1: 77 21411100 7201577

queue 2: 0 0 0

queue 3: 0 0 2211926973

Thus can be a good idea to have queue2 not decreased at least (so queue set 2 can be adjusted to keep queue 2 at 10%).

Hope this helps.

Nik

HTH,
Niko

yantiscompany · ‎11-28-2011

Thanks for the help. I've gotten to the point where I have it set to "mls qos queue-set output 2 buffers 1 10 1 88", and I am still getting output queues dropped on the 4th queue (queue 3). Should I keep going and start taking away from the second queue, or is there something else I can do?

To make the change I did:

company-core-sw1#config t

company-core-sw1(config)#mls qos queue-set output 2 buffers 1 10 1 88

company-core-sw1(config)#int gig2/0/46

company-core-sw1(config-if)#queue-set 2

I then cleared the counters, and sure enough after a few minutes I see output queues dropped on queue 3 threshold 3. Did I do this right?

yantiscompany · ‎11-28-2011

Also, I was curious as to why the buffers are filling up? Looking at the link graph utilization on the ports showing dropped queues, the max the graph is showing is about 75% with a 60% average. Does this mean the the Server/SAN connected to the port is not able to process even 75% of the traffic on a gigabit port, or am I looking at this completely the wrong way?

nkarpysh · ‎11-28-2011

Hi Justin,

What you see is a bursty traffic. Average rate you noticed is 60-70% of the wire speed. But it can go memntally much higher and lower. When it goes higher - it creates a burst and interface is not able to store all that thus drops happening. As you have almost all traffic in queue4 - I guess buffer tuning will not help here.

I need to think of splitting that traffic to a different ports or bundle that connection in a port-channel to increase throughput.

The other thing to try is move some DSCps from that queue:

mls qos srr-queue output dscp-map queue 1 threshold 3 40 41 42 43 44 45 46 47

mls qos srr-queue output dscp-map queue 2 threshold 3 24 25 26 27 28 29 30 31

mls qos srr-queue output dscp-map queue 2 threshold 3 48 49 50 51 52 53 54 55

mls qos srr-queue output dscp-map queue 2 threshold 3 56 57 58 59 60 61 62 63

mls qos srr-queue output dscp-map queue 3 threshold 3 16 17 18 19 20 21 22 23

mls qos srr-queue output dscp-map queue 3 threshold 3 32 33 34 35 36 37 38 39

mls qos srr-queue output dscp-map queue 4 threshold 1 8

mls qos srr-queue output dscp-map queue 4 threshold 2 9 10 11 12 13 14 15

mls qos srr-queue output dscp-map queue 4 threshold 3 0 1 2 3 4 5 6 7

You see you have lower DSCP values assisgned to it so you may try to move some to other queus and see if it helps. But I think you get DSCP0 there and issue will follow it to the other queues. Thus may be a good idea to understand what is that traffic and mark a part of it on ingress with a different value.

Anyway - as you said rate of interface is 60-70% - this is always of a high posibility to have bursts even with correct DSCP assigned. Some queues will drop. With this type of traffic - you need to increase the link to that server as you just oversubscribing it momentarilly.

Nik

HTH,
Niko