10-26-2018 02:07 AM - edited 03-08-2019 04:28 PM
Hi,
We see a problem with our 9300-48T switches.
Client devices mostly use intel NIC's (I219-V for example). Links are 1Gbit Full duplex, autonegotiate.
When a client copies data from network share A to local disk he gets steady 1Gbit download speed.
When a client copies data from local disk to network share B he gets steady 1Gbit upload speed.
When client copies data from network share A to network share B he gets about 500Mbit upload and 500Mbit download.
at the same time the switch starts Increasing "Total output drops" and "Excess Defer frames" counters. Please note that these counters are equal to each other, so Excess Defer is the only reason why output frames are being dropped.
swfarm01#sh interfaces GigabitEthernet 1/0/1 controller
GigabitEthernet1/0/1 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 68ca.e499.7d81 (bia 68ca.e499.7d81)
Description: Bder12
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 955839364
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 46000 bits/sec, 5 packets/sec
5 minute output rate 4000 bits/sec, 4 packets/sec
5938362788 packets input, 8175427171217 bytes, 0 no buffer
Received 298662 broadcasts (275191 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 275191 multicast, 0 pause input
0 input packets with dribble condition detected
2362141175 packets output, 2461912216984 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
35572 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
Transmit GigabitEthernet1/0/1 Receive
2461912216984 Total bytes 8175427171217 Total bytes
2346111759 Unicast frames 5938064126 Unicast frames
2460045343032 Unicast bytes 8175376018743 Unicast bytes
13673248 Multicast frames 275191 Multicast frames
1497859827 Multicast bytes 46033210 Multicast bytes
2356168 Broadcast frames 23471 Broadcast frames
369014125 Broadcast bytes 5119264 Broadcast bytes
0 System FCS error frames 0 IpgViolation frames
0 MacUnderrun frames 0 MacOverrun frames
0 Pause frames 0 Pause frames
0 Cos 0 Pause frames 0 Cos 0 Pause frames
0 Cos 1 Pause frames 0 Cos 1 Pause frames
0 Cos 2 Pause frames 0 Cos 2 Pause frames
0 Cos 3 Pause frames 0 Cos 3 Pause frames
0 Cos 4 Pause frames 0 Cos 4 Pause frames
0 Cos 5 Pause frames 0 Cos 5 Pause frames
0 Cos 6 Pause frames 0 Cos 6 Pause frames
0 Cos 7 Pause frames 0 Cos 7 Pause frames
0 Oam frames 0 OamProcessed frames
0 Oam frames 0 OamDropped frames
576308413 Minimum size frames 439499728 Minimum size frames
136856374 65 to 127 byte frames 72238453 65 to 127 byte frames
39493563 128 to 255 byte frames 22431829 128 to 255 byte frames
12634989 256 to 511 byte frames 18315630 256 to 511 byte frames
4321021 512 to 1023 byte frames 5146451 512 to 1023 byte frames
1592526823 1024 to 1518 byte frames 5380730701 1024 to 1518 byte frames
0 1519 to 2047 byte frames 0 1519 to 2047 byte frames
0 2048 to 4095 byte frames 0 2048 to 4095 byte frames
0 4096 to 8191 byte frames 0 4096 to 8191 byte frames
0 8192 to 16383 byte frames 0 8192 to 16383 byte frames
0 16384 to 32767 byte frame 0 16384 to 32767 byte frame
0 > 32768 byte frames 0 > 32768 byte frames
0 Late collision frames 0 SymbolErr frames
955839364 Excess Defer frames 0 Collision fragments
0 Good (1 coll) frames 0 ValidUnderSize frames
0 Good (>1 coll) frames 0 InvalidOverSize frames
0 Deferred frames 0 ValidOverSize frames
0 Gold frames dropped 0 FcsErr frames
We can also see with "sh platform hardware fed switch 1 qos queue stats interface gigabitEthernet 1/0/1" command that Drop-TH2 counter for Queue 1 is equal to "Total output drops" and "Excess Defer frames" counters.
DATA Port:1 Drop Counters
-------------------------------
Queue Drop-TH0 Drop-TH1 Drop-TH2 SBufDrop QebDrop
----- ----------- ----------- ----------- ----------- -----------
0 0 0 0 0 0
1 0 0 955839364 0 0
2 0 0 0 0 0
3 0 0 0 0 0
4 0 0 0 0 0
5 0 0 0 0 0
6 0 0 0 0 0
7 0 0 0 0 0
interface configuration is as basic as it can be:
interface GigabitEthernet1/0/1
switchport access vlan 1903
switchport mode access
no cdp enable
!
Configuring "qos queue-softmax-multiplier 1200" improved the situation a bit: the client now gets 700-1000Mbit speed in a saw-like pattern (see attachments). Frame drop counters are not increasing constantly anymore, but in small bursts, and these bursts are synchronized with speed drops seen by client. That seems to happen across all 9300 switches we have, on every port. Their configurations are pretty much equal.
How can we get true 1Gbit Full duplex on our clients, without observing that huge amount of frame drops?
10-26-2018 07:17 AM
There was a recent discussion in the community about output drops and excess defer frames. The explanation was that the traffic being sent over the link was greater than the capacity of the link. The suggestion for improvement was either to try changing buffers on the switch or to deploy qos. Since your first attempt at sos seems to have helped, perhaps you want to try a more extensive sos implementation. Here is the link if you want to see that discussion
HTH
Rick
10-29-2018 02:01 AM
10-26-2018 08:03 AM
Hello,
in addition to Richard's post, try and increase the hold queue on the interface:
interface GigabitEthernet1/0/1
switchport access vlan 1903
switchport mode access
no cdp enable
hold-queue out 2000
10-29-2018 01:53 AM
10-29-2018 02:23 AM - edited 10-29-2018 02:53 AM
Just to clarify: The client PC connected to the switch port can easily reach 1Gbps when it only receives, or only transmits. The problem appears when the PC transmits and receives traffic at the same time. In that case the download speed is affected by the fact that the client transmits data at the same time which should not happen with full-duplex links.
One way to reproduce I gave in my original post: initiate file copy on the client PC from Server A to Server B. In that case speed is capped at ~550Mbit each way and receive speed equals transmit speed. (1-1.1Gbps). "Excess defer" counter is growing
Another way to reproduce the issue is to start uploading data to a Server A from PC, and downloading data from server B to PC at the same time. In that case we observe 1Gbit upload speed and 200Mbit download speed. We can see that "excess defer" counter is growing. As soon as the upload is finished (or cancelled) the download speed jumps up to 1Gbit and "Excess defer" counter stops growing.
Both of these methods show that upload traffic affects download traffic which shouldn't happen on full duplex-links. Links are not over-subscribed. The issue is seen on ~10x 9300 stacks and on hundreds of clients all over the campus.
The issue does not affect our Force10 switches that are in the same network. The same client PC constantly gets 2Gbps aggregate bandwidth while communicating with the same servers while being connected to Force10 switch.
12-04-2018 11:19 AM
Hi
Do you have a policy-map configured on the device?
Can you try to enable a policy-map on an interface? this example should enable the use of the soft-max configuration you mentioned earlier.
// Example
conf t
policy-map test
exit
interface gx/y
service-policy output test
end
Then verify with the commands to see the queues.
Additionally, can you share a show run?
I also share an interesting link related to output drops on 3850
BR
Gaston
12-05-2018 01:56 AM
Adding policy map didn't affect anything.
Building configuration... Current configuration : 12270 bytes ! ! Last configuration change at 11:50:37 EET Wed Dec 5 2018 by ! NVRAM config last updated at 13:56:48 EET Fri Nov 2 2018 by ! version 16.6 no service pad service timestamps debug datetime msec localtime service timestamps log datetime msec localtime service password-encryption no platform punt-keepalive disable-kernel-core ! hostname X ! ! vrf definition Mgmt-vrf ! address-family ipv4 exit-address-family ! address-family ipv6 exit-address-family ! ! aaa new-model ! ! aaa group server radius userauth server-private X auth-port 1812 key 7 server-private X auth-port 1812 key 7 ! aaa authentication login default group userauth local aaa authorization console aaa authorization exec default group userauth local if-authenticated ! ! ! ! ! ! aaa session-id common clock timezone EET 2 0 clock summer-time EEST recurring last Sun Mar 3:00 last Sun Oct 4:00 switch 1 provision c9300-48t ! ! ! ! ! ip domain name X ! ! ! ! ! ! ! ! qos queue-softmax-multiplier 1200 cpp system-default ! ! ! ! ! diagnostic bootup level minimal spanning-tree mode rapid-pvst spanning-tree portfast default spanning-tree portfast bpduguard default spanning-tree extend system-id archive path X write-memory time-period 1440 errdisable recovery cause udld errdisable recovery cause bpduguard errdisable recovery cause security-violation errdisable recovery cause channel-misconfig errdisable recovery cause pagp-flap errdisable recovery cause dtp-flap errdisable recovery cause link-flap errdisable recovery cause sfp-config-mismatch errdisable recovery cause gbic-invalid errdisable recovery cause l2ptguard errdisable recovery cause psecure-violation errdisable recovery cause port-mode-failure errdisable recovery cause dhcp-rate-limit errdisable recovery cause pppoe-ia-rate-limit errdisable recovery cause mac-limit errdisable recovery cause vmps errdisable recovery cause storm-control errdisable recovery cause inline-power errdisable recovery cause arp-inspection errdisable recovery cause link-monitor-failure errdisable recovery cause oam-remote-failure errdisable recovery cause loopback errdisable recovery cause psp ! username admin privilege 15 secret 5 X ! redundancy mode sso ! ! ! class-map match-any system-cpp-police-topology-control description Topology control class-map match-any system-cpp-police-sw-forward description Sw forwarding, L2 LVX data, LOGGING class-map match-any system-cpp-default description DHCP Snooping, EWLC control, EWCL data class-map match-any system-cpp-police-sys-data description Learning cache ovfl, Crypto Control, Exception, EGR Exception, NFL SAMPLED DATA, Gold Pkt, RPF Failed class-map match-any system-cpp-police-punt-webauth description Punt Webauth class-map match-any system-cpp-police-l2lvx-control description L2 LVX control packets class-map match-any system-cpp-police-forus description Forus Address resolution and Forus traffic class-map match-any system-cpp-police-multicast-end-station description MCAST END STATION class-map match-any system-cpp-police-multicast description Transit Traffic and MCAST Data class-map match-any system-cpp-police-l2-control description L2 control class-map match-any system-cpp-police-dot1x-auth description DOT1X Auth class-map match-any system-cpp-police-data description ICMP redirect, ICMP_GEN and BROADCAST class-map match-any system-cpp-police-stackwise-virt-control description Stackwise Virtual class-map match-any system-cpp-police-control-low-priority description General punt class-map match-any non-client-nrt-class class-map match-any system-cpp-police-routing-control description Routing control class-map match-any system-cpp-police-protocol-snooping description Protocol snooping ! policy-map system-cpp-policy class system-cpp-police-data police rate 600 pps class system-cpp-police-sys-data police rate 100 pps class system-cpp-police-sw-forward police rate 1000 pps class system-cpp-police-multicast police rate 500 pps class system-cpp-police-multicast-end-station police rate 2000 pps class system-cpp-police-punt-webauth police rate 1000 pps class system-cpp-police-l2-control police rate 2000 pps class system-cpp-police-stackwise-virt-control police rate 8000 pps class system-cpp-police-routing-control police rate 5400 pps class system-cpp-police-control-low-priority police rate 200 pps class system-cpp-police-l2lvx-control police rate 1000 pps class system-cpp-police-topology-control police rate 13000 pps class system-cpp-police-dot1x-auth police rate 1000 pps class system-cpp-police-protocol-snooping police rate 2000 pps class system-cpp-police-forus police rate 1000 pps class system-cpp-default police rate 1000 pps ! ! ! ! ! ! ! ! ! ! ! ! ! ! interface Port-channel1 description to X switchport mode trunk ! interface GigabitEthernet0/0 vrf forwarding Mgmt-vrf no ip address shutdown speed 1000 negotiation auto ! interface GigabitEthernet1/0/1 description X switchport access vlan 1903 switchport mode access no cdp enable ! interface GigabitEthernet1/0/2 description X switchport access vlan 1903 switchport mode access no cdp enable ! interface GigabitEthernet1/0/3 description X switchport access vlan 1903 switchport mode access no cdp enable ! interface GigabitEthernet1/0/4 description X switchport access vlan 1903 switchport mode access no cdp enable ! interface GigabitEthernet1/0/5 description X switchport access vlan 1903 switchport mode access no cdp enable ! interface GigabitEthernet1/0/6 description X switchport access vlan 1903 switchport mode access no cdp enable ! interface GigabitEthernet1/0/7 description X switchport access vlan 1903 switchport mode access no cdp enable ! interface GigabitEthernet1/0/8 description X switchport access vlan 1903 switchport mode access no cdp enable ! interface GigabitEthernet1/0/9 description X switchport access vlan 1903 switchport mode access no cdp enable ! interface GigabitEthernet1/0/10 description X switchport access vlan 1903 switchport mode access no cdp enable ! interface GigabitEthernet1/0/11 description X switchport access vlan 1903 switchport mode access no cdp enable ! interface GigabitEthernet1/0/12 description X switchport access vlan 1903 switchport mode access no cdp enable ! interface GigabitEthernet1/0/13 description X switchport access vlan 1903 switchport mode access no cdp enable ! interface GigabitEthernet1/0/14 description X switchport access vlan 1903 switchport mode access no cdp enable ! interface GigabitEthernet1/0/15 description X switchport access vlan 1903 switchport mode access no cdp enable ! interface GigabitEthernet1/0/16 description X switchport access vlan 1903 switchport mode access no cdp enable ! interface GigabitEthernet1/0/17 description X switchport access vlan 1903 switchport mode access no cdp enable ! interface GigabitEthernet1/0/18 switchport access vlan 1903 switchport mode access no cdp enable ! interface GigabitEthernet1/0/19 description X switchport access vlan 2003 switchport mode access no cdp enable ! interface GigabitEthernet1/0/20 description X switchport access vlan 2003 switchport mode access no cdp enable ! interface GigabitEthernet1/0/21 description X switchport access vlan 1903 switchport mode access no cdp enable ! interface GigabitEthernet1/0/22 description X switchport access vlan 2003 switchport mode access no cdp enable ! interface GigabitEthernet1/0/23 description X switchport access vlan 1903 switchport mode access no cdp enable ! interface GigabitEthernet1/0/24 description X switchport access vlan 1903 switchport mode access no cdp enable ! interface GigabitEthernet1/0/25 description X switchport access vlan 1903 switchport mode access no cdp enable ! interface GigabitEthernet1/0/26 description X switchport access vlan 1903 switchport mode access no cdp enable ! interface GigabitEthernet1/0/27 description X switchport access vlan 2003 switchport mode access no cdp enable ! interface GigabitEthernet1/0/28 description X switchport access vlan 1903 switchport mode access no cdp enable ! interface GigabitEthernet1/0/29 description X switchport access vlan 1903 switchport mode access no cdp enable ! interface GigabitEthernet1/0/30 description X switchport access vlan 1903 switchport mode access no cdp enable ! interface GigabitEthernet1/0/31 description X switchport access vlan 2003 switchport mode access no cdp enable ! interface GigabitEthernet1/0/32 description X switchport access vlan 1903 switchport mode access no cdp enable ! interface GigabitEthernet1/0/33 description X switchport access vlan 1903 switchport mode access no cdp enable ! interface GigabitEthernet1/0/34 description X switchport access vlan 1903 switchport mode access no cdp enable ! interface GigabitEthernet1/0/35 switchport access vlan 1903 switchport mode access no cdp enable ! interface GigabitEthernet1/0/36 description X switchport access vlan 2001 switchport mode access no cdp enable ! interface GigabitEthernet1/0/37 description X switchport access vlan 1903 switchport mode access no cdp enable ! interface GigabitEthernet1/0/38 description X switchport access vlan 1903 switchport mode access no cdp enable ! interface GigabitEthernet1/0/39 description X switchport access vlan 1903 switchport mode access no cdp enable ! interface GigabitEthernet1/0/40 description X switchport access vlan 1903 switchport mode access no cdp enable ! interface GigabitEthernet1/0/41 description X switchport access vlan 1903 switchport mode access no cdp enable ! interface GigabitEthernet1/0/42 description X switchport access vlan 1903 switchport mode access no cdp enable ! interface GigabitEthernet1/0/43 description X switchport access vlan 2001 switchport mode access no cdp enable ! interface GigabitEthernet1/0/44 switchport access vlan 1903 switchport mode access no cdp enable ! interface GigabitEthernet1/0/45 switchport access vlan 1903 switchport mode access no cdp enable ! interface GigabitEthernet1/0/46 description cisco test switchport access vlan 2003 switchport mode access no cdp enable ! interface GigabitEthernet1/0/47 switchport access vlan 1903 switchport mode access no cdp enable ! interface GigabitEthernet1/0/48 switchport access vlan 1903 switchport mode access no cdp enable ! interface GigabitEthernet1/1/1 ! interface GigabitEthernet1/1/2 ! interface GigabitEthernet1/1/3 ! interface GigabitEthernet1/1/4 ! interface TenGigabitEthernet1/1/1 ! interface TenGigabitEthernet1/1/2 ! interface TenGigabitEthernet1/1/3 ! interface TenGigabitEthernet1/1/4 ! interface TenGigabitEthernet1/1/5 ! interface TenGigabitEthernet1/1/6 ! interface TenGigabitEthernet1/1/7 ! interface TenGigabitEthernet1/1/8 ! interface FortyGigabitEthernet1/1/1 switchport mode trunk channel-group 1 mode desirable ! interface FortyGigabitEthernet1/1/2 switchport mode trunk channel-group 1 mode desirable ! interface Vlan1 no ip address shutdown ! interface Vlan9 ip address X ! ip default-gateway X ip forward-protocol nd no ip http server no ip http secure-server ip ssh version 2 ! ! logging trap debugging logging facility local0 logging host X ! snmp-server community X RO ! ! ! ! control-plane service-policy input system-cpp-policy ! ! line con 0 stopbits 1 line vty 5 15 ! ntp server X ntp server X prefer ! wsma agent exec ! wsma agent config ! wsma agent filesys ! wsma agent notify ! ! end
12-05-2018 02:11 AM
So after more investigation the problem is a bit different:
applying "qos queue-softmax-multiplier 1200" makes everything better, so it is configured now on all our switches.
The problem still exists, but at a lower scale:
If a port has only ingress or only egress traffic then we see 1 Gigabit throughput, and the graph is flat, and no output drops.
If a port has both ingress and egress traffic then the total throughput is less that 2Gigabit and you can see on the graph that traffic is "fighting" for throughput (see attached file). Output drops are seen in this scenario.
It is Interesting that Intel NIC's are quite close to 2Gbit aggregate, but Realtek NICs that we have stay at around 1.4Gbit aggregate.
The same behavior is seen at our Force10 switch, so the problem is not switch dependent anymore.
Could it be that even though all consumer NICs are marketed as 1Gbit full duplex they actually can't hold 2Gbit aggregate, and start showing that weird behavior?
12-05-2018 05:42 AM
03-17-2021 04:00 AM
@IT7 we've been reading your thread with great interest as we are facing the exact same problem on a Catalyst 9500 SWV pair. setting the "qos queue-softmax-multiplier 1200" gave us also an increase of factor 2-3 on the transfer speed.
Did you ever manage to solve this problem in the end?
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide