Excess Defer frames on full-duplex link on 9300

[IT7]

Hi,

We see a problem with our 9300-48T switches.

Client devices mostly use intel NIC's (I219-V for example). Links are 1Gbit Full duplex, autonegotiate.

When a client copies data from network share A to local disk he gets steady 1Gbit download speed.

When a client copies data from local disk to network share B he gets steady 1Gbit upload speed.

When client copies data from network share A to network share B he gets about 500Mbit upload and 500Mbit download.

at the same time the switch starts Increasing "Total output drops" and "Excess Defer frames" counters. Please note that these counters are equal to each other, so Excess Defer is the only reason why output frames are being dropped.

swfarm01#sh interfaces GigabitEthernet 1/0/1 controller
GigabitEthernet1/0/1 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 68ca.e499.7d81 (bia 68ca.e499.7d81)
Description: Bder12
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 955839364
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 46000 bits/sec, 5 packets/sec
5 minute output rate 4000 bits/sec, 4 packets/sec
5938362788 packets input, 8175427171217 bytes, 0 no buffer
Received 298662 broadcasts (275191 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 275191 multicast, 0 pause input
0 input packets with dribble condition detected
2362141175 packets output, 2461912216984 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
35572 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
Transmit GigabitEthernet1/0/1 Receive
2461912216984 Total bytes 8175427171217 Total bytes
2346111759 Unicast frames 5938064126 Unicast frames
2460045343032 Unicast bytes 8175376018743 Unicast bytes
13673248 Multicast frames 275191 Multicast frames
1497859827 Multicast bytes 46033210 Multicast bytes
2356168 Broadcast frames 23471 Broadcast frames
369014125 Broadcast bytes 5119264 Broadcast bytes
0 System FCS error frames 0 IpgViolation frames
0 MacUnderrun frames 0 MacOverrun frames
0 Pause frames 0 Pause frames
0 Cos 0 Pause frames 0 Cos 0 Pause frames
0 Cos 1 Pause frames 0 Cos 1 Pause frames
0 Cos 2 Pause frames 0 Cos 2 Pause frames
0 Cos 3 Pause frames 0 Cos 3 Pause frames
0 Cos 4 Pause frames 0 Cos 4 Pause frames
0 Cos 5 Pause frames 0 Cos 5 Pause frames
0 Cos 6 Pause frames 0 Cos 6 Pause frames
0 Cos 7 Pause frames 0 Cos 7 Pause frames
0 Oam frames 0 OamProcessed frames
0 Oam frames 0 OamDropped frames
576308413 Minimum size frames 439499728 Minimum size frames
136856374 65 to 127 byte frames 72238453 65 to 127 byte frames
39493563 128 to 255 byte frames 22431829 128 to 255 byte frames
12634989 256 to 511 byte frames 18315630 256 to 511 byte frames
4321021 512 to 1023 byte frames 5146451 512 to 1023 byte frames
1592526823 1024 to 1518 byte frames 5380730701 1024 to 1518 byte frames
0 1519 to 2047 byte frames 0 1519 to 2047 byte frames
0 2048 to 4095 byte frames 0 2048 to 4095 byte frames
0 4096 to 8191 byte frames 0 4096 to 8191 byte frames
0 8192 to 16383 byte frames 0 8192 to 16383 byte frames
0 16384 to 32767 byte frame 0 16384 to 32767 byte frame
0 > 32768 byte frames 0 > 32768 byte frames
0 Late collision frames 0 SymbolErr frames
955839364 Excess Defer frames 0 Collision fragments
0 Good (1 coll) frames 0 ValidUnderSize frames
0 Good (>1 coll) frames 0 InvalidOverSize frames
0 Deferred frames 0 ValidOverSize frames
0 Gold frames dropped 0 FcsErr frames

We can also see with "sh platform hardware fed switch 1 qos queue stats interface gigabitEthernet 1/0/1" command that Drop-TH2 counter for Queue 1 is equal to "Total output drops" and "Excess Defer frames" counters.

DATA Port:1 Drop Counters
-------------------------------
Queue Drop-TH0 Drop-TH1 Drop-TH2 SBufDrop QebDrop
----- ----------- ----------- ----------- ----------- -----------
0 0 0 0 0 0
1 0 0 955839364 0 0
2 0 0 0 0 0
3 0 0 0 0 0
4 0 0 0 0 0
5 0 0 0 0 0
6 0 0 0 0 0
7 0 0 0 0 0

interface configuration is as basic as it can be:

interface GigabitEthernet1/0/1
switchport access vlan 1903
switchport mode access
no cdp enable
!

Configuring "qos queue-softmax-multiplier 1200" improved the situation a bit: the client now gets 700-1000Mbit speed in a saw-like pattern (see attachments). Frame drop counters are not increasing constantly anymore, but in small bursts, and these bursts are synchronized with speed drops seen by client. That seems to happen across all 9300 switches we have, on every port. Their configurations are pretty much equal.

How can we get true 1Gbit Full duplex on our clients, without observing that huge amount of frame drops?

[Richard Burts]

There was a recent discussion in the community about output drops and excess defer frames. The explanation was that the traffic being sent over the link was greater than the capacity of the link. The suggestion for improvement was either to try changing buffers on the switch or to deploy qos. Since your first attempt at sos seems to have helped, perhaps you want to try a more extensive sos implementation. Here is the link if you want to see that discussion

https://community.cisco.com/t5/switching/catalyst-3850-high-total-output-drops-and-output-errors/td-p/3726497

HTH

Rick

[IT7]

Hi,

It is hard to say if the OP of the post that you pointed me to had exactly the same issue or not, as he never stated what transfer speeds they get, and if all of his links were fully utilized, so it's hard to say if he even had an issue, or everything was working up to the spec. Moreover, there was no solution provided that could help to improve the situation.

[Georg Pauwen]

Hello,

in addition to Richard's post, try and increase the hold queue on the interface:

interface GigabitEthernet1/0/1
switchport access vlan 1903
switchport mode access
no cdp enable

hold-queue out 2000

[IT7]

Hi,

Thank you for your suggestion. Unfortunately it did not help. With "qos queue-softmax-multiplier 1200" and "hold-queue 2000 out" configured the speed was very unstable, sometimes it was dropping to 0 and the max bandwidth observed was ~600Mbps one way (aggregate 1.2Gbps).

With "qos queue-softmax-multiplier" set to default configuring "hold-queue 2000 out" did not make any difference, and we were still observing ~500Mbps one way (1Gbps aggregate).

[IT7]

Just to clarify: The client PC connected to the switch port can easily reach 1Gbps when it only receives, or only transmits. The problem appears when the PC transmits and receives traffic at the same time. In that case the download speed is affected by the fact that the client transmits data at the same time which should not happen with full-duplex links.

One way to reproduce I gave in my original post: initiate file copy on the client PC from Server A to Server B. In that case speed is capped at ~550Mbit each way and receive speed equals transmit speed. (1-1.1Gbps). "Excess defer" counter is growing
Another way to reproduce the issue is to start uploading data to a Server A from PC, and downloading data from server B to PC at the same time. In that case we observe 1Gbit upload speed and 200Mbit download speed. We can see that "excess defer" counter is growing. As soon as the upload is finished (or cancelled) the download speed jumps up to 1Gbit and "Excess defer" counter stops growing.

Both of these methods show that upload traffic affects download traffic which shouldn't happen on full duplex-links. Links are not over-subscribed. The issue is seen on ~10x 9300 stacks and on hundreds of clients all over the campus.

The issue does not affect our Force10 switches that are in the same network. The same client PC constantly gets 2Gbps aggregate bandwidth while communicating with the same servers while being connected to Force10 switch.

[gaston.benitez]

Hi

Do you have a policy-map configured on the device?

Can you try to enable a policy-map on an interface? this example should enable the use of the soft-max configuration you mentioned earlier.

// Example

conf t

policy-map test

exit

interface gx/y

 service-policy output test

end

Then verify with the commands to see the queues.

Additionally, can you share a show run?

I also share an interesting link related to output drops on 3850

https://www.cisco.com/c/en/us/support/docs/switches/catalyst-3850-series-switches/200594-Catalyst-3850-Troubleshooting-Output-dr.html

BR

Gaston

[IT7]

Adding policy map didn't affect anything.

Building configuration...

Current configuration : 12270 bytes
!
! Last configuration change at 11:50:37 EET Wed Dec 5 2018 by 
! NVRAM config last updated at 13:56:48 EET Fri Nov 2 2018 by 
!
version 16.6
no service pad
service timestamps debug datetime msec localtime
service timestamps log datetime msec localtime
service password-encryption
no platform punt-keepalive disable-kernel-core
!
hostname X
!
!
vrf definition Mgmt-vrf
 !
 address-family ipv4
 exit-address-family
 !
 address-family ipv6
 exit-address-family
!
!
aaa new-model
!
!
aaa group server radius userauth
 server-private X auth-port 1812 key 7 
 server-private X auth-port 1812 key 7 
!
aaa authentication login default group userauth local
aaa authorization console
aaa authorization exec default group userauth local if-authenticated
!
!
!
!
!
!
aaa session-id common
clock timezone EET 2 0
clock summer-time EEST recurring last Sun Mar 3:00 last Sun Oct 4:00
switch 1 provision c9300-48t
!
!
!
!
!
ip domain name X
!
!
!
!
!
!
!
!
qos queue-softmax-multiplier 1200
cpp system-default
!
!
!
!
!
diagnostic bootup level minimal
spanning-tree mode rapid-pvst
spanning-tree portfast default
spanning-tree portfast bpduguard default
spanning-tree extend system-id
archive
 path X
 write-memory
 time-period 1440
errdisable recovery cause udld
errdisable recovery cause bpduguard
errdisable recovery cause security-violation
errdisable recovery cause channel-misconfig
errdisable recovery cause pagp-flap
errdisable recovery cause dtp-flap
errdisable recovery cause link-flap
errdisable recovery cause sfp-config-mismatch
errdisable recovery cause gbic-invalid
errdisable recovery cause l2ptguard
errdisable recovery cause psecure-violation
errdisable recovery cause port-mode-failure
errdisable recovery cause dhcp-rate-limit
errdisable recovery cause pppoe-ia-rate-limit
errdisable recovery cause mac-limit
errdisable recovery cause vmps
errdisable recovery cause storm-control
errdisable recovery cause inline-power
errdisable recovery cause arp-inspection
errdisable recovery cause link-monitor-failure
errdisable recovery cause oam-remote-failure
errdisable recovery cause loopback
errdisable recovery cause psp
!
username admin privilege 15 secret 5 X
!
redundancy
 mode sso
!
!
!
class-map match-any system-cpp-police-topology-control
  description Topology control
class-map match-any system-cpp-police-sw-forward
  description Sw forwarding, L2 LVX data, LOGGING
class-map match-any system-cpp-default
  description DHCP Snooping, EWLC control, EWCL data
class-map match-any system-cpp-police-sys-data
  description Learning cache ovfl, Crypto Control, Exception, EGR Exception, NFL SAMPLED DATA, Gold Pkt, RPF Failed
class-map match-any system-cpp-police-punt-webauth
  description Punt Webauth
class-map match-any system-cpp-police-l2lvx-control
  description L2 LVX control packets
class-map match-any system-cpp-police-forus
  description Forus Address resolution and Forus traffic
class-map match-any system-cpp-police-multicast-end-station
  description MCAST END STATION
class-map match-any system-cpp-police-multicast
  description Transit Traffic and MCAST Data
class-map match-any system-cpp-police-l2-control
  description L2 control
class-map match-any system-cpp-police-dot1x-auth
  description DOT1X Auth
class-map match-any system-cpp-police-data
  description ICMP redirect, ICMP_GEN and BROADCAST
class-map match-any system-cpp-police-stackwise-virt-control
  description Stackwise Virtual
class-map match-any system-cpp-police-control-low-priority
  description General punt
class-map match-any non-client-nrt-class
class-map match-any system-cpp-police-routing-control
  description Routing control
class-map match-any system-cpp-police-protocol-snooping
  description Protocol snooping
!
policy-map system-cpp-policy
 class system-cpp-police-data
  police rate 600 pps
 class system-cpp-police-sys-data
  police rate 100 pps
 class system-cpp-police-sw-forward
  police rate 1000 pps
 class system-cpp-police-multicast
  police rate 500 pps
 class system-cpp-police-multicast-end-station
  police rate 2000 pps
 class system-cpp-police-punt-webauth
  police rate 1000 pps
 class system-cpp-police-l2-control
  police rate 2000 pps
 class system-cpp-police-stackwise-virt-control
  police rate 8000 pps
 class system-cpp-police-routing-control
  police rate 5400 pps
 class system-cpp-police-control-low-priority
  police rate 200 pps
 class system-cpp-police-l2lvx-control
  police rate 1000 pps
 class system-cpp-police-topology-control
  police rate 13000 pps
 class system-cpp-police-dot1x-auth
  police rate 1000 pps
 class system-cpp-police-protocol-snooping
  police rate 2000 pps
 class system-cpp-police-forus
  police rate 1000 pps
 class system-cpp-default
  police rate 1000 pps
!
!
!
!
!
!
!
!
!
!
!
!
!
!
interface Port-channel1
 description to X
 switchport mode trunk
!
interface GigabitEthernet0/0
 vrf forwarding Mgmt-vrf
 no ip address
 shutdown
 speed 1000
 negotiation auto
!
interface GigabitEthernet1/0/1
 description X
 switchport access vlan 1903
 switchport mode access
 no cdp enable
!
interface GigabitEthernet1/0/2
 description X
 switchport access vlan 1903
 switchport mode access
 no cdp enable
!
interface GigabitEthernet1/0/3
 description X
 switchport access vlan 1903
 switchport mode access
 no cdp enable
!
interface GigabitEthernet1/0/4
 description X
 switchport access vlan 1903
 switchport mode access
 no cdp enable
!
interface GigabitEthernet1/0/5
 description X
 switchport access vlan 1903
 switchport mode access
 no cdp enable
!
interface GigabitEthernet1/0/6
 description X
 switchport access vlan 1903
 switchport mode access
 no cdp enable
!
interface GigabitEthernet1/0/7
 description X
 switchport access vlan 1903
 switchport mode access
 no cdp enable
!
interface GigabitEthernet1/0/8
 description X
 switchport access vlan 1903
 switchport mode access
 no cdp enable
!
interface GigabitEthernet1/0/9
 description X
 switchport access vlan 1903
 switchport mode access
 no cdp enable
!
interface GigabitEthernet1/0/10
 description X
 switchport access vlan 1903
 switchport mode access
 no cdp enable
!
interface GigabitEthernet1/0/11
 description X
 switchport access vlan 1903
 switchport mode access
 no cdp enable
!
interface GigabitEthernet1/0/12
 description X
 switchport access vlan 1903
 switchport mode access
 no cdp enable
!
interface GigabitEthernet1/0/13
 description X
 switchport access vlan 1903
 switchport mode access
 no cdp enable
!
interface GigabitEthernet1/0/14
 description X
 switchport access vlan 1903
 switchport mode access
 no cdp enable
!
interface GigabitEthernet1/0/15
 description X
 switchport access vlan 1903
 switchport mode access
 no cdp enable
!
interface GigabitEthernet1/0/16
 description X
 switchport access vlan 1903
 switchport mode access
 no cdp enable
!
interface GigabitEthernet1/0/17
 description X
 switchport access vlan 1903
 switchport mode access
 no cdp enable
!
interface GigabitEthernet1/0/18
 switchport access vlan 1903
 switchport mode access
 no cdp enable
!
interface GigabitEthernet1/0/19
 description X
 switchport access vlan 2003
 switchport mode access
 no cdp enable
!
interface GigabitEthernet1/0/20
 description X
 switchport access vlan 2003
 switchport mode access
 no cdp enable
!
interface GigabitEthernet1/0/21
 description X
 switchport access vlan 1903
 switchport mode access
 no cdp enable
!
interface GigabitEthernet1/0/22
 description X
 switchport access vlan 2003
 switchport mode access
 no cdp enable
!
interface GigabitEthernet1/0/23
 description X
 switchport access vlan 1903
 switchport mode access
 no cdp enable
!
interface GigabitEthernet1/0/24
 description X
 switchport access vlan 1903
 switchport mode access
 no cdp enable
!
interface GigabitEthernet1/0/25
 description X
 switchport access vlan 1903
 switchport mode access
 no cdp enable
!
interface GigabitEthernet1/0/26
 description X
 switchport access vlan 1903
 switchport mode access
 no cdp enable
!
interface GigabitEthernet1/0/27
 description X
 switchport access vlan 2003
 switchport mode access
 no cdp enable
!
interface GigabitEthernet1/0/28
 description X
 switchport access vlan 1903
 switchport mode access
 no cdp enable
!
interface GigabitEthernet1/0/29
 description X
 switchport access vlan 1903
 switchport mode access
 no cdp enable
!
interface GigabitEthernet1/0/30
 description X
 switchport access vlan 1903
 switchport mode access
 no cdp enable
!
interface GigabitEthernet1/0/31
 description X
 switchport access vlan 2003
 switchport mode access
 no cdp enable
!
interface GigabitEthernet1/0/32
 description X
 switchport access vlan 1903
 switchport mode access
 no cdp enable
!
interface GigabitEthernet1/0/33
 description X
 switchport access vlan 1903
 switchport mode access
 no cdp enable
!
interface GigabitEthernet1/0/34
 description X
 switchport access vlan 1903
 switchport mode access
 no cdp enable
!
interface GigabitEthernet1/0/35
 switchport access vlan 1903
 switchport mode access
 no cdp enable
!
interface GigabitEthernet1/0/36
 description X
 switchport access vlan 2001
 switchport mode access
 no cdp enable
!
interface GigabitEthernet1/0/37
 description X
 switchport access vlan 1903
 switchport mode access
 no cdp enable
!
interface GigabitEthernet1/0/38
 description X
 switchport access vlan 1903
 switchport mode access
 no cdp enable
!
interface GigabitEthernet1/0/39
 description X
 switchport access vlan 1903
 switchport mode access
 no cdp enable
!
interface GigabitEthernet1/0/40
 description X
 switchport access vlan 1903
 switchport mode access
 no cdp enable
!
interface GigabitEthernet1/0/41
 description X
 switchport access vlan 1903
 switchport mode access
 no cdp enable
!
interface GigabitEthernet1/0/42
 description X
 switchport access vlan 1903
 switchport mode access
 no cdp enable
!
interface GigabitEthernet1/0/43
 description X
 switchport access vlan 2001
 switchport mode access
 no cdp enable
!
interface GigabitEthernet1/0/44
 switchport access vlan 1903
 switchport mode access
 no cdp enable
!
interface GigabitEthernet1/0/45
 switchport access vlan 1903
 switchport mode access
 no cdp enable
!
interface GigabitEthernet1/0/46
 description cisco test
 switchport access vlan 2003
 switchport mode access
 no cdp enable
!
interface GigabitEthernet1/0/47
 switchport access vlan 1903
 switchport mode access
 no cdp enable
!
interface GigabitEthernet1/0/48
 switchport access vlan 1903
 switchport mode access
 no cdp enable
!
interface GigabitEthernet1/1/1
!
interface GigabitEthernet1/1/2
!
interface GigabitEthernet1/1/3
!
interface GigabitEthernet1/1/4
!
interface TenGigabitEthernet1/1/1
!
interface TenGigabitEthernet1/1/2
!
interface TenGigabitEthernet1/1/3
!
interface TenGigabitEthernet1/1/4
!
interface TenGigabitEthernet1/1/5
!
interface TenGigabitEthernet1/1/6
!
interface TenGigabitEthernet1/1/7
!
interface TenGigabitEthernet1/1/8
!
interface FortyGigabitEthernet1/1/1
 switchport mode trunk
 channel-group 1 mode desirable
!
interface FortyGigabitEthernet1/1/2
 switchport mode trunk
 channel-group 1 mode desirable
!
interface Vlan1
 no ip address
 shutdown
!
interface Vlan9
 ip address X
!
ip default-gateway X
ip forward-protocol nd
no ip http server
no ip http secure-server
ip ssh version 2
!
!
logging trap debugging
logging facility local0
logging host X
!
snmp-server community X RO
!
!
!
!
control-plane
 service-policy input system-cpp-policy
!
!
line con 0
 stopbits 1
line vty 5 15
!
ntp server X
ntp server X prefer
!
wsma agent exec
!
wsma agent config
!
wsma agent filesys
!
wsma agent notify
!
!
end

[IT7]

So after more investigation the problem is a bit different:

applying "qos queue-softmax-multiplier 1200" makes everything better, so it is configured now on all our switches.

The problem still exists, but at a lower scale:

If a port has only ingress or only egress traffic then we see 1 Gigabit throughput, and the graph is flat, and no output drops.

If a port has both ingress and egress traffic then the total throughput is less that 2Gigabit and you can see on the graph that traffic is "fighting" for throughput (see attached file). Output drops are seen in this scenario.

It is Interesting that Intel NIC's are quite close to 2Gbit aggregate, but Realtek NICs that we have stay at around 1.4Gbit aggregate.

The same behavior is seen at our Force10 switch, so the problem is not switch dependent anymore.

Could it be that even though all consumer NICs are marketed as 1Gbit full duplex they actually can't hold 2Gbit aggregate, and start showing that weird behavior?

[Joseph W. Doherty]

"applying "qos queue-softmax-multiplier 1200" makes everything better, so it is configured now on all our switches."

That might be the equivalent of bumping of the drop limits on earlier Catalyst 3K switches, which I found often decreased port drops.

"Could it be that even though all consumer NICs are marketed as 1Gbit full duplex they actually can't hold 2Gbit aggregate, and start showing that weird behavior?"

Sure, it might be. Even on the Catalyst 6500 gig port line cards, all did not support the same performance.

[Lennart Jung]

@IT7 we've been reading your thread with great interest as we are facing the exact same problem on a Catalyst 9500 SWV pair. setting the "qos queue-softmax-multiplier 1200" gave us also an increase of factor 2-3 on the transfer speed.

Did you ever manage to solve this problem in the end?