Cat9500-24Q Performance Issues

[Dan Werntz]

I have a Catalyst 9500-24Q that is not passing traffic well.  It is connected to a Cat9300 and several UCS servers via 40G bidi transceivers.  I cannot get iperf to push more than 4 or 5Gbps port to port in the same VLAN on the switch.  This is far from line rate and is concerning me that the switch is defective.

I know it isn't the servers because if I connect the 40G links direct I can get about 20G of throughput.

Has anyone else experienced this or have some ideas for performance tuning?

[Leo Laohoo]

1. What firmware is the switch running on? 
2. Is/Are "Total Output Drops" counters incrementing?

[Dan Werntz]

1.  It's running 16.12.4.

2.  No output drops at all.

[Leo Laohoo]

Please post the output to the following commands: 

sh version
sh platform software status con brief
sh interface <PORT> controll

[Dan Werntz]

Datacenter-9500#show version
Cisco IOS XE Software, Version 16.12.04
Cisco IOS Software [Gibraltar], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 16.12.4, RELEASE SOFTWARE (fc5)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2020 by Cisco Systems, Inc.
Compiled Thu 09-Jul-20 21:49 by mcpre

Cisco IOS-XE software, Copyright (c) 2005-2020 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0. For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.

ROM: IOS-XE ROMMON
BOOTLDR: System Bootstrap, Version 16.9.1r [FC2], RELEASE SOFTWARE (P)

Datacenter-9500 uptime is 1 week, 7 hours, 9 minutes
Uptime for this control processor is 1 week, 7 hours, 11 minutes
System returned to ROM by PowerOn at 00:54:01 EST Thu Dec 24 2020
System restarted at 00:56:51 EST Thu Dec 24 2020
System image file is "flash:packages.conf"
Last reload reason: PowerOn

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

Technology Package License Information:

------------------------------------------------------------------------------
Technology-package Technology-package
Current Type Next reboot
------------------------------------------------------------------------------
network-advantage Smart License network-advantage
dna-advantage Subscription Smart License dna-advantage
AIR License Level: AIR DNA Advantage
Next reload AIR license Level: AIR DNA Advantage

Smart Licensing Status: UNREGISTERED/EVAL EXPIRED

cisco C9500-24Q (X86) processor with 1343576K/6147K bytes of memory.
Processor board ID FCW2204A25B
2 Virtual Ethernet interfaces
96 Ten Gigabit Ethernet interfaces
24 Forty Gigabit Ethernet interfaces
2048K bytes of non-volatile configuration memory.
16777216K bytes of physical memory.
1638400K bytes of Crash Files at crashinfo:.
11264000K bytes of Flash at flash:.
0K bytes of WebUI ODM Files at webui:.

Base Ethernet MAC Address : 00:bf:77:b9:1b:00
Motherboard Assembly Number : 73-18138-02
Motherboard Serial Number : FOC22028N77
Model Revision Number : E0
Motherboard Revision Number : B0
Model Number : C9500-24Q
System Serial Number : FCW2204A25B

Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
* 1 120 C9500-24Q 16.12.4 CAT9K_IOSXE INSTALL

Configuration register is 0x102

Datacenter-9500#sh platform software status con brief
Load Average
Slot Status 1-Min 5-Min 15-Min
1-RP0 Healthy 0.20 0.17 0.17

Memory (kB)
Slot Status Total Used (Pct) Free (Pct) Committed (Pct)
1-RP0 Healthy 15956456 3015584 (19%) 12940872 (81%) 3556728 (22%)

CPU Utilization
Slot CPU User System Nice Idle IRQ SIRQ IOwait
1-RP0 0 2.60 0.60 0.00 96.80 0.00 0.00 0.00
1 2.20 0.30 0.00 97.50 0.00 0.00 0.00
2 2.99 0.49 0.00 96.50 0.00 0.00 0.00
3 4.19 0.39 0.00 95.40 0.00 0.00 0.00
4 0.70 0.00 0.00 99.30 0.00 0.00 0.00
5 5.69 0.09 0.00 94.20 0.00 0.00 0.00
6 0.00 0.00 0.00 100.00 0.00 0.00 0.00
7 0.00 0.00 0.00 100.00 0.00 0.00 0.00

FortyGigabitEthernet1/0/3 is up, line protocol is up (connected)
Hardware is Forty Gigabit Ethernet, address is 00bf.77b9.1b03 (bia 00bf.77b9.1b03)
MTU 9000 bytes, BW 40000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 40Gb/s, link type is auto, media type is QSFP 40G SR BD SFP
input flow-control is on, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:07, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
30 second input rate 0 bits/sec, 0 packets/sec
30 second output rate 0 bits/sec, 0 packets/sec
337648979 packets input, 510635757904 bytes, 0 no buffer
Received 3364 broadcasts (2736 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 2736 multicast, 0 pause input
0 input packets with dribble condition detected
27980199 packets output, 26326321056 bytes, 0 underruns
Output 6929 broadcasts (0 multicasts)
0 output errors, 0 collisions, 2 interface resets
1864 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
Transmit FortyGigabitEthernet1/0/3 Receive
26326321056 Total bytes 510635757904 Total bytes
27892885 Unicast frames 337645615 Unicast frames
26317451220 Unicast bytes 510635206118 Unicast bytes
80385 Multicast frames 2736 Multicast frames
8250750 Multicast bytes 379415 Multicast bytes
6929 Broadcast frames 628 Broadcast frames
619086 Broadcast bytes 172371 Broadcast bytes
0 System FCS error frames 0 IpgViolation frames
0 MacUnderrun frames 0 MacOverrun frames
0 Pause frames 0 Pause frames
0 Cos 0 Pause frames 0 Cos 0 Pause frames
0 Cos 1 Pause frames 0 Cos 1 Pause frames
0 Cos 2 Pause frames 0 Cos 2 Pause frames
0 Cos 3 Pause frames 0 Cos 3 Pause frames
0 Cos 4 Pause frames 0 Cos 4 Pause frames
0 Cos 5 Pause frames 0 Cos 5 Pause frames
0 Cos 6 Pause frames 0 Cos 6 Pause frames
0 Cos 7 Pause frames 0 Cos 7 Pause frames
0 Oam frames 0 OamProcessed frames
0 Oam frames 0 OamDropped frames
10998043 Minimum size frames 1392549 Minimum size frames
51169 65 to 127 byte frames 5824 65 to 127 byte frames
8035 128 to 255 byte frames 840 128 to 255 byte frames
3552 256 to 511 byte frames 839 256 to 511 byte frames
4148 512 to 1023 byte frames 1395 512 to 1023 byte frames
16915237 1024 to 1518 byte frames 3734858 1024 to 1518 byte frames
15 1519 to 2047 byte frames 332512674 1519 to 2047 byte frames
0 2048 to 4095 byte frames 0 2048 to 4095 byte frames
0 4096 to 8191 byte frames 0 4096 to 8191 byte frames
0 8192 to 16383 byte frames 0 8192 to 16383 byte frames
0 16384 to 32767 byte frame 0 16384 to 32767 byte frame
0 > 32768 byte frames 0 > 32768 byte frames
0 Late collision frames 0 SymbolErr frames
0 Excess Defer frames 0 Collision fragments
0 Good (1 coll) frames 0 ValidUnderSize frames
0 Good (>1 coll) frames 0 InvalidOverSize frames
0 Deferred frames 0 ValidOverSize frames
0 Gold frames dropped 0 FcsErr frames
0 Gold frames truncated
1 Gold frames successful
0 1 collision frames
0 2 collision frames
0 3 collision frames
0 4 collision frames
0 5 collision frames
0 6 collision frames
0 7 collision frames
0 8 collision frames
0 9 collision frames
0 10 collision frames
0 11 collision frames
0 12 collision frames
0 13 collision frames
0 14 collision frames
0 15 collision frames
0 Excess collision frames

LAST UPDATE 3068 msecs AGO

---------------------------------------------

Datacenter-9500#sh interface fo1/0/7 controll
FortyGigabitEthernet1/0/7 is up, line protocol is up (connected)
Hardware is Forty Gigabit Ethernet, address is 00bf.77b9.1b07 (bia 00bf.77b9.1b07)
Description: Tanker-A-Port1
MTU 9000 bytes, BW 40000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 40Gb/s, link type is auto, media type is QSFP 40G SR BD SFP
input flow-control is on, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:27, output 00:00:17, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 13000 bits/sec, 9 packets/sec
1542388483 packets input, 2512088822077 bytes, 0 no buffer
Received 21055 broadcasts (20762 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 20762 multicast, 2 pause input
0 input packets with dribble condition detected
511122574 packets output, 925838143313 bytes, 0 underruns
Output 365258 broadcasts (0 multicasts)
0 output errors, 0 collisions, 3 interface resets
13435 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
Transmit FortyGigabitEthernet1/0/7 Receive
925838143313 Total bytes 2512088822077 Total bytes
508383199 Unicast frames 1542367428 Unicast frames
925588852032 Unicast bytes 2512083898924 Unicast bytes
2374117 Multicast frames 20762 Multicast frames
216212039 Multicast bytes 4896353 Multicast bytes
365258 Broadcast frames 293 Broadcast frames
33079242 Broadcast bytes 26800 Broadcast bytes
0 System FCS error frames 0 IpgViolation frames
0 MacUnderrun frames 0 MacOverrun frames
0 Pause frames 2 Pause frames
0 Cos 0 Pause frames 0 Cos 0 Pause frames
0 Cos 1 Pause frames 0 Cos 1 Pause frames
0 Cos 2 Pause frames 0 Cos 2 Pause frames
0 Cos 3 Pause frames 0 Cos 3 Pause frames
0 Cos 4 Pause frames 0 Cos 4 Pause frames
0 Cos 5 Pause frames 0 Cos 5 Pause frames
0 Cos 6 Pause frames 0 Cos 6 Pause frames
0 Cos 7 Pause frames 0 Cos 7 Pause frames
0 Oam frames 0 OamProcessed frames
0 Oam frames 0 OamDropped frames
537284 Minimum size frames 11372387 Minimum size frames
98144164 65 to 127 byte frames 17152733 65 to 127 byte frames
317046 128 to 255 byte frames 15489586 128 to 255 byte frames
42533791 256 to 511 byte frames 1949691 256 to 511 byte frames
4420542 512 to 1023 byte frames 12501319 512 to 1023 byte frames
289536516 1024 to 1518 byte frames 4762320 1024 to 1518 byte frames
24325696 1519 to 2047 byte frames 1444072778 1519 to 2047 byte frames
5642169 2048 to 4095 byte frames 2413962 2048 to 4095 byte frames
532239 4096 to 8191 byte frames 1718483 4096 to 8191 byte frames
45133127 8192 to 16383 byte frames 30955226 8192 to 16383 byte frames
0 16384 to 32767 byte frame 0 16384 to 32767 byte frame
0 > 32768 byte frames 0 > 32768 byte frames
0 Late collision frames 0 SymbolErr frames
0 Excess Defer frames 0 Collision fragments
0 Good (1 coll) frames 0 ValidUnderSize frames
0 Good (>1 coll) frames 0 InvalidOverSize frames
0 Deferred frames 0 ValidOverSize frames
0 Gold frames dropped 0 FcsErr frames
0 Gold frames truncated
1 Gold frames successful
0 1 collision frames
0 2 collision frames
0 3 collision frames
0 4 collision frames
0 5 collision frames
0 6 collision frames
0 7 collision frames
0 8 collision frames
0 9 collision frames
0 10 collision frames
0 11 collision frames
0 12 collision frames
0 13 collision frames
0 14 collision frames
0 15 collision frames
0 Excess collision frames

LAST UPDATE 10170 msecs AGO

[balaji.bandi]

Can you post more information, how these connected, and some port configuration information.

and interface output.

[Dan Werntz]

They are connected from the VIC on the servers directly to the 9500 with 40G bidis.  I've tried with and without port channels, currently one is not in a PC and the other is.  Interface configs below.

interface FortyGigabitEthernet1/0/3
switchport trunk native vlan 100
switchport mode trunk
load-interval 30

interface FortyGigabitEthernet1/0/7
description Tanker-A-Port1
switchport mode trunk
channel-group 20 mode active

interface FortyGigabitEthernet1/0/8
description Tanker-A-Port2
switchport mode trunk
channel-group 20 mode active

interface Port-channel20
description Tanker-A
switchport mode trunk

FortyGigabitEthernet1/0/3 is up, line protocol is up (connected)
Hardware is Forty Gigabit Ethernet, address is 00bf.77b9.1b03 (bia 00bf.77b9.1b03)
MTU 9000 bytes, BW 40000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 40Gb/s, link type is auto, media type is QSFP 40G SR BD SFP
input flow-control is on, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
30 second input rate 2000 bits/sec, 1 packets/sec
30 second output rate 19000 bits/sec, 9 packets/sec
337649686 packets input, 510636039874 bytes, 0 no buffer
Received 3534 broadcasts (2883 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 2883 multicast, 0 pause input
0 input packets with dribble condition detected
27986208 packets output, 26327847690 bytes, 0 underruns
Output 7282 broadcasts (0 multicasts)
0 output errors, 0 collisions, 2 interface resets
1880 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

FortyGigabitEthernet1/0/7 is up, line protocol is up (connected)
Hardware is Forty Gigabit Ethernet, address is 00bf.77b9.1b07 (bia 00bf.77b9.1b07)
Description: Tanker-A-Port1
MTU 9000 bytes, BW 40000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 40Gb/s, link type is auto, media type is QSFP 40G SR BD SFP
input flow-control is on, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:01, output 00:00:14, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 5000 bits/sec, 8 packets/sec
1542388717 packets input, 2512088845367 bytes, 0 no buffer
Received 21079 broadcasts (20786 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 20786 multicast, 2 pause input
0 input packets with dribble condition detected
511125602 packets output, 925838693710 bytes, 0 underruns
Output 365286 broadcasts (0 multicasts)
0 output errors, 0 collisions, 3 interface resets
13448 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

FortyGigabitEthernet1/0/8 is up, line protocol is up (connected)
Hardware is Forty Gigabit Ethernet, address is 00bf.77b9.1b08 (bia 00bf.77b9.1b08)
Description: Tanker-A-Port2
MTU 9000 bytes, BW 40000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 40Gb/s, link type is auto, media type is QSFP 40G SR BD SFP
input flow-control is on, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:10, output 00:00:10, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 2000 bits/sec, 3 packets/sec
9834446 packets input, 55053057294 bytes, 0 no buffer
Received 20713 broadcasts (20552 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 20552 multicast, 0 pause input
0 input packets with dribble condition detected
341377567 packets output, 514652889148 bytes, 0 underruns
Output 206595 broadcasts (0 multicasts)
0 output errors, 0 collisions, 3 interface resets
13442 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

Port-channel20 is up, line protocol is up (connected)
Hardware is EtherChannel, address is 00bf.77b9.1b08 (bia 00bf.77b9.1b08)
Description: Tanker-A
MTU 9000 bytes, BW 80000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 40Gb/s, link type is auto, media type is N/A
input flow-control is on, output flow-control is unsupported
Members in this channel: Fo1/0/7 Fo1/0/8
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 7000 bits/sec, 10 packets/sec
1552223124 packets input, 2567141898612 bytes, 0 no buffer
Received 41786 broadcasts (41332 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 41332 multicast, 2 pause input
0 input packets with dribble condition detected
852502650 packets output, 1440491540725 bytes, 0 underruns
Output 571828 broadcasts (0 multicasts)
0 output errors, 0 collisions, 1 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

[Leo Laohoo]

---

@Dan Werntz wrote:

I know it isn't the servers because if I connect the 40G links direct I can get about 20G of throughput.

---

Wait a second ... The output provided above are from the 9500, right? 
Can you provide the output from the 9300 switch directly-connected to the servers?

[Dan Werntz]

I think I was unclear.  The servers are only connected to the 9500.  Only users are connected to the 9300.  There is a 2x40G port channel between the 9500 and 9300 but it is not in play for server to server throughput testing.

I connected the servers back to back with no switch in between and was able to get around 20Gbps.

[Leo Laohoo]

---

@Dan Werntz wrote:

What if I kick off the change tomorrow? Will that still fall under "number of staff are coming back to work"?

---

I would be interested to see the interface counters for the ports connected to the servers.  I suspect there is a huge counter under "Total Output Drops".

[Dan Werntz]

That's what was in my output, server ports only.  Identified in bold below.

FortyGigabitEthernet1/0/3 - Server A

337649686 packets input, 510636039874 bytes, 0 no buffer
Received 3534 broadcasts (2883 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 2883 multicast, 0 pause input
0 input packets with dribble condition detected
27986208 packets output, 26327847690 bytes, 0 underruns
Output 7282 broadcasts (0 multicasts)
0 output errors, 0 collisions, 2 interface resets
1880 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

FortyGigabitEthernet1/0/7 - Server B Port 1

1542388717 packets input, 2512088845367 bytes, 0 no buffer
Received 21079 broadcasts (20786 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 20786 multicast, 2 pause input
0 input packets with dribble condition detected
511125602 packets output, 925838693710 bytes, 0 underruns
Output 365286 broadcasts (0 multicasts)
0 output errors, 0 collisions, 3 interface resets
13448 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

FortyGigabitEthernet1/0/8 - Server B Port 2

9834446 packets input, 55053057294 bytes, 0 no buffer
Received 20713 broadcasts (20552 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 20552 multicast, 0 pause input
0 input packets with dribble condition detected
341377567 packets output, 514652889148 bytes, 0 underruns
Output 206595 broadcasts (0 multicasts)
0 output errors, 0 collisions, 3 interface resets
13442 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

Port-channel20 - Server B Port Channel

1552223124 packets input, 2567141898612 bytes, 0 no buffer
Received 41786 broadcasts (41332 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 41332 multicast, 2 pause input
0 input packets with dribble condition detected
852502650 packets output, 1440491540725 bytes, 0 underruns
Output 571828 broadcasts (0 multicasts)
0 output errors, 0 collisions, 1 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

[balaji.bandi]

interface FortyGigabitEthernet1/0/3
switchport trunk native vlan 100
switchport mode trunk
load-interval 30

is this interface connected to cat 9300? can you show us the config of Cat 9300 interface config and output same as you posted?

how are you testing? from the device connected to Cat 9300? what port did it connect?

[Dan Werntz]

The interfaces in my output are a port channel to one server and a single interface to another server.  The 9300 is only user devices and is not involved in the testing.  Only devices connected to the 9500 are where I am running iperf, both client and server.

[balaji.bandi]

Ok so you mean, both Iperf Server and client in Cat 9500 only

so where is the Server IPERF running and where is the IPERF client running.

are  these under esxi  as VM ? or end device has iperf client ?

[Dan Werntz]

Both boxes are bare metal linux servers to remove any vswitch limitations.  Server iperf is on the port channel and client is on the single.  But I have run it both directions.