I've a very simple producer - consumer configuration using UDP unicast over a 10 Gbit connection.
I can achieve around 1100 MB/s using a point to point connection before having packet loss. The producer, an FPGA with custom VHDL, can be configured to send n packets at a time with a specified delay between each packets. And the producer honors 802.3 pause frames when sent by the consumer. I think this is all good.
Now I connect these two devices to a Nexus 3064 switch. These are the only 2 devices connected to the switch. MTU is set to 9000; CDP is disabled; LLDP is disabled. Receive and send Flowcontrol is enabled on both ports being used. Wireshark on the consumer only shows the expected data traffic and ARP packets.
I see less than 600 MB/s using the switch using the same packet count and interpacket delay time that provided 100 MB/s point-point. No packets are lost. The switch is receiving the pause frames from the consumer and is not sending pause frames to the producer, so the switch buffers don't appear to be overflowing. How do I improve performance?
Here some supporting switch config info:
bigten# show interface eth1/17
Ethernet1/17 is up
Dedicated Interface
Hardware: 100/1000/10000 Ethernet, address: a493.4c80.1418 (bia a493.4c80.1418
)
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA
Port mode is access
full-duplex, 10 Gb/s, media type is 10G
Beacon is turned off
Input flow-control is on, output flow-control is on
Rate mode is dedicated
Switchport monitor is off
EtherType is 0x8100
Last link flapped 1d01h
Last clearing of "show interface" counters never
5 interface resets
30 seconds input rate 40 bits/sec, 0 packets/sec
30 seconds output rate 0 bits/sec, 0 packets/sec
Load-Interval #2: 5 minute (300 seconds)
input rate 24 bps, 0 pps; output rate 0 bps, 0 pps
RX
328739 unicast packets 6544 multicast packets 11 broadcast packets
13325432 input packets 857826412 bytes
18 jumbo packets 0 storm suppression bytes
0 runts 0 giants 0 CRC 0 no buffer
0 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
12990138 Rx pause
TX
50095175 unicast packets 678 multicast packets 151 broadcast packets
50096004 output packets 206404726437 bytes
49437788 jumbo packets
0 output errors 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 1115 output discard
0 Tx pause
bigten# show interface eth1/21
Ethernet1/21 is down (Link not connected)
Dedicated Interface
Hardware: 100/1000/10000 Ethernet, address: a493.4c80.141c (bia a493.4c80.141c
)
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA
Port mode is access
Full-duplex, 10 Gb/s, media type is 10G
Beacon is turned off
Input flow-control is on, output flow-control is on
Rate mode is dedicated
Switchport monitor is off
EtherType is 0x8100
Last link flapped 00:00:29
Last clearing of "show interface" counters never
23 interface resets
30 seconds input rate 0 bits/sec, 0 packets/sec
30 seconds output rate 0 bits/sec, 0 packets/sec
Load-Interval #2: 5 minute (300 seconds)
input rate 0 bps, 0 pps; output rate 0 bps, 0 pps
RX
50096180 unicast packets 0 multicast packets 0 broadcast packets
50096180 input packets 206408826460 bytes
49438800 jumbo packets 0 storm suppression bytes
0 runts 0 giants 0 CRC 0 no buffer
0 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
TX
328712 unicast packets 308 multicast packets 19 broadcast packets
329039 output packets 25669490 bytes
0 jumbo packets
0 output errors 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 48 output discard
0 Tx pause
bigten# show queuing interface eth1/17
Ethernet1/17 queuing information:
TX Queuing
qos-group sched-type oper-bandwidth
0 WRR 100
RX Queuing
qos-group 0
HW MTU: 9000 (9000 configured)
drop-type: drop, xon: 0, xoff: 0
Statistics:
Ucast pkts sent over the port : 50095175
Ucast bytes sent over the port : 206404618670
Mcast pkts sent over the port : 180
Mcast bytes sent over the port : 39410
Ucast pkts dropped : 1031
Ucast bytes dropped : 4299298
Mcast pkts dropped : 0
Mcast bytes dropped : 0
bigten# show queuing interface eth1/21
Ethernet1/21 queuing information:
TX Queuing
qos-group sched-type oper-bandwidth
0 WRR 100
RX Queuing
qos-group 0
HW MTU: 9000 (9000 configured)
drop-type: drop, xon: 0, xoff: 0
Statistics:
Ucast pkts sent over the port : 328712
Ucast bytes sent over the port : 25639032
Mcast pkts sent over the port : 36
Mcast bytes sent over the port : 7200
Ucast pkts dropped : 0
Ucast bytes dropped : 0
Mcast pkts dropped : 0
Mcast bytes dropped : 0
