Nexus 9300 port- large amounts of "rx pause" & "output discard"

[MicJameson1]

Hello.

GOAL: On two Nexus in HSRP config, to alleviate excessive "rx pause" and "output discard"s.

GIVEN: Packet loss exists in cloud-to-LAN-server circuit. All other device inspections reveal healthy operation. The last ports on the last hop inspected (as always!), on a pair of Nexus 9300, each with a port channel (consisting of two ports in each port channel) to the destination server, are showing many "output discard". (4 ports showing symptoms.)

Only one port is showing many "rx pause". This is the first port of the port channel in the primary Nexus.

Below is explanation from some  Cisco literature...

---
"Information on show interface pause counters
When the receive part (Rx) of the port has its Rx FIFO queue filled and reaches the high water mark, the transmit part (Tx) of the port starts to generate pause frames with an interval value mentioned in it. The remote device is expected to stop / reduce the transmission of packets for the interval time mentioned in the pause frame.

If the Rx is able to clear the Rx queue or reach low water mark within this interval, Tx sends out a special pause frame that mentions the interval as zero (0x0). This enables the remote device to start to transmit packets. If the Rx still works on the queue, once the interval time expires, the Tx sends a new pause frame again with a new interval value.

Pause frames are used to flow control traffic and its use can be found in platform specific configuration guides though most commands are common. An example on Nexus9000 can be seen below:
https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/7-x/qos/configuration/guide/b_Cisco_Nexus_9000_Series_NX-OS_Quality_of_Service_Configuration_Guide_7x.html
---

show_interface_counters_errors_output_discards
The number of outbound packets chosen to be discarded even though no errors have been detected.

The cause of output discards is very specific depending on Nexus platform.

For example, the N7K platform will experience output discards in different ways depending on the linecard used. For M series, traffic will be dropped in the egress direction due to congested egress buffers. The F series, will only drop BUM(broadcast, unknown unicast, and Multicast) traffic in the egress direction."

=====

QUESTION: May you please suggest/assist how to remediate this symptom?

Below is data from each of four ports. Symptoms in red bold.

Thank you!

-----

(obfuscated) 9K1# sh int e1/301

Ethernet1/301 is up

admin state is up, Dedicated Interface

  Belongs to Po206

  Hardware: 1000/10000 Ethernet, address: (obfuscated)

  Description: ESXiHost2_10GB

  MTU 1500 bytes, BW 10000000 Kbit , DLY 10 usec

  reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA, medium is broadcast

  Port mode is trunk

  full-duplex, 10 Gb/s, media type is 10G

  Beacon is turned off

  Auto-Negotiation is turned on  FEC mode is Auto

  Input flow-control is off, output flow-control is off

  Auto-mdix is turned off

  Rate mode is dedicated

  Switchport monitor is off 

  EtherType is 0x8100 

  EEE (efficient-ethernet) : n/a

    admin fec state is auto, oper fec state is off

  Last link flapped 64week(s) 5day(s)

  Last clearing of "show interface" counters 64w5d

  2 interface resets

  Load-Interval #1: 30 seconds

    30 seconds input rate 24105592 bits/sec, 2448 packets/sec

    30 seconds output rate 44125120 bits/sec, 4185 packets/sec

    input rate 24.10 Mbps, 2.45 Kpps; output rate 44.12 Mbps, 4.18 Kpps

  Load-Interval #2: 5 minute (300 seconds)

    300 seconds input rate 24716088 bits/sec, 2340 packets/sec

    300 seconds output rate 34712392 bits/sec, 3670 packets/sec

    input rate 24.72 Mbps, 2.34 Kpps; output rate 34.71 Mbps, 3.67 Kpps

  RX

    250190024753 unicast packets  4260491 multicast packets  10643134 broadcast packets

    250204928378 input packets  339469040229327 bytes

    200056026819 jumbo packets  0 storm suppression packets

    0 runts  0 giants  0 CRC  0 no buffer

    0 input error  0 short frame  0 overrun   0 underrun  0 ignored

    0 watchdog  0 bad etype drop  0 bad proto drop  0 if down drop

    0 input with dribble  0 input discard

    657223 Rx pause   !! <<<<<<<<<<<<<<<<<< !!

  TX

    367629585405 unicast packets  40376617 multicast packets  35106507 broadcast packets

    367705068529 output packets  516699228055464 bytes

    327187524973 jumbo packets

    0 output error  0 collision  0 deferred  0 late collision

    0 lost carrier  0 no carrier  0 babble  8449947 output discard  !! <<<<<<<<<<<< !!

    0 Tx pause

-----

9K1# sh int e1/305

Ethernet1/305 is up

admin state is up, Dedicated Interface

  Belongs to Po206

  Hardware: 1000/10000 Ethernet, address: (obfuscated)

  Description: ESXiHost2_10GB

  MTU 1500 bytes, BW 10000000 Kbit , DLY 10 usec

  reliability 255/255, txload 4/255, rxload 1/255

  Encapsulation ARPA, medium is broadcast

  Port mode is trunk

  full-duplex, 10 Gb/s, media type is 10G

  Beacon is turned off

  Auto-Negotiation is turned on  FEC mode is Auto

  Input flow-control is off, output flow-control is off

  Auto-mdix is turned off

  Rate mode is dedicated

  Switchport monitor is off 

  EtherType is 0x8100 

  EEE (efficient-ethernet) : n/a

    admin fec state is auto, oper fec state is off

  Last link flapped 64week(s) 5day(s)

  Last clearing of "show interface" counters 64w5d

  3 interface resets

  Load-Interval #1: 30 seconds

    30 seconds input rate 5254088 bits/sec, 702 packets/sec

    30 seconds output rate 159668728 bits/sec, 15701 packets/sec

    input rate 5.25 Mbps, 702 pps; output rate 159.67 Mbps, 15.70 Kpps

  Load-Interval #2: 5 minute (300 seconds)

    300 seconds input rate 28271848 bits/sec, 2666 packets/sec

    300 seconds output rate 126514392 bits/sec, 12273 packets/sec

    input rate 28.27 Mbps, 2.67 Kpps; output rate 126.51 Mbps, 12.27 Kpps

  RX

    91376558292 unicast packets  5468879 multicast packets  8544588 broadcast packets

    91390571759 input packets  104849545537913 bytes

    48804963778 jumbo packets  0 storm suppression packets

    0 runts  0 giants  0 CRC  0 no buffer

    0 input error  0 short frame  0 overrun   0 underrun  0 ignored

    0 watchdog  0 bad etype drop  0 bad proto drop  0 if down drop

    0 input with dribble  0 input discard

    0 Rx pause

  TX

    407212749834 unicast packets  45161618 multicast packets  21930056 broadcast packets

    407279841508 output packets  513351517818806 bytes

    315134321175 jumbo packets

    0 output error  0 collision  0 deferred  0 late collision

    0 lost carrier  0 no carrier  0 babble  31018127 output discard  !! <<<<<<<<<<<<<<<<<<< !!

    0 Tx pause

=============

9K2# sh int e1/301

Ethernet1/301 is up

admin state is up, Dedicated Interface

  Belongs to Po206

  Hardware: 1000/10000 Ethernet, address: (obfuscated)

  Description: ESXiHost2_10GB

  MTU 1500 bytes, BW 10000000 Kbit , DLY 10 usec

  reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA, medium is broadcast

  Port mode is trunk

  full-duplex, 10 Gb/s, media type is 10G

  Beacon is turned off

  Auto-Negotiation is turned on  FEC mode is Auto

  Input flow-control is off, output flow-control is off

  Auto-mdix is turned off

  Rate mode is dedicated

  Switchport monitor is off 

  EtherType is 0x8100 

  EEE (efficient-ethernet) : n/a

    admin fec state is auto, oper fec state is off

  Last link flapped 64week(s) 5day(s)

  Last clearing of "show interface" counters never

  1 interface resets

  Load-Interval #1: 30 seconds

    30 seconds input rate 26935336 bits/sec, 2757 packets/sec

    30 seconds output rate 48847032 bits/sec, 5091 packets/sec

    input rate 26.93 Mbps, 2.76 Kpps; output rate 48.85 Mbps, 5.09 Kpps

  Load-Interval #2: 5 minute (300 seconds)

    300 seconds input rate 25824968 bits/sec, 2562 packets/sec

    300 seconds output rate 30321928 bits/sec, 3528 packets/sec

    input rate 25.82 Mbps, 2.56 Kpps; output rate 30.32 Mbps, 3.53 Kpps

  RX

    177777321510 unicast packets  23641213 multicast packets  16385642 broadcast packets

    177817348365 input packets  234014568192315 bytes

    131059139942 jumbo packets  0 storm suppression packets

    0 runts  0 giants  0 CRC  0 no buffer

    0 input error  0 short frame  0 overrun   0 underrun  0 ignored

    0 watchdog  0 bad etype drop  0 bad proto drop  0 if down drop

    0 input with dribble  0 input discard

    0 Rx pause

  TX

    418538249507 unicast packets  44202632 multicast packets  29950857 broadcast packets

    418612402996 output packets  571288839175700 bytes

    363755872582 jumbo packets

    0 output error  0 collision  0 deferred  0 late collision

    0 lost carrier  0 no carrier  0 babble  2223093 output discard   !! <<<<<<<<<<<<<<<<<<< !!

-----

9K2# sh int e1/305

Ethernet1/305 is up

admin state is up, Dedicated Interface

  Belongs to Po206

  Hardware: 1000/10000 Ethernet, address: (obfuscated)

  Description: ESXiHost2_10GB

  MTU 1500 bytes, BW 10000000 Kbit , DLY 10 usec

  reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA, medium is broadcast

  Port mode is trunk

  full-duplex, 10 Gb/s, media type is 10G

  Beacon is turned off

  Auto-Negotiation is turned on  FEC mode is Auto

  Input flow-control is off, output flow-control is off

  Auto-mdix is turned off

  Rate mode is dedicated

  Switchport monitor is off 

  EtherType is 0x8100 

  EEE (efficient-ethernet) : n/a

    admin fec state is auto, oper fec state is off

  Last link flapped 64week(s) 5day(s)

  Last clearing of "show interface" counters never

  1 interface resets

  Load-Interval #1: 30 seconds

    30 seconds input rate 36705872 bits/sec, 3467 packets/sec

    30 seconds output rate 5068760 bits/sec, 857 packets/sec

    input rate 36.71 Mbps, 3.47 Kpps; output rate 5.07 Mbps, 857 pps

  Load-Interval #2: 5 minute (300 seconds)

    300 seconds input rate 47576184 bits/sec, 4530 packets/sec

    300 seconds output rate 4891600 bits/sec, 997 packets/sec

    input rate 47.58 Mbps, 4.53 Kpps; output rate 4.89 Mbps, 997 pps

  RX

    213883589204 unicast packets  4450657 multicast packets  1812612 broadcast packets

    213889852473 input packets  262763534007905 bytes

    146625107607 jumbo packets  0 storm suppression packets

    0 runts  0 giants  0 CRC  0 no buffer

    0 input error  0 short frame  0 overrun   0 underrun  0 ignored

    0 watchdog  0 bad etype drop  0 bad proto drop  0 if down drop

    0 input with dribble  0 input discard

    0 Rx pause

  TX

    491623591936 unicast packets  176736262 multicast packets  43668043 broadcast packets

    491843996241 output packets  451063671692595 bytes

    261123112585 jumbo packets

    0 output error  0 collision  0 deferred  0 late collision

    0 lost carrier  0 no carrier  0 babble  3953256 output discard  !! <<<<<<<<<<<<<<<<<<< !!

    0 Tx pause

[Cisco_Virtual_Engineer]

To remediate the output discards and Rx pause issues, you can try the following steps:

1. Check for any misconfigurations in the port channels, VLANs, and trunk settings on both Nexus switches. Ensure that the settings are consistent across all devices.

2. Enable Quality of Service (QoS) on the Nexus switches to better manage the traffic and prioritize critical traffic. This can help alleviate congestion and minimize output discards and Rx pause issues. Refer to the Cisco Nexus 9000 Series NX-OS Quality of Service Configuration Guide for more information: https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/7-x/qos/configuration/guide/b_Cisco_Nexus_9000_Series_NX-OS_Quality_of_Service_Configuration_Guide_7x.html

3. Monitor the traffic patterns and utilization of the affected interfaces. If the traffic is exceeding the capacity of the interfaces, consider upgrading to higher bandwidth interfaces or load balancing the traffic across multiple interfaces.

4. Check for any faulty hardware, such as cables, transceivers, or line cards, and replace them if necessary.

5. Consider enabling flow control on the affected interfaces to better manage the traffic and prevent congestion. This can be done using the `flowcontrol receive on` and `flowcontrol send on` commands on the interface configuration mode.

6. If the issue persists, consider opening a support case with Cisco TAC for further assistance and troubleshooting.