Team, before we start, I have to say that I have middle of the road switch / networking skills.
We have:
We have a leaf/spine 9K topology. All the compute nodes on the network have LACP mellanox connectX3 cards, 40Gb interfaces. APIC 2.2(1o)
Our compute node symptoms:
The compute nodes (red hat 7) with the mellanox cards see a ton of dropped packets.
Looking at the NIC's status, shows it has transmitted 300K+ pause frames. Received 0 pause frames. The following settings are at:
rx_pause_duration 4014377
rx_pause_transition 155778
I don't know if that's high or normal.
9K switch symptoms:
There are a large numer of buffer drops on the uplinks. One of the interfaces Rx pause counters are incrementing.
What I know:
We are barelly pushing this network. Bursts of 4 Gb/s, but average is about 500Mb.
Cicso 9K's in ACI mode, don't support the IEEE 802.3x port based Flow Control, which was the original flow control mechanism. It only supports IEEE 802.1Qbb , priority based Flow Control. Our Mellanox cards don't support this flow control.
Question:
Sorry, this many not be a fair question as there may not be enough information given. But based on the symptoms above, our configuration, are we having a flow control problem? Buffer drops to me mean the ends of the pipe can't coordinate the payload.
Any help would be stellar.
tks
