flow control on Cisco Nexus fabric extenders

[LA-Engineer]

Cisco Nexus Fabric Extender host ports are configured for 'flow-control send' by default.

This is understandable since the fabric extender fabric interfaces are oversubscribed.  However, I just wanted to know under what condition will the fabric extender send tx-pauses?  

For instance, will the fabric extender send tx-pauses when the fabric interfaces are at a certain percentage utilized?

Thank you in advance!

[Reza Sharifi · Accepted solution]

What we have seen with 2232 series FEXs connecting to VM servers is that the unlink ports on the FEX send tx-pause frames when there is a pretty large transfer between ports. One of the problems with 2232 series is that all 8 uplink ports have only 12Mb shared buffer and so It can run out of buffer when there is a large transfer.  We have tried with ports having flow-control send off and on and also receive on and off and we are still seen pause frames.

What type of FEX are you using?

Are they connected to VM?

HTH

[LA-Engineer]

Hi Reza, thanks for getting back to me.

We're are actually using a N2K-C2232PP-10GE!

The host ports are connected to a NetApp and are carrying iSCSI.

One other thing I noticed is that the host ports connected to the NetApp that are showing large numbers of Tx pauses are also recording output errors.  Its only one every few minutes but I'm wondering if this may be related.  I don't see any CRC errors anywhere to explain the output errors.

The reason I'm asking about this is because we are having performance issues on one of our servers that uses the NetApp connected to this FEX.  However, BW utilization on the fabric uplink port-channel doesn't really go above 250Mbps.  Perhaps the load interval isn't catching the spikes caused by iSCSI block requests?

[Reza Sharifi]

Hi,

We don't see any output errors on the FEX uplinks.  Regarding performance issue, we also did some testing and noticed we don't get line rate performance on the 10Gig FEX interfaces. Since we have a ticket open with Cisco, they recommend to add "no hardware n2232p queue-limit" on the FEX uplink ports. After deploying this command, we are geeing much better performance (almost line rate) on the FEXs.  BTW, deploying this command does not require an outage and can be done any time.

I am interested to know to if this command will help you with NetApp performance issue.

HTH

[LA-Engineer]

Thank you.  I will setup a time to apply this and test whether or not it worked.

BTW, is there a way to see the status of these buffers such as the allocated size and whether there are drops?  Is it just "show queueing interface".

[Reza Sharifi]

I think, in order to see the buffer parameters, you have to logging to the FEX.  From the switch, I think the command is "attach fex xxx" and then look for commands

HTH

[LA-Engineer]

Thank you.  I will try it this evening and let you know if there were any improvements.  By the way, would you happen to know why there is a limitation on the queue?  Why would anyone want to set the queue-limit?

[LA-Engineer]

Nevermind, I figured out the answer to that question.

Removing the queue-limit will allow a port to use up all of the shared buffer and blocking it for any other port.

[LA-Engineer]

I talked to TAC and they recommended the same thing - to remove the queue-limit on the host ports.  It looks like it made the performance of our SQL servers worse.  Although, our Netapp admin tells me that he's seeing fewer network related alarms.  Very strange.

I am going to move the netapp off the FEX and onto the parent switch.

Thanks for you help!