Nexus 7k FCoE Traffic Load Balancing

[ataranen]

I can't find a clear answer on FCoE traffic load balancing on 7k storage VDC. The following Cisco 7k FCoE reference has obvious configuration mistakes in load-balancing configuration: http://www.cisco.com/en/US/products/ps9670/products_configuration_example09186a0080c13d2f.shtml

It shows load-balancing command being applied under the storage-vdc, which is not possible. Load-balancing can only go under default VDC:

fcoe(config)# port-channel load-balance ethernet source-dest-port

In addition the 7k takes "port-channel load-balance src-dst l4port", not "ethernet source-dest-port".

Based on this example, which is full of mistakes, one would conclude that in order to set load balacing to src-dst-ox id on 7k we have to do:

"port-channel load-balance src-dst l4port" unde the default VDC.

However, the following configuratio guide, tells that the "Traffic is load balanced across equal cost E_Ports and VE_Ports based on SID, DID, and OXID" automatically without the need to configure port-based load balancing for the underlying Ethernet links.

http://www.cisco.com/en/US/docs/switches/datacenter/sw/nx-os/fcoe/configuration/guide/b_Cisco_NX-OS_FCoE_Configuration_Guide_chapter_0100.html#concept_013970837994428BB207A1A79DBFA22A

On Nexus 5k, it is very clear: we have to set Ethernet to "port-channel load-balance ethernet source-dest-port" in order to load balance storage traffic in SID-DID-OXID mode, but I can't find any conclusive documentation on 7k

[bogdan.sass]

The document you linked to (

http://www.cisco.com/en/US/products/ps9670/products_configuration_example09186a0080c13d2f.shtml

) does state that:

On Nexus 7000, by default the "source-destination-oxid" load balancing mechanism is used for FCoE traffic.

Also, I cannot see the "port-channel load-balance" command under the storage VDC (indeed, changing the load balancing method for the storage VDC of a 7K is not possible).

Maybe they updated the documentation?

[ataranen]

That document was updated Nov. 25 2013 and the FCoE load-balancing section has changed. Now it clearly states that FCoE traffic will automatically be load-balanced using src-dst-oxid. What is not clear however how it affects non FCoE traffic. On Nexus 7k the load-balacing mechanism can be changed only per module or the entire system:

port-channel load-balance [algorithm [module module]]  . If FCoE automatically changes load-balancing scheme, will that also affect other non-FCoE ports on that module?

[bogdan.sass · Accepted solution]

Well, I see it this way:

-if an interface is allocated to a storage VDC, it will always use src-dst-oxid load balancing

-if an interface is allocated to a "regular" VDC, it will use the load balancing configured for that particular module, or for the system (with the module-level configuration taking precedence).

Now the question is - what happens to shared interfaces (allocated to both a storage and a "regular" VDC)? Well, it looks like Cisco found a very simple solution - according to this, "You cannot configure a shared interface to be part of a port channel"

[dsampath]

Well, I see still the question hasn't narrowed down. I hit the same one as ataranen

Say if I don't want the default "src-dst-oxid load balancing" for storage VDC and need to change to a different type. Does this mean, I have to change only on default-vdc? IF so that will have a major hit on rest of the non-fcoe traffic.

While this, can you please post if there is any improved situation from last communication? Hope to see some clarity.

[Walter Dey]

Please explain, why you are not happy with default "src-dst-oxid load balancing"

This is widely used in storage, where a out of order FC frame can cause big trouble; this default guarantees, that all FC frames of any read and/or write transcation are sent over the same link (no out of order).

[ataranen]

I think the question is more about the device capability rather than about advantages of src-dst over src-dst-oxid. It is hard to argue that src-dst-oxid is always better. However we do allow src-dst load balancing on other devices for FCoE. On Nexus 5k there is no issue configuring Ethernet port-channel load balancing to be source-destination-mac, which will provide SID/DID load-balancing to FCoE. Why can't we do the same on 7k ?

What if someone evil-minded set's one side of the FCoE port-channel on 5k to be source-destination-mac, and the other side on 7k is the default source-destination-L4port ? That may have more negative impact .   

[Surya ARBY]

Hi.

Could someone get a definitive answer for F1 cards and FCOE load balancing over ethernet port-channels ?

It seems that this src-dst-oxid is not done by default on those cards...