FCoE Port Channels on Single Nexus 5K - possible?

[Mierdin21]

I'm looking for information regarding FCoE on a single Nexus 5548. I'm trying to set up a port channel from a Netapp filer's CNA adapters (2 twinax cables). I was told that for some reason port channels do not work on a single Nexus 5K design (i.e. no vPC), but I didn't know if that meant simply without the use of LACP. I've configured it in both fashions, and it seems that the VFCs do not want to come up, meaning that my filers cannot log in to the SAN. This behavior is what was described to me as what would happen if I tried doing a port channel in this way - basically I've bound the VFCs to the port channels, and since the port channels are composed of ports on the same switch, it just doesn't work. Seems odd that this would bet the case, though it would normally force me to simply buy a second N5K (sneaky sneaky).

Any tips?

[Jeremy Waldrop]

I will have to check the most recent version of the documentation to see if your configuraiton is supported but I know that in previous NX-OS versions vfc interfaces had to be bound to an Ethernet interface and not a port-channel.

I did a config like this with NetApp and 2 5548s and in my config I had 1 NetApp FCoE port in one 5548 and the other NetApp port in the other 5548. I then created single legged port-channels on each 5548 and then a single vPC from the 2. I bound my vfc interfaces to the Ethernet interfaces and it worked great.

I heard that the latest code allows for vfcs to be bound to a port-channel but I am sure how that would work. You may need to set the FCoE VLAN on the NetApp side to whatever your FCoE VLAN is on the Nexus.

[Mierdin21]

Yeah I know it works with 2 N5Ks because you can use vPC and this problem goes away. Unfortunately I only have one, which means I have to have two links to the same Nexus. I am running new code and am therefore able to bind the VFCs to the port channel directly. Unfortunately I am not able to see the VFCs come up when I try to get the filers to log in to the SAN. This is the behavior I was told to expect.

I found the below image on another thread. It said it was from a Networkers presentation. I'm looking for official documentation that goes over this problem, and it would be nice if it shows a configuration that allows this to work, though that's not terribly important. At this point I'm just looking for an official answer about this.

[usujlana]

This might be helpfull :

http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/operations/n5k_fcoe_ops.html

"

Note

FCoE  is not supported on a FEX interface or port channel interfaces when the  FEX is connected to two switches in a FEX active-active topology.

"

[Mierdin21]

That's not my issue. I'm using only one Nexus 5K, meaning that if I want to have a port channel running between the 5K and the Netapp filers, I have to bind the virtual fibre channel interfaces to that port channel. I believe that in doing so, when there are more than one links as a member of that port channel, the VFCs simply go down by design and will not come up. For some reason VFCs require to be bound to a single physical interface, whether that interfaces is a single phyiscal port or a port channel that's only configured to have one port in it.

Either way, I had to configure the Netapp side to be active/passive. I've bound the VFCs to each port (total of 4, so 4 VFCs as well). No port channels are being used on the Nexus side. This is not ideal but it works.

I would still like to see some documentation regarding the behavior of the VFCs when presented with multiple links in this way. Has anyone seen any documentation regarding this?

[Jeremy Waldrop · Accepted solution]

In this scenario wouldn't you create 2 vfc interfaces and then bind each to their respective Ethernet interface? You could still use the port-channel for Ethernet traffic.

[scott.pendleton]

This may sound convuluted, but this is how I understand it as it was explained to me.

The reason you can't bind the vfc to a port-channel with multiple interfaces on a single Nexus stems from the function of fibre-channel logic processing on the port-asic. In the Nexus 5548/96, each port has it's own asic. The VFC borrows the processor for Fibre-channel on the port-assic of the physcial interface. When the interface is a port-channel with multiple active links the fibre-channel commands get round-robbined to the various physical ports which prevents the vFC itself form compelting a flogi into the fibre-channel fabric and this is why the interface goes down. If you only have one interface, no round-robbin occurs and the flogi completes.

Its either a flaw in the software that it allows you to do it in the first place or a feature that is expected but not yet fully implemented. 

[vdsudame]

In Fiber Channel world a when a FC port logs into a switch it gets assigned a unique pwwn which is a port world wide name. Whenever a different host or server CNA logs into a different switch port it gets assigned another unique pwwn. We essentially cannot have two ports log in with the same unique wwn as the port is differentiated by its unique wwn. So its not really the asic but its pwwn because of which we cannot bundle more than one port in a port channel and bind that in the vfc. The software will also not allow you to do that. So one cannot even bundle two different ports across two different asics in a port channel and bind them in a vfc.

let me know if this clarifies.

Thx, Vin.

[scott.pendleton]

Yeah, but in the case he is describing the you would have two unique pwwns and a common nwwn, which is allowed and has been supported on the MDS for years.

[vdsudame]

while bringing up the vfc we factor in the port wwn and not the node wwn. hence we cannot bind two ports in vfc, be it physical or logical. the vfc will not come up that way.

[Cisco]

Hi, i know this is quite old, but i just stumbled over the same issue today...

Single 5548 software 7.0(1)N1(1) 

I dont seem to be able to bind individual interfaces to a vfc if the interface is in a Port-Channel.

nmc(config-if)# bind interface ethernet 1/19
ERROR: fcoe_mgr: VFC cannot be bound to Port Channel as it has more than one member (err_id 0x4207002C)

I tried to remove the interface from the Port-Channel, then i can add it to the vfc, but this prevents me from putting it back into the port-channel

Any ideas?

Thanks

[Jeremy Waldrop]

yeah, make sure the port channel has only one member, this configuration is designed for hosts with CNAs where there is one connection to each vPC peer switch and you need boot from SAN functionality. Or it is for NetApp CNA where there is one connection to each vPC peer

[Cisco]

Thanks, Jeremy, this is for a single HP Host with a dual port CNA CN1100E connected to a single 5548 via LACP.

Thanks

[Mierdin21]

I really hate it when people open threads and don't close them properly.

My bad guys...

Anyways, long story short, I did exactly what Jeremy suggested. I kept the port channel interfaces in place to properly aggregate ethernet traffic, but bound each vfc interface to a specific ethernet interface, rather than the port channel.

All FC target interfaces logged in via each unique VFC and things worked fine and dandy.

A colleague of mine reminded me that more detail on this can be found in the NX-OS Fibre Channel over Ethernet Configuration Guide.

Thanks for the help, everyone!