Hi, I am a networking guy and i need to understand which configuration do i need to connect 2 UCS 6120 to my Nexus 5000 (2 with VPC),
I know that i must activate npiv and fcoe feature, and read about creating vsans and vfc, my first question is does anybody have a procedure? (for example: 1.- You must create a vsan, 2.- You need to associate vsan with vlans, 3.- etc)..
Secondly, i have some doubts,
1.- If i have fc interfaces do i need to create an vfc interface? (or this is only if i am going to use the 10G ethernet interfaces?.)
2.- Why i need to bind the vsan to the vlans?.
Please help me with this, the SAN world is new to me!!
Hope you could help me,
Juan Pablo Hidalgo
Hi Juan Pablo,
Please follow this guide for details:
UCS Fabric Interconnect can work in two modes: native (NPV) or switch.
By default it's NPV, so it means you need to create TF-N link with Nexus 5k (with NPIV feature active on N5k) for FC links. There is no direct FCoE support between N5k and FI at this point (
-On each N5k (fabric A, VSAN10 and fabric B, VSAN20):
interface fcx/y (attached to UCS FI)
switchport mode F
switchport trunk VSAN 10 (20 on B)
-For FC interfaces, there is no need to create a vfc interface, it's mandatory indeed for 10GigE ports that attach to a CNA: you bind each GigE to a vfc.
-VLAN-VSAN binding is mandatory: you need to have at least one VLAN in fcoe mode that binds to a VSAN, in order for CNA to register pWWN to the name server, then receive FCID. You must also trunk default VLAN on the GigE port attached to CNA in order to allow the DCBX exchange with switch.
vsan 10 interface fcx/y
vlan 10 (20 on B)
fcoe vsan 10 (20 on B)
# sh vlan fcoe
Original VLAN ID Translated VSAN ID Association State
---------------- ------------------ -----------------
10 10 Operational
switchport mode trunk
spanning-tree port type edge trunk
bind interface Ethernetx/y
switchport trunk allowed vsan 1 (default VLAN)
switchport trunk allowed vsan add 10 (20 on B)
vsan 10 interface vfcy
You also need to configure zoning on N5k.
I strongly recommend you to acquire this book if you're new to storage networking, as it covers FCoE and FC integration on Cisco product, procedure-like:
Thanks for your reply and information!!,
I only have one more question, it's about the vlan and vsan, i don't understand what is the purpose of binding the vlan and vsan, i understand that the UCS and N5K needs the vsan, but why the binding, it is because i use the fcoe?,
is this vlan used for traffic between switches, or it is only used for the vsan and that's it.
again thanks you very much for your help,
Hi Juan Pablo,
Binding VLAN-VSAN is needed only for FCoE: switch needs to know on which VLAN mapping it's expected to receive FC data, as FC is encapsulated in the ethernet frame while EtherType will be FCoE.
Also it will first receive FIP (FCoE initialization protocol=control plane) frames on that link, in order to discover/negociate FCoE-capable devices (CNA). This will be used for FCoE VLAN discovery, FCF and fabric login.
FCoE traffic (data plane) will then be received by switch.
Then binded VFC interface will collect the FCoE frames and send them to the FC fabric, decapsulated from ethernet.
So basically this VLAN-VSAN is used within switch (or between switches for multi-hop FCoE), but won't carry classic ethernet traffic. For this one you need to use other VLAN(s) trunked on the GigE port of that CNA.
Also, be aware that you need to disallow these FCoE VLANs over the vPC peer-link of your N5k pair since you need to enforce strict FC fabric A/B segregation for redundancy (load-balancing over fabrics is done by MPIO driver at host level and storage array side).
Jumping in in this thread as a latecomer, but I've not been able to find a straight answer.
"There is no direct FCoE support between N5k and FI at this point", is this still the case?
I'm a newbie in the Cisco (and pretty much Ethernet) world, I come from a traditional FC background. I'm trying to get my head around this, but if I can't connect FCoE natively between a Nexus 5548 and 6120XP FI, I have to rethink my design.
This is no more a limitation, we introduced support for north-bound FCoE (multihop) in latest UCS-M release 2.1(1a).
Please find more info in that article: http://blogs.cisco.com/datacenter/ucs-multihop-fcoe-in-under-an-hour/
I hope this answers your question.
Thanks for the quick reply. I've read the post you linked, but I still feel I'm missing something quite fundamental.
I have a Cisco UCS system with the latest images, and it looks like its running 2.1(1a), so I should be covered.
There are two Fabric Interconnects, 6120XP connected to a single Nexus 5548UP. This is just a lab setup, nothing in production.
I've created a VLAN 1001, mapped this to VSAN 10. Created two vfc's, one on the port where Fabric Interconnet A connects, and one vfc on a port on the Nexus 5548 where I've got a Netapp unified storage port connected.
Both ports are trunk ports.
Created a VSAN 10 in UCS, linked it to VLAN 1001. I have an ESXi with a vHBA connected to VSAN 10.
Allowed VLAN 1001 on the ports where I have the link to the 6120 and the port where the NetApp connects.
show flogi database shows nothing
fcoe is enabled.
The 6120XP is in switch mode, but that should be ok now that we have FCoE multihop, or?
I've googled and googled, but since the multihop support is so recent, I'm still searching for some useful documentation on how to configure all this.
From our deployment experience, it should work fine if FI are in end-host mode (NPV), but in FC switching mode there may be a problem (hardware limitation on 1st generation FI 61xx). Ethernet switching mode is not in the scope for this limitation.
Please check if you have similar fault reported in your 6120XP switch-mode:
FI-A# scope fc-up FI-A /fc-uplink # scope fab a FI-A /fc-uplink/fabric # scope fcoeinterface x y FI-A /fc-uplink/fabric/fcoeinterface # show fault Severity Code Last Transition Time ID Description --------- -------- ------------------------ -------- ----------- Major F1084 2013-01-10T17:49:38.352 154788 FCoE uplink port 1/9 cannot be supported on springfields when FI is in eth endhost mode and FC switching mode Major F1083 2013-01-10T15:27:45.126 105418 FCoE uplink is down on Vsan 1
If so, please modify design for FC NPV, or test with a 2nd gen FI 62xx.
I get this on the 6120:
fi-ucs-got-demo-A /fc-uplink/fabric/fcoeinterface # show fault
Severity Code Last Transition Time ID Description
--------- -------- ------------------------ -------- -----------
Cleared F1082 2013-01-18T14:32:26.882 886546 FCoE uplink port 1/3 is down
Major F1083 2013-01-18T14:32:11.505 886550 FCoE uplink is down on Vsan 1
Major F1083 2013-01-18T14:32:11.505 886549 FCoE uplink is down on Vsan 10
fi-ucs-got-demo-A /fc-uplink/fabric/fcoeinterface #
And this on the Nexus 5548 where I have the link to the 6120 (Eth 1/3)
ucs# show int vfc 13
vfc13 is trunking (Not all VSANs UP on the trunk)
Bound interface is Ethernet1/3
Hardware is Ethernet
Port WWN is 20:0c:54:7f:ee:e2:af:3f
Admin port mode is F, trunk mode is on
snmp link state traps are enabled
Port mode is TF
Port vsan is 10
Trunk vsans (admin allowed and active) (1,10,20)
Trunk vsans (up) ()
Trunk vsans (isolated) ()
Trunk vsans (initializing) (1,10,20)
1 minute input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
1 minute output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
0 frames input, 0 bytes
0 discards, 0 errors
0 frames output, 0 bytes
0 discards, 0 errors
last clearing of "show interface" counters never
Interface last changed at Fri Jan 18 12:00:06 2013
Here is sample configuration to enable FCoE between UCS and N5K
##### On UCSM 2.1 ######
1) FC end mode
2) Configure a port as FCoE uplink and assign the interface to VSAN 10
## Verifying the output
6248-01-A(nxos)# show run int vfc 1255
bind interface Ethernet1/24
switchport mode NP
##### Configuration on N5K #####
1) Enable NPIV,fcoe and lldp features
2) Create vlan, configure ethernet and VFC interfaces
----------- sample configuration ----------------
fcoe vsan 10
switchport mode trunk
spanning-tree port type edge
spanning-tree bpdufilter enable
int vfc 14
bind interface Ethernet1/19
switchport trunk allowed vsan 1
switchport trunk allowed vsan add 10
vsan 10 interface vfc 119
#### Comands to verify the configuration #####
show fcoe vlan
show int vfc 119
show fcoe database
show flogi table
I did not understand why you used vfc14, and after vfc119? Could you explain please?
Another question is why did you allowed VSAN 1??
Lastly, if you would allow VLANs in the ethernet port, would allow VLAN 1001 or also VLAN 1?? I'm question this because I believe FIP relies in the native VLAN.. Is it right?
Thanks for the detailed reply. I'm probably way off here, but what if I wanted to run the FI in "switch mode" for FC? This is based on the assumption that now with FCoE multihop support, I can connect the FI to the Nexus 5548 as a switch and still have end to end FCoE connectivity-
The Nexus 5000 have hardware limitation for this, but issue don't exist for Nexus 5500 to run in FC host mode and FCoE multihop.
We do not need FI to be in FC switch mode. FC end host mode ( NPV) allows FI ports ( NP ) to connect NPIV ( N5K switch ) .
If N5K software version allows multihop, you can then extend FCoE from N5K to another switch via VE port.