ā06-05-2012 07:06 AM
Hi, I am a networking guy and i need to understand which configuration do i need to connect 2 UCS 6120 to my Nexus 5000 (2 with VPC),
I know that i must activate npiv and fcoe feature, and read about creating vsans and vfc, my first question is does anybody have a procedure? (for example: 1.- You must create a vsan, 2.- You need to associate vsan with vlans, 3.- etc)..
Secondly, i have some doubts,
1.- If i have fc interfaces do i need to create an vfc interface? (or this is only if i am going to use the 10G ethernet interfaces?.)
2.- Why i need to bind the vsan to the vlans?.
Please help me with this, the SAN world is new to me!!
Hope you could help me,
Regards,
Juan Pablo Hidalgo
ā01-24-2013 10:35 AM
Well, I got it all working after much tinkering. One lingering question, probably answered by reading the specifications but still: I had to configure the port from the FI (6120) as an "FCoE uplink port" in order to see the fabric logins in the Nexus switch. My initial assumption was that a "Network" port would be sufficient, Unified Fabric and all. The VLANs still carry, so I'm all good in this configuration, but to me it appears a bit limiting to only allow exactly one VSAN on an FCoE uplink. What if I want to have a multi-tennant solution with many more VSANs (although the FI supports 32 VSANs unless I'm mistaken).
ā01-24-2013 12:18 PM
Hi,
We are planning to deploy a FCoE Multihop scenario where we have two FIC 6248UP connecting to two Nexus 5548UP configured in vPC. The question here is if we can carry ethernet and FCoE traffic through the same cable, as the hosts do using CNA's. When I heard about FCoE Multihop northbound support in Fabric Interconnect I always thought that FCoE traffic would pass on the same cable as the ethernet traffic. Is it not possible??
The only possible way I am seeing to do FCoE Multihop is to link FIC-A to N5K-A and FIC-B to N5K-B for FCoE traffic. And keep ethernet traffic using vPC in a classic fashion. Is it?
ā01-24-2013 12:32 PM
You can use the current vPC uplink for both Ethernet and FCoE traffic if you configure the port as Unified within UCSM. If you configure the existing uplink port as an FCoE Uplink it will change the role to Unified Uplink.
After you change the port to Unified you will need to map the appropriate VSAN to it. To switch over from traditional FC to FCoE disable the FC uplinks.
ā02-13-2013 07:42 PM
Hi Jeremy,
I just read this article http://blogs.cisco.com/datacenter/ucs-multihop-fcoe-in-under-an-hour/ and it seems that use vPC to carry FCoE + Ethernet traffic is not a valid scenario. See bellow:
3. With the Nexus 5k systems, you can PortChannel FCoE and Ethernet links together, or you can use vPCs. You cannot do both. If you want need to use vPCs on your Ethernet links, you will need to use separate links for FCoE (but you can port-channel those together, if you wish). Using UCS with a Nexus 7k you will need to use dedicated FCoE links between the Fabric Interconnects and the 7k.
Do you have any documentation about this?? I googled a lot, but it seems that Cisco did not release many related content yet.
Thank you!
ā02-22-2013 04:09 AM
Hey Rafael, I was doing this in a lab just to see if it would work. I was able to get it working but due to the horrible performance of accessing a LUN from an ESXi host over multihop FCoE I had to revert back to traditional FC connections.
This performance thing is a major bug in my opinion. I tested it again last week on gen 2 hardware and a brand new VNX and had the same results. It was so bad I couldn't even get a Windows VM install to finish. In this setup I had 2 dedicated Ethernet uplinks configured in an FCoE Port Channel. Once I moved to traditional FC there were no performance issues.
ā02-28-2013 04:06 AM
Hi Jeremy, So you reverted to native FC between the FI and Nexus or was it dedicated āSANā FCoE uplink with separate LAN uplinks (vPC)
ā02-28-2013 04:57 AM
Reverted to native FC.
When I had dedicated FCoE links there was a serious performance issue. I only intialy setup the FCoE on the vPC links just to see if it could be done but I never used it.
I have tested this on both gen 1 6100s/2104/M81KR and gen 2 6200/2208/1240/1280 hardware and it is the same result. I can boot the ESXi hosts from the SAN boot LUN but when I power on a VM it takes 20+ minutes and I can't ever login to it.
I am using ESXi 5.1 with the latest VIC enic/fnic drivers.
ā02-28-2013 05:04 AM
Sounds bad! I better order some FC SFPs....
One last question, from the Nexus to storage was this FCoE or native?
Thanks for the heads up.
ā02-28-2013 05:26 AM
Native FC from an EMC VNX. In this same lab I have 2 rack mount ESXi servers with QLogic CNAs with FCoE going through the same Nexus 5548s and same VNX SAN with no performance issues.
ā02-28-2013 08:33 AM
So this is a multi-hop issue then. I have done end to end fcoe deployments before direct from host > nexus > storage without issues.
ā04-10-2013 05:16 PM
it appears that when you can't bind the vfc interface to a portchannel with more than 1 link.
i have 2 nexus 5K uplinked via 2 vpc etherchannels to the ucs 6296
each vpc has 4 links (2 from each nexus)
i will see if i can make the uplinks in 4 different vpc (with 2 total connections each)
anybody has any feedback on this?
ā04-12-2013 02:06 PM
loizosko,
To configure ethernet network infrastructure, you can do the following - { just like yours, 4 links total, 2 links from each nexus }
Nexus 5k-1
interface ethernet 1/1
description Connected to Fabric-Interconnect-A Eth 1/13
channel-group 10 mode active
no shut
interface ethernet 1/2
description Connected to Fabric-Interconnect-B Eth 1/13
channel-group 11 mode active
no shut
interface port-channel10
switchport mode trunk
spanning-tree port type edge trunk
vpc 10
interface port-channel11
switchport mode trunk
spanning-tree port type edge trunk
vpc 11
Nexus 5k-2
interface ethernet 1/1
description Connected to Fabric-Interconnect-A Eth 1/14
channel-group 10 mode active
no shut
interface ethernet 1/2
description Connected to Fabric-Interconnect-B Eth 1/14
channel-group 11 mode active
no shut
interface port-channel10
switchport mode trunk
spanning-tree port type edge trunk
vpc 10
interface port-channel11
switchport mode trunk
spanning-tree port type edge trunk
vpc 11
Go to UCSM, and configure ports 1/13,1/14 (in Fabric-A) as Uplink Ports, Under Lan > Lan Cloud > Fabric A, you can create Port-channels. Repeat the above for Fabric B
Ethernet Uplink Port-channels should come up.
To extend SAN from Data center core to Fabric Interconnects, do the following - (assuming you are trying to configure using FCoE - between 5k and FI and FI is in end host mode)
To maintain SAN seperation, as an example, connect 2 links from 5k-1 to FI-A and 2 links from 5k-2 to FI-B. Create SAN Port-channel
On Nexus 5k-1
feature npiv
feature fcoe
feature lacp
vlan 100
name FCOE_VLAN_100
vsan database
vsan 100
vsan 100 name SAN100
vlan 100
fcoe vsan 100
interface Ethernet 1/21
description Connected to Fabric_Interconnect_A Eth 1/21
channel-group 30 mode active
no shut
interface Ethernet 1/22
description Connected to Fabric_Interconnect_A Eth 1/22
channel-group 30 mode active
no shut
interface port-channel 30
switchport mode trunk
switchport trunk allowed vlan 100
spanning-tree port type edge trunk
interface vfc 30
bind interface port-channel30
switchport trunk allowed vsan 100
switchport mode F
no shut
On Nexus 5k-2
feature npiv
feature fcoe
feature lacp
vlan 101
name FCOE_VLAN_101
vsan database
vsan 101
vsan 101 name SAN101
vlan 101
fcoe vsan 101
interface Ethernet 1/21
description Connected to Fabric_Interconnect_B Eth 1/21
channel-group 30 mode active
no shut
interface Ethernet 1/22
description Connected to Fabric_Interconnect_B Eth 1/22
channel-group 30 mode active
no shut
interface port-channel 30
switchport mode trunk
switchport trunk allowed vlan 101
spanning-tree port type edge trunk
interface vfc 30
bind interface port-channel30
switchport trunk allowed vsan 101
switchport mode F
no shut
Go to UCSM, define VSANS (find in SAN Cloud), under Fabric A and Fabric B, create FCoE Port-channels (select appropriate interfaces), associate the Port-channels to correct VSANs.
Refer
ā04-12-2013 02:39 PM
i wanted to have 4 10GB links out of each FI
your solution only has 2 out of each FI
now i have 2 interfaces on each port channel (4 total per vpc)
your solution calls for only 2 uplinks from each FI
review the 2 diagrams below
ā04-12-2013 02:57 PM
Ioizosko,
Nexus 5k-1
feature lacp
interface ethernet 1/1
description Connected to Fabric-Interconnect-A Eth 1/13
channel-group 10 mode active
no shut
interface ethernet 1/2
description Connected to Fabric-Interconnect-A Eth 1/14
channel-group 10 mode active
no shut
interface ethernet 1/3
description Connected to Fabric-Interconnect-B Eth 1/13
channel-group 11 mode active
no shut
interface ethernet 1/4
description Connected to Fabric-Interconnect-B Eth 1/14
channel-group 11 mode active
no shut
interface port-channel10
switchport mode trunk
spanning-tree port type edge trunk
vpc 10
interface port-channel11
switchport mode trunk
spanning-tree port type edge trunk
vpc 11
Nexus 5k-2
feature lacp
interface ethernet 1/1
description Connected to Fabric-Interconnect-A Eth 1/15
channel-group 10 mode active
no shut
interface ethernet 1/2
description Connected to Fabric-Interconnect-A Eth 1/16
channel-group 10 mode active
no shut
interface ethernet 1/3
description Connected to Fabric-Interconnect-B Eth 1/15
channel-group 11 mode active
no shut
interface ethernet 1/4
description Connected to Fabric-Interconnect-B Eth 1/16
channel-group 11 mode active
no shut
interface port-channel10
switchport mode trunk
spanning-tree port type edge trunk
vpc 10
interface port-channel11
switchport mode trunk
spanning-tree port type edge trunk
vpc 11
In UCSM, configure appropriate Ethernet uplink ports (and create Port-channel configurations) for both Fabrics.
ā04-12-2013 03:34 PM
this is what i currently have
but with this setup cannot pass FCOE
when i try associate the vfc interface to the port channel i get this errorr
N5K-SW1(config-if)# bind interface port-channel 2
ERROR: fcoe_mgr: VFC cannot be bound to Port Channel as it has more than one member (err_id 0x4207002C)
N5K-SW1(config-if)#
all the articles indicate that can't do in this case.
if i do the scenario i indicated (possible solution??) might work but did not test it.
as alternative i could have a separate FC port channel but i would like to see if i can get it via fcoe
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide