08-12-2011 09:41 AM
Hello,
We are about to dive in the FCoE pool but before I but a few hundred CNAs I want to test them.
Here is my LAB hardware:
1 Server with 1 Emulex OCE1102-FX CNA
1 Nexus 5596
1 Netapp 3240
Questions:
1. Does anyone have what the configuration on the 5k would look like?
2. Do I need to communicate some information to the Netapp admin so the server can see the storage?
3. Does the Netapp admin need to comminicate anything to me?
4. Is this enough hardware to just test the CNAs?
Thanks for youe help,
P.
08-12-2011 12:17 PM
So here is what I have so far:
Config changes for Nexus 5596k & FCoE where Interface ethernet 1/1 has a twinax cable going to a server's
CNA and interface 1/2 has a twinax cable going to a Netapp controller
QoS Stuff:
nexus5k(config-sys-qos)# service-policy type qos input fcoe-default-in-policy
nexus5k(config-sys-qos)# service-policy type queuing input fcoe-default-in-policy
nexus5k(config-sys-qos)# service-policy type queuing output fcoe-default-out-policy
nexus5k(config-sys-qos)# service-policy type network-qosfcoe-default-nq-policy
------------------------------------------
Setting up VSAN and VLAN mapping
nexus5k(config)# vsan database
nexus5k(config-vsan-db)# vsan 100
nexus5k(config)# vlan 100
nexus5k(config-vlan)# name VSAN100_map
nexus5k(config-vlan)# fcoe vsan 100
------------------------------------------------------
Create Virtual Fiber Channel Interfaces (vfc)
nexus5k(config)# interface vfc 1
nexus5k(config-if)# bind interface ethernet 1/1
nexus5k(config-if)# no shutdown
nexus5k(config-if)# exit
--------------------------------------------------------
Assign vfc to appropriate VSAN
nexus5k(config)# vsan database
nexus5k(config-vsan-db)# vsan 100 interface vfc 1
--------------------------------------------------------
Setup Interface for CNA interface
nexus5k(config)# interface ethernet 1/1
description To ServerX with Emulex CNA
nexus5k(config-if)# switchport mode trunk
nexus5k(config-if)# spanning-tree port type edge trunk
nexus5k(config-if)# switchport trunk allowed vlan 100
Now I still dont know what I need to do on the interface going to the NetApp. Also I dont know if what I have above is complete or not. Once I do I will be sure to document in this thread.
08-12-2011 12:26 PM
Example configurations can be found in the Nexus 5500 Operations guide:
The Netapp storage controller can be attached 2 ways to allow Block level storage to be served upo to your FCoE host, attached via FC to a native FCon the 5500 or UP port configured as FC port on the 5500., Or the Netapp can attach via FCoE to a Ethernet port on the 5500. What method is the NetApp attaching by?
Zoning will need to be done on the 5500 to allow the host to see the storage Target.
A single Nexus 5500 switch with proper FCoE license, cables/optics and proper ports to match the connection requirements of the NetApp is all you need.
08-12-2011 12:31 PM
>What method is the NetApp attaching by?
I am not too sure what you mean by this question. Physically it is attaching itself with a twinax cable to the 5596. Perhaps you mean native FC or FCoE? What is your recommendation?
08-12-2011 12:36 PM
Also the link you gave me gives me a 403.
08-12-2011 12:51 PM
What method you use to attach NetApp tp 5500 will depend on where you are going with Unified IO in yor Network. If you wish to move to Ethernet as common DataCenter transport then go with FCoE attached storage. If you are still well establisled with Fibre Channel and have the Network SAN skills and investment you will continue with, then you can look towards FC attached.
The link I attached works for me, but I am full access to Cisco.com, look for Design guides in the Nexus 5000 document area of Cisco.com, you will need to log into cisco.com.
08-12-2011 12:54 PM
08-12-2011 01:02 PM
Our roadmap is Unified fabric, and we are not entrenced in FC at the moment. The great thing about this buildout is it's green pastures. We get to build it from the ground up.
That link works for me now. I have a logon for Cisco too but for some reason it wasnt letting me get to it.
Many thanks!!
Also I will put up the whole config when I get it working.
08-15-2011 08:12 AM
Hello again,
I have the CNA installed on the Windows Machine and after patching the server with
WindowsServer2003-KB932755-x86-ENU which updates storport.sys I was able to install the dirvers for FCoE and for Ethernet. I now can see the 2 HBAs and two new 10 GB NICs.
On the switch I can do a show flogi database vsan 100 and here is what I get:
nexus5k# sh flogi database vsan 100
--------------------------------------------------------------------------------
INTERFACE VSAN FCID PORT NAME NODE NAME
--------------------------------------------------------------------------------
vfc1 100 0x270000 10:00:00:00:c9:bb:bf:03 20:00:00:00:c9:bb:bf:03
Total number of flogi = 1.
Woohoo! Now I just need to get another twinax cable for the NETApp controller. I setup the ethernet interface on the 5k for NETApp the same as the Windows box:
interface Ethernet1/1
description To Server X with Emulex CNA
switchport mode trunk
switchport trunk allowed vlan 100
spanning-tree port type edge trunk
interface Ethernet1/2
description To Netapp Controller
switchport mode trunk
switchport trunk allowed vlan 100
spanning-tree port type edge trunk
Does this look right? Also, is zoning a requirement?
Thanks,
P.
08-15-2011 11:56 AM
looks good, yes zoning is required. At this point you treat it as a regular FC connected host.
08-16-2011 06:45 AM
You could use "Permit" zoning in the FCoE vsan for the lab, not a best practice in real production.
Your also going to want to allow other VLANs to be trunked to the host, real world would not just allow FCoE VLAN
08-17-2011 04:05 AM
Thanks for the input.
I have a drawing here of what I think my expanded lab will look like in the few weeks (right now I have 1 5k and a NETApp)
A few things I should explain:
S is the server with a two port CNA
Red Circles are for VPCs
Each racktop 2232 goes goes to one 5k. I read that you have to keep it that way in order to maintain the fabric paths.
Each rack has two 2232s, so you will see I have one port from the CNA going to each one.
When I look at this I see that if the 5k on the right goes down I am in big trouble no? Does one port from the CNA need to go to the other racks 2232?
Any other problems with this design?
Many many thanks for looking this over
08-17-2011 12:42 PM
OK my first test lab is done. A quick reminder of what I am working with (dont be fooled by the diagram above, That is the next build):
1 Server with 1 Emulex OCE1102-FX CNA
1 Nexus 5596
1 Netapp 3240
And here is the complete config with some descriptions:
Config for Nexus 5596k for FCoE where Interface ethernet 1/1 has a twinax cable going to a server's
CNA and interface 1/2 has a twinax cable going to a Netapp controller
QoS Stuff:
nexus5k(config-sys-qos)# service-policy type qos input fcoe-default-in-policy
nexus5k(config-sys-qos)# service-policy type queuing input fcoe-default-in-policy
nexus5k(config-sys-qos)# service-policy type queuing output fcoe-default-out-policy
nexus5k(config-sys-qos)# service-policy type network-qosfcoe-default-nq-policy
------------------------------------------------------------------------------------
Setting up VSAN and VLAN mapping
nexus5k(config)# vsan database
nexus5k(config-vsan-db)# vsan 100
nexus5k(config)# vlan 100
nexus5k(config-vlan)# name VSAN100_map
nexus5k(config-vlan)# fcoe vsan 100
------------------------------------------------------------------------------------
Create Virtual Fiber Channel Interfaces (vfc)
nexus5k(config)# interface vfc 1
nexus5k(config-if)# bind interface ethernet 1/1
nexus5k(config-if)# no shutdown
nexus5k(config-if)# exit
------------------------------------------------------------------------------------
Assign vfc to appropriate VSAN
nexus5k(config)# vsan database
nexus5k(config-vsan-db)# vsan 100 interface vfc 1
---------------------------------------------------------------------------------------
Setup Interface for CNA Interface
nexus5k(config)# interface ethernet 1/1
description To Server X with Emulex CNA
nexus5k(config-if)# switchport mode trunk
nexus5k(config-if)# spanning-tree port type edge trunk
nexus5k(config-if)# switchport trunk allowed vlan 100
--------------------------------------------------------------------------------
Here is what the 5k see as part of its fabric:
nexus5k(config)# sh flogi database
--------------------------------------------------------------------------------
INTERFACE VSAN FCID PORT NAME NODE NAME
--------------------------------------------------------------------------------
vfc1 100 0x270000 10:00:00:00:c9:bb:bf:03 20:00:00:00:c9:bb:bf:03
vfc2 100 0x270001 50:0a:09:82:9d:d4:a2:05 50:0a:09:80:8d:d4:a2:05
Total number of flogi = 2.
The top one is the CNA and the bottom is the NETApp
----------------------------------------------------------
Create Zone for vsan 100. Set zone, Activate Zone and Distribute
nexus5k(config)#zone name windows_server_cna1 vsan 100
Nexus-5020-2(config-zone)# member pwwn 10:00:00:00:c9:bb:bf:03
Nexus-5020-2(config-zone)# member pwwn 50:0a:09:82:9d:d4:a2:05
Nexus-5020-2(config-zone)# exit
Nexus-5020-2(config)# zoneset name FCoE_100 vsan 100
Nexus-5020-2(config-zoneset)# member windows_server_cna1
Nexus-5020-2(config-zoneset)# zoneset activate name FCoE_100 vsan 100
Zoneset activation initiated. check zone status
Nexus-5020-2(config)# zoneset distribute full vsan 100
----------------------------------------------------------------------------------------
This was all I did on the switch. My NetApp guy added the CNAs wwn to the
initiators list in NetApp and voila! After that I rescanned the disks
in Disk Manager in Windows I could see the drive on the NETApp.
I put Iometer on the Windows box and the app said I was only getting about
2 megabytes a sec. Very strange. 3 gig file copies to and from the NETApp
took about 10 secs. Iometer is what EMC uses to benchmark. Maybe I was doing
something wrong.
08-18-2011 06:15 AM
Also a note on the design for LAB 2 which I included a drawing of in an earlier post. That design WONT work. Here is a link that explains why:
What I get out of it is you need to do it this way in order to keep the facric paths seperate. This design is the correct one (I hope):
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide