Buy or Renew
Buy or Renew
Cisco + Splunk: It’s a new day for your data. Learn more.
 
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
cancel
Start a conversation
2111
Views
5
Helpful
13
Replies

Trying to Test CNAs

Pete89
Level 2
Level 2

‎08-12-2011 09:41 AM

Hello,

We are about to dive in the FCoE pool but before I but a few hundred CNAs I want to test them.

Here is my LAB hardware:

1 Server with 1 Emulex OCE1102-FX CNA

1 Nexus 5596

1 Netapp 3240

Questions:

1. Does anyone have what the configuration on the 5k would look like?

2. Do I need to communicate some information to the Netapp admin so the server can see the storage?

3. Does the Netapp admin need to comminicate anything to me?

4. Is this enough hardware to just test the CNAs?

Thanks for youe help,

P.

Labels:
0 Helpful
13 Replies 13

Pete89
Level 2
Level 2

‎08-12-2011 12:17 PM

So here is what I have so far:

Config changes for Nexus 5596k & FCoE where Interface ethernet 1/1 has a twinax cable going to a server's

CNA and interface 1/2 has a twinax cable going to a Netapp controller

QoS Stuff:

nexus5k(config-sys-qos)# service-policy type qos input fcoe-default-in-policy

nexus5k(config-sys-qos)# service-policy type queuing input fcoe-default-in-policy

nexus5k(config-sys-qos)# service-policy type queuing output fcoe-default-out-policy

nexus5k(config-sys-qos)# service-policy type network-qosfcoe-default-nq-policy

------------------------------------------

Setting up VSAN and VLAN mapping

nexus5k(config)# vsan database

nexus5k(config-vsan-db)# vsan 100

nexus5k(config)# vlan 100

nexus5k(config-vlan)# name VSAN100_map

nexus5k(config-vlan)# fcoe vsan 100

------------------------------------------------------

Create Virtual Fiber Channel Interfaces (vfc)

nexus5k(config)# interface vfc 1

nexus5k(config-if)# bind interface ethernet 1/1

nexus5k(config-if)# no shutdown

nexus5k(config-if)# exit

--------------------------------------------------------

Assign vfc to appropriate VSAN

nexus5k(config)# vsan database

nexus5k(config-vsan-db)# vsan 100 interface vfc 1

--------------------------------------------------------

Setup Interface for CNA interface

nexus5k(config)# interface ethernet 1/1

description To ServerX with Emulex CNA

nexus5k(config-if)# switchport mode trunk

nexus5k(config-if)# spanning-tree port type edge trunk

nexus5k(config-if)# switchport trunk allowed vlan 100

Now I still dont know what I need to do on the interface going to the NetApp. Also I dont know if what I have above is complete or not. Once I do I will be sure to document in this thread.

0 Helpful

mfrase
Level 1
Level 1

‎08-12-2011 12:26 PM

Example configurations can be found in the Nexus 5500 Operations guide:


http://www.cisco.com/en/US/partner/docs/switches/datacenter/nexus5000/sw/operations/n5k_fcoe_ops_appB.html

The Netapp storage controller can be attached 2 ways to allow Block level storage to be served upo to your FCoE host, attached via FC to a native FCon the 5500 or UP port configured as FC port on the 5500., Or the Netapp can attach via FCoE to a Ethernet port on the 5500. What method is the NetApp attaching by?

Zoning will need to be done on the 5500 to allow the host to see the storage Target.

A single Nexus 5500 switch with proper FCoE license, cables/optics and proper ports to match the connection requirements of the NetApp is all you need.

http://www.cisco.com/en/US/partner/docs/switches/datacenter/nexus5000/sw/operations/n5k_fcoe_ops_appB.html

0 Helpful

Pete89
Level 2
Level 2
In response to mfrase

‎08-12-2011 12:31 PM

>What method is the NetApp attaching by?

I am not too sure what you mean by this question. Physically it is attaching itself with a twinax cable to the 5596. Perhaps you mean native FC or FCoE? What is your recommendation?

0 Helpful

Pete89
Level 2
Level 2
In response to Pete89

‎08-12-2011 12:36 PM

Also the link you gave me gives me a 403.

0 Helpful

mfrase
Level 1
Level 1
In response to Pete89

‎08-12-2011 12:51 PM

What method you use to attach NetApp tp 5500 will depend on where you are going with Unified IO in yor Network.  If you wish to move to Ethernet as common DataCenter transport then go with FCoE attached storage.  If you are still well establisled with Fibre Channel and have the Network SAN skills and investment you will continue with, then you can look towards FC attached.

The link I attached works for me, but I am full access to Cisco.com,  look for Design guides in the Nexus 5000 document area of Cisco.com, you will need to log into cisco.com.

0 Helpful

Pete89
Level 2
Level 2
In response to mfrase

‎08-12-2011 01:02 PM

Our roadmap is Unified fabric, and we are not entrenced in FC at the moment. The great thing about this buildout is it's green pastures. We get to build it from the ground up.

That link works for me now. I have a logon for Cisco too but for some reason it wasnt letting me get to it.

Many thanks!!

Also I will put up the whole config when I get it working.

0 Helpful

Pete89
Level 2
Level 2
In response to Pete89

‎08-15-2011 08:12 AM

Hello again,

I have the CNA installed on the Windows Machine and after patching the server with

WindowsServer2003-KB932755-x86-ENU which updates storport.sys I was able to install the dirvers for FCoE and for Ethernet. I now can see the 2 HBAs and two new 10 GB NICs.

On the switch I can do a show flogi database vsan 100 and here is what I get:

nexus5k# sh flogi database vsan 100

--------------------------------------------------------------------------------

INTERFACE        VSAN    FCID           PORT NAME               NODE NAME

--------------------------------------------------------------------------------

vfc1             100   0x270000  10:00:00:00:c9:bb:bf:03 20:00:00:00:c9:bb:bf:03

Total number of flogi = 1.

Woohoo! Now I just need to get another twinax cable for the NETApp controller. I setup the ethernet interface on the 5k for NETApp the same as the Windows box:

interface Ethernet1/1

  description To Server X with Emulex CNA

  switchport mode trunk

  switchport trunk allowed vlan 100

  spanning-tree port type edge trunk

interface Ethernet1/2

  description To Netapp Controller

  switchport mode trunk

  switchport trunk allowed vlan 100

  spanning-tree port type edge trunk

Does this look right? Also, is zoning a requirement?

Thanks,

P.

0 Helpful

dynamoxxx
Level 5
Level 5
In response to Pete89

‎08-15-2011 11:56 AM

looks good, yes zoning is required. At this point you treat it as a regular FC connected host.

@dynamoxxx
0 Helpful

mfrase
Level 1
Level 1
In response to dynamoxxx

‎08-16-2011 06:45 AM

You could use "Permit" zoning in the FCoE vsan for the lab, not a best practice in real production.

Your also going to want to allow other VLANs to be trunked to the host, real world would not just allow FCoE VLAN

0 Helpful

Pete89
Level 2
Level 2
In response to mfrase

‎08-17-2011 04:05 AM

Thanks for the input.

I have a drawing here of what I think my expanded lab will look like in the few weeks (right now I have 1 5k and a NETApp)

A few things I should explain:

S is the server with a two port CNA

Red Circles are for VPCs

Each racktop 2232 goes goes to one 5k. I read that you have to keep it that way in order to maintain the fabric paths.

Each rack has two 2232s, so you will see I have one port from the CNA going to each one.

When I look at this I see that if the 5k on the right goes down I am in big trouble no? Does one port from the CNA need to go to the other racks 2232?

Any other problems with this design?

Many many thanks for looking this over

0 Helpful

Pete89
Level 2
Level 2
In response to Pete89

‎08-17-2011 12:42 PM

OK my first test lab is done. A quick reminder of what I am working with (dont be fooled by the diagram above, That is the next build):

1 Server with 1 Emulex OCE1102-FX CNA

1 Nexus 5596

1 Netapp 3240

And here is the complete config with some descriptions:

Config for Nexus 5596k for FCoE where Interface ethernet 1/1 has a twinax cable going to a server's

CNA and interface 1/2 has a twinax cable going to a Netapp controller

QoS Stuff:

nexus5k(config-sys-qos)# service-policy type qos input fcoe-default-in-policy

nexus5k(config-sys-qos)# service-policy type queuing input fcoe-default-in-policy

nexus5k(config-sys-qos)# service-policy type queuing output fcoe-default-out-policy

nexus5k(config-sys-qos)# service-policy type network-qosfcoe-default-nq-policy

------------------------------------------------------------------------------------

Setting up VSAN and VLAN mapping

nexus5k(config)# vsan database

nexus5k(config-vsan-db)# vsan 100

nexus5k(config)# vlan 100

nexus5k(config-vlan)# name VSAN100_map

nexus5k(config-vlan)# fcoe vsan 100

------------------------------------------------------------------------------------

Create Virtual Fiber Channel Interfaces (vfc)

nexus5k(config)# interface vfc 1

nexus5k(config-if)# bind interface ethernet 1/1

nexus5k(config-if)# no shutdown

nexus5k(config-if)# exit

------------------------------------------------------------------------------------

Assign vfc to appropriate VSAN

nexus5k(config)# vsan database

nexus5k(config-vsan-db)# vsan 100 interface vfc 1

---------------------------------------------------------------------------------------

Setup Interface for CNA Interface

nexus5k(config)# interface ethernet 1/1

description To Server X with Emulex CNA

nexus5k(config-if)# switchport mode trunk

nexus5k(config-if)# spanning-tree port type edge trunk

nexus5k(config-if)# switchport trunk allowed vlan 100

--------------------------------------------------------------------------------

Here is what the 5k see as part of its fabric:

nexus5k(config)# sh flogi database

--------------------------------------------------------------------------------

INTERFACE        VSAN    FCID           PORT NAME               NODE NAME

--------------------------------------------------------------------------------

vfc1             100   0x270000  10:00:00:00:c9:bb:bf:03 20:00:00:00:c9:bb:bf:03

vfc2             100   0x270001  50:0a:09:82:9d:d4:a2:05 50:0a:09:80:8d:d4:a2:05

Total number of flogi = 2.

The top one is the CNA and the bottom is the NETApp

----------------------------------------------------------

Create Zone for vsan 100. Set zone, Activate Zone and Distribute

nexus5k(config)#zone name windows_server_cna1 vsan 100

Nexus-5020-2(config-zone)# member pwwn 10:00:00:00:c9:bb:bf:03

Nexus-5020-2(config-zone)# member pwwn 50:0a:09:82:9d:d4:a2:05

Nexus-5020-2(config-zone)# exit

Nexus-5020-2(config)# zoneset name FCoE_100 vsan 100

Nexus-5020-2(config-zoneset)# member windows_server_cna1

Nexus-5020-2(config-zoneset)# zoneset activate name FCoE_100 vsan 100

Zoneset activation initiated. check zone status

Nexus-5020-2(config)# zoneset distribute full vsan 100

----------------------------------------------------------------------------------------

This was all I did on the switch. My NetApp guy added the CNAs wwn to the

initiators list in NetApp and voila! After that I rescanned the disks

in Disk Manager in Windows I could see the drive on the NETApp.

I put Iometer on the Windows box and the app said I was only getting about

2 megabytes a sec. Very strange. 3 gig file copies to and from the NETApp

took about 10 secs. Iometer is what EMC uses to benchmark. Maybe I was doing

something wrong.

0 Helpful

Pete89
Level 2
Level 2
In response to Pete89

‎08-18-2011 06:15 AM

Also a note on the design for LAB 2 which I included a drawing of in an earlier post. That design WONT work.  Here is a link that explains why:

http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/operations/n5k_fcoe_ops.html#wp1096248

What I get out of it is you need to do it this way in order to keep the facric paths seperate. This design is the correct one (I hope):

0 Helpful
Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Customers Also Viewed These Support Documents