Learning FCoE: which switch?

[JimCh]

I'm new to this technology, so please forgive any misuses of terminology.

I'm interested in learning how to configure FC over Ethernet so I can experiment "talking" to one of my old FC tape drives over a 10GBE network.

To this end I've bought a 10GBE CNA HBA and configured it in my workstation, and now I need a switch to convert those FCoE packets to FC.

There are a number of affordable used units on eBay: Brocade/Dell/EMC MP-8000P/HP AP802, and the Cisco 6120XP & 6248UP.

The Cisco switches seem to be cheaper so - as I don't have much money to spend on this - they're the ones I'm drawn to.

Am I right in assuming they can be used for this purpose (assuming they have the N10-E0060 FC module installed)? And am I right in thinking the 6248 is just a newer "version" of the 6120 in terms of what it does?

Is there anything I need to be aware of when purchasing?

TIA!

[balaji.bandi]

how about nexus 5K UP

[Sergiu.Daniluk · Accepted solution]

Hi @JimCh 

6248 is indeed the next generation of 6120. However, the 6120XP & 6248UP are Fabric Interconnects, which are part of Cisco UCS ecosystem. You won't be able to use them as you intend - as switches. 

The equivalent of the 6248UP in terms of dc switches is Nexus 5548UP (they literally have the same ASIC).

https://www.cisco.com/c/en/us/products/switches/nexus-5548up-switch/index.html 

However, on top of the switch, you will also need the FC_FEATURES_PKG license to be able to run the FC/FCoE features.

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/sw/nx-os/licensing/guide/b_Cisco_NX-OS_Licensing_Guide/b_Cisco_NX-OS_Licensing_Guide_chapter_01.html 

Or you can use the grace period every 120 days :-)

Stay safe,

Sergiu

[JimCh]

Hi Sergiu,

Thanks for the reply. I was under the impression the 6248 could be used in "switch mode" so I'm glad you steered me away from buying one.

Wow. Cisco's licensing options are... erm... complex, aren't they ;-)

Am I right in thinking that because the 5548UP is EOL I'd find it difficult to get hold of a FC_FEATURES_PKG license? If I can just keep renewing the Trial/Grace one I suppose it doesn't matter as I won't be using it in a production environment.

The fact it has "universal" ports is a nice feature. Do you know if the unit's fans are particularly noisy?

[balaji.bandi]

yes Nexus 5K end of Life, but they still can be used for your testing, any DC based switches or infast any Cat 9300 onwards very noisy, if you looking no sound, then there may be other desktop switch support low end FCoE 

[JimCh]

OK, I've bitten the bullet and bought a second user Nexus 5548UP on eBay, along with a RJ45 serial cable and have fired it up (man, it's noisy). I'm "in" and can play around with various commands in the CLI.

I'm a bit stuck on how to get the FC_FEATURES_PKG going. My interpretation of the documentation implies I can somehow switch this feature on for eval without a license file, but it doesn't tell me how to do this.

[balaji.bandi]

here is the document how to enable :

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/sw/nx-os/licensing/guide/b_Cisco_NX-OS_Licensing_Guide/b_Cisco_NX-OS_Licensing_Guide_chapter_01.html#con_40633

what NXOS OS you have currently - show version can show you?

you can enable by adding feature XXX to enable features

[JimCh]

Thanks.
Maybe I'm a bit slow, but I don't find their documentation all that easy to follow. In case anyone else wants to do this, here it is a step-by-step:

switch# show license usage
Feature                      Ins  Lic   Status Expiry Date Comments
                                 Count
--------------------------------------------------------------------------------
FM_SERVER_PKG                 No    -   Unused             -
ENTERPRISE_PKG                No    -   Unused             -
FC_FEATURES_PKG               No    -   Unused             -
LAN_BASE_SERVICES_PKG         No    -   Unused             -
LAN_ENTERPRISE_SERVICES_PKG   No    -   Unused             -
--------------------------------------------------------------------------------

switch# configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.

switch(config)# feature fcoe
FC license checked out successfully
fc_plugin extracted successfully
FC plugin loaded successfully
FCoE manager enabled successfully
FC enabled on all modules successfully
Warning: Ensure class-fcoe is included in qos policy-maps of all types

switch(config)# show license usage
Feature                      Ins  Lic   Status Expiry Date Comments
                                 Count
--------------------------------------------------------------------------------
FM_SERVER_PKG                 No    -   Unused             -
ENTERPRISE_PKG                No    -   Unused             -
FC_FEATURES_PKG               No    -   In use             Grace 119D 23H
LAN_BASE_SERVICES_PKG         No    -   Unused             -
LAN_ENTERPRISE_SERVICES_PKG   No    -   Unused             -
--------------------------------------------------------------------------------

Thanks for your help, guys.

I also found this blog:
https://brasstacksblog.typepad.com/brass-tacks/2011/01/fcoe-login-failure-when-connecting-to-nexus-5548.html

Which discusses the "Warning: Ensure class-fcoe is included in qos policy-maps of all types"
error message above. Seems the 5548 does not transmit any DCBX or FIP frames by default, so you need to enable QoS class-fcoe:

switch# config terminal
Enter configuration commands, one per line.  End with CNTL/Z.

switch(config)# system qos
switch(config-sys-qos)# service-policy type qos input fcoe-default-in-policy
switch(config-sys-qos)# service-policy type queuing input fcoe-default-in-policy                    
switch(config-sys-qos)# service-policy type queuing output fcoe-default-out-policy
switch(config-sys-qos)# service-policy type network-qos fcoe-default-nq-policy

and, if you want the switch to "remember" this, copy the running config to the startup config:

switch(config-sys-qos)# copy running-config startup-config
[########################################] 100%

PS: The switch is running the following:
   25136128    Apr 28 23:31:10 2011  n5000-uk9-kickstart.5.0.3.N1.1b.bin
  188700150    Apr 28 23:32:29 2011  n5000-uk9.5.0.3.N1.1b.bin
Is it worth upgrading to later versions? I'm having trouble registering my switch with Cisco, which is apparently necessary before downloading any of their upgrades :-P

[balaji.bandi]

Good stuff -- don't worry eventually you learn better.

[Sergiu.Daniluk]

Hi @JimCh 

Yep, that looks good with the grace period there ;-) When it expires, you just need a write erase and reload.

Regarding the upgrade - absolutely. 5.0 is stone age. Upgrade to 7.3(7)N1(1b) - which is the current recommended release. You have more features and less bugs vs 5.0.

Have fun and enjoy the learning!

Stay safe,

Sergiu

[JimCh]

Thanks!

I've logged a support ticket with Cisco as their product registration software is refusing to let me register it. One of their support guys opened the case after he said that he couldn't find my serial number, which is odd... there's a label on the shipping carton, and chassis which all tally with the "show inventory" response. What's also odd is that their "product family" drop box on the Add Devices dialogue doesn't include Nexus 5000 switches.

Am I right in thinking I need to be registered with them to download the latest software? Or is there some other source?

[Sergiu.Daniluk]

No you do not need to register your switch to be able to dld software. You just have to have your CCO (cisco account) completed with all details (name, company, address).

Once you have all details you will be able to download software.

Regards,

Sergiu