Nexus 5000 and MDS 9509 FC

[Michael Dougherty]

Greetings Everyone,

I have been beating my head on the desk for the last few days tring to get a FC connection established between a NX5010 and a MDS9509.  The 9509 is running the legacy storage systems but we are moving to FCoE on the NEXUS.  I am not getting link on the fiber between the 2 devices.  I have checked and verified the MM cable both SFP units have been swapped out.

I keep getting the following on the MDS when I bring up the interface.

2011 Apr 22 20:39:42 MDS9509_01 %PORT-5-IF_DOWN_INACTIVE: %$VSAN 1%$ Interface fc1/9 is down (Inactive)

fc1/9 is down (Inactive)
    Port description is Connection to Nexus 5kSW01 [FC2/1]
    Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN)
    Port WWN is xxxxxxxxxxxxxxx

    Admin port mode is auto, trunk mode is on

On the NX 5010 I have

fc2/1 is down (Link failure or not-connected)
    Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN)
    Port WWN is xxxxxxxxxxxxxxxxx

    Admin port mode is auto, trunk mode is on
    snmp link state traps are enabled
    Port vsan is 1
    Receive data field Size is 2112

I dont have anything really configured I am just tring to get a link to establish.  If someone has a sample configuration for this or a suggestion, I would certainly appreciate it.

Cheers

Mike

[Roman Rodichev]

can you paste "show vsan 1" from MDS 9509, is it suspended?

[Michael Dougherty]

Thanks for the quick reply.

This is from the MDS

vsan 1 information
         name:VSAN0001  state:suspended
         interoperability mode:default
         loadbalancing:src-id/dst-id/oxid
         operational state:down

From the NX5010

vsan 1 information
         name:VSAN0001  state:active
         interoperability mode:default
         loadbalancing:src-id/dst-id/oxid
         operational state:down

[Roman Rodichev · Accepted solution]

As you can see VSAN 1 is suspended on the 9509. Someone did that intentionally to avoid using the default VSAN. I assume you don't have any HBAs or storage in VSAN 1. Put the fc port into a different VSAN. Even if it's an TE port for trunking you need to have the port assigned into a non-suspended VSAN.

vsan database

vsan X interface fc1/9, where X is an active VSAN

[Michael Dougherty]

Well that would make sense now that I see it. So I just need to make a new VSAN entry on the MDS only and/ or do I need a matching one on the NX5k as well?

Sent from Cisco Technical Support iPhone App

[Roman Rodichev]

You can just put that port into any active VSAN. I assume you are going to run a VSAN trunk between 9500 and 5k (TE port) therefore it doesn't matter what VSAN the port is in, it's going to trunk anyway.

[Michael Dougherty]

I added the VSAN ID's to the 5k to match the 9500.  I then configured a VSAN trunk on the interfaces that connect each other.  I still show the interface being inactive.  Please see below.

MDS9500

fc1/9 is down (Inactive)

    Port description is Connection to Nexus 5kSW01 [FC2/1]

    Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN)

    Port WWN is xxxxxxxxxxx

    Admin port mode is E, trunk mode is auto

    snmp link state traps are enabled

    Port vsan is 1

    Receive data field Size is 2112

interface fc1/9

  switchport description Connection to Nexus 5kSW01 [FC2/1]

  switchport trunk mode auto

  switchport speed 4000

  switchport rate-mode dedicated

  no shutdown

  switchport mode E

NX5010

fc2/1 is down (Inactive)

    Port description is Connection to MDS9509 SW01 [FC1/9]

    Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN)

    Port WWN is xxxxxxxxxxxxxxxx

    Admin port mode is E, trunk mode is auto

    snmp link state traps are enabled

    Port vsan is 1

    Receive data field Size is 2112

interface fc2/1

  switchport speed 4000

  switchport mode E

  switchport description Connection to MDS9509 SW01 [FC1/9]

  switchport trunk mode auto

  no shutdown

[Roman Rodichev · Accepted solution]

Your MDS fc1/9 port is still in VSAN 1 though, and VSAN 1 is suspended. Move the port into a different VSAN, any active VSAN

vsan database

vsan X interface fc1/9

replace X with any active vsan

[Michael Dougherty]

Thanks for your help on this issue.  I am still trying to get familiar with these NX based switches.  I created a new VSAN on both sides and assigned my 2 interfaces to it and they came up after a minute or 2 of what I assume is a convergence time.

So even if the FC interfaces are in a trunk they still have to be assigned to an active VSAN, is that basically the bottom line?

Also I am planning to create a mesh connection between the 2 NX5k's and the 2 MDS 9500's can I just configure channels between them or will STP kick my links out or do I have to VPC configurations?

[Roman Rodichev]

Yes, the trunk interfaces must be assigned into active VSAN, even though they are trunk (TE) ports and might not even be trunking that VSAN.

There is no concept of Spanning-Tree in Fibre Channel. Don't confuse Ethernet with FC, a lot of things a different. VSANs are similar to VLANs and  they allow you to segregate traffic, and VSAN trunking also has the same goal as VLAN trunking, but that's where the similarity with Ethernet ends.

The loop prevention is achieved through FSPF (FC shortest path first) algorithm. Think of it as OSPF. Each VSAN runs its own FSPF process. Each FC switch has a unique Domain ID (value from 1 to 254) assigned by the FC Domain process in each VSAN. FSPF process is automatically on, it detects other switches and determines a best path between domain IDs based on FSPF interface costs (per VSAN). So it's actually more like IP routing than Ethernet STP. FC addresses are 3-byte FCID addresses where the first byte is the domain ID of the switch that the node (storage or HBA) is connected to. You are probably familiar with WWN's, but WWN's are not actually used in fibre channel frame headers, instead they are mapped to FCIDs.

FC/SANOS is a fun topic, consider becoming CCIE Storage expert (http://www.iementor.com/ccie-storage-workbook-walkthrough-bundle-p-338.html)

I'm not sure why it took 2 minutes for interface to come up, it should have been instanteneous, are you saying the actual interfaces didn't go Up for two minutes after changes were made?

There's no VPC concept in fibre channel. There is a regular port channel that Cisco has on MDS/NX-OS FC switches. Yes, you can configure Port channel between NX-OS on N5K and SAN-OS/NX-OS on MDS9K. It must be configured between two switches and can't span multiple switches.

[Michael Dougherty]

Well sir, you have provided alot of good information and I only say thank you for sharing it with me.  I think that I have most of my answer to keep moving forward with this project and I can share this information with my co-workers.

Cheers and hats off

-mike

[Michael Dougherty]

Greetings Roman,

I have run into another strange problem with the FC between the MDS and the NX5k.  I am not getting any FLOGI sessions in the 5k I also am not getting any FCDomain database information from.  Any suggestions?

[dynamoxxx]

if you run "sh fcdomain domain-list" from either switch ..do you see your VSANs ?

[vdsudame]

for fcoe logins, you will need to configure separately for vfc interfaces and following the configuration steps documented in

http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/fcoe/502_n1_1/b_Cisco_n5k_fcoe_config_gd_re_502_n1_1.html

please let us know if this is done. after this make sure that 'trunk protocol' is enabled on both the n5k and mds. it is enabled by default but we have seen sometimes that its disabled inappropriately. You can execute 'show trunk protocol' command on the n5k to see whether its enabled or not.

http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/san_switching/502_n2_1/Cisco_n5k_nxos_sanswitching_config_guide_rel502_n2_1_chapter6.html#con_1125649

If you need to get this working as soon, then please open a TAC Case with us and we can webex into your switches and look at the configs.

Thanks, Vinayak