npv to npiv Trunking

[ataranen]

I find that only 1 VSAN can be in the UP state at any given time on a trunking link between NPV and NPIV switches. Here is my simplified scenario; Nexus 5k in NPV mode connected to MDS 9222i in NPIV mode.

NPIV Side: MDS 9222i with NX-OS 5.2(2d)

feature npiv

feature fport-channel-trunk

vsan database

  vsan 10

  vsan 20

interface fc1/1

  switchport mode F

  switchport trunk allowed vsan 10

  switchport trunk allowed vsan add 20

  no shutdown

NPV Side: Nexus 5548UP with NX-OS 5.1(3)N2(1c)

feature fcoe

feature npv

vsan database

  vsan 10

  vsan 20

interface fc1/31

  switchport mode NP

  switchport trunk allowed vsan 10

  switchport trunk allowed vsan add 20

  switchport trunk mode on

  no shutdown

mds9222i# show interface fc1/1

fc1/1 is trunking

    Hardware is Fibre Channel, SFP is unknown

    Port WWN is 20:01:00:0d:ec:dd:5b:40

    Admin port mode is F, trunk mode is on

    snmp link state traps are enabled

    Port mode is TF

    Port vsan is 1

    Speed is 4 Gbps

    Rate mode is shared

    Transmit B2B Credit is 16

    Receive B2B Credit is 16

    Receive data field Size is 2112

    Beacon is turned off

    Trunk vsans (admin allowed and active) (10,20)

    Trunk vsans (up)                       (10)

    Trunk vsans (isolated)                 ()

    Trunk vsans (initializing)             (20)

    5 minutes input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec

    5 minutes output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec

mds9222i# show flogi database

--------------------------------------------------------------------------------

INTERFACE        VSAN    FCID           PORT NAME               NODE NAME

--------------------------------------------------------------------------------

fc1/1            10    0x050000  20:1f:54:7f:ee:c9:c1:40 20:0a:54:7f:ee:c9:c1:41

5k1# show interface fc1/31

fc1/31 is trunking

    Hardware is Fibre Channel, SFP is unknown

    Port WWN is 20:1f:54:7f:ee:c9:c1:40

    Admin port mode is NP, trunk mode is on

    snmp link state traps are enabled

    Port mode is TNP

    Port vsan is 1

    Speed is 4 Gbps

    Transmit B2B Credit is 16

    Receive B2B Credit is 16

    Receive data field Size is 2112

    Beacon is turned off

   Trunk vsans (admin allowed and active) (10,20)

    Trunk vsans (up)                       (10)

    Trunk vsans (isolated)                 ()

    Trunk vsans (initializing)             (20)

    1 minute input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec

    1 minute output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec

I can make VSAN 20 come up on the trunk if I disable VSAN 10, but I cannot make both of them to work on the trunk. Any ideas?

[dynamoxxx]

i have tested UCS FI (host-end mode, same as NPV) and MDS 9513 (same features enabled as on your 9222i) ..and i was able to trunk VSAN between them. Could be something 5k specific.

@dynamoxxx

[ataranen]

I have tested this configuration between UCS FI 5548UP and MDS922i - same problem. Trunking only works for a single VSAN.

DC1-A(nxos)# show int san-port-channel 100

san-port-channel 100 is trunking

    Hardware is Fibre Channel

    Port WWN is 24:64:54:7f:ee:b4:23:c0

    Admin port mode is NP, trunk mode is on

    snmp link state traps are enabled

    Port mode is TNP

    Port vsan is 100

    Speed is 16 Gbps

    Trunk vsans (admin allowed and active) (1,100)

    Trunk vsans (up)                       (100)

    Trunk vsans (isolated)                 ()

    Trunk vsans (initializing)             (1)

Anyone using 9222i with NXOS 5.2 who got it working?

[dynamoxxx]

i had it working between 9513 and UCS FI. Did you enable f-port-trunking, did you also create another VSAN in UCS ?

@dynamoxxx

[richbarb]

The VSAN 1 is suspend or pruned in the trunk.

All vsans configured in a trunk bring up from VSAN 1 (default), if the VSAN 1 is suspended or pruned, the trunk will not work.

To resolve this, configure all trunks interfaces in a different vsan, under vsan database, this vsan must be allowed in the trunk.

Restart all fc interfaces/san-port-channels.

[matao]

I have tested with Nexus 5K NPV to Nexus 7K, and it ONLY 1 vsan can be brought up with  at a given time.

on the Nexus 7K, I have enabled npiv and fport-trunking

It is not possible to configure an fc/vfc interface to different vsan under vsan database. an fc/vfc can only belong to one VSAN.

from the chart below, it suggests that we can NPV several VSANs into an NPIV switch at the same time, but in reality the config does not work.

http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/san_switching/503_n2_1/Cisco_n5k_nxos_sanswitching_config_guide_rel503_n2_1_chapter6.html

Not sure why it does not work, if we can only bring up  1 VSAN on FCoE NPV, why should we call it a "trunk" port?

[ataranen]

Thanks for looking into it. I just take it as a given: we can't trunk on 5k NP port. I ran multiple tests with different NX version and get the same result every time - only 1 vsan is up.

[praytick · Accepted solution]

Hello,

You can trunk multiple VSANs over a TNP-TF link for both FC and FCoE. I suspect what you are seeing, only 1 VSAN up, is because the VSAN that is up is the one the NPV switch used to login with. (Verified by looking at the OUI of the PWWN in your show flogi database output) The other VSANs that are trunked on the link will stay down until a downstream server off the NPV switch that is placed in that VSAN tries to FLOGI in. At that time the VSAN will come up.