cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
3239
Views
0
Helpful
16
Replies

FCoE Northbound Connectivity ISSUES

marco.mosangini
Level 1
Level 1

Hi all,

i've configured this topology for Ethernet and FCoE uplink from 6248UP to 5548UP:

Multi-Hop FCoE

When i check flogi database from n5k the result is:

  • n5kA --> no vfc to 6248A was logged
  • n5kB --> vfc to 6248B is up and running

This is sh int vfc21 (n5kA FCoE Po to 6248A):

vfc21 is down (Initializing)

    Bound interface is port-channel21

    Hardware is Ethernet

    Port WWN is 20:14:54:7f:ee:b2:fa:7f

    Admin port mode is F, trunk mode is on

    snmp link state traps are enabled

    Port vsan is 101

    1 minute input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec

    1 minute output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec

      0 frames input, 0 bytes

        0 discards, 0 errors

      0 frames output, 0 bytes

        0 discards, 0 errors

    last clearing of "show interface" counters never

This is sh int vfc22 (n5kB FCoE Po to 6248A):

vfc22 is trunking

    Bound interface is port-channel22

    Hardware is Ethernet

    Port WWN is 20:15:54:7f:ee:af:49:bf

    Admin port mode is F, trunk mode is on

    snmp link state traps are enabled

    Port mode is TF

    Port vsan is 102

    Trunk vsans (admin allowed and active) (102)

    Trunk vsans (up)                       (102)

    Trunk vsans (isolated)                 ()

    Trunk vsans (initializing)             ()

    1 minute input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec

    1 minute output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec

      740 frames input, 73824 bytes

        0 discards, 648 errors

      759 frames output, 132936 bytes

        0 discards, 0 errors

    last clearing of "show interface" counters never

    Interface last changed at Wed Apr  3 09:32:57 2013

This is sh ru int vfc755 (6248A FCoE Po to n5kA):

vfc755 is trunking (Not all VSANs UP on the trunk)

    Bound interface is port-channel21

    Hardware is Virtual Fibre Channel

    Port WWN is 22:f2:54:7f:ee:aa:cf:3f

    Admin port mode is NP, trunk mode is on

    snmp link state traps are enabled

    Port mode is TNP

    Port vsan is 101

    Trunk vsans (admin allowed and active) (1,101)

    Trunk vsans (up)                       ()

    Trunk vsans (isolated)                 ()

    Trunk vsans (initializing)             (1,101)

    1 minute input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec

    1 minute output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec

      0 frames input, 0 bytes

        0 discards, 0 errors

      0 frames output, 0 bytes

        0 discards, 0 errors

    last clearing of "show interface" counters never

    Interface last changed at Wed Apr  3 19:53:21 2013

This is sh ru int vfc756 (6248B FCoE Po to n5kB):

vfc756 is trunking (Not all VSANs UP on the trunk)

    Bound interface is port-channel22

    Hardware is Virtual Fibre Channel

    Port WWN is 22:f3:54:7f:ee:aa:d2:3f

    Admin port mode is NP, trunk mode is on

    snmp link state traps are enabled

    Port mode is TNP

    Port vsan is 102

    Trunk vsans (admin allowed and active) (1,102)

    Trunk vsans (up)                       (102)

    Trunk vsans (isolated)                 ()

    Trunk vsans (initializing)             (1)

    1 minute input rate 8 bits/sec, 1 bytes/sec, 0 frames/sec

    1 minute output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec

      1187 frames input, 163868 bytes

        0 discards, 0 errors

      1930 frames output, 202972 bytes

        0 discards, 0 errors

    last clearing of "show interface" counters never

    Interface last changed at Wed Apr  3 07:22:45 2013

Could someone help me solving this painful issue??

Thanks

16 Replies 16

Jeremy Waldrop
Level 4
Level 4

What are vfc 755 and 756? Looks like there are 2 vfc bond to the same port channel? Send the output of these commands

show run | section "system qos"

show vlan fcoe

show run int vfc21

show run int vfc755

show run int po21

show int po21 fcoe

show fcoe database

  • vfc755 is vfc 6248UPA side (connected to vfc21 n5kA)
  • vfc756 is vfc 6248UPB side (connected to vfc22 n5kB)

The commands are the output from 5548UP-A

show run | section "system qos"

system qos

  service-policy type qos input fcoe-default-in-policy

  service-policy type queuing input fcoe-default-in-policy

  service-policy type queuing output fcoe-default-out-policy

  service-policy type network-qos jumbo

show vlan fcoe

Original VLAN ID        Translated VSAN ID      Association State

----------------        ------------------      -----------------

      101                       101              Operational

      4048                      1                Operational

show running-config interface vfc21

interface vfc21

  bind interface port-channel21

  switchport trunk allowed vsan 101

  no shutdown

show run int po21

!Command: show running-config interface port-channel21

!Time: Wed Apr  3 23:00:38 2013

version 6.0(2)N1(2)

interface port-channel21

  description UCS_FabIn_A_FCoE

  switchport mode trunk

  switchport trunk allowed vlan 101

  spanning-tree port type edge trunk

  spanning-tree bpdufilter enable

show int po21 fcoe

port-channel21 is FCoE UP

    vfc21 is Up

show fcoe database

-------------------------------------------------------------------------------

INTERFACE       FCID            PORT NAME               MAC ADDRESS

-------------------------------------------------------------------------------

swp01201#

Thanks in advance

M.

This is the connection topology:

n5kA-->po21-->vfc21-----------vfc755--->po21---6248A (not working correcly)

n5kB-->po22-->vfc22-----------vfc756--->po22---6248B (working correctly)

M.


I assume the FCoE VLAN on UCS Fabric A is 101?

Is the port-channel LACP?

send these command outputs from n5kA

show port-c summ

show int po21 trunk

show vsan 101

these from nxos on UCS Fabric A

show vlan fcoe

show port-c summ

show int po21 trunk

True.

######################################

n5kA# show port-c summ

Flags:  D - Down        P - Up in port-channel (members)

        I - Individual  H - Hot-standby (LACP only)

        s - Suspended   r - Module-removed

        S - Switched    R - Routed

        U - Up (port-channel)

        M - Not in use. Min-links not met

--------------------------------------------------------------------------------

Group Port-       Type     Protocol  Member Ports

      Channel

--------------------------------------------------------------------------------

1     Po1(SU)     Eth      LACP      Eth1/21(P)   Eth1/22(P)   Eth1/23(D)

                                     Eth1/24(D)

10    Po10(SU)    Eth      LACP      Eth1/5(P)    Eth1/6(P)

11    Po11(SU)    Eth      LACP      Eth1/1(P)

12    Po12(SU)    Eth      LACP      Eth1/2(P)

13    Po13(SU)    Eth      LACP      Eth1/3(P)

14    Po14(SU)    Eth      LACP      Eth1/4(P)

15    Po15(SU)    Eth      LACP      Eth1/17(P)   Eth1/18(P)

16    Po16(SU)    Eth      LACP      Eth1/15(P)

21    Po21(SU)    Eth      LACP      Eth2/11(P)   Eth2/12(P)

n5kA# show int po21 trunk

--------------------------------------------------------------------------------

Port          Native  Status        Port

              Vlan                  Channel

--------------------------------------------------------------------------------

Po21          1       trunking      --

--------------------------------------------------------------------------------

Port          Vlans Allowed on Trunk

--------------------------------------------------------------------------------

Po21          101

--------------------------------------------------------------------------------

Port          Vlans Err-disabled on Trunk

--------------------------------------------------------------------------------

Po21          none

--------------------------------------------------------------------------------

Port          STP Forwarding

--------------------------------------------------------------------------------

Po21          none

--------------------------------------------------------------------------------

Port          Vlans in spanning tree forwarding state and not pruned

--------------------------------------------------------------------------------

--------------------------------------------------------------------------------

Port          Vlans Forwarding on FabricPath

--------------------------------------------------------------------------------

Po21          none

n5kA# show vsan 101

vsan 101 information

         name:FABRIC_A  state:active

         interoperability mode:default

         loadbalancing:src-id/dst-id/oxid

         operational state:up

######################################

6248A(nxos)# show vlan fcoe

Original VLAN ID        Translated VSAN ID      Association State

----------------        ------------------      -----------------

      4048                      1                Operational

      101                       101              Operational

6248A(nxos)# show port-c summ

Flags:  D - Down        P - Up in port-channel (members)

        I - Individual  H - Hot-standby (LACP only)

        s - Suspended   r - Module-removed

        S - Switched    R - Routed

        U - Up (port-channel)

--------------------------------------------------------------------------------

Group Port-       Type     Protocol  Member Ports

      Channel

--------------------------------------------------------------------------------

13    Po13(SU)    Eth      LACP      Eth1/19(P)   Eth1/20(P)

21    Po21(SU)    Eth      LACP      Eth1/27(P)   Eth1/28(P)

1025  Po1025(SU)  Eth      NONE      Eth1/1(P)    Eth1/2(P)    Eth1/3(P)

                                     Eth1/4(P)

1280  Po1280(SU)  Eth      NONE      Eth1/1/5(P)  Eth1/1/7(P)

1282  Po1282(SU)  Eth      NONE      Eth1/1/1(P)  Eth1/1/3(P)

1284  Po1284(SD)  Eth      NONE      Eth1/1/9(D)  Eth1/1/11(D)

1286  Po1286(SD)  Eth      NONE      Eth1/1/13(D)  Eth1/1/15(D)

6248A(nxos)# show int po21 trunk

--------------------------------------------------------------------------------

Port          Native  Status        Port

              Vlan                  Channel

--------------------------------------------------------------------------------

Po21          4049    trunking      --

--------------------------------------------------------------------------------

Port          Vlans Allowed on Trunk

--------------------------------------------------------------------------------

Po21          101,4048-4049

--------------------------------------------------------------------------------

Port          Vlans Err-disabled on Trunk

--------------------------------------------------------------------------------

Po21          none

--------------------------------------------------------------------------------

Port          STP Forwarding

--------------------------------------------------------------------------------

Po21          4049

--------------------------------------------------------------------------------

Port          Vlans in spanning tree forwarding state and not pruned

--------------------------------------------------------------------------------

Po21          --

######################################

at this point I am not sure what the issue is, everything looks correct from what I can tell. Usually when vfc is stuck in the initializing state the system FCoE QoS policies aren't applied but yours look ok.

Another command you can run to check the qos is "show run ipqos" the output should look like this

lab-n5k-a# show run ipqos

!Command: show running-config ipqos

!Time: Thu Apr  4 12:33:37 2013

version 6.0(2)N1(1)

class-map type qos class-fcoe

class-map type qos match-all test

class-map type qos match-all test-vlan

class-map type qos match-all match-all-traffic

  match cos 0-7

class-map type queuing class-fcoe

  match qos-group 1

class-map type queuing class-all-flood

  match qos-group 2

class-map type queuing class-ip-multicast

  match qos-group 2

class-map type network-qos class-fcoe

  match qos-group 1

class-map type network-qos class-all-flood

  match qos-group 2

class-map type network-qos class-ip-multicast

  match qos-group 2

system qos

  service-policy type queuing input fcoe-default-in-policy

  service-policy type queuing output fcoe-default-out-policy

  service-policy type qos input fcoe-default-in-policy

  service-policy type network-qos fcoe-default-nq-policy

Configuration is exactly the same. In N5KB works correctly but in N5KA does not.

I've done another test: i've moved FCoE northbound to FC port-channel but issue was the same.

Marco

Ok, so it isn't an FCoE issue. Sounds like a VSAN/FC issue, do you have other devices or SAN connected to the Nexus over FC that are working? Anything in the flogi database for VSAN 101?

I've an FC tape drive connected to N5KA correctly inside flogi database. If I connect this trive to N5KA (VSAN 101 Fabric A) i couldn't see it from any blade. If i connect the drive to N5KB (VSAN 102 Fabric B) it's all ok. Comparing confs from nexus A and B the conf is exaclty the same (except VSAN 101 ID and FCOE VLAN ID).

In every blade that has a path to fabric A i saw this fault:

Affected object: sys/chassis-1/fabric-A/path-1/vc-815

Cause: link-down

Description: fc VIF 1 / 1 A-815 down, reason: NPV upstream port not available

M.

that typically means the NPIV feature isn't enabled. Did you check that?

N5KVA:

version 6.0(2)N1(2)

feature fcoe

hostname swp01201

feature npiv

feature fport-channel-trunk

feature telnet

cfs eth distribute

feature interface-vlan

feature hsrp

feature lacp

feature vpc

feature lldp

logging level flogi 5

N5KB:

version 6.0(2)N1(2)

feature fcoe

hostname swp01202

feature npiv

feature fport-channel-trunk

feature telnet

cfs eth distribute

feature interface-vlan

feature hsrp

feature lacp

feature vpc

feature lldp

logging level flogi 5

I don't know then, I think it is time to open a TAC case

Can you paste the output of

FI-A (nxos)

show run int e2/11-12

show int e2/11-12

show vsan database

FI-B (nxos)

show run int e2/11-12

show int e2/11-12

show vsan database

Thanks,

Robert

Also attach a "show run" from both N5Ks.  This is a config problem somewhere...

Robert

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: