FCoE Northbound Connectivity ISSUES

marco.mosangini · ‎04-03-2013

Hi all,

i've configured this topology for Ethernet and FCoE uplink from 6248UP to 5548UP:

When i check flogi database from n5k the result is:

n5kA --> no vfc to 6248A was logged
n5kB --> vfc to 6248B is up and running

This is sh int vfc21 (n5kA FCoE Po to 6248A):

vfc21 is down (Initializing)

Bound interface is port-channel21

Hardware is Ethernet

Port WWN is 20:14:54:7f:ee:b2:fa:7f

Admin port mode is F, trunk mode is on

snmp link state traps are enabled

Port vsan is 101

1 minute input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec

1 minute output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec

0 frames input, 0 bytes

0 discards, 0 errors

0 frames output, 0 bytes

0 discards, 0 errors

last clearing of "show interface" counters never

This is sh int vfc22 (n5kB FCoE Po to 6248A):

vfc22 is trunking

Bound interface is port-channel22

Hardware is Ethernet

Port WWN is 20:15:54:7f:ee:af:49:bf

Admin port mode is F, trunk mode is on

snmp link state traps are enabled

Port mode is TF

Port vsan is 102

Trunk vsans (admin allowed and active) (102)

Trunk vsans (up) (102)

Trunk vsans (isolated) ()

Trunk vsans (initializing) ()

1 minute input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec

1 minute output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec

740 frames input, 73824 bytes

0 discards, 648 errors

759 frames output, 132936 bytes

0 discards, 0 errors

last clearing of "show interface" counters never

Interface last changed at Wed Apr 3 09:32:57 2013

This is sh ru int vfc755 (6248A FCoE Po to n5kA):

vfc755 is trunking (Not all VSANs UP on the trunk)

Bound interface is port-channel21

Hardware is Virtual Fibre Channel

Port WWN is 22:f2:54:7f:ee:aa:cf:3f

Admin port mode is NP, trunk mode is on

snmp link state traps are enabled

Port mode is TNP

Port vsan is 101

Trunk vsans (admin allowed and active) (1,101)

Trunk vsans (up) ()

Trunk vsans (isolated) ()

Trunk vsans (initializing) (1,101)

1 minute input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec

1 minute output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec

0 frames input, 0 bytes

0 discards, 0 errors

0 frames output, 0 bytes

0 discards, 0 errors

last clearing of "show interface" counters never

Interface last changed at Wed Apr 3 19:53:21 2013

This is sh ru int vfc756 (6248B FCoE Po to n5kB):

vfc756 is trunking (Not all VSANs UP on the trunk)

Bound interface is port-channel22

Hardware is Virtual Fibre Channel

Port WWN is 22:f3:54:7f:ee:aa:d2:3f

Admin port mode is NP, trunk mode is on

snmp link state traps are enabled

Port mode is TNP

Port vsan is 102

Trunk vsans (admin allowed and active) (1,102)

Trunk vsans (up) (102)

Trunk vsans (isolated) ()

Trunk vsans (initializing) (1)

1 minute input rate 8 bits/sec, 1 bytes/sec, 0 frames/sec

1 minute output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec

1187 frames input, 163868 bytes

0 discards, 0 errors

1930 frames output, 202972 bytes

0 discards, 0 errors

last clearing of "show interface" counters never

Interface last changed at Wed Apr 3 07:22:45 2013

Could someone help me solving this painful issue??

Thanks

Jeremy Waldrop · ‎04-03-2013

What are vfc 755 and 756? Looks like there are 2 vfc bond to the same port channel? Send the output of these commands

show run | section "system qos"

show vlan fcoe

show run int vfc21

show run int vfc755

show run int po21

show int po21 fcoe

show fcoe database

marco.mosangini · ‎04-03-2013

vfc755 is vfc 6248UPA side (connected to vfc21 n5kA)
vfc756 is vfc 6248UPB side (connected to vfc22 n5kB)

The commands are the output from 5548UP-A

show run | section "system qos"

system qos

service-policy type qos input fcoe-default-in-policy

service-policy type queuing input fcoe-default-in-policy

service-policy type queuing output fcoe-default-out-policy

service-policy type network-qos jumbo

show vlan fcoe

Original VLAN ID Translated VSAN ID Association State

---------------- ------------------ -----------------

101 101 Operational

4048 1 Operational

show running-config interface vfc21

interface vfc21

bind interface port-channel21

switchport trunk allowed vsan 101

no shutdown

show run int po21

!Command: show running-config interface port-channel21

!Time: Wed Apr 3 23:00:38 2013

version 6.0(2)N1(2)

interface port-channel21

description UCS_FabIn_A_FCoE

switchport mode trunk

switchport trunk allowed vlan 101

spanning-tree port type edge trunk

spanning-tree bpdufilter enable

show int po21 fcoe

port-channel21 is FCoE UP

vfc21 is Up

show fcoe database

-------------------------------------------------------------------------------

INTERFACE FCID PORT NAME MAC ADDRESS

-------------------------------------------------------------------------------

swp01201#

Thanks in advance

M.

marco.mosangini · ‎04-03-2013

This is the connection topology:

n5kA-->po21-->vfc21-----------vfc755--->po21---6248A (not working correcly)

n5kB-->po22-->vfc22-----------vfc756--->po22---6248B (working correctly)

M.

Jeremy Waldrop · ‎04-03-2013

I assume the FCoE VLAN on UCS Fabric A is 101?

Is the port-channel LACP?

send these command outputs from n5kA

show port-c summ

show int po21 trunk

show vsan 101

these from nxos on UCS Fabric A

show vlan fcoe

show port-c summ

show int po21 trunk

marco.mosangini · ‎04-03-2013

True.

######################################

n5kA# show port-c summ

Flags: D - Down P - Up in port-channel (members)

I - Individual H - Hot-standby (LACP only)

s - Suspended r - Module-removed

S - Switched R - Routed

U - Up (port-channel)

M - Not in use. Min-links not met

--------------------------------------------------------------------------------

Group Port- Type Protocol Member Ports

Channel

--------------------------------------------------------------------------------

1 Po1(SU) Eth LACP Eth1/21(P) Eth1/22(P) Eth1/23(D)

Eth1/24(D)

10 Po10(SU) Eth LACP Eth1/5(P) Eth1/6(P)

11 Po11(SU) Eth LACP Eth1/1(P)

12 Po12(SU) Eth LACP Eth1/2(P)

13 Po13(SU) Eth LACP Eth1/3(P)

14 Po14(SU) Eth LACP Eth1/4(P)

15 Po15(SU) Eth LACP Eth1/17(P) Eth1/18(P)

16 Po16(SU) Eth LACP Eth1/15(P)

21 Po21(SU) Eth LACP Eth2/11(P) Eth2/12(P)

n5kA# show int po21 trunk

--------------------------------------------------------------------------------

Port Native Status Port

Vlan Channel

--------------------------------------------------------------------------------

Po21 1 trunking --

--------------------------------------------------------------------------------

Port Vlans Allowed on Trunk

--------------------------------------------------------------------------------

Po21 101

--------------------------------------------------------------------------------

Port Vlans Err-disabled on Trunk

--------------------------------------------------------------------------------

Po21 none

--------------------------------------------------------------------------------

Port STP Forwarding

--------------------------------------------------------------------------------

Po21 none

--------------------------------------------------------------------------------

Port Vlans in spanning tree forwarding state and not pruned

--------------------------------------------------------------------------------

Port Vlans Forwarding on FabricPath

--------------------------------------------------------------------------------

Po21 none

n5kA# show vsan 101

vsan 101 information

name:FABRIC_A state:active

interoperability mode:default

loadbalancing:src-id/dst-id/oxid

operational state:up

######################################

6248A(nxos)# show vlan fcoe

Original VLAN ID Translated VSAN ID Association State

---------------- ------------------ -----------------

4048 1 Operational

101 101 Operational

6248A(nxos)# show port-c summ

Flags: D - Down P - Up in port-channel (members)

I - Individual H - Hot-standby (LACP only)

s - Suspended r - Module-removed

S - Switched R - Routed

U - Up (port-channel)

--------------------------------------------------------------------------------

Group Port- Type Protocol Member Ports

Channel

--------------------------------------------------------------------------------

13 Po13(SU) Eth LACP Eth1/19(P) Eth1/20(P)

21 Po21(SU) Eth LACP Eth1/27(P) Eth1/28(P)

1025 Po1025(SU) Eth NONE Eth1/1(P) Eth1/2(P) Eth1/3(P)

Eth1/4(P)

1280 Po1280(SU) Eth NONE Eth1/1/5(P) Eth1/1/7(P)

1282 Po1282(SU) Eth NONE Eth1/1/1(P) Eth1/1/3(P)

1284 Po1284(SD) Eth NONE Eth1/1/9(D) Eth1/1/11(D)

1286 Po1286(SD) Eth NONE Eth1/1/13(D) Eth1/1/15(D)

6248A(nxos)# show int po21 trunk

--------------------------------------------------------------------------------

Port Native Status Port

Vlan Channel

--------------------------------------------------------------------------------

Po21 4049 trunking --

--------------------------------------------------------------------------------

Port Vlans Allowed on Trunk

--------------------------------------------------------------------------------

Po21 101,4048-4049

--------------------------------------------------------------------------------

Port Vlans Err-disabled on Trunk

--------------------------------------------------------------------------------

Po21 none

--------------------------------------------------------------------------------

Port STP Forwarding

--------------------------------------------------------------------------------

Po21 4049

--------------------------------------------------------------------------------

Port Vlans in spanning tree forwarding state and not pruned

--------------------------------------------------------------------------------

Po21 --

######################################

Jeremy Waldrop · ‎04-04-2013

at this point I am not sure what the issue is, everything looks correct from what I can tell. Usually when vfc is stuck in the initializing state the system FCoE QoS policies aren't applied but yours look ok.

Another command you can run to check the qos is "show run ipqos" the output should look like this

lab-n5k-a# show run ipqos

!Command: show running-config ipqos

!Time: Thu Apr 4 12:33:37 2013

version 6.0(2)N1(1)

class-map type qos class-fcoe

class-map type qos match-all test

class-map type qos match-all test-vlan

class-map type qos match-all match-all-traffic

match cos 0-7

class-map type queuing class-fcoe

match qos-group 1

class-map type queuing class-all-flood

match qos-group 2

class-map type queuing class-ip-multicast

match qos-group 2

class-map type network-qos class-fcoe

match qos-group 1

class-map type network-qos class-all-flood

match qos-group 2

class-map type network-qos class-ip-multicast

match qos-group 2

system qos

service-policy type queuing input fcoe-default-in-policy

service-policy type queuing output fcoe-default-out-policy

service-policy type qos input fcoe-default-in-policy

service-policy type network-qos fcoe-default-nq-policy

marco.mosangini · ‎04-04-2013

Configuration is exactly the same. In N5KB works correctly but in N5KA does not.

I've done another test: i've moved FCoE northbound to FC port-channel but issue was the same.

Marco

Jeremy Waldrop · ‎04-04-2013

Ok, so it isn't an FCoE issue. Sounds like a VSAN/FC issue, do you have other devices or SAN connected to the Nexus over FC that are working? Anything in the flogi database for VSAN 101?

marco.mosangini · ‎04-04-2013

I've an FC tape drive connected to N5KA correctly inside flogi database. If I connect this trive to N5KA (VSAN 101 Fabric A) i couldn't see it from any blade. If i connect the drive to N5KB (VSAN 102 Fabric B) it's all ok. Comparing confs from nexus A and B the conf is exaclty the same (except VSAN 101 ID and FCOE VLAN ID).

In every blade that has a path to fabric A i saw this fault:

Affected object: sys/chassis-1/fabric-A/path-1/vc-815

Cause: link-down

Description: fc VIF 1 / 1 A-815 down, reason: NPV upstream port not available

M.

Jeremy Waldrop · ‎04-04-2013

that typically means the NPIV feature isn't enabled. Did you check that?

marco.mosangini · ‎04-04-2013

N5KVA:

version 6.0(2)N1(2)

feature fcoe

hostname swp01201

feature npiv

feature fport-channel-trunk

feature telnet

cfs eth distribute

feature interface-vlan

feature hsrp

feature lacp

feature vpc

feature lldp

logging level flogi 5

N5KB:

version 6.0(2)N1(2)

feature fcoe

hostname swp01202

feature npiv

feature fport-channel-trunk

feature telnet

cfs eth distribute

feature interface-vlan

feature hsrp

feature lacp

feature vpc

feature lldp

logging level flogi 5

Jeremy Waldrop · ‎04-05-2013

I don't know then, I think it is time to open a TAC case

Robert Burns · ‎04-05-2013

Can you paste the output of

FI-A (nxos)

show run int e2/11-12

show int e2/11-12

show vsan database

FI-B (nxos)

show run int e2/11-12

show int e2/11-12

show vsan database

Thanks,

Robert

Robert Burns · ‎04-05-2013

Also attach a "show run" from both N5Ks. This is a config problem somewhere...

Robert