problem with fcoe (brocade cna & nexus 5596)

desrochj1 · ‎02-16-2012

Hello,

We are currently troubleshooting an FCOE connection from a server to a nexus 5596.

the cna card is a brocade 1860 firmware is at 3.0.0.0 and the nexus os is at 5.1(3)N1(1)

the pwwn of the server cna card never shows up in the flogi database....

the cna interface is connected to ethernet 1/15 interface

i did a show lldp interface 1/15 and i can see that there's a peer at the other end.

Ethernet1/15 is up

Hardware: 1000/10000 Ethernet, address: 547f.ee26.fd16 (bia 547f.ee26.fd16)

MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA

Port mode is trunk

full-duplex, 10 Gb/s, media type is 10G

Beacon is turned off

Input flow-control is off, output flow-control is off

Rate mode is dedicated

Switchport monitor is off

EtherType is 0x8100

Last link flapped 00:00:06

Last clearing of "show interface" counters never

30 seconds input rate 240 bits/sec, 0 packets/sec

30 seconds output rate 0 bits/sec, 0 packets/sec

Load-Interval #2: 5 minute (300 seconds)

input rate 32 bps, 0 pps; output rate 760 bps, 1 pps

RX

0 unicast packets 2323 multicast packets 160 broadcast packets

2483 input packets 249974 bytes

0 jumbo packets 0 storm suppression bytes

0 runts 0 giants 0 CRC 0 no buffer

0 input error 0 short frame 0 overrun 0 underrun 0 ignored

0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop

0 input with dribble 0 input discard

0 Rx pause

TX

1 unicast packets 145578 multicast packets 241 broadcast packets

145820 output packets 11055457 bytes

0 jumbo packets

0 output errors 0 collision 0 deferred 0 late collision

0 lost carrier 0 no carrier 0 babble 0 output discard

0 Tx pause

17 interface resets

Ethernet1/15 is FCoE UP

vfc1 is Up

vfc1 is trunking

Bound interface is Ethernet1/15

Hardware is Ethernet

Port WWN is 20:00:54:7f:ee:26:fd:3f

Admin port mode is F, trunk mode is on

snmp link state traps are enabled

Port mode is TF

Port vsan is 1

Trunk vsans (admin allowed and active) (1)

Trunk vsans (up) ()

Trunk vsans (isolated) ()

Trunk vsans (initializing) (1)

1 minute input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec

1 minute output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec

0 frames input, 0 bytes

0 discards, 0 errors

0 frames output, 0 bytes

0 discards, 0 errors

last clearing of "show interface" counters never

Interface last changed at Thu Feb 16 15:21:40 2012

sh vsan 1

vsan 1 information

name:VSAN0001 state:active

interoperability mode:default

loadbalancing:src-id/dst-id/oxid

operational state:up

vsan 1 interfaces:

fc1/44 fc1/45 fc1/46 fc1/47

fc1/48 vfc1

Ethernet1/15 is FCoE UP
vfc1 is Up

sh vsan 1
vsan 1 information
         name:VSAN0001 state:active
         interoperability mode:default
         loadbalancing:src-id/dst-id/oxid
         operational state:up

vsan 1 interfaces:
fc1/44 fc1/45 fc1/46 fc1/47
fc1/48 vfc1

can you help me on this ?

thanks

zawoo · ‎02-16-2012

Can you do "show run interface e1/15" and also "show fcoe vlan"? If we are not allowing the native vlan and fcoe vlan on the Ethernet interface you might see a issue similar to what you are seeing now. Also make sure "spanning-tree port type edge trunk" on ethernet interface assuming you have all the latest and greatest firmware update for your cna already.

Sent from Cisco Technical Support iPhone App

erikpsmith · ‎02-17-2012

Hi, I ran into a similar problem with the Brocade 1010 and 1020 CNAs, see http://brasstacksblog.typepad.com/brass-tacks/2011/06/interoperability-issue-between-brocade-cna-and-cisco-nexus-5596.html . The resolution was to upgrade to CNA driver 3.0.3.0 . This may help since you are running 3.0.0.0.

Erik

Kris Vandecruys · ‎02-17-2012

Hi all,

I can shed some light here. It's very likely that you're indeed running into the issue Erik described. What's happening is that the CNA might be using the old Gen1 protocol, CIN. Nexus5500 series no longer support this, in favor of the more modern and standard CEE.

You can easily check this. Run this command on the affected interface:

# show system internal dcbx info interface Ethernet1/15

You'll see an output similar to this:

Interface info for if_index: 0x1a002000(Eth1/15)

tx_enabled: TRUE

rx_enabled: TRUE

dcbx_enabled: TRUE

DCX Protocol: CIN

That last line should read 'DCX Protocol: CEE' for it to work on a 5500 series.

Hope this helps!

/Kris

zawoo · ‎02-21-2012

Also make sure you have the following FCoE QoS policies explicitly define:

system qos

service-policy type qos input fcoe-default-in-policy

service-policy type queuing input fcoe-default-in-policy

service-policy type queuing output fcoe-default-out-policy

service-policy type network-qos fcoe-default-nq-policy

Dennis Goh · ‎03-05-2012

Hi,

I've also happened to encounter this issue. Any reason why system qos service policies can affect the protocols used? If I do not explicitly define these policies, I can see that the DCX protocol used is still CIN. Once these lines are implemented, you perform a "show system internal dcbx info int eX/X" DCX protocol you will see CEE.

desrochj1 · ‎02-17-2012

Hello Kris,

the cna card seems to run CEE protocol

sh system internal dcbx info interface ethernet 1/15

Interface info for if_index: 0x1a00e000(Eth1/15)
tx_enabled: TRUE
rx_enabled: TRUE
dcbx_enabled: TRUE
DCX Protocol: CEE
DCX CEE NIV extension: disabled

here's the show run for the server interface

interface Ethernet1/15

switchport mode trunk

spanning-tree port type edge

here's the show vlan fcoe

sh vlan fcoe

Original VLAN ID Translated VSAN ID Association State
---------------- ------------------ -----------------

889 1 Operational

lamav · ‎02-19-2012

Have you bound the VSAN to the FCoE VLAN?

example:

vlan 1000

fcoe vsan 2

name FCoE_VLAN

desrochj1 · ‎03-05-2012

problem was with cna