Solved: Nexus 4001i no fip MACs showing

ArrowSIVAC · ‎12-05-2011

Environment:

IBM BCH blade in bay 12 with QLogic 10Gb Convergence adapter.

CISCO Nexus 5010 (sw10) ports 3, 4 trunked via port group to BCH Chassis bay 7 ports 15, 16 port group.

First time setting this up. Following the examples guide for the 4001i and 5010 as well as the read the admin guide.

Issue: It seems I have everything setup. I simply need to see the WWPN show up in the fabric so I can map it through to the Fibre Storage devices, but when I run "show fip-snooping vlan-discovery" no MACs show. I manually went into the blade managment module, recorded the MAC and built the vrf table entry, but not working, so I beleive something is missconfigured. Need some guidance... clue I beleive is that no MAC shows up in the FIP query.

Current Configuration (I tried to cut it down to just things related to pertinent data)

Nexus 5010

Nexus 4001i

feature fcoe

feature npiv

feature telnet

feature lacp

feature lldp

feature fex

class-map type qos class-fcoe

class-map type queuing class-all-flood

match qos-group 2

class-map type queuing class-ip-multicast

match qos-group 2

class-map type network-qos class-all-flood

match qos-group 2

class-map type network-qos class-ip-multicast

match qos-group 2

vlan 11

name serverdata

vlan 28

fcoe vsan 28

name vsan

vsan database

vsan 28 name "vsan"

fcdomain domain 29 static vsan 28

interface port-channel11

description sw10-03_04 to sw14-15_16

switchport mode trunk

switchport trunk native vlan 11

speed 10000

interface vfc112

bind mac-address 00:c0:dd:17:8e:10

switchport description bch112

no shutdown

vsan database

vsan 28 interface vfc9

vsan 28 interface vfc10

vsan 28 interface vfc28

vsan 28 interface vfc112

vsan 28 interface fc2/1

vsan 28 interface fc2/2

vsan 28 interface fc2/3

vsan 28 interface fc2/4

vsan 28 interface fc2/5

vsan 28 interface fc2/6

vsan 28 interface fc2/7

vsan 28 interface fc2/8

interface fc2/1

switchport description fc01-09-12 (E Mode Port)

no shutdown

interface Ethernet1/3

description sw14-15

switchport mode trunk

switchport trunk native vlan 11

channel-group 11

interface Ethernet1/4

description sw14-16

switchport mode trunk

switchport trunk native vlan 11

channel-group 11

boot kickstart bootflash:/n5000-uk9-kickstart.5.0.2.N2.1.bin

boot system bootflash:/n5000-uk9.5.0.2.N2.1.bin

feature telnet

feature lacp

feature fip-snooping

policy-map type queuing policy-fcoe-bandwidth

class type queuing 1p7q4t-out-q-default

bandwidth percent 50

class type queuing 1p7q4t-out-pq1

bandwidth percent 7

class type queuing 1p7q4t-out-q2

bandwidth percent 7

class type queuing 1p7q4t-out-q3

bandwidth percent 7

class type queuing 1p7q4t-out-q4

bandwidth percent 7

class type queuing 1p7q4t-out-q5

bandwidth percent 7

class type queuing 1p7q4t-out-q6

bandwidth percent 7

class type queuing 1p7q4t-out-q7

bandwidth percent 7

class-map type network-qos class-fcoe

match cos 3

class-map type network-qos class-non-fcoe

match cos 0-2,4-7

policy-map type network-qos policy-fcoe

class type network-qos class-fcoe

pause no-drop

mtu 2500

class type network-qos class-non-fcoe

system qos

service-policy type network-qos policy-fcoe

service-policy type queuing output policy-fcoe-bandwidth

vlan 11

name serverdata

vlan 28

fip-snooping enable

fip-snooping fc-map 0x0efc00

name vsan

port-channel load-balance ethernet source-dest-mac

interface port-channel1

description sw10-03_04 to sw14-15_16

switchport mode trunk

switchport trunk native vlan 11

spanning-tree port type edge trunk

speed 10000

interface Ethernet1/12

description bch112

switchport mode trunk

switchport trunk native vlan 11

fip-snooping port-mode fcf

spanning-tree port type edge

speed 10000

interface Ethernet1/15

description sw10-03

switchport mode trunk

switchport trunk native vlan 11

speed 10000

channel-group 1

interface Ethernet1/16

description sw10-04

switchport mode trunk

switchport trunk native vlan 11

speed 10000

channel-group 1

boot kickstart bootflash:/n4000-bk9-kickstart.4.1.2.E1.1d.bin

boot system bootflash:/n4000-bk9.4.1.2.E1.1d.bin

sw10# show license usage

Feature Ins Lic Status Expiry Date Comments

Count

--------------------------------------------------------------------------------

FM_SERVER_PKG No - Unused -

ENTERPRISE_PKG Yes - Unused Never -

FC_FEATURES_PKG Yes - In use Never -

--------------------------------------------------------------------------------

sw10# sh int po 11

port-channel11 is up

sw10# show vsan membership

vsan 1 interfaces:

vsan 28 interfaces:

fc2/1 fc2/2 fc2/3 fc2/4

fc2/5 fc2/6 fc2/7 fc2/8

vfc9 vfc10 vfc28 vfc112

sw10# show vlan fcoe

Original VLAN ID Translated VSAN ID Association State

---------------- ------------------ -----------------

28 28 Operational

sw14# show license usage

Feature Ins Lic Status Expiry Date Comments

Count

--------------------------------------------------------------------------------

BASIC_STORAGE_SERVICES_PKG No - In use Grace 96D 1H

--------------------------------------------------------------------------------

sw14# sh int po1

port-channel1 is up

sw14# sh port-channel summary

--------------------------------------------------------------------------------

Group Port- Type Protocol Member Ports

Channel

--------------------------------------------------------------------------------

1 Po1(SU) Eth NONE Eth1/15(P) Eth1/16(P)

sw14# show fip-snooping vlan-discovery

sw14# sh int e1/12

Ethernet1/12 is up

Hardware: 1000/10000 Ethernet, address: 588d.090e.cfcd (bia 588d.090e.cfcd)

Any thoughts on what I am missing would be appreciated.

Thanks,

Surya ARBY · ‎12-07-2011

the fip-snooping port-mode fcf command must be applied on the uplink etherchannel, not the server facing port.

Change your config and reboot.

View solution in original post

Surya ARBY · ‎12-07-2011

your config is wrong. The nexus 4000 has to know where to forward FIP messages, and the command is applied on the wrong interface

View solution in original post

Surya ARBY · ‎12-07-2011

the fip-snooping port-mode fcf command must be applied on the uplink etherchannel, not the server facing port.

Change your config and reboot.

ArrowSIVAC · ‎12-07-2011

No details on whichever interface I try to query.

sw14# show fip-snooping vlan-discovery interface ?

ethernet Ethernet IEEE 802.3z

port-channel Port Channel interface

sw14# show fip-snooping vlan-discovery interface port-channel 1

sw14# show fip-snooping vlan-discovery interface ethernet 1/12

sw14#

Is this some kind of real time query in that the adapter must have an active kernel / driver sending out packets. I booted the server which rotates through the PXE boot mode which would (I A$$ume) accomolish this.

Surya ARBY · ‎12-07-2011

your config is wrong. The nexus 4000 has to know where to forward FIP messages, and the command is applied on the wrong interface

ArrowSIVAC · ‎12-08-2011

Thanks for the response but If you could provide some better guidance as to the correct command and where my following of the configuation guide was not correct:

http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CCcQFjAA&url=http%3A%2F%2Fwww.cisco.com%2Fen%2FUS%2Fdocs%2Fswitches%2Fdatacenter%2Fnexus5000%2Fsw%2Foperations%2Fn5k_fcoe_ops_appD.pdf&ei=lLDgTqCJC4eYgwfSsJzwBQ&usg=AFQjCNEBzTVhhD01t...

(page 8) Enable FIP Snooping on the Port channel on the 4000 for link between the 4000 and 5000

interface port-channel1

description sw10-03_04 to sw14-15_16

switchport mode trunk

switchport trunk native vlan 11

fip-snooping port-mode fcf

spanning-tree port type edge trunk

speed 10000

I was missing this and now when I run

sw14# show fip-snooping vlan-discovery

Legend:

-------------------------------------------------------------------------------

Interface VLAN FIP MAC

-------------------------------------------------------------------------------

Eth1/2 1 00:c0:dd:17:8d:f1

I see fip mac.... but not the slot 12 like I had defined. I will have to review this again to see what I am missing.

Surya ARBY · ‎12-08-2011

Issue a shut / no shut on the interface 1/12 ? Do you see LLDP negociation correctly ?

ArrowSIVAC · ‎12-09-2011

The issue was that my host facing port configuation was incorrect:

Incorrect:

interface Ethernet1/12

description bch112

switchport mode trunk

switchport trunk native vlan 11

fip-snooping port-mode fcf

spanning-tree port type edge

speed 10000

Once I changed it to:

interface Ethernet1/12

description bch112

switchport mode trunk

switchport trunk native vlan 11

spanning-tree port type edge trunk

speed 10000

then the fip snooping started showing devicse

sw14# show fip-snooping vlan-discovery

Legend:

-------------------------------------------------------------------------------

Interface VLAN FIP MAC

-------------------------------------------------------------------------------

Eth1/14 11 00:c0:dd:17:90:5d

Eth1/12 11 00:c0:dd:17:8e:11

Eth1/2 1 00:c0:dd:17:8d:f1

Now I added the binding to the vfc interface on the 5k...

interface vfc102

bind mac-address 00:c0:dd:17:8d:f1

switchport description bch102

no shutdown

interface vfc112

bind mac-address 00:c0:dd:17:8e:11

switchport description bch112

no shutdown

interface vfc114

bind mac-address 00:c0:dd:17:90:5d

switchport description bch114

no shutdown

Make sure the vfc are in the VSAN:

sw10# conf t

Enter configuration commands, one per line. End with CNTL/Z.

sw10(config)# vsan database

sw10(config)# show vsan membership

vsan 1 interfaces:

vsan 28 interfaces:

fc2/1 fc2/2 fc2/3 fc2/4

fc2/5 fc2/6 fc2/7 fc2/8

vfc9 vfc10 vfc102 vfc112

vfc113 vfc114

(Redbook and CISCO guide shows that I should be done here and the adapters show up in MDS Manager)

Now that the interfaces are present in the VSAN I would "assume" they would show up in fabric manager.. but they do not. I go into BIOS for QLogic (ctrl+Q) and tell it to scan the SAN, but the device still do not show.

I think something is wrong with the VSAN binding..

sw10# sh interface vfc114

vfc114 is trunking (Not all VSANs UP on the trunk)

Bound MAC is 00:c0:dd:17:90:5d

Port description is bch114

Hardware is Virtual Fibre Channel

Port WWN is 20:71:00:05:9b:21:60:bf

Admin port mode is F, trunk mode is on

snmp link state traps are enabled

sw10# sh interface port-channel 11

port-channel11 is up

Hardware: Port-Channel, address: 0005.9b21.608a (bia 0005.9b21.608a)

Description: sw10-03_04 to sw14-15_16

MTU 1500 bytes, BW 20000000 Kbit, DLY 10 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA

Port mode is trunk

full-duplex, 10 Gb/s

Beacon is turned off

Input flow-control is off, output flow-control is off

Switchport monitor is off

EtherType is 0x8100

Members in this channel: Eth1/3, Eth1/4

Last clearing of "show interface" counters never

30 seconds input rate 48 bits/sec, 6 bytes/sec, 0 packets/sec

30 seconds output rate 173032 bits/sec, 21629 bytes/sec, 73 packets/sec

Load-Interval #2: 5 minute (300 seconds)

input rate 112 bps, 0 pps; output rate 91.87 Kbps, 31 pps

VSAN 28 shows status as "active" and oper state of "up"

I am not sure about the "Inorder Delivery". No details on this and when I tried no change seems to have been made.

You have been very helpful and it is appreciated. Any thought would be of help

Surya ARBY · ‎12-11-2011

Is there anything not working now ? Everything seems to be good to me.