02-25-2012 08:52 AM
Hello every one.
I have two IBM BladeCenter H Series each one equiped with seven IBM HS22, as an I/O device one Qlogic 2 10 GE CNA addpater has been installed on IBM HS22s. in I/O bay 7 and 9 Nexus 4001i has been installed, These NX4001is are connected to two Nexus 5010 and from there Nexus 5010 are connected to MDS9134 and EMC VNX 5300 (FCoE). I have been trying to configure the whole network using FCoE for storage access on HS22 servers VMware ESX 4U1 is installed. after three weeks of strugle i still can not see any mac address when I issue command "fip-snooping vlaqn-discovery".
I have minimized my whole topology into one single link between Nexus 4001i and Nexus 5010 and still it's not working, when i enable the debug for dcbx i see the following error:
170) Event:E_DEBUG, length:77, at 76332 usecs after Sat Feb 25 16:12:29 2012
[102] dcx_lldp_get_lls:429 Could not find LLDP Feature on if_index=0x1a003000
even though when i check LLDP on all interfaces it shows it is enabled and receiving and sending traffics.
when i issue "show mac-address-table" on nexus 4001i, i can not see any macc address. follwing is the configuration for both switches:
Nexus 4001i | Nexus 5010 |
---|---|
version 4.1(2)E1(1f) feature telnet feature lacp feature fip-snooping ip domain-lookup ip host MapNX4001-B1-7 192.168.10.154 switchname MapNX4001-B1-7 vrf context management ip route 0.0.0.0/0 172.27.66.1 vlan 1 vlan 9 fip-snooping enable interface Ethernet1/1 switchport mode trunk switchport trunk allowed vlan 1,9 spanning-tree port type edge trunk speed auto interface Ethernet1/2 switchport mode trunk switchport trunk allowed vlan 1,9 spanning-tree port type edge trunk speed auto interface Ethernet1/3 switchport mode trunk switchport trunk allowed vlan 1,9 spanning-tree port type edge trunk speed auto interface Ethernet1/4 switchport mode trunk switchport trunk allowed vlan 1,9 spanning-tree port type edge trunk speed auto interface Ethernet1/5 switchport mode trunk switchport trunk allowed vlan 1,9 spanning-tree port type edge trunk speed auto interface Ethernet1/6 switchport mode trunk switchport trunk allowed vlan 1,9 spanning-tree port type edge trunk speed auto interface Ethernet1/7 switchport mode trunk switchport trunk allowed vlan 1,9 spanning-tree port type edge trunk speed auto interface Ethernet1/8 shutdown speed auto interface Ethernet1/9 shutdown speed auto interface Ethernet1/10 shutdown speed auto interface Ethernet1/11 shutdown speed auto interface Ethernet1/12 shutdown speed auto interface Ethernet1/13 shutdown speed auto interface Ethernet1/14 shutdown speed auto interface Ethernet1/15 shutdown speed 10000 interface Ethernet1/16 shutdown speed 10000 interface Ethernet1/17 !!! Link to Nexus 5010 switchport mode trunk switchport trunk allowed vlan 9 fip-snooping port-mode fcf spanning-tree port type trunk speed 10000 interface Ethernet1/18 shutdown speed 10000 interface Ethernet1/19 shutdown speed 10000 interface Ethernet1/20 shutdown speed 10000 interface mgmt0 ip address 192.168.10.154/24 interface mgmt1 boot kickstart bootflash:/n4000-bk9-kickstart.4.1.2.E1.1f.bin boot system bootflash:/n4000-bk9.4.1.2.E1.1f.bin system health loopback frequency 60 ip route 0.0.0.0/0 192.168.10.100 | version 5.0(3)N2(1) feature fcoe feature telnet feature lldp ip domain-lookup hostname MapNX5010-1 class-map type qos class-fcoe class-map type queuing class-all-flood match qos-group 2 class-map type queuing class-ip-multicast match qos-group 2 class-map type network-qos class-all-flood match qos-group 2 class-map type network-qos class-ip-multicast match qos-group 2 vrf context management ip route 0.0.0.0/0 192.168.10.100 vlan 1 vlan 9 fcoe vsan 9 vsan database vsan 9 interface fc2/1 interface fc2/2 interface fc2/3 interface fc2/4 interface fc2/5 interface fc2/6 interface Ethernet1/1 shutdown interface Ethernet1/2 shutdown interface Ethernet1/3 shutdown interface Ethernet1/4 shutdown interface Ethernet1/5 shutdown interface Ethernet1/6 shutdown interface Ethernet1/7 !!! Link to Nexus 4001i switchport mode trunk switchport trunk allowed vlan 1,9 interface Ethernet1/8 shutdown interface Ethernet1/9 shutdown interface Ethernet1/10 shutdown interface Ethernet1/11 shutdown interface Ethernet1/12 shutdown interface Ethernet1/13 shutdown interface Ethernet1/14 shutdown interface Ethernet1/15 shutdown interface Ethernet1/16 shutdown interface Ethernet1/17 shutdown interface Ethernet1/18 shutdown interface Ethernet1/19 shutdown interface Ethernet1/20 shutdown interface mgmt0 ip address 192.168.10.151/24 line console line vty boot kickstart bootflash:/n5000-uk9-kickstart.5.0.3.N2.1.bin boot system bootflash:/n5000-uk9.5.0.3.N2.1.bin interface fc2/1 interface fc2/2 interface fc2/3 interface fc2/4 interface fc2/5 interface fc2/6 MapNX5010-1# |
Regards
MAK
Solved! Go to Solution.
02-26-2012 04:56 PM
On 5K
Down link
interface Ethernet1/7
!!! Link to Nexus 4001i
switchport mode trunk
switchport trunk allowed vlan 1,9 < --- Remove
On 4K
Up-link
interface Ethernet1/17
!!! Link to Nexus 5010
switchport mode trunk
switchport trunk allowed vlan 9 < --- Remove
fip-snooping port-mode fcf
spanning-tree port type trunk < --- Remove
speed 10000
On Server interfaces e1/1-7
interface Ethernet1/1
switchport mode trunk
switchport trunk native vlan 1 < ---- Add
switchport trunk allowed vlan 1,9
spanning-tree port type edge trunk
speed auto
- shut/no shut after modify these paramenters
- Make sure QOS add to N4K
policy-map type queuing policy-fcoe-bandwidth class type queuing 1p7q4t-out-q-default bandwidth percent 50 class type queuing 1p7q4t-out-pq1 bandwidth percent 7 class type queuing 1p7q4t-out-q2 bandwidth percent 7 class type queuing 1p7q4t-out-q3 bandwidth percent 7 class type queuing 1p7q4t-out-q4 bandwidth percent 7 class type queuing 1p7q4t-out-q5 bandwidth percent 7 class type queuing 1p7q4t-out-q6 bandwidth percent 7 class type queuing 1p7q4t-out-q7 bandwidth percent 7 class-map type network-qos Jumbo1 match cos 4 class-map type network-qos class-fcoe match cos 3 class-map type network-qos class-non-fcoe match cos 0-2,4-7 policy-map type network-qos Jumbo class type network-qos Jumbo1 mtu 9216 policy-map type network-qos policy-fcoe class type network-qos class-fcoe pause no-drop mtu 2500 class type network-qos class-non-fcoe system qos service-policy type network-qos policy-fcoe service-policy type queuing output policy-fcoe-bandwidth
02-26-2012 05:20 AM
I don't see any configuration for your vfc interface?
To bind to a MAC address when the CNA is remotely connected through a DCB bridging device such as a Nexus 4000 to the Nexus 5x00.
config on the 5k:
(config-if)# interface vfc n
(config-if)# bind mac-address 00:0a:00:00:00:36
---
Posted by WebUser Adriaan Steyn
02-26-2012 06:04 AM
Thank you Adriaan,
As you mentioned correctly i need to create a vfc and bind it to discoverd MACs of Server's CNAs, but that is exactly where i got stuck, when i issue the command "fip-snooping vlan-discovery" to get the resolved mac address for vfc binding, the comannd does not have ny output.
02-26-2012 04:56 PM
On 5K
Down link
interface Ethernet1/7
!!! Link to Nexus 4001i
switchport mode trunk
switchport trunk allowed vlan 1,9 < --- Remove
On 4K
Up-link
interface Ethernet1/17
!!! Link to Nexus 5010
switchport mode trunk
switchport trunk allowed vlan 9 < --- Remove
fip-snooping port-mode fcf
spanning-tree port type trunk < --- Remove
speed 10000
On Server interfaces e1/1-7
interface Ethernet1/1
switchport mode trunk
switchport trunk native vlan 1 < ---- Add
switchport trunk allowed vlan 1,9
spanning-tree port type edge trunk
speed auto
- shut/no shut after modify these paramenters
- Make sure QOS add to N4K
policy-map type queuing policy-fcoe-bandwidth class type queuing 1p7q4t-out-q-default bandwidth percent 50 class type queuing 1p7q4t-out-pq1 bandwidth percent 7 class type queuing 1p7q4t-out-q2 bandwidth percent 7 class type queuing 1p7q4t-out-q3 bandwidth percent 7 class type queuing 1p7q4t-out-q4 bandwidth percent 7 class type queuing 1p7q4t-out-q5 bandwidth percent 7 class type queuing 1p7q4t-out-q6 bandwidth percent 7 class type queuing 1p7q4t-out-q7 bandwidth percent 7 class-map type network-qos Jumbo1 match cos 4 class-map type network-qos class-fcoe match cos 3 class-map type network-qos class-non-fcoe match cos 0-2,4-7 policy-map type network-qos Jumbo class type network-qos Jumbo1 mtu 9216 policy-map type network-qos policy-fcoe class type network-qos class-fcoe pause no-drop mtu 2500 class type network-qos class-non-fcoe system qos service-policy type network-qos policy-fcoe service-policy type queuing output policy-fcoe-bandwidth
02-27-2012 06:05 AM
Dear Vu,
Thank you so much for the help, the problem has been resolved, I guess QoS Policy resolved the issue. the system in now working fine. Thanx again .
Regards
CoolMAK
05-14-2012 11:47 AM
Hi All,
I also encounter the same issue. Is this N4k QOS setting is Cisco recommended one?
Can anyone provide Cisco document for QOS requirement for N4k FIP snooping/ FCOE?
Thanks,
Ye
05-14-2012 11:52 AM
Hello Ye,
Here is the document for you
Thanks
Vu
05-14-2012 12:53 PM
Hi Vu,
Thanks for your kind response. I saw this one but it's general guide how to configure. For customer point of view, they need Cisco's confirmation as it's production network.
For N7k and N5k, Cisco mention this QOS requirement for FCOE very clearly. But for N4k, there is no such info, I also check below design guide, Cisco didn't mention about QOS requirement for N4k.
http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps10596/deployment_guide_c07-574724.pdf
Thank you,
Ye
05-14-2012 04:01 PM
Hello Ye,
Confirmed QOS is required for FCOE fip-snooping
QoS must be configured for FCoE before FIP snooping is enabled. MTU, PFC, and ETS are required for FIP snooping. During initial configuration of the switch, QoS is configured by default if you configure FCoE at the time. If you want to change the default QoS configuration, you should configure QoS.
05-14-2012 08:54 PM
Hi Vu,
That means there will be QOS config if we configured FCOE during the initial setup. Can we assume that the below qos setting is the default config requirement?
As all our customer's N4k are in production and we need to make sure not to be any impact. Also not possible to get a spare to test, that's why, we need Cisco confirmation or official document.
policy-map type queuing policy-fcoe-bandwidth
class type queuing 1p7q4t-out-q-default
bandwidth percent 50
class type queuing 1p7q4t-out-pq1
bandwidth percent 7
class type queuing 1p7q4t-out-q2
bandwidth percent 7
class type queuing 1p7q4t-out-q3
bandwidth percent 7
class type queuing 1p7q4t-out-q4
bandwidth percent 7
class type queuing 1p7q4t-out-q5
bandwidth percent 7
class type queuing 1p7q4t-out-q6
bandwidth percent 7
class type queuing 1p7q4t-out-q7
bandwidth percent 7
class-map type network-qos Jumbo1
match cos 4
class-map type network-qos class-fcoe
match cos 3
class-map type network-qos class-non-fcoe
match cos 0-2,4-7
policy-map type network-qos Jumbo
class type network-qos Jumbo1
mtu 9216
policy-map type network-qos policy-fcoe
class type network-qos class-fcoe
pause no-drop
mtu 2500
class type network-qos class-non-fcoe
system qos
service-policy type network-qos policy-fcoe
service-policy type queuing output policy-fcoe-bandwidth policy-map type queuing policy-fcoe-bandwidth
Thanks you so much!
Ye
05-17-2012 07:34 AM
Hello Ye,
confirmed, the QOS listed above is a requirement for FCOE
thanks
Vu
05-17-2012 08:15 AM
Thanks a lot. After enabling QOS on N4k site, VSANs are UP on N5k site.
Regards,
Ye
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide