Multihop FCoE question

[krun_shah]

is it possible to bind a portchannel carrying multiple fcoe vlan to more than one vfc in diff vsan?

Lets  say with one N5K connected to N7K using a eth-portchannel carrying FCoE vlans 100(vsan 100) and 200 (vsan 200). Is it possible to configure two vfc  interfaces in vsan 100 and 200 respectively on each switch? That way one  link failure in port-channel bundle will not cause the whole SAN fabric  to be down and hashes FCoE traffic to available port.

[vdsudame]

i dont think there is a need to have multiple vfcs on n5k/n7k and dont believe we can do this either. the vfc will go down only when all the ports in the port channel go down. until that point the vfc is going to stay up. also you can implement "port channel load balance source-dest-port" for lacp po so as to have all the links in port channel utilized uniformly. by default we use single link only with source dest.

Vin.

[krun_shah]

Sorry If I confused anyone. What I mean is one vfc in vsan 100 and one vfc in vsan 200. Both vfc has bind itnerface po 10 command under it as well as port chanenl 10 is trunking FCOE vlans 100 and 200. Is this config possible?

[vdsudame]

we cannot bind one physica or port channel interface in multiple vfcs.

[krun_shah]

           

unfortunatly this is not documented on nexus 5000 and/or 7000 configuration guides.

http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/fcoe/513_n1_1/b_Cisco_n5k_fcoe_config_gd_re_513_n1_1_chapter_0100.html#task_12735881956

http://www.cisco.com/en/US/partner/docs/switches/datacenter/sw/nx-os/fcoe/configuration_guide/b_Cisco_NX-OS_FCoE_Configuration_Guide_chapter_0100.html#task_12735881956

Conceptually it should be possible, I do not have lab equipment to test so just want to know what is technical reason behind it.

feature fcoe

vsan database

vsan 100 name Fabric-A

vsan 200 name Fabric-B

vlan 100

fcoe vsan 100

vlan 200

fcoe vsan 200

int eth 1/1-2

channe-group 10 mode active

int po 10

switchport mode trunk

switchport trunk allowed vlan 1,100,200

int vfc 1

  switchport mode E

  bind interface po 10

int vfc 2

  switchport mode E

  bind interface po 10

vsan database

vsan 100 interface vfc 1

vsan 200 interface vfc 2

[vdsudame]

at fiber channel level when a device logs in or does a flogi (fabric login) it gets a unique wwn (which is analogous to a mac address in ethernet world). we cannot have that same unique wwn log in on two different ports on a nexus switch at the same time. For this same reason you cannot bind same port channel in two different vfcs as we cannot have a fabric login with the same wwn across two different vfcs.

[Carlo Schmidt]

Krun,

I think what you are trying to achieve is dedicating one vFC per vsan. This is possible but instead of creating a port-channel with two links and binding two vFCs to it, you would have to create two SINGLE link port channels. Once you have two port channels, you can bind ONE vFC to each.

By doing this, you lose 'link level redundancy' which you would have had by using a port channel with multiple physical  interfaces.

Now if you determine you want link level redundancy, you have to create a two interface port channel. This port channel will be between two different Cisco switches so from a Fiber Channel perspective, this will automatically be a Trunking E port. This port channel, which is a "virtual link" since you are combining multiple interfaces into one logical link will have to follow the same rules any single interface follows, which is ONE TE (Trunking Expansion) port.

[krun_shah]

Do you mean, combining multiple vfc interfaces in to a single vfc-port-channel ?? This is only possible in N7K.

Further more, it is not possible to VSAN trunking on vfc or vfc-portchannel interfaces....This is clearly documented in config guide.

I can do something like following

int vfc 1

bind interface eth1/1

channel-group 10

int vfc 2

bind interface eth1/2

channel-group 10

int vfc-port-channel 10

But above vfc-port-channel can only be E or F port not trunk carrying multiple vsans.

[Carlo Schmidt]

Can you provide a link to the documentation + page number? Here is an example of what I mean:

• vFC-Port-channel on N7K

7010-1-fcoe# show int vfc-port-channel 20

vfc-po20 is trunking

    Bound interface is port-channel20

    Hardware is Ethernet

    Port WWN is 25:25:00:22:55:79:66:00

    Admin port mode is E, trunk mode is on

    snmp link state traps are enabled

    Port mode is TE

    Port vsan is 1

    Speed is 20 Gbps

    Trunk vsans (admin allowed and active) (200,3011)

   Trunk vsans (up)                       (200,3011)

    Trunk vsans (isolated)                 ()

    Trunk vsans (initializing)             ()

vfc-po20

    304630 fcoe in packets

    52662280 fcoe in octets

    304747 fcoe out packets

    23547260 fcoe out octets

    Interface last changed at Fri Apr 27 16:04:17 2012

• Running config:

!Command: show running-config interface vfc-po20

!Time: Sat Apr 28 09:54:38 2012

version 5.2(4)

interface vfc-po20

  bind interface port-channel20

  switchport mode E

  switchport trunk allowed vsan 200

  switchport trunk allowed vsan add 3011

  no shutdown

• vFC on the Nexus 5000:

5020-1(config-if)# show int vfc 20

vfc20 is trunking (Not all VSANs UP on the trunk)

    Bound interface is port-channel20

    Hardware is Ethernet

    Port WWN is 20:13:00:0d:ec:a3:81:bf

    Admin port mode is E, trunk mode is on

    snmp link state traps are enabled

    Port mode is TE

    Port vsan is 1

    Trunk vsans (admin allowed and active) (1,200,3011)

    Trunk vsans (up)                       (200, 3011)

    Trunk vsans (isolated)                 ()

    Trunk vsans (initializing)             (1)

    1 minute input rate 2640 bits/sec, 330 bytes/sec, 3 frames/sec

    1 minute output rate 6928 bits/sec, 866 bytes/sec, 3 frames/sec

      302132 frames input, 23339536 bytes

        0 discards, 0 errors

      302016 frames output, 57040752 bytes

        0 discards, 0 errors

    last clearing of "show interface" counters Fri Apr 27 14:23:37 2012

• Running config on N5K

!Command: show running-config interface vfc20

!Time: Sat Apr 28 09:56:51 2012

version 5.1(3)N2(1)

interface vfc20

  bind interface port-channel20

  switchport mode E

  no shutdown

[krun_shah]

Thanks Carlo that helped a lot.

Terminology is bit confusing because even though you configured switchport mode E under int vfc-port-channel 20, show ocmmand shows TE_port. Normally in MDS world you would configure "switchport trunk mode on|auto" then it becomes a TE_port.

Can you also share config from actual vfc interfaces you bundled in vfc-port-channel? Configuration guide does not show how to bundle vfc interface into vfc-port-channel and confuses a lot. see below

http://www.cisco.com/en/US/partner/docs/switches/datacenter/sw/nx-os/fcoe/configuration_guide/b_Cisco_NX-OS_FCoE_Configuration_Guide_chapter_0100.html#task_8E581CB3695C4748B49157F26DB439F9

Step 3

switchport mode                      mode                                Example: switch(config-if)# switchport mode e

Configures the switchport mode for a  virtual Fibre Channel interface. The mode is E or F. The default is F mode.

So I thought vfc interfaces cannot be a trunk ( carrying multiple vsans) ports TE_port it can only be E_port (expanssion port) or F_port.

In nexus 5000 config guide clearly says in limitation you can assign only one vsan per vfc interface and looks like you are managed to trunk multiple vsans on single vfc.

Each vFC interface is                     associated with only one VSAN.

http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/fcoe/513_n1_1/b_Cisco_n5k_fcoe_config_gd_re_513_n1_1_chapter_0100.html#con_1288667

[vdsudame]

on n7k this is auto bind, ie once you create say regular port channel 10, you have to create the same vfc-port-chanel number 10 and it auto binds the po10 to vfc-port-channel 10 implicitly. As you already know, this is not the same on n5k. on n5k we use the regular 'bind interface po' command to associate the interface vfc with the port channel.

[Carlo Schmidt · Accepted solution]

There is no vFC port channel, the way the N7K and MDS do FCoE is slightly different from the 5K.

On the 7K/MDS, you create a regular port channel. example:

!Command: show running-config interface port-channel22 membership

!Time: Tue May  1 09:54:35 2012

version 5.2(4)

interface port-channel22

  switchport

  switchport mode trunk

  switchport trunk allowed vlan 200

interface Ethernet7/22

  switchport mode trunk

  switchport trunk allowed vlan 200

  channel-group 22 mode active

  no shutdown

interface Ethernet7/23

  switchport mode trunk

  switchport trunk allowed vlan 200

  channel-group 22 mode active

  no shutdown

Now to trunk VSANs over this, you need to allow the vlan mapped to the vsan and configure the following:

N7K-24-12-7010-1-fcoe# show run int vfc-port-channel 22

!Command: show running-config interface vfc-po22

!Time: Tue May  1 09:56:40 2012

version 5.2(4)

interface vfc-po22

  bind interface port-channel22

  switchport mode E

  switchport trunk allowed vsan 200

  no shutdown

The MDS and 7K map the vfc-port-channel to the port-channel automatically as long as you are using the same number.

On the 5K, you would create a vFC and bind it to a port-channel.

Does this answer your questions?

[krun_shah]

Wwn? Or fcid i thought wwn are hardcoded in hba

[Vu Phan]

WWN and PWWN are unique and hardcode or burn to HBA. This number is not changeable.

WWN is unique number for HBA

PWWN is unique number for specific port on that HBA

FCID is assigned by the switch when the HBA flogi to fabric