MDS configuration for port channel

[simon.geary]

I've tried to set up the new FC port channel upload to a pair of MDS 9124s, but as I don't know enough about the MDS side I can't get the link to come up. Are there any references available anywhere that tell you blow by blow exactly how to configure the MDS side of the port channel? Or perhaps a sample working config?

Thanks

[Robert Burns]

There's a bit of MDS config required.  Please reference:

http://www.cisco.com/en/US/docs/switches/datacenter/mds9000/sw/5_0/configuration/guides/int/nxos/pc.html

Essentially you have to

1. Put the member ports in the appropriate VSAN

2. Ensure the member ports are set to the appropriate mode (Fx usually)

3. Create the Port Channel interface

4. Add the first member

5. Add second member

6. No shut the port channel interface.

Let me whip this up in my lab quick and I'll provide exact steps/config example.

Regards,

Robert

[David McFarland]

Simon,

A few items:

1) You are on 1.4.1i Balboa code

2) FI's are in FC switch mode as MDS 9124's don't support F-port channel (NPV/NPIV)

3) It's a lot easier the first time if you have matched speed sfp+ all around. Although, with the correct configs, unmatched speeds can be made to work.

If the FIs are in switch mode and your MDS is running a minimum 3.3 here are some configs that may help:

MDS side interfaces 4/11 and 4/12 are plugged into the UCS fc ports. First setup and ensure ISLs are working properly between UCS and MDS, then configure the port-channels

interface fc4/11

  switchport rate-mode dedicated

  switchport mode E

  switchport trunk mode auto

  no shutdown

interface fc4/12

  switchport rate-mode dedicated

  switchport mode E

  switchport trunk mode auto

  no shutdown

The UCS GUI Equipment -> Fabric Interconnects -> FI-A -> Set FC Switching Mode. This will cause both FI's to reboot into FC switch mode. This is VERY DISRUPTIVE, both FI's will reboot.

On the UCS CLI you should see this on the UCS fc ports connected to the MDS

cae-sj-ca3-A(nxos)# show running-config interface fc 2/1-2

!Command: show running-config interface fc2/1-2

!Time: Wed Oct 20 16:49:39 2010

version 4.2(1)N1(1.4)

interface fc2/1

  switchport mode E

  no shutdown

interface fc2/2

  switchport mode E

  no shutdown

Until you have VSAN trunks enabled, make sure the ports on the UCS and MDS are in the same VSAN. VSAN 1 for example.

Once you have working ISLs, then proceed to turn them into port-channels and enable VSAN trunking. Start with the MDS

Create a channel group, it should look like this

cae-sj-9506-1# show run interface port-channel 3

!Command: show running-config interface port-channel 3

!Time: Wed Oct 20 17:06:01 2010

version 5.0(1a)

interface port-channel 3

  channel mode active

  switchport mode E

  switchport rate-mode dedicated

  switchport trunk mode auto  <-- VSAN trunking

Enable VSAN trunking on the MDS ISL interfaces, it'll look like this:

cae-sj-9506-1# show run interface fc 4/11-12

!Command: show running-config interface fc4/11-12

!Time: Wed Oct 20 17:07:05 2010

version 5.0(1a)

interface fc4/11

  switchport rate-mode dedicated

  switchport mode E

  switchport trunk mode auto

  no shutdown

interface fc4/12

  switchport rate-mode dedicated

  switchport mode E

  switchport trunk mode auto

  no shutdown

Add the channel group information to the MDS ISL interfaces:

Enter configuration commands, one per line.  End with CNTL/Z.

cae-sj-9506-1(config)# interface fc 4/11-12

cae-sj-9506-1(config-if)# channel-group 3 force

fc4/11 fc4/12 added to port-channel 3 and disabled

please do the same operation on the switch at the other end of the port-channel,

then do "no shutdown" at both ends to bring it up

cae-sj-9506-1(config-if)# show run interface fc 4/11-12

!Command: show running-config interface fc4/11-12

!Time: Wed Oct 20 17:07:39 2010

version 5.0(1a)

interface fc4/11

  switchport rate-mode dedicated

  switchport mode E

  switchport trunk mode auto

  channel-group 3 force

  no shutdown

interface fc4/12

  switchport rate-mode dedicated

  switchport mode E

  switchport trunk mode auto

  channel-group 3 force

  no shutdown

Create the SAN port channel on the UCS side

SAN -> SAN Cloud -> Fabric A -> FC Port Channels -> Create Port Channel

After you create the SAN port channel in UCS, make sure and enable it. I also bounce the MDS port channel at this point.

If all is correct, in a few agonizing minutes, the port-channel will be formed and passing data.

cae-sj-ca3-A(nxos)# show interface san-port-channel 1

san-port-channel 1 is trunking

    Hardware is Fibre Channel

    Port WWN is 24:01:00:0d:ec:d3:5d:c0

    Admin port mode is E, trunk mode is on

    snmp link state traps are enabled

    Port mode is TE

    Port vsan is 1

    Speed is 8 Gbps

    Trunk vsans (admin allowed and active) (1,10,26,50,66,100-101,103,123,222,24

0)

    Trunk vsans (up)                       (1,10,50,100,103)

    Trunk vsans (isolated)                 (26,66,101,123,222,240)

    Trunk vsans (initializing)             ()

    5 minute input rate 2312 bits/sec, 289 bytes/sec, 2 frames/sec

    5 minute output rate 1440 bits/sec, 180 bytes/sec, 2 frames/sec

      2669 frames input, 194760 bytes

        0 discards, 0 errors

        0 CRC,  0 unknown class

        0 too long, 0 too short

      2677 frames output, 158316 bytes

        0 discards, 0 errors

      0 input OLS, 1 LRR, 0 NOS, 0 loop inits

Again, make sure you have the ISLs up an running first before configuring the port-channels. It makes troubleshooting much easier.

Let me know if you need any help.

[Jeremy Waldrop · Accepted solution]

I don't think it is possible to configure the Fabric Interconnects in FC switch mode as only NPV is supported on those. The only switch mode that is supported and possible to configure is the Ethernet LAN switching mode.

I configured this last week on a pair of MDS 9124s and had to to enable the F port trunking and channeling protocol feature using this command

"feature fport-channel-trunk"

After that I was to get SAN Port Channeling working. Here is a snipit of my configuration:

feature npiv
feature fport-channel-trunk

interface port-channel 11
  channel mode active
  switchport rate-mode dedicated

interface fc1/1
  port-license acquire
  channel-group 11 force
  no shutdown

interface fc1/2
  port-license acquire
  channel-group 11 force
  no shutdown

[Robert Burns]

@Jeremey

As of Balboa (1.4) there is FC Switch mode & FC End Host mode - in addition to Ethernet Switch mode & Ethernet End Host mode.

Robert

[Jeremy Waldrop]

Oh, I didn't realize that Robert. Thank you for correcting me. I guess that is for direct attach FC storage configurtions?

[Robert Burns]

Exactly right - the FC switch mode runs the bare minimum required to directly attach NetApp/EMC targets directly to the FI's.

Let me know when you've had a chance to play around with 1.4 and/or direct attach storage - would love to hear your feedback.

Regards,

Robert

[JEFFREY SESSLER]

jwaldropvarrow wrote:
I don't think it is possible to configure the Fabric Interconnects in FC switch mode as only NPV is supported on those. The only switch mode that is supported and possible to configure is the Ethernet LAN switching mode.
I configured this last week on a pair of MDS 9124s and had to to enable the F port trunking and channeling protocol feature using this command
"feature fport-channel-trunk"
After that I was to get SAN Port Channeling working. Here is a snipit of my configuration:
feature npiv
feature fport-channel-trunk
interface port-channel 11
  channel mode active
  switchport rate-mode dedicated
interface fc1/1
  port-license acquire
  channel-group 11 force
  no shutdown
interface fc1/2
  port-license acquire
  channel-group 11 force
  no shutdown

Jeremy,

Would this setup be the same on the MDS 9148 switches? 

[David McFarland]

The MDS 9148 supports f-port channel running NX-OS 4.2.7a. You can use Rob Burns' config example for f-port channel with UCS in default End-Host-Mode (NPV) or the config I provided if you want to run in switch mode. The 9148 will support both configurations. Be warned, depending on the sfp speeds, you will have to out-of-service ports in the port group when running multiple ports at dedicated rate mode of 8GB or 4GB in the same port group.

[simon.geary]

Thanks Jeremy, feature fport-channel-trunk did it for me and I have the port channels working now on MDS 9124 running 5.0.4.

As a side note to all this, it looks like a compatibility matrix for FC switches is required for UCS. Prior to 1.4, it seems that all you needed to know about your FC switch was that it supported NPIV. The new 1.4 features mix this up a little and it is no longer easy to understand what FC switch supports what new feature. A blog post for someone perhaps!!

Thanks to everyone who chipped in with help.

[Robert Burns]

Good point Simon.  We're just polishing up some Whitepapers that should shed some practical light on the new FC features of 1.4.

Stay tuned.

Robert

[Robert Burns]

In my lab I'm using 9222i's so I can use FC End Host mode with my UCS.  FYI - I'm not using FC trunking, just FC Port channel.

Here's how I have set my system up.

6100-A Ports fc2/7, fc2/8

MDS-A Ports fc1/7, fc1/8

UCS GUI Side

1. From equipment tab add each interface (fc2/7, fc2/8) to the appropriate VSAN 2000

2. From the SAN tab, under Fab-A, create Port Channel 200, added both ports fc2/7, fc2/8 as members

(Port Channel will be disabled).

3. Enable all members ports and the parente port channel

MDS CLI Side

1. Add member ports to appropriate VSAN 200

  mds9222i(config)# vsan database

  mds9222i(config-vsan-db)# vsan 2000 interface fc1/13-14

2. Set Ports to correct mode - fabric ports

  mds9222i(config)# int fc1/13-14

  mds9222i(config-if)# switchport mode f

3. Create & configure the Port Channel

  mds9222i(config)# int port-channel 200

  mds9222i(config-if)# switchport mode F

  mds9222i(config-if)# channel mode active

  mds9222i(config-if)# vsan database

  mds9222i(config-vsan-db)# vsan 2000 interface fc1/13-14

4.  Add FC members to Port Channel

  mds9222i(config)# int fc1/13-14

  mds9222i(config-if)# channel-group 200 force

5. Unshut the Port Channel

   mds9222i(config)# int po200

   mds9222i(config-if)# no shut

The relevant config from the UCS (NXOS show run) and the MDS are below:

UCS Side

interface san-port-channel 200
  channel mode active
  switchport mode NP
  no shutdown
 
  interface fc2/7
  channel-group 200 force
  no shutdown

interface fc2/8
  channel-group 200 force
  no shutdown
 
MDS Side
 
  interface port-channel 200
  channel mode active
  switchport mode F
  switchport rate-mode shared
  no shutdown

interface fc1/13
  switchport rate-mode shared
  switchport mode F
  channel-group 200 force
  no shutdown

interface fc1/14
  switchport rate-mode shared
  switchport mode F
  channel-group 200 force
  no shutdown

You can verify if everything is working correctly with:

show int brief (ensure both member ports are showing up)

show int port-channel x (ensure Port channel is up and configured correctly)

show port-channel summary (should show two member ports)

Regards,

Robert

[Jeremy Waldrop]

On our lab 9124s we have NX-OS 5 and the only way I could get the port channels to come up was to enable the "fport-channel-trunk" feature. My 6120s are in the default End-host mode.

[iain.hardcastle]

@Jeremy,

Hi, I'm the customer that the OP is working with on this configuration.

I have a 9124 running NXOS 4.1.3a. I can see the  "fport-channel-trunk" feature and it is disabled.

I dont need F port trunking, just the channeling. It seems from the difference between our configs is that all I need to do is enable that feature to be able to configure the F port trunk. With the feature disabled I get 'port disabled', 'incompatible port types' etc.

So, questions:

1 - will it work with 4.1.3a? This 9124 is the only switch in the fabric right now so upgrading would be service affecting. The commands appear to be there, I just get the errors. I will upgrade when I can build some redundancy into the solution.

2 - is enabling the  "fport-channel-trunk" feature disruptive (see 1 above)?

3 - with the feature disabled, do you see the errors I describe above?

FC and MDS config is a grey area for me so I do appreciate any input...!

thanks a million

iain