UCS FI 6248 to Nexus 5548 San port-channel - not working

[vjmendoza]

Hi all,

I'm sure I am missing something fairly obvious and stupid but I need several sets of eyes and help.

Here is the scenario:

I want to be able to create san port-channels between the FI and Nexus.  I don't need to trunk yet as I can't even get the channel to come up.

UCS FI 6248:

Interfaces fc1/31-32

Nexus 5548

interfaces fc2/15-16

FI is in end-host mode and Nexus is running NPIV mode with fport-channel-trunk feature enabled.

I'm going to output the relevants configurations below.

Nexus 5548:

NX5KA(config)# show feature | include enabled

fcoe                  1         enabled

fex                   1         enabled

fport-channel-trunk   1         enabled

hsrp_engine           1         enabled

interface-vlan        1         enabled

lacp                  1         enabled

lldp                  1         enabled

npiv                  1         enabled

sshServer             1         enabled

vpc                   1         enabled

interface san-port-channel 133

  channel mode active

  no switchport trunk allowed  vsan all

  switchport trunk mode off

interface fc2/15

  switchport trunk mode off

  channel-group 133 force

no shutdown

interface fc2/16

  switchport trunk mode off

  channel-group 133 force

no shutdown

NX5KA# show vsan membership

vsan 1 interfaces:

    fc2/13            fc2/14          

vsan 133 interfaces:

    fc2/15            fc2/16            san-port-channel 133

vsan 4079(evfp_isolated_vsan) interfaces:

vsan 4094(isolated_vsan) interfaces:

NX5KA# show san-port-channel summary

U-Up D-Down B-Hot-standby S-Suspended I-Individual link

summary header

--------------------------------------------------------------------------------

Group  Port-      Type    Protocol               Member Ports

       Channel

--------------------------------------------------------------------------------

133    San-po133  FC      PCP      (D)  FC       fc2/15(D)    fc2/16(D)   

UCS Fabric Interconnect outputs:

UCS-FI-A-A(nxos)# show san-port-channel summary

U-Up D-Down B-Hot-standby S-Suspended I-Individual link

summary header

--------------------------------------------------------------------------------

Group  Port-      Type    Protocol               Member Ports

       Channel

--------------------------------------------------------------------------------

133    San-po133  FC      PCP      (D)  FC       fc1/31(D)    fc1/32(D)   

UCS-FI-A-A(nxos)#

UCS-FI-A-A(nxos)# show run int fc1/31-32

!Command: show running-config interface fc1/31-32

!Time: Fri Dec 20 22:58:51 2013

version 5.2(3)N2(2.21b)

interface fc1/31

  switchport mode NP

  channel-group 133 force

  no shutdown

interface fc1/32

  switchport mode NP

  channel-group 133 force

  no shutdown

UCS-FI-A-A(nxos)#

UCS-FI-A-A(nxos)# show run int san-port-channel 133

!Command: show running-config interface san-port-channel 133

!Time: Fri Dec 20 22:59:09 2013

version 5.2(3)N2(2.21b)

interface san-port-channel 133

  channel mode active

  switchport mode NP

[vjmendoza]

I decided to turn on terminal monitor on the Nexus 5k and this is the output I get:

2009 May 15 23:01:21 NX5KA May 15 23:01:21 %KERN-3-SYSTEM_MSG: fc2_nsh_tx_frame: FC2 s_id/d_id/vsan error: sid=0xfffffe,did=0x0,vsan=133,rctl:0x23,type:0x1,oxid 0x207,rxid:0x285 - kernel

2009 May 15 23:01:54 NX5KA %PORT-5-IF_PORT_QUIESCE_FAILED: Interface fc2/16 port quiesce failed due to failure reason: Force Abort Due to Link Failure (NOS/LOS) (0x119)

2009 May 15 23:01:54 NX5KA %PORT-5-IF_DOWN_OLS_RCVD: %$VSAN 133%$ Interface fc2/16 is down (OLS received) san-port-channel 133 

2009 May 15 23:01:55 NX5KA May 15 23:01:55 %KERN-3-SYSTEM_MSG: fc2_nsh_tx_frame: FC2 s_id/d_id/vsan error: sid=0xfffffe,did=0x0,vsan=133,rctl:0x23,type:0x1,oxid 0x21f,rxid:0x29e - kernel

2009 May 15 23:02:00 NX5KA %PORT-5-IF_PORT_QUIESCE_FAILED: Interface fc2/15 port quiesce failed due to failure reason: Force Abort Due to Link Failure (NOS/LOS) (0x119)

2009 May 15 23:02:00 NX5KA %PORT-5-IF_DOWN_OLS_RCVD: %$VSAN 133%$ Interface fc2/15 is down (OLS received) san-port-channel 133 

2009 May 15 23:02:01 NX5KA May 15 23:02:01 %KERN-3-SYSTEM_MSG: fc2_nsh_tx_frame: FC2 s_id/d_id/vsan error: sid=0xfffffe,did=0x0,vsan=133,rctl:0x23,type:0x1,oxid 0x225,rxid:0x2a4 - kernel

2009 May 15 23:02:04 NX5KA May 15 23:02:04 %KERN-3-SYSTEM_MSG: fc2_nsh_tx_frame: FC2 s_id/d_id/vsan error: sid=0xfffffe,did=0x0,vsan=133,rctl:0x23,type:0x1,oxid 0x229,rxid:0x2aa - kernel

[vjmendoza]

Incidentally,

If I turn off or don't create san-port channels, the UCS logs onto the fabric and all the FC interfaces come up.

I'm missing something..

[richbarb]

Hello vjmendoza,

Your Nexus 5548 FC interfaces should have this configuration:

interface san-port-channel 133

  channel mode active

  switchport mode f

  switchport trunk mode off

interface fc2/15

  switchport trunk mode off

  switchport mode f

  channel-group 133 force

no shutdown

interface fc2/16

  switchport trunk mode off

  switchport mode f

  channel-group 133 force

no shutdown

Richard

[vjmendoza]

interface san-port-channel 133

  channel mode active

  switchport mode F

  no switchport trunk allowed  vsan all

  switchport trunk mode off

version 5.1(3)N1(1)

interface fc2/15

  switchport mode F

  switchport trunk mode off

  channel-group 133 force

  no shutdown

interface fc2/16

  switchport mode F

  switchport trunk mode off

  channel-group 133 force

  no shutdown

Same results - I downed the san-port-channel on both sides.

NX5KA(config-if)# show san-port-channel summary

U-Up D-Down B-Hot-standby S-Suspended I-Individual link

summary header

--------------------------------------------------------------------------------

Group  Port-      Type    Protocol               Member Ports

       Channel

--------------------------------------------------------------------------------

133    San-po133  FC      PCP      (D)  FC       fc2/15(D)    fc2/16(D)   

NX5KA(config-if)#

[richbarb]

Ok, I understand what's going on.

The VSANs created in the UCS should be applied in the SAN port-channel, like this:

(I didn't find a san port-channel configuration image, but is the same configuration type).

In the Nexus you must create those VSANs and leave like this:

Nexus A

#vsan database

#vsan 10

#vsan 10 interface san-port-channel 133

Nexus B

#vsan database

#vsan 20

#vsan 20 interface san-port-channel 133

Richard

[vjmendoza]

NX5KA(config-if)# show vsan membership

vsan 1 interfaces:

    fc2/13            fc2/14          

vsan 133 interfaces:

    fc2/15            fc2/16            san-port-channel 133

vsan 4079(evfp_isolated_vsan) interfaces:

vsan 4094(isolated_vsan) interfaces:

NX5KA(config-if)#

vsan database

  vsan 133

interface san-port-channel 133

  channel mode active

  switchport mode F

  no switchport trunk allowed  vsan all

  switchport trunk mode off

vsan database

  vsan 133 interface san-port-channel 133

So this is the configuration from the beginning on the Nexus side.

And the screenshot on the UCSM shows that I did attached the port-channel to VSAN133 in UCS

[vjmendoza]

For further clarity, here is the show output from the nxos (FI)

UCS-FI-A-A(nxos)# show vsan membership

vsan 1 interfaces:

    fc1/29            fc1/30          

vsan 100 interfaces:

    vfc687          

vsan 133 interfaces:

    fc1/31            fc1/32            san-port-channel 133

vsan 4079(evfp_isolated_vsan) interfaces:

vsan 4094(isolated_vsan) interfaces:

UCS-FI-A-A(nxos)#

[richbarb]

Considering you said that it works when disabled the port-channel, I will not ask you if the transceivers and cabling are ok.

This command in the san-port channel isn't necessary:

no switchport trunk allowed  vsan all

Please leave this way:

switchport trunk allowed  vsan all (this is the deafult behavior)

Try shutdown all interfaces including the physical in both sides, after that, enable in this order:

Nexus FC -> Nexus SAN-PC -> FI FC -> FI SAN-PC.

If still not working, please give me the commands outputs:

NexusA# sh int san-port-channel 133

NexusA# sh int bri

[vjmendoza]

!Command: show running-config interface san-port-channel 133

!Time: Sat May 16 04:59:07 2009

version 5.1(3)N1(1)

interface san-port-channel 133

  channel mode active

  switchport mode F

  switchport trunk mode off

Changed it as you suggested...

Followed the order of operations for "no shut"

Nexus FC -> Nexus SAN-PC -> FI FC -> FI SAN-PC.

Didn't work:

NX5KA(config-if)# show san-port-channel summary

U-Up D-Down B-Hot-standby S-Suspended I-Individual link

summary header

--------------------------------------------------------------------------------

Group  Port-      Type    Protocol               Member Ports

       Channel

--------------------------------------------------------------------------------

133    San-po133  FC      PCP      (D)  FC       fc2/15(D)    fc2/16(D)

NX5KA(config-if)#

Here is the output as you requested:

NX5KA(config-if)# show int san-port-channel 133

san-port-channel 133 is down (No operational members)

    Hardware is Fibre Channel

    Port WWN is 24:85:00:2a:6a:5a:81:00

    Admin port mode is F, trunk mode is off

    snmp link state traps are enabled

    Port vsan is 133

    1 minute input rate 1256 bits/sec, 157 bytes/sec, 0 frames/sec

    1 minute output rate 248 bits/sec, 31 bytes/sec, 0 frames/sec

      3966 frames input, 615568 bytes

        0 discards, 0 errors

        0 CRC,  0 unknown class

        0 too long, 0 too short

      2956 frames output, 143624 bytes

        0 discards, 0 errors

      46 input OLS, 41 LRR, 73 NOS, 0 loop inits

      257 output OLS, 189 LRR, 219 NOS, 0 loop inits

    last clearing of "show interface" counters never

    Member[1] : fc2/15

    Member[2] : fc2/16

NX5KA(config-if)#

NX5KA(config-if)# show int brief

-------------------------------------------------------------------------------

Interface  Vsan   Admin  Admin   Status          SFP    Oper  Oper   Port

                  Mode   Trunk                          Mode  Speed  Channel

                         Mode                                 (Gbps)

-------------------------------------------------------------------------------

fc2/13     1      auto   on      sfpAbsent        --     --           --

fc2/14     1      auto   on      sfpAbsent        --     --           --

fc2/15     133    F      off     init             swl    --           133

fc2/16     133    F      off     init             swl    --           133

[richbarb]

This configuration examples to 5548UP can help you.

http://www.cisco.com/en/US/partner/docs/switches/datacenter/nexus5000/sw/san_switching/513_n2_1/b_Cisco_n5k_nxos_sanswitching_config_guide_rel513_n1_1_chapter_01000.html#task_B5E9AC0B14614A95868D519145D47F7E

About the UCS.

All transceivers have the same speed capability?

The FI are in end-host-mode?

The FI aren't using the "trunk uplink" option in the SAN configuration?

The cables could be cross connected, did you check it?

Try build all configuration again step-by-step calmly, if still not work I only can think some kind of bug in your versions combined.

Richard