12-20-2013 03:08 PM
Hi all,
I'm sure I am missing something fairly obvious and stupid but I need several sets of eyes and help.
Here is the scenario:
I want to be able to create san port-channels between the FI and Nexus. I don't need to trunk yet as I can't even get the channel to come up.
UCS FI 6248:
Interfaces fc1/31-32
Nexus 5548
interfaces fc2/15-16
FI is in end-host mode and Nexus is running NPIV mode with fport-channel-trunk feature enabled.
I'm going to output the relevants configurations below.
Nexus 5548:
NX5KA(config)# show feature | include enabled
fcoe 1 enabled
fex 1 enabled
fport-channel-trunk 1 enabled
hsrp_engine 1 enabled
interface-vlan 1 enabled
lacp 1 enabled
lldp 1 enabled
npiv 1 enabled
sshServer 1 enabled
vpc 1 enabled
interface san-port-channel 133
channel mode active
no switchport trunk allowed vsan all
switchport trunk mode off
interface fc2/15
switchport trunk mode off
channel-group 133 force
no shutdown
interface fc2/16
switchport trunk mode off
channel-group 133 force
no shutdown
NX5KA# show vsan membership
vsan 1 interfaces:
fc2/13 fc2/14
vsan 133 interfaces:
fc2/15 fc2/16 san-port-channel 133
vsan 4079(evfp_isolated_vsan) interfaces:
vsan 4094(isolated_vsan) interfaces:
NX5KA# show san-port-channel summary
U-Up D-Down B-Hot-standby S-Suspended I-Individual link
summary header
--------------------------------------------------------------------------------
Group Port- Type Protocol Member Ports
Channel
--------------------------------------------------------------------------------
133 San-po133 FC PCP (D) FC fc2/15(D) fc2/16(D)
UCS Fabric Interconnect outputs:
UCS-FI-A-A(nxos)# show san-port-channel summary
U-Up D-Down B-Hot-standby S-Suspended I-Individual link
summary header
--------------------------------------------------------------------------------
Group Port- Type Protocol Member Ports
Channel
--------------------------------------------------------------------------------
133 San-po133 FC PCP (D) FC fc1/31(D) fc1/32(D)
UCS-FI-A-A(nxos)#
UCS-FI-A-A(nxos)# show run int fc1/31-32
!Command: show running-config interface fc1/31-32
!Time: Fri Dec 20 22:58:51 2013
version 5.2(3)N2(2.21b)
interface fc1/31
switchport mode NP
channel-group 133 force
no shutdown
interface fc1/32
switchport mode NP
channel-group 133 force
no shutdown
UCS-FI-A-A(nxos)#
UCS-FI-A-A(nxos)# show run int san-port-channel 133
!Command: show running-config interface san-port-channel 133
!Time: Fri Dec 20 22:59:09 2013
version 5.2(3)N2(2.21b)
interface san-port-channel 133
channel mode active
switchport mode NP
12-20-2013 03:09 PM
I decided to turn on terminal monitor on the Nexus 5k and this is the output I get:
2009 May 15 23:01:21 NX5KA May 15 23:01:21 %KERN-3-SYSTEM_MSG: fc2_nsh_tx_frame: FC2 s_id/d_id/vsan error: sid=0xfffffe,did=0x0,vsan=133,rctl:0x23,type:0x1,oxid 0x207,rxid:0x285 - kernel
2009 May 15 23:01:54 NX5KA %PORT-5-IF_PORT_QUIESCE_FAILED: Interface fc2/16 port quiesce failed due to failure reason: Force Abort Due to Link Failure (NOS/LOS) (0x119)
2009 May 15 23:01:54 NX5KA %PORT-5-IF_DOWN_OLS_RCVD: %$VSAN 133%$ Interface fc2/16 is down (OLS received) san-port-channel 133
2009 May 15 23:01:55 NX5KA May 15 23:01:55 %KERN-3-SYSTEM_MSG: fc2_nsh_tx_frame: FC2 s_id/d_id/vsan error: sid=0xfffffe,did=0x0,vsan=133,rctl:0x23,type:0x1,oxid 0x21f,rxid:0x29e - kernel
2009 May 15 23:02:00 NX5KA %PORT-5-IF_PORT_QUIESCE_FAILED: Interface fc2/15 port quiesce failed due to failure reason: Force Abort Due to Link Failure (NOS/LOS) (0x119)
2009 May 15 23:02:00 NX5KA %PORT-5-IF_DOWN_OLS_RCVD: %$VSAN 133%$ Interface fc2/15 is down (OLS received) san-port-channel 133
2009 May 15 23:02:01 NX5KA May 15 23:02:01 %KERN-3-SYSTEM_MSG: fc2_nsh_tx_frame: FC2 s_id/d_id/vsan error: sid=0xfffffe,did=0x0,vsan=133,rctl:0x23,type:0x1,oxid 0x225,rxid:0x2a4 - kernel
2009 May 15 23:02:04 NX5KA May 15 23:02:04 %KERN-3-SYSTEM_MSG: fc2_nsh_tx_frame: FC2 s_id/d_id/vsan error: sid=0xfffffe,did=0x0,vsan=133,rctl:0x23,type:0x1,oxid 0x229,rxid:0x2aa - kernel
12-20-2013 03:09 PM
Incidentally,
If I turn off or don't create san-port channels, the UCS logs onto the fabric and all the FC interfaces come up.
I'm missing something..
12-20-2013 06:07 PM
Hello vjmendoza,
Your Nexus 5548 FC interfaces should have this configuration:
interface san-port-channel 133
channel mode active
switchport mode f
switchport trunk mode off
interface fc2/15
switchport trunk mode off
switchport mode f
channel-group 133 force
no shutdown
interface fc2/16
switchport trunk mode off
switchport mode f
channel-group 133 force
no shutdown
Richard
12-20-2013 06:48 PM
interface san-port-channel 133
channel mode active
switchport mode F
no switchport trunk allowed vsan all
switchport trunk mode off
version 5.1(3)N1(1)
interface fc2/15
switchport mode F
switchport trunk mode off
channel-group 133 force
no shutdown
interface fc2/16
switchport mode F
switchport trunk mode off
channel-group 133 force
no shutdown
Same results - I downed the san-port-channel on both sides.
NX5KA(config-if)# show san-port-channel summary
U-Up D-Down B-Hot-standby S-Suspended I-Individual link
summary header
--------------------------------------------------------------------------------
Group Port- Type Protocol Member Ports
Channel
--------------------------------------------------------------------------------
133 San-po133 FC PCP (D) FC fc2/15(D) fc2/16(D)
NX5KA(config-if)#
12-20-2013 07:01 PM
Ok, I understand what's going on.
The VSANs created in the UCS should be applied in the SAN port-channel, like this:
(I didn't find a san port-channel configuration image, but is the same configuration type).
In the Nexus you must create those VSANs and leave like this:
Nexus A
#vsan database
#vsan 10
#vsan 10 interface san-port-channel 133
Nexus B
#vsan database
#vsan 20
#vsan 20 interface san-port-channel 133
Richard
12-20-2013 07:50 PM
NX5KA(config-if)# show vsan membership
vsan 1 interfaces:
fc2/13 fc2/14
vsan 133 interfaces:
fc2/15 fc2/16 san-port-channel 133
vsan 4079(evfp_isolated_vsan) interfaces:
vsan 4094(isolated_vsan) interfaces:
NX5KA(config-if)#
vsan database
vsan 133
interface san-port-channel 133
channel mode active
switchport mode F
no switchport trunk allowed vsan all
switchport trunk mode off
vsan database
vsan 133 interface san-port-channel 133
So this is the configuration from the beginning on the Nexus side.
And the screenshot on the UCSM shows that I did attached the port-channel to VSAN133 in UCS
12-20-2013 07:52 PM
For further clarity, here is the show output from the nxos (FI)
UCS-FI-A-A(nxos)# show vsan membership
vsan 1 interfaces:
fc1/29 fc1/30
vsan 100 interfaces:
vfc687
vsan 133 interfaces:
fc1/31 fc1/32 san-port-channel 133
vsan 4079(evfp_isolated_vsan) interfaces:
vsan 4094(isolated_vsan) interfaces:
UCS-FI-A-A(nxos)#
12-20-2013 08:13 PM
Considering you said that it works when disabled the port-channel, I will not ask you if the transceivers and cabling are ok.
This command in the san-port channel isn't necessary:
no switchport trunk allowed vsan all
Please leave this way:
switchport trunk allowed vsan all (this is the deafult behavior)
Try shutdown all interfaces including the physical in both sides, after that, enable in this order:
Nexus FC -> Nexus SAN-PC -> FI FC -> FI SAN-PC.
If still not working, please give me the commands outputs:
NexusA# sh int san-port-channel 133
NexusA# sh int bri
12-20-2013 09:06 PM
!Command: show running-config interface san-port-channel 133
!Time: Sat May 16 04:59:07 2009
version 5.1(3)N1(1)
interface san-port-channel 133
channel mode active
switchport mode F
switchport trunk mode off
Changed it as you suggested...
Followed the order of operations for "no shut"
Nexus FC -> Nexus SAN-PC -> FI FC -> FI SAN-PC.
Didn't work:
NX5KA(config-if)# show san-port-channel summary
U-Up D-Down B-Hot-standby S-Suspended I-Individual link
summary header
--------------------------------------------------------------------------------
Group Port- Type Protocol Member Ports
Channel
--------------------------------------------------------------------------------
133 San-po133 FC PCP (D) FC fc2/15(D) fc2/16(D)
NX5KA(config-if)#
Here is the output as you requested:
NX5KA(config-if)# show int san-port-channel 133
san-port-channel 133 is down (No operational members)
Hardware is Fibre Channel
Port WWN is 24:85:00:2a:6a:5a:81:00
Admin port mode is F, trunk mode is off
snmp link state traps are enabled
Port vsan is 133
1 minute input rate 1256 bits/sec, 157 bytes/sec, 0 frames/sec
1 minute output rate 248 bits/sec, 31 bytes/sec, 0 frames/sec
3966 frames input, 615568 bytes
0 discards, 0 errors
0 CRC, 0 unknown class
0 too long, 0 too short
2956 frames output, 143624 bytes
0 discards, 0 errors
46 input OLS, 41 LRR, 73 NOS, 0 loop inits
257 output OLS, 189 LRR, 219 NOS, 0 loop inits
last clearing of "show interface" counters never
Member[1] : fc2/15
Member[2] : fc2/16
NX5KA(config-if)#
NX5KA(config-if)# show int brief
-------------------------------------------------------------------------------
Interface Vsan Admin Admin Status SFP Oper Oper Port
Mode Trunk Mode Speed Channel
Mode (Gbps)
-------------------------------------------------------------------------------
fc2/13 1 auto on sfpAbsent -- -- --
fc2/14 1 auto on sfpAbsent -- -- --
fc2/15 133 F off init swl -- 133
fc2/16 133 F off init swl -- 133
12-22-2013 10:29 AM
This configuration examples to 5548UP can help you.
About the UCS.
All transceivers have the same speed capability?
The FI are in end-host-mode?
The FI aren't using the "trunk uplink" option in the SAN configuration?
The cables could be cross connected, did you check it?
Try build all configuration again step-by-step calmly, if still not work I only can think some kind of bug in your versions combined.
Richard
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide