Solved: Re: 4 FC ports between Cisco UCS FI-A and MDS-1 are down

gkhnsmsk7 · ‎09-12-2019

Hi,

I am having issues with 4 fibre channel ports on my Cisco UCS and MDS environment.

The connection and configuration on my FI-A is like below:

interface fc2/1
  channel-group 1 force
  no shutdown
interface fc2/2
  channel-group 1 force
  no shutdown
interface fc2/3
  channel-group 2 force
  no shutdown
interface fc2/4
  channel-group 2 force
  no shutdown
interface san-port-channel 1
channel mode active
switchport mode NP
interface san-port-channel 2
channel mode active
switchport mode NP

sh int fc2/1
fc2/1 is down (Error disabled)
Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN)
Port WWN is 20:41:00:2a:6a:0b:94:80
Admin port mode is NP, trunk mode is off
snmp link state traps are enabled
Port vsan is 10
Receive data field Size is 2112
Beacon is turned off
Belongs to san-port-channel 1
1 minute input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
1 minute output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
46 frames input, 4168 bytes
16 discards, 0 errors
0 CRC, 0 unknown class
0 too long, 0 too short
30 frames output, 8880 bytes
0 discards, 0 errors
19 input OLS, 19 LRR, 0 NOS, 0 loop inits
56 output OLS, 6 LRR, 25 NOS, 0 loop inits
last clearing of "show interface" counters never

sh int fc2/2
fc2/2 is down (Error disabled)
Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN)
Port WWN is 20:42:00:2a:6a:0b:94:80
Admin port mode is NP, trunk mode is off
snmp link state traps are enabled
Port vsan is 10
Receive data field Size is 2112
Beacon is turned off
Belongs to san-port-channel 1
1 minute input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
1 minute output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
45 frames input, 3856 bytes
12 discards, 0 errors
0 CRC, 0 unknown class
0 too long, 0 too short
33 frames output, 9768 bytes
0 discards, 0 errors
22 input OLS, 22 LRR, 0 NOS, 0 loop inits
33 output OLS, 0 LRR, 11 NOS, 0 loop inits
last clearing of "show interface" counters never

sh int san-port-channel 1
san-port-channel 1 is down (No operational members)
Hardware is Fibre Channel
Port WWN is 24:01:00:2a:6a:0b:94:80
Admin port mode is NP, trunk mode is off
snmp link state traps are enabled
Port vsan is 10
1 minute input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
1 minute output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
78 frames input, 6848 bytes
24 discards, 0 errors
0 CRC, 0 unknown class
0 too long, 0 too short
54 frames output, 15984 bytes
0 discards, 0 errors
35 input OLS, 35 LRR, 0 NOS, 0 loop inits
79 output OLS, 5 LRR, 32 NOS, 0 loop inits
last clearing of "show interface" counters never
Member[1] : fc2/1
Member[2] : fc2/2


sh int san-port-channel 2
san-port-channel 2 is down (No operational members)
Hardware is Fibre Channel
Port WWN is 24:02:00:2a:6a:0b:94:80
Admin port mode is NP, trunk mode is off
snmp link state traps are enabled
Port vsan is 10
1 minute input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
1 minute output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
19 frames input, 1672 bytes
6 discards, 0 errors
0 CRC, 0 unknown class
0 too long, 0 too short
12 frames output, 3232 bytes
0 discards, 0 errors
6 input OLS, 6 LRR, 0 NOS, 0 loop inits
14 output OLS, 3 LRR, 6 NOS, 0 loop inits
last clearing of "show interface" counters never
Member[1] : fc2/3
Member[2] : fc2/4

The connection and configuration on my MDS devices is like below:

interface fc1/1
  switchport rate-mode dedicated
  switchport description FI-A port1
  switchport trunk mode off
  port-license acquire
  channel-group 1 force
  no shutdown
interface fc1/2
  switchport rate-mode dedicated
  switchport description FI-A port2
  switchport trunk mode off
  port-license acquire
  channel-group 1 force
  no shutdown
interface fc1/3
  switchport rate-mode dedicated
  switchport description FI-A port3
  switchport trunk mode off
  port-license acquire
  channel-group 2 force
  no shutdown
interface fc1/4
  switchport rate-mode dedicated
  switchport description FI-A port4
  switchport trunk mode off
  port-license acquire
  channel-group 2 force
  no shutdown
interface port-channel 1
  channel mode active
  switchport rate-mode dedicated
  switchport trunk mode off
interface port-channel 2
  channel mode active
  switchport rate-mode dedicated
  switchport trunk mode off
sh int fc1/1
fc1/1 is down (Link failure or not-connected)
    Port description is FI-A port1
    Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN)
    Port WWN is 20:01:54:7f:ee:9b:1d:d8
    Admin port mode is auto, trunk mode is off
    snmp link state traps are enabled
    Port vsan is 10
    Receive data field Size is 2112
    Beacon is turned off
    Belongs to port-channel 1
    5 minutes input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
    5 minutes output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
      61 frames input, 17876 bytes
        0 discards, 0 errors
        0 CRC,  0 unknown class
        0 too long, 0 too short
      87 frames output, 7764 bytes
        0 discards, 0 errors
      5343 input OLS, 13 LRR, 654 NOS, 0 loop inits
      589 output OLS, 32 LRR, 5699 NOS, 562 loop inits

sh int fc1/2
fc1/2 is down (Link failure or not-connected)
    Port description is FI-A port2
    Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN)
    Port WWN is 20:02:54:7f:ee:9b:1d:d8
    Admin port mode is auto, trunk mode is off
    snmp link state traps are enabled
    Port vsan is 10
    Receive data field Size is 2112
    Beacon is turned off
    Belongs to port-channel 1
    5 minutes input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
    5 minutes output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
      55 frames input, 16188 bytes
        0 discards, 0 errors
        0 CRC,  0 unknown class
        0 too long, 0 too short
      74 frames output, 6496 bytes
        0 discards, 0 errors
      20 input OLS, 2 LRR, 2 NOS, 0 loop inits
      36 output OLS, 34 LRR, 18 NOS, 18 loop inits

sh int fc1/3
fc1/3 is down (Link failure or not-connected)
    Port description is FI-A port3
    Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN)
    Port WWN is 20:03:54:7f:ee:9b:1d:d8
    Admin port mode is auto, trunk mode is off
    snmp link state traps are enabled
    Port vsan is 10
    Receive data field Size is 2112
    Beacon is turned off
    Belongs to port-channel 2
    5 minutes input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
    5 minutes output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
      22 frames input, 6340 bytes
        0 discards, 0 errors
        0 CRC,  0 unknown class
        0 too long, 0 too short
      30 frames output, 2588 bytes
        0 discards, 0 errors
      7 input OLS, 0 LRR, 0 NOS, 0 loop inits
      14 output OLS, 14 LRR, 7 NOS, 7 loop inits

sh int fc1/4
fc1/4 is down (Link failure or not-connected)
    Port description is FI-A port4
    Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN)
    Port WWN is 20:04:54:7f:ee:9b:1d:d8
    Admin port mode is auto, trunk mode is off
    snmp link state traps are enabled
    Port vsan is 10
    Receive data field Size is 2112
    Beacon is turned off
    Belongs to port-channel 2
    5 minutes input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
    5 minutes output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
      25 frames input, 7300 bytes
        0 discards, 0 errors
        0 CRC,  0 unknown class
        0 too long, 0 too short
      33 frames output, 2980 bytes
        0 discards, 0 errors
      11 input OLS, 3 LRR, 4 NOS, 0 loop inits
      15 output OLS, 12 LRR, 9 NOS, 8 loop inits

sh int port-channel 1
port-channel 1 is down (No operational members)
    Hardware is Fibre Channel
    Port WWN is 24:01:54:7f:ee:9b:1d:d8
    Admin port mode is auto, trunk mode is off
    snmp link state traps are enabled
    Port vsan is 10
    5 minutes input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
    5 minutes output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
      124 frames input, 36104 bytes
        0 discards, 0 errors
        0 CRC,  0 unknown class
        0 too long, 0 too short
      171 frames output, 15220 bytes
        0 discards, 0 errors
      5367 input OLS, 17 LRR, 658 NOS, 0 loop inits
      629 output OLS, 68 LRR, 5719 NOS, 582 loop inits
    Member[1] : fc1/1
    Member[2] : fc1/2

sh int port-channel 2
port-channel 2 is down (No operational members)
    Hardware is Fibre Channel
    Port WWN is 24:02:54:7f:ee:9b:1d:d8
    Admin port mode is auto, trunk mode is off
    snmp link state traps are enabled
    Port vsan is 10
    5 minutes input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
    5 minutes output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
      47 frames input, 13640 bytes
        0 discards, 0 errors
        0 CRC,  0 unknown class
        0 too long, 0 too short
      63 frames output, 5568 bytes
        0 discards, 0 errors
      18 input OLS, 3 LRR, 4 NOS, 0 loop inits
      29 output OLS, 26 LRR, 16 NOS, 15 loop inits
    Member[1] : fc1/3
    Member[2] : fc1/4

When I try to disable/enable the ports on both sides I see the following error messages and the ports are automatically errDisabled on FI-A.

MDS-1
MDS-1 %PORT-5-IF_DOWN_OLS_RCVD: %$VSAN 10%$ Interface fc1/1 is down (OLS received) port-channel 1 FI-A port1
MDS-1 %PORT-5-IF_DOWN_OLS_RCVD: %$VSAN 10%$ Interface fc1/2 is down (OLS received) port-channel 1 FI-A port2
MDS-1 %PORT-5-IF_DOWN_OLS_RCVD: %$VSAN 10%$ Interface fc1/3 is down (OLS received) port-channel 2 FI-A port3
MDS-1 %PORT-5-IF_DOWN_OLS_RCVD: %$VSAN 10%$ Interface fc1/4 is down (OLS received) port-channel 2 FI-A port4

FI-A
FI-A %PORT-2-IF_DOWN_ERROR_DISABLED: %$VSAN 10%$ Interface fc2/1 is down (Error disabled)
FI-A %PORT-2-IF_DOWN_ERROR_DISABLED: %$VSAN 10%$ Interface fc2/2 is down (Error disabled)
FI-A %PORT-2-IF_DOWN_ERROR_DISABLED: %$VSAN 10%$ Interface vfc1478 is down (Error disabled)  server 5/4, VHBA fc1
FI-A %PORT-2-IF_DOWN_ERROR_DISABLED: %$VSAN 10%$ Interface vfc1466 is down (Error disabled)  server 5/5, VHBA fc1
FI-A %PORT-2-IF_DOWN_ERROR_DISABLED: %$VSAN 10%$ Interface vfc1410 is down (Error disabled)  server 5/2, VHBA fc1
.
.
.
.

I hope someone can help me out on this issue.

Thanks.

gkhnsmsk7 · ‎09-12-2019

I could the solve the problem by enabling trunk protocol on my MDS switch.

Before

MDS-1# sh trunk protocol
Trunk Protocol is disabled

After

MDS-1# sh trunk protocol
Trunk Protocol is enabled

Many thanks for your help!

View solution in original post

Kirk J · ‎09-12-2019

What version of MDS and UCSM are you running? Need to make sure you aren't hitting an OUI issue (https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvk63036/?rfs=iqvred )

Also please do a #show int fc x/y transceiver details, for ports on the FIs and the MDS (don't need all of them unless they are different transceiver model numbers)

Kirk...

neilmart · ‎09-12-2019

Hello,

Definitely looks like the bug Kirk mentioned. If you do not verify the traces of the bug, provide the transceiver details as he mentioned. Check fill pattern on FI and the MDS match. Check the speed on each side. See if you are able to bring up the interfaces without using port-channel.

Best Regards,

Neil Martin .ılı.ılı.

Technical Consultant Engineer

Unified Computing Systems and Server Virtualization

SUSE Certified Administrator

In Enterprise Linux 12

Vmware Certified Professional

Data Center Virtualization 2019

gkhnsmsk7 · ‎09-12-2019

Hi,

thanks for your replies. Actually the ports stopped working after a power outage in our "datacenter". We have the same configuration on our FI-B and MDS-2, which are up and running.

Versions:

UCSM: 2.2(8f)

Fabric Interconnect: 5.2(3)N2(2.28f)

MDS: 5.2(8a)

As far as I can see I'm not hitting that bug

MDS-1# show wwn switch
Switch WWN is 20:00:54:7f:ee:9b:1d:d8
MDS-1# sh fcid company-id-from-wwn 20:00:54:7f:ee:9b:1d:d8
Extracted oui: 0x547FEE

Below MDS transceiver outputs. I think I see high Optical Rx Power dbm levels because the port is errDisabled on FI-A.

MDS-1# sh int fc1/1 transceiver
fc1/1 sfp is present
    Name is CISCO-AVAGO
    Manufacturer's part number is SFBR-5780APZ-CS2
    Revision is G2.3
    Serial number is AGD170687JL
    Cisco part number is 10-2418-01
    Cisco pid is DS-SFP-FC8G-SW
    FC Transmitter type is short wave laser w/o OFC (SN)
    FC Transmitter supports short distance link length
    Transmission medium is multimode laser with 62.5 um aperture (M6)
    Supported speeds are - Min speed: 2000 Mb/s, Max speed: 8000 Mb/s
    Nominal bit rate is 8500 Mb/s
    Link length supported for 50/125mm fiber is 50 m
    Link length supported for 62.5/125mm fiber is 20 m
    Cisco extended id is unknown (0x0)

    No tx fault, rx loss, no sync exists, diagnostic monitoring type is 0x68
    SFP Diagnostics Information:
        Temperature           :  33.28 C
        Voltage               :   3.28 V
        Current               :   7.88 mA
        Optical Tx Power      :  -3.53 dBm
        Optical Rx Power      : -25.85 dBm      --
        Tx Fault count  : 0
    Note: ++  high-alarm; +  high-warning; --  low-alarm; -low-warning

MDS-1# sh int fc1/2 transceiver
fc1/2 sfp is present
    Name is CISCO-AVAGO
    Manufacturer's part number is SFBR-5780APZ-CS2
    Revision is G2.3
    Serial number is AGD170681U6
    Cisco part number is 10-2418-01
    Cisco pid is DS-SFP-FC8G-SW
    FC Transmitter type is short wave laser w/o OFC (SN)
    FC Transmitter supports short distance link length
    Transmission medium is multimode laser with 62.5 um aperture (M6)
    Supported speeds are - Min speed: 2000 Mb/s, Max speed: 8000 Mb/s
    Nominal bit rate is 8500 Mb/s
    Link length supported for 50/125mm fiber is 50 m
    Link length supported for 62.5/125mm fiber is 20 m
    Cisco extended id is unknown (0x0)

    No tx fault, rx loss, no sync exists, diagnostic monitoring type is 0x68
    SFP Diagnostics Information:
        Temperature           :  33.72 C
        Voltage               :   3.28 V
        Current               :   6.10 mA
        Optical Tx Power      :  -2.51 dBm
        Optical Rx Power      : -27.96 dBm      --
        Tx Fault count  : 0
    Note: ++  high-alarm; +  high-warning; --  low-alarm; -low-warning

MDS-1# sh int fc1/3 transceiver
fc1/3 sfp is present
    Name is CISCO-AVAGO
    Manufacturer's part number is SFBR-5780APZ-CS2
    Revision is G2.3
    Serial number is AGD170687J3
    Cisco part number is 10-2418-01
    Cisco pid is DS-SFP-FC8G-SW
    FC Transmitter type is short wave laser w/o OFC (SN)
    FC Transmitter supports short distance link length
    Transmission medium is multimode laser with 62.5 um aperture (M6)
    Supported speeds are - Min speed: 2000 Mb/s, Max speed: 8000 Mb/s
    Nominal bit rate is 8500 Mb/s
    Link length supported for 50/125mm fiber is 50 m
    Link length supported for 62.5/125mm fiber is 20 m
    Cisco extended id is unknown (0x0)

    No tx fault, rx loss, no sync exists, diagnostic monitoring type is 0x68
    SFP Diagnostics Information:
        Temperature           :  32.77 C
        Voltage               :   3.28 V
        Current               :   5.59 mA
        Optical Tx Power      :  -2.44 dBm
        Optical Rx Power      : -24.20 dBm      --
        Tx Fault count  : 0
    Note: ++  high-alarm; +  high-warning; --  low-alarm; -low-warning

MDS-1# sh int fc1/4 transceiver
fc1/4 sfp is present
    Name is CISCO-AVAGO
    Manufacturer's part number is SFBR-5780APZ-CS2
    Revision is G2.3
    Serial number is AGD170681UN
    Cisco part number is 10-2418-01
    Cisco pid is DS-SFP-FC8G-SW
    FC Transmitter type is short wave laser w/o OFC (SN)
    FC Transmitter supports short distance link length
    Transmission medium is multimode laser with 62.5 um aperture (M6)
    Supported speeds are - Min speed: 2000 Mb/s, Max speed: 8000 Mb/s
    Nominal bit rate is 8500 Mb/s
    Link length supported for 50/125mm fiber is 50 m
    Link length supported for 62.5/125mm fiber is 20 m
    Cisco extended id is unknown (0x0)

    No tx fault, rx loss, no sync exists, diagnostic monitoring type is 0x68
    SFP Diagnostics Information:
        Temperature           :  33.98 C
        Voltage               :   3.27 V
        Current               :   6.51 mA
        Optical Tx Power      :  -2.44 dBm
        Optical Rx Power      : -26.58 dBm      --
        Tx Fault count  : 0
    Note: ++  high-alarm; +  high-warning; --  low-alarm; -low-warning

FI-A transceiver outputs:

FI-A(nxos)# sh int fc2/1 transceiver
fc2/1 sfp is present
    Name is CISCO-AVAGO
    Manufacturer's part number is SFBR-5780APZ-CS2
    Revision is G2.3
    Serial number is AGA17078XHN
    FC Transmitter type is short wave laser w/o OFC (SN)
    FC Transmitter supports short distance link length
    Transmission medium is multimode laser with 62.5 um aperture (M6)
    Supported speeds are - Min speed: 2000 Mb/s, Max speed: 8000 Mb/s
    Nominal bit rate is 8500 Mb/s
    Link length supported for 50/125um fiber is 50 m
    Link length supported for 62.5/125um fiber is 20 m
    Link length supported for 50/125um OM3 fiber is 150 m
    Cisco extended id is unknown (0x0)

    No tx fault, no rx loss, in sync state, diagnostic monitoring type is 0x68
    Digital diagnostics feature not supported in SFP

FI-A(nxos)# sh int fc2/2 transceiver
fc2/2 sfp is present
    Name is CISCO-AVAGO
    Manufacturer's part number is SFBR-5780APZ-CS2
    Revision is G2.3
    Serial number is AGA170383FF
    FC Transmitter type is short wave laser w/o OFC (SN)
    FC Transmitter supports short distance link length
    Transmission medium is multimode laser with 62.5 um aperture (M6)
    Supported speeds are - Min speed: 2000 Mb/s, Max speed: 8000 Mb/s
    Nominal bit rate is 8500 Mb/s
    Link length supported for 50/125um fiber is 50 m
    Link length supported for 62.5/125um fiber is 20 m
    Link length supported for 50/125um OM3 fiber is 150 m
    Cisco extended id is unknown (0x0)

    No tx fault, no rx loss, in sync state, diagnostic monitoring type is 0x68
    Digital diagnostics feature not supported in SFP

FI-A(nxos)# sh int fc2/3 transceiver
fc2/3 sfp is present
    Name is CISCO-AVAGO
    Manufacturer's part number is SFBR-5780APZ-CS2
    Revision is G2.3
    Serial number is AGA170383F5
    FC Transmitter type is short wave laser w/o OFC (SN)
    FC Transmitter supports short distance link length
    Transmission medium is multimode laser with 62.5 um aperture (M6)
    Supported speeds are - Min speed: 2000 Mb/s, Max speed: 8000 Mb/s
    Nominal bit rate is 8500 Mb/s
    Link length supported for 50/125um fiber is 50 m
    Link length supported for 62.5/125um fiber is 20 m
    Link length supported for 50/125um OM3 fiber is 150 m
    Cisco extended id is unknown (0x0)

    No tx fault, no rx loss, no sync exists, diagnostic monitoring type is 0x68
    Digital diagnostics feature not supported in SFP

FI-A(nxos)# sh int fc2/4 transceiver
fc2/4 sfp is present
    Name is CISCO-AVAGO
    Manufacturer's part number is SFBR-5780APZ-CS2
    Revision is G2.3
    Serial number is AGA170383GH
    FC Transmitter type is short wave laser w/o OFC (SN)
    FC Transmitter supports short distance link length
    Transmission medium is multimode laser with 62.5 um aperture (M6)
    Supported speeds are - Min speed: 2000 Mb/s, Max speed: 8000 Mb/s
    Nominal bit rate is 8500 Mb/s
    Link length supported for 50/125um fiber is 50 m
    Link length supported for 62.5/125um fiber is 20 m
    Link length supported for 50/125um OM3 fiber is 150 m
    Cisco extended id is unknown (0x0)

    No tx fault, no rx loss, in sync state, diagnostic monitoring type is 0x68
    Digital diagnostics feature not supported in SFP

I didn't try the ports without port-channel yet, going to give it a try.

gkhnsmsk7 · ‎09-12-2019

I could the solve the problem by enabling trunk protocol on my MDS switch.

Before

MDS-1# sh trunk protocol
Trunk Protocol is disabled

After

MDS-1# sh trunk protocol
Trunk Protocol is enabled

Many thanks for your help!