MDS 9148 NX-OS 5.0(1a) Link failure Problem

Fernando Bello · ‎09-05-2012

Hi all,

I'm finding the following problem on my 2 MDS9148 switches.

2012 May  7 14:15:54 SAN01 %PORT-4-IF_SFP_WARNING: Interface fc1/7, Low Rx Power Warning  BK01
2012 May  7 14:15:54 SAN01 %PORT-3-IF_SFP_7ALARM: Interface fc1/7, Low Rx Power Alarm  BK01

And finally the FC interfaces go down. We checked the fiber cable to be sure that it is not a physical issue and the fiber cables are ok.

This happens with fc1/1, 1/3, 1/4 and several more.

Does anyone have an idea why this could be happening and what the error might be?

Your help will be much appreciated.

Thanks in advance,

Kind regards

krun_shah · ‎09-05-2012

Are those ISL ports? What is connected to those ports?

Collect following and post it here

sh int fc1/7 counters details

sh int fc 1/7 transceiver details

Fernando Bello · ‎09-05-2012

Krun_shah,

I am posting the fc1/1 and fc1/5 instead of fc1/7. All of them are experiencing the same
problem.

To those ports is connected one of the drives of a backup robot.

THe interfaces I believe they are not ISL. (You mean ISL as a trunking method, right?)
Here it is the config of each interface.

interface fc1/1

switchport description LTO3
port-license acquire
no shutdown

interface fc1/5

switchport description LTO3

port-license acquire

no shutdown

Here are the details for fc1/1 and fc1/5 that are experiencing the same problem as fc1/7.

#sh int fc1/1 counters details fc1/1 4417757720 frames, 5859624695228 bytes received 0 class-2 frames, 0 bytes received 0 class-2 discards 0 F_BSY frames, 0 F_RJT frames generated against class-2 frames 0 port reject frames 4417757720 class-3 frames, 5859624695228 bytes received 0 class-f frames, 0 bytes received 0 discards, 0 errors received 0 discards, 117 errors transmitted 100604456388 frames, 207142346384892 bytes transmitted 0 class-2 frames, 0 bytes transmitted 100604456388 class-3 frames, 207142346384892 bytes transmitted 0 class-3 frames discarded 0 class-f frames, 0 bytes transmitted 0 class-f frames discarded 0 multicast packets received, 0 transmitted 0 broadcast packets received, 0 transmitted 4417756190 unicast packets received, 100611090558 transmitted 85 link failures, 82 sync losses, 77 signal losses 0 primitive sequence protocol errors 163601 invalid transmission words 0 invalid CRCs, 0 Delimiter Errors 0 address identifier errors 0 link reset protocol errors received 0 link reset protocol errors transmitted 0 Offline Sequence errors received 173 Offline Sequence errors transmitted 0 frames received that are shorter than the minimum allowable frame length regardless of the CRC error 0 frames received that are longer than the maximum frame length and also have a CRC error 0 waits due to lack of transmit credits 0 frames received with length greater than what was agreed to in FLOGI/PLOGI 0 frames received with length less than the minimum indicated by the frame header 0 link reset responses received 0 link reset responses transmitted 0 non-operational sequences received 88 non-operational sequences transmitted 0 fragmented frames received 0 frames received with EOF aborts 0 unknown class frames received 0 8b10b disparity errors 117 frames discarded 0 Exchange Link Parameters switch fabric internal link service request failures 110542953 transitions of BB credit out of zero state 0 transitions of BB credit to zero state 0 Enhanced Inter Switch Link (EISL) frames discarded 0 framing errors 0 F8 type LIP sequence errors received 0 F8 type LIP sequence errors issued 193 Non F8 type LIP sequence errors received 173 Non F8 type LIP sequence errors issued # sh int fc 1/1 transceiver details fc1/1 sfp is present Name is CISCO-AVAGO Manufacturer's part number is SFBR-5780APZ-CS2 Revision is G2.3 Serial number is AGD143284VM FC Transmitter type is short wave laser w/o OFC (SN) FC Transmitter supports short distance link length Transmission medium is multimode laser with 62.5 um aperture (M6) Supported speeds are - Min speed: 2000 Mb/s, Max speed: 8000 Mb/s Nominal bit rate is 8500 Mb/s Link length supported for 50/125mm fiber is 50 m Link length supported for 62.5/125mm fiber is 20 m Cisco extended id is unknown (0x0) No tx fault, no rx loss, in sync state, diagnostic monitoring type is 0x68 SFP Diagnostics Information: ---------------------------------------------------------------------------- Alarms Warnings High Low High Low ---------------------------------------------------------------------------- Temperature 34.66 C 75.00 C -5.00 C 70.00 C 0.00 C Voltage 3.27 V 3.63 V 2.97 V 3.46 V 3.13 V Current 7.09 mA 8.50 mA 2.00 mA 8.50 mA 2.00 mA Tx Power -2.91 dBm 1.70 dBm -14.00 dBm -1.30 dBm -10.00 dBm Rx Power -6.76 dBm 3.00 dBm -17.30 dBm 0.00 dBm -13.30 dBm Transmit Fault Count = 0 ---------------------------------------------------------------------------- Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning # sh int fc1/5 counters details fc1/5 4486944237 frames, 5759105445416 bytes received 0 class-2 frames, 0 bytes received 0 class-2 discards 0 F_BSY frames, 0 F_RJT frames generated against class-2 frames 0 port reject frames 4486944237 class-3 frames, 5759105445416 bytes received 0 class-f frames, 0 bytes received 0 discards, 0 errors received 0 discards, 2 errors transmitted 108664953452 frames, 223758725747704 bytes transmitted 0 class-2 frames, 0 bytes transmitted 108664953452 class-3 frames, 223758725747704 bytes transmitted 0 class-3 frames discarded 0 class-f frames, 0 bytes transmitted 0 class-f frames discarded 0 multicast packets received, 0 transmitted 0 broadcast packets received, 0 transmitted 4486942834 unicast packets received, 108671587794 transmitted 63 link failures, 62 sync losses, 6 signal losses 0 primitive sequence protocol errors 124 invalid transmission words 0 invalid CRCs, 0 Delimiter Errors 0 address identifier errors 0 link reset protocol errors received 0 link reset protocol errors transmitted 0 Offline Sequence errors received 129 Offline Sequence errors transmitted 0 frames received that are shorter than the minimum allowable frame length regardless of the CRC error 0 frames received that are longer than the maximum frame length and also have a CRC error 0 waits due to lack of transmit credits 0 frames received with length greater than what was agreed to in FLOGI/PLOGI 0 frames received with length less than the minimum indicated by the frame header 0 link reset responses received 0 link reset responses transmitted 0 non-operational sequences received 66 non-operational sequences transmitted 0 fragmented frames received 0 frames received with EOF aborts 0 unknown class frames received 0 8b10b disparity errors 2 frames discarded 0 Exchange Link Parameters switch fabric internal link service request failures 294071266 transitions of BB credit out of zero state 0 transitions of BB credit to zero state 0 Enhanced Inter Switch Link (EISL) frames discarded 0 framing errors 0 F8 type LIP sequence errors received 0 F8 type LIP sequence errors issued 134 Non F8 type LIP sequence errors received 129 Non F8 type LIP sequence errors issued # sh int fc 1/5 transceiver details fc1/5 sfp is present Name is CISCO-AVAGO Manufacturer's part number is SFBR-5780APZ-CS2 Revision is G2.3 Serial number is AGD143284VS FC Transmitter type is short wave laser w/o OFC (SN) FC Transmitter supports short distance link length Transmission medium is multimode laser with 62.5 um aperture (M6) Supported speeds are - Min speed: 2000 Mb/s, Max speed: 8000 Mb/s Nominal bit rate is 8500 Mb/s Link length supported for 50/125mm fiber is 50 m Link length supported for 62.5/125mm fiber is 20 m Cisco extended id is unknown (0x0) No tx fault, no rx loss, in sync state, diagnostic monitoring type is 0x68 SFP Diagnostics Information: ---------------------------------------------------------------------------- Alarms Warnings High Low High Low ---------------------------------------------------------------------------- Temperature 36.53 C 75.00 C -5.00 C 70.00 C 0.00 C Voltage 3.26 V 3.63 V 2.97 V 3.46 V 3.13 V Current 7.08 mA 8.50 mA 2.00 mA 8.50 mA 2.00 mA Tx Power -2.72 dBm 1.70 dBm -14.00 dBm -1.30 dBm -10.00 dBm Rx Power -3.94 dBm 3.00 dBm -17.30 dBm 0.00 dBm -13.30 dBm Transmit Fault Count = 0 ---------------------------------------------------------------------------- Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning

Fernando Bello · ‎09-05-2012

Krun_shah

Seems that the other post has not a good layout.

Here it is: THe error:

2012 Sep 4 09:34:43 SAN-01 %PORT-2-IF_DOWN_LINK_FAILURE: %$VSAN 2%$ Interface fc1/1 is down (Link failure) LTO3

2012 Sep 4 09:34:44 SAN-01 %PORT-2-IF_DOWN_LINK_FAILURE: %$VSAN 2%$ Interface fc1/1 is down (Link failure) LTO3

2012 Sep 4 09:34:44 SAN-01 %PORT-2-IF_DOWN_LINK_FAILURE: %$VSAN 2%$ Interface fc1/5 is down (Link failure) LTO3

2012 Sep 4 09:35:31 SAN-01 %PORT-2-IF_DOWN_LINK_FAILURE: %$VSAN 2%$ Interface fc1/5 is down (Link failure) LTO3

2012 Sep 4 09:35:31 SAN-01 %PORT-5-IF_UP: %$VSAN 2%$ Interface fc1/1 is up in mode FL LTO3

2012 Sep 4 09:35:33 SAN-01 %PORT-5-IF_UP: %$VSAN 2%$ Interface fc1/5 is up in mode FL LTO3

2012 Sep 4 09:36:57 SAN-01 %PORT-4-IF_SFP_WARNING: Interface fc1/1, Low Rx Power Warning LTO3

2012 Sep 4 09:36:57 SAN-01 %PORT-4-IF_SFP_WARNING: Interface fc1/5, Low Rx Power Warning LTO3

2012 Sep 4 09:36:57 SAN-01 %PORT-3-IF_SFP_ALARM: Interface fc1/5, Low Rx Power Alarm LTO3

2012 Sep 4 09:46:58 SAN-01 %PORT-4-IF_SFP_WARNING: Interface fc1/1, Low Rx Power Warning cleared LTO3

2012 Sep 4 09:46:58 SAN-01 %PORT-3-IF_SFP_ALARM: Interface fc1/1, Low Rx Power Alarm cleared LTO3

2012 Sep 4 09:46:58 SAN-01 %PORT-4-IF_SFP_WARNING: Interface fc1/5, Low Rx Power Warning cleared LTO3

2012 Sep 4 09:46:58 SAN-01 %PORT-3-IF_SFP_ALARM: Interface fc1/5, Low Rx Power Alarm cleared

----

The fc1/1 and fc1/5 ports are experiencing the same problem as fc1/7 so I am pasting the show command output for those ports.

The fc1/1 and fc1/5 are not ISL ports I believe. You mean ISL as the trunking protocol, right?

Here it is the config for both ports.

interface fc1/1

switchport description LTO3

port-license acquire

no shutdown

interface fc1/5

switchport description LTO3

port-license acquire

no shutdown

WE have connected to those ports a drive that is used by a backup robot.

Here are the ouput for the show commands you asked.

sh int fc1/1 counters details

fc1/1

4417757720 frames, 5859624695228 bytes received

0 class-2 frames, 0 bytes received

0 class-2 discards

0 F_BSY frames, 0 F_RJT frames

generated against class-2 frames

0 port reject frames

4417757720 class-3 frames, 5859624695228 bytes received

0 class-f frames, 0 bytes received

0 discards, 0 errors received

0 discards, 117 errors transmitted

100604456388 frames, 207142346384892 bytes transmitted

0 class-2 frames, 0 bytes transmitted

100604456388 class-3 frames, 207142346384892 bytes transmitted

0 class-3 frames discarded

0 class-f frames, 0 bytes transmitted

0 class-f frames discarded

0 multicast packets received, 0 transmitted

0 broadcast packets received, 0 transmitted

4417756190 unicast packets received, 100611090558 transmitted

85 link failures, 82 sync losses, 77 signal losses

0 primitive sequence protocol errors

163601 invalid transmission words

0 invalid CRCs, 0 Delimiter Errors

0 address identifier errors

0 link reset protocol errors received

0 link reset protocol errors transmitted

0 Offline Sequence errors received

173 Offline Sequence errors transmitted

0 frames received that are shorter than

the minimum allowable frame length

regardless of the CRC error

0 frames received that are longer than

the maximum frame length and also have a

CRC error

0 waits due to lack of transmit credits

0 frames received with length greater

than what was agreed to in FLOGI/PLOGI

0 frames received with length less than

the minimum indicated by the frame header

0 link reset responses received

0 link reset responses transmitted

0 non-operational sequences received

88 non-operational sequences transmitted

0 fragmented frames received

0 frames received with EOF aborts

0 unknown class frames received

0 8b10b disparity errors

117 frames discarded

0 Exchange Link Parameters switch fabric

internal link service request failures

110542953 transitions of BB credit out of zero state

0 transitions of BB credit to zero state

0 Enhanced Inter Switch Link (EISL) frames

discarded

0 framing errors

0 F8 type LIP sequence errors received

0 F8 type LIP sequence errors issued

193 Non F8 type LIP sequence errors received

173 Non F8 type LIP sequence errors issued

SAN-01# sh int fc 1/1 transceiver details

fc1/1 sfp is present

Name is CISCO-AVAGO

Manufacturer's part number is SFBR-5780APZ-CS2

Revision is G2.3

Serial number is AGD143284VM

FC Transmitter type is short wave laser w/o OFC (SN)

FC Transmitter supports short distance link length

Transmission medium is multimode laser with 62.5 um aperture (M6)

Supported speeds are - Min speed: 2000 Mb/s, Max speed: 8000 Mb/s

Nominal bit rate is 8500 Mb/s

Link length supported for 50/125mm fiber is 50 m

Link length supported for 62.5/125mm fiber is 20 m

Cisco extended id is unknown (0x0)

No tx fault, no rx loss, in sync state, diagnostic monitoring type is 0x68

SFP Diagnostics Information:

----------------------------------------------------------------------------

Alarms Warnings

High Low High Low

----------------------------------------------------------------------------

Temperature 34.66 C 75.00 C -5.00 C 70.00 C 0.00 C

Voltage 3.27 V 3.63 V 2.97 V 3.46 V 3.13 V

Current 7.09 mA 8.50 mA 2.00 mA 8.50 mA 2.00 mA

Tx Power -2.91 dBm 1.70 dBm -14.00 dBm -1.30 dBm -10.00 dBm

Rx Power -6.76 dBm 3.00 dBm -17.30 dBm 0.00 dBm -13.30 dBm

Transmit Fault Count = 0

----------------------------------------------------------------------------

Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning

SAN-01# sh int fc1/5 counters details

fc1/5

4486944237 frames, 5759105445416 bytes received

0 class-2 frames, 0 bytes received

0 class-2 discards

0 F_BSY frames, 0 F_RJT frames

generated against class-2 frames

0 port reject frames

4486944237 class-3 frames, 5759105445416 bytes received

0 class-f frames, 0 bytes received

0 discards, 0 errors received

0 discards, 2 errors transmitted

108664953452 frames, 223758725747704 bytes transmitted

0 class-2 frames, 0 bytes transmitted

108664953452 class-3 frames, 223758725747704 bytes transmitted

0 class-3 frames discarded

0 class-f frames, 0 bytes transmitted

0 class-f frames discarded

0 multicast packets received, 0 transmitted

0 broadcast packets received, 0 transmitted

4486942834 unicast packets received, 108671587794 transmitted

63 link failures, 62 sync losses, 6 signal losses

0 primitive sequence protocol errors

124 invalid transmission words

0 invalid CRCs, 0 Delimiter Errors

0 address identifier errors

0 link reset protocol errors received

0 link reset protocol errors transmitted

0 Offline Sequence errors received

129 Offline Sequence errors transmitted

0 frames received that are shorter than

the minimum allowable frame length

regardless of the CRC error

0 frames received that are longer than

the maximum frame length and also have a

CRC error

0 waits due to lack of transmit credits

0 frames received with length greater

than what was agreed to in FLOGI/PLOGI

0 frames received with length less than

the minimum indicated by the frame header

0 link reset responses received

0 link reset responses transmitted

0 non-operational sequences received

66 non-operational sequences transmitted

0 fragmented frames received

0 frames received with EOF aborts

0 unknown class frames received

0 8b10b disparity errors

2 frames discarded

0 Exchange Link Parameters switch fabric

internal link service request failures

294071266 transitions of BB credit out of zero state

0 transitions of BB credit to zero state

0 Enhanced Inter Switch Link (EISL) frames

discarded

0 framing errors

0 F8 type LIP sequence errors received

0 F8 type LIP sequence errors issued

134 Non F8 type LIP sequence errors received

129 Non F8 type LIP sequence errors issued

SAN-01# sh int fc 1/5 transceiver details

fc1/5 sfp is present

Name is CISCO-AVAGO

Manufacturer's part number is SFBR-5780APZ-CS2

Revision is G2.3

Serial number is AGD143284VS

FC Transmitter type is short wave laser w/o OFC (SN)

FC Transmitter supports short distance link length

Transmission medium is multimode laser with 62.5 um aperture (M6)

Supported speeds are - Min speed: 2000 Mb/s, Max speed: 8000 Mb/s

Nominal bit rate is 8500 Mb/s

Link length supported for 50/125mm fiber is 50 m

Link length supported for 62.5/125mm fiber is 20 m

Cisco extended id is unknown (0x0)

No tx fault, no rx loss, in sync state, diagnostic monitoring type is 0x68

SFP Diagnostics Information:

----------------------------------------------------------------------------

Alarms Warnings

High Low High Low

----------------------------------------------------------------------------

Temperature 36.53 C 75.00 C -5.00 C 70.00 C 0.00 C

Voltage 3.26 V 3.63 V 2.97 V 3.46 V 3.13 V

Current 7.08 mA 8.50 mA 2.00 mA 8.50 mA 2.00 mA

Tx Power -2.72 dBm 1.70 dBm -14.00 dBm -1.30 dBm -10.00 dBm

Rx Power -3.94 dBm 3.00 dBm -17.30 dBm 0.00 dBm -13.30 dBm

Transmit Fault Count = 0

----------------------------------------------------------------------------

Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning

thanks in advance,

Fernando

dynamoxxx · ‎09-06-2012

do you have other tape drives that are connected to this switch and are working correctly ? If yes, can you move them to these ports and move the other drives to "working" ports and see if the problem follows those specific tape drives.

@dynamoxxx

Fernando Bello · ‎09-06-2012

Hi dynamoxxx,

From port fc1/1 to f1/16 we have connected different things, servers, drives and other devices. And all ports are experiencing the same problem.

Thanks!

Fernando

dynamoxxx · ‎09-07-2012

i would not think that 6 SFPs would go bad but have you tried to swap those from other know "working" ports ? Also are these direct connects or you are going through a patch panel of some sort ?

@dynamoxxx

Fernando Bello · ‎09-12-2012

Hi Dynamoxx,

Sorry for the late response, but I was sure that I have answered the question you asked previously.

Yes, fibers go through a patch panel. Above, I pasted the output of "show fc1/1 transceiver detail" and the output I got was the following:

SAN-01# sh int fc 1/5 transceiver details

fc1/5 sfp is present

Name is CISCO-AVAGO

Manufacturer's part number is SFBR-5780APZ-CS2

Revision is G2.3

Serial number is AGD143284VS

FC Transmitter type is short wave laser w/o OFC (SN)

FC Transmitter supports short distance link length

Transmission medium is multimode laser with 62.5 um aperture (M6)*********************************************

Supported speeds are - Min speed: 2000 Mb/s, Max speed: 8000 Mb/s

Nominal bit rate is 8500 Mb/s******************************************************************************************

Link length supported for 50/125mm fiber is 50 m

Link length supported for 62.5/125mm fiber is 20 m

Cisco extended id is unknown (0x0)

As you can see, it says that the transmission medium is multimode laser with 62.5 um and the nominal rate is 8500 Mb/s.

Is it possible that the transceiver is able to detect what type of fiber is it connected? Is that a default value? May I change that value from the command line?

Besides, the nominal rate, what does exaclty mean and is it possible to modify it?

the GBIC model is DS-SFP-FC8G-SW

thanks in advance for your help,

Fernando

robertoharamita · ‎09-26-2012

Fernando,

I'm having the same problem, did you solved your?

Thanks in advance,

Roberto

Fernando Bello · ‎09-26-2012

Roberto,

No, some tests still need to be done, but unfortunately, the problem still exists. If I find the solution, I will let you know ASAP.

thanks,

Fernando

robertoharamita · ‎09-26-2012

Fernando,

Thank you for your prompt reply. I have one more question, have you tried to upgrade the firmware to the 5.2 version?

Regards,

Roberto

Filiph Westman · ‎10-09-2012

This is a known software problem (no functional impact). The alarm displays a few minutes after the interface comes up, then clears about 10 minutes later. Solved in NX-OS 4.2(7e) or 5.0.7 and later (CSCtk10527)...

--filiph

Fernando Bello · ‎11-07-2012

Hi Filiph,

thanks for your response. I will check if the upgrade is possible and then let you know the results.

Thanks!

Fernando

Filiph Westman · ‎11-12-2012

Just to clarify....

you still need to investigate why ports go offline with LINK FAILURE as the cause (check logs on the LTO device and the physical connection).

The following "Low Rx Power Warning" is a cosmetic issue that happens when link goes online again.

--filiph

raghu.grandhi · ‎06-10-2013

Hi All,

I'm having the same problem currently using NX-OS Version 5.1(3)N1(1a) why this is updated version but same problem any one help.

2013 Jun 9 14:05:50 NXS01 %PORT-4-IF_SFP_WARNING: Interface fc2/4, Low Rx Power Warning

2013 Jun 9 14:05:50 NXS01 %PORT-3-IF_SFP_ALARM: Interface fc2/4, Low Rx Power Alarm