MDS9140 port status changed to "Hardware Failure"

[msotomaldonado]

I am connecting a McData Eclipse 1620 to a Cisco MDS 9140, at the moment I connect the McData to a FC port in the MDS switch, The MDS port status changes to "Hardware Failure", once the port is in this state it is not possible to shut it down nor do anything with it, this is making troubleshooting very difficult. Is there a way to clear this "Hardware Failure" from the port without having to reboot the MDS switch?

--------------------------------

FDSW4# sh int fc1/2

fc1/2 is down (Hardware failure)

--------------------------------

For those interested in the MDS log:

2007 Jan 5 12:48:19 FDSW4 %MODULE-2 MOD_DIAG_FAIL: Module 1 reported failure on ports 1/2-1/2 (Fibre Channel) due to FC MAC Experienced an error in device 3 (device error 0x22)

2007 Jan 5 12:48:19 FDSW4 %MODULE-2-MOD_SOMEPORTS_FAILED: Module 1 reported failure on ports 1/2-1/2 (Fibre Channel) due to FC MAC Experienced an error in device 3 (error 0x22)

2007 Jan 5 12:48:19 FDSW4 %PORT-5-IF_DOWN_HW_FAILURE: %$VSAN 167%$ Interface fc1/2 is down (Hardware Failure)

[colin.mcnamara]

You might want to check out the following documentation -

http://www.cisco.com/en/US/products/ps5989/prod_troubleshooting_guide_chapter09186a00805a735d.html

It goes into some depth regarding port troubleshooting, including the FC MAC system.

--Colin

[msotomaldonado]

Colin,

Thanks for the link, I checked it out, but it seems the "Hardware failure" message is a dead end, most of the diags (including the external loopback test) fail.

I borrowed a MDS9216 from Cisco for testing, but I cannot recreate the problem, I never get the "hardware failure" problem, it would seem the problem is solely related to the MDS 9100 series.

Cheers!

[inch]

Howdy,

Have you got interop mode turned on?

Cheers

Andrew

[stephen2615]

This is probably too simplistic but I had a port that randomly shut itself down. I could enable the port again so this probably is of no use to you but the alarms were very similiar. When it was down, I swapped the "dead" SFP with another SFP and I was able to bring it up again. I also believe the switch was cooking itself as it was above a bunch of HP blade enclosures.

Am I right in assuming that your problem seems to have happened when you plugged the other switch into the MDS?

Stephen

[msotomaldonado]

Stephen,

Yes, the problem only occurs when I plug the McData box, the MDS ports goes into the "initializing" phase and then changes to "Hardware Failure", after this the port is useless until you reboot the MDS.

Cheers!

[msotomaldonado]

Hi Andrew,

Yes, I've tried with interop 1 and 4 (I upgraded the SANOS to 3.0(3)), and the result is the same.

Cheers!

[inch]

If you are keen....

Ask your cisco SE if you can loan a PAA-1 or PAA-2 and span ingres/egress of the port in question to a span port of the PAA.

The PAA with then convert the FC frames into ethernet frames and with a laptop and ethereal you will be able to do a full trace of what the hell is going on :-)

If your cisco SE cant track one down for you and depending on where you are located i might be able to help you out....

I have used the PAA's quite a lot and you really cant beat them for low level debugging :)

Cheers

Andrew

[msotomaldonado]

At this point I think the "Hardware failure" problem is beyond an on-the-field solution as it looks more like a hardware issue rather than software, I will ask Cisco to open a TAC case.

Cheers!

M