Faulty fabric module - N9K-C9504-FM-E

[tuanquangnguyen]

Got a fabric module that refused to boot up on a new switch. Upon inspecting the log I've come across this.

2020 Mar 26 17:14:17 N9K-C9504 %$ VDC-1 %$ %MODULE-2-MOD_DIAG_FAIL: Module 22 (Serial number: ) reported failure due to fatal error in device DEV_TAHOE (device error 0xc0c00200)

2020 Mar 26 17:14:25 N9K-C9504 %$ VDC-1 %$ %USBHSD-STANDBY-2-MOUNT: logflash: online

2020 Mar 26 17:15:42 N9K-C9504 %$ VDC-1 %$ %PLATFORM-2-MOD_DETECT: Module 22 detected (Serial number ) Module-Type 4-slot Fabric Module Model N9K-C9504-FM-E

2020 Mar 26 17:15:42 N9K-C9504 %$ VDC-1 %$ %PLATFORM-2-MOD_PWRUP: Module 22 powered up (Serial number )

2020 Mar 26 17:15:53 N9K-C9504 %$ VDC-1 %$ %MODULE-2-MOD_DIAG_FAIL: Module 22 (Serial number: ) reported failure due to fatal error in device DEV_TAHOE (device error 0xc0c00200)

2020 Mar 26 17:17:33 N9K-C9504 %$ VDC-1 %$ %MODULE-2-MOD_FAIL: Initialization of module 22 (Serial number: ) failed

2020 Mar 26 17:17:33 N9K-C9504 %$ VDC-1 %$ %PLATFORM-2-MOD_PWRDN: Module 22 powered down (Serial number )

Show module also indicates the module is powered down. Have tried reloading the module, but apparently I'm not allowed to do so as it wasn't even booted.

Mod Ports Module-Type Model Status

--- ----- ------------------------------------- --------------------- ---------

22 0 4-slot Fabric Module N9K-C9504-FM-E powered-dn

The switch is running on NX-OS 7.0(3)I7(7) which is the recommended long-lived release.

Has anyone seen this kind of error before? Apparently we would open a TAC case but would need to wait until the next business day. Just want to get a head start so I asked here first.

[Sergiu.Daniluk]

Hi,

The error code points to a hardware error:

N9K# show system error-id 0xc0c00200

Error_id: 0xC0C00200
Error Facility: 
Error Description: Device Name:[lacrosse] Instance:[0] Error Type:[hw error] code:[0]

But at the same time might be a software issue.

Have you tried re-seating the FM? Or try swapping the bad FM with a good FM (from the same chassis) and see if the error follows the FM or is the slot which is problematic.

If the problem follows the FM, then is a FM issue.

If the known good FM starts reloading, then is the slot.

Cheers,

Sergiu

[tuanquangnguyen]

Hi,

I've tried reseating as soon as we found out the problem. The FM was at fault since when we replaced it with one in another slot (e.g. switching slot 4 FM to slot 2 and vice versa), then the switch started throwing error on slot 4.

I was just not sure if it was due to hardware or software. Since it's a new purchase, I couldn't open a case with TAC until the starting date of warranty though, so probably no RMA nor TAC until then.(?) Is there any other viable action should I take before then?

[Sergiu.Daniluk]

Hi,

Just to make sure I understood correctly: when you received the new FM, you inserted it in slot 4 first, and it was working ok. After that you swapped it with the FM in slot 2, and back to slot 4. Only after that the errors started to be generated? If my understanding is not correct, then please clarify.

If the device is DOA (dead on arrival) you can still open the case, but since the product is new, they will generate a manufacturing RMA, meaning that the replacement product will be shipped to you in 30 days as it is a brand new manufactured product.

Regards,

Sergiu

[tuanquangnguyen]

Hi,

Initially, the (faulty) FM was in slot 2 and the issue was already raised (for slot 2). Then, I tried reseating it to slot 4, then the error was raised on slot 4.

[Sergiu.Daniluk · Accepted solution]

That is an indication that there is a h/w issue with the FM. You should call Cisco Frontline and ask for a DOA RMA.

Make sure you mention that this is a new FM. Also share the steps you have performed to troubleshoot this issue and the relevant logs (show inventory, show module, show logging log).

Regards,

Sergiu