N7K-SUP2 occured IO error and comfact flash failed message

[Hongju Jung]

Dear on duty

PID: N7K-SUP2

NX-OS: 6.1.3

I have some log issue with below one

2015 Apr 17 20:57:51 SA_N7K_LTE_PASW4 Apr 17 20:57:50 %KERN-3-SYSTEM_MSG: [46639557.531764] end_request: I/O error, dev sde, sector 408528 - kernel

2015 Apr 17 20:57:51 SA_N7K_LTE_PASW4 Apr 17 20:57:50 %KERN-3-SYSTEM_MSG: [46639557.531814] end_request: I/O error, dev sde, sector 408528 - kernel

2015 Apr 17 20:57:51 SA_N7K_LTE_PASW4 Apr 17 20:57:50 %KERN-3-SYSTEM_MSG: [46639557.601261] end_request: I/O error, dev sde, sector 173456 - kernel

2015 Apr 17 20:57:51 SA_N7K_LTE_PASW4 Apr 17 20:57:50 %KERN-3-SYSTEM_MSG: [46639557.601307] end_request: I/O error, dev sde, sector 173456 - kernel

2015 Apr 17 20:57:51 SA_N7K_LTE_PASW4 Apr 17 20:57:50 %KERN-3-SYSTEM_MSG: [46639557.651248] end_request: I/O error, dev sde, sector 329472 - kernel

2015 Apr 17 20:57:51 SA_N7K_LTE_PASW4 Apr 17 20:57:50 %KERN-3-SYSTEM_MSG: [46639557.651294] end_request: I/O error, dev sde, sector 329472 - kernel

2015 Apr 17 20:57:51 SA_N7K_LTE_PASW4 Apr 17 20:57:50 %KERN-3-SYSTEM_MSG: [46639557.735636] end_request: I/O error, dev sde, sector 4013200 - kernel

2015 Apr 17 20:57:51 SA_N7K_LTE_PASW4 Apr 17 20:57:50 %KERN-3-SYSTEM_MSG: [46639557.735696] end_request: I/O error, dev sde, sector 4013200 - kernel

2015 Apr 18 06:30:55 SA_N7K_LTE_PASW4 %DIAGCLIENT-2-EEM_ACTION_HM_SHUTDOWN: Test <CompactFlash> has been disabled as a part of default EEM action

2015 Apr 18 06:30:55 SA_N7K_LTE_PASW4 %DEVICE_TEST-2-COMPACT_FLASH_FAIL: Module 5 has failed test CompactFlash 20 times on device Compact Flash due to error The compact flash power test failed

2015 Apr 18 06:30:55 SA_N7K_LTE_PASW4 %MODULE-4-MOD_WARNING: Module 5 (Serial number: JAF1729ANPP) reported warning due to The compact flash power test failed in device DEV_UNDEF (device error 0x0)

2015 Apr 18 06:30:56 SA_N7K_LTE_PASW4 %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by admin on vsh.19785

Regarding above meesage have 4 times different chssais occured

Anydoby know about what happen ? that is really H/W concern about compact flash or software bug issue and what is the action plan from now.

I got communicated with TAC engineer that is not verified H/W issue or software issue now, still nexus Hardware team investingated H/W problem

and software team not make sure there is not software issue, now they just reported not occured 6.2.x higer version that's why they are trusted fixed the problem now.

anybody know which code fixed it or someones make sure above issue will softsware issue or not

thanks

[Rafatur Rahman]

Hello,

From the symptom it seems to be software issue.  Fix is available in 6.2(14), 7.2(x), and later. 

Workaround:
While system can operate with only one flash device, its highly recommended to recover and add the removed flash back into RAID configuration. Second flash device can also get into this condition over time triggering the read-only mode.

Flash Recovery Tool:
n7000-s2-flash-recovery-tool.10.0.1.tar.gz is available to be downloaded from Cisco support site. This works as a custom plug-in that can be run using the 'load' CLI.

- To run the tool, download and copy it to bootflash/volatile/slot0 and run the load command.
- Tool automatically fixes any single flash errors when present.
- If a standby available, it will copy itself to standby and run there.
- No side effects if there are no errors reported at the time.
- Tool will not attempt dual flash recovery either on active or standby.

Explanation of the Error:

Each Nexus 7000 Supervisor board are equipped with 2 identical eUSB flash devices in a RAID1 mirror configuration. Called primary and mirror, they together provide for non-volatile repositories for storing boot images, startup configuration and other persistent application data.

Over years in service, one of these devices may get disconnected from the USB bus. This causes the RAID software to drop the affected device to be removed from its configuration. System still can function normally with the remaining working device.

However, if the second device also experiences similar issue and drops out of the RAID array, boot flash devices will re-mounted as read-only preventing configuration copying. Even though there is no operational impact on systems running in this state, a reload of the affected supervisor is needed to recover from this situation. Moreover, the latest running configuration may be lost in the event of a power outage.

HTH.

Regards,

[Hongju Jung]

N7K bootflash failure issue with NX-OS 6.1.3

above issue how to make sure software issue or H/W issue.

if there reload device with bootflash failure part and will be make fix it that is called software problem or something else ?

if there H/W Problem, we can check H/W problem via show system internal command or not. even we can changed the RMA one and collected failure part will check that out later ?

anybody knows about how to exactly which kind issue

thanks.

[francoso]

i cannot issue the 'load' CLI ...

is this an internal command ?

[Madhukrishnan Gopinathan Nair]

If you are using NPE image , load command may not work