Troubleshooting CS-MARS fsck issues

[athukral]

Introduction

This document describes the solution for a MARS appliance that is stuck and not responding due to Disc issues.

Prerequisites

HW: CS-MARS

SW: 6.0

Symptoms

On rebooting you will see the following  message on console and boot process will stop—

Automatic reboot in progress.

Restarting system.

LILO

Loading linux.............

/: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY. (i.e., without -a or -p options)

[/sbin/fsck.ext2 -- /] fsck.ext2 -a /dev/sda1   [FAILED]

*** An error occurred during the file system check.

*** Dropping you to a shell; the system will reboot

*** when you leave the shell.

Give root password for maintenance

(or type Control-D for normal startup):

Solution

In order to troubleshoot  please perform the following steps—

Please re-image the box using a recovery boot DVD (in iso format), which can be found at the following link:

http://www.cisco.com/cgi-bin/tablebuild.pl/cs-mars-recovery

Hope this procedure will be helpful. Thanks for viewing.

[Siddharth Chandrachud]

Please be aware of the following enhancement bug I filed for this issue: CSCth73356

Symptom:

Mars under certain data corruption conditions might give out the following error:

/dev/sda5: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY,
(i.e., without -a or -p options)
/dev/sda6: clean, ......files, .......blocks
/dev/sda7: clean, ...................
/dev/sda8........................
/dev/.......
/dev/sda10:clean.........
[FAILED]

****An Error occurred during the file system check.
****Dropping you to a shell: the system will reboot
****when you leave the shell.
Give root password for maintenance

1)  Running fsck manually may corrupt the file system. CS-MARS will  automatically run fsck to fix some inconsistency issues during bootup.  Only under extreme conditions does it require a manual fsck.   Manual  fsck requires user intervention (sometimes hundreds of times) and  intimate understanding of file system internal design.  We are not  trying to repair a linux system here, we are trying to repair a software  appliance.  In the vast majority of cases, the corruption is deep  enough that even if the linux filesystem is repaired, the overlying MARS  software (DB, binaries, etc.) is usually damaged beyond repair,  requiring a clean install.

2)  Per Cisco policy and security reasons we don't give out the root password.

3)   Hence this option should be replaced with recommended procedure of  trying OS recovery (in case of Mars Gen2) and if that fails proceed to  re-imaging the Mars appliance.

Conditions:
Data/File system corruption.

Workaround:
OS recovery (in case of Mars Gen2) and if that fails proceed to re-imaging the Mars appliance