- Summary
- Solution
- Symptoms
- How to determine the current version of firmware on the hard drive
- What should be done if filesystem issues persist after applying the patches?
- Related Defects
- Related Links
Summary
Cisco Media Convergence Servers 7816-I4, 7825-I4 (and IBM x3250-M2 equivalent) and 7828-I4 have recently been experiencing technical issues. These servers are used by Cisco Unified Communications Manager and various other Cisco Collaboration software products.
The symptom is that the local disk drives' file-system goes into read-only mode, which can manifest as application services going down, the server becoming unresponsive via the network or the management interfaces, or worst case data corruption necessitating a reinstall and restore from backup.
Root cause has been identified by Cisco and its suppliers as a disk drive issue stemming from interaction with system firmware.
Field Notice 63374 has been published and includes more technical details regarding this issue. Cisco and its suppliers are committed to high quality and apologize for any disruptions or impact caused by this issue.
Solution
The file-system going read-only issue which has recently been affecting server models MCS-7816-I4, MCS-7825-I4, and MCS-7828-I4 (or their IBM equivilants) in the field is addressed by CSCti52867 - "IBM 7816-I4 and 782x-I4 READONLY file system".
The fix for CSCti52867 is now available and requires the application of two patch files. Install both of these patch files in the order listed below.
1. First install ciscocm.ibm-diskex-1.0.cop.sgn
The Readme file ciscocm.ibm-diskex-1.0.cop.sgn includes installation instructions for this .cop.sgn.
Make sure to only install this utility when show hardware CLI output indicates the array is in a healthy state.
If your server has never had the filesystem go readonly then this step is optional.
2. Next install Cisco-HDD-FWUpdate-3.0.1-I.ISO .
The Readme file Cisco-HDD-FWUpdate-3.0.1-I.Readme.pdf includes installation instructions for this ISO.
This installer is completely independant of the OS installed on the server.
Note: Installing the FWUpdate v3.0(1) or later will get you firmware with the fix for this defect. It is always recommended that you apply the latest FWUCD available for your server.
Refer to the Release Note of CSCti52867 and the Readme file for each of the above mentioned patch files for more details.
Symptoms
- The file system goes READONLY, then CUCM services may go down, the server may become "unresponsive" meaning that it is not possible to ssh into the server, login to the console, or web into the server although it may still respond to pings.
- Traces from all services stop writing (including syslog)
- You see the following error on the server console
EXT3-fs error (device sda6) in start_transaction: Jornal has aborted
- If you are able to login to the server via SSH, the following output may be displayed.
Last login: Mon Aug X XX:XX:XX XXXX from XXX.XXX.XXX.XXX Command Line Interface is starting up, please wait ... java.io.FileNotFoundException: /var/log/active/platform/log/cli.bin (Read-only file system) at java.io.RandomAccessFile.open(Native Method) at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212) : : : at org.apache.log4j.Category.info(Category.java:674) at sdMain.main(sdMain.java:611) log4j:ERROR No output stream or file set for the appender named [CLI_LOG]. Welcome to the Platform Command Line Interface WARNING: The /common file system is mounted read only. <<<<<<<<<<<<<<<<<< Please use Recovery Disk to check the file system using fsck. admin:
How to determine the current version of firmware on the hard drive
For MCS-7825-I4 and MCS-7828-I4, running Cisco UCM 7.1 and above, you can use the CLI command 'show hardware' to verify the firmware version.
admin:show hardware
HW Platform : 7828I4
Processors : 1
Type
: Intel(R) Core(TM)2 Quad CPU Q9400 @ 2.66GHz
CPU Speed : 2660
Memory : 8192
MBytes
Object ID : 1.3.6.1.4.1.9.1.899
OS Version : UCOS 4.0.0.0-34
Serial Number : KQRBVVB
RAID Version :
Raid firmware version: 1.26.81.00
Raid Bios version: 6.16.00.00
BIOS Information :
IBM IBMBIOSVersion1.44-[M9E144AUS-1.44]- 06/11/2009
RAID Details :
LSI Logic IR Configuration Utility 2.00.15
Read configuration has been initiated for controller 0
------------------------------------------------------------------------
Controller information
------------------------------------------------------------------------
Controller
type
: SAS1064E
BIOS
version
: 6.16.00.00
Firmware version
: 1.26.81.00
Channel
description
: 1 Serial Attached SCSI
Initiator
ID
: 112
Maximum physical
devices
: 62
Concurrent commands
supported : 266
Slot
: 0
Bus
: 1
Device
: 0
Function
: 0
RAID
Support
: Yes
------------------------------------------------------------------------
IR Volume information
------------------------------------------------------------------------
IR volume 1
Volume
ID
: 7
Status of
volume
: Okay (OKY)
RAID
level
: 1
Size (in
MB)
: 237464
Physical hard disks (Target
ID) : 9 8
------------------------------------------------------------------------
Physical device information
------------------------------------------------------------------------
Initiator at ID #112
Target on ID #8
Device is a Hard disk
Enclosure
#
: 1
Slot
#
: 1
Target
ID
: 8
State
: Online (ONL)
Size (in MB)/(in
sectors)
: 238475/488397168
Manufacturer
: ATA
Model
Number
: WD2502ABYS-23B7A
Firmware Revision
: 3B04
Serial
No
: WD-WCAT1D712130
Drive
Type
: SATA
Target on ID #9
Device is a Hard disk
Enclosure
#
: 1
Slot
#
: 0
Target
ID
: 9
State
: Online (ONL)
Size (in MB)/(in
sectors)
: 238475/488397168
Manufacturer
: ATA
Model Number
: WD2502ABYS-23B7A
Firmware
Revision
: 3B04
Serial
No
: WD-WCAT1D723848
Drive
Type
: SATA
------------------------------------------------------------------------
Enclosure information
------------------------------------------------------------------------
Enclosure#
: 1
Logical
ID
: 5005076b:0648afc0
Numslots
: 4
StartSlot
: 0
Start
TargetID
: 0
Start
Bus
: 0
------------------------------------------------------------------------
The text highlited in red are the info you need. This output shows a server with two drives with model number WD2502ABYS on 3B04 firmware. These drives should be upgraded as soon as possible.
- For MCS-7825-I4 and MCS-7828-I4 models running Cisco UCM versions previous to 7.1, as well as any version of Cisco UCM running on a MCS-7816-I4 model server, you must download and boot off of a CD burned with Cisco-HDD-FWUpdate-3.0.1-I.ISO (refer to the "Solution" section for links to download the ISO and readme). Upon successful boot of the Cisco-HDD-FWUpdate-3.0.1-I.ISO CD, you will be presented with the current HDD FW version as well as the opportunity to upgrade to HDD FW version 02.03B06.
What should be done if filesystem issues persist after applying the patches?
As of 16 February 2011 if you encounter any further filesystem or hard drive issues after applying both the firmware and disk exerciser you should proceed to replace the affected drive(s).
There are three ways you can replace the drive(s).
- If you have an MCS server with an active Cisco support contract open a TAC service request.
- If you purchased the IBM x3250 M2 MCS equivalent and have an IBM support contract contact IBM support.
- If you do not have any support contract for the server you can purchase a new drive from Cisco or IBM.
- The Cisco part number for the hard drive is HDD-7825-I4-250=.
- Contact your IBM reseller to confirm the correct part number for the 250GB SATA simple-swap HD for the x3250 M2 server.
If you have any questions you can leave a comment on this document. The ibm-fs-failure@cisco.com email address is no longer active as of 1 September 2014.
Sending the email will not generate a TAC SR but will allow us to collect more information. This is an informal submission with no associated SLA and we will make every effort to follow up submissions but cannot guarantee a response.
Related Defects
- http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCti52867CSCti52867 - "IBM 7816-I4 and 782x-I4 READONLY file system"
- CSCtg26203 - utils create report hardware shows Internal CLI failure
- CSCtn17205 - Drive Exerciser Utility can not be installed on UCCX VOS platform
- http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCti28336CSCti28336 - Need document using the recovery CD when file system mounted read only
Related Links
Hello, I have UCM Pub and SUB (7816 I4). After inputting the CLI command "Show Hardware" I do not receive the full output, I only receive what is displayed below. Any ideas? Thanks!
Hello,
I am facing a similar issue and I need to confirm if I am hitting the same bug or not. It happened before 3 monthes, and I fixed it using the recovery CD and it was fixed. Now it is happening again and we need a permanent solution.
It happens when the customer will not be able to take a backup. When he will login to the server, he will get the attached error.
Can you confirm if we applied the mentioned patches in your article it will resolve my problem ?