N7K-C7010--Copy failed: Read-only file system (0x807B001E)

weiyong wu · ‎08-22-2016

HI:

i have a N7K-C7010 with 2 Supervisors basic information as following:

sho modu---------------------------------------------------------------------

Mod Ports Module-Type Model Status
--- ----- ----------------------------------- ------------------ ----------
1 24 10 Gbps Ethernet Module N7K-M224XP-23L ok
3 48 1/10 Gbps Ethernet Module N7K-F248XP-25 ok
5 0 Supervisor Module-2 N7K-SUP2 active *
6 0 Supervisor Module-2 N7K-SUP2 ha-standby
10 48 1000 Mbps Optical Ethernet XL Modul N7K-M148GS-11L ok

Mod Sw Hw
--- -------------- ------
1 6.2(2a) 2.0
3 6.2(2a) 1.2
5 6.2(2a) 3.0
6 6.2(2a) 3.0
10 6.2(2a) 1.0

show version---------------------------------------------------------------------

Software
BIOS: version 2.12.0
kickstart: version 6.2(6a)
system: version 6.2(2a)
BIOS compile time: 05/29/2013
kickstart image file is: bootflash:///n7000-s2-kickstart.6.2.6a.bin
kickstart compile time: 1/29/2014 15:00:00 [02/26/2014 16:19:34]
system image file is: bootflash:///n7000-s2-dk9.6.2.2a.bin
system compile time: 8/23/2013 15:00:00 [10/15/2013 11:58:57]
Hardware
cisco Nexus7000 C7010 (10 Slot) Chassis ("Supervisor Module-2")
Intel(R) Xeon(R) CPU with 12224948 kB of memory.

when i copy runn bootflash:test22
Copy failed: Read-only file system (0x807B001E)

show logg---------------------------------------------------------------------

2016 Aug 21 17:04:48 N7K Aug 21 17:04:47 %KERN-3-SYSTEM_MSG: [76987803.072249] end_request: I/O error, dev sdb, sector 155776 - kernel
2016 Aug 21 17:04:48 N7K Aug 21 17:04:48 %KERN-3-SYSTEM_MSG: [76987803.082670] Buffer I/O error on device md6, logical block 1 - kernel
2016 Aug 21 17:04:48 N7K Aug 21 17:04:48 %KERN-3-SYSTEM_MSG: [76987803.271597] Buffer I/O error on device md6, logical block 32257 - kernel
2016 Aug 21 17:04:48 N7K %SYSMGR-3-CFGWRITE_FAILED: Configuration copy failed (error-id 0x401E0000).
2016 Aug 21 17:04:48 N7K Aug 21 17:04:48 %KERN-3-SYSTEM_MSG: [76987803.292594] end_request: I/O error, dev sdb, sector 155776 - kernel
2016 Aug 21 17:07:42 N7K aaad: File Writing Failed for /mnt/pss/aaa_vdc_1.seqnum
2016 Aug 21 17:09:57 N7K last message repeated 1 time
2016 Aug 21 17:09:57 N7K Aug 21 17:09:57 %KERN-2-SYSTEM_MSG: [76988112.304129] ext3_abort called. - kernel
2016 Aug 21 17:09:57 N7K Aug 21 17:09:57 %KERN-2-SYSTEM_MSG: [76988112.304168] EXT3-fs error (device md3): ext3_journal_start_sb: Detected aborted journal
- kernel
2016 Aug 21 17:09:57 N7K Aug 21 17:09:57 %KERN-2-SYSTEM_MSG: [76988112.304278] Remounting filesystem read-only - kernel
2016 Aug 21 17:09:57 N7K aaad: File Writing Failed for /mnt/pss/aaa_vdc_1.seqnum
2016 Aug 21 17:10:36 N7K last message repeated 1 time
2016 Aug 21 17:12:01 N7K last message repeated 1 time
2016 Aug 21 17:20:48 N7K last message repeated 3 times
2016 Aug 21 18:20:51 N7K last message repeated 3 times
2016 Aug 21 18:20:53 N7K last message repeated 2 times
2016 Aug 21 18:20:53 N7K Aug 21 18:20:53 %KERN-2-SYSTEM_MSG: [76992368.140442] EXT3-fs error (device md5): ext3_readdir: directory #9801 contains a hole at
offset 0 - kernel
2016 Aug 21 18:20:53 N7K Aug 21 18:20:53 %KERN-2-SYSTEM_MSG: [76992368.140626] EXT3-fs error (device md5): ext3_readdir: directory #7841 contains a hole at
offset 0 - kernel
2016 Aug 21 18:20:53 N7K Aug 21 18:20:53 %KERN-2-SYSTEM_MSG: [76992368.140768] EXT3-fs error (device md5): ext3_readdir: directory #1961 contains a hole at
offset 0 - kernel
2016 Aug 21 18:20:53 N7K Aug 21 18:20:53 %KERN-2-SYSTEM_MSG: [76992368.140919] EXT3-fs error (device md5): ext3_readdir: directory #9802 contains a hole at
offset 0 - kernel
2016 Aug 21 18:20:53 N7K Aug 21 18:20:53 %KERN-2-SYSTEM_MSG: [76992368.141058] EXT3-fs error (device md5): ext3_readdir: directory #1962 contains a hole at
offset 0 - kernel
2016 Aug 21 18:20:53 N7K Aug 21 18:20:53 %KERN-2-SYSTEM_MSG: [76992368.141197] EXT3-fs error (device md5): ext3_readdir: directory #17642 contains a hole a
t offset 0 - kernel
2016 Aug 21 18:20:53 N7K Aug 21 18:20:53 %KERN-2-SYSTEM_MSG: [76992368.141400] EXT3-fs error (device md5): ext3_readdir: directory #15681 contains a hole a
t offset 0 - kernel
2016 Aug 21 18:20:53 N7K %SYSMGR-3-CFGWRITE_FAILED: Configuration copy failed (error-id 0x401E0004).
2016 Aug 21 18:20:53 N7K Aug 21 18:20:53 %KERN-2-SYSTEM_MSG: [76992368.148771] EXT3-fs error (device md5): ext3_readdir: directory #9801 contains a hole at
offset 0 - kernel
2016 Aug 21 18:20:53 N7K Aug 21 18:20:53 %KERN-2-SYSTEM_MSG: [76992368.148959] EXT3-fs error (device md5): ext3_readdir: directory #7841 contains a hole at
offset 0 - kernel
2016 Aug 21 18:20:53 N7K Aug 21 18:20:53 %KERN-2-SYSTEM_MSG: [76992368.149103] EXT3-fs error (device md5): ext3_readdir: directory #1961 contains a hole at
offset 0 - kernel
2016 Aug 21 18:20:53 N7K Aug 21 18:20:53 %KERN-2-SYSTEM_MSG: [76992368.149259] EXT3-fs error (device md5): ext3_readdir: directory #9802 contains a hole at
offset 0 - kernel
2016 Aug 21 18:20:53 N7K Aug 21 18:20:53 %KERN-2-SYSTEM_MSG: [76992368.149402] EXT3-fs error (device md5): ext3_readdir: directory #1962 contains a hole at
offset 0 - kernel
2016 Aug 21 18:20:53 N7K Aug 21 18:20:53 %KERN-2-SYSTEM_MSG: [76992368.149544] EXT3-fs error (device md5): ext3_readdir: directory #17642 contains a hole a
t offset 0 - kernel
2016 Aug 21 18:20:53 N7K Aug 21 18:20:53 %KERN-2-SYSTEM_MSG: [76992368.149744] EXT3-fs error (device md5): ext3_readdir: directory #15681 contains a hole a
t offset 0 - kernel
2016 Aug 21 18:20:53 N7K Aug 21 18:20:53 %KERN-2-SYSTEM_MSG: [76992368.156369] EXT3-fs error (device md6): ext3_readdir: directory #2 contains a hole at of
fset 0 - kernel
2016 Aug 21 18:20:53 N7K Aug 21 18:20:53 %KERN-3-SYSTEM_MSG: [76992368.156506] end_request: I/O error, dev sdb, sector 155776 - kernel
2016 Aug 21 18:20:53 N7K Aug 21 18:20:53 %KERN-3-SYSTEM_MSG: [76992368.163877] Buffer I/O error on device md6, logical block 1 - kernel
2016 Aug 21 18:20:53 N7K Aug 21 18:20:53 %KERN-3-SYSTEM_MSG: [76992368.353323] Buffer I/O error on device md6, logical block 32257 - kernel
2016 Aug 21 18:20:53 N7K %SYSMGR-3-CFGWRITE_FAILED: Configuration copy failed (error-id 0x401E0000).
2016 Aug 21 18:20:53 N7K Aug 21 18:20:53 %KERN-3-SYSTEM_MSG: [76992368.373906] end_request: I/O error, dev sdb, sector 155776 - kernel
2016 Aug 21 18:21:00 N7K aaad: File Writing Failed for /mnt/pss/aaa_vdc_1.seqnum
2016 Aug 21 19:11:57 N7K last message repeated 2 times
2016 Aug 21 19:52:43 N7K %USER-3-SYSTEM_MSG: frequency file /mnt/pss/ntp.drift.TEMP: Read-only file system - ntpd
2016 Aug 22 02:52:43 N7K last message repeated 1 time
2016 Aug 22 09:52:43 N7K last message repeated 1 time
2016 Aug 22 15:27:56 N7K aaad: File Writing Failed for /mnt/pss/aaa_vdc_1.seqnum
2016 Aug 22 15:28:40 N7K last message repeated 2 times
2016 Aug 22 15:31:27 N7K last message repeated 1 time
2016 Aug 22 15:34:09 N7K last message repeated 2 times
2016 Aug 22 15:41:55 N7K last message repeated 3 times
2016 Aug 22 15:49:54 N7K last message repeated 2 times
2016 Aug 22 15:52:43 N7K last message repeated 4 times
2016 Aug 22 15:52:43 N7K %USER-3-SYSTEM_MSG: frequency file /mnt/pss/ntp.drift.TEMP: Read-only file system - ntpd
2016 Aug 22 15:55:58 N7K aaad: File Writing Failed for /mnt/pss/aaa_vdc_1.seqnum
2016 Aug 22 15:59:14 N7K last message repeated 1 time
2016 Aug 22 16:01:26 N7K last message repeated 20 times
2016 Aug 22 16:08:59 N7K last message repeated 3 times

Anybody help me solved this iusse,thx

Mark Malone · ‎08-22-2016

works ok on mine as a test below on v6.2.12

xxxxxxxxx# copy running-config bootflash:backup.22.8.16.txt
Copy complete, now saving to disk (please wait)...

********************************************************************

You could be hitting a bug this looks very similar on 9k same error code

I/O Error when accessing bootflash - Read Only File system

CSCuo44829

Description

Unable to access bootflash
Nexus9500# Copy running-config bootflash:test_run
Copy failed: Read-only file system (0x807B001E)
Copy complete, now saving to disk (please wait)...

Also see I/O errors in the log
Apr 12 05:58:09 %KERN-3-SYSTEM_MSG: [6968471.092073] Buffer I/O error on device sda7, logical block 4307 - kernel
Apr 12 05:58:09 %KERN-3-SYSTEM_MSG: [6968471.092091] Aborting journal on device sda4-8. - kernel
2014 Apr 12 05:58:10 tukuscore01 Apr 12 05:58:09 %KERN-3-SYSTEM_MSG: [6968471.092102] end_request: I/O error, dev sda, sector 100753408 - kernel

Symptom:
Unable to access bootflash
Nexus9500# Copy running-config bootflash:test_run
Copy failed: Read-only file system (0x807B001E)
Copy complete, now saving to disk (please wait)...

Also see I/O errors in the log
Apr 12 05:58:09 %KERN-3-SYSTEM_MSG: [6968471.092073] Buffer I/O error on device sda7, logical block 4307 - kernel
Apr 12 05:58:09 %KERN-3-SYSTEM_MSG: [6968471.092091] Aborting journal on device sda4-8. - kernel
2014 Apr 12 05:58:10 tukuscore01 Apr 12 05:58:09 %KERN-3-SYSTEM_MSG: [6968471.092102] end_request: I/O error, dev sda, sector 100753408 - kernel

Conditions:
Seen on systems that has been up and running for over 70 days, however it can be seen in new setups as well

Workaround:
1. A power cycle of the switch is required to clear the condition in case of a ToR
2. In case of redundant supervisor, if standby supervisor is not affected, one can switchover to standby supervisor.

Further Problem Description:
Issue is fixed in 6.1(2)I2(2b) and 6.1(2)I3(1) and later.

A firmware patch is available that will permanently fix the issue on older versions of code. Please contact your local account SE or Cisco TAC on instructions to get the firmware patch and update the system.

weiyongwu · ‎08-28-2016

hi mark malone,

thank you for you reply, n7k's software verison is 6.2(2a),i don't know whether it is bug or not

.bug CSCuo44829 is suit for n9k...

Mark Malone · ‎08-31-2016

Hi

That bug effects the linux kernel not the nx-os , it would operate across all platforms 5.7.9 , bugs that are on one platform can be on the other as there running underneath same Linux kernel versions , I understand its not the exact match but it shows its most likely a bug your being effected as the error code is the same whether its 5,7 or 9k

weiyongwu · ‎08-31-2016

hi ,any software version can fix my problem?