9500 wont save cfg, flash errors

[aburrows]

Hi All,

Got a pair of 9500 in a stack-wise virtual setup running 16.9.4.  Has been up and running for 10 weeks since last reboot and with no config changes.  Tried to apply a save to the 9500 and it gave the following errors.  

9500-SW1#copy run startup-config
Destination filename [startup-config]?
Building configuration...

% Configuration buffer full, can't add command: name TR1_015_SECU
%Aborting Save. Compress the config, Save it to flash or Free up space on deviceCompressed configuration from 28236 bytes to 11734 bytes[OK]
9500-SW1#

There is plenty of space in NVRAM but the flash is also having issues with the following errors on bootflash.  bootflash on switch1 show no files, but on standby flash is show the full files.

Directory of nvram:/

32769 -rw- 0 Jan 3 2025 15:39:01 -04:00 startup-config
32770 ---- 3832 <no date> private-config
32771 -rw- 11796 <no date> underlying-config
1 ---- 513 <no date> persistent-data
2 -rw- 530 <no date> C9500-24Y4C_1_0_0
3 ---- 4 <no date> private-KS1
4 -rw- 17 <no date> ecfm_ieee_mib
5 -rw- 802 <no date> IOS-Self-Sig#1.cer
6 -rw- 0 <no date> ifIndex-table
7 -rw- 805 <no date> CiscoLicensi#1CA.cer
8 -rw- 530 <no date> C9500-24Y4C_3_0_0
9 -rw- 820 <no date> IOS-Self-Sig#2.cer
10 -rw- 530 <no date> C9500-24Y4C_5_0_0
11 -rw- 937 <no date> CiscoDNACent#3F27CA.cer

33554432 bytes total (33527488 bytes free)

------------------ dir stby-nvram: ------------------

Directory of stby-nvram:/

32769 -rw- 0 Jan 3 2025 15:39:02 -04:00 startup-config
32770 ---- 3832 <no date> private-config
32771 -rw- 12219 <no date> underlying-config
1 ---- 502 <no date> persistent-data
2 -rw- 530 <no date> C9500-24Y4C_1_0_0
3 ---- 4 <no date> private-KS1
4 -rw- 17 <no date> ecfm_ieee_mib
5 -rw- 802 <no date> IOS-Self-Sig#1.cer
6 -rw- 0 <no date> ifIndex-table
7 -rw- 805 <no date> CiscoLicensi#1CA.cer
8 -rw- 530 <no date> C9500-24Y4C_5_0_0
9 -rw- 820 <no date> IOS-Self-Sig#2.cer
10 -rw- 530 <no date> C9500-24Y4C_3_0_0
11 -rw- 923 <no date> CiscoDNACent#4F2ECA.cer
12 -rw- 937 <no date> CiscoDNACent#3F27CA.cer

Log Buffer (4096 bytes):
n
Jan 6 13:16:39.933 AST: %BTRACE_MANAGER-3-BTRT_ERR_MSG: Chassis 1 R0/0: btman: Btrace Rotate Error: Move file failed, Delete file: reflector_R0-0.6223_371865.20250106171619.bin
Jan 6 13:16:39.933 AST: %IOSXE-3-PLATFORM: Chassis 1 R0/0: kernel: EXT2-fs (sda1): previous I/O error to superblock detected
Jan 6 13:16:39.933 AST: %IOSXE-2-PLATFORM: Chassis 1 R0/0: kernel: EXT2-fs (sda1): error: ext2_readdir: bad page in #63106
Jan 6 13:16:39.933 AST: %IOSXE-3-PLATFORM: Chassis 1 R0/0: kernel: EXT2-fs (sda1): previous I/O error to superblock detected

#dir bootflash:
Directory of bootflash:/

No files in directory

11250098176 bytes total (7216844800 bytes free)

dir stby-bootflash:
Directory of stby-bootflash:/

340706 drwx 4096 Mar 23 2019 19:44:50 -03:00 .dbpersist
340708 -rw- 400 Oct 25 2024 15:54:23 -03:00 boothelper.log.old
340712 drwx 4096 Oct 25 2024 16:00:14 -03:00 .installer
340713 -rw- 1702 Oct 25 2024 16:00:15 -03:00 bootloader_evt_handle.log
340714 drwx 4096 Mar 23 2019 19:52:59 -03:00 .ssh
340721 drwx 4096 Aug 10 2023 04:24:57 -03:00 core
389377 drwx 4096 Oct 25 2024 15:40:14 -03:00 .prst_sync
405601 drwx 4096 Nov 27 2019 12:58:33 -04:00 .rollback_timer
413713 drwx 4096 Mar 23 2019 19:53:14 -03:00 gs_script
421825 drwx 4096 Nov 21 2019 12:03:34 -04:00 tech_support
429937 drwx 4096 Oct 25 2024 16:00:30 -03:00 dc_profile_dir
340711 -rw- 400 Oct 25 2024 16:00:30 -03:00 boothelper.log
340724 -rw- 123888 Oct 25 2024 16:00:51 -03:00 memleak.tcl
340725 drwx 4096 Mar 23 2019 19:53:50 -03:00 onep
340726 -rw- 705009898 Nov 27 2019 12:55:14 -04:00 cat9k_iosxe.16.09.04.SPA.bin
340733 -rw- 2656 Oct 25 2024 16:05:12 -03:00 vlan.dat
397498 -rw- 27231232 Nov 27 2019 12:56:38 -04:00 cat9k-cc_srdriver.16.09.04.SPA.pkg
397499 -rw- 81241084 Nov 27 2019 12:56:38 -04:00 cat9k-espbase.16.09.04.SPA.pkg
397500 -rw- 1647612 Nov 27 2019 12:56:38 -04:00 cat9k-guestshell.16.09.04.SPA.pkg
397501 -rw- 425411576 Nov 27 2019 12:56:39 -04:00 cat9k-rpbase.16.09.04.SPA.pkg
397507 -rw- 34612458 Nov 27 2019 12:56:55 -04:00 cat9k-rpboot.16.09.04.SPA.pkg
397502 -rw- 28931068 Nov 27 2019 12:56:39 -04:00 cat9k-sipbase.16.09.04.SPA.pkg
397503 -rw- 55047160 Nov 27 2019 12:56:39 -04:00 cat9k-sipspa.16.09.04.SPA.pkg
397504 -rw- 35243004 Nov 27 2019 12:56:39 -04:00 cat9k-srdriver.16.09.04.SPA.pkg
397505 -rw- 15856632 Nov 27 2019 12:56:39 -04:00 cat9k-webui.16.09.04.SPA.pkg
397506 -rw- 9208 Nov 27 2019 12:56:39 -04:00 cat9k-wlc.16.09.04.SPA.pkg
340709 -rw- 0 Jan 6 2025 13:27:59 -04:00 packages.conf

11250098176 bytes total (9151950848 bytes free)

[alancelliott]

Looks like you're running out of buffer size when saving your config. The suggestions here may solve your issue: https://community.cisco.com/t5/other-network-architecture-subjects/configuration-buffer-full-can-t-add-command/td-p/373348

Compress the config with the

Router(config)# service compress-config

command, or use the boot buffersize command to increase the buffer size for your the config. Cisco documentation: https://www.cisco.com/c/en/us/td/docs/ios/ios_xe/fundamentals/configuration/guide/2_xe/cf_xe_book/cf_config-files_xe.html#wp1077418

Hope that helps,

Alan

[Leo Laohoo]

---

@aburrows wrote: 

Jan 6 13:16:39.933 AST: %IOSXE-3-PLATFORM: Chassis 1 R0/0: kernel: EXT2-fs (sda1): previous I/O error to superblock detected
Jan 6 13:16:39.933 AST: %IOSXE-2-PLATFORM: Chassis 1 R0/0: kernel: EXT2-fs (sda1): error: ext2_readdir: bad page in #63106
Jan 6 13:16:39.933 AST: %IOSXE-3-PLATFORM: Chassis 1 R0/0: kernel: EXT2-fs (sda1): previous I/O error to superblock detected

---

Good luck. 

These kernel-based error messages are serious and is a cry for help.  If someone chose to ignore them then do not be alarmed with strange things happening to the switch or the network.

[marce1000]

  - FYI : https://www.cisco.com/c/en/us/support/docs/ip/syslog/221138-troubleshoot-disk-errors.html

  M.