CAT3850 missing flash: filesystem

[thomas.werner]

I have been trying to recover this CAT3850 since last week.  I learned yesterday that usb drive needed to be formatted FAT16.  I reformatted usb drive which contained IOS XE bin files.  I tried to install 16.12.05b.  Installer fails.  Here is output at terminal (I am typing this manually):

switch: dir

List of filesystems currently registered:

     xmodem[0]: (read-only)

       null[1]: (read-write)

         bs[3]: (read-only)

       tftp[17]: (read-write)

        ftp[18]: (read-only)

       http[19]: (read-only)

 usbflash0:[20]: (read-write)

switch: dir usbflash0:

Directory of usbflash0:/

     1  -rw-  450568480  cat3k_caa-universalk9.16.09.06.SPA.bin

     2  -rw-  450577620  cat3k_caa-universalk9.16.09.07.SPA.bin

     3  -rw-  480018036  cat3k_caa-universalk9.16.12.05b.SPA.bin

     4  -rw-  480157511  cat3k_caa-universalk9.16.12.06.SPA.bin

231710720 bytes available (1861332992 bytes used)

switch: set

ABNORMAL_RESET_COUNT=1

ASIC_PCI_RESET=1

BOOT=flash:packages.conf

BOOT_LOADER_UPGRADE_DISABLE=1

CFG_MODEL_NUM=WS-C3850-24S-E

CLEI_CODE_NUMBER=IPMW100ARC

DDR_SPEED=667

DEFAULT_ROUTER=192.0.2.1

ECI_CODE_NUMBER=468647

IP_ADDR=192.0.2.123/255.255.255.0

LICENSE_BOOT_LEVEL=ipservicesk9,all:3850_XS_24;

MAC_ADDR=70:1F:53:29:EE:80

MANUAL_BOOT=no

MODEL_NUM=WS-C3850-24S

MODEL_REVISION_NUM=L0

MOTHERBOARD_ASSEMBLY_NUM=73-14445-07

MOTHERBOARD_REVISION_NUM=B0

MOTHERBOARD_SERIAL_NUM=FOC2133284K

RECOVERY_BUDLE=sda9:cat3k_caa-recovery.bin

SWITCH_NUMBER=1

SYSTEM_SERIAL_NUM=FOC2133U182

TAN_NUM=800-41087-07

TAN_REVISION_NUMBER=C0

TERMLINES=0

USB_ASSEMBLY_NUM=73-16167-02

USB_REVISION_NUM=A0

USB_SERIAL_NUM=FOC213245S0W

VERSION_ID=V03

Switch: emergency-install usbflash0:cat3k_caa-universalk9.16.12.05b.SPA.bin

The bootflash will be erased during installation operation, continue (y/n)?y

Starting emergency recovery (usbflash0:cat3k_caa-universalk9.16.12.05b.SPA.bin)...

Reading full image into memory.................done

Bundle image

..........................

Kernel Address    : 0x537945d4

Kernel Size       : 0x41852f/4293935

Initramfs Address : 0x53bacb04

Initramfs Size    : 0xe77ac6/15170246

Compression Format: .mzip

Bootable image at @ ram:0x537945d4

Bootable image segment 0 address range [0x81100000, 0x82140000] is in range [0x80180000, 0x90000000].

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

File "sda9:cat3k_caa-recovery.bin" uncompressed and installed, entry point: 0x8166a600

Loading Linux kernel with entry point 0x8166a600 ...

Bootloader: Done loading app on core_mask: 0x3f

### Launching Linux Kernel (flags = 0x5)

Initialting Emergency Installation of bundle usbflash0:cat3k_caa-universalk9.16.12.05b.SPA.bin

Reading bundle usbflash0:cat3k_caa-universalk9.16.12.05b.SPA.bin...

Validating bundle usbflash0:cat3k_caa-universalk9.16.12.05b.SPA.bin...

Installing bundle usbflash0:cat3k_caa-universalk9.16.12.05b.SPA.bin...

Verifying bundle usbflash0:cat3k_caa-universalk9.16.12.05b.SPA.bin...

Package cat3k_caa-guestshell.16.12.05b.SPA.pkg is Digitally Signed

Package cat3k_caa-rpbase.16.12.05b.SPA.pkg is Digitally Signed

Package cat3k_caa-rpcore.16.12.05b.SPA.pkg is Digitally Signed

Package cat3k_caa-srdriver.16.12.05b.SPA.pkg is Digitally Signed

Package cat3k_caa-webui.16.12.05b.SPA.pkg is Digitally Signed

Preparing flash.../FAT: bogus number of reserved sectors

Failed to cleanly mount flash partition

Failed preparing flash - Emergency Install aborted

Restating system.

Booting...(use DDR clock 667 MHz)sda3: format unknown, not mounted.

Up 1000 Mbps Full duplex (port 0) (SGMII)

flash:packages.conf: no such device

Cannot open file flash:packages.conf

flash:packages.conf: no such device

flash:packages.conf: no such device

Error loading "flash:packages.conf"

Interrupt within 5 seconds to abort boot process.

Boot process failed...

The system is unable to boot automatically.  The

BOOT environment variable needs to be set to a

bootable image.

Switch:

I tried to resolve by "flash_init".  The command appeared to work, but the "flash:" filesystem is still missing.

Switch: flash_init

Initializing Flash...

flashfs[7]: 0 files, 1 directories

flashfs[7]: 0 orphaned files, 0 orphaned directories

flashfs[7]: Total bytes: 6784000

flashfs[7]: Bytes used: 1024

flashfs[7]: Bytes available: 6782976

flashfs[7]: flashfs fsck took 1 seconds....done Initializing Flash.

switch: dir

List of filesystems currently registered:

     xmodem[0]: (read-only)

       null[1]: (read-write)

         bs[3]: (read-only)

       tftp[17]: (read-write)

        ftp[18]: (read-only)

       http[19]: (read-only)

 usbflash0:[20]: (read-write)

[marce1000]

 - Try fsck flash:

 M.

[thomas.werner]

Thanks for reply.  Unfortunately, when I type the command, I get this output:

switch: fsck flash:

Unknown filesystem "flash:"

switch:

[Leo Laohoo · Accepted solution]

Ok, I have seen this before.  A lot of times.

1. Do this from ROMMON:  boot usbflash0:IOS.bin
2. After the switch boots the IOS-XE, copy the IOS-XE file into the switch's flash.
3. Unpack the firmware using the command "request platform software package install switch all file flash:cat3k_caa-universalk9.16.12.06.SPA.bin on-reboot new auto-copy verbose". 
4. Make sure the boot variable is pointing to "packages.conf" file. 
5. Cold reboot (pull the power cable and DO NOT USE the "reload" command) to check if it works.  

[thomas.werner]

It took a couple of iterations, but I now have a flash: and the switch survives a power cycle.  Now working on restoring configuration.  Thank you for your assistance.

[keka289]

And how did you fix it? We've got the same problem with an old 3850 and I'm trying to get the flash: filesystem back. After stuck in ROMMON mode and lots of iterations I'm able to boot again, but the flash: FS is still missing. Maybe someone can help (I know its an old topic). Thank you and Regards.

[Leo Laohoo]

Post the complete output to the following ROMMON commands:

• set
• dir flash:
• dir usbflash0:

[keka289]

switch: set
ABNORMAL_RESET_COUNT=0
ASIC_PCI_RESET=1
AUTO_SWITCH_CONSOLE_DISABLE=0
BAUD=9600
BOOT=flash:packages.conf
BOOT_LOADER_UPGRADE_DISABLE=1
BSI=0
CALL_HOME_DEBUG=0000000000000
CFG_MODEL_NUM=WS-C3850-48XS-F-E
CLEI_CODE_NUMBER=IPMDH00BRB
CSR_PCIERST_DISCONNECTED=yes
DC_COPY=yes
DDR_SPEED=667
DEFAULT_ROUTER=192.0.2.1
D_STACK_DOMAIN_NUM=1
ECI_CODE_NUMBER=469464
IP_ADDR=192.0.2.123/255.255.255.0
LICENSE_BOOT_LEVEL=ipservicesk9,all:ngwc;
MAC_ADDR=00:BF:77:09:52:00
MANUAL_BOOT=yes
MODEL_NUM=WS-C3850-48XS
MODEL_REVISION_NUM=H0
MOTHERBOARD_ASSEMBLY_NUM=73-16622-05
MOTHERBOARD_REVISION_NUM=B0
MOTHERBOARD_SERIAL_NUM=FOC22011CZE
RANDOM_NUM=404031664
RECOVERY_BUNDLE=sda9:cat3k_caa-recovery.bin
SBC_CFG=17-G48CSR-04.SBC_cfg
SWITCH_IGNORE_STARTUP_CFG=1
SWITCH_NUMBER=1
SYSTEM_SERIAL_NUM=FOC2202T2FA
TAN_NUM=68-5295-02
TAN_REVISION_NUMBER=G0
TEMPLATE=advanced
TERMLINES=0
USB_ASSEMBLY_NUM=73-16167-02
USB_REVISION_NUM=A0
USB_SERIAL_NUM=FOC21514F2K
VERSION_ID=V02

------------------

switch: dir flash:
unable to stat flash:/: no such device

-------------------

switch: dir usbflash0:
Directory of usbflash0:/

2 -rw- 16384 FSCK0000.REC
3 -rw- 16384 FSCK0001.REC
4 -rw- 189743104 cat3k_caa-universalk9.16.12.07.SPA.bin
5 -rw- 0 cat3k_caa-universalk9.16.09.07.SPA.bin
6 -rw- 480231424 cat3k_caa-universalk9.16.12.09.SPA.bin
7 -rw- 0 cat3k_caa-universalk9.SPA.03.07.05.E.152-3.E5.bin

[keka289]

After doing an emergency-install, the flash file system disappeared.

sda9: the emergency-install BIN also disappeared after install.

I can only boot directly with the "cat3k_caa-universalk9.SPA.03.07.05.E.152-3.E5.bin" via TFTP, but I dont know how to recover SDAs, flash file system, etc. - it looks like its completely broken.

I've set MANUAL_BOOT=yes and SWITCH_IGNORE_STARTUP_CFG=1 for testing purposes.

[Leo Laohoo]

---

@keka289 wrote:

BOOT_LOADER_UPGRADE_DISABLE=1

---

Enter "BOOT_LOADER_UPGRADE_DISABLE=0". 

---

@keka289 wrote:

SWITCH_IGNORE_STARTUP_CFG=1

---

SWITCH_IGNORE_STARTUP_CFG=0

---

@keka289 wrote:

MANUAL_BOOT=yes

---

This is the source of the issue. 

Enter "MANUAL_BOOT=no".

[keka289]

As I wrote in my last post, I've did the MANUAL_BOOT and SWITCH_IGNORE_STARTUP_CFG Option just for testing purposes. Now I've changed it back to "0" and "no" again.

---

As you wrote - I've changed this now:

BOOT_LOADER_UPGRADE_DISABLE=1

... to "0". But without any effect.

Booting...(use DDR clock 667 MHz)sda3: format unkown, not mounted.
Up 1000 Mbps Full duplex (port 0) (SGMII)
flash:packages.conf: no such device
Cannot open file flash:packages.conf
flash:packages.conf: no such device
flash:packages.conf: no such device

Error loading "flash:packages.conf"

Interrupt within 5 seconds to abort boot process.
Boot process failed...

switch: dir

List of filesystems currently registered:

xmodem[0]: (read-only)
null[1]: (read-write)
bs[3]: (read-only)
usbflash0[10]: (read-write)
tftp[13]: (read-write)
ftp[14]: (read-only)
http[15]: (read-only)

switch: dir sda9:
unable to stat sda9:/: no such device

[Leo Laohoo]

Ok, try this: 

boot usbflash0:cat3k_caa-universalk9.16.12.07.SPA.bin

If the switch boots up, follow up with the following commands: 

dir flash:
sh boot
sh romvar

[keka289]

First try with usbflash0:

### Launching Linux Kernel (flags = 0x5)

Linux version 4.9.187 (sumikum7@sjc-ads-5670) (gcc version 5.3.0 (GCC) ) #1 SMP Fri Nov 19 03:09:46 PST 2021
CVMSEG size: 2 cache lines (256 bytes)
Cavium Inc. SDK-5.1.0
bootconsole [early0] enabled
CPU0 revision is: 000d9202 (Cavium Octeon II)
Checking for the multiply/shift bug... no.
Checking for the daddiu bug... no.
EXT2-fs (sdb7): error: unable to read group descriptors
fsck_or_mkfs.sh[376]: mount: wrong fs type, bad option, bad superblock on /dev/sdb7,
fsck_or_mkfs.sh[376]:        missing codepage or helper program, or other error
fsck_or_mkfs.sh[376]:        In some cases useful info is found in syslog - try
fsck_or_mkfs.sh[376]:        dmesg | tail or so.
fsck_or_mkfs.sh[376]: ERROR: couldn't mount /dev/bootflash7
fsck_or_mkfs.sh[376]: Executing mkfs on /dev/bootflash7
fsck_or_mkfs.sh[376]: mke2fs 1.43-WIP (18-May-2015)
fsck_or_mkfs.sh[376]: Successfully mounted /dev/bootflash7
fsck_or_mkfs.sh[381]: mount: wrong fs type, bad option, bad superblock on /dev/sdb3,
fsck_or_mkfs.sh[381]:        missing codepage or helper program, or other error
fsck_or_mkfs.sh[381]:        In some cases useful info is found in syslog - try
fsck_or_mkfs.sh[381]:        dmesg | tail or so.
fsck_or_mkfs.sh[381]: ERROR: couldn't mount /dev/bootflash3
fsck_or_mkfs.sh[381]: Executing mkfs on /dev/bootflash3
fsck_or_mkfs.sh[381]: mke2fs 1.43-WIP (18-May-2015)
fsck_or_mkfs.sh[381]: mount: wrong fs type, bad option, bad superblock on /dev/sdb3,
fsck_or_mkfs.sh[381]:        missing codepage or helper program, or other error
fsck_or_mkfs.sh[381]:        In some cases useful info is found in syslog - try
fsck_or_mkfs.sh[381]:        dmesg | tail or so.
fsck_or_mkfs.sh[381]: Reformat failed
● systemd-fsck@dev-bootflash3.service - File System Check on /dev/bootflash3
   Loaded: loaded (/lib/systemd/system/systemd-fsck@.service; static; vendor preset: enabled)
  Drop-In: /lib/systemd/system/systemd-fsck@.service.d
           └─fsck_or_mkfs.conf
   Active: failed (Result: exit-code) since Fri 2023-07-28 04:48:28 Universal; 126ms ago
     Docs: man:systemd-fsck@.service(8)
  Process: 381 ExecStart=/sbin/fsck_or_mkfs.sh %f (code=exited, status=32)
 Main PID: 381 (code=exited, status=32)

[    6.958411] localhost systemd[1]: Starting File System Check on /dev/bootflash3...
[    6.999165] localhost fsck_or_mkfs.sh[381]: e2fsck 1.43-WIP (18-May-2015)
[    7.144596] localhost fsck_or_mkfs.sh[381]: ext2fs_open2: Bad magic number in super-block
[    7.145082] localhost fsck_or_mkfs.sh[381]: fsck.ext2: Superblock invalid, trying backup blocks...
[    7.151724] localhost fsck_or_mkfs.sh[381]: /dev/sdb3 was not cleanly unmounted, check forced.
[    7.597779] localhost fsck_or_mkfs.sh[381]: Pass 1: Checking inodes, blocks, and sizes
[   26.789775] localhost fsck_or_mkfs.sh[381]: Pass 2: Checking directory structure
[   26.798140] localhost fsck_or_mkfs.sh[381]: Pass 3: Checking directory connectivity
[   26.799449] localhost fsck_or_mkfs.sh[381]: Unconnected directory inode 8097 (/lost+found/???)
[   26.799981] localhost fsck_or_mkfs.sh[381]: Connect to /lost+found? yes
[   26.801370] localhost fsck_or_mkfs.sh[381]: Unconnected directory inode 161921 (/lost+found/???)
[   26.801898] localhost fsck_or_mkfs.sh[381]: Connect to /lost+found? yes
[   26.803288] localhost fsck_or_mkfs.sh[381]: Pass 4: Checking reference counts
[   26.804265] localhost fsck_or_mkfs.sh[381]: Inode 11 ref count is 2, should be 4.  Fix? yes
[   26.806037] localhost fsck_or_mkfs.sh[381]: Inode 8097 ref count is 3, should be 2.  Fix? yes
[   26.821948] localhost fsck_or_mkfs.sh[381]: Inode 161921 ref count is 3, should be 2.  Fix? yes
[   26.859110] localhost fsck_or_mkfs.sh[381]: Pass 5: Checking group summary information
[   27.023451] localhost fsck_or_mkfs.sh[381]: Free blocks count wrong for group #1 (31816, counted=18387).
[   27.023988] localhost fsck_or_mkfs.sh[381]: Fix? yes
[   27.024880] localhost fsck_or_mkfs.sh[381]: Free blocks count wrong for group #4 (32260, counted=26936).
[   27.025431] localhost fsck_or_mkfs.sh[381]: Fix? yes
[   27.026323] localhost fsck_or_mkfs.sh[381]: Free blocks count wrong for group #5 (31816, counted=30676).
[   27.026862] localhost fsck_or_mkfs.sh[381]: Fix? yes
[   27.027747] localhost fsck_or_mkfs.sh[381]: Free blocks count wrong for group #20 (32260, counted=32259).
[   27.028298] localhost fsck_or_mkfs.sh[381]: Fix? yes
[   27.029067] localhost fsck_or_mkfs.sh[381]: Free blocks count wrong (1780886, counted=1760992).
[   27.029595] localhost fsck_or_mkfs.sh[381]: Fix? yes
[   27.104968] localhost fsck_or_mkfs.sh[381]: Free inodes count wrong for group #1 (8096, counted=8090).
[   27.105521] localhost fsck_or_mkfs.sh[381]: Fix? yes
[   27.106344] localhost fsck_or_mkfs.sh[381]: Directories count wrong for group #1 (0, counted=1).
[   27.106883] localhost fsck_or_mkfs.sh[381]: Fix? yes
[   27.107775] localhost fsck_or_mkfs.sh[381]: Free inodes count wrong for group #20 (8096, counted=8095).
[   27.108332] localhost fsck_or_mkfs.sh[381]: Fix? yes
[   27.109152] localhost fsck_or_mkfs.sh[381]: Directories count wrong for group #20 (0, counted=1).
[   27.109680] localhost fsck_or_mkfs.sh[381]: Fix? yes
[   27.110439] localhost fsck_or_mkfs.sh[381]: Free inodes count wrong (453365, counted=453358).
[   27.110981] localhost fsck_or_mkfs.sh[381]: Fix? yes
[   36.198585] localhost fsck_or_mkfs.sh[381]: /dev/sdb3: ***** FILE SYSTEM WAS MODIFIED *****
[   36.199086] localhost fsck_or_mkfs.sh[381]: /dev/sdb3: 18/453376 files (11.1% non-contiguous), 52344/1813336 blocks
[   36.252411] localhost fsck_or_mkfs.sh[381]: fsck return value for /dev/bootflash3: 1
[   36.267151] localhost fsck_or_mkfs.sh[381]: Mounting /dev/bootflash3 at /tmp/fsck-or-mkfs-bootflash3
[   45.155628] localhost fsck_or_mkfs.sh[381]: mount: wrong fs type, bad option, bad superblock on /dev/sdb3,
[   45.241636] localhost fsck_or_mkfs.sh[381]:        missing codepage or helper program, or other error
[   45.327632] localhost fsck_or_mkfs.sh[381]:        In some cases useful info is found in syslog - try
[   45.382577] localhost fsck_or_mkfs.sh[381]:        dmesg | tail or so.
[   45.383237] localhost fsck_or_mkfs.sh[381]: Creating filesystem with 1813336 4k blocks and 453376 inodes
[   45.383720] localhost fsck_or_mkfs.sh[381]: Filesystem UUID: 94aade04-d5da-445c-9899-9ef57faadf41
[   45.384215] localhost fsck_or_mkfs.sh[381]: Superblock backups stored on blocks:
[   45.384695] localhost fsck_or_mkfs.sh[381]:         32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632
[   45.385190] localhost fsck_or_mkfs.sh[381]: [49B blob data]
[   45.385671] localhost fsck_or_mkfs.sh[381]: [46B blob data]
[   45.450447] localhost fsck_or_mkfs.sh[381]: ERROR: couldn't mount /dev/bootflash3
[   45.512446] localhost fsck_or_mkfs.sh[381]: Executing mkfs on /dev/bootflash3
[   45.570445] localhost fsck_or_mkfs.sh[381]: mke2fs 1.43-WIP (18-May-2015)
[   59.306811] localhost fsck_or_mkfs.sh[381]: [83B blob data]
[   68.206244] localhost fsck_or_mkfs.sh[381]: mount: wrong fs type, bad option, bad superblock on /dev/sdb3,
[   68.206522] localhost fsck_or_mkfs.sh[381]:        missing codepage or helper program, or other error
[   68.206786] localhost fsck_or_mkfs.sh[381]:        In some cases useful info is found in syslog - try
[   68.207030] localhost fsck_or_mkfs.sh[381]:        dmesg | tail or so.
[   68.207725] localhost systemd[1]: systemd-fsck@dev-bootflash3.service: Main process exited, code=exited, status=32/n/a
[   68.208102] localhost fsck_or_mkfs.sh[381]: Reformat failed
[   68.208809] localhost systemd[1]: Failed to start File System Check on /dev/bootflash3.
[   68.266193] localhost systemd[1]: systemd-fsck@dev-bootflash3.service: Unit entered failed state.
[   68.266692] localhost systemd[1]: systemd-fsck@dev-bootflash3.service: Failed with result 'exit-code'.
Warning: systemd-fsck@dev-bootflash3.service changed on disk. Run 'systemctl daemon-reload' to reload units.
>>> Rebooting
Failed to connect to bus: No such file or directory
Failed to connect to bus: No such file or directory
watchdog: watchdog0: watchdog did not stop!
reboot: Restarting system

[keka289]

Second try via TFTP:

### Launching Linux Kernel (flags = 0x5)

Linux version 4.9.187 (sumikum7@sjc-ads-5670) (gcc version 5.3.0 (GCC) ) #1 SMP Fri Nov 19 03:09:46 PST 2021
CVMSEG size: 2 cache lines (256 bytes)
Cavium Inc. SDK-5.1.0
bootconsole [early0] enabled
CPU0 revision is: 000d9202 (Cavium Octeon II)
Checking for the multiply/shift bug... no.
Checking for the daddiu bug... no.
fsck_or_mkfs.sh[320]: mount: wrong fs type, bad option, bad superblock on /dev/sda3,
fsck_or_mkfs.sh[320]:        missing codepage or helper program, or other error
fsck_or_mkfs.sh[320]:        In some cases useful info is found in syslog - try
fsck_or_mkfs.sh[320]:        dmesg | tail or so.
fsck_or_mkfs.sh[320]: ERROR: couldn't mount /dev/bootflash3
fsck_or_mkfs.sh[320]: Executing mkfs on /dev/bootflash3
fsck_or_mkfs.sh[320]: mke2fs 1.43-WIP (18-May-2015)
fsck_or_mkfs.sh[320]: mount: wrong fs type, bad option, bad superblock on /dev/sda3,
fsck_or_mkfs.sh[320]:        missing codepage or helper program, or other error
fsck_or_mkfs.sh[320]:        In some cases useful info is found in syslog - try
fsck_or_mkfs.sh[320]:        dmesg | tail or so.
fsck_or_mkfs.sh[320]: Reformat failed
● systemd-fsck@dev-bootflash3.service - File System Check on /dev/bootflash3
   Loaded: loaded (/lib/systemd/system/systemd-fsck@.service; static; vendor preset: enabled)
  Drop-In: /lib/systemd/system/systemd-fsck@.service.d
           └─fsck_or_mkfs.conf
   Active: failed (Result: exit-code) since Fri 2023-07-28 04:56:00 Universal; 125ms ago
     Docs: man:systemd-fsck@.service(8)
  Process: 320 ExecStart=/sbin/fsck_or_mkfs.sh %f (code=exited, status=32)
 Main PID: 320 (code=exited, status=32)

[   18.467602] localhost fsck_or_mkfs.sh[320]: Error reading block 360712 (Attempt to read block from filesystem resulted in short read) while getting next inode from scan.  Ignore error? yes
[   18.468117] localhost fsck_or_mkfs.sh[320]: Force rewrite? yes
[   18.468620] localhost fsck_or_mkfs.sh[320]: Error reading block 360713 (Attempt to read block from filesystem resulted in short read) while getting next inode from scan.  Ignore error? yes
[   18.469152] localhost fsck_or_mkfs.sh[320]: Force rewrite? yes
[   18.469754] localhost fsck_or_mkfs.sh[320]: Error reading block 360714 (Attempt to read block from filesystem resulted in short read) while getting next inode from scan.  Ignore error? yes
[   18.470285] localhost fsck_or_mkfs.sh[320]: Force rewrite? yes
[   18.470772] localhost fsck_or_mkfs.sh[320]: Error reading block 360715 (Attempt to read block from filesystem resulted in short read) while getting next inode from scan.  Ignore error? yes
[   18.471300] localhost fsck_or_mkfs.sh[320]: Force rewrite? yes
[   18.471832] localhost fsck_or_mkfs.sh[320]: Error reading block 360716 (Attempt to read block from filesystem resulted in short read) while getting next inode from scan.  Ignore error? yes
[   18.472382] localhost fsck_or_mkfs.sh[320]: Force rewrite? yes
[   18.472881] localhost fsck_or_mkfs.sh[320]: Error reading block 360717 (Attempt to read block from filesystem resulted in short read) while getting next inode from scan.  Ignore error? yes
[   18.473432] localhost fsck_or_mkfs.sh[320]: Force rewrite? yes
[   18.473920] localhost fsck_or_mkfs.sh[320]: Error reading block 360718 (Attempt to read block from filesystem resulted in short read) while getting next inode from scan.  Ignore error? yes
[   18.474456] localhost fsck_or_mkfs.sh[320]: Force rewrite? yes
[   18.474958] localhost fsck_or_mkfs.sh[320]: Error reading block 360719 (Attempt to read block from filesystem resulted in short read) while getting next inode from scan.  Ignore error? yes
[   18.475516] localhost fsck_or_mkfs.sh[320]: Force rewrite? yes
[   18.476033] localhost fsck_or_mkfs.sh[320]: Error reading block 360720 (Attempt to read block from filesystem resulted in short read) while getting next inode from scan.  Ignore error? yes
[   18.476542] localhost fsck_or_mkfs.sh[320]: Force rewrite? yes
[   18.477072] localhost fsck_or_mkfs.sh[320]: Error reading block 360721 (Attempt to read block from filesystem resulted in short read) while getting next inode from scan.  Ignore error? yes
[   18.477586] localhost fsck_or_mkfs.sh[320]: Force rewrite? yes
[   18.478120] localhost fsck_or_mkfs.sh[320]: Error reading block 360722 (Attempt to read block from filesystem resulted in short read) while getting next inode from scan.  Ignore error? yes
[   18.478624] localhost fsck_or_mkfs.sh[320]: Force rewrite? yes
[   18.479185] localhost fsck_or_mkfs.sh[320]: Error reading block 360723 (Attempt to read block from filesystem resulted in short read) while getting next inode from scan.  Ignore error? yes
[   18.479707] localhost fsck_or_mkfs.sh[320]: Force rewrite? yes
[   18.480228] localhost fsck_or_mkfs.sh[320]: Error reading block 360724 (Attempt to read block from filesystem resulted in short read) while getting next inode from scan.  Ignore error? yes
[   18.480716] localhost fsck_or_mkfs.sh[320]: Force rewrite? yes
[   18.481263] localhost fsck_or_mkfs.sh[320]: Error reading block 360725 (Attempt to read block from filesystem resulted in short read) while getting next inode from scan.  Ignore error? yes
[   18.481752] localhost fsck_or_mkfs.sh[320]: Force rewrite? yes
[   18.482352] localhost fsck_or_mkfs.sh[320]: Error reading block 360726 (Attempt to read block from filesystem resulted in short read) while getting next inode from scan.  Ignore error? yes
[   18.482854] localhost fsck_or_mkfs.sh[320]: Force rewrite? yes
[   18.483359] localhost fsck_or_mkfs.sh[320]: Error reading block 360727 (Attempt to read block from filesystem resulted in short read) while getting next inode from scan.  Ignore error? yes
[   18.483849] localhost fsck_or_mkfs.sh[320]: Force rewrite? yes
[   18.484414] localhost fsck_or_mkfs.sh[320]: Error reading block 360728 (Attempt to read block from filesystem resulted in short read) while getting next inode from scan.  Ignore error? yes
[   18.484917] localhost fsck_or_mkfs.sh[320]: Force rewrite? yes
[   18.485423] localhost fsck_or_mkfs.sh[320]: Error reading block 360729 (Attempt to read block from filesystem resulted in short read) while getting next inode from scan.  Ignore error? yes
[   18.485912] localhost fsck_or_mkfs.sh[320]: Force rewrite? yes
[   18.486415] localhost fsck_or_mkfs.sh[320]: Error reading block 360730 (Attempt to read block from filesystem resulted in short read) while getting next inode from scan.  Ignore error? yes
[   18.486914] localhost fsck_or_mkfs.sh[320]: Force rewrite? yes
[   18.487418] localhost fsck_or_mkfs.sh[320]: Error reading block 360731 (Attempt to read block from filesystem resulted in short read) while getting next inode from scan.  Ignore error? yes
[   18.487907] localhost fsck_or_mkfs.sh[320]: Force rewrite? yes
[   18.488547] localhost fsck_or_mkfs.sh[320]: Error reading block 360732 (Attempt to read block from filesystem resulted in short read) while getting next inode from scan.  Ignore error? yes
[   18.489050] localhost fsck_or_mkfs.sh[320]: Force rewrite? yes
[   18.489539] localhost fsck_or_mkfs.sh[320]: Error reading block 360733 (Attempt to read block from filesystem resulted in short read) while getting next inode from scan.  Ignore error? yes
[   18.490044] localhost fsck_or_mkfs.sh[320]: Force rewrite? yes
[   88.233932] localhost fsck_or_mkfs.sh[320]: Pass 2: Checking directory structure
[   88.242983] localhost fsck_or_mkfs.sh[320]: Pass 3: Checking directory connectivity
[   88.243583] localhost fsck_or_mkfs.sh[320]: Unconnected directory inode 8097 (/lost+found/???)
[   88.244088] localhost fsck_or_mkfs.sh[320]: Connect to /lost+found? yes
[   88.244588] localhost fsck_or_mkfs.sh[320]: Unconnected directory inode 161921 (/lost+found/???)
[   88.245089] localhost fsck_or_mkfs.sh[320]: Connect to /lost+found? yes
[   88.245620] localhost fsck_or_mkfs.sh[320]: Pass 4: Checking reference counts
[   88.246122] localhost fsck_or_mkfs.sh[320]: Inode 11 ref count is 2, should be 4.  Fix? yes
[   88.246609] localhost fsck_or_mkfs.sh[320]: Inode 8097 ref count is 3, should be 2.  Fix? yes
[   88.260220] localhost fsck_or_mkfs.sh[320]: Inode 161921 ref count is 3, should be 2.  Fix? yes
[   88.296785] localhost fsck_or_mkfs.sh[320]: Pass 5: Checking group summary information
[   88.592479] localhost fsck_or_mkfs.sh[320]: Free blocks count wrong for group #1 (31816, counted=18387).
[   88.592976] localhost fsck_or_mkfs.sh[320]: Fix? yes
[   88.593567] localhost fsck_or_mkfs.sh[320]: Free blocks count wrong for group #4 (32260, counted=26936).
[   88.594070] localhost fsck_or_mkfs.sh[320]: Fix? yes
[   88.594555] localhost fsck_or_mkfs.sh[320]: Free blocks count wrong for group #5 (31816, counted=30676).
[   88.595056] localhost fsck_or_mkfs.sh[320]: Fix? yes
[   88.595538] localhost fsck_or_mkfs.sh[320]: Free blocks count wrong for group #20 (32260, counted=32259).
[   88.596036] localhost fsck_or_mkfs.sh[320]: Fix? yes
[   88.596521] localhost fsck_or_mkfs.sh[320]: Free blocks count wrong (1780886, counted=1760992).
[   88.597023] localhost fsck_or_mkfs.sh[320]: Fix? yes
[   88.667189] localhost fsck_or_mkfs.sh[320]: Free inodes count wrong for group #1 (8096, counted=8090).
[   88.667748] localhost fsck_or_mkfs.sh[320]: Fix? yes
[   88.668248] localhost fsck_or_mkfs.sh[320]: Directories count wrong for group #1 (0, counted=1).
[   88.668749] localhost fsck_or_mkfs.sh[320]: Fix? yes
[   88.669322] localhost fsck_or_mkfs.sh[320]: Free inodes count wrong for group #20 (8096, counted=8095).
[   88.669863] localhost fsck_or_mkfs.sh[320]: Fix? yes
[   88.670418] localhost fsck_or_mkfs.sh[320]: Directories count wrong for group #20 (0, counted=1).
[   88.670971] localhost fsck_or_mkfs.sh[320]: Fix? yes
[   88.671523] localhost fsck_or_mkfs.sh[320]: Free inodes count wrong (453365, counted=453358).
[   88.672064] localhost fsck_or_mkfs.sh[320]: Fix? yes
[   97.754537] localhost fsck_or_mkfs.sh[320]: /dev/sda3: ***** FILE SYSTEM WAS MODIFIED *****
[   97.755047] localhost fsck_or_mkfs.sh[320]: /dev/sda3: 18/453376 files (11.1% non-contiguous), 52344/1813336 blocks
[   97.906315] localhost fsck_or_mkfs.sh[320]: fsck return value for /dev/bootflash3: 1
[   97.921284] localhost fsck_or_mkfs.sh[320]: Mounting /dev/bootflash3 at /tmp/fsck-or-mkfs-bootflash3
[  104.575612] localhost fsck_or_mkfs.sh[320]: mount: wrong fs type, bad option, bad superblock on /dev/sda3,
[  104.661611] localhost fsck_or_mkfs.sh[320]:        missing codepage or helper program, or other error
[  104.749596] localhost fsck_or_mkfs.sh[320]:        In some cases useful info is found in syslog - try
[  104.803572] localhost fsck_or_mkfs.sh[320]:        dmesg | tail or so.
[  104.804238] localhost fsck_or_mkfs.sh[320]: Creating filesystem with 1813336 4k blocks and 453376 inodes
[  104.804731] localhost fsck_or_mkfs.sh[320]: Filesystem UUID: efd161b9-4d74-4aa0-bdf4-20ef15d013d0
[  104.805230] localhost fsck_or_mkfs.sh[320]: Superblock backups stored on blocks:
[  104.805715] localhost fsck_or_mkfs.sh[320]:         32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632
[  104.806216] localhost fsck_or_mkfs.sh[320]: [49B blob data]
[  104.806701] localhost fsck_or_mkfs.sh[320]: [46B blob data]
[  104.872442] localhost fsck_or_mkfs.sh[320]: ERROR: couldn't mount /dev/bootflash3
[  104.934443] localhost fsck_or_mkfs.sh[320]: Executing mkfs on /dev/bootflash3
[  104.992442] localhost fsck_or_mkfs.sh[320]: mke2fs 1.43-WIP (18-May-2015)
[  118.725106] localhost fsck_or_mkfs.sh[320]: [83B blob data]
[  127.627236] localhost fsck_or_mkfs.sh[320]: mount: wrong fs type, bad option, bad superblock on /dev/sda3,
[  127.627515] localhost fsck_or_mkfs.sh[320]:        missing codepage or helper program, or other error
[  127.627758] localhost fsck_or_mkfs.sh[320]:        In some cases useful info is found in syslog - try
[  127.628022] localhost fsck_or_mkfs.sh[320]:        dmesg | tail or so.
[  127.628288] localhost fsck_or_mkfs.sh[320]: Reformat failed
[  127.628932] localhost systemd[1]: systemd-fsck@dev-bootflash3.service: Main process exited, code=exited, status=32/n/a
[  127.629779] localhost systemd[1]: Failed to start File System Check on /dev/bootflash3.
[  127.687527] localhost systemd[1]: systemd-fsck@dev-bootflash3.service: Unit entered failed state.
[  127.688026] localhost systemd[1]: systemd-fsck@dev-bootflash3.service: Failed with result 'exit-code'.
Warning: systemd-fsck@dev-bootflash3.service changed on disk. Run 'systemctl daemon-reload' to reload units.
>>> Rebooting
Failed to connect to bus: No such file or directory
Failed to connect to bus: No such file or directory
watchdog: watchdog0: watchdog did not stop!
reboot: Restarting system

[Leo Laohoo]

Contact Cisco TAC and organize an RMA (before end-of-support date).  

The switch's flash is dead.