%IOSXE-3-PLATFORM: Switch 2 R0/0: kernel: mmcblk0: recovery failed!

[rasmus.elmholt]

Hi

I have a 9300 stack where the log is filled with the following messages:

262866921: Jul 10 08:46:43.188: %IOSXE-3-PLATFORM: Switch 2 R0/0: kernel: mmcblk0: recovery failed!
262866922: Jul 10 08:46:43.195: %IOSXE-3-PLATFORM: Switch 2 R0/0: kernel: sdhci-pci 0000:00:1c.0: error -110 requesting status
262866923: Jul 10 08:46:43.195: %IOSXE-3-PLATFORM: Switch 2 R0/0: kernel: mmcblk0: recovery failed!
262866924: Jul 10 08:46:43.202: %IOSXE-3-PLATFORM: Switch 2 R0/0: kernel: sdhci-pci 0000:00:1c.0: error -110 requesting status
262866925: Jul 10 08:46:43.202: %IOSXE-3-PLATFORM: Switch 2 R0/0: kernel: mmcblk0: recovery failed!
262866926: Jul 10 08:46:43.209: %IOSXE-3-PLATFORM: Switch 2 R0/0: kernel: sdhci-pci 0000:00:1c.0: error -110 requesting status
262866927: Jul 10 08:46:43.209: %IOSXE-3-PLATFORM: Switch 2 R0/0: kernel: mmcblk0: recovery failed!
262866928: Jul 10 08:46:43.209: %BTRACE_MANAGER-3-BTRT_ERR_MSG: Switch 2 R0/0: btman: Btrace Rotate Error: Move file failed, Delete file: chasync_R0-0.6600_3356106.20230710064606.bin.gz
262866929: Jul 10 08:46:43.217: %IOSXE-3-PLATFORM: Switch 2 R0/0: kernel: sdhci-pci 0000:00:1c.0: error -110 requesting status
262866930: Jul 10 08:46:43.217: %IOSXE-3-PLATFORM: Switch 2 R0/0: kernel: mmcblk0: recovery failed!
262866931: Jul 10 08:46:43.224: %IOSXE-3-PLATFORM: Switch 2 R0/0: kernel: sdhci-pci 0000:00:1c.0: error -110 requesting status
262866932: Jul 10 08:46:43.224: %IOSXE-3-PLATFORM: Switch 2 R0/0: kernel: mmcblk0: recovery failed!
262866933: Jul 10 08:46:43.231: %IOSXE-3-PLATFORM: Switch 2 R0/0: kernel: sdhci-pci 0000:00:1c.0: error -110 requesting status
262866934: Jul 10 08:46:43.231: %IOSXE-3-PLATFORM: Switch 2 R0/0: kernel: mmcblk0: recovery failed!

Does anyone know what the cause of this is, and how to fix it?

It is a 9300 stack running 17.3.4 and missing the last switch in the stack.

Cisco IOS Software [Amsterdam], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 17.3.4

#show switch 
Switch/Stack Mac Address : c4b2.****.**** - Local Mac Address
Mac persistency wait time: Indefinite
                                             H/W   Current
Switch#   Role    Mac Address     Priority Version  State 
-------------------------------------------------------------------------------------
*1       Active   c4b2.****.****     15     V01     Ready                               
 2       Standby  c4b2.****.****     14     V01     Ready                               
 3       Member   0000.****.****     0      V01     Unprovisioned                       

[marce1000]

 - Smells like a hardware problem ; 

sdhci-pci

   stands for   Secure Digital Host Controller Interface PCI driver ; don't let your sleep about it , only meaning when the device gives that kind of internal messaging it points to hardware issues (usually) . Ways to go forward are : power down and cold boot and follow up on the self test errors (if production conditions permit). Also look into the latest advisory release : https://software.cisco.com/download/home/286326357/type/282046477/release/Cupertino-17.9.3  , and check if upgrading can help (if it would only be a bug)

 M.

[rasmus.elmholt]

And how do I see the POST messages if I reboot switch 2 without having to connect with a console cable?

[Leo Laohoo]

Error messages are coming from Switch 2.  
Reboot Switch 2 or upgrade the entire stack to 17.9.3.

[rasmus.elmholt]

This is the number of the different Errors i have gotten from this stack today:

 255466 IOSXE-3-PLATFORM: Switch 2 R0/0: kernel: sdhci-pci 0000:00:1c.0: error -110 requesting status
 255444 IOSXE-3-PLATFORM: Switch 2 R0/0: kernel: mmcblk0: recovery failed!
    100 IOSXE-3-PLATFORM: Switch 2 R0/0: kernel: mmc0: cache flush error -110
    100 IOSXE-3-PLATFORM: Switch 2 R0/0: kernel: EXT2-fs (mmcblk0p3): previous I/O error to superblock detected
    100 IOSXE-2-PLATFORM: Switch 2 R0/0: kernel: EXT2-fs (mmcblk0p3): error: ext2_fsync: detected IO error when writing metadata buffers
     50 IOSXE-3-PLATFORM: Switch 2 R0/0: kernel: EXT2-fs (mmcblk0p1): previous I/O error to superblock detected
     50 IOSXE-2-PLATFORM: Switch 2 R0/0: kernel: EXT2-fs (mmcblk0p1): error: ext2_fsync: detected IO error when writing metadata buffers

[marce1000]

                    >...Errors i have gotten from this stack today:
  - Flash is corrupt or getting corrupt : please note , in worst case the system may fail a next reboot (attempt)  , 

 M.