CSCue10556 - standby reload due to Bulk-sync failure due to PRC mismatch

SaravanakumarR6943 · ‎06-22-2021

Hi All,

Getting following error frequently and device is not forming redundancy.

Its WS-C6509-E Model.

As per Cisoc Bug tool, it had mentioned the following reason

standby reload due to Bulk-sync failure due to PRC mismatch

CSCue10556

Anyone please validate the logs and share the next plan of action to fix this issue.

Logs:

-

032590: Jun 22 17:11:49.674 UTC: Config Sync: Bulk-sync failure due to PRC mismatch. Please check the full list of PRC failures via:
show redundancy config-sync failures prc

032591: Jun 22 17:11:49.674 UTC: Config Sync: Starting lines from PRC file:
archive
! <submode> "archive"
- path disk0://$h_
- maximum 14
- write-memory
- time-period 2880
! </submode> "archive"

032592: Jun 22 17:11:49.678 UTC: Config Sync: Bulk-sync failure, Reloading Standby

032593: Jun 22 17:11:50.422 UTC: %RF-SP-5-RF_RELOAD: Peer reload. Reason: Proxy request to reload peer
032594: Jun 22 17:11:50.422 UTC: %OIR-SP-3-PWRCYCLE: Card in module 6, is being power-cycled 'Proxy request to reload peer'
032595: Jun 22 17:11:51.426 UTC: %PFREDUN-SP-6-ACTIVE: Standby processor removed or reloaded, changing to Simplex mode
032596: Jun 22 17:11:52.974 UTC: %SNMP-5-MODULETRAP: Module 6 [Down] Trap
032597: Jun 22 17:14:50.520 UTC: %PFREDUN-SP-6-ACTIVE: Standby initializing for SSO mode
032598: Jun 22 17:14:50.716 UTC: %SYS-SP-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging output.
032599: Jun 22 17:15:01.980 UTC: %PFINIT-SP-5-CONFIG_SYNC: Sync'ing the startup configuration to the standby Router.
032600: Jun 22 17:15:37.752 UTC: %FABRIC-SP-5-CLEAR_BLOCK: Clear block option is off for the fabric in slot 6.
032601: Jun 22 17:15:37.752 UTC: %FABRIC-SP-5-FABRIC_MODULE_BACKUP: The Switch Fabric Module in slot 6 became standby
032602: Jun 22 17:15:52.804 UTC: %DIAG-SP-6-RUN_COMPLETE: Module 6: Running Complete Diagnostics...
032603: Jun 22 17:15:54.652 UTC: %DIAG-SP-6-DIAG_OK: Module 6: Passed Online Diagnostics
032604: Jun 22 17:15:57.184 UTC: %SNMP-5-MODULETRAP: Module 6 [Up] Trap
032605: Jun 22 17:15:57.200 UTC: %OIR-SP-6-INSCARD: Card inserted in slot 6, interfaces are now online
032606: *Jun 22 17:13:52.031: %SYS-SPSTBY-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging output.
032607: *Jun 22 17:15:23.904: %SPANTREE-SPSTBY-5-EXTENDED_SYSID: Extended SysId enabled for type vlan. The Bridge IDs of all active STP instances have been updated, which might change the spanning tree topology
*Jun 22 17:15:23.916: SPSTBY: SP: Currently running ROMMON from S (Gold) region
032608: Jun 22 17:15:40.632: %DIAG-SPSTBY-6-RUN_COMPLETE: Module 6: Running Complete Diagnostics...
032609: Jun 22 17:15:52.544: %DIAG-SPSTBY-6-DIAG_OK: Module 6: Passed Online Diagnostics
032610: Jun 22 17:16:40.964: %SYS-SPSTBY-5-RESTART: System restarted --
Cisco IOS Software, s72033_sp Software (s72033_sp-ADVIPSERVICESK9_WAN-M), Version 12.2(33)SXI8a, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2011 by Cisco Systems, Inc.
Compiled Sat 03-Dec-11 09:00 by prod_rel_team
032611: Jun 22 17:16:58.013 UTC: %CHKPT-4-GET_HUGE_BUF: Client 162 buffer requested (size = 4256) is too large
032612: Jun 22 17:17:00.021 UTC: Config Sync: Bulk-sync failure due to PRC mismatch. Please check the full list of PRC failures via:
show redundancy config-sync failures prc

032613: Jun 22 17:17:00.021 UTC: Config Sync: Starting lines from PRC file:
archive
! <submode> "archive"
- path disk0://$h_
- maximum 14
- write-memory
- time-period 2880
! </submode> "archive"

032614: Jun 22 17:17:00.021 UTC: Config Sync: Bulk-sync failure, Reloading Standby

032615: Jun 22 17:17:00.769 UTC: %RF-SP-5-RF_RELOAD: Peer reload. Reason: Proxy request to reload peer
032616: Jun 22 17:17:00.769 UTC: %OIR-SP-3-PWRCYCLE: Card in module 6, is being power-cycled 'Proxy request to reload peer'
032617: Jun 22 17:17:01.781 UTC: %PFREDUN-SP-6-ACTIVE: Standby processor removed or reloaded, changing to Simplex mode
032618: Jun 22 17:17:03.373 UTC: %SNMP-5-MODULETRAP: Module 6 [Down] Trap
032619: Jun 22 17:20:01.242 UTC: %PFREDUN-SP-6-ACTIVE: Standby initializing for SSO mode
032620: Jun 22 17:20:01.434 UTC: %SYS-SP-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging output.
032621: Jun 22 17:20:12.774 UTC: %PFINIT-SP-5-CONFIG_SYNC: Sync'ing the startup configuration to the standby Router.
032622: Jun 22 17:20:48.494 UTC: %FABRIC-SP-5-CLEAR_BLOCK: Clear block option is off for the fabric in slot 6.
032623: Jun 22 17:20:48.494 UTC: %FABRIC-SP-5-FABRIC_MODULE_BACKUP: The Switch Fabric Module in slot 6 became standby
032624: Jun 22 17:21:03.875 UTC: %DIAG-SP-6-RUN_COMPLETE: Module 6: Running Complete Diagnostics...
032625: Jun 22 17:21:08.111 UTC: %DIAG-SP-6-DIAG_OK: Module 6: Passed Online Diagnostics
032626: Jun 22 17:21:10.931 UTC: %SNMP-5-MODULETRAP: Module 6 [Up] Trap
032627: Jun 22 17:21:10.935 UTC: %OIR-SP-6-INSCARD: Card inserted in slot 6, interfaces are now online
032628: *Jun 22 17:19:02.607: %SYS-SPSTBY-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging output.
032629: *Jun 22 17:20:34.598: %SPANTREE-SPSTBY-5-EXTENDED_SYSID: Extended SysId enabled for type vlan. The Bridge IDs of all active STP instances have been updated, which might change the spanning tree topology
*Jun 22 17:20:34.614: SPSTBY: SP: Currently running ROMMON from S (Gold) region
032630: Jun 22 17:20:51.214: %DIAG-SPSTBY-6-RUN_COMPLETE: Module 6: Running Complete Diagnostics...
032631: Jun 22 17:21:03.699: %DIAG-SPSTBY-6-DIAG_OK: Module 6: Passed Online Diagnostics
032632: Jun 22 17:21:51.437: %SYS-SPSTBY-5-RESTART: System restarted --
Cisco IOS Software, s72033_sp Software (s72033_sp-ADVIPSERVICESK9_WAN-M), Version 12.2(33)SXI8a, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2011 by Cisco Systems, Inc.
Compiled Sat 03-Dec-11 09:00 by prod_rel_team
032633: Jun 22 17:22:08.359 UTC: %CHKPT-4-GET_HUGE_BUF: Client 162 buffer requested (size = 4256) is too large
032634: Jun 22 17:22:09.951 UTC: Config Sync: Bulk-sync failure due to PRC mismatch. Please check the full list of PRC failures via:
show redundancy config-sync failures prc

Thank you in advance.

Aleksandar Aleksiev · ‎11-30-2023

Hi,

It seems like this is almost obsolete topic... However, I'll share some experience here that can help somebody with similar problem.

First to address the reload-loop issue. The above reload is reported as caused by bulk-sync mismatch:
032612: Jun 22 17:17:00.021 UTC: Config Sync: Bulk-sync failure due to PRC mismatch. Please ...
... ...
032614: Jun 22 17:17:00.021 UTC: Config Sync: Bulk-sync failure, Reloading Standby

That should never happen as long as you have in your redundancy configuration (could be the default one) such a reload policy disabled like that:

#sh runn all | sec ^redun
redundancy
main-cpu
auto-sync running-config
mode sso
no policy config-sync lbl prc reload
no policy config-sync bulk prc reload
#

If it's still reloading, that simply means a BUG and you should contact Cisco.

Now, the actual root cause for the mismatch. It's visible in the log here:
032613: Jun 22 17:17:00.021 UTC: Config Sync: Starting lines from PRC file:
archive
! <submode> "archive"
- path disk0://$h_
- maximum 14
- write-memory
- time-period 2880
! </submode> "archive"

For some reason, the above part of the configuration differs on the slave-system and together with the above BUG is causing a reload-loop that's not allowing you to troubleshoot further.
The simplest way to avoid the mismatch is to remove the above part of the configuration on the active-system. Like that:
conf t
archive
no path disk0://$h_
no maximum 14
no write-memory
no time-period 2880
end

Now, if there are no other mismatches, the standby-system will start correctly and allow you to troubleshoot the reason of the missing config on it.

The most probably reason is that slavedisk0: do not exist and as a result the configuration cannot be applied.
That situation can happen relatively easy, since disk0: is the CF (compact flash) on the front of the SUP card and any physical intervention can partially remove it in making it useless. It can also be damaged...

I hope that will help.
Cheers !

compass131 · ‎11-03-2024

I had the same problem with ASR1009,

*Oct 31 09:01:18.075: Config Sync: Bulk-sync failure due to PRC mismatch. Please check the full list of PRC failures via:
show redundancy config-sync failures prc

*Oct 31 09:01:18.075: Config Sync: Starting lines from PRC file:
interface GigabitEthernet0
! <submode> "interface"
- vrf forwarding Mgmt-intf
! </submode> "interface"

Using the following command:

redundancy
main-cpu
auto-sync running-config
mode sso
no policy config-sync lbl prc reload
no policy config-sync bulk prc reload