4510 Standby Sup 7-E will not come back online

[jedavis]

This switch is running 3.2.2.SG and I was trying to upgrade to 3.8.6.E.  Even though these are not ISSU compatible images I used ISSU loadversion to start the process on the standby supervisor.  It never came back.  It appeared to be stuck in a reboot loop.

Nov 10 04:19:21: %C4K_REDUNDANCY-3-COMMUNICATION: Communication with the peer Supervisor has been established
Nov 10 04:19:33: %C4K_REDUNDANCY-5-CONFIGSYNC: The bootvar has been successfully synchronized to the standby supervisor
Nov 10 04:19:33: %C4K_REDUNDANCY-5-CONFIGSYNC: The config-reg has been successfully synchronized to the standby supervisor
Nov 10 04:19:33: %C4K_REDUNDANCY-5-CONFIGSYNC: The startup-config has been successfully synchronized to the standby supervisor
Nov 10 04:19:33: %C4K_REDUNDANCY-5-CONFIGSYNC: The private-config has been successfully synchronized to the standby supervisor
Nov 10 04:19:35: %C4K_REDUNDANCY-5-CONFIGSYNC_RATELIMIT: The vlan database has been successfully synchronized to the standby supervisor
Nov 10 04:20:04: Config Sync: Bulk-sync failure due to PRC mismatch. Please check the full list of PRC failures via:
 show redundancy config-sync failures prc

Nov 10 04:20:04: Config Sync: Starting lines from PRC file:
-ntp clock-period 17495702

Nov 10 04:20:04: Config Sync: Bulk-sync failure, Reloading Standby

I connected to the standby console port and interrupted the boot process.  I then loaded the old image from rommon.  And this is what happens:

loading image

Checking digital signature
bootflash:/cat4500e-universalk9.SPA.03.02.02.SG.150-2.SG2.bin: Digitally Signed Release Software with key version A

Rommon reg: 0x00004F80
Reset2Reg: 0x00000F00

Image load status: 0x00000000
####
Snowtrooper 220 controller 0x042F2975..0x0447E526 Size:0x0056CB6D Program Done!
#############
Linux version 2.6.24.4.51.23.k10 (susingh@build-lnx-005) (gcc version 4.2.1 p1 (Cisco c4.2.1-p1)) #1 SMP Wed Dec 7 19:14:43 PST 2011
Starting System Services

diagsk10-post version 5.0.3

prod: WS-X45-SUP7-E part: 73-12064-08 serial: CAT1542L5K9


Power-on-self-test for Module 6: WS-X45-SUP7-E

CPU Subsystem Tests ...
 seeprom: Pass

Traffic: L3 Loopback ...
 Test Results: Pass

Traffic: L2 Loopback ...
 Test Results: Pass
post done
Exiting to ios...
Starting IOS Services

Restricted Rights Legend

Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
(c) of the Commercial Computer Software - Restricted
Rights clause at FAR sec. 52.227-19 and subparagraph
(c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.

cisco Systems, Inc.
 170 West Tasman Drive
 San Jose, California 95134-1706

 

Cisco IOS Software, IOS-XE Software, Catalyst 4500 L3 Switch Software (cat4500e-UNIVERSALK9-M), Version 03.02.02.SG RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2011 by Cisco Systems, Inc.
Compiled Wed 07-Dec-11 19:55 by prod_rel_team

 

Cisco IOS-XE software, Copyright (c) 2005-2010 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0. For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.

 

Image text-base: 0x100A17F4, data-base: 0x13F1C3E0

 

***********************************
* STANDBY SUPERVISOR *
* REDUNDANCY mode is SSO *
* Continue bootup *
***********************************

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco WS-C4510R+E (MPC8572) processor (revision 8) with 2097152K/20480K bytes of memory.
Processor board ID FOX1539GTUS
MPC8572 CPU at 1.5GHz, Supervisor 7
Last reset from Reset by peer
120 Gigabit Ethernet interfaces
8 Ten Gigabit Ethernet interfaces
511K bytes of non-volatile configuration memory.

Warning: bootflash:/c2960x-universalk9-tar.152-4.E6.tar does not exist. Command retained.
%NTP: unreasonable value


Press RETURN to get started!


*Nov 10 16:06:41.193: %C4K_IOSSYS-6-IMAGELEVEL: Supervisor booting in image level 'entservices'
*Nov 10 16:06:51.270: %C4K_REDUNDANCY-6-INIT: STANDBY:Initializing as STANDBY supervisor
*Nov 10 16:06:56.298: %C4K_REDUNDANCY-6-DUPLEX_MODE: STANDBY:The peer Supervisor has been detected
*Nov 10 16:06:56.307: %C4K_REDUNDANCY-3-COMMUNICATION: STANDBY:Communication with the peer Supervisor has been established
Please stand by while rebooting the system...STANDBY:STANDBY supervisor initializing for sso mode
 Restarting system.

At this point it boots the new version and I am back to where I started.  I cleared the boot variable in rommon and tried it again but it made no difference.  

After doing a little research I think this may be a manifestation of bug CSCua74618 which contains a workaround.  I just need to make some configuration changes.  However, anytime I enter anything in configuration mode I get this:

Switch(config)#god help us all!!!!!!
Config mode locked out until standby initializes

configuration mode locked.'Please try later.'

 Anyone know how to solve this paradox?  Any way to break the config lock?

Thanks!

-Jeff

[Georg Pauwen]

Hello,

not sure if this is helpful, but in the reviews of this release, another user described the problem below. Does this apply to you ?

-->"The code is very stable once the switch is running it however when upgrading from an old XO code, there are many issues. In order to upgrade from 03.03.02.XO you will have to boot the switch to ROMON and then point to the new code - 03.08.06.E on both sups. Once that is completed you will then have to delete the old iOS from Flash as it will point to the old code if rebooted."<--

[jedavis]

Thanks for the reply Georg.

It may be relevant, I am not sure.  I didn't see a delete command in the rommon command list.  And I can't get to the standby flash from IOS because I can't get this thing to stay loaded and It doesn't boot far enough to reach it from the active supervisor.  I will keep it in mind if I have issues after I get past this config lock problem.

[Georg Pauwen]

Hello,

did you try to pull both supervisors, possibly power of the entire chassis, and reseat the supervisors ?

[jedavis]

This is the production core switch in a manufacturing facility in England.  I am in the US.  I did not power down my lone remaining supervisor.  I need to get my resume updated before I try that,

[balaji.bandi]

just wan anxiety looking at your boot.

Warning: bootflash:/c2960x-universalk9-tar.152-4.E6.tar does not exist. Command retained.

why this image, when you working on 4510? 

[jedavis]

I was just using the 4510 as a tftp server to distribute that image to the 2960X access switches at the site.  It did exist on the active supervisor bootflash.

[jedavis]

Thanks for the reply BB.

Switch#show redundancy states
 my state = 13 -ACTIVE
 peer state = 1 -DISABLED
 Mode = Duplex
 Unit = Primary
 Unit ID = 5

Redundancy Mode (Operational) = Stateful Switchover
 Redundancy Mode (Configured) = Stateful Switchover
 Redundancy State = Stateful Switchover
 Manual Swact = disabled (the peer unit is still initializing)

Communications = Down Reason: Failure

client count = 66
 client_notification_TMR = 240000 milliseconds
 keep_alive TMR = 9000 milliseconds
 keep_alive count = 0
 keep_alive threshold = 18
 RF debug mask = 0

---

Switch#show redundancy
Redundant System Information :

------------------------------
 Available system uptime = 1 year, 30 weeks, 1 day, 6 hours, 28 minutes
Switchovers system experienced = 0
 Standby failures = 57
 Last switchover reason = none

Hardware Mode = Duplex
 Configured Redundancy Mode = Stateful Switchover
 Operating Redundancy Mode = Stateful Switchover
 Maintenance Mode = Disabled
 Communications = Down Reason: Failure

Current Processor Information :
------------------------------
 Active Location = slot 5
 Current Software state = ACTIVE
 Uptime in current state = 1 year, 30 weeks, 1 day, 6 hours, 25 minutes
 Image Version = Cisco IOS Software, IOS-XE Software, Catalyst 4500 L3 Switch Software (cat4500e-UNIVERSALK9-M), Version 03.02.02.SG RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2011 by Cisco Systems, Inc.
Compiled Wed 07-Dec-11 19:55 by prod
 BOOT = bootflash:/cat4500e-universalk9.SPA.03.08.06.E.152-4.E6.bin,12;
 Configuration register = 0x102

Peer (slot: 6) information is not available because it is in 'DISABLED' state

 

---

Switch#show module
Chassis Type : WS-C4510R+E

Power consumed by backplane : 40 Watts

Mod Ports Card Type Model Serial No.
---+-----+--------------------------------------+------------------+-----------
 1 12 1000BaseX (SFP) WS-X4612-SFP-E JAE154608EW
 2 12 1000BaseX (SFP) WS-X4612-SFP-E JAE1546086T
 5 4 Sup 7-E 10GE (SFP+), 1000BaseX (SFP) WS-X45-SUP7-E CAT1543L00H
 6 Supervisor
 7 48 10/100/1000BaseT (RJ45) WS-X4648-RJ45-E JAE1539057U
 9 48 10/100/1000BaseT (RJ45) WS-X4648-RJ45-E JAE15390561

M MAC addresses Hw Fw Sw Status
--+--------------------------------+---+------------+----------------+---------
 1 649e.f31a.4f44 to 649e.f31a.4f4f 1.1 Ok
 2 70ca.9b13.587c to 70ca.9b13.5887 1.1 Ok
 5 ccef.481e.2200 to ccef.481e.2203 1.0 15.0(1r)SG2 03.02.02.SG Ok
 6 Unknown Unknown Unknown Other
 7 ccef.483a.96c2 to ccef.483a.96f1 1.0 Ok
 9 ccef.483a.9b8a to ccef.483a.9bb9 1.0 Ok

Mod Redundancy role Operating mode Redundancy status
----+-------------------+-------------------+----------------------------------
 5 Active Supervisor SSO Active
 6 Standby Supervisor SSO Disabled

[CWolf1]

---

@jedavis wrote:

Warning: bootflash:/c2960x-universalk9-tar.152-4.E6.tar does not exist. Command retained. 

---

Why is there a boot variable statement instructing the Sup7 to boot the IOS meant for a 2960X?

---

@jedavis wrote:

This switch is running 3.2.2.SG and I was trying to upgrade to 3.8.6.E.  Even though these are not ISSU compatible images I used ISSU loadversion to start the process on the standby supervisor.  It never came back.  It appeared to be stuck in a reboot loop. 

---

Kindly read the Release Notes.  Straight upgrade from 3.2.X to 3.8.X is not possible because a ROMmon upgrade is required first before the IOS can be upgraded.

What IOS version did the primary supervisor card boot up in?

[jedavis]

There is no boot variable pointing to a 2960X image.  THe configuraton statement referencing that image is:

tftp-server bootflash:/c2960x-universalk9-tar.152-4.E6.tar alias ios

This seems to be causing some confusion.  I was simply using the 4500 as a tftp server to upgrade 2960X's at the site.  

As to reading the release notes you have a point.  I did not notice this very important note:

• If you are upgrading to Cisco IOS XE Release 3.8.xE and using Supervisor Engine 7-E or 7L-E, you must use ROMMON version 15.0(1r)SG10 or a higher version (if available).

I have 15.0(1r)SG2.  15.0(1r)SG15 is available.

Thank you for pointing that out.  I will certainly perform that upgrade just as soon as I figure out how to get the standby supervisor back online. 

[balaji.bandi]

Glad to hear you pointed our and nailed down the issue to working solution soon.

post us the update how it goes.

[jedavis]

I still don't have a path forward with this.  In order to get the standby supervisor up and running I need to make some configuration changes.  But the configuration is locked.  Is there any way to get around this configuration lock?

Switch(config)#I should have been a lawyer
Config mode locked out until standby initializes

configuration mode locked.'Please try later.'
Switch(config)#

Do you suppose if we slide the standby out of the chassis it will kick the config mode open?

[balaji.bandi]

 This error message is usually seen when standby sup is trying to reach RPR or SSO redundancy. It prevents config changes until standby sup is done syncing with active sup.

Can you post both the chasis below output :

show redundancy states

show redundancy

show modules

[jedavis]

BB, please see my response above

[jedavis]

OK, progress.  I reset the standby supervisor using "redundancy reload peer".  Then I interrupted the boot process using Ctrl-C from the standby console and put it into rommon.  I was then able to access config mode and remove the "ntp clock period" statement.  Removing this allowed me to boot 3.2.2SG into SSO mode.

So now that it was recovered I upgraded the standby rommon to 15.0(1r)SG15.  Then I booted it with the 3.8.6E image.  This still never establishes communication with the active supervisor.  On the standby console this is the last sequence of messages that I see:

Exiting to ios...
Loading gsbu64atomic as gdb64atomic
Loading isp1362_hcd_k10
Using 6 for MTS slot
Platform Manager: starting in standalone mode (standby)

And on the active console:

Nov 14 15:32:27: %C4K_REDUNDANCY-6-DUPLEX_MODE: The peer Supervisor has been detected
Nov 14 15:34:27: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
(HANDSHAKE_TIMEOUT message repeated every 5 minutes)

I didn't expect it to come up in SSO mode, but I at least expected it to come up in RPR mode.  I even tried to set the redundancy mode to RPR explicitly but it didn't change the behavior.

Switch(config-red)#do sho red states
 my state = 13 -ACTIVE
 peer state = 1 -DISABLED
 Mode = Duplex
 Unit = Primary
 Unit ID = 5

Redundancy Mode (Operational) = RPR
 Redundancy Mode (Configured) = RPR
 Redundancy State = RPR
 Manual Swact = disabled (the peer unit is still initializing)

Communications = Down Reason: Failure

client count = 66
 client_notification_TMR = 240000 milliseconds
 keep_alive TMR = 9000 milliseconds
 keep_alive count = 0
 keep_alive threshold = 18
 RF debug mask = 0

 Changing the redundancy mode to SSO and booting back to 3.2.2SG brings it up as "peer state =  8 -STANDBY HOT".

Is the initialization failure expected with the mismatched IOS?  If I forced switchover would the standby complete initialization?  I am very hesitant to do this since this is a production switch and I don't have physical access to it.  Release notes do not address this condition.

Thanks for all the help!

-Jeff