Solved: Re: Replaced standby SUP-7L-E boot as standalone

Sergej Barkovski · ‎07-02-2021

ello,

c4507R+E chassis had two Sup-7L-E as active and standby units. The standby SUP failed and was replaced with the same model. After the replacement SUP was started it bootet till this moment:

Loading image !!!!!!!!!!!!!!!!

Checking digital signature....

[slot0:cat4500e-universalk9.SPA.03.04.05.SG.151-2.SG5.bin]

Digitally Signed Release Software with key version A

Rommon reg: 0x00004F80

Reset2Reg: 0x00000F00

*

Image load status: 0x00000003

####

Radtrooper 065 controller 0x0481ADD2..0x049B696C Size:0x00661EDC Program Done!

##############

Linux version 2.6.24.4.3.70.k10 (abhakat@build-lnx-036) (gcc version 4.2.1 p7 (Cisco c4.2.1-p7)) 4

Starting System Services

diagsk10-post version 5.1.4.0

prod: WS-X45-SUP7L-E part: 73-14453-05 serial: CAT1827L2Z3

Power-on-self-test for Module 3: WS-X45-SUP7L-E

CPU Subsystem Tests ...

seeprom: Pass

Traffic: L3 Loopback ...

Test Results: Pass

Traffic: L2 Loopback ...

Test Results: Pass

post done

Exiting to ios...

Starting IOS Services

Platform Manager: acquire hwlock chassis()

Platform Manager: starting in standalone mode

After that nothing happenes and the active SUP shown in "show module" only "Supervisor" without model, type, MAC-Adresses (screenshot attached).

The romon version on active and standby replacement are the same - 15.0.1(r)SG10. The IOS XE on the Active-

Cisco IOS Software, IOS-XE Software, Catalyst 4500 L3 Switch Software (cat4500e-UNIVERSALK9-M), Version 03.04.05.SG RELEASE SOFTWARE (fc1).

And on the replacement standby it was first 03.04.04 without encryption (without K9 in the image name).

It was not possible to copy the same IOS XE image as on the Active to the bootflash of the Standby. We copied the 03.04.05 image on a SD card and set it to the slot0 of the standby Sup. We stopped the boot process and set the boot image to boot slot0:cat4500e-universalk9.SPA.03.04.05.SG.151-2.SG5.bin (the same image as on teh active sup). But it also ended with the message - "starting in standalone mode" (see the output above)

Any ideas, what did we do wrong?

Thanks a lot.

Best regards,

Sergej

marce1000 · ‎07-05-2021

- https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst4500/12-2/37sg/system/messages/message/emsg.html#wp1869509

(correction)

M.

-- Each morning when I wake up and look into the mirror I always say ' Why am I so brilliant ? '
When the mirror will then always repond to me with ' The only thing that exceeds your brilliance is your beauty! '

View solution in original post

marce1000 · ‎07-02-2021

- FYI : https://www.cisco.com/c/en/us/support/docs/switches/catalyst-4500-series-switches/24052-165.html

https://www.cisco.com/c/en/us/support/docs/switches/catalyst-4500-series-switches/24052-165.html#standby

M.

-- Each morning when I wake up and look into the mirror I always say ' Why am I so brilliant ? '
When the mirror will then always repond to me with ' The only thing that exceeds your brilliance is your beauty! '

Sergej Barkovski · ‎07-05-2021

Hello Marce1000,

yes, I checked this document at the start of our investigations. I cannot imagine it could be a hardware problem, because we have the same behaviour with two different replacement supervisor engines of teh same model.

The second replacement has the same rommon and ios at the boot as teh primary. But it did not change anything in the issue.

The replacement sup does not have configuration on it. It should get it from active, after the boot and sync with the active unit.

Best regards,

Sergej

Leo Laohoo · ‎07-02-2021

@Sergej Barkovski wrote:

[slot0:cat4500e-universalk9.SPA.03.04.05.SG.151-2.SG5.bin]

I can see the replacement card is trying to boot 3.4.5. POST indicates everything is fine.

Console into the ACTIVE Supervisor and post the complete output to the command "sh logs".
NOTE: DO not "over edit' the output. Remove the IP addresses, Serial Numbers, Host names but no more.

Sergej Barkovski · ‎07-05-2021

Hello Leo,

it takes some time till I got the fresh show log. But I have a screenshot of the show logging and some others, as the first replacement supervior engine was inserted and bootet. Take a look into the screenshots made on teh active unit, I attached.

I'll share the fresh outputs, as soon as I got them.

Thank you.

Best regards,

Sergej

marce1000 · ‎07-05-2021

- https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst4500/12-2/37sg/system/messages/message/emsg.html#wp1869509

(correction)

M.

-- Each morning when I wake up and look into the mirror I always say ' Why am I so brilliant ? '
When the mirror will then always repond to me with ' The only thing that exceeds your brilliance is your beauty! '

Sergej Barkovski · ‎07-05-2021

Hello marce1000,

than you for the hint. As soon as I have the fresh logs I'll check, if they are the same. If yes, I'll suggest this recovery procedure.

Just for my understanding, does the "reset the active superviser" means to reload it, as the standby sup still seats in the chassis, or should the standby first be removed from the chassis, and the active reloaded only after that?

Thank you.

Best reagrds,

Sergej

marce1000 · ‎07-05-2021

- I would assume the standby sup to remain in place, since a communication-problem between the two is denoted.

M

-- Each morning when I wake up and look into the mirror I always say ' Why am I so brilliant ? '
When the mirror will then always repond to me with ' The only thing that exceeds your brilliance is your beauty! '

Sergej Barkovski · ‎07-13-2021

Hello,

we will plan a maintenanace window and try the solution from cisco documentation. I'll update in a couple of weeks, if it helped.

Best regards,

Sergej

Sergej Barkovski · ‎07-23-2021

Hello Marce1000,

a reboot of the active SUP resolved the issue. After it came up, the standby sup was detected correct and they built the rpr redundancy. After the standby got the same image as the active, they both run in SSO.

Thanks a lot.

Best regards,

Sergej

Leo Laohoo · ‎07-05-2021

Console into the replacement line card in slot 3. I think it is in ROMMON but we would not be able to determine why until we get console access.

Sergej Barkovski · ‎07-06-2021

Hello Leo,

the replacement unit is not in rommon. It boots the ios and stays at the "Platform Manager: starting in standalone mode ". the boot process is shown in my first post. If we reload btw. re-seat it, it boots again till this line and stays in this state forever. The screenhots of the logs from the active unit, i attached, shows, that the negotiation beetween active and the other sup fails over several days.

Thanks,

Bets regrads,

Sergej

Sergej Barkovski · ‎07-06-2021

sorry, it seems I forgot to upload the fresh logs of the active unit.

---snip---

Jul 1 20:02:01: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 1 20:32:01: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 1 21:02:01: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 1 21:32:01: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 1 22:02:01: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 1 22:32:01: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 1 23:02:01: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 1 23:32:01: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 00:02:01: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 00:32:01: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 01:02:01: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 01:32:01: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 02:02:01: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 02:32:01: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 03:02:01: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 03:32:01: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 04:02:01: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 04:32:01: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 05:02:01: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 05:32:01: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 06:02:01: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 06:32:01: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 07:02:01: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 07:32:01: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 08:02:01: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 08:32:01: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 09:02:01: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 09:32:01: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 10:02:01: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 10:32:01: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 11:02:01: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 11:27:59: %C4K_REDUNDANCY-3-SIMPLEX_MODE: The peer Supervisor has been lost
Jul 2 11:28:01: %C4K_IOSMODPORTMAN-6-MODULEREMOVED: Module 3 is removed
Jul 2 11:28:33: %C4K_IOSMODPORTMAN-6-MODULEINSERTED: Module 3 is inserted
Jul 2 11:38:15: %C4K_REDUNDANCY-6-DUPLEX_MODE: The peer Supervisor has been detected
Jul 2 11:40:15: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 11:45:15: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 11:50:15: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 11:55:15: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 12:00:15: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 12:05:15: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 12:10:15: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 12:40:15: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 13:10:15: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 13:40:15: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 14:10:15: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 14:40:15: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 15:10:15: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 15:40:15: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 16:10:15: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 16:40:15: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 17:10:15: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 17:40:15: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 18:10:15: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 18:40:15: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 19:10:15: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 19:40:15: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 20:10:15: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 20:40:15: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 21:10:15: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 21:40:15: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 22:10:15: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 22:40:15: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 23:10:15: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 2 23:40:15: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 3 00:10:15: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 3 00:40:15: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 3 01:10:15: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 3 01:40:15: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 3 02:10:15: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 3 02:40:15: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 3 03:10:15: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 3 03:40:15: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 3 04:10:15: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 3 04:40:15: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 3 05:10:15: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 3 05:40:15: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 3 06:10:15: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 3 06:40:15: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 3 07:10:15: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 3 07:40:15: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 3 08:10:15: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
Jul 3 08:40:15: %C4K_REDUNDANCY-2-HANDSHAKE_TIMEOUT_ACTIVE: The handshake messaging between active and standby has not yet started.
--More--

---snip---

One thing more. The redundancy config just have two lines:

redundancy

mode sso

I'm not sure, if need "auto-config standard" to synchronize the startup config and config-register. Or it's done by default.

Thanks

Best regards,

Sergej