VSS standby booting into ROMMOM mode after reload

[Kamil Sheriff]

Hello,

I have an issue with the newly build VSS pair, where once configured with appropriate commands the primary VSS pair is booting normally and becomes Active but when the secondary VSS Pair is booted up it's getting crashed and booting in ROMMON mode. The troubleshooting steps have done are:

1. Tried uploading the new stable version of the image but unfortunately still the same issue.

2. The register value gets changed to 0x200 everytime it goes to ROMMON mode. Configured manually 0x2102 to boot normally and erase the config in seconds before crashing it again.

3. boot versions are the same in both primary and secondary but still the standby is having trouble.

4. VSL portchannels has been configured correctly and are up.

Please review the logs below when the standby is booted up.

Router#switch convert mode virtual

This command will convert all interface names
to naming convention "interface-type switch-number/slot/port",
save the running config to startup-config and
reload the switch.

NOTE: Make sure to configure one or more dual-active detection methods
once the conversion is complete and the switches have come up in VSS mode.

Do you want to proceed? [yes/no]: yes
Converting interface names
Building configuration...
[OK]
Saving converted configuration to bootdisk: ...
Destination filename [startup-config.converted_vs-20180905-092640]?
11
*Sep 5 09:26:42.419: %SYS-5-RELOAD: Reload requested by Exec. Reload Reason: Conversion.157 bytes copied in 0.448 secs (24904 bytes/sec)
Resetting .......

System Bootstrap, Version 12.2(50r)SYS3, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 2012 by cisco Systems, Inc.
PYRAMID platform with 2097152 Kbytes of main memory

Autoboot executing command: "boot bootdisk:s2t54-ipservicesk9-mz.SPA.155-1.SY1.bin"

bootdisk:%s72044-atafslib-m: Digitally Signed Release Software with key version A

Initializing ATA monitor library...

bootdisk:s2t54-ipservicesk9-mz.SPA.155-1.SY1.bin: Digitally Signed Release Software with key version A

Self extracting the image... [OK]
Self decompressing the image : ####################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################### [OK]
*** No sreloc section
Restricted Rights Legend

Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
(c) of the Commercial Computer Software - Restricted
Rights clause at FAR sec. 52.227-19 and subparagraph
(c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.

cisco Systems, Inc.
170 West Tasman Drive
San Jose, California 95134-1706

Cisco IOS Software, s2t54 Software (s2t54-IPSERVICESK9-M), Version 15.5(1)SY1, RELEASE SOFTWARE (fc6)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2018 by Cisco Systems, Inc.
Compiled Sun 04-Mar-18 06:00 by prod_rel_team
System detected Virtual Switch configuration...
Interface TenGigabitEthernet 2/3/4 is member of PortChannel 2
Interface TenGigabitEthernet 2/3/5 is member of PortChannel 2

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco C6807-XL (M8572) processor (revision ) with 1785856K/262144K bytes of memory.
Processor board ID SMC19170021
CPU: MPC8572_E, Version: 2.2, (0x80E80022)
CORE: E500, Version: 3.0, (0x80210030)
CPU:1500MHz, CCB:600MHz, DDR:600MHz
L1: D-cache 32 kB enabled
I-cache 32 kB enabled

Last reset from s/w reset
1 Virtual Ethernet interface
51 Gigabit Ethernet interfaces
34 Ten Gigabit Ethernet interfaces
2543K bytes of non-volatile configuration memory.

Press RETURN to get started!

Firmware compiled 04-Mar-18 06:00 by binugopa Build [26112]

*Sep 5 09:28:24.907: %PFREDUN-6-ACTIVE: Initializing as ACTIVE processor for this switch

*Sep 5 09:28:36.239: %VSL_BRINGUP-6-MODULE_UP: VSL module in slot 3 switch 2 brought up
*Sep 5 09:30:41.451: %VSLP-5-RRP_PEER_TIMEOUT: VSLP peer timer expired without detecting peer. Resolving role as ActiveInitializing as Virtual Switch ACTIVE processor

*Sep 5 09:30:41.451: %VSLP-2-VSL_DOWN: VSL links down and not ready for any traffic

*Sep 5 09:30:57.651: SW2: SP: Currently running ROMMON from S (Gold) region
*Sep 5 09:31:02.895: %SYS-SW2-5-CONFIG_I: Configured from memory by console
*Sep 5 09:31:03.163: %SYS-SW2-5-RESTART: System restarted --
Cisco IOS Software, s2t54 Software (s2t54-IPSERVICESK9-M), Version 15.5(1)SY1, RELEASE SOFTWARE (fc6)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2018 by Cisco Systems, Inc.
Compiled Sun 04-Mar-18 06:00 by prod_rel_team
*Sep 5 09:31:04.443: %SYS-SW2-6-BOOTTIME: Time taken to reboot after reload = 261 seconds
*Sep 5 09:31:04.511: %OIR-SW2-6-INSPS: Power supply inserted in slot 1
*Sep 5 09:31:04.747: %C6KPWR-SW2-4-PSOK: power supply 1 turned on.
*Sep 5 09:31:04.875: %OIR-SW2-6-INSPS: Power supply inserted in slot 2
*Sep 5 09:31:04.951: %C6KPWR-SW2-4-PSOK: power supply 2 turned on.
*Sep 5 09:31:05.187: %OIR-SW2-6-INSPS: Power supply inserted in slot 3
*Sep 5 09:31:05.267: %C6KPWR-SW2-4-PSOK: power supply 3 turned on.
*Sep 5 09:31:05.403: %OIR-SW2-6-INSPS: Power supply inserted in slot 4
*Sep 5 09:31:05.483: %C6KPWR-SW2-4-PSOK: power supply 4 turned on.
*Sep 5 09:31:07.327: %OIR-SW2-6-INSPC: Power converter inserted in slot 1
*Sep 5 09:31:08.203: %OIR-SW2-6-INSPC: Power converter inserted in slot 2
*Sep 5 09:31:08.315: %C6KENV-SW2-4-FANHPMODE: Switch 2 Fan-tray 1 is operating in high power mode
*Sep 5 09:31:08.399: %FABRIC-SW2-5-FABRIC_MODULE_ACTIVE: The Switch Fabric Module in slot 3 became active.
*Sep 5 09:31:08.507: %PNP-SW2-6-PNP_DISCOVERY_STOPPED: PnP Discovery stopped (Startup Config Present)
*Sep 5 09:31:14.259: %OIR-SW2-6-INSREM: Switch 2 Physical Slot 1 - Module Type LINE_CARD inserted
*Sep 5 09:31:14.259: %DIAG-SW2-6-RUN_MINIMUM: Switch 2 Module 3: Running Minimal Diagnostics...
*Sep 5 09:31:14.263: %OIR-SW2-6-INSREM: Switch 2 Physical Slot 2 - Module Type LINE_CARD inserted
*Sep 5 09:31:14.263: %OIR-SW2-6-INSREM: Switch 2 Physical Slot 3 - Module Type LINE_CARD inserted
*Sep 5 09:31:14.271: %CONST_DIAG-SW2-6-DIAG_PORT_SKIPPED: Module 3 port 4 is skipped in TestLoopback due to: the port is used as a VSL link.
*Sep 5 09:31:14.271: %CONST_DIAG-SW2-6-DIAG_PORT_SKIPPED: Module 3 port 5 is skipped in TestLoopback due to: the port is used as a VSL link.
*Sep 5 09:31:16.411: %CONST_DIAG-SW2-6-DIAG_PORT_SKIPPED: Module 3 port 4 is skipped in TestL2CTSLoopback due to: the port is used as a VSL link.
*Sep 5 09:31:16.411: %CONST_DIAG-SW2-6-DIAG_PORT_SKIPPED: Module 3 port 5 is skipped in TestL2CTSLoopback due to: the port is used as a VSL link.
*Sep 5 09:31:24.271: %CONST_DIAG-SW2-6-DIAG_PORT_SKIPPED: Module 3 port 4 is skipped in TestL3CTSLoopback due to: the port is used as a VSL link.
*Sep 5 09:31:24.271: %CONST_DIAG-SW2-6-DIAG_PORT_SKIPPED: Module 3 port 5 is skipped in TestL3CTSLoopback due to: the port is used as a VSL link.
*Sep 5 09:31:31.107: %CONST_DIAG-SW2-6-DIAG_PORT_SKIPPED: Module 3 port 4 is skipped in TestProtocolMatchChannel due to: port loopback not allowed.
*Sep 5 09:31:36.399: %DIAG-SW2-3-MAJOR: Switch 2 Module 3: Online Diagnostics detected a Major Error. Please use 'show diagnostic result <target>' to see test results.
*Sep 5 09:31:36.399: %CONST_DIAG-SW2-3-BOOTUP_TEST_FAIL: Switch 2 Module 3: TestVslBridgeLink failed
*Sep 5 09:31:36.399: %CONST_DIAG-SW2-3-BOOTUP_TEST_FAIL: Switch 2 Module 3: TestVslLocalLoopback failed on port(s) 4-5
*Sep 5 09:31:38.107: %C6KENV-SW2-4-HIGHER_SLOT_EMPTY: The higher adjacent slot of module 3 might be empty. Airdam must be installed in that slot to be NEBS compliant

%Software-forced reload

*Sep 5 09:31:42.107: SW2: %HA_EM-6-LOG: Mandatory.go_bootup.tcl: GOLD EEM TCL policy for boot up diagnostic
*Sep 5 09:31:42.627: %CONST_DIAG-SW2-3-SUP_FAILURE: Active supervisor has MAJOR online di

09:31:43 UTC Wed Sep 5 2018: Unexpected exception to CPU: vector 1500, PC = 0x7BBEBD4 , LR = 0x7BBEB6C
-Traceback= 0x7BBEBD4z 0x96222C8z 0x94270ACz 0x94276ECz 0x790F374z 0x49F15D8z 0x49F4048z 0x76C0404z 0x4A05350z 0x7BC01A4z 0x7BBA560z

CPU Register Context:
MSR = 0x00029200 CR = 0x28020022 CTR = 0x00000005 XER = 0x20000000
R0 = 0x07BBEB6C R1 = 0x5963C5B0 R2 = 0xFFF7FFF7 R3 = 0x15C14F48
R4 = 0x00000006 R5 = 0x00000009 R6 = 0x0000002B R7 = 0x00000009
R8 = 0x0000001F R9 = 0x00000000 R10 = 0x00000744 R11 = 0xF9172A10
R12 = 0x000000BC R13 = 0x0121F000 R14 = 0x00000000 R15 = 0x0C0DDE2C
R16 = 0x00000000 R17 = 0x100B0000 R18 = 0x1C1635B8 R19 = 0x100C0000
R20 = 0x0A5C2D28 R21 = 0x100C0000 R22 = 0xFFFFFFFF R23 = 0x00000001
R24 = 0x00000003 R25 = 0x5963C83C R26 = 0x00000000 R27 = 0x1C161B4C
R28 = 0x0A5C3F38 R29 = 0x0C6BC548 R30 = 0x5963C5D0 R31 = 0x00000000

TEXT_START : 0x04100270
DATA_START : 0x0C000000
Writing crashinfo to bootdisk:crashinfo_20180905-093143-UTCagnostic failure 0x0: UNKNOWN
*Sep 5 09:31:42.627: %PFREDUN-SW2-4-AUTOBOOT: Disable autoboot for Supervisor in slot 35
*Sep 5 09:31:42.627: %PFREDUN-SW2-6-ACTIVE: Sup in slot 35 is crashing due to Supervisor online diag failur
*Sep 5 09:31:43.339: %OIR-SW2-6-SP_INSCARD: Card inserted in Switch_number = 2, physical slot 3, i
*** System received a Software Forced Crash ***
signal= 0x17, code= 0x1500, context= 0xe32e8ec
PC = 0x7bbebd4, Vector = 0x1500, SP = 0x5963c5b0

System Bootstrap, Version 12.2(50r)SYS3, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 2012 by cisco Systems, Inc.
PYRAMID platform with 2097152 Kbytes of main memory

rommon 1 >
rommon 1 >

[Leo Laohoo]

---

@Kamil Sheriff wrote:

*Sep 5 09:31:36.399: %CONST_DIAG-SW2-3-BOOTUP_TEST_FAIL: Switch 2 Module 3: TestVslBridgeLink failed
*Sep 5 09:31:36.399: %CONST_DIAG-SW2-3-BOOTUP_TEST_FAIL: Switch 2 Module 3: TestVslLocalLoopback failed on port(s) 4-5

---

Raise a TAC Case.  This could either be CSCuv85387 or CSCte46962.