6509 stuck in Rommon mode

german_molina1 · ‎11-26-2018

Hi everyone!

I have been in trouble for more than two weeks trying to restore a 6509 Switch from ROMMON mode.

Today I did a procedure seen on internet which was the following:

1. BOOT=s72033-adventerprisek9_wan-mz.122-33.SXI7.bin

2. Sync

3. Confreg

4. Reset

After doing that procedure, the Switch started in normal operation but after 30 seconds, it returned to ROMMON mode. Please have a look to the following and if you see any issue don´t hesitate to reply me, Thanks!!

System Bootstrap, Version 12.2(17r)SX7, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 2009 by cisco Systems, Inc.
Cat6k-Sup720/RP platform with 1048576 Kbytes of main memory

Download Start
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Download Completed! Booting the image.
Self decompressing the image : ########################################################################################################################################################################################################################################################################################################################################################################################################## [OK]

Restricted Rights Legend

Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
(c) of the Commercial Computer Software - Restricted
Rights clause at FAR sec. 52.227-19 and subparagraph
(c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.

cisco Systems, Inc.
170 West Tasman Drive
San Jose, California 95134-1706

Cisco IOS Software, s72033_rp Software (s72033_rp-ADVENTERPRISEK9_WAN-M), Version 12.2(33)SXI7, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2011 by Cisco Systems, Inc.
Compiled Mon 18-Jul-11 05:49 by prod_rel_team
Image text-base: 0x40101328, data-base: 0x44856430

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco WS-C6509-E (R7000) processor (revision 1.5) with 983008K/65536K bytes of memory.
Processor board ID SMG1445N035
SR71000 CPU at 600Mhz, Implementation 0x504, Rev 1.2, 512KB L2 Cache
Last reset from s/w reset
1 Virtual Ethernet interface
114 Gigabit Ethernet interfaces
68 Ten Gigabit Ethernet interfaces
1917K bytes of non-volatile configuration memory.
8192K bytes of packet buffer memory.

65536K bytes of Flash internal SIMM (Sector size 512K).
System's EARLs will be forced to come up in PFC3C mode due to pfc mode
configuration in startup configPlease note switch 1 slot 4 is empty, the configuration is accepted.

Warning: enabling multiple VLAN interfaces may result in traffic bypassing the FWSM - use with caution!Please note switch 1 slot 4 is empty, the configuration is accepted.
Please note switch 2 slot 4 is empty, the configuration is accepted.

Press RETURN to get started!

*******************************************************************************
This computer system (including all hardware, software, and peripheral
equipment) is the property of Ecopetrol. Use of this computer system is
restricted to official Ecopetrol business. Ecopetrol reserves the
right to monitor use of the computer system at any time. Use of this system
constitutes consent to such monitoring. Any unauthorized access, use, or
modification of the computer system can result in civil liability and/or
criminal penalties.
*******************************************************************************

User Access Verification

Username: admin
Password:

SWN50001A#

%Software-forced reload

13:59:21 UTC Mon Nov 26 2018: Breakpoint exception, CPU signal 23, PC = 0x42B020F8

--------------------------------------------------------------------
Possible software fault. Upon reccurence, please collect
crashinfo, "show tech" and contact Cisco Technical Support.
--------------------------------------------------------------------

-Traceback= 42B020F8 42AFFC4C 4233F064 423406E8 426CC910 426CCA68 42AF4F3C
$0 : 00000000, AT : 44EF0000, v0 : 46AE0000, v1 : 00000000
a0 : 481D2D7C, a1 : 00000005, a2 : 00000000, a3 : 00000000
t0 : 42AF5080, t1 : 34008101, t2 : 42AF50A8, t3 : FFFF00FF
t4 : 42AF5080, t5 : 500122E8, t6 : 00000000, t7 : 000076FE
s0 : 00000000, s1 : 44D40000, s2 : 00000000, s3 : 44CB0000
s4 : 44CB0000, s5 : 44CB0000, s6 : 44240000, s7 : 00000001
t8 : 5001234C, t9 : 00000000, k0 : 00000000, k1 : 00000000
gp : 44EF1808, sp : 50012418, s8 : 44240000, ra : 42AFFC4C
EPC : 42B020F8, ErrorEPC : BFC1CED4, SREG : 34008103
MDLO : 00000000, MDHI : 00000000, BadVaddr : 00000000
DATA_START : 0x44856430
Cause 00000824 (Code 0x9): Breakpoint exception

Writing crashinfo to bootflash:crashinfo_20181126-135921-UTC

=== Flushing messages (13:59:21 UTC Mon Nov 26 2018) ===

Buffered messages:

*Nov 26 13:58:25.323: %SPANTREE-5-EXTENDED_SYSID: Extended SysId enabled for type vlan. The Bridge IDs of all active STP instances have been updated, which might change the spanning tree topology
*Nov 26 13:58:30.923 UTC: %SVCLC-5-SVCLCMULTI: Group 1 being tied to more than one module
*Nov 26 13:58:35.619 UTC: %SYS-5-CONFIG_I: Configured from memory by console
*Nov 26 13:58:39.540 UTC: %SYS-5-RESTART: System restarted --
Cisco IOS Software, s72033_rp Software (s72033_rp-ADVENTERPRISEK9_WAN-M), Version 12.2(33)SXI7, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2011 by Cisco Systems, Inc.
Compiled Mon 18-Jul-11 05:49 by prod_rel_team
*Nov 26 13:58:39.552 UTC: %SSH-5-ENABLED: SSH 2.0 has been enabled
*Nov 26 13:58:39.952 UTC: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.11.147.121 port 514 started - CLI initiated
*Nov 26 13:58:44.668 UTC: %SNMP-5-COLDSTART: SNMP agent on host SWN50001A is undergoing a cold start
*Nov 26 13:35:22.599: %SYS-SW2_SP-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging output.
*Nov 26 13:58:17.587: %SPANTREE-SW2_SP-5-EXTENDED_SYSID: Extended SysId enabled for type vlan. The Bridge IDs of all active STP instances have been updated, which might change the spanning tree topology
*Nov 26 13:58:21.427: %C6KERRDETECT-SW2_SP-2-SWBUSSTALL: The switching bus is experiencing stall for 3 seconds
*Nov 26 13:58:38.661: %SYS-SW2_SP-5-RESTART: System restarted --
*Nov 26 13:58:38.749: %OIR-SW2_SP-6-INSPS: Power supply inserted in slot 1
*Nov 26 13:58:38.917: %OIR-SW2_SP-6-INSPS: Power supply inserted in slot 2
*Nov 26 13:58:38.921: %DUAL_ACTIVE-SW2_SP-4-CONFIG: No dual-active detection methods configured - it is recommended to have at least one configured
*Nov 26 13:58:39.333 UTC: %SYS-SW2_SP-6-BOOTTIME: Time taken to reboot after reload = 836 seconds
*Nov 26 13:58:39.973 UTC: %C6K_POWER-SW2_SP-4-GLITCH: Recovered from glitch in system power supply.
*Nov 26 13:58:40.077 UTC: %C6K_POWER-SW2_SP-4-GLITCH: Recovered from glitch in system power supply.
*Nov 26 13:58:40.177 UTC: %C6K_POWER-SW2_SP-4-GLITCH: Recovered from glitch in system power supply.
*Nov 26 13:58:40.181 UTC: %C6K_POWER-SW2_SP-4-GLITCH: Recovered from glitch in system power supply.
*Nov 26 13:58:40.858 UTC: %C6KPWR-SW2_SP-4-UNDERPOWERED: insufficient power to operate all FRUs in system.
*Nov 26 13:58:40.858 UTC: %C6KPWR-SW2_SP-2-INSUFFICIENTPOWER: Powering down all linecards as there is not enough power to operate all critical cards
*Nov 26 13:58:40.858 UTC: %C6KPWR-SW2_SP-2-INSUFFICIENTPOWER: Powering down all linecards as there is not enough power to operate all critical cards
*Nov 26 13:58:42.010 UTC: %C6KPWR-SW2_SP-4-DISABLED: power to module in slot 2 set off (admin request)
*Nov 26 13:58:42.484 UTC: %C6KPWR-SW2_SP-4-DISABLED: power to module in slot 3 set off (admin request)
*Nov 26 13:58:42.612 UTC: %C6KPWR-SW2_SP-4-POWERDENIED: insufficient power, module in slot 4 power denied.
Queued messages:
*Nov 26 13:59:24.276 UTC: %SYS-3-LOGGER_FLUSHING: System pausing to ensure console debugging output.

*Nov 26 13:59:21.008 UTC: %C6K_PLATFORM-2-PEER_RESET: RP is being reset by the SP
*** System received a Software forced crash ***
signal= 0x17, code= 0x24, context= 0x46ad9ae4
PC = 0x42af50dc, SP = 0x44d3d238, RA = 0x416c008c
Cause Reg = 0x0000b820, Status Reg = 0x34008002

*Nov 26 13:59:25.564 UTC: %SYS-SW2_SP-3-LOGGER_FLUSHING: System pausing to ensure console debugging output.

*Nov 26 13:59:25.564 UTC: %OIR-SW2_SP-6-CONSOLE: Changing console ownership to switch processor

No warm reboot Storage
*** System received an unknown failure ***
signal= 0x0, code= 0x0, context= 0x443a4b64
PC = 0x417bd650, Cause = 0x1820, Status Reg = 0x34008102
rommon 2 >

balaji.bandi · ‎11-26-2018

Can you explain more about the setup, is this VSS ? what is modules in it.

is this working one ? or new one you upgrading ?

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

german_molina1 · ‎11-26-2018

Yeah.

There are 2 switches 6509 configured as a VSS. One of them (what I mention here in the previous post) is failed.

Actually this switch has the following modules:

1. WS-X6748-GE-TX

2. WS-X6716-10GE

3. WS-X6716-10GE

4. WS-SVC-FWM-1

5. VS-S720-10G

The another 6509 is working right now.

Thanks!

balaji.bandi · ‎11-26-2018

i do not have other post reference, You need to provide more information about the issue or background to understand the problem and suggest best we can

so this is working one and failed ? what is failed ?

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

german_molina1 · ‎11-27-2018

Mr Balaji,

The client don´t know what happened with the 6509 switch. It seems to be a corrupt IOS running in the Switch.

There are two 6509, one is working under good conditions and the another is failed. They were working as VSS (High availability), but now VSS is not working because the ROMMON condition on the failed switch.

Thanks!

Dan Lukes · ‎11-26-2018

@german_molina1 wrote:
1. BOOT=s72033-adventerprisek9_wan-mz.122-33.SXI7.bin
2. Sync

3. Confreg

4. Reset

After doing that procedure, the Switch started in normal operation but after 30 seconds, it returned to ROMMON mode.

You may consider to delete configuration of the switch. If switch will panic even with no configuration, you are facing hardware issue.

@german_molina1 wrote:

13:59:21 UTC Mon Nov 26 2018: Breakpoint exception, CPU signal 23, PC = 0x42B020F8
Cause 00000824 (Code 0x9): Breakpoint exception

Breakpoint exception is software-triggered panic. Internal firmware has detected a condition that's unable to handle. Message "Software forced crash" fired later claims the same. Following messages seems to be related to the issue:

Writing crashinfo to bootflash:crashinfo_20181126-135921-UTC
*Nov 26 13:58:40.858 UTC: %C6KPWR-SW2_SP-4-UNDERPOWERED: insufficient power to operate all FRUs in system.

*Nov 26 13:58:40.858 UTC: %C6KPWR-SW2_SP-2-INSUFFICIENTPOWER: Powering down all linecards as there is not enough power to operate all critical cards
*Nov 26 13:58:42.010 UTC: %C6KPWR-SW2_SP-4-DISABLED: power to module in slot 2 set off (admin request)
*Nov 26 13:58:42.484 UTC: %C6KPWR-SW2_SP-4-DISABLED: power to module in slot 3 set off (admin request)
*Nov 26 13:58:42.612 UTC: %C6KPWR-SW2_SP-4-POWERDENIED: insufficient power, module in slot 4 power denied.

You are facing "insufficient power" issue. Either a power module is faulty or switch power requirements has increased (changed configuration ? added card ? firmware upgrade ?).

german_molina1 · ‎11-27-2018

Hi Dan,

Do you have any procedure to delete the configuration of the switch and upload the new firmware?

Thanks.

Dan Lukes · ‎11-27-2018

Well, for the purpose of the test, erasing of configuration is not necessary. Use ROMMON commands:

confreg 0x2142
reset

The switch will (re-)start but it will not load configuration saved. Use either standard config mode command "config-register 0x2102" or ROMMON command "confreg 0x2102" to revert back to standard boot mode (e.g. saved configuration is loaded during boot).

Leo Laohoo · ‎11-26-2018

Go into the bootflash ("dir bootflash:") and see if the file "crashinfo_20181126-135921-UTC" is present. If it is, attach it to the thread.

german_molina1 · ‎11-27-2018

Hi Leo,

File "crashinfo_20181126-135921-UTC" is in the bootflash. How can I see more information about it? Don´t know how to download it from 6509.

If you have any procedure to do, let me know.

Thanks!

German

gaston.benitez · ‎11-27-2018

Hi

To see this file you can do the next command and havving your session logging to a file.

more bootflash:crashinfo_20181126-135921-UTC

Also try to verify if your power suplies are connected and working.

BR

Gaston

Leo Laohoo · ‎11-27-2018

@german_molina1 wrote:

File "crashinfo_20181126-135921-UTC" is in the bootflash. How can I see more information about it? Don´t know how to download it from 6509.

The supervisor card has a CF slot. Copy the file into the CF and take the CF to your computer.

gaston.benitez · ‎12-03-2018

Hi

Please do the next procedure.

Remove all the linecards from the box except supervisors.

Then, power on the box.

It ought to be enough to let the complete boot up process finilize

If this work ok, then please check your power supplies to certify that you have power enough to boot up all the linecards at the same time.

BR

Gaston