Stack member switch 3750X in a booting loop!

sonicimage1 · ‎12-27-2017

Hi Guys,

Last week I tried to upgrade our edge 3750X switch (two switches in a stack), from c3750e-universalk9-mz.152-2.E.bin to c3750e-universalk9-mz.152-4.E5.bin.

Previously I copied the new IOS to both flash: and flash2: from TFTP, ran command 'boot system switch all flash:c3750e-universalk9-mz.152-4.E5.bin, and then 'wr mem'.

Later I consoled into the master switch and reloaded. The master switch booted into the new image successfully, but the member switch got stuck in a booting loop. The new IOS was loading, and went nearly to the end, and the master switch could see the member at some point. Then the process started all over again, and it was endless!

I tried powering the member switch off, and then on again - still the same. Tried removing stacking cables, and restarting - still no joy. Then booted the member into SWITCH: prompt, and then into the previous IOS - still a loop! I didn't know what else I could try, so at the end I powered the offending switch off and left it as it was (as the master switch was providing all the services).

Please could you advise how to resolve this issue? Any advice would be appreciated.

Regards,

Lucia

Leo Laohoo · ‎12-27-2017

Console into the switch and reboot. Post the entire boot-up/boot-crash-loop. We want to see what's causing it.

sonicimage1 · ‎12-27-2017

How would I do that? As the switch doesn't actually boot so I can't get into # prompt to copy anything?

Leo Laohoo · ‎12-27-2017

Console into the switch (the same one that is boot-crash-loop), enable "terminal logging" and then reboot the switch.

sonicimage1 · ‎12-27-2017

I can't enable anything as the switch never loads the IOS but keeps restarting.

Leo Laohoo · ‎12-27-2017

@sonicimage1 wrote:

switch never loads the IOS but keeps restarting.

We want to see this.

Can you console into this switch? We want to see something.

Besides, if you can't console into this switch how can we help?

sonicimage1 · ‎12-27-2017

I can console into it tomorrow as I'm not on site today (the school is closed). I understand that you need to see it, but my question is - how can I get that info for you as I've never done it, and the switch doesn't let me do anything in CLI as it never loads completely? Please advise.

Leo Laohoo · ‎12-27-2017

@sonicimage1 wrote:

how can I get that info for you as I've never done it, and the switch doesn't let me do anything in CLI as it never loads completely?

Like I've said before, "console in". Console into the switch that is having difficulties using a laptop/PC and a console cable.
It's a 3750X so this means you need an easy-to-find USB cable with a mini-USB at the end. This is not a special cable but a mini-USB at one end and a regular USB male at the other end.

sonicimage1 · ‎12-27-2017

Ok, I think you misunderstand what I'm trying to ask. I know how to console into a switch and I know that mini USB console cable is required as I already used it. But when I'm in CLI and reboot the switch, the output that I see - how will I get it out so that I could send it to you? Would I be able to copy everything I see in the terminal while the switch is still booting? As I mentioned I'd never done that, I only copied the command output in a working switch.

Leo Laohoo · ‎12-27-2017

@sonicimage1 wrote:

the output that I see - how will I get it out so that I could send it to you? Would I be able to copy everything I see in the terminal while the switch is still booting?

Again, I've already mentioned this before. Different softwares use different term. It's either "terminal logging" or "logging console". The console terminal gets logged into a file.

sonicimage1 · ‎12-28-2017

Here is the output of the switch booting process - at the end failed to boot at all. It went into ROMMON, and it looks like it doesn't see flash at all!

File "flash:c3750e-universalk9-mz.152-4.E5.bin" uncompressed and installed, entry point: 0x3000
executing...

Restricted Rights Legend

Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
(c) of the Commercial Computer Software - Restricted
Rights clause at FAR sec. 52.227-19 and subparagraph
(c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.

           cisco Systems, Inc.
           170 West Tasman Drive
           San Jose, California 95134-1706

Cisco IOS Software, C3750E Software (C3750E-UNIVERSALK9-M), Version 15.2(4)E5, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2017 by Cisco Systems, Inc.
Compiled Mon 18-Sep-17 06:58 by prod_rel_team
Initializing flashfs...
Using driver version 4 for media type 1
Using driver version 4 for media type 2
mifs[6]: 10 files, 1 directories
mifs[6]: Total bytes : 2097152
mifs[6]: Bytes used : 741376
mifs[6]: Bytes available : 1355776
mifs[6]: mifs fsck took 1 seconds.
mifs[6]: Initialization complete.

mifs[7]: 0 files, 1 directories
mifs[7]: Total bytes : 4194304
mifs[7]: Bytes used : 8192
mifs[7]: Bytes available : 4186112
mifs[7]: mifs fsck took 0 seconds.
mifs[7]: Initialization complete.

mifs[8]: 5 files, 1 directories
mifs[8]: Total bytes : 524288
mifs[8]: Bytes used : 49152
mifs[8]: Bytes available : 475136
mifs[8]: mifs fsck took 0 seconds.
mifs[8]: Initialization complete.

mifs[9]: 5 files, 1 directories
mifs[9]: Total bytes : 524288
mifs[9]: Bytes used : 49152
mifs[9]: Bytes available : 475136
mifs[9]: mifs fsck took 1 seconds.
mifs[9]: Initialization complete.

mifs[10]: 38 files, 3 directories
mifs[10]: Total bytes : 119144448
mifs[10]: Bytes used : 55144448
mifs[10]: Bytes available : 64000000
mifs[10]: mifs fsck took 0 seconds.
mifs[10]: Initialization complete.

...done Initializing flashfs.
Checking for Bootloader upgrade..

Boot Loader upgrade not required (Stage 2)

FIPS: Flash Key Check : Begin
FIPS: Flash Key Check : End, Not Found, FIPS Mode Not Enabled

POST: CPU MIC register Tests : Begin
POST: CPU MIC register Tests : End, Status Passed

POST: MA BIST : Begin
POST: MA BIST : End, Status Passed

POST: TCAM BIST : Begin
POST: TCAM BIST : End, Status Passed

POST: SF ASIC BIST : Begin
POST: SF ASIC BIST : End, Status Passed

POST: Switch Fabric Memory Tests : Begin
POST: Switch Fabric Memory Tests : End, Status Passed

POST: CPU MIC interface Loopback Tests : Begin
POST: CPU MIC interface Loopback Tests : End, Status Passed

POST: PortASIC RingLoopback Tests : Begin
POST: PortASIC RingLoopback Tests : End, Status Passed

extracting front_end/front_end_ucode_info (309 bytes)
SM: Detected stack cables at PORT1 PORT2

Waiting for Stack Master Election...
POST: Thermal, Fan Tests : Begin
POST: Thermal, Fan Tests : End, Status Passed

POST: PortASIC Port Loopback Tests : Begin
POST: PortASIC Port Loopback Tests : End, Status Passed

POST: PortASIC Macsec Loopback Tests : Begin
POST: PortASIC Macsec Loopback Tests : End, Status Passed

POST: EMAC Loopback Tests : Begin
POST: EMAC Loopback Tests : End, Status Passed

Election Complete
Switch 2 booting as Member, Switch 1 elected Master
HCOMP: Compatibility check PASSED
Waiting for feature sync....
Waiting for Port download...Complete
Stack Master is ready

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco WS-C3750X-24S (PowerPC405) processor (revision M0) with 524288K bytes of memory.
Processor board ID FDO1828P38X
Last reset from power-on
1 FastEthernet interface
56 Gigabit Ethernet interfaces
4 Ten Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.

512K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address       : 7C:0E:CE:41:4F:80
Motherboard assembly number     : 73-13061-06
Motherboard serial number       : FDO18281DA8
Model revision number           : M0
Motherboard revision number     : A0
Model number                    : WS-C3750X-24S-S
Daughterboard assembly number   : 800-32727-03
Daughterboard serial number     : FDO18281F9N
System serial number            : FDO1828P38X
Top Assembly Part Number        : 800-33746-06
Top Assembly Revision Number    : B0
Version ID                      : V05
CLEI Code Number                : CMMP200DRA
Hardware Board Revision Number : 0x05
3383280K bytes of USB Flash usbflash0 (Read/Write)

Press RETURN to get started!

10:43:08 UTC Thu Dec 28 2017: Unexpected exception to CPUvector 2000, PC = 350A4A4
-Traceback= 0x350A4A4z 0x25DD8E4z 0x26121D0z 0x25D1F14z 0x32DBCC8z 0x32D8048z

=== Flushing messages (10:43:08 UTC Thu Dec 28 2017) ===

Buffered messages:

*Mar 1 00:00:06.903: Read env variable - LICENSE_BOOT_LEVEL =
*Jan 2 00:00:06.090: %IOS_LICENSE_IMAGE_APPLICATION-6-LICENSE_LEVEL: Module name = c3750x Next reboot level = ipbase and License = ipbase
*Jan 2 00:02:08.698: %STACKMGR-4-SWITCH_ADDED: Switch 1 has been ADDED to the stack
*Jan 2 00:02:08.698: %STACKMGR-4-SWITCH_ADDED: Switch 2 has been ADDED to the stack
*Jan 2 00:02:11.516: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0, changed state to down
*Jan 2 00:02:13.429: %SMI-5-CLIENT: Smart Install Client feature is enabled. It is recommended to disable the Smart Install feature when it is not actively used. To disable feature execute 'no vstack' in configuration mode
*Dec 28 10:42:49.944: %USBFLASH-5-CHANGE: usbflash0 has been inserted!
*Dec 28 10:42:51.580: %SPANTREE-5-EXTENDED_SYSID: Extended SysId enabled for type vlan
*Dec 28 10:42:59.415: %PLATFORM_ENV-6-FRU_PS_OIR: FRU Power Supply 1 inserted
*Dec 28 10:42:59.415: %PLATFORM_ENV-6-FRU_PS_OIR: FRU Power Supply 2 inserted
*Dec 28 10:43:01.588: %SYS-5-LOG_CONFIG_CHANGE: Console logging disabled
*Dec 28 10:43:02.821: %SYS-5-CONFIG_I: Configured from memory by console
*Dec 28 10:43:02.846: %SYS-5-RESTART: System restarted --
Cisco IOS Software, C3750E Software (C3750E-UNIVERSALK9-M), Version 15.2(4)E5, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2017 by Cisco Systems, Inc.
Compiled Mon 18-Sep-17 06:58 by prod_rel_team
*Dec 28 10:43:03.039: %STACKMGR-5-SWITCH_READY: Switch 1 is READY
*Dec 28 10:43:03.534: %STACKMGR-5-MASTER_READY: Master Switch 1 is READY
*Dec 28 10:43:03.634: %STACKMGR-5-SWITCH_READY: Switch 2 is READY
Dec 28 10:43:06.276: %USB_CONSOLE-6-MEDIA_USB: Console media-type is USB.
Queued messages:
Dec 28 10:43:10.881: %SYS-3-LOGGER_FLUSHING: System pausing to ensure console debugging output.

Dec 28 10:43:06.855: %LINK-5-CHANGED: Interface GigabitEthernet2/1/1, changed state to administratively down
Dec 28 10:43:07.039: %LINK-5-CHANGED: Interface GigabitEthernet2/1/2, changed state to administratively down
Dec 28 10:43:07.039: %LINK-5-CHANGED: Interface GigabitEthernet2/1/3, changed state to administratively down
Dec 28 10:43:07.081: %LINK-5-CHANGED: Interface GigabitEthernet2/1/4, changed state to administratively down
Dec 28 10:43:08.532: %Flash file system is active. Unable to write extended crashinfo.
Cisco IOS Software, C3750E Software (C3750E-UNIVERSALK9-M), Version 15.2(4)E5, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2017 by Cisco Systems, Inc.
Compiled Mon 18-Sep-17 06:58 by prod_rel_team

Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!
SRR0 = 0x02770FA0 SRR1 = 0x00029230 SRR2 = 0x0350A4A4 SRR3 = 0x00029230
ESR = 0x00000000 DEAR = 0x00000000 TSR = 0x84000000 DBSR = 0x01000000

CPU Register Context:
Vector = 0x00002000 PC = 0x0350A4A4 MSR = 0x00029230 CR = 0x40224048
LR = 0x025DD8E4 CTR = 0x000000A0 XER = 0x00000000
R0 = 0x00000003 R1 = 0x09F70B10 R2 = 0x00000000 R3 = 0x00000000
R4 = 0x09F70C5C R5 = 0x000000A1 R6 = 0x00005180 R7 = 0x00000040
R8 = 0x001A1B00 R9 = 0x052D0000 R10 = 0x052F0000 R11 = 0x052F0000
R12 = 0x00027A28 R13 = 0x1B981A10 R14 = 0x20208044 R15 = 0x00000001
R16 = 0x0A124CD8 R17 = 0x000013FC R18 = 0x09F70BBC R19 = 0x00000000
R20 = 0x09F92C02 R21 = 0x00000001 R22 = 0x00000000 R23 = 0xFFFFFFFF
R24 = 0x09F70C5C R25 = 0x00000000 R26 = 0x09F70C5C R27 = 0x0A123844
R28 = 0x00000003 R29 = 0x00000064 R30 = 0x09F92BF8 R31 = 0x09F94330

Stack trace:
PC = 0x0350A4A4, SP = 0x09F70B10
Frame 00: SP = 0x09F70D68    PC = 0x025DD7FC
Frame 01: SP = 0x09F70E28    PC = 0x026121D0
Frame 02: SP = 0x09F70FD0    PC = 0x025D1F14
Frame 03: SP = 0x09F70FD8    PC = 0x032DBCC8
Frame 04: SP = 0x00000000    PC = 0x032D8048

BHS-SW-08-2 uptime is 2 minutes, 47 seconds

cisco WS-C3750X-24S (PowerPC405) processor (revision M0) with 524288K bytes of memory.
Processor board ID FDO1828P38X

10:43:13 UTC Thu Dec 28 2017: Unexpected exception to CPUvector 200, PC = 2FB36B4
-Traceback= 0x2FB36B4z 0x3146C84z

Cisco IOS Software, C3750E Software (C3750E-UNIVERSALK9-M), Version 15.2(4)E5, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2017 by Cisco Systems, Inc.
Compiled Mon 18-Sep-17 06:58 by prod_rel_team

Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!
SRR0 = 0x02770FA0 SRR1 = 0x00029230 SRR2 = 0x0350A4A4 SRR3 = 0x00029230
ESR = 0x00000000 DEAR = 0x00000000 TSR = 0x84000000 DBSR = 0x01000000

CPU Register Context:
Vector = 0x00002000 PC = 0x0350A4A4 MSR = 0x00029230 CR = 0x40224048
LR = 0x025DD8E4 CTR = 0x000000A0 XER = 0x00000000
R0 = 0x00000003 R1 = 0x09F70B10 R2 = 0x00000000 R3 = 0x00000000
R4 = 0x09F70C5C R5 = 0x000000A1 R6 = 0x00005180 R7 = 0x00000040
R8 = 0x001A1B00 R9 = 0x052D0000 R10 = 0x052F0000 R11 = 0x052F0000
R12 = 0x00027A28 R13 = 0x1B981A10 R14 = 0x20208044 R15 = 0x00000001
R16 = 0x0A124CD8 R17 = 0x000013FC R18 = 0x09F70BBC R19 = 0x00000000
R20 = 0x09F92C02 R21 = 0x00000001 R22 = 0x00000000 R23 = 0xFFFFFFFF
R24 = 0x09F70C5C R25 = 0x00000000 R26 = 0x09F70C5C R27 = 0x0A123844
R28 = 0x00000003 R29 = 0x00000064 R30 = 0x09F92BF8 R31 = 0x09F94330

Stack trace:
PC = 0x0350A4A4, SP = 0x09F70B10
Frame 00: SP = 0x09F70D68    PC = 0x025DD7FC
Frame 01: SP = 0x09F70E28    PC = 0x026121D0
Frame 02: SP = 0x09F70FD0    PC = 0x025D1F14
Frame 03: SP = 0x09F70FD8    PC = 0x032DBCC8
Frame 04: SP = 0x00000000    PC = 0x032D8048

BHS-SW-08-2 uptime is 2 minutes, 49 seconds

cisco WS-C3750X-24S (PowerPC405) processor (revision M0) with 524288K bytes of memory.
Processor board ID FDO1828P38X

Using driver version 3 for media type 1
Invalid parameter block -- no mac address available.
Xmodem file system is available.
Can't set variable with illegal char "ÿ" (0xffffffff) in name.
Can't set variable with illegal char "ÿ" (0xffffffff) in name.
Can't set variable with illegal char "÷" (0xfffffff7) in name.
Can't set variable with illegal char "ï" (0xffffffef) in name.
Can't set variable with illegal char "ç" (0xffffffe7) in name.
Can't set variable with illegal char "ß" (0xffffffdf) in name.
Can't set variable with illegal char "×" (0xffffffd7) in name.
Can't set variable with illegal char "Ï" (0xffffffcf) in name.
Can't set variable with illegal char "Ç" (0xffffffc7) in name.
Can't set variable with illegal char "¿" (0xffffffbf) in name.
Can't set variable with illegal char "·" (0xffffffb7) in name.
Can't set variable with illegal char "¯" (0xffffffaf) in name.
Can't set variable with illegal char "§" (0xffffffa7) in name.
Can't set variable with illegal char "" (0xffffff9f) in name.
Can't set variable with illegal char "" (0xffffff97) in name.
Can't set variable with illegal char "" (0xffffff8f) in name.
Can't set variable with illegal char "" (0xffffff87) in name.
Can't set variable with illegal char "" (0x7f) in name.
Can't set variable with illegal char "â" (0xffffffe2) in name.
Can't set variable with illegal char "â" (0xffffffe2) in name.
Can't set variable with illegal char "â" (0xffffffe2) in name.
Can't set variable with illegal char "â" (0xffffffe2) in name.
Can't set variable with illegal char "â" (0xffffffe2) in name.
Can't set variable with illegal char "â" (0xffffffe2) in name.
Can't set variable with illegal char "â" (0xffffffe2) in name.
Can't set variable with illegal char "â" (0xffffffe2) in name.
Can't set variable with illegal char "â" (0xffffffe2) in name.
Can't set variable with illegal char "/" (0x2f) in name.
Can't set variable with illegal char "'" (0x27) in name.
Can't set variable with illegal char "" (0x1f) in name.
Can't set variable with illegal char "" (0x17) in name.
Can't set variable with illegal char "" (0x0f) in name.
Can't set variable with illegal char "" (0x07) in name.
The password-recovery mechanism is enabled.
The "MAC_ADDR" environment variable is not set.

Initializing Flash...
mifs[2]: 1 files, 1 directories
mifs[2]: Total bytes     :    6838272
mifs[2]: Bytes used      :    1956352
mifs[2]: Bytes available :    4881920
mifs[2]: mifs fsck took 1 seconds.
...done Initializing Flash.
done.
Loading ""...: no such device

Error loading ""

Interrupt within 5 seconds to abort boot process.
Boot process failed...

The system is unable to boot automatically. The BOOT
environment variable needs to be set to a bootable
image.

switch: dir

List of filesystems currently registered:

               bsdcs[0]: (read-only)
              bstage[1]: (read-only)
              fstage[2]: (read-write)
              xmodem[3]: (read-only)
                null[4]: (read-write)
           usbflash0[5]: (read-write)
                tftp[6]: (read-only)

switch: dir flash:
unable to stat flash:/: no such device

switch:

sonicimage1 · ‎12-28-2017

This is the output when I tried to boot off the USB with the same image:

File "usbflash0:c3750e-universalk9-mz.152-4.E5.bin" uncompressed and installed, entry point: 0x3000
executing...

Restricted Rights Legend

Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
(c) of the Commercial Computer Software - Restricted
Rights clause at FAR sec. 52.227-19 and subparagraph
(c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.

           cisco Systems, Inc.
           170 West Tasman Drive
           San Jose, California 95134-1706

Cisco IOS Software, C3750E Software (C3750E-UNIVERSALK9-M), Version 15.2(4)E5, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2017 by Cisco Systems, Inc.
Compiled Mon 18-Sep-17 06:58 by prod_rel_team
Initializing flashfs...
Using driver version 4 for media type 1
Can't set variable with illegal char "" (0xFFFFFFFF) in name.
...done Initializing flashfs.

Checking for Bootloader upgrade..
Boot Loader upgrade not required (Stage 2)
vb:: No such device
FIPS: Fatal failure: sp800108 failed

%Software-forced reload

00:00:23 UTC Mon Mar 1 1993: Unexpected exception to CPUvector 2000, PC = 32DA9E4
-Traceback= 0x32DA9E4z 0x2CFE290z 0x2CFDB88z 0x274D0A0z 0x274D26Cz 0x274D3BCz 0x70EBFCz 0x70EE5Cz 0x32DBCC8z 0x32D8048z

=== Flushing messages (00:00:24 UTC Mon Mar 1 1993) ===

Buffered messages:
Queued messages:
*Mar 1 00:00:24.175: %SYS-3-LOGGER_FLUSHING: System pausing to ensure console debugging output.

*Mar 1 00:00:19.092: Read env variable - LICENSE_BOOT_LEVEL =
*Mar 1 00:00:19.100: %LICENSE-2-VLS_ERROR: 'VLSaddLicenseStorage' failed with an error - rc = 145 - 'Error[145]: Failure in creating/opening the store.'
-Traceback= 6FF140z 285CCz 8528z 70E910z 70EE5Cz 32DBCC8z 32D8048z
*Mar 1 00:00:19.100: %HULC_LICENSE-1-LICENSE_REGISTER_FAILED: license register license storage 'flash:' failed - rc = 24
*Mar 1 00:00:19.100: %IOS_LICENSE_IMAGE_APPLICATION-6-LICENSE_LEVEL: Module name = c3750x Next reboot level = lanbase and License = No valid license found
Cisco IOS Software, C3750E Software (C3750E-UNIVERSALK9-M), Version 15.2(4)E5, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2017 by Cisco Systems, Inc.
Compiled Mon 18-Sep-17 06:58 by prod_rel_team

Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!
SRR0 = 0x02791418 SRR1 = 0x00029230 SRR2 = 0x032DA9E4 SRR3 = 0x00029230
ESR = 0x00000000 DEAR = 0x00000000 TSR = 0x84000000 DBSR = 0x10000000

CPU Register Context:
Vector = 0x00002000 PC = 0x032DA9E4 MSR = 0x00029230 CR = 0x24000022
LR = 0x032DA980 CTR = 0x006E9A5C XER = 0x00000000
R0 = 0x032DA980 R1 = 0x06581480 R2 = 0x00000000 R3 = 0x04A6F968
R4 = 0x00000000 R5 = 0x00000000 R6 = 0x06581450 R7 = 0x05300000
R8 = 0x00029230 R9 = 0x052F0000 R10 = 0x00008000 R11 = 0x00000000
R12 = 0x24000028 R13 = 0x1D981A10 R14 = 0x0070EE14 R15 = 0x00000000
R16 = 0x00000000 R17 = 0x00000000 R18 = 0x00000000 R19 = 0x00000000
R20 = 0x00000000 R21 = 0x00000000 R22 = 0x00000000 R23 = 0x00000000
R24 = 0x00000000 R25 = 0x00000000 R26 = 0x00000000 R27 = 0x04E50000
R28 = 0x00000009 R29 = 0x0523101C R30 = 0x04E520F8 R31 = 0x00000000

Stack trace:
PC = 0x032DA9E4, SP = 0x06581480
Frame 00: SP = 0x06581490    PC = 0x032DA980
Frame 01: SP = 0x06581498    PC = 0x02CFE290
Frame 02: SP = 0x065814A0    PC = 0x02CFDB88
Frame 03: SP = 0x065814B0    PC = 0x0274D0A0
Frame 04: SP = 0x065814D0    PC = 0x0274D26C
Frame 05: SP = 0x065814E8    PC = 0x0274D3BC
Frame 06: SP = 0x06581518    PC = 0x0070EBFC
Frame 07: SP = 0x06581620    PC = 0x0070EE5C
Frame 08: SP = 0x06581628    PC = 0x032DBCC8
Frame 09: SP = 0x00000000    PC = 0x032D8048

Switch uptime is 27 seconds

cisco SW-6750X-24S (PowerPC405) processor with 524288K bytes of memory.

Leo Laohoo · ‎12-28-2017

@sonicimage1 wrote:

*Mar 1 00:00:19.100: %LICENSE-2-VLS_ERROR: 'VLSaddLicenseStorage' failed with an error - rc = 145 - 'Error[145]: Failure in creating/opening the store.'
-Traceback= 6FF140z 285CCz 8528z 70E910z 70EE5Cz 32DBCC8z 32D8048z
*Mar 1 00:00:19.100: %HULC_LICENSE-1-LICENSE_REGISTER_FAILED: license register license storage 'flash:' failed - rc = 24
*Mar 1 00:00:19.100: %IOS_LICENSE_IMAGE_APPLICATION-6-LICENSE_LEVEL: Module name = c3750x Next reboot level = lanbase and License = No valid license found

Contact Cisco TAC and organize for the switch to be RMA-ed.

The switch isn't booting because the portion of the flash where the license is stored could be corrupt.

sonicimage1 · ‎12-29-2017

I tried an experiment - disconnected the stack cables from member switch, went into ROMMON, booted off the USB into the new IOS as a standalone switch. It came up fine and didn't restart itself. Then I gave a higher priority to switch 2, and connected the stack cables again. Switch 2 took over as a master, but switch 1 went on restarting in the same way as switch 2 previously did. This made me think that the problem may lie in the stack cables or stack modules, not the switches themselves.

sonicimage1 · ‎01-04-2018

I had to revert the switches to the previous IOS, they are up and running now. But I still can't understand why they didn't like the new one, or maybe it's just not compatible with that type of switches.