dead 3850-48F after upgrade from 03.02.03.SE to16.12.6

kriza · ‎10-06-2021

Hello.

I upgraded five of 3850-48F-L from 03.02.03.SE to16.12.6. After reload SYST was blinking fast (for about 2 hours) and nothing done. No output from console. After turning them off and turning them on there is no light on front panel and still no console output. After turning on fans just max out two short times and then spinning slowly.

I upgrade using cat3k_caa-universalk9.16.12.06.SPA.bin that was downloaded to flash:.

Then as written here

https://www.cisco.com/c/en/us/support/docs/switches/catalyst-3850-series-switches/117552-technote-cat3850-00.html?referring_site=RE&pos=3&page=https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3850/hardware/quick/guide/cat3850_gsg.html#anc5

I use command to upgrade:

software install file flash:cat3k_caa-universalk9.16.12.06.SPA.bin

At the end I answer no to question

[1 2]: Do you want to proceed with reload? [yes/no]:

Because I wanted them to reload later.

After reloading they are in state I described at the beginning.

I tried holding mode button when plugging in, holding reset button, still nothing. All I was able to get to console was one character after turning on - � .

I am out of ideas. I hope they are not dead. Please help me.

balaji.bandi · ‎10-06-2021

Is this part of the STACK ?

Turn off the Device completly, wait for 10min,(connect the console cable - post complete output here) Turn ON follow below factory reset : ( try to change the baudrate on the console cable see if you get any output ?)

https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3850/software/release/16-9/configuration_guide/sys_mgmt/b_169_sys_mgmt_3850_cg/performing_factory_reset.html

SYST was blinking fast - After that what LED show front ?

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

kriza · ‎10-06-2021

No switches are not stacked.

After turning on device:

-when using mini usb console output is nothing.

-when using serial console output is:

"

�

"

So I can not perform factory reset as described in link you provided

After more than 2 hours blinking SYST I tried to turn of and on device and then replace them with other devices because it was part of working infrastructure.

Leo Laohoo · ‎10-06-2021

@kriza wrote:

After turning them off and turning them on there is no light on front panel and still no console output.

All 5 units are behaving like this?

kriza · ‎10-06-2021

Yes all five of them.

Leo Laohoo · ‎10-06-2021

@kriza wrote:

Yes all five of them.

I find it hard to believe all five switches are dead.

Please verify if the console terminal is correctly set.

kriza · ‎10-07-2021

Yes I do not believe too.

I am trying:

screen /dev/ttyS0 9600

or

screen /dev/ttyACM0 9600

As described here

https://software.cisco.com/download/home/284455420/type/282855122/release/3.1

tmartins1 · ‎09-16-2022

Hi guys !

I have a switch same model with the same problem.
Yestarday i try to upgrade firmware switch and after install firmware in the realod process switch is not back.
I and many people on the support team try all to start the switch we can not turn-on the switch.
We can not believe so the switch is dead.

morcowbel293 · ‎01-24-2023

I'm having this exact problem with a 3850-24P. Performed update using "software install" command. After a reboot I am in the same state as OP. No CLI output and no lights on the front. It sounds like a normal boot, with the fans running up to max and then winding down. I've tried every speed on the console line. I've tried holding the button on the front as well as pressing rapidly on power up and sending a break my terminal program. Nothing works. Hoping for an RMA at this point.

Oleg8r · ‎02-12-2023

I have the same issue, after installing Gibraltar iOS switch is dead. No lights, no console connection.

Leo Laohoo · ‎02-12-2023

Contact Cisco TAC and raise an RMA.

Oleg8r · ‎02-13-2023

Yes, I've organised RMA already, however I have a bunch of switches which need to be upgraded and I would like to know what went wrong before I touch them.

Leo Laohoo · ‎02-13-2023

@Oleg8r wrote:
however I have a bunch of switches which need to be upgraded and I would like to know what went wrong before I touch them.

Ok, then I have a "big ask" (feel free to say "no"): List down the serial numbers of the dead switches. I am particularly interested in the first seven characters of the serial number (do not post the last four characters).

Oleg8r · ‎02-13-2023

Thank for your reply.
I have only one dead switch so far. But I’m afraid to kill those are in production with firmware upgrade cause RMA takes time.
Here is a serial of a dead switch - FCW1835C___
Also please find attached a log from software installation.

Unmounting ng3k filesystems...
Unmounted /dev/sda3...
Warning! - some ng3k filesystems may not have unmounted cleanly...
Please stand by while rebooting the system...
Restarting system.

Booting...Initializing and Testing RAM ++++@@@@####...########################## ######++@@++@@++@@++@@++@@++@@++@@++@@++@@++@@++@@++@@++@@++@@++@@++@@done.
Memory Test Pass!

Base ethernet MAC Address: bc:16:f5:9c:6f:00

Interface GE 0 link down***ERROR: PHY link is down
Initializing Flash...

flashfs[7]: 0 files, 1 directories
flashfs[7]: 0 orphaned files, 0 orphaned directories
flashfs[7]: Total bytes: 6784000
flashfs[7]: Bytes used: 1024
flashfs[7]: Bytes available: 6782976
flashfs[7]: flashfs fsck took 1 seconds....done Initializing Flash.
Getting rest of image
Reading full image into memory....done
Reading full base package into memory...: done = 32791733
Nova Bundle Image
--------------------------------------
Kernel Address : 0x6042e3c4
Kernel Size : 0x4381df/4424159
Initramfs Address : 0x608665a3
Initramfs Size : 0x1ab4f12/28004114
Compression Format: mzip

Bootable image at @ ram:0x6042e3c4
Bootable image segment 0 address range [0x81100000, 0x81da5280] is in range [0x80180000, 0x90000000].
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @@@@@@@@@@@@@@@@@@@@@@boot_system: 380
Loading Linux kernel with entry point 0x81895bf0 ...
Bootloader: Done loading app on core_mask: 0xf

### Launching Linux Kernel (flags = 0x5)

Linux version 4.9.187 (sumikum7@sjc-ads-5670) (gcc version 5.3.0 (GCC) ) #1 SMPFri Nov 19 03:09:46 PST 2021
CVMSEG size: 2 cache lines (256 bytes)
Cavium Inc. SDK-5.1.0
bootconsole [early0] enabled
CPU0 revision is: 000d900a (Cavium Octeon II)
Checking for the multiply/shift bug... no.
Checking for the daddiu bug... no.
%IOSXEBOOT-4-FLASH: (rp/0): deleting cat3k_caa-universalk9.16.12.08.SPA.bin because free space on flash is less than 300MB
%IOSXEBOOT-3748f25d3076de52df7db2641f93e50c-new_cksum: (rp/0): 4
%IOSXEBOOT-3748f25d3076de52df7db2641f93e50c-saved_cksum: (rp/0): 4
%IOSXEBOOT-Thu-###: (rp/0): Feb 9 23:55:05 Universal 2023 PLEASE DO NOT POWER CYCLE ### BOOT LOADER UPGRADING 4
● tdl.service - TDL Resolve
Loaded: loaded (/lib/systemd/system/tdl.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Thu 2023-02-09 23:55:07 Universal; 74ms ago
Process: 4044 ExecStart=/usr/binos/conf/tdl_boottime.sh (code=exited, status=1/FAILURE)
Main PID: 4044 (code=exited, status=1/FAILURE)

[ 246.117709] linux systemd[1]: Starting TDL Resolve...
[ 264.527899] linux tdl_boottime.sh[4044]: luajit: /usr/binos/conf/epoch_lib.lua:850: attempt to compare two nil values
[ 264.528759] linux tdl_boottime.sh[4044]: stack traceback:
[ 264.529566] linux tdl_boottime.sh[4044]: /usr/binos/conf/epoch_lib.lua:850: in function 'add_and_merge_record'
[ 264.530380] linux tdl_boottime.sh[4044]: /usr/binos/conf/epoch_lib.lua:868: in function 'generate_merged_metadata'
[ 264.531170] linux tdl_boottime.sh[4044]: /usr/binos/conf/epoch_lib.lua:958: in function 'generate_domain'
[ 264.531936] linux tdl_boottime.sh[4044]: /usr/binos/conf/epoch_resolve.lua:209: in main chunk
[ 264.532788] linux tdl_boottime.sh[4044]: [C]: at 0xaab8d9cce0
[ 264.540383] linux tdl_boottime.sh[4044]: Time to check parameters: 0.000241 seconds.
[ 264.541814] linux tdl_boottime.sh[4044]: Time to build package table: 0.002847 seconds.
[ 264.542611] linux tdl_boottime.sh[4044]: Time to cache objects: 8.208803 seconds.
[ 264.548671] linux systemd[1]: tdl.service: Main process exited, code=exited, status=1/FAILURE
[ 264.551950] linux systemd[1]: Failed to start TDL Resolve.
[ 264.565987] linux systemd[1]: tdl.service: Unit entered failed state.
[ 264.566838] linux systemd[1]: tdl.service: Failed with result 'exit-code'.
Feb 9 23:55:07.710: %BOOT-3-SYSD_STARTFAIL: R0/0: Failed to launch boot task tdl.service ( exit-code )
>>> Rebooting
Failed to set wall message, ignoring: The name org.freedesktop.login1 was not provided by any .service files
Failed to reboot system via logind: The name org.freedesktop.login1 was not provided by any .service files
% FACTORYRESET - Exiting without calling factory_reset
watchdog: watchdog0: watchdog did not stop!
reboot: Restarting system