Catalyst 3650-48FQ-E Switch Update

[hbell]

I inherited about 60 2960x and 3650 switches in my new position, all of which were not updated after installation in 2016-2017. I have gotten through all the 2960s and have started on the 3560s.  First up are a couple stand-alone Catalyst 3650-48FQ-E switches, and they are giving me hell.  I can update them through Fuji 16.9.8, but when I try to move to Gibraltar 16.9.1 I keep betting a boot loop. 

Booting...
Interface GE 0 link down***ERROR: PHY link is down

Getting rest of image
Reading full image into memory...Check base package header ...: done = 16384
Getting rest of image
Reading full image into memory....done
Reading full base package into memory...: done = 34200944
Bundle Image
--------------------------------------
Kernel Address : 0x534303c0
Kernel Size : 0x43824e/4424270
Initramfs Address : 0x5386860e
Initramfs Size : 0x1c0cf62/29413218
Compression Format: mzip

Bootable image at @ ram:0x534303c0
Bootable image segment 0 address range [0x81100000, 0x81da5280] is in range [0x80180000, 0x90000000].
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@boot_system: 377
Loading Linux kernel with entry point 0x81894d40 ...
Bootloader: Done loading app on core_mask: 0xf

### Launching Linux Kernel (flags = 0x5)

Linux version 4.9.177 (xelinux@sjc-xelinux1) (gcc version 5.3.0 (GCC) ) #1 SMP Fri Jun 7 18:13:03 PDT 2019
CVMSEG size: 2 cache lines (256 bytes)
Cavium Inc. SDK-5.1.0
bootconsole [early0] enabled
CPU0 revision is: 000d900a (Cavium Octeon II)
Checking for the multiply/shift bug... no.
Checking for the daddiu bug... no.
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(1,0)

Booting...

Any ideas about what's going on here?

[Reza Sharifi]

The only suggested version of Gibraltar on Cisco's site is:

cat3k_caa-universalk9.16.12.10a.SPA.bin

HTH

[hbell]

Yeah, that update leads to boot looping, too. As I said, 16.9.8 works, but any subsequent IOS update loops.  DJX995's post is exactly what's I'm experiencing with these devices that have reached or are approaching EOL.  I can't have them replaced until summer 2025, so it would be a bit reassuring to have them updated with all patches and bug fixes.

https://community.cisco.com/t5/switching/catalyst-c3850-gibraltar-16-12-1-ed/td-p/3907723

[Reza Sharifi]

Ok. Does the box have enough room on the flash for multiple images? 

[balaji.bandi]

I can update them through Fuji 16.9.8

that means the switch working as expected when you are in this version right ? what is the reason for upgrade to new code ?

can you post complete boot logs and also suggest to  check some issue reported on higher version :

https://community.cisco.com/t5/switching/catalyst-c3850-gibraltar-16-12-1-ed/td-p/3907723

[DanielP211 · Accepted solution]

Hello!

Perform a cold reboot by disconnecting the power cable from the switch.

If the issue persists, initiate a boot into ROMMON and extract the configuration from the switch.

Transfer the firmware to a USB flash drive and proceed to utilize the command provided to perform a full re-flash of the flash memory. - emergency-install usbflash:IMAGE
It that dosent work I belive you are encountering a bug that will require a RMA of the switches. The switches are still under support till 2026, so in case you have smartnet I suggest you contact TAC.

BR

[Leo Laohoo]

---

@hbell wrote:
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(1,0)

---

Why the need to downgrade to 16.9.1???

[hbell]

Luckily my 3650 was still under warranty, and TAC has sent me a replacement which I want to put back into service. The switch shipped with IOS ver  03.03.03SE which, of course, I would like to update to the newest version possible.  I do not want to send this replacement into "Mainboard hardware authentication failed, abort init..." effectively bricking the device.  I have sent a query to TAC, but I' will ask here, too. Is there an identifier (chassis serial, motherboard serial, etc.) that let's me know what IOS version is the last, best update for 3650s?

[Leo Laohoo]

---

@hbell wrote:
let's me know what IOS version is the last, best update for 3650s?

---

That's a trick question. 

The "official" Cisco response would be 16.12.10a is the latest "star" release. 

My answer is the latest 3.6.X or 3.7.X firmware. 

Why 3.X.X over 16.X.X firmware?  Because 16.X.X is bloatware and a memory leak hog.  A 3650/3850 on 16.X.X will need to be rebooted every 4 to 6 months because there are many extra processes added to the code that will cause memory leaks.  For purely a layer 2 function, switches on 3.6.X/3.7.X can have an uptime of several years without any issues.