3650 Switch Upgrade Failure

[johnmedcraft]

Hi everyone -

currently got a 3650 switch that is failing to upgrade to code cat3k_caa-universalk9.SPA.03.06.03.E.152-2.E3.bin.  once copyied to flash: and added the boot system command (boot system switch all flash:cat3k_caa-universalk9.SPA.03.06.03.E.152-2.E3.bin) and rebooted it fails to load.

Booting...Initializing RAM +++++++@@@@@@@@...++++++++++++++++++++++++++++++++
Base ethernet MAC Address: 84:b2:61:4b:99:80

Interface GE 0 link down***ERROR: PHY link is down
Initializing Flash...

flashfs[7]: 0 files, 1 directories
flashfs[7]: 0 orphaned files, 0 orphaned directories
flashfs[7]: Total bytes: 6784000
flashfs[7]: Bytes used: 1024
flashfs[7]: Bytes available: 6782976
flashfs[7]: flashfs fsck took 2 seconds....done Initializing Flash.
Reading full image into memory....................................................................................................................................................................................................................................................................................................done
Bundle Image
--------------------------------------
Kernel Address    : 0x6042d274
Kernel Size       : 0x418331/4293425
Initramfs Address : 0x608455a8
Initramfs Size    : 0xe780c1/15171777
Compression Format: .mzip

Bootable image at @ ram:0x6042d274
Bootable image segment 0 address range [0x81100000, 0x82140000] is in range [0x80180000, 0x90000000].
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
File "flash:cat3k_caa-universalk9.SPA.03.06.03.E.152-2.E3.bin" uncompressed and installed, entry point: 0x8166a500
Loading Linux kernel with entry point 0x8166a500 ...
Bootloader: Done loading app on core_mask: 0xf

### Launching Linux Kernel (flags = 0x5)

EXT2-fs error (device sda1): ext2_check_descriptors: Block bitmap for group 16 not in group (block 195328)!
Waiting to save verification log to crashinfo
.............................Verification log saved to crashinfo as verify-drivers-1452183952076982340.log

Signature Verification Failed on /ucode0/cat3k_caa-drivers.SSA.03.06.01.PSR3.63.pkg

%IOSXEBOOT-4-SWTREE: (rp/0): Signature Verification Failed on /ucode0/cat3k_caa-drivers.SSA.03.06.01.PSR3.63.pkg

%IOSXEBOOT-4-ABORT: (rp/0): Signature Verification Failed on /ucode0/cat3k_caa-drivers.SSA.03.06.01.PSR3.63.pkg Failed to process provisioning file (/tmp/packages.conf).
%IOSXEBOOT-4-ABORT: (rp/0): Signature Verification Failed on /ucode0/cat3k_caa-drivers.SSA.03.06.01.PSR3.63.pkg Failed to process provisioning file (/tmp/packages.conf).

I have done an incremental upgrade and have been able to get to code cat3k_caa-universalk9.SPA.03.06.02a.E.152-2a.E2.bin.  this is the only switch out of 6 that has had this issue.  Just to confirm the IOS did not get corrupt I re-downloaded the IOS from the cisco site which did not fix the issue.  Install methods have been attempted via USB recovery and TFTP with code 03.06.02a running.

Does anyone have any ideas before I contact TAC?

Many Thanks

John

[Rajeshkumar Gatti · Accepted solution]

I suspect it is CSCuw82216

-Raj

[johnmedcraft]

Hi Raj

Based on the Bug that is the exact issue.  I haven't had a chance to test but will mark as correct.

Many Thanks,

john

[John Cook]

We are running into this too, but once corrupt we can't consistently get it to boot again with 3.6.3 (even after doing emergency recovery).  Will try a previous version to see if we can get them "stable" again.  I'm about to open a case, but was wondering if TAC had given you any other info yet.

Thanks,

John

[Landon]

We ran into this last night on 2 switches as well. We were able to get everything functional again by running the previous version of code (3.3.5 or 3.6.2). We opened a TAC case, but the engineer wasn't helpful at all. We even gave him the bug ID and got nowhere.

It seems like problem could easily be fixed if the right person worked on it. They just need to figure out why the flash is being mounted read-only and fix that. Surely there will be an interim release to fix it sometime soon as this is a pretty bad bug.

[John Cook]

Looks like the flash corruption bug is addressed in 3.7.3 that was released on Friday.  TAC has steered me toward that release but also said that 3.6.4 is due in Feb.  The case initially started with them wanting me to RMA due to flash failure, but after some webex and demonstrations that going back to earlier code resolved the issue, I got a little further.  What I found is that even with a seemingly good upgrade to 3.6.3, subsequent reboots could still dump it into rommon or a boot loop.  I left one in a loop over the weekend and this morning it had finally booted correctly.  Next reload it was back in the loop though.  We'll see what new bugs we hit in 3.7.3...

[alicato43]

hi all,

I have 200+ 3650 switches in my fleet and experienced this issue on one switch so far,

I was upgrading from a 3.6.2a to 3.6.3 for ise 2.0 and trustsec 5.3 recommended versions but stopped when experienced this bug. 3.6.4 just came out and the release notes doesn't mention that bug id CSCuw82216 isn't in the resolved caveats. as mentioned by john cook, we RMA's the switch due to being remotely. please advise. thanks

[Rajeshkumar Gatti]

Hi,

3.6.4 has a fix for CSCuw82216.

-Raj

[alicato43]

hi raj,

I cannot see this bug fixed in the release notes:

http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3650/software/release/3e/release_notes/OL3264701.html

[Rajeshkumar Gatti]

Good point. I will work with the doc team and get this updated.

For now the information about the fix is listed here as well. It mentions for 3850 but the same info is applicable for 3650 as well.

Note: Software defect CSCuw82216may result in switch inoperability due to flash memory corruption when upgrading from a pre-3.6.3 or pre-3.7.2 software release to 3.6.3/3.7.2 in install mode. This issue is resolved in 3.6.4 or 3.7.3 or later software therefore the issue is not seen when upgrading to these software versions. As a result of this software defect, bundle mode is recommended for upgrades involving the effected software versions

-Raj

[John Cook]

Hello Raj,

Just to follow up on this - it looks like 3.6.3 is still listed as the starred version ("Cisco Suggested release based on software quality, stability and longevity.") as of this morning.  Since this bug is listed as catastrophic, it seems it would make sense to remove this as the suggested release in the 3.6 train.  

Just my 2 cents.  Thanks for your feedback though.

-John

[marco.merlo]

Hi John,

I agree with you,  maybe I am not a lucky guy but my experiences with "starred" versions (IOS, XE, XR, ASA) have been quite painful.  I even asked our TAM to ask Cisco to remove the temptress star ....

MM