2960x switch booted an image that is not present in the flash.

CubeNetworksNOC · ‎05-15-2020

We have a switch 2960x-48LPS-L switch. We uploaded an new image c2960x-universalk9-mz.152-7.E2.bin to the flash and I can see it in flash. We changed the boot variable to boot the new image.

2#show boot
BOOT path-list : flash:c2960x-universalk9-mz.152-7.E2.bin

But still switch managed to get an old ios while reloading.

show version
Cisco IOS Software, C2960X Software (C2960X-UNIVERSALK9-M), Version 15.0(2)EX5, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2014 by Cisco Systems, Inc.
Compiled Fri 21-Feb-14 05:54 by prod_rel_team

ROM: Bootstrap program is C2960X boot loader
BOOTLDR: C2960X Boot Loader (C2960X-HBOOT-M) Version 15.2(4r)E3, RELEASE SOFTWARE (fc4)

on7241ax102 uptime is 1 day, 13 hours, 26 minutes
System returned to ROM by power-on
System restarted at 23:45:57 AEST Thu May 14 2020
System image file is "flash:/c2960x-universalk9-mz.150-2.EX5/c2960x-universalk9-mz.150-2.EX5.bin"

2#dir flash:
Directory of flash:/

644 -rwx 16374 May 14 2020 23:38:46 +10:00 config.text
646 -rwx 1756 May 15 2020 10:49:20 +10:00 vlan.dat
647 -rwx 17281 Jul 18 2017 21:31:24 +10:00 config_backup_17-07-2017_23-31-39_before_mera
648 -rwx 26542080 May 14 2020 19:19:41 +10:00 c2960x-universalk9-mz.152-7.E2.bin
651 -rwx 5144 May 15 2020 10:49:30 +10:00 multiple-fs
650 -rwx 3838 May 14 2020 23:38:54 +10:00 private-config.text

What is causiing the switch to pull an ios which is not even present in the flash? Really confusing . Please assist.

Leo Laohoo · ‎05-15-2020

648 -rwx 26542080 May 14 2020 19:19:41 +10:00 c2960x-universalk9-mz.152-7.E2.bin

Reboot the switch again and there is a strong chance the switch will go into ROMMON.

I suspect the IOS BIN file is corrupt because the filesize does not match with what is found in the Cisco website.

So what has happened are the following:

At around 19:19:41 AEST, 14 May 2020, someone copied (take note I did not use the term "uploaded") the file "c2960x-universalk9-mz.152-7.E2.bin" into the switch.
At 23:38:46 AEST, 14 May 2020, someone saved the config and rebooted the switch.
During the reboot, the switch tried to load "c2960x-universalk9-mz.152-7.E2.bin" but since the file is corrupt, the switch then proceeded to load the old firmware.
The switch recovered at 23:45:57 AEST, 14 May 2020 but still running 15.0(2)EX5, instead of 15.2(7)E2.
After the switch has recovered, someone then proceeded to delete the entire "c2960x-universalk9-mz.150-2.EX5" folder.

NOTE: Had someone did #5 first and then rebooted the switch, the switch would have gone into ROMMON.

The practice of "copying" the BIN file into a switch is something I have been warning everyone about. There is no way in the copy command to verify if the file is corrupt or not. Verifying the file is an added step and not everyone knows about it.

The only way to correct this mistake is to download the file "c2960x-universalk9-tar.152-7.E2.tar" from the Cisco website and use the tried-and-tested method:

archive download-sw tftp://<TFTP_IP_ADDRESS>/c2960x-universalk9-tar.152-7.E2.tar

IOS Upgrade on Catalyst Switches 2900, 3500 and 3700 - Easy as Pi

CubeNetworksNOC · ‎05-15-2020

Looks like the ios indeed was corrupted. worng md5 checksum value.

Martin L · ‎05-15-2020

it is weird; is it possible image is not what is says it is? or file bin was renamed ?

Leo Laohoo · ‎05-15-2020

@Martin L wrote:
or file bin was renamed ?

The BIN file can be renamed but the output to the "sh version" cannot be changed.

Martin L · ‎05-16-2020

Right, so you think you copied 15.2 while actually image is still 15.0 as shown by running switch. just 15.0 file was renamed as 15.2.

Leo Laohoo · ‎05-16-2020

@Martin L wrote:
Right, so you think you copied 15.2 while actually image is still 15.0 as shown by running switch. just 15.0 file was renamed as 15.2.

My suspicion was confirmed by someone who works there.