IOS Upgrading

salemmahara · ‎04-14-2018

Hello everyone

As you know, nowadays "keeping devices up to date" is massively emphasizing by everyone.

So we tried to upgrade our switches to a more secure version recommended by Cisco.

During upgrade process something went wrong! We have a set of 2960X with the same feature set and ... . After upgrading , 4 of them didn't come up! Just stuck in booting with high fan speed . We tried to use another IOS version but it didn't work again. We had to downgrade to the version stored on flash.

But one of them didn't recover and Syst LED keeps amber ! When I checked logs, there was something like this:

This switch may not have manufactured by Cisco or with Cisco's authorization. This product may contain software that was copied in violation of Cisco's license term..............

This logs removed after restart.

Note: We download IOS versions from Cisco Website. But no one of them worked on these switches.

We could recover 3 of 4 to the version stored on flash but the last one didn't work even with the IOS on it. There was an interesting note! Clients directly connected to switch could telnet to it, everything was OK , switch was operational, could ping other switches on network from the switch console but it didn't forward client traffics out! ( Syst LED was Amber) .

What do you recommend ? I think these switches are not original ! Because the seller is not a authorized person. BTW, here is the output of Show version, Show Boot and Show post on the switch with amber Syst LED :

Switch#sho boot
BOOT path-list      : flash:c2960x-universalk9-mz.152-4.E/c2960x-universalk9-mz.152-4.E.bin
Config file         : flash:/config.text
Private Config file : flash:/private-config.text
Enable Break        : yes
Manual Boot         : no
Allow Dev Key         : yes
HELPER path-list    :
Auto upgrade        : yes
Auto upgrade path   :
Boot optimization   : disabled
NVRAM/Config file
      buffer size:   524288
Timeout for Config
          Download:    0 seconds
Config Download
       via DHCP:       disabled (next boot: disabled)
==========================
cisco WS-C2960X-48FPD-L (APM86XXX) processor (revision E0) with 524288K bytes of memory.
Processor board ID -------
Last reset from reload command
2 Virtual Ethernet interfaces
1 FastEthernet interface
50 Gigabit Ethernet interfaces
2 Ten Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.

512K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address       : -------
Motherboard assembly number     : -------
Power supply part number        : -------
Motherboard serial number       : -------
Power supply serial number      : -------
Model revision number           : E0
Motherboard revision number     : A0
Model number                    : WS-C2960X-48FPD-L
Daughterboard assembly number   : -------
Daughterboard serial number     : -------
System serial number            : -------
Top Assembly Part Number        : -------
Top Assembly Revision Number    : C0
Version ID                      : V01
CLEI Code Number                : CMMLJ00ARA
Daughterboard revision number   : A0
Hardware Board Revision Number : 0x05


Switch Ports Model                     SW Version            SW Image
------ ----- -----                     ----------            ----------
*    1 52    WS-C2960X-48FPD-L         15.2(4)E              C2960X-UNIVERSALK9-M
=========================

POST: MA BIST : Begin
POST: MA BIST : End, Status Passed

POST: TCAM BIST : Begin
POST: TCAM BIST : End, Status Passed

POST: Inline Power Controller Tests : Begin
POST: Inline Power Controller Tests : End, Status Passed

POST: Thermal, Fan Tests : Begin
POST: Thermal, Fan Tests : End, Status Passed

POST: PortASIC Stack Port Loopback Tests : Begin
POST: PortASIC Stack Port Loopback Tests : End, Status Passed

POST: PortASIC Port Loopback Tests : Begin
POST: PortASIC Port Loopback Tests : End, Status Passed

POST: EMAC Loopback Tests : Begin
POST: EMAC Loopback Tests : MAC Loopback Passed
POST: EMAC Loopback Tests : PHY Loopback Passed
POST: EMAC Loopback Tests : End, Status Passed

   We tried 15.2(6)E1 and 15.2(4)E6 and tried to rollback to 15.2(4)E (Stored on flash) but ....

Other switches are working fine with these IOS versions , downloaded from the same FTP and TFTP server with the same MD5 hash. We are extremely sure about files' health .

Georg Pauwen · ‎04-14-2018

Hello,

--> This switch may not have manufactured by Cisco or with Cisco's authorization. This product may contain software that was copied in violation of Cisco's license term.............. <--

Check if your serial numbers are valid using the website below::

https://cway.cisco.com/mydevices/devices

or give TAC a call:

https://www.cisco.com/c/en/us/support/web/tsd-cisco-worldwide-contacts.html

salemmahara · ‎04-15-2018

Hello

Thanks for replying

Could you check my reply to Leo please?

Leo Laohoo · ‎04-14-2018

The biggest mistake a lot of people make, when it comes to picking the IOS version to move to, is choosing one with a gold star next to it. Big mistake!

Next big mistake a lot of people make? Because they've picked one with a bright and shiny gold star, no one cares to read about the Release Notes. Hey, it's THE gold star. What can go wrong, right? Uh-huh.

Allow me to let you "in" a little secret: See the IOS version you've picked, 15.2(4)E? Notice the last character is a letter? Guess what is missing? A number. ANY number. Here's the secret: An IOS version that ends with a letter means this is "version 0". This means that no one has openly tested this in the wild. If a stable IOS is what you're aiming for, pick a version with a high number, like "3" and above.

I remembered the first time I've ever upgraded a 2960X, running EX3, there was a bug with the microcode. The bug means that when I upgraded from EX3 to any version, I have to manually power-down the stack. Issuing the command "reload" would force the each individual switch to halt/stop the boot-up process midway. The switch won't even boot into ROMmon. It'll just stop.

Apologies for the long response and I hope it helps (and makes sense).

salemmahara · ‎04-15-2018

Hello Leo

Thanks for replying.

As I mentioned before, I went for 15.2(6)E1 at first . 15.2(4)E is the shipped IOS on switch. When I tried the new IOS and it didn't work, I tried the original one.

BTW, I found something strange . There is log I attached here. The switch is showing Authentication_Faild and .... .

I checked the license . Switch lists 2 Licenses( LanLite and LanBase) but when I try Show license right-to-use detail command, both licenses are in Inactive state !

How can I make it active ? IOS commands showed on the internet just don't work and are not supported on my switch.

Leo Laohoo · ‎04-15-2018

If you went from 15.2(4)E to 15.2(6)E1 the I want to see the switch attempt to boot up 15.2(6)E1.

@salemmahara wrote:

The switch is showing Authentication_Faild and .... .

You mean like this:

POST: ACT2 Authentication : Begin  
POST: ACT2 Authentication : End, Status Passed FlexStack Module SmartChip
                            Authentication Failed

If this is the case then read THIS.

salemmahara · ‎04-15-2018

NO, like This:

POST: ACT2 Authentication : Begin  
POST: ACT2 Authentication : End, Status Failed

Nothing more. I tried a hard reboot with no result.

I'll send the output of upgrading to 15.2(6)E1 but everything is same as 15.2(4)E. This Failure happens again. I'll try other IOS versions like 15.2.4E5, E3 .

Leo Laohoo · ‎04-15-2018

Is there a FlexStack module installed? If it does, remove it and try again with the new IOS.

WS-C2960X False “%ILET-1-DEVICE_AUTHENTICATION_FAIL:” Messages and No Link Up on SFP Uplinks

salemmahara · ‎04-15-2018

No Flexstack or.... . Only 2 GLC-T transceiver . I removed them and had a hard boot. Nothing changed!

Leo Laohoo · ‎04-15-2018

Console into the switch and re-boot. We want to see the entire boot-up process.

salemmahara · ‎04-18-2018

System configuration has been modified. Save? [yes/no]: y
Building configuration...
[OK]
Proceed with reload? [confirm]

Apr 15 20:06:38: %SYS-5-RELOAD: Reload requested by -------- on console. Reload Reason: Reload command.
CPU rev: B
Image passed digital signature verification
Board rev: 5
Testing DataBus...
Testing AddressBus...
Testing Memory from 0x00000000 to 0x1fffffff.../
Using driver version 4 for media type 1
Xmodem file system is available.
Base ethernet MAC Address: ---------
The password-recovery mechanism is enabled.
USB EHCI 1.00
USB EHCI 1.00
USB Console INIT
Initializing Flash...
mifs[5]: 12 files, 1 directories
mifs[5]: Total bytes : 1806336
mifs[5]: Bytes used : 833536
mifs[5]: Bytes available : 972800
mifs[5]: mifs fsck took 1 seconds.
mifs[6]: 1 files, 1 directories
mifs[6]: Total bytes : 3870720
mifs[6]: Bytes used : 971264
mifs[6]: Bytes available : 2899456
mifs[6]: mifs fsck took 0 seconds.
mifs[7]: 5 files, 1 directories
mifs[7]: Total bytes : 258048
mifs[7]: Bytes used : 8192
mifs[7]: Bytes available : 249856
mifs[7]: mifs fsck took 1 seconds.
mifs[8]: 5 files, 1 directories
mifs[8]: Total bytes : 258048
mifs[8]: Bytes used : 8192
mifs[8]: Bytes available : 249856
mifs[8]: mifs fsck took 0 seconds.
mifs[9]: 677 files, 19 directories
mifs[9]: Total bytes : 122185728
mifs[9]: Bytes used : 57441280
mifs[9]: Bytes available : 64744448
mifs[9]: mifs fsck took 55 seconds.
...done Initializing Flash.
Loading "flash:c2960x-universalk9-mz.152-6.E1.bin"...Verifying image flash:c2960x-universalk9-mz.152-6.E1.bin.......................................................................................................................................................................................................................................................................................................................................................................................................................Image passed digital signature verification
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
File "flash:c2960x-universalk9-mz.152-6.E1.bin" uncompressed and installed, entry point: 0x3000
executing...

Restricted Rights Legend

Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
(c) of the Commercial Computer Software - Restricted
Rights clause at FAR sec. 52.227-19 and subparagraph
(c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.

cisco Systems, Inc.
170 West Tasman Drive
San Jose, California 95134-1706

Cisco IOS Software, C2960X Software (C2960X-UNIVERSALK9-M), Version 15.2(6)E1, RELEASE SOFTWARE (fc4)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2018 by Cisco Systems, Inc.
Compiled Tue 06-Mar-18 06:58 by prod_rel_team
Initializing flashfs...
Using driver version 4 for media type 1
mifs[7]: 12 files, 1 directories
mifs[7]: Total bytes : 1806336
mifs[7]: Bytes used : 833536
mifs[7]: Bytes available : 972800
mifs[7]: mifs fsck took 0 seconds.
mifs[7]: Initialization complete.

mifs[8]: 1 files, 1 directories
mifs[8]: Total bytes : 3870720
mifs[8]: Bytes used : 971264
mifs[8]: Bytes available : 2899456
mifs[8]: mifs fsck took 0 seconds.
mifs[8]: Initialization complete.

mifs[9]: 5 files, 1 directories
mifs[9]: Total bytes : 258048
mifs[9]: Bytes used : 8192
mifs[9]: Bytes available : 249856
mifs[9]: mifs fsck took 0 seconds.
mifs[9]: Initialization complete.

mifs[10]: 5 files, 1 directories
mifs[10]: Total bytes : 258048
mifs[10]: Bytes used : 8192
mifs[10]: Bytes available : 249856
mifs[10]: mifs fsck took 0 seconds.
mifs[10]: Initialization complete.

mifs[11]: 677 files, 19 directories
mifs[11]: Total bytes : 122185728
mifs[11]: Bytes used : 57441280
mifs[11]: Bytes available : 64744448
mifs[11]: mifs fsck took 1 seconds.
mifs[11]: Initialization complete.

...done Initializing flashfs.
Checking for Bootloader upgrade..
Boot Loader upgrade not required (Stage 2)

FIPS: Flash Key Check : Begin
FIPS: Flash Key Check : End, Not Found, FIPS Mode Not Enabled

extracting front_end/front_end_ucode_info (43 bytes)
Software version is different so extracting
the whole bundle

POST: MA BIST : Begin
POST: MA BIST : End, Status Passed

POST: TCAM BIST : Begin
POST: TCAM BIST : End, Status Passed

POST: ACT2 Authentication : Begin
POST: ACT2 Authentication : End, Status Failed
Waiting for Stack Master Election...
POST: Thermal, Fan Tests : Begin
POST: Thermal, Fan Tests : End, Status Passed

POST: PortASIC Stack Port Loopback Tests : Begin
POST: PortASIC Stack Port Loopback Tests : End, Status Passed

POST: PortASIC Port Loopback Tests : Begin
POST: PortASIC Port Loopback Tests : End, Status Passed

POST: EMAC Loopback Tests : Begin
POST: EMAC Loopback Tests : MAC Loopback Passed
POST: EMAC Loopback Tests : PHY Loopback Passed
POST: EMAC Loopback Tests : End, Status Passed

Election Complete
Switch 1 booting as Master
Waiting for Port download...Complete
Initializing Port Extension Feature Support...

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco WS-C2960X-48FPD-L (APM86XXX) processor (revision E0) with 524288K bytes of memory.
Processor board ID ------
Last reset from power-on
1 Virtual Ethernet interface
1 FastEthernet interface
50 Gigabit Ethernet interfaces
4 Ten Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.

512K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address : -------
Motherboard assembly number : ------
Power supply part number : --------
Motherboard serial number : -------
Power supply serial number : -------
Model revision number : E0
Motherboard revision number : A0
Model number : WS-C2960X-48FPD-L
Daughterboard assembly number : -------
Daughterboard serial number : --------
System serial number : --------
Top Assembly Part Number : -------
Top Assembly Revision Number : C0
Version ID : V01
CLEI Code Number : --------
Daughterboard revision number : A0
Hardware Board Revision Number : 0x05

Switch Ports Model SW Version SW Image
------ ----- ----- ---------- ----------
* 1 54 WS-C2960X-48FPD-L 15.2(6)E1 C2960X-UNIVERSALK9-M

Press RETURN to get started!

********************

The Syst LED is amber after boot process and switch generates attached log periodically.

Leo Laohoo · ‎04-19-2018

@salemmahara wrote:

POST: ACT2 Authentication : End, Status Failed

Error message still tells me it's THIS.

Leo Laohoo · ‎07-15-2020

@salemmahara wrote:

CPU rev: B
Image passed digital signature verification

...
Image passed digital signature verification

...
POST: ACT2 Authentication : Begin
POST: ACT2 Authentication : End, Status Failed

...
Model number : WS-C2960X-48FPD-L

...
Version ID : V01

Based on the recently-released F-Secure document about fake Cisco kit, I suspect this switch is a fake.

jsharpe · ‎10-14-2018

Hi Everyone,

I encountered the exact same issue after upgrading IOS from 150-2.EX5 to 152-4.E6 on a stack of 5 2960x switches, after remotely reloading the stack, it wouldn't come back up. After arriving on site to diagnose the problem, 2 of the 5 switches were displaying the orange system light.

After searching a number of forum posts I could see that the ACT2 failure/ILET-1-Device_Authentication_Fail issue was resolved by cold starting the switches by removing the power cable for 5 minutes before turning the switch back on. This did not work for me, I tried this numerous time for the two affected switches as well as booting the switches without the stacking module and rolling back the IOS as well as upgrading to an even newer version of IOS.

If this happens to you and the power cycle doesn't fix the issue, get hold of TAC as the switches will need to be replaced. luckily enough we had spare port capacity on other members of the stack to cable endpoints into the working switches but it was extremely tight. Hope this helps the next person that encounters the issue, I would've saved myself a few hours of panic if I knew the switches couldn't be fixed!