C2960X switch issues after image upgrade

[mmccallebaut]

Our client has 1600 switches installed of the type WS-C2960X-24PS-L.

We did an image upgrade from IOS 15.2(4)E to 15.2(4)E7 (done by Prime).

Unfortuantely we encountered several problems:

1) 2% didn't start up. They were blocked on "POST : Thermal, Fan Tests : Begin". I found some bugs that is also referring to this problem namely CSCvf24186 and CSCtx90729. Some switches were OK after a power reset but not all. For about 2 or 3 we need to do a RMA.

2) because of all those problems with the image upgrade by Prime we did 201 switches via an interactive shell script. Now all the switches came back on line , even upgraded BUT we encountered issues with POE in some switches . No POE was granted on the interfaces so AP's and IP phones stayed off line. Here we didn't find a bug nor a solution yet. A power reset did not always result in  a full POE working switch.

So my big question here is are those C2960X switches so difficult to do an upgrade on and why we have so many problems in doing so? Our client wants us to change the image every year !

Thanks for a reaction.

Regards

Mike

[balaji.bandi]

First of all i am not great fan of upgrading the so many switches with Prime.

Second - before upgrading so many switches to the desired version you choose to upgrade, have you tested any limited number of switches before you go and deploy on mass switches?

if i were you i compare the config any changes done - in related to Power and other configs - which are not working.

if this is impacting the business, i raise a TAC case to P1 level since you have many switches gone down it impacting the business.

Next step - read the release notes, and plan a correct upgrade with stage level few switches until it proves all working as expected in the stage level, then test for couple of days, then do the mass upgrade.

make sense?

Coming to the problem you have, what is the Logs show? any indications?

[mmccallebaut]

Some extra information:

1) Upgrade was done in waves . 15 Waves in total of about 100 switches per wave. So not all 1600 switches were done at the same time !

2) of course we did an upgrade of 60 switches first in order to evaluate the working of the new IOS. No issues after that upgrade.

3) All the switches are having the same startup-configuration (loaded via tftp).

4) I'm trying to figure out if I can avoid such things in the future before raising a P1 TAC case

5) As said before when a switch doesn't boot it is stuck on the POST thermal FAN test . The only way so far we found to deblock this situation is a power reset of the switch with ALL cables deconnected , even the console cable !

6) an upgrade of 201 switches was done afterwards , using an interactive Expect script by 10 switches at a time. Here before doing the reload I configured the command "boot fast". All the switches were ok after reload but we encountered POE problems afterwards. Switch was sending snmptrap "POWER USAGE ABOVE THRESHOLD" all the time ... Nothing found in loggings. Power inline never and again auto on the interfaces didn't change anything. Again a power reset had to be done with no cables connected and "boot fast" OFF !

For the moment I'm still searching for a solution.

Thanks

Regards

Mike

[balaji.bandi]

Nice, that seems to be a reasonable approach.  just to understand if you physically down the stack and bring is that works ?

or do you still having issue with PoE after hard boot the switch ?

[mmccallebaut]

The C2960X switches are stand-alone devices , no stacking !

[telmo.castor]

---

@mmccallebaut wrote:

Our client has 1600 switches installed of the type WS-C2960X-24PS-L.

We did an image upgrade from IOS 15.2(4)E to 15.2(4)E7 (done by Prime).

Unfortuantely we encountered several problems:

1) 2% didn't start up. They were blocked on "POST : Thermal, Fan Tests : Begin". I found some bugs that is also referring to this problem namely CSCvf24186 and CSCtx90729. Some switches were OK after a power reset but not all. For about 2 or 3 we need to do a RMA.

2) because of all those problems with the image upgrade by Prime we did 201 switches via an interactive shell script. Now all the switches came back on line , even upgraded BUT we encountered issues with POE in some switches . No POE was granted on the interfaces so AP's and IP phones stayed off line. Here we didn't find a bug nor a solution yet. A power reset did not always result in  a full POE working switch.

So my big question here is are those C2960X switches so difficult to do an upgrade on and why we have so many problems in doing so? Our client wants us to change the image every year !

Thanks for a reaction.

Regards

Mike

---

I am facing same issue, my SW C2960XR recovered after 4th reboot. Does those SW back to facing problems?