2960X overheat, upgrading to 15.2(3)E2

[jeckert]

FYI:

We ran into an overheating issue upgrading 2960x switches to 15.2(3)E2. I just wanted to get the word out, maybe save someone the trouble of an RMA.

We upgraded switch to 15.2(3)E2 on the reload it updates Front-end Microcode and fails post for inline power. The issue is that the system temp reads 0 and the fan never kicks on. If you don't do anything more with the switch, after a couple hours it will become hot enough to burn your hand and damage the switch.  Reload right away and the switch will come up fine and the fans work normally.

[Martin Hruby]

Hello

This sounds pretty serious. Have you opened a case with Cisco Support?

Best regards,
Martin

[jeckert]

Yep I started a case, they are going to RMA me a new switch.  I stated a few times what happened. I don't now if the RMA guy is going to log anything related to the actual cause but I tried.

It was kind of scary, I had a stack of 4 switches I was staging.  We moved away from using 15.0.2a-EX5 because a reload issue we ran into when connecting with SSH from a remote network.  After upgrade the stack, I simply started staging them.  The were not in the same room as me so I didn't notice that the fans were not running.  I while later I heard a loud pop noise and the top switch in stack was reloading. I ran out and they were too hot to touch with no fans running. After cooling 3 of the 4 still worked.

We did a test with another switch, upgraded from 15.0.2a-EX5  15.2(3)E2. after the initial  reload and microcode upload it fails the post and the Fans shut off.  a "show env all" shows that it thinks the temp is 0 so why would the fans kick on.  Another reload and switch came fine, post fine fans fine temp reading 32.

Switch#show env all
FAN is OK
SYSTEM TEMPERATURE is OK
System Temperature Value: 0 Degree Celsius
System Temperature State: GREEN
Yellow Threshold : 58 Degree Celsius
Red Threshold    : 68 Degree Celsius
SW  PID                 Serial#     Status           Sys Pwr  PoE Pwr  Watts
--  ------------------  ----------  ---------------  -------  -------  -----
 1  Built-in                                         Good

SW  Status          RPS Name          RPS Serial#  RPS Port#
--  -------------   ----------------  -----------  ---------
1   <>              <>

[zedat-voice]

We had the same issue here. We nearly escpaded an serious hardware replacement procedure of 9 Stacks (with more then 4 Switches per Stack) which we updated for testing. We also commented the release and provided a link to this forum post.

[jeckert]

Just think if someone upgraded a remote closet somewhere. At best, it only burns out the switch.

[Oscar Olsson]

Did you open a case with Cisco TAC? Did you receive a bugID? It would be great if we could follow that bug in order to know when it's safe to upgrade, and to what release. As I understand it, the recommended release, 15.0.2a-EX5, has other issues with SSH.

[jeckert]

I am thinking there isn't a good release out there for 2960X switches.  You have to pick your evil.

We moved to that release because of a customer that needed only ssh access. The switches cashed each time we tried to connect ssh running on 15.0.2a-EX5.  Go any further back an you have a CPU issue.

Anyway, I opened a case and got a replacement for the toasted switch but they did really want to listen to the details of what happened, only that it failed in a way the met the criteria for replacement. We opened another case strictly on the software issue, gave detailed logs showing the event and they said they could replicate it, but they were not using a 2060X just an older 2060. Yesterday we finally got an engineer that understood that they need to test on a 2960X to simulate an issue on 2960X switches.  So maybe soon they will have a bug for it.

[Oscar Olsson]

Ok, thanks for doing this for the community as well. FYI, I did manage to reproduce it as well just now. Went from 15.0(2)EX5 to 15.2(3)E2, in a stack of two switches, using the archive download-sw upgrade method. So it should be pretty trivial for the TAC to reproduce it as well. Sadly I can't open my own TAC case since we're too cheap to buy smartnet support contracts for the lower end switch ranges.

Regarding the software quality - there seems to be something wrong with the QA testing. Regardless of how bad code development tries to release, it shouldn't make it through solid QA testing. I wonder why the 2960-X series have software quality issues, while I don't really see the same thing with 3850. My best guess would be that 2960-X is based on an older (non-linux) architecture, and it might be difficult to get good people to keep developing that..? A bit like Nokia and Symbian.