PoE Switches keep failing

[Sean Lair]

We have an issue where switches are failing weekly in a switch closet.  In the past month we have gone through several 3750G switches and a couple 4510s.  The power supplies have eventually made a popping noise and had to be replaced.  on the 4510s we've tried two chassis and gone through several power supplies.

The switches have been behind UPS systems so should be receiving conditioned power.

Could load from the PoE devices really be causing this?  I wouldn't think it's power since they are behind a UPS.

Any ideas on how to tackle this issue?  We're going through switches like crazy...

Thanks

Sean

[InayathUlla Sharieff]

Sean,

Your troubleshooting approach is good.

First thing to check the UPS issue on the server room.

Are there any issues with non-poe switches?

The Switches which are failing POE are in the same rack? or they are placed in different racks and facing this issue?

Whats the version running on the devices?

Since how long you are seeing this issue? Was this working fne before? If yes were there any H/W or S/W configuration changes been made?

HTH

Regards

Inayath

[Sean Lair]

Thank you for your response.

I've received a little more information from the team working directly on this issue.

There are only 5 switches i the closet, 3 of them or PoE.  The PoE switches appear to be the ones that keep failing so far.  They are all in the same rack and on the same power.  I just checked the version of one of the currently running switches, it is at 12.2(35)SE5.

This issue started within the last month and was working fine before.  There haven't been any configuration changes.  However, there have been power changes/issues.  There was one electrical storm in the area that did cause other equipment on campus to fail, there haven't been any of these storms again yet and we still have failures.  There was also some remodling going on in the building.  This makes me think it is something to do with the power, but the electricians keep checking everything and say it's ok.  They have also moved the circuits to other panels, but the issues still occur.  They think the problem is on the load side, but load hasn't changed.

When one of the 4510 power supplies failed, it at first stopped delivering power to PoE devices and then within minutes, there was a very loud bang (like a shotgun) and the power supplies went offline.  At that time we also lost the UPS.

After that the UPS was replaced with a new APC 2200.  Today we had a 3750G fail and it is on that APC UPS.  It seems that each time a switch has failed, it has been accompanied by a popping noise.

Any thoughts?  Could PoE devices pull so much power that it could cause a power supply to fail?  I feel it's on the line side not the load side becaue of the timing, but the electricians feel it's on the load side and that the switches are pulling huge spikes in power before they blow.  They said they measured 40 amps to the 4510 before its power supply blew.

We are just in a bad spot as we are burning through our spare switches at fast rate here.

Thanks for any suggestions

Sean

[glen.grant]

    The power supplies will only supply so much power after which it will not power any more POE devices . Depends on what cards are the chassis etc.  To me it sounds like its on the power  side where there is something going on . If you can't figure anything out then  you may have to get some kind of power monitoring equipment in the closet to look at  what's going on with the power.  Use your show power commands to see how is being used and is still available.  If they pop before they going that still sounds like a line surge.

[jamescox3]

How are the temps in the room? I kind of doubt that they are anywhere near high enough to be problematic. How about humidity, or dust build up?

[Sean Lair]

The temperatures are more on the colder side so that should be OK.  Humidity levels are normal and while it may be a little dusty, it isn't bad.  In addition, the other switches and new power supplies failed quickly and didn't have build-up.