Solved: SG300-28P: POE power drops every several minutes

richlafferty · ‎09-11-2012

Hi all,

We have an SG300-28P which, at intervals between 5 and 30 minutes or so, drops power to all of its POE-powered ports and then immediately restores power. The switch logs nothing when this happens other than recording that the ports went down and back up, even with debug logging enabled. Non-POE ports are not affected.

When the problem started there were only four POE-powered devices on the switch and we are far under our power budget. (While testing I added a fifth device, see below.)

Not all combinations of devices cause the drops to occur -- specifically, it looks like having devices on ports near each other causes the drops. When the drops occur, the absent counters of all of the POE devices increment. I noticed this earlier thread here seemed similar:

https://supportforums.cisco.com/thread/2165089

but in that case, he got no power at all in the adjacent configurations, instead of having it drop every several minutes.

I'm running

sw06#sh ver

SW version 1.1.2.0 ( date 12-Nov-2011 time 23:34:26 )

Boot version 1.1.0.6 ( date 11-May-2011 time 18:31:00 )

HW version V02

I have the new 1.2.2.7.6 firmware ready to go, I'll update this when I have a chance to test it. Unfortunately it is a production switch, but it's one with relatively few devices on it so far. If need be, I've got an sf300-48p I can swap in to dedicate this sg300 to testing.

Thanks for anything you might suggest.

Here's the details:

- "Phone" is an Avaya 4610SW or 5610SW IP phone (class 2)

- "Watchdog" is a IT Watchdogs Watchdog 100 environmental monitor (class 0)

- "WAP" is a Cisco WAP4410N (class 3)

First failing combination:

gi10 phone #1

gi22 phone #2

gi23 watchdog

gi24 wap

would cause POE power to drop on all those ports every 5-30 minutes.

I turned off Green Ethernet entirely -- no change.

Then did the following tests:

1. "power inline never" on gi23 and gi24: problem went away

2. "power inline auto" on gi23 and gi24: problem came back

3. "power inline never" on gi23 only: problem went away

4. "power inline auto" on gi23: problem came back

5. "power inline never" on gi24 only: problem went away

6. "power inline auto" on gi24: problem came back.

I left gi23 powered down overnight and it was stable: made it ~10h without any issues.

This morning tried the following configs:

08:54 - worked fine

gi4 watchdog

gi10 phone #1

gi22 phone #2

gi24 wap

10:31 - added another phone, problem came back

gi4 watchdog

gi10 phone #1

gi22 phone #2

gi23 phone #3

gi24 wap

10:43 - moved phone #3 to gi11 (adjacent to gi12) -- problem continued

gi4 watchdog

gi10 phone #1

gi11 phone #3

gi22 phone #2

gi24 wap

11:00 - moved phone #3 to gi8 -- problem continued

gi4 watchdog

gi8 phone #3

gi10 phone #1

gi22 phone #2

gi24 wap

11:31 - moved phone #3 to gi14 -- problem went away

gi4 watchdog

gi10 phone #1

gi14 phone #3

gi22 phone #2

gi24 wap

and that's where we've left things right now, been stable for about four hours.

All of the following outputs are from the latest (11:31) config.

sw06#show power inline

Port based power-limit mode

Unit Power Nominal Power Consumed Power Usage Threshold Traps

---- ------- ------------- ------------------ --------------- ---------

1 On 180 Watts 18 Watts (10%) 95 Disable

Port Powered Device State Status Priority Class

-------- -------------------- ---------------- ------------ -------- ---------

gi1 Auto Searching low class0

gi2 Auto Searching low class0

gi3 Auto Searching low class0

gi4 Auto On low class0

gi5 Auto Searching low class0

gi6 Auto Searching low class0

gi7 Auto Searching low class0

gi8 Auto Searching low class0

gi9 Auto Searching low class0

gi10 Auto On low class2

gi11 Auto Searching low class0

gi12 Auto Searching low class0

gi13 Auto Searching low class0

gi14 Auto On low class2

gi15 Auto Searching low class0

gi16 Auto Searching low class0

gi17 Auto Searching low class0

gi18 Auto Searching low class0

gi19 Auto Searching low class0

gi20 Auto Searching low class0

gi21 Auto Searching low class0

gi22 Auto On low class2

gi23 Auto Searching low class0

gi24 - Auto On low class3

-- watchdog:

sw06#show power inline gi4

Port Powered Device State Status Priority Class

-------- -------------------- ---------------- ------------ -------- ---------

gi4 Auto On low class0

Power limit (for port power-limit mode): 15.400W

Port Status: Port is on - valid resistor detected

Overload Counter: 0

Short Counter: 0

Denied Counter: 0

Absent Counter: 4

Invalid Signature Counter: 0

sw06#show interfaces advertise gi4

Port: gi4

Type: 1G-Copper

Link state: Up

Auto negotiation: Enabled

1000f 1000h 100f 100h 10f 10h

----- ----- ---- ---- --- ---

Admin Local link Advertisement yes no yes yes yes yes

Oper Local link Advertisement yes no yes yes yes yes

Oper Remote link Advertisement no no yes yes yes yes

Priority Resolution - - yes - - -

-- phone:

sw06#show power inline gi10

Port Powered Device State Status Priority Class

-------- -------------------- ---------------- ------------ -------- ---------

gi10 Auto On low class2

Power limit (for port power-limit mode): 15.400W

Port Status: Port is on - valid resistor detected

Overload Counter: 0

Short Counter: 0

Denied Counter: 0

Absent Counter: 465

Invalid Signature Counter: 0

sw06#show int adv gi10

Port: gi10

Type: 1G-Copper

Link state: Up

Auto negotiation: Enabled

1000f 1000h 100f 100h 10f 10h

----- ----- ---- ---- --- ---

Admin Local link Advertisement yes no yes yes yes yes

Oper Local link Advertisement yes no yes yes yes yes

Oper Remote link Advertisement no no yes yes yes yes

Priority Resolution - - yes - - -

-- phone:

sw06#show power inline gi14

Port Powered Device State Status Priority Class

-------- -------------------- ---------------- ------------ -------- ---------

gi14 Auto On low class2

Power limit (for port power-limit mode): 15.400W

Port Status: Port is on - valid resistor detected

Overload Counter: 0

Short Counter: 0

Denied Counter: 0

Absent Counter: 0

Invalid Signature Counter: 0

sw06#show int adv gi14

Port: gi14

Type: 1G-Copper

Link state: Up

Auto negotiation: Enabled

1000f 1000h 100f 100h 10f 10h

----- ----- ---- ---- --- ---

Admin Local link Advertisement yes no yes yes yes yes

Oper Local link Advertisement yes no yes yes yes yes

Oper Remote link Advertisement no no yes yes yes yes

Priority Resolution - - yes - - -

sw06#

-- phone:

sw06#show power inline gi22

Port Powered Device State Status Priority Class

-------- -------------------- ---------------- ------------ -------- ---------

gi22 Auto On low class2

Power limit (for port power-limit mode): 15.400W

Port Status: Port is on - valid resistor detected

Overload Counter: 0

Short Counter: 0

Denied Counter: 0

Absent Counter: 591

Invalid Signature Counter: 166

sw06#show int adv gi22

Port: gi22

Type: 1G-Copper

Link state: Up

Auto negotiation: Enabled

1000f 1000h 100f 100h 10f 10h

----- ----- ---- ---- --- ---

Admin Local link Advertisement yes no yes yes yes yes

Oper Local link Advertisement yes no yes yes yes yes

Oper Remote link Advertisement no no yes yes yes yes

Priority Resolution - - yes - - -

-- was watchdog, now empty:

sw06#show power inline gi23

Port Powered Device State Status Priority Class

-------- -------------------- ---------------- ------------ -------- ---------

gi23 Auto Searching low class0

Power limit (for port power-limit mode): 15.400W

Port Status: Port is off - detection is in process

Overload Counter: 0

Short Counter: 0

Denied Counter: 0

Absent Counter: 587

Invalid Signature Counter: 0

-- wap:

sw06#show power inline gi24

Port Powered Device State Status Priority Class

-------- -------------------- ---------------- ------------ -------- ---------

gi24 - Auto On low class3

Power limit (for port power-limit mode): 15.400W

Port Status: Port is on - valid resistor detected

Overload Counter: 0

Short Counter: 0

Denied Counter: 0

Absent Counter: 607

Invalid Signature Counter: 3

sw06#show int adv gi24

Port: gi24

Type: 1G-Copper

Link state: Up

Auto negotiation: Enabled

1000f 1000h 100f 100h 10f 10h

----- ----- ---- ---- --- ---

Admin Local link Advertisement yes no yes yes yes yes

Oper Local link Advertisement yes no yes yes yes yes

Oper Remote link Advertisement no no yes no no no

Priority Resolution - - yes - - -

Other useful bits:

sw06#show spanning-tree active

Spanning tree disabled (BPDU flooding) mode RSTP

Default port cost method: long

sw06#sh ver

SW version 1.1.2.0 ( date 12-Nov-2011 time 23:34:26 )

Boot version 1.1.0.6 ( date 11-May-2011 time 18:31:00 )

HW version V02

Tom Watts · ‎09-12-2012

Hi Rich, reference this picture I just drew- This is a rough interpretation of how a 24 port switch has for guts. Each + block represents a transformer. The corresponding numbers below represent the port bank which corresponds to the transformer. Of course POE MODULE is clearly identified. Each POE module services 12 ports.

I think you have a completely different issue. If the POE is working for 5-30 minutes, it may be a layer 1 issue (bad wires) or a different condition, perhaps equipment being disturbed. If the switch is at fault, it would seem more likely to be a defective than a buggy unit. I have done a lot of extensive research on how the switch ASIC, transformer and POE modules work in cohesion.

Quite frankly, I think you should upgrade to the 1.2.7.76 firmware, factory reset the unit and test any ports you'd like using the same equipment. If the POE keeps dropping off every 5-30 minutes and you are absolutely positive it is not a layer 1 issue or a configuration issue on your products, I'd just replace the switch.

-Tom
Please rate helpful posts

-Tom Please mark answered for helpful posts http://blogs.cisco.com/smallbusiness/

View solution in original post

Tom Watts · ‎09-11-2012

Hello Rich, the case I created for the other gentlemen, John White is 622799757. Currently, this case status is pending more testing from the development team. However, his scenario is using Ruckus 7982 AP.

The important thing to note I think is how the port transform and poe modules interact. The port transformer services 4 ports

1,2,13,14

3,4,15,16

etc

In theory you should be able to connect every other port without issues such as;

1, 3, 5, 7

13, 15, 17, 19

etc

To keep this very simple;

Connect phone #1 to port 1

Connect phone #2 to port 2

Connect phone #3 to port 13

Connect ap #1 to port 14

Do these devices 1 at a time until they are fully powered. Do not connect them all at the same time.

What is the result?

-Tom
Please rate helpful posts

-Tom Please mark answered for helpful posts http://blogs.cisco.com/smallbusiness/

richlafferty · ‎09-11-2012

Did so (with ports 3, 4, 15, 16 -- port 1 is a trunk to another switch so I didn't want to repatch it.)

I haven't seen POE power bounce in the last ten minutes or so. I did notice one slight anomaly when the AP was coming up:

11-Sep-2012 17:05:41 %LINK-I-Up: gi16

11-Sep-2012 17:05:46 %LINK-W-Down: gi16

11-Sep-2012 17:05:48 %LINK-I-Up: gi16

11-Sep-2012 17:05:50 %LINK-W-Down: gi16

11-Sep-2012 17:05:51 %LINK-I-Up: gi16

but I suspect that might just be a weirdness of the AP. I'll let this soak overnight -- I'm about to head out for the day, and sometimes the cycle is as long as a half hour when it does happen. If there's something else you'd like me to test following this test, let me know, and I'll provide that tomorrow morning along with whether or not this was stable overnight.

The watchdog is currently disconnected, so these four are the only POE devices on the switch.

I'm out of the office for most of tomorrow.

richlafferty · ‎09-12-2012

Hey Tom,

The switch was stable overnight with three phones and the WAP in 3, 4, 15, 16. The Watchdog is still disconnected. What's the next step?

Tom Watts · ‎09-12-2012

Hi Rich, reference this picture I just drew- This is a rough interpretation of how a 24 port switch has for guts. Each + block represents a transformer. The corresponding numbers below represent the port bank which corresponds to the transformer. Of course POE MODULE is clearly identified. Each POE module services 12 ports.

I think you have a completely different issue. If the POE is working for 5-30 minutes, it may be a layer 1 issue (bad wires) or a different condition, perhaps equipment being disturbed. If the switch is at fault, it would seem more likely to be a defective than a buggy unit. I have done a lot of extensive research on how the switch ASIC, transformer and POE modules work in cohesion.

Quite frankly, I think you should upgrade to the 1.2.7.76 firmware, factory reset the unit and test any ports you'd like using the same equipment. If the POE keeps dropping off every 5-30 minutes and you are absolutely positive it is not a layer 1 issue or a configuration issue on your products, I'd just replace the switch.

-Tom
Please rate helpful posts

-Tom Please mark answered for helpful posts http://blogs.cisco.com/smallbusiness/

richlafferty · ‎10-29-2012

Hi Thomas,

Upgrading the firmware seems to have done it. We've been stable (knock on wood!) for about a month now. Thanks for your help!

myanuary · ‎04-22-2013

yesterday, i've an issue like that on catalyst 4507...

two ip phone suddenly "off", and when i try to connect it on another port, it's "on" again..

the poe status is auto...

so i try to connect it to my pc, and the link is up, then i connect it again to ip phone again and magically it's "on" again...

i don't know why suddenly the poe "die" and "live" again, i am still curious if this happen again..

what a strange problem...