3750E and RPS 2300 PoE Problem

[andrew-hunt]

I have a problem with a 4-switch stack of 3750E-48PD-SF catalysts that are connected to a single RPS 2300.

Each switch has 1 x 1150W power supply.  The RPS 2300 has 2 x 1150W power supplies.

The problem is that whenever we lose mains power to a single 3750E switch we lose PoE on that switch, even though the other three switches in the stack retain mains power and the RPS is backing up the switch with the failed PSU.  The switch remains online, we just lose PoE.

We're running the c3750e-universalk9-mz.150-1.SE version of IOS

#sho env rps
SW  Status          RPS Name          RPS Serial#  RPS Port#
--  -------------   ----------------  -----------  ---------
1   Active          RPS_2300          FDO1529YJ64  4
2   Active          RPS_2300          FDO1529YJ64  3
3   Active          RPS_2300          FDO1529YJ64  2
4   Active          RPS_2300          FDO1529YJ64  1

RPS Name: RPS_2300
  State: Active
  PID: PWR-RPS2300
  Serial#: FDO1529YJ64
  Fan: Good
  Temperature: Green

  RPS Power Supply A: Present
    PID             : C3K-PWR-1150WAC
    Serial#         : ART1524W04X
    System Power    : Good
    PoE Power       : Good
    Watts           : 300/800 (System/PoE)

  RPS Power Supply B: Present
    PID             : C3K-PWR-1150WAC
    Serial#         : ART1524W07U
    System Power    : Good
    PoE Power       : Good
    Watts           : 300/800 (System/PoE)

DCOut  State   Connected  Priority  BackingUp  WillBackup  Portname        SW#
-----  ------- ---------  --------  ---------  ----------  --------------- ---
    1  Active  Yes               1  No         Yes         AD_RPS_2300       4
    2  Active  Yes               2  No         Yes         AD_RPS_2300       3
    3  Active  Yes               3  No         Yes         AD_RPS_2300       2
    4  Active  Yes               4  No         Yes         AD_RPS_2300       1
    5  Active  No                5  No         Yes         <>                -
    6  Active  No                6  No         Yes         <>                -

We only have a single powered device on the switch at the moment (as it's not yet in production).  Each switch has been set a separate priority with the intention that in the event of a total power failure the two most critical switches will remain online.

When we simulate a power failure the debug shows:

Oct 12 19:03:58.693: %PLATFORM_ENV-6-FRU_PS_OIR: FRU Power Supply 1 powered off(CatAD14-4)

Oct 12 19:03:59.406: %ILPOWER-7-DETECT: Interface Gi4/0/2: Power Device detected: IEEE PD (CatAD14-4)

Oct 12 19:03:59.739: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/2, changed state to down

Oct 12 19:03:59.406: %ILPOWER-5-ILPOWER_POWER_DENY: Interface Gi4/0/2: inline power denied. Reason: Insufficient total available power (CatAD14-4)

Has anyone else come across this issue or have any inspired ideas as to what may be the reason?

Thanks!

Andrew.

[nkarpysh]

Hi Andrew,

Can you please get these outputs when device is running on RPS:

sh power inline

show env all

show log (for new error messages)

Nik

[andrew-hunt]

Hello Nik,

The info you need is in the attached text file. 

Just to reiterate one point and that is that this is a new install, so we only have a single PoE device plugged in to the stack at the moment.  The behaviour is the same whether we physically unplug the power lead from the catalyst or use the power supply 4 off command.

Thanks for taking a look at this - much appreciated.

Andrew.

[Ven Taylor]

I've seen this before with 3750E's.

I don't know why, but when the power is reduced to these switches, they immediately stop providing PoE.

I assumed it was a survival mechanism, but never looked into it.

The immediate fix is to reload the switch and make sure it's using the internal power supply and NOT the RPS.

Personally, I would avoid the RPS and go with a strong UPS.  RPS's gave us nothing but trouble.

If you're looking for redundant power supplies, go with the 3750X.  It actually has them.

Ven

[nkarpysh]

This is not correct behaviour indeed. However not clear if that is switch or RPS problem. I would try to use same RPS with other switch and see if that is grantinf power. Also another test is to use different RPS with first switch and see if all symptoms reproducible.

By this we can first of all isolate if problem sepcific to

- Single RPS

- or Single Switch

- All RPSs

- All switches

- All RPS and switches

That can confirm if it is HW problem on one of the devices or a bug.

Nik

[andrew-hunt]

Hello Nik,

We have tested using all four 3750E switches in the stack and the behaviour is the same  - the switch will remain online but we lose PoE

We have another RPS in our Dubai office so we will ship that to Abu Dhabi on Thursday and test with that.

I will follow up on Thursday with the results of the test.

Just FYI we also changed the RPS cable order so Port 1 on the RPS goes to Switch 1 in the stack, Port 2 to Switch 2 and so forth just in case there was some weird logic regarding that.  We also tried with only two switches connected to RPS power - but the results were identical.

Thanks for your continued help.  I'll let you know what happens on Thursday.

Best wishes,

Andrew.

[andrew-hunt]

Hello Nik,

We have done more testing.  We have an identical set-up in our Dubai office: 4 x 3750E switches and an RPS 2300.  It works as designed.  The switches in Dubai are running 12.2(50)SE2 of the 3750 universalk9 image

Downgrading Abu Dhabi to the same version of IOS results in the RPS working there too.  I have attached a couple of text files showing the results.

The question now is in which version of IOS does this become a bug, and how do we get it resolved?  I am happy to run with 12.2(50)SE2 for now as we've seen no issues in Dubai but there are a lot of issues with that version and I'd prefer to upgrade to something later - but don't really want to test each release until I find one that breaks...

Any thoughts?

Thanks again,

Andrew.

[nkarpysh · Accepted solution]

Hi Andrew,

SO far I found only one DDTS for the same:

http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?caller=pluginredirector&method=fetchBugDetails&bugId=CSCth76268

In internal notes it is saying that this bug is a trace of code changed which broke original behaviour. However I see that fixed only in 12.2(55)SE, 15.0 is not mentioned there. There is a posisbility that no one consider that defect in new release, or it still can be another issue. If possible you can try 12.2(55)SE to see if you still have an issue to undertsand if this bug fixing your problem and the same fix should be ported to 15.0.

Nik,

[andrew-hunt]

Hello Nik,

Looks like you're right.  I have just tried with 12.2(55)SE4 and the RPS works as it should.  Something's definitely broken in 12.2(58)SE2 and also 15.0

I really appreciate your help with this!  Thanks again!

Best wishes,

Andrew.

[fernandess]

Just upgraded to 15.0(2)SE2 and got hit with this bug. No mention in the release notes either. Hard to believe that a 'fixed' bug is still around after this long & completely undocumented.

[brandon.case]

I know this is an old thread but I need to add to it that this bug seems to be resolved in 15.0(2)SE4.

I ran across this problem twice in the past week when we've had switch power supply failures and it's resulted in no wireless service in two entire buildings until we could get the respective power supplies replaced (luckily same-day from on-hand inventory). After getting fed up with it I built a lab in my office running 15.0(2)SE4 and sure enough, I was able to force the switch over to RPS and back again without the APs on the test switch losing PoE.