3650 PoE issue - Tstart error detected

[andrewswanson]

Hello

I am seeing an issue with PoE on a switch stack (WS-C3650-48FD-S running 03.06.05E).

Issue affects both non Cisco and Cisco devices. Not all switches in the stack are affected by this - each switch has a single PWR-C2-1025WAC PSU and the power consumption is low for the affected switch (652.6 W remaining with 122.4 W used)

The affected port(s) has the following status:

Gi7/0/24 auto faulty 0.0 n/a n/a 30.0

Cabling has been checked and issue can happen with long/short patch cables (as with bug CSCsw18530 https://bst.cloudapps.cisco.com/bugsearch/bug/CSCsw18530/?referring_site=bugquickviewclick)

debug output when issue occurs:

%ILPOWER-7-DETECT: Interface Gi7/0/24: Power Device detected: Cisco PD
%ILPOWER-5-IEEE_DISCONNECT: Interface Gi7/0/24: PD removed
%ILPOWER-3-CONTROLLER_PORT_ERR: Controller port error, Interface Gi7/0/24: Power Controller reports power Tstart error detected

A restart will "solve" the issue but what can be done to avoid this in future. Disabling and re-enabling inline power and the interface doesn't help either.

Thanks
Andy

[Leo Laohoo]

Andrew, 

I think this is due to CSCux65429 (and not CSCsw18530).  The only recommendation I can give, believe me or not, is to upgrade the IOS.  

[andrewswanson]

Thanks for the response Leo

The bug search tool states that CSCux65429 is resolved with 3.6(5)E (which the switches are currently using) - I had tried the suggested workaround with no success. I was planning to upgrade the stacks to the now recommended 3.6.6E  - I'll report back if this resolves the issue.

Thanks
Andy

[Jordan1212]

Can you confirm if a reload or upgrade fixes the issue.

Just had this happen today with WS-C3650-48PS running 03.06.04.E.  Bouncing the port does not help.

020759: *Jun 20 13:59:33.977 EST: %ILPOWER-3-CONTROLLER_PORT_ERR: Controller port error, Interface Gi1/0/42: Power Controller reports power Tstart error detected
020761: *Jun 20 13:59:40.004 EST: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/42: PD removed
020762: *Jun 20 13:59:40.004 EST: %ILPOWER-3-CONTROLLER_PORT_ERR: Controller port error, Interface Gi1/0/42: Power Controller reports power Tstart error detected
020764: *Jun 20 13:59:46.077 EST: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/42: PD removed

Most of my other switches are running 3.6.6E and this one needs to get upgraded anyhow, but for the few that are still running 3.6.4E, just want to know if there is any stable IOS that fixes this.

Currently 3.6.6E is Cisco suggested release.

[andrewswanson]

Hi

Haven't done the upgrade as yet - planning to move to 3.6.6E soon. Reloading the affected switch(es) resolves the issue for me.

hth

Andy

[Jordan1212]

Thanks for the reply Andy.  I will plan on upgrading to 3.6.6E as well since I will need to reload either way.

Regards,

Jordan

[andrewswanson]

Upgraded a stack of 7 switches from  3.6.5E to the recommended 3.6.6E and am seeing this issue again. I'll replace the affected switch with a known good one and test again

Andy

[Jordan1212]

Had a co-worker say a similar issue occurred with a brand new different 3650-48PS running 3.6.6E. 

I didn't see the logs and cannot confirm 100% that the logs were the same, but the issue was PoE not working/dead phones until a reboot of the switch.

[andrewswanson]

I replaced the switch in the stack where I was seeing the PoE "faulty" errors.

Once I had repatched all devices, I checked the inline power and found that the new switch had 1 interface already in a "faulty" operational state.

The device connected to the faulty interface is a VDI zero client that doesn't even support PoE!

I done some testing on the switch removed from the stack and can't replicate the PoE issue. I don't have TAC support for these switches and they are outwith their warranty - suppose its a case of finding an IOS release where this issue is resolved.

Andy

[andrewswanson]

Think this could be related to bug CSCux65429 - at least Cisco are aware of the issue

[brannonsweet@cox.net]

I was getting log entries for:

%ILPOWER-3-CONTROLLER_PORT_ERR: Controller port error, Interface Gi2/0/34: Power Controller reports power Tstart error detected

%ILPOWER-3-CONTROLLER_PORT_ERR: Controller port error, Interface Gi2/0/6: Power Controller reports power Tstart error detected

A "show power inline" indicates I have to bad POE ports.  I'm RMA'ing the switch.

#show power inline
Module Available Used Remaining
(Watts) (Watts) (Watts)
------ --------- -------- ---------
1 462.0   218.4   243.6
2 462.0    0.0 4    62.0
Interface Admin Oper Power Device Class Max
(Watts)
Gi2/0/6 auto faulty 0.0 n/a n/a 30.0
Gi2/0/34 auto faulty 0.0 n/a n/a 30.0

[swfowler58]

Are there any updates on this issue? I too have a WS-C3650-48PQ running 03.07.03E and experiencing the same Poe interface faulty. Log's show the following:

1805143: May 17 15:30:12.021: %ILPOWER-7-DETECT: Interface Gi2/0/17: Power Device detected: IEEE PD
1805145: May 17 15:30:12.588: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi2/0/17: PD removed
1805146: May 17 15:30:12.589: %ILPOWER-3-CONTROLLER_PORT_ERR: Controller port error, Interface Gi2/0/17: Power Controller reports power Tstart error detected

[Leo Laohoo]

Are there any updates on this issue?

Try upgrading the firmware.

[andrewswanson]

Hi - I'm seeing this issue on only 4 switches in a deployment of 90 (over the last 6 months or so) and can't replicate the problem or predict where/when it will happen. If I find a release where it doesn't happen I'll post back here.

Cheers

Andy

[sunilferrao]

hi 

is anyone found soulution for this issue

Cheers

Snl