CSCvf33653 - Controller port error, Power given, but State Machine Power Good wait timer timed out

[757VA]

this bug is not resolved in 16.12.4 for the 3650 48p sw. i just encountered it today. 

the work around was to move the PD to another switch in the stack. which is by no means a fix

further testing revealed that the poe power is not available on the remaining free ports on my second switch in the stack.

i would like more feedback on how to resolve this issue.

[Leo Laohoo]

Tell us what you are seeing:  Are you actually seeing the above error message(s) or something else? 

Have you also tried cold rebooting (completely remove the power cables)?

[JOeKeenan2659]

Seeing the same issue on one module in a 3850 stack on 16.12.4 , the other 4 modules in the stack aren't affected. The option I'm looking at is moving the PDs to another switch in the stack but agree that's not a proper solution 

TDR tests on the drops came back clean, tried various config changes at the individual port level:

Bouncing ports

power inline never / power inline auto,

Setting Max power draw to 15.4 W

Setting POE to 4 wires

As it is a state machine time out message I think that would likely suggest a bug, but I'm not seeing a fix anywhere

Any ideas?

[Art Astafiev]

I am seeing same problem on 16.12.3a code which also listed as "fixed" under bug description. Any more suggestions?

[Leo Laohoo]

---

@JOeKeenan2659 wrote:

Seeing the same issue on one module in a 3850 stack on 16.12.4 , the other 4 modules in the stack aren't affected. 

---

Wait ... I think I know what this is ...

So you have some PDs, right?  And they do not "talk" CDP or LLDP, am I correct? 

To check if they talk CDP or LLDP, use the command "sh power inline <PORT> detail". 

[Art Astafiev]

CDP is disabled on this port. LLDP is enabled on switch and on this interface. Device is Axis IP camera. 

Switch1# show power inline GigabitEthernet 2/0/9 detail
Interface: Gi2/0/9
Inline Power Mode: auto
Operational status: on
Device Detected: yes
Device Type: Ieee PD
IEEE Class: 2
Discovery mechanism used/configured: Ieee and Cisco
Police: off

Power Allocated
Admin Value: 30.0
Power drawn from the source: 7.0
Power available to the device: 7.0

Actual consumption
Measured at the port: 0.0
Maximum Power drawn by the device since powered on: 0.0

Absent Counter: 0
Over Current Counter: 1
Short Current Counter: 0
Invalid Signature Counter: 0
Power Denied Counter: 0

Power Negotiation Used: None
LLDP Power Negotiation --Sent to PD-- --Rcvd from PD--
Power Type: - -
Power Source: - -
Power Priority: - -
Requested Power(W): - -
Allocated Power(W): - -

Four-Pair PoE Supported: No
Spare Pair Power Enabled: No
Four-Pair PD Architecture: N/A
------------------

Power drawn is flipping from 7 to zero every few minutes. 

[Art Astafiev]

Basically port is detecting POE device every 5 minutes. This is the same device which is constantly connected to port and powered via POE, so device cannot just be in slow reboot loop because in the middle for 4 minutes port is in disconnect state and device has no power. So, this is some sort of timer on interface side which is probing device and provide it some power every 5 minutes. 

I know if I reboot switch device will be powered, but I am waiting to see if anyone can suggest something other than reboot. 

[Leo Laohoo]

---

@Art Astafiev wrote:

CDP is disabled on this port. LLDP is enabled on switch and on this interface. Device is Axis IP camera. 

---

We have this same issue now and ours is probably very bad because our stacks are running Dot1X and the flapping ports are causing the stack members to crash every 36 hours.  

Based on the output to the command "sh power inline <PORT> detail", the cameras DO NOT TALK LLDP.  

This is a bug with the entire 16.12.X train (until 16.12.4).  It is not known if 16.12.5, slated for 1Q2021 release, will fix this issue. 

TAC is talking about "downgrading" to 16.3.11 and "see if this helps" (aka TAC is not even sure if this will fix the issue or not). 

Our workaround, so far is to apply SMU to fix CSCvv28324 (cat3k_caa-universalk9.16.12.04.CSCvv28324.SPA.smu.bin).

WARNING:  This SMU will reboot the stack.  

This bug gets triggered when a PD does not talk CDP nor LLDP gets connected to the port.  (Normally, if the same PD gets plugged into a "classic" IOS switch, it will just work -- But IOS-XE is "special".)   

The bug emerges after 3 weeks uptime.  If the stack members do not crash, the constant flapping of PoE will cause the ports to "throw hands in the air and give up" after >3 weeks -- This means the port will stop to "work", PoE or not.  If a non-PoE is connected to the port, NO data traffic will pass.  The port will report up/up but the traffic will be reported as 0 input and 0 output traffic.  

[WayneTW]

Any one upgrade any version and get fixed?

[Leo Laohoo]

---

@WayneTW wrote:

Any one upgrade any version and get fixed?

---

What issue(s) are you seeing/getting, @WayneTW? 
TAC is saying 16.12.5, scheduled for release on 21 January 2021, "will" contain thie fix for this bug and CSCvv28324.  I am, however, skeptical about this claim because this bug and CSCvv28324 has been present since 16.12.1 and previous releases have failed to fix this bug.  

[WayneTW]

Hi Leo,

I got the same the issue, here's the message, several ports cannot boot up successfully today with IP Phones.

Here's is one of the port message, this port was working with IP Phone and after I re-plugged the cable, no more function.

*Jan  8 18:18:28.659: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi2/0/28: PD removed
*Jan  8 18:18:28.662: %ILPOWER-3-CONTROLLER_PORT_ERR: Controller port error, Interface Gi2/0/28: Power given, but State Machine Power Good wait timer timed out
*Jan  8 18:18:38.672: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi2/0/28: PD removed
*Jan  8 18:18:49.055: %ILPOWER-5-DETECT: Interface Gi2/0/28: Power Device detected: Cisco PD
*Jan  8 18:19:27.591: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/28, changed state to up
*Jan  8 18:19:28.591: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/28, changed state to up
*Jan  8 18:19:39.943: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/28, changed state to down
*Jan  8 18:19:41.947: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/28, changed state to up

ServerRoom-1#show power inline gigabitEthernet 2/0/28 detail
 Interface: Gi2/0/28
 Inline Power Mode: auto
 Operational status: faulty
 Device Detected: no
 Device Type: n/a
 IEEE Class: n/a
 Discovery mechanism used/configured: Ieee and Cisco
 Police: off

 Power Allocated
 Admin Value: 30.0
 Power drawn from the source: 0.0
 Power available to the device: 0.0

 Actual consumption
 Measured at the port: 0.0
 Maximum Power drawn by the device since powered on: 0.0

 Absent Counter: 0
 Over Current Counter: 0
 Short Current Counter: 0
 Invalid Signature Counter: 0
 Power Denied Counter: 0

 Power Negotiation Used: None
 LLDP Power Negotiation --Sent to PD--      --Rcvd from PD--
   Power Type:          -                    -
   Power Source:        -                    -
   Power Priority:      -                    -
   Requested Power(W):  -                    -
   Allocated Power(W):  -                    -

Four-Pair PoE Supported: No
Spare Pair Power Enabled: No
Four-Pair PD Architecture: N/A

Wayne

[Leo Laohoo]

Is the stack running 16.12.X?

[WayneTW]

Hi Leo,

Yes! the stacks version are Version 16.12.03a

[Leo Laohoo]

---

@WayneTW wrote:

Yes! the stacks version are Version 16.12.03a

---

Wait for 16.12.5 to be released shortly (21 January 2021) and upgrade the firmware.

Hopefully, if the bug is indeed fixed in 16.12.5, then you will NOT see any problems after >8 weeks of uptime.  

[Art Astafiev]

We have 43 stacks of Catalyst 3850's on code 16.12.5b and another 51 standalone 3850 on this code. We had been running on this code for around 10 months now. I would say that it is relatively stable and has maximum amount of security fixes. So, I would recommend everyone to move from 16.9.x code due to security vulnerabilities. I don't remember that PoE issue was reported since we did upgrade.