%ILPOWER-3-CONTROLLER_PORT_ERR: Controller port error on Everest-16.6.6 (Possible Bug?)

[ekim4ds89]

We have a bunch of Cisco WS-C3650-48PD across the network as our access layer switches that are running Everest-16.6.6.

Recently in the past couple of weeks we have had an issue going around the network where these 3650's are reporting Controller port errors int he logs. Below is what we are seeing in the logs. It's very random and will flap the port over and over so the workstations connected will keep loosing network access. None of these ports so far have had a PoE device plugged into it that would request power.

Dec 11 08:47:29.320 EST: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/39: PD removed
Dec 11 08:47:49.325 EST: %ILPOWER-7-DETECT: Interface Gi1/0/39: Power Device detected: IEEE PD
Dec 11 08:47:59.312 EST: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/39: PD removed
Dec 11 08:47:59.313 EST: %ILPOWER-3-CONTROLLER_PORT_ERR: Controller port error, Interface Gi1/0/39: Power given, but Power Controller does not report Power Good

If i go into the interface and do a power inline never to shut off power inline on that interface the issue stops. This is a workaround for now.

I found some similar Bug Reports but none that show our ios version as a known affected release

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvf50867

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvf75518

I tested on one of the problem switches upgrading to Everest-16.6.7 which is the newest in the Everest-16.6 train but that did not resolve the issue.

Has anyone else seen this issue with their Catalyst 3650's running Everest-16.6.6?

[Leo Laohoo]

Hard reboot (as in, pull the power) the stack and see if it makes any improvements.

[ekim4ds89]

I will try that and report back my findings. Thanks!

[ekim4ds89]

Did a hard reboot on the switch yesterday and the issue came back today. Same interface:

"Dec 13 11:09:16.477 EST: %ILPOWER-3-CONTROLLER_PORT_ERR: Controller port error, Interface Gi1/0/39: Power given, but
Power Controller does not report Power Good"

What I have noticed is that on all of these WS-C3650-48PD's having the issue it's always the same interface but nothing in common on what is plugged into them. Most are workstations (some windows, some Mac) and one is a copier but none of the ports that do this have anything requiring PoE. It always seems to be the same interface on the switch as well which is weird. None of the other interfaces, always just one interface on each switch having the issue.

Any information from the switches themselves that I can grab to help?

[ekim4ds89]

Any other ideas?

[Leo Laohoo]

Contact Cisco TAC and organize for an RMA.
This could be CSCvj76259.

[ekim4ds89]

Thank you for your reply. I have a case going with TAC now.