Re: Cisco 3850 16.12.3a POE issues - Page 2

AdamF1 · ‎09-01-2020

Good morning,

Is anyone else running 16.12.3a IOS on 3850 switches?

Here is an issue we are facing but I can not find any documentation of a bug in this code and its still recommended as the code to go to...

Issue: POE Stops functioning on random ports but works on others. POE will not work for Avaya phones, cameras, cisco phones, or Cisco AP's ( 3602,3702,3802).

Work around: Reboot switch, downgrade, or find a port that will provide POE.

We began upgrading and testing on several stacks for a month or 2 with no issues prior to deploying to approximately 30 stacks of 3850's. After we did a mass deployment we began to see POE issues on switches that seem to be triggered when removing or adding a POE device. Once the condition has been triggered it will not go away until rebooted or downgraded. Logs will state " Controller port error, interface x/x/x, power given, but machine power good wait timer timed out.

I have found similar issues or bugs in older codes, have we regressed?

atrevido43 · ‎03-01-2021

We have this issue too. Upgraded to 16.12.4 end of December and 4 weeks later a bunch of devices won't power up, but some will. Log error is same

.Mar 1 18:11:58.371: %ILPOWER-5-DETECT: Interface Gi1/0/14: Power Device detected: IEEE PD
.Mar 1 18:13:28.386: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/14: PD removed

.Mar 1 18:13:28.387: %ILPOWER-3-CONTROLLER_PORT_ERR: Controller port error, Interface Gi1/0/14: Power given, but State Machine Power Good wait timer timed out

AdamF1 · ‎03-01-2021

Issue seems to be resolved in 16.12.5 per their bug documentation and from my own

testing. I’ve been running the code on several different stacks for several weeks without issue yet.

Leo Laohoo · ‎03-01-2021

I am also testing 16.12.5 and I am going to wait for about 12 weeks of no-issue uptime before making a decision.

Art Astafiev · ‎03-05-2021

We also start upgrade to 16.12.5 on large scale. It will take us 2-3 weeks to complete upgrade. Hope this will be the end of this issue, but I will report back in few weeks after upgrade will be completed.

Leo Laohoo · ‎03-05-2021

@Art Astafiev wrote:

We also start upgrade to 16.12.5 on large scale. It will take us 2-3 weeks to complete upgrade.

Slow down. Test 16.12.5 PROPERLY & THOROUGHLY -- If Cisco no longer tests their codes, then who is/are ~~the guinea pigs~~ testing the codes?

Wisco6977 · ‎04-22-2021

The bug is still in 16.12.5. After about 4 weeks of running the code, I had to downgrade around 20 3850 stacks back to 16.9.6 to remove the POE bug. I see 16.12.5b was released but I do not see a fix for the issue there either.

AdamF1 · ‎04-28-2021

Did you open a Cisco Case? Bug CSCvv50628 states it was resolved in 16.12.5.

Anyone else have POE issues with 16.12.5?

I'd like move to the recommended version 16.12.5b but am hesitant on upgrading if people are still experiencing it.

Wisco6977 · ‎04-28-2021

We opened a case with 16.12.3, 16.12.4 and 16.12.5 but we have not heard anything back. We have had too much downtime with the whole 16.12.x code train. We are going back to 16.9.6 on all 3850s as that is the last stable code for the 3850s in my opinion.

AdamF1 · ‎04-28-2021

Are you able to share the TAC case # for the 16.12.5 poe issue?

Leo Laohoo · ‎04-28-2021

@Wisco6977 wrote:

We opened a case with 16.12.3, 16.12.4 and 16.12.5 but we have not heard anything back.

Which TAC "region" did the case get assigned to?

Wisco6977 · ‎04-29-2021

Looks like "Global CX Centers - Enterprise Routing and Switching" and the Engineer has an San Jose area code.

Not saying 16.12.5b wont work for people, but we have close to 600 3850 stacks in a healthcare environment. That is not counting the numerous 3750X and 9300 stacks. Getting downtime to upgrade or downgrade can be a pain and I had close to 30 stacks that had the POE issue in my region alone. Most all of them had the issue around the same time, 4 weeks in. If you don't have phones or WAPs on the switch, or any device requiring POE, that code might be okay, but anything requiring POE, you will have issues.

We have had numerous issues with 9500s and now 9300s on the 16.12.x train as well. While typing this up, and sitting in on a teams meeting, one of our Sr. engineers is moving some 9300s from 16.12.5 to 17.3.3 as the we are not able to see device profiling in ISE. Worked on 16.9.6, doesn't work on the 16.12.x platform, works on 17.3.3.

Leo Laohoo · ‎04-29-2021

@Wisco6977 wrote:

Looks like "Global CX Centers - Enterprise Routing and Switching" and the Engineer has an San Jose area code.

I am not going to say anything else other than this: Contact TAC and get them to re-assign the case to LATAM.

Do not ask me for any more information.

We are in the same boat: I have a medical facility campus and I have 3850/9300 running on 16.12.4 and they are, to sum it up politely, atrocious. I am hitting several bugs with Dot1x + flapping port + 16.12.4 + 3850/9300 where the secondary will crash every 3 to 4 weeks. All I need is one, just one, flapping port and then *toink!*, secondary crashes.

I have created several TAC Cases in the past and no one could diagnose it. I currently two TAC Cases and they are now "converging". A TAC engineer was able to confirm the issue was caused by Dot1x.

Art Astafiev · ‎04-22-2021

This is bad news. 16.12.5b has 3 major fixes in compare to 16.12.5, but they are not related to POE bug fix.

We have few stacks on 16.12.5 now and didn't see bug, but we anyway is moving to 16.12.5b due to issues which 16.12.5b fixes.

Unfortunately any upgrade will reset clock, so we will not be able to say if 16.12.5b has issue or not for another few weeks.

Leo Laohoo · ‎04-22-2021

The release of the re-build 16.12.5b does not bode well for us. This means we have to "reset the clock" and test this newly released re-build (and discovery what other nasties this re-build brings). Two of the three "fixed" bugs in 16.12.5b concerns us to some degree.

Anyone who is reading this post and has 3650/3850 need to be of concern: End of SW Maintenance Releases Date is 30 October 2021.

Anyone who have the luxury of ~~talking~~ ~~howling~~ screaming at your Cisco AM/SE, now is the time.

AdamF1 · ‎04-23-2021

Oh I have been letting our SA know... At this point I don't think 16.12.x will be a functioning IOS for 3850s and they should honestly just pull it and extend the life of 16.9.x. 16.12.x has been plagued with bugs in both 3K and 9K lines.

Cisco sure doesn't disappoint when it comes to software.