Phones rebooting themselves

steve pearson · ‎03-05-2015

Hi

I have recently deployed 200 Cisco 8841 IP Phones to a new build, using Cisco 3850X switches with 1400W Power and making use of PowerStack.

I am finding phones are randomly rebooting themselves.

I have run some debugging and seen the following:

Mar 5 18:35:47.686: ILP:: Gi1/0/34: State=NGWC_ILP_DETECTING_S-2, Event=NGWC_ILP_IEEE_DET_SHORT_EV-29
Mar 5 18:35:47.686: detected short from IEEE. still waiting to hear from phy Gi1/0/34

Mar 5 18:35:48.287: ILP:: Gi2/0/10: State=NGWC_ILP_DETECTING_S-2, Event=NGWC_ILP_WAIT_IEEE_SHORT_TIMER_EV-41
Mar 5 18:35:48.287: ILP_DETECTING_S: Gi2/0/10 phy still didn't get back to us.
It looks like a real short and not a csco pd. don't power up the pd

Mar 5 18:35:48.410: ILP:: Gi1/0/35: State=NGWC_ILP_DETECTING_S-2, Event=NGWC_ILP_WAIT_IEEE_SHORT_TIMER_EV-41
Mar 5 18:35:48.410: ILP_DETECTING_S: Gi1/0/35 phy still didn't get back to us.
It looks like a real short and not a csco pd. don't power up the pd

Mar 5 18:35:48.411: ILP:: Gi1/0/36: State=NGWC_ILP_DETECTING_S-2, Event=NGWC_ILP_WAIT_IEEE_SHORT_TIMER_EV-41
Mar 5 18:35:48.411: ILP_DETECTING_S: Gi1/0/36 phy still didn't get back to us.
It looks like a real short and not a csco pd. don't power up the pd

Mar 5 18:35:48.547: ILP:: Gi2/0/19: State=NGWC_ILP_DETECTING_S-2, Event=NGWC_ILP_WAIT_IEEE_SHORT_TIMER_EV-41
Mar 5 18:35:48.547: ILP_DETECTING_S: Gi2/0/19 phy still didn't get back to us.
It looks like a real short and not a csco pd. don't power up the pd

Mar 5 18:35:48.548: ILP:: Gi2/0/21: State=NGWC_ILP_DETECTING_S-2, Event=NGWC_ILP_WAIT_IEEE_SHORT_TIMER_EV-41
Mar 5 18:35:48.548: ILP_DETECTING_S: Gi2/0/21 phy still didn't get back to us.
It looks like a real short and not a csco pd. don't power up the pd

Mar 5 18:35:48.549: ILP:: Gi2/0/22: State=NGWC_ILP_DETECTING_S-2, Event=NGWC_ILP_WAIT_IEEE_SHORT_TIMER_EV-41
Mar 5 18:35:48.549: ILP_DETECTING_S: Gi2/0/22 phy still didn't get back to us.
It looks like a real short and not a csco pd. don't power up the pd

A show logging, shows the following (this I expect is not PCs being unpatched and repatched as the building is fully established):

Mar 5 15:26:01.658: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/29, changed state to up
Mar 5 15:26:04.638: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/4, changed state to down
Mar 5 15:26:05.649: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/4, changed state to down
Mar 5 15:26:08.677: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/4, changed state to up
Mar 5 15:26:09.679: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/4, changed state to up
Mar 5 15:26:09.778: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/33, changed state to down
Mar 5 15:26:10.790: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/33, changed state to down
Mar 5 15:26:13.828: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/33, changed state to up
Mar 5 15:26:14.830: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/33, changed state to up
Mar 5 15:26:16.477: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/31, changed state to down
Mar 5 15:26:17.476: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/31, changed state to down
Mar 5 15:26:20.517: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/31, changed state to up
Mar 5 15:26:21.516: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/31, changed state to up
Mar 5 15:26:32.096: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/33, changed state to down
Mar 5 15:26:33.096: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/33, changed state to down
Mar 5 15:26:38.652: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/31, changed state to down
Mar 5 15:26:39.640: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/33, changed state to up
Mar 5 15:26:39.647: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/31, changed state to down
Mar 5 15:26:40.640: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/33, changed state to up
Mar 5 15:26:46.658: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/31, changed state to up
Mar 5 15:26:47.656: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/31, changed state to up
Mar 5 15:27:07.756: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/33, changed state to down
Mar 5 15:27:08.760: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/33, changed state to down
Mar 5 15:27:11.736: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/33, changed state to up
Mar 5 15:27:12.741: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/33, changed state to up
Mar 5 15:27:14.137: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/31, changed state to down
...... and so it goes on and on, lots of ports going up and down.

On all switches in the stack, the remaining power is upwards of 600W.

All cabling has been tested by a trusted contractor.

The switches are all layer 3 with resilient links and no known issues with any link. It is also happening on multiple stacks, leading me to suspect this is a manufactur fault?

Thanks

Steve

rickey.keith · ‎06-14-2017

I am having this exact same issue on my 3850 stacks. Did you ever find a solution to this?

csco10401830 · ‎01-25-2019

weird, but two ways I've fixed this in the past is by turning off POE at the port level with the command

shut

power inline never

no shut

SWITCH03(config-if)#do sh int status | i 5/0/12
Gi5/0/12 connected 1 a-full a-1000 10/100/1000BaseTX

something strange I tried in a different scenario was to change the port speed to 100MB and the phones stayed up after changing the port speed... it absolutely makes no sense.. but it worked.