Cisco 3650 - Port detection issue with POE

[Davemac10]

Good day.  I have a Cisco 3650 48 port switch that is acting strange.  POE seemed to jump around at first, but now after pulling the switch from the closet and testing it with a single cisco phone I am realizing an issue with the switch detecting the correct port and I was curious if anyone else has seen this before.  I plugged a cisco phone into port g1/0/1 and I am watching the logs as it comes up.  It detects the incorrect port at first and then changes the correct port state to up...  Any port I plug into seems to do this allows power to the incorrect interface

After plugging phone into port g1/0/1 look below:

*Jun 6 16:06:16.401: %ILPOWER-7-DETECT: Interface Gi1/0/3: Power Device detected: IEEE PD
*Jun 6 16:06:17.401: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/3: Power granted
*Jun 6 16:06:25.992: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to up
*Jun 6 16:06:26.992: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to up

HOWEVER.., If you look at the int status & the power inline stats you can see that power appears to be applied to g1/0/3 with nothing actually plugged into that interface.  During the interface status we are showing the correct interface connected.

Interface Admin Oper Power Device Class Max
(Watts)
--------- ------ ---------- ------- ------------------- ----- ----
Gi1/0/1 auto off 0.0 n/a n/a 30.0
Gi1/0/2 auto off 0.0 n/a n/a 30.0
Gi1/0/3 auto on 15.4 Ieee PD 3 30.0
Gi1/0/4 auto off 0.0 n/a n/a 30.0
Gi1/0/5 auto off 0.0 n/a n/a 30.0
Gi1/0/6 auto off 0.0 n/a n/a 30.0
Gi1/0/7 auto off 0.0 n/a n/a 30.0
Gi1/0/8 auto off 0.0 n/a n/a 30.0
Gi1/0/9 auto off 0.0 n/a n/a 30.0
Gi1/0/10 auto off 0.0 n/a n/a 30.0
Gi1/0/11 auto off 0.0 n/a n/a 30.0
Gi1/0/12 auto off 0.0 n/a n/a 30.0
Gi1/0/13 auto off 0.0 n/a n/a 30.0
Gi1/0/14 auto off 0.0 n/a n/a 30.0
Gi1/0/15 auto off 0.0 n/a n/a 30.0

Switch#sh int status

Port Name Status Vlan Duplex Speed Type
Gi1/0/1 connected 1 a-full a-1000 10/100/1000BaseTX
Gi1/0/2 notconnect 1 auto auto 10/100/1000BaseTX
Gi1/0/3 notconnect 1 auto auto 10/100/1000BaseTX
Gi1/0/4 notconnect 1 auto auto 10/100/1000BaseTX
Gi1/0/5 notconnect 1 auto auto 10/100/1000BaseTX
Gi1/0/6 notconnect 1 auto auto 10/100/1000BaseTX
Gi1/0/7 notconnect 1 auto auto 10/100/1000BaseTX
Gi1/0/8 notconnect 1 auto auto 10/100/1000BaseTX
Gi1/0/9 notconnect 1 auto auto 10/100/1000BaseTX
Gi1/0/10 notconnect 1 auto auto 10/100/1000BaseTX
Gi1/0/11 notconnect 1 auto auto 10/100/1000BaseTX

Any suggestions? I'm about to contact TACS but I figured I'd check the blogs first.

[Johannes_Grimm]

Hi Dave,

that sounds very weird. Which IOS version do you use? Have you tried different versions?

Best regards,

Johannes

[Davemac10]

It's the strangest thing I ever saw. The switch was part of a stack and was
up to date with 16.6.5 release. I haven't tried a different version because
all of our 3650s are running the same iOS. I'm thinking it's a hardware
issue but was just wondering if anyone else has seen this.

[omz]

Seen a similar issue. Software release may not match but do the Symptom/Conditions/Workaround match?

CSCvc51344

Description

Symptom:
Few connected Ip Phones are not coming up after switch reload . Issue happening on random ports . Each reload we found different phones connected on the switch not coming up .

No issue in available power .

Issue seen in below builds
3.6.3
3.6.5
3.6.5 ES
3.6.6

workaround : We need to bounce the connected interface .

Conditions:
Switch reload

Workaround:
Bounce the POE connected interfaces .

[Davemac10]

The whole switch has been rebooted and still has the issue. I plug Cisco
phone into one port and power States it's granted to another, yet the phone
does power on which is strange. It would lead to connectivity issues with
a PC attached to the phone.

[Leo Laohoo]

Run a TDR and see if layer 1 is OK or not.

[Davemac10]

Test came back normal:

Interface Speed Local pair Pair length Remote pair Pair status
--------- ----- ---------- ------------------ ----------- --------------------
Gi1/0/5 1000M

Pair A 5 +/- 10 meters N/A Normal
Pair B 5 +/- 10 meters N/A Normal
Pair C 7 +/- 10 meters N/A Normal
Pair D 5 +/- 10 meters N/A Normal

After further testing; plugging a phone into port 1 causes g1/0/3 to grant power (as well as g1/0/1).  Plugging into g1/0/5 gives me a message that g1/0/7 has PD Removed and then power device detected when no device is plugged into that port.  Here is a readout of me unplugging and plugging in g1/0/5 for example.  

Switch#
*Jul 31 17:27:17.358: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/7: PD removed
*Jul 31 17:27:17.360: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/7: PD removed
*Jul 31 17:27:17.489: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/5, changed state to down
*Jul 31 17:27:17.993: %ILPOWER-7-DETECT: Interface Gi1/0/7: Power Device detected: IEEE PD
*Jul 31 17:27:18.371: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/7: Power granted
*Jul 31 17:27:18.491: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/5, changed state to down
*Jul 31 17:27:25.963: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/5, changed state to up
*Jul 31 17:27:26.963: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/5, changed state to up

[Leo Laohoo]

CSCvj76259