Catalyst 3850 swich stack PoE faulty

479130 · ‎02-26-2019

Hello,

I have a stack of 6x 3850 PoE capable switches, recently some ports started to fail to give PoE and/or going up/down where PoE devices are connected to, like Cisco Access Points, Digium desk phones, other PoE devices.

I have:

1) changed cabling

2) perform shut / no shut

3) check PoE module is not overloaded

4) set power inline auto / static

5) connected devices to different switch ports on the same switch

6) connected devices to different switch ports on a different switch

7) reboot switch stack

From time to time some ports seems to fail to negotiate PoE and go to faulty state.

This is failing from switch 1,2 oldest to switch 5,6 newest devices added no more than a year ago.

I am having trouble diagnosing where the issue could be

below configuration:

#show env power all

SW PID Serial# Status Sys Pwr PoE Pwr Watts

-- ------------------ ---------- --------------- ------- ------- -----

1A PWR-C1-350WAC ART2003F757 OK Good Good 350

1B PWR-C1-715WAC DCA1949G04Z OK Good Good 715

2A PWR-C1-350WAC ART2003F78P OK Good Good 350

2B PWR-C1-715WAC DCA1949G052 OK Good Good 715

3A PWR-C1-350WAC ART2003F72S OK Good Good 350

3B PWR-C1-715WAC DCA1949G057 OK Good Good 715

4A PWR-C1-350WAC LIT17461E6H OK Good Good 350

4B PWR-C1-715WAC DCB1910G12V OK Good Good 715

5A PWR-C1-715WAC DCA2149G499 OK Good Good 715

5B PWR-C1-715WAC DCA2149G49K OK Good Good 715

6A PWR-C1-715WAC DCA2149G4DT No Input Power Bad Bad 715

#show switch detail

Switch/Stack Mac Address : 042a.e23f.7700 - Local Mac Address

Mac persistency wait time: Indefinite

H/W Current

Switch# Role Mac Address Priority Version State

------------------------------------------------------------

*1 Active 042a.e23f.7700 15 V07 Ready

2 Standby 042a.e23f.7600 14 V07 Ready

3 Member 042a.e23f.7680 13 V07 Ready

4 Member 8890.8dd8.6300 12 V04 Ready

5 Member 00f8.2c9b.9a00 1 V08 Ready

6 Member 00f8.2c9b.9700 1 V08 Ready

Stack Port Status Neighbors

Switch# Port 1 Port 2 Port 1 Port 2

--------------------------------------------------------

1 OK OK 2 6

2 OK OK 3 1

3 OK OK 4 2

4 OK OK 5 3

5 OK OK 6 4

6 OK OK 1 5

6B PWR-C1-715WAC DCA2149G4D2 OK Good Good 715

#show inventory

NAME: "c38xx Stack", DESCR: "c38xx Stack"

PID: WS-C3850-24P , VID: V07 , SN: FOC2012U0LV

NAME: "Switch 1", DESCR: "WS-C3850-24P-S"

PID: WS-C3850-24P-S , VID: V07 , SN: FOC2012U0LV

NAME: "Switch 1 - Power Supply A", DESCR: "Switch 1 - Power Supply A"

PID: PWR-C1-350WAC , VID: V01 , SN: ART2003F757

NAME: "Switch 1 - Power Supply B", DESCR: "Switch 1 - Power Supply B"

PID: PWR-C1-715WAC , VID: V01 , SN: DCA1949G04Z

NAME: "Switch 1 FRU Uplink Module 1", DESCR: "2x1G 2x10G Uplink Module"

PID: C3850-NM-2-10G , VID: V01 , SN: FOC20010HPK

NAME: "TenGigabitEthernet1/1/4", DESCR: "SFP-10GBase-SR"

PID: SFP-10G-SR , VID: V03 , SN: JUR2004GFHW

NAME: "Switch 2", DESCR: "WS-C3850-24P-S"

PID: WS-C3850-24P-S , VID: V07 , SN: FCW2012C0L6

NAME: "Switch 2 - Power Supply A", DESCR: "Switch 2 - Power Supply A"

PID: PWR-C1-350WAC , VID: V01 , SN: ART2003F78P

NAME: "Switch 2 - Power Supply B", DESCR: "Switch 2 - Power Supply B"

PID: PWR-C1-715WAC , VID: V01 , SN: DCA1949G052

NAME: "Switch 2 FRU Uplink Module 1", DESCR: "2x1G 2x10G Uplink Module"

PID: C3850-NM-2-10G , VID: V01 , SN: FOC20010HMB

NAME: "TenGigabitEthernet2/1/4", DESCR: "SFP-10GBase-SR"

PID: SFP-10G-SR , VID: V03 , SN: JUR2007G1DH

NAME: "Switch 3", DESCR: "WS-C3850-24P-S"

PID: WS-C3850-24P-S , VID: V07 , SN: FOC2012U0LW

NAME: "Switch 3 - Power Supply A", DESCR: "Switch 3 - Power Supply A"

PID: PWR-C1-350WAC , VID: V01 , SN: ART2003F72S

NAME: "Switch 3 - Power Supply B", DESCR: "Switch 3 - Power Supply B"

PID: PWR-C1-715WAC , VID: V01 , SN: DCA1949G057

NAME: "Switch 4", DESCR: "WS-C3850-24T-S"

PID: WS-C3850-24T-S , VID: V04 , SN: FOC1823U10S

NAME: "Switch 4 - Power Supply A", DESCR: "Switch 4 - Power Supply A"

PID: PWR-C1-350WAC , VID: V01 , SN: LIT17461E6H

NAME: "Switch 4 - Power Supply B", DESCR: "Switch 4 - Power Supply B"

PID: PWR-C1-715WAC , VID: V01 , SN: DCB1910G12V

NAME: "Switch 5", DESCR: "WS-C3850-24P-S"

PID: WS-C3850-24P-S , VID: V08 , SN: FCW2204C0VC

NAME: "Switch 5 - Power Supply A", DESCR: "Switch 5 - Power Supply A"

PID: PWR-C1-715WAC , VID: V02 , SN: DCA2149G499

NAME: "Switch 5 - Power Supply B", DESCR: "Switch 5 - Power Supply B"

PID: PWR-C1-715WAC , VID: V02 , SN: DCA2149G49K

NAME: "Switch 6", DESCR: "WS-C3850-24P-S"

PID: WS-C3850-24P-S , VID: V08 , SN: FCW2204C0V2

NAME: "Switch 6 - Power Supply A", DESCR: "Switch 6 - Power Supply A"

PID: PWR-C1-715WAC , VID: V02 , SN: DCA2149G4DT

NAME: "Switch 6 - Power Supply B", DESCR: "Switch 6 - Power Supply B"

PID: PWR-C1-715WAC , VID: V02 , SN: DCA2149G4D2

#show power inline

Module Available Used Remaining

(Watts) (Watts) (Watts)

------ --------- -------- ---------

1 720.0 0.0 720.0

Interface Admin Oper Power Device Class Max

(Watts)

--------- ------ ---------- ------- ------------------- ----- ----

Gi1/0/1 auto off 0.0 n/a n/a 30.0

Gi1/0/2 auto faulty 0.0 n/a n/a 30.0

Gi1/0/3 auto off 0.0 n/a n/a 30.0

Gi1/0/4 off off 0.0 n/a n/a 30.0

Gi1/0/5 auto faulty 0.0 n/a n/a 30.0

Gi1/0/6 auto off 0.0 n/a n/a 30.0

Gi1/0/7 auto faulty 0.0 n/a n/a 30.0

Gi1/0/8 auto off 0.0 n/a n/a 30.0

Gi1/0/9 auto off 0.0 n/a n/a 30.0

Gi1/0/10 auto off 0.0 n/a n/a 30.0

Gi1/0/11 auto off 0.0 n/a n/a 30.0

Gi1/0/12 auto off 0.0 n/a n/a 30.0

Gi1/0/13 auto off 0.0 n/a n/a 30.0

Gi1/0/14 auto off 0.0 n/a n/a 30.0

Gi1/0/15 auto off 0.0 n/a n/a 30.0

Gi1/0/16 auto off 0.0 n/a n/a 30.0

Gi1/0/17 auto off 0.0 n/a n/a 30.0

Gi1/0/18 auto off 0.0 n/a n/a 30.0

Gi1/0/19 auto off 0.0 n/a n/a 30.0

Gi1/0/20 auto off 0.0 n/a n/a 30.0

Gi1/0/21 auto off 0.0 n/a n/a 30.0

Gi1/0/22 auto off 0.0 n/a n/a 30.0

Gi1/0/23 auto off 0.0 n/a n/a 30.0

Gi1/0/24 auto off 0.0 n/a n/a 30.0

Module Available Used Remaining

(Watts) (Watts) (Watts)

------ --------- -------- ---------

2 720.0 79.7 640.3

Interface Admin Oper Power Device Class Max

(Watts)

--------- ------ ---------- ------- ------------------- ----- ----

Gi2/0/1 auto off 0.0 n/a n/a 30.0

Gi2/0/2 auto on 3.9 Ieee PD 2 30.0

Gi2/0/3 auto off 0.0 n/a n/a 30.0

Gi2/0/4 auto off 0.0 n/a n/a 30.0

Gi2/0/5 auto on 3.9 Ieee PD 2 30.0

Gi2/0/6 auto off 0.0 n/a n/a 30.0

Gi2/0/7 auto on 3.9 Ieee PD 2 30.0

Gi2/0/8 auto off 0.0 n/a n/a 30.0

Gi2/0/9 auto off 0.0 n/a n/a 30.0

Gi2/0/10 auto off 0.0 n/a n/a 30.0

Gi2/0/11 auto off 0.0 n/a n/a 30.0

Gi2/0/12 auto on 5.8 Ieee PD 2 30.0

Gi2/0/13 auto off 0.0 n/a n/a 30.0

Interface Admin Oper Power Device Class Max

(Watts)

--------- ------ ---------- ------- ------------------- ----- ----

Gi2/0/14 auto faulty 0.0 n/a n/a 30.0

Gi2/0/15 static faulty 0.0 n/a n/a 30.0

Gi2/0/16 auto off 0.0 n/a n/a 30.0

Gi2/0/17 auto off 0.0 n/a n/a 30.0

Gi2/0/18 auto off 0.0 n/a n/a 30.0

Gi2/0/19 auto off 0.0 n/a n/a 30.0

Gi2/0/20 auto off 0.0 n/a n/a 30.0

Gi2/0/21 auto on 16.8 AIR-CAP3602I-E-K9 4 30.0

Gi2/0/22 auto on 15.4 AIR-CAP3602I-E-K9 4 30.0

Gi2/0/23 auto off 0.0 n/a n/a 30.0

Gi2/0/24 auto off 0.0 n/a n/a 30.0

Module Available Used Remaining

(Watts) (Watts) (Watts)

------ --------- -------- ---------

3 720.0 78.3 641.7

Interface Admin Oper Power Device Class Max

(Watts)

--------- ------ ---------- ------- ------------------- ----- ----

Gi3/0/1 auto off 0.0 n/a n/a 30.0

Gi3/0/2 auto off 0.0 n/a n/a 30.0

Gi3/0/3 auto off 0.0 n/a n/a 30.0

Gi3/0/4 auto off 0.0 n/a n/a 30.0

Gi3/0/5 auto off 0.0 n/a n/a 30.0

Gi3/0/6 auto off 0.0 n/a n/a 30.0

Gi3/0/7 auto on 3.9 Ieee PD 2 30.0

Gi3/0/8 auto off 0.0 n/a n/a 30.0

Gi3/0/9 auto on 3.9 Ieee PD 2 30.0

Gi3/0/10 auto on 5.8 Ieee PD 2 30.0

Gi3/0/11 auto on 3.9 Ieee PD 2 30.0

Gi3/0/12 auto off 0.0 n/a n/a 30.0

Gi3/0/13 auto off 0.0 n/a n/a 30.0

Gi3/0/14 auto off 0.0 n/a n/a 30.0

Gi3/0/15 static off 30.0 n/a n/a 30.0

Gi3/0/16 auto off 0.0 n/a n/a 30.0

Gi3/0/17 auto off 0.0 n/a n/a 30.0

Gi3/0/18 auto off 0.0 n/a n/a 30.0

Gi3/0/19 auto off 0.0 n/a n/a 30.0

Gi3/0/20 auto faulty 0.0 n/a n/a 30.0

Gi3/0/21 auto on 15.4 AIR-CAP3602I-E-K9 4 30.0

Gi3/0/22 auto on 15.4 AIR-CAP3602I-E-K9 4 30.0

Gi3/0/23 auto faulty 0.0 n/a n/a 30.0

Gi3/0/24 auto off 0.0 n/a n/a 30.0

Module Available Used Remaining

(Watts) (Watts) (Watts)

------ --------- -------- ---------

4 n/a n/a n/a

Interface Admin Oper Power Device Class Max

(Watts)

--------- ------ ---------- ------- ------------------- ----- ----

Module Available Used Remaining

Interface Admin Oper Power Device Class Max

(Watts)

--------- ------ ---------- ------- ------------------- ----- ----

(Watts) (Watts) (Watts)

------ --------- -------- ---------

5 720.0 11.7 708.3

Interface Admin Oper Power Device Class Max

(Watts)

--------- ------ ---------- ------- ------------------- ----- ----

Gi5/0/1 auto off 0.0 n/a n/a 30.0

Gi5/0/2 auto off 0.0 n/a n/a 30.0

Gi5/0/3 auto off 0.0 n/a n/a 30.0

Gi5/0/4 auto off 0.0 n/a n/a 30.0

Gi5/0/5 auto off 0.0 n/a n/a 30.0

Gi5/0/6 auto off 0.0 n/a n/a 30.0

Gi5/0/7 auto off 0.0 n/a n/a 30.0

Gi5/0/8 auto off 0.0 n/a n/a 30.0

Gi5/0/9 auto off 0.0 n/a n/a 30.0

Gi5/0/10 auto off 0.0 n/a n/a 30.0

Gi5/0/11 auto off 0.0 n/a n/a 30.0

Gi5/0/12 auto on 3.9 Ieee PD 2 30.0

Gi5/0/13 auto off 0.0 n/a n/a 30.0

Gi5/0/14 auto on 3.9 Ieee PD 2 30.0

Gi5/0/15 auto off 0.0 n/a n/a 30.0

Gi5/0/16 auto off 0.0 n/a n/a 30.0

Gi5/0/17 auto off 0.0 n/a n/a 30.0

Gi5/0/18 auto off 0.0 n/a n/a 30.0

Gi5/0/19 auto on 3.9 Ieee PD 2 30.0

Gi5/0/20 auto off 0.0 n/a n/a 30.0

Gi5/0/21 auto off 0.0 n/a n/a 30.0

Gi5/0/22 auto off 0.0 n/a n/a 30.0

Gi5/0/23 auto off 0.0 n/a n/a 30.0

Gi5/0/24 auto off 0.0 n/a n/a 30.0

Module Available Used Remaining

(Watts) (Watts) (Watts)

------ --------- -------- ---------

6 720.0 16.8 703.2

Interface Admin Oper Power Device Class Max

(Watts)

--------- ------ ---------- ------- ------------------- ----- ----

Gi6/0/1 auto off 0.0 n/a n/a 30.0

Gi6/0/2 auto off 0.0 n/a n/a 30.0

Gi6/0/3 auto off 0.0 n/a n/a 30.0

Gi6/0/4 auto off 0.0 n/a n/a 30.0

Gi6/0/5 auto off 0.0 n/a n/a 30.0

Gi6/0/6 auto off 0.0 n/a n/a 30.0

Gi6/0/7 auto off 0.0 n/a n/a 30.0

Gi6/0/8 auto off 0.0 n/a n/a 30.0

Gi6/0/9 auto off 0.0 n/a n/a 30.0

Gi6/0/10 auto faulty 0.0 n/a n/a 30.0

Gi6/0/11 auto off 0.0 n/a n/a 30.0

Gi6/0/12 auto off 0.0 n/a n/a 30.0

Gi6/0/13 auto off 0.0 n/a n/a 30.0

Gi6/0/14 auto off 0.0 n/a n/a 30.0

Gi6/0/15 auto off 0.0 n/a n/a 30.0

Interface Admin Oper Power Device Class Max

(Watts)

--------- ------ ---------- ------- ------------------- ----- ----

Gi6/0/16 auto off 0.0 n/a n/a 30.0

Gi6/0/17 auto off 0.0 n/a n/a 30.0

Gi6/0/18 auto off 0.0 n/a n/a 30.0

Gi6/0/19 auto off 0.0 n/a n/a 30.0

Gi6/0/20 auto off 0.0 n/a n/a 30.0

Gi6/0/21 auto off 0.0 n/a n/a 30.0

Gi6/0/22 auto on 16.8 AIR-CAP3602I-E-K9 4 30.0

Gi6/0/23 auto off 0.0 n/a n/a 30.0

Gi6/0/24 auto off 0.0 n/a n/a 30.0

#show ver

Cisco IOS Software, IOS-XE Software, Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 03.06.06E RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Compiled Sat 17-Dec-16 00:22 by prod_rel_team

licensed under the GNU General Public License ("GPL") Version 2.0. The

software code licensed under GPL Version 2.0 is free software that comes

with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such

GPL code under the terms of GPL Version 2.0.

(http://www.gnu.org/licenses/gpl-2.0.html) For more details, see the

documentation or "License Notice" file accompanying the IOS-XE software,

or the applicable URL provided on the flyer accompanying the IOS-XE

software.

ROM: IOS-XE ROMMON

BOOTLDR: CAT3K_CAA Boot Loader (CAT3K_CAA-HBOOT-M) Version 3.58, RELEASE SOFTWARE (P)

LLSASW01 uptime is 4 weeks, 1 hour, 59 minutes

Uptime for this control processor is 4 weeks, 2 hours, 6 minutes

System returned to ROM by reload at 07:20:26 GMT Tue Jan 29 2019

System restarted at 07:29:13 GMT Tue Jan 29 2019

System image file is "flash:packages.conf"

Last reload reason: Reload command

This product contains cryptographic features and is subject to United

States and local country laws governing import, export, transfer and

use. Delivery of Cisco cryptographic products does not imply

third-party authority to import, export, distribute or use encryption.

Importers, exporters, distributors and users are responsible for

compliance with U.S. and local country laws. By using this product you

agree to comply with applicable laws and regulations. If you are unable

to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:

http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to

export@cisco.com.

License Level: Ipbase

License Type: Permanent

Next reload license Level: Ipbase

cisco WS-C3850-24P (MIPS) processor with 4194304K bytes of physical memory.

Processor board ID FOC2012U0LV

18 Virtual Ethernet interfaces

168 Gigabit Ethernet interfaces

24 Ten Gigabit Ethernet interfaces

2048K bytes of non-volatile configuration memory.

4194304K bytes of physical memory.

250456K bytes of Crash Files at crashinfo:.

250456K bytes of Crash Files at crashinfo-2:.

250456K bytes of Crash Files at crashinfo-3:.

250456K bytes of Crash Files at crashinfo-4:.

252000K bytes of Crash Files at crashinfo-5:.

252000K bytes of Crash Files at crashinfo-6:.

1609272K bytes of Flash at flash:.

1609272K bytes of Flash at flash-2:.

1609272K bytes of Flash at flash-3:.

1609272K bytes of Flash at flash-4:.

1611414K bytes of Flash at flash-5:.

1611414K bytes of Flash at flash-6:.

0K bytes of Dummy USB Flash at usbflash0:.

0K bytes of Dummy USB Flash at usbflash0-2:.

0K bytes of Dummy USB Flash at usbflash0-3:.

0K bytes of Dummy USB Flash at usbflash0-4:.

0K bytes of Dummy USB Flash at usbflash0-5:.

0K bytes of Dummy USB Flash at usbflash0-6:.

0K bytes of at webui:.

Base Ethernet MAC Address : 04:2a:e2:3f:77:00

Motherboard Assembly Number : 73-15805-04

Motherboard Serial Number : FOC20123AG3

Model Revision Number : AA0

Motherboard Revision Number : B0

Model Number : WS-C3850-24P

System Serial Number : FOC2012U0LV

Switch Ports Model SW Version SW Image Mode

------ ----- ----- ---------- ---------- ----

* 1 32 WS-C3850-24P 03.06.06E cat3k_caa-universalk9 INSTALL

2 32 WS-C3850-24P 03.06.06E cat3k_caa-universalk9 INSTALL

3 32 WS-C3850-24P 03.06.06E cat3k_caa-universalk9 INSTALL

4 32 WS-C3850-24T 03.06.06E cat3k_caa-universalk9 INSTALL

5 32 WS-C3850-24P 03.06.06E cat3k_caa-universalk9 INSTALL

6 32 WS-C3850-24P 03.06.06E cat3k_caa-universalk9 INSTALL

Switch 02

---------

Switch uptime : 4 weeks, 2 hours, 5 minutes

Base Ethernet MAC Address : 04:2a:e2:3f:76:00

Motherboard Assembly Number : 73-15805-04

Motherboard Serial Number : FOC20123AHM

Model Revision Number : AA0

Motherboard Revision Number : B0

Model Number : WS-C3850-24P

System Serial Number : FCW2012C0L6

Switch 03

---------

Switch uptime : 4 weeks, 2 hours, 6 minutes

Base Ethernet MAC Address : 04:2a:e2:3f:76:80

Motherboard Assembly Number : 73-15805-04

Motherboard Serial Number : FOC20123AT2

Model Revision Number : AA0

Motherboard Revision Number : B0

Model Number : WS-C3850-24P

System Serial Number : FOC2012U0LW

Switch 04

---------

Switch uptime : 4 weeks, 2 hours, 5 minutes

Base Ethernet MAC Address : 88:90:8d:d8:63:00

Motherboard Assembly Number : 73-14443-08

Motherboard Serial Number : FOC1815006Y

Model Revision Number : S0

Motherboard Revision Number : A0

Model Number : WS-C3850-24T

System Serial Number : FOC1823U10S

Switch 05

---------

Switch uptime : 4 weeks, 2 hours, 6 minutes

Base Ethernet MAC Address : 00:f8:2c:9b:9a:00

Motherboard Assembly Number : 73-15805-05

Motherboard Serial Number : FOC22035QYP

Model Revision Number : AD0

Motherboard Revision Number : B0

Model Number : WS-C3850-24P

System Serial Number : FCW2204C0VC

Switch 06

---------

Switch uptime : 4 weeks, 2 hours, 6 minutes

Base Ethernet MAC Address : 00:f8:2c:9b:97:00

Motherboard Assembly Number : 73-15805-05

Motherboard Serial Number : FOC22035QMH

Model Revision Number : AD0

Motherboard Revision Number : B0

Model Number : WS-C3850-24P

System Serial Number : FCW2204C0V2

Configuration register is 0x102

Note:

we have an unmanaged 4ports PoE switch that is connected to Gi2/0/15 and throws out PoE messages constantly but I don't think this is related to the current issue as it was in this hardware configuration way before any problems came up, here is the term mon just in case:

Feb 26 08:21:24: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/15, changed state to up

Feb 26 08:21:54: %ILPOWER-7-DETECT: Interface Gi2/0/15: Power Device detected: IEEE PD

Feb 26 08:21:57: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi2/0/15: PD removed

Feb 26 08:21:57: %ILPOWER-3-CONTROLLER_PORT_ERR: Controller port error, Interface Gi2/0/15: Power given, but Power Controller does not report Power Good

Feb 26 08:23:31: %ILPOWER-7-DETECT: Interface Gi2/0/15: Power Device detected: IEEE PD

Feb 26 08:23:34: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi2/0/15: PD removed

Feb 26 08:23:34: %ILPOWER-3-CONTROLLER_PORT_ERR: Controller port error, Interface Gi2/0/15: Power given, but Power Controller does not report Power Good

Feb 26 08:23:36: %ILPOWER-7-DETECT: Interface Gi2/0/15: Power Device detected: IEEE PD

Feb 26 08:23:39: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi2/0/15: PD removed

Feb 26 08:23:39: %ILPOWER-3-CONTROLLER_PORT_ERR: Controller port error, Interface Gi2/0/15: Power given, but Power Controller does not report Power Good

thanks for your suggestions in advance

Leo Laohoo · ‎02-26-2019

@479130 wrote:

Feb 26 08:23:39: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi2/0/15: PD removedFeb 26 08:23:39: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi2/0/15: PD removed

Is this one of the ports?

If yes, post the complete output to the command "sh interface Gi2/0/15 controll".

479130 · ‎02-26-2019

No, 2/0/15 is unmanaged PoE 4-ports switch

#show interfaces gigabitEthernet 2/0/15 controll

GigabitEthernet2/0/15 is up, line protocol is up (connected)

Hardware is Gigabit Ethernet, address is 042a.e23f.760f (bia 042a.e23f.760f)

MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX

input flow-control is off, output flow-control is unsupported

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:06:20, output never, output hang never

Last clearing of "show interface" counters never

Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 5062561

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 30000 bits/sec, 4 packets/sec

5 minute output rate 36000 bits/sec, 14 packets/sec

29029458 packets input, 25985163070 bytes, 0 no buffer

Received 369741 broadcasts (48309 multicasts)

0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 watchdog, 48309 multicast, 0 pause input

0 input packets with dribble condition detected

36787289 packets output, 28952678333 bytes, 0 underruns

5062561 output errors, 0 collisions, 1 interface resets

0 unknown protocol drops

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier, 0 pause output

0 output buffer failures, 0 output buffers swapped out

Transmit GigabitEthernet2/0/15 Receive

28952678333 Total bytes 25985163070 Total bytes

26947533 Unicast frames 28659717 Unicast frames

27215681791 Unicast bytes 25895137100 Unicast bytes

5822034 Multicast frames 48309 Multicast frames

1224545189 Multicast bytes 15298753 Multicast bytes

4017722 Broadcast frames 321432 Broadcast frames

512451353 Broadcast bytes 74727217 Broadcast bytes

0 System FCS error frames 0 IpgViolation frames

0 MacUnderrun frames 0 MacOverrun frames

0 Pause frames 0 Pause frames

0 Cos 0 Pause frames 0 Cos 0 Pause frames

0 Cos 1 Pause frames 0 Cos 1 Pause frames

0 Cos 2 Pause frames 0 Cos 2 Pause frames

0 Cos 3 Pause frames 0 Cos 3 Pause frames

0 Cos 4 Pause frames 0 Cos 4 Pause frames

0 Cos 5 Pause frames 0 Cos 5 Pause frames

0 Cos 6 Pause frames 0 Cos 6 Pause frames

0 Cos 7 Pause frames 0 Cos 7 Pause frames

0 Oam frames 0 OamProcessed frames

0 Oam frames 0 OamDropped frames

9043267 Minimum size frames 6963726 Minimum size frames

4285438 65 to 127 byte frames 3692984 65 to 127 byte frames

3710326 128 to 255 byte frames 712406 128 to 255 byte frames

1744712 256 to 511 byte frames 947491 256 to 511 byte frames

732120 512 to 1023 byte frames 619115 512 to 1023 byte frames

17271426 1024 to 1518 byte frames 16093736 1024 to 1518 byte frames

0 1519 to 2047 byte frames 0 1519 to 2047 byte frames

0 2048 to 4095 byte frames 0 2048 to 4095 byte frames

0 4096 to 8191 byte frames 0 4096 to 8191 byte frames

0 8192 to 16383 byte frames 0 8192 to 16383 byte frames

0 16384 to 32767 byte frame 0 16384 to 32767 byte frame

0 > 32768 byte frames 0 > 32768 byte frames

0 Late collision frames 0 SymbolErr frames

5062561 Excess Defer frames 0 Collision fragments

0 Good (1 coll) frames 0 ValidUnderSize frames

0 Good (>1 coll) frames 0 InvalidOverSize frames

0 Deferred frames 0 ValidOverSize frames

0 Gold frames dropped 0 FcsErr frames

0 Gold frames truncated

0 Gold frames successful

0 1 collision frames

0 2 collision frames

0 3 collision frames

0 4 collision frames

0 5 collision frames

0 6 collision frames

0 7 collision frames

0 8 collision frames

0 9 collision frames

0 10 collision frames

0 11 collision frames

0 12 collision frames

0 13 collision frames

0 14 collision frames

0 15 collision frames

0 Excess collision frames

LAST UPDATE 100 msecs AGO

One device that fails to receive PoE is 3/0/15 and it shows not connect at the minute (it's a Cisco AP)

LLSASW01#show interfaces gigabitEthernet 3/0/15 controll

GigabitEthernet3/0/15 is down, line protocol is down (notconnect)

Hardware is Gigabit Ethernet, address is 042a.e23f.768f (bia 042a.e23f.768f)

Description: ===> temp AP C20 <===

MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

Auto-duplex, Auto-speed, media type is 10/100/1000BaseTX

input flow-control is off, output flow-control is unsupported

ARP type: ARPA, ARP Timeout 04:00:00

Last input never, output never, output hang never

Last clearing of "show interface" counters never

Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 0 bits/sec, 0 packets/sec

5 minute output rate 0 bits/sec, 0 packets/sec

0 packets input, 0 bytes, 0 no buffer

Received 0 broadcasts (0 multicasts)

0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 watchdog, 0 multicast, 0 pause input

0 input packets with dribble condition detected

0 packets output, 0 bytes, 0 underruns

0 output errors, 0 collisions, 1 interface resets

0 unknown protocol drops

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier, 0 pause output

0 output buffer failures, 0 output buffers swapped out

Transmit GigabitEthernet3/0/15 Receive

0 Total bytes 0 Total bytes

0 Unicast frames 0 Unicast frames

0 Unicast bytes 0 Unicast bytes

0 Multicast frames 0 Multicast frames

0 Multicast bytes 0 Multicast bytes

0 Broadcast frames 0 Broadcast frames

0 Broadcast bytes 0 Broadcast bytes

0 System FCS error frames 0 IpgViolation frames

0 MacUnderrun frames 0 MacOverrun frames

0 Pause frames 0 Pause frames

0 Cos 0 Pause frames 0 Cos 0 Pause frames

0 Cos 1 Pause frames 0 Cos 1 Pause frames

0 Cos 2 Pause frames 0 Cos 2 Pause frames

0 Cos 3 Pause frames 0 Cos 3 Pause frames

0 Cos 4 Pause frames 0 Cos 4 Pause frames

0 Cos 5 Pause frames 0 Cos 5 Pause frames

0 Cos 6 Pause frames 0 Cos 6 Pause frames

0 Cos 7 Pause frames 0 Cos 7 Pause frames

0 Oam frames 0 OamProcessed frames

0 Oam frames 0 OamDropped frames

0 Minimum size frames 0 Minimum size frames

0 65 to 127 byte frames 0 65 to 127 byte frames

0 128 to 255 byte frames 0 128 to 255 byte frames

0 256 to 511 byte frames 0 256 to 511 byte frames

0 512 to 1023 byte frames 0 512 to 1023 byte frames

0 1024 to 1518 byte frames 0 1024 to 1518 byte frames

0 1519 to 2047 byte frames 0 1519 to 2047 byte frames

0 2048 to 4095 byte frames 0 2048 to 4095 byte frames

0 4096 to 8191 byte frames 0 4096 to 8191 byte frames

0 8192 to 16383 byte frames 0 8192 to 16383 byte frames

0 16384 to 32767 byte frame 0 16384 to 32767 byte frame

0 > 32768 byte frames 0 > 32768 byte frames

0 Late collision frames 0 SymbolErr frames

0 Excess Defer frames 0 Collision fragments

0 Good (1 coll) frames 0 ValidUnderSize frames

0 Good (>1 coll) frames 0 InvalidOverSize frames

0 Deferred frames 0 ValidOverSize frames

0 Gold frames dropped 0 FcsErr frames

0 Gold frames truncated

0 Gold frames successful

0 1 collision frames

0 2 collision frames

0 3 collision frames

0 4 collision frames

0 5 collision frames

0 6 collision frames

0 7 collision frames

0 8 collision frames

0 9 collision frames

0 10 collision frames

0 11 collision frames

0 12 collision frames

0 13 collision frames

0 14 collision frames

0 15 collision frames

0 Excess collision frames

LAST UPDATE 2429356970 msecs AGO

Leo Laohoo · ‎02-26-2019

No line errors.
Can you do the following:
1. Command: test cable tdr int Gi 2/0/15;
2. Wait for approximately 5 to 7 seconds for the test to run;
3. Command: sh cable tdr int Gi 2/0/15; and
4. Post the output to #3.
Do the same to Gi 3/0/15.
NOTE: I recommend you upgrade the firmware of the switch. I believe you're hitting CSCvb65304.

479130 · ‎02-26-2019

Does that makes any sense?

pins are shorted? and inverted for PairA,B?

#show cable-diagnostics tdr int gig 1/0/5
TDR test last run on: February 26 16:41:28

Interface Speed Local pair Pair length Remote pair Pair status
--------- ----- ---------- ------------------ ----------- --------------------
Gi1/0/5 100M Pair A 0 +/- 1 meters Pair B Normal
Pair B 0 +/- 1 meters Pair A Normal
Pair C 63 +/- 1 meters N/A Short
Pair D 65 +/- 1 meters N/A Short

#test cable-diagnostics tdr int gig 2/0/15
TDR test started on interface Gi2/0/15
A TDR test can take a few seconds to run on an interface
Use 'show cable-diagnostics tdr' to read the TDR results.

#show cable-diagnostics tdr int gig 2/0/15
TDR test last run on: February 26 16:43:35

Interface Speed Local pair Pair length Remote pair Pair status
--------- ----- ---------- ------------------ ----------- --------------------
Gi2/0/15 1000M Pair A 50 +/- 10 meters Pair A Normal
Pair B 50 +/- 10 meters Pair B Normal
Pair C 50 +/- 10 meters Pair C Normal
Pair D 50 +/- 10 meters Pair D Normal

Leo Laohoo · ‎02-26-2019

Short at 63 metres. That means the cable nearest the client (or furthest away from the switch port) is faulty. Replace that and re-run the test if it still doesn't work.
If the cable has been replaced and the issue persist, then remove the cable and re-run the TDR so we can determine if the punch block is faulty or not.

One important thing: Notice the TDR result that says Pairs A and B are "Normal" and Pairs C & D are "short"? This tells me that the cable used was never meant for PoE and will not support GigabitEthernet. This also means that the cable used was either manually (vs machine molded) terminated or someone picked up a cable from a junk shop.

For more information about TDR, read How to use Time-Domain Reflectometer (TDR).

479130 · ‎03-05-2019

Hello Leo,

thanks for your support.

As you said, we probably have shorts on LAN cable connecting VoIP phone, let's just mark that as faulty cable.

But we recently had other devices going PoE down after creating this ticket, one in particular is a watchdog connected directly to the switch with a cat6 0.5m cable, I wouldn't believe this has a short aswell.

switchport in question is 1/0/1, devices got no PoE for over a week, I wasn't able to grant PoE to it this morning by shut/no shut or physically disconnect and reconnect LAN cable. At the end I have connected a power supply to the device.

I came back few hours later and disconnect power supply from watchdog and now it seems to have PoE granted from switch again.

What could it be? As said, it's seems that recently we have this behaviour to random switchports.

Mar 5 12:54:39: %ILPOWER-7-DETECT: Interface Gi5/0/7: Power Device detected: IEEE PD
Mar 5 12:54:42: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi5/0/7: PD removed
Mar 5 12:54:42: %ILPOWER-3-CONTROLLER_PORT_ERR: Controller port error, Interface Gi5/0/7: Power given, but Power Controller does not report Power Good
Mar 5 12:56:32: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/15, changed state to down
Mar 5 12:56:33: %LINK-3-UPDOWN: Interface GigabitEthernet4/0/15, changed state to down
Mar 5 12:56:50: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/17, changed state to down
Mar 5 12:56:51: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/17, changed state to down
Mar 5 12:57:11: %ILPOWER-7-DETECT: Interface Gi3/0/23: Power Device detected: IEEE PD
Mar 5 12:57:14: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi3/0/23: PD removed
Mar 5 12:57:14: %ILPOWER-3-CONTROLLER_PORT_ERR: Controller port error, Interface Gi3/0/23: Power given, but Power Controller does not report Power Good
Mar 5 12:57:28: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet5/0/4, changed state to down
Mar 5 12:57:29: %LINK-3-UPDOWN: Interface GigabitEthernet5/0/4, changed state to down
Mar 5 12:57:36: %LINK-3-UPDOWN: Interface GigabitEthernet5/0/4, changed state to up
Mar 5 12:57:37: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet5/0/4, changed state to up
Mar 5 12:58:37: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet5/0/4, changed state to down
Mar 5 12:58:38: %LINK-3-UPDOWN: Interface GigabitEthernet5/0/4, changed state to down
Mar 5 12:59:17: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to down
Mar 5 12:59:18: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to down
Mar 5 12:59:24: %ILPOWER-7-DETECT: Interface Gi1/0/1: Power Device detected: IEEE PD
Mar 5 12:59:25: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/1: Power granted
Mar 5 12:59:28: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/1: PD removed
Mar 5 12:59:29: %ILPOWER-7-DETECT: Interface Gi1/0/1: Power Device detected: IEEE PD
Mar 5 12:59:30: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/1: Power granted
Mar 5 12:59:32: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/1: PD removed
Mar 5 12:59:33: %ILPOWER-7-DETECT: Interface Gi1/0/1: Power Device detected: IEEE PD
Mar 5 12:59:34: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/1: Power granted
Mar 5 12:59:36: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/1: PD removed
Mar 5 12:59:40: %ILPOWER-7-DETECT: Interface Gi1/0/1: Power Device detected: IEEE PD
Mar 5 12:59:41: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/1: Power granted
Mar 5 12:59:44: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/1: PD removed
Mar 5 12:59:45: %ILPOWER-7-DETECT: Interface Gi1/0/1: Power Device detected: IEEE PD
Mar 5 12:59:46: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/1: Power granted
Mar 5 12:59:46: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/1: PD removed
Mar 5 12:59:46: %ILPOWER-7-DETECT: Interface Gi1/0/1: Power Device detected: IEEE PD
Mar 5 12:59:47: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/1: Power granted
Mar 5 12:59:51: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to up
Mar 5 12:59:52: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/1: PD removed
Mar 5 12:59:52: %ILPOWER-7-DETECT: Interface Gi1/0/1: Power Device detected: IEEE PD
Mar 5 12:59:53: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to down
Mar 5 12:59:54: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/1: Power granted
Mar 5 12:59:56: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to up
Mar 5 12:59:57: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to up

479130 · ‎03-05-2019

and switchport 1/0/1 device went down again as no PoE granted

#test cable-diagnostics tdr interface gigabitEthernet 1/0/1

TDR test started on interface Gi1/0/1

A TDR test can take a few seconds to run on an interface

Use 'show cable-diagnostics tdr' to read the TDR results.

#show cable-diagnostics tdr int gig 1/0/1

TDR test last run on: March 05 14:07:55

Interface Speed Local pair Pair length Remote pair Pair status

--------- ----- ---------- ------------------ ----------- --------------------

Gi1/0/1 auto Pair A 0 +/- 1 meters N/A Normal

Pair B 0 +/- 1 meters N/A Normal

Pair C 1 +/- 1 meters N/A Short

Pair D 1 +/- 1 meters N/A Short

At this stage it's very unlikely we can have another short

andrew.butterworth · ‎07-20-2019

I have a 3650 with some 100Mbps PoE devices (IP cameras) and I get a similar output with the TDR test showing a short on pairs C & D however the ports are working fine.

cat-3650#show cable-diagnostics tdr interface gigabitEthernet 1/0/20
TDR test last run on: July 20 11:24:47

Interface Speed Local pair Pair length Remote pair Pair status
--------- ----- ---------- ------------------ ----------- --------------------
Gi1/0/20 100M Pair A 0 +/- 1 meters Pair A Normal
Pair B 0 +/- 1 meters Pair B Normal
Pair C 13 +/- 1 meters N/A Short
Pair D 14 +/- 1 meters N/A Short

I think the PoE device is actually causing this result so is probably nothing to worry about. For 1000Mbps connected ports I get 'normal' on all pairs:

cat-3650#show cable-diagnostics tdr interface gigabitEthernet 1/0/38
TDR test last run on: July 20 11:27:26

Interface Speed Local pair Pair length Remote pair Pair status
--------- ----- ---------- ------------------ ----------- --------------------
Gi1/0/38 1000M Pair A 4 +/- 10 meters Pair B Normal
Pair B 4 +/- 10 meters Pair A Normal
Pair C 4 +/- 10 meters Pair C Normal
Pair D 4 +/- 10 meters Pair D Normal

Leo Laohoo · ‎07-20-2019

Try using LLDP instead of CDP.

479130 · ‎03-07-2019

Any more ideas or suggestions to look at?

appreciated

Georg Pauwen · ‎03-07-2019

Hello,

the one thing left to try is to upgrade your stack to release Denali-16.3.7 MD (which apparently is the recommended release)...

479130 · ‎03-07-2019

Thanks for your reply Georg,

I'm not inclined to upgrade if not extremely necessary as this creates downtime.

Fact is I have 3 switchstacks in 3 different offices on exactly same release

Office 1: stack of 6 switches 3850

Office 2: stack of 6 switches 3850 -> this is the one with PoE issues

Office 3: stack of 4 switches 3850

they are all on 03.06.06E, and we use similar connected PoE devices to all of them (ex. Cisco APs, Digium D40 VoIP, watchdogs etc) therefore I would exclude it as OS version issue

Leo Laohoo · ‎03-08-2019

TDR report shows faulty cable. Distance shows the cable nearest the switch port is faulty.

479130 · ‎03-12-2019

thanks, I will swap cable and test again port 1/0/1

Today we have PoE fault on:

Interface Speed Local pair Pair length Remote pair Pair status
--------- ----- ---------- ------------------ ----------- --------------------
Gi1/0/2 auto Pair A 48 +/- 1 meters N/A Open
Pair B 50 +/- 1 meters N/A Open
Pair C 49 +/- 1 meters N/A Open
Pair D 51 +/- 1 meters N/A Open

#show cable-diagnostics tdr interface gigabitEthernet 1/0/7
TDR test last run on: March 12 10:32:52

Interface Speed Local pair Pair length Remote pair Pair status
--------- ----- ---------- ------------------ ----------- --------------------
Gi1/0/7 auto Pair A 64 +/- 1 meters N/A Open
Pair B 65 +/- 1 meters N/A Open
Pair C 63 +/- 1 meters N/A Short
Pair D 65 +/- 1 meters N/A Short

#show cable-diagnostics tdr interface gigabitEthernet 2/0/14
TDR test last run on: March 12 10:32:59

Interface Speed Local pair Pair length Remote pair Pair status
--------- ----- ---------- ------------------ ----------- --------------------
Gi2/0/14 1000M Pair A 50 +/- 10 meters Pair A Normal
Pair B 50 +/- 10 meters Pair B Normal
Pair C 50 +/- 10 meters Pair C Normal
Pair D 50 +/- 10 meters Pair D Normal

#show cable-diagnostics tdr interface gigabitEthernet 3/0/20
TDR test last run on: March 12 10:33:07

Interface Speed Local pair Pair length Remote pair Pair status
--------- ----- ---------- ------------------ ----------- --------------------
Gi3/0/20 1000M Pair A 33 +/- 10 meters Pair B Normal
Pair B 33 +/- 10 meters Pair A Normal
Pair C 33 +/- 10 meters Pair D Normal
Pair D 33 +/- 10 meters Pair C Normal

#show cable-diagnostics tdr interface gigabitEthernet 3/0/23
TDR test last run on: March 12 10:33:11

Interface Speed Local pair Pair length Remote pair Pair status
--------- ----- ---------- ------------------ ----------- --------------------
Gi3/0/23 100M Pair A 0 +/- 1 meters Pair A Normal
Pair B 0 +/- 1 meters Pair B Normal
Pair C 12 +/- 1 meters N/A Short
Pair D 13 +/- 1 meters N/A Short

#show cable-diagnostics tdr interface gigabitEthernet 5/0/7
TDR test last run on: March 12 10:33:17

Interface Speed Local pair Pair length Remote pair Pair status
--------- ----- ---------- ------------------ ----------- --------------------
Gi5/0/7 100M Pair A 0 +/- 1 meters Pair B Normal
Pair B 0 +/- 1 meters Pair A Normal
Pair C 30 +/- 1 meters N/A Short
Pair D 31 +/- 1 meters N/A Short

would that indicate we have more cable fault on 1/0/2, 1/0/7, 3/0/23, 5/0/7 ?