Device won't turn on via PoE. Link up, link down, link up, etc.

PatrickG_SF · ‎10-28-2020

Hi all,

I have a device made by a manufacturer called Crestron that takes PoE plus. I have a pair Catalyst 3850s with stacking cable, and quite a few PoE devices on it, all of them 15.4W, but still far within my PoE budget. My issues is, when I plug in 2 Crestron device, I get console feedback 'link is up', 'link is down', etc. I get so much chatter that it is hard to debug what is going on without unplugging it. I have LLDP enabled and CDR disabled, and no luck. I have noticed a few odd things:

- When I run a test cable diagnostic, I get a reported short on 2 pairs. This happened with 3 different cables, on 2 different devices. One of these cables, I was able to power the device with an unmanaged switch and there were no apparent issues.

- I have seen the switch report 'power granted' for the respective interface, but no connectivity.

- As mentioned, I have an unmanaged Ubiquiti PoE+ switch and the unit fires up fine with 0 issues.

Any help is appreciated here. Also,

- Here is a snippet of the console when I run a debug ilpower powerman command:

BMR-GOP(config-if)#
*Oct 28 22:13:56.038: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/1: PD removed
*Oct 28 22:13:56.955: %ILPOWER-7-DETECT: Interface Gi1/0/1: Power Device detected: IEEE PD
*Oct 28 22:13:56.955: (Gi1/0/1) data power pool 1
*Oct 28 22:13:56.955: Ilpower PD device 3 class 7 from interface (Gi1/0/1)
*Oct 28 22:13:56.955: (Gi1/0/1) state auto
*Oct 28 22:13:56.955: (Gi1/0/1) data power pool: 1, pool 1
*Oct 28 22:13:56.955: (Gi1/0/1) curr pwr usage 30800
*Oct 28 22:13:56.956: (Gi1/0/1) req pwr 15400
*Oct 28 22:13:56.956: (Gi1/0/1) total pwr 405000
*Oct 28 22:13:56.956: (Gi1/0/1) power_status OK
*Oct 28 22:13:56.956: ilpower new power from pd discovery Gi1/0/1, power_status ok
*Oct 28 22:13:56.956: Ilpower interface (Gi1/0/1) power status change, allocated power 15400
*Oct 28 22:13:56.956: ilpower_notify_lldp_tlv: lldp power class tlv:shut
BMR-GOP(config-if)#
*Oct 28 22:13:56.956: (curr/prev) pwr value 15400/0
*Oct 28 22:13:56.958: ilpower_notify_lldp_power_via_mdi_tlv Gi1/0/1 pwr alloc 15400
*Oct 28 22:13:56.958: Gi1/0/1 AUTO PORT PWR Alloc 130 Request 130
*Oct 28 22:13:56.958: Gi1/0/1: LLDP NOTIFY TLV:
(curr/prev) PSE Allocation: 13000/0
(curr/prev) PD Request : 13000/0
(curr/prev) PD Class : Class 4/
(curr/prev) PD Priority : low/unknown
(curr/prev) Power Type : Type 2 PSE/Type 2 PSE
(curr/prev) mdi_pwr_support: 7/0
(curr/prev Power Pair) : Signal/
(curr/prev) PSE Pwr Source : Primary/Unknown
*Oct 28 22:13:57.653: ilpower delete power from pd linkdown Gi1/0/1
*Oct 28 22:13:57.653: Ilpower interface (Gi1/0/1), delete allocated power 15400
*Oct 28 22:13:57.653: Ilpower interface (Gi1/0/1) setting ICUT_OFF threshold to 0.
*Oct 28 22:13:57.653: ilpower_notify_lldp_power_via_mdi_tlv Gi1/0/1 pwr alloc 0
*Oct 28 22:13:57.653: Gi1/0/1 AUTO PORT PWR Alloc 130 Request 130
*Oct 28 22:13:57.654: Gi1/0/1: LLDP NOTIFY TLV:
(curr/prev) PSE Allocation: 13000/0
(curr/prev) PD Request : 13000/0
(curr/prev) PD Class : Class 4/
(curr/prev) PD Priority : low/unknown
(curr/prev) Power Type : Type 2 PSE/Type 2 PSE
(curr/prev) mdi_pwr_support: 7/0
(curr/prev Power Pair) : Signal/
(curr/prev) PSE Pwr Source : Primary/Unknown
*Oct 28 22:13:57.654: ilpower_notify_lldp_tlv: lldp power class tlv:
*Oct 28 22:13:57.654: (curr/prev) pwr value 15400/0
*Oct 28 22:13:59.659: %LINK-5-CHANGED: Interface GigabitEthernet1/0/1, changed state to administratively downAllocAlloc
^

Leo Laohoo · ‎10-28-2020

I think I have a good idea what is going on.

Post the complete output to the following commands:

sh version

In this output, I want to see the IOS-XE version you are running as well as the serial numbers of the switches.

PatrickG_SF · ‎10-28-2020

Thank you!

BMR-GOP>enable
BMR-GOP#sh ver
Cisco IOS Software [Denali], Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9- M), Version 16.3.6, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2018 by Cisco Systems, Inc.
Compiled Wed 28-Feb-18 15:23 by mcpre

Cisco IOS-XE software, Copyright (c) 2005-2018 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0. For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.

ROM: IOS-XE ROMMON
BOOTLDR: CAT3K_CAA Boot Loader (CAT3K_CAA-HBOOT-M) Version 3.58, RELEASE SOFTWAR E (P)

BMR-GOP uptime is 1 hour, 59 minutes
Uptime for this control processor is 2 hours, 3 minutes
System returned to ROM by Power Failure at 17:19:47 utc Thu Jun 18 2020
System image file is "flash:packages.conf"
Last reload reason: Power Failure

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

Technology Package License Information:

-----------------------------------------------------------------
Technology-package Technology-package
Current Type Next reboot
------------------------------------------------------------------
lanbasek9 Permanent lanbasek9

cisco WS-C3850-48P (MIPS) processor (revision AC0) with 864936K/6147K bytes of memory.
Processor board ID FOC2222L1SS
1 Virtual Ethernet interface
104 Gigabit Ethernet interfaces
8 Ten Gigabit Ethernet interfaces
2048K bytes of non-volatile configuration memory.
4194304K bytes of physical memory.
252000K bytes of Crash Files at crashinfo:.
252000K bytes of Crash Files at crashinfo-1:.
1611414K bytes of Flash at flash:.
1611414K bytes of Flash at flash-1:.
0K bytes of at webui:.

Base Ethernet MAC Address : 00:45:1d:f9:ce:00
Motherboard Assembly Number : 73-15800-08
Motherboard Serial Number : FOC22221FFA
Model Revision Number : AC0
Motherboard Revision Number : B0
Model Number : WS-C3850-48P
System Serial Number : FOC2222L1SS

Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
1 56 WS-C3850-48P 16.3.6 CAT3K_CAA-UNIVERSALK9 INSTALL
* 2 56 WS-C3850-48P 16.3.6 CAT3K_CAA-UNIVERSALK9 INSTALL

Switch 01
---------
Switch uptime : 2 hours, 3 minutes

Base Ethernet MAC Address : 00:b6:70:96:27:80
Motherboard Assembly Number : 73-15800-08
Motherboard Serial Number : FOC22221F8J
Model Revision Number : AC0
Motherboard Revision Number : B0
Model Number : WS-C3850-48P
System Serial Number : FOC2222L1SK

Configuration register is 0x142 (will be 0x102 at next reload)

Leo Laohoo · ‎10-28-2020

Cool. If this is what I think this is, can you COLD REBOOT (remove the power cables) the entire switch stack?

Let us know if it works or not.

PatrickG_SF · ‎10-28-2020

The power failure in the sh ver I linked above refers to a COLD REBOOT aka bounce that I performed a few hours ago. No change there.

Leo Laohoo · ‎10-28-2020

Just want to confirm the power cables were pulled and did not use the "reload" command. Is this correct?

NOTE: Some IOS-XE versions will (incorrectly) report "power failure" even with the "reboot" command or a systems crash -- This is a known "feature".

PatrickG_SF · ‎10-28-2020

Thanks for the info. Yes, I pulled the cables and waited about 30 seconds and plugged them back in

Leo Laohoo · ‎10-28-2020

Cool.

Post the complete output to the command "sh interface <PORTS>".

I want to see the output to both ports to determine if the CRESTONS are talking or not.

Can you also post the output to the command "sh post"?

PatrickG_SF · ‎10-28-2020

Somewhat difficult to do with all the flapping, so forgive the link up/down feedback in between:

This is for the only Crestron unit I'm troubleshooting, on gi1/0/1

Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
0 packets input, 0 bytes, 0 no buffer
Received 0 broadcasts (0 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
0 packets output, 0 bytes, 0 underruns
--More--
*Oct 29 00:24:31.121: %ILPOWER-7-DETECT: Interface Gi1/0/1: Power Device detected: IEEE PD
*Oct 29 00:24:31.650: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/1: Power granted
*Oct 29 00:24:32.534: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/1: PD removed
*Oct 29 00:24:32.538: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/1: PD removed
*Oct 29 00:24:33.156: %ILPOWER-7-DETECT: Interface Gi1/0/1: Power Device detected: IEEE PD
*Oct 29 00:24:34.534: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/1: PD removed
*Oct 29 00:24:34.538: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/1: PD removed
*Oct 29 00:24:35.155: %ILPOWER-7-DETECT: Interface Gi1/0/1: Power Device detected: IEEE PD
*Oct 29 00:24:36.533: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/1: PD removed
*Oct 29 00:24:36.536: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/1: 0 output errors, 0 collisions, 8 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
BMR-GOP#
*Oct 29 00:24:37.159: %ILPOWER-7-DETECT: Interface Gi1/0/1: Power Device detected: IEEE PD
*Oct 29 00:24:38.535: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/1: PD removed
*Oct 29 00:24:38.539: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/1: PD removed
*Oct 29 00:24:39.156: %ILPOWER-7-DETECT: Interface Gi1/0/1: Power Device detected: IEEE PD
*Oct 29 00:24:40.534: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/1: PD removed
*Oct 29 00:24:40.537: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/1: PD removed
*Oct 29 00:24:41.157: %ILPOWER-7-DETECT: Interface Gi1/0/1: Power Device detected: IEEE PD
*Oct 29 00:24:41.663: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/1: Power granted
*Oct 29 00:24:42.534: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/1: PD removed
*Oct 29 00:24:42.538: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/1: PD removed
*Oct 29 00:24:43.787: %ILPOWER-7-DETECT: Interface Gi1/0/1: Power Device detected: IEEE PD
*Oct 29 00:24:44.531: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/1: Power granted
*Oct 29 00:24:45.534: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/1: PD removed
*Oct 29 00:24:45.536: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/1: PD removed
*Oct 29 00:24:46.531: %ILPOWER-7-DETECT: Interface Gi1/0/1: Power Device detected: IEEE PD
*Oct 29 00:24:47.537: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/1: PD removed
*Oct 29 00:24:47.541: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/1: PD removed
*Oct 29 00:24:48.532: %ILPOWER-7-DETECT: Interface Gi1/0/1: Power Device detected: I

Leo Laohoo · ‎10-28-2020

Log out and log back in and the terminal monitor should go away.
Alternatively, disable the port or shut down the port.

Try the command again.

PatrickG_SF · ‎10-28-2020

Sorry I assumed you wouldn't get the info you are looking for if I shut the port down.

BMR-GOP#sh interface gi1/0/1
GigabitEthernet1/0/1 is administratively down, line protocol is down (disabled)
Hardware is Gigabit Ethernet, address is 00b6.7096.2781 (bia 00b6.7096.2781)
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Auto-duplex, 100Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
0 packets input, 0 bytes, 0 no buffer
Received 0 broadcasts (0 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
0 packets output, 0 bytes, 0 underruns
0 output errors, 0 collisions, 8 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

Leo Laohoo · ‎10-28-2020

No input nor output traffic.

Can you move the connection to the 2nd switch? Any port.

Another thing, do you have any Cisco VoIP phones? If you do, can you plug it to Gi 1/0/1 and see if the behaviour is the same?

PatrickG_SF · ‎10-29-2020

The behavior is the same on both switches and after trying different ports on each. The link flaps and the NIC on the Crestron device never lights up. I will attach the config file here but I'm still at a loss for what is or even could be causing this issue. I am yet to see an err state reported. when powering up. I see 'power granted' and then 'PD removed', over and over.

Leo Laohoo · ‎10-29-2020

Stick a Cisco PoE phone and see if power gets delivered.
Make sure to use the same patch cable.

PatrickG_SF · ‎10-30-2020

Leo, thank you for your continued help. I will be back onsite today, however I do not have a Cisco PoE phone or any other PoE+ device. I am wondering if you saw anything in the config file that might be suspect. I seem to have stumped many people with this issue, and I would like to think its a hardware failure, but I cannot be sure. When I examine the ilpower debug that I posted at the top of the thread it appears the device is requesting 15400W, which if I am not mistaken is not PoE plus, its just regular PoE like the handful of other devices on the network. The power requirements for the device are a little cryptic for me, reporting the following:

IEEE 802.3at Type 2 PoE+ Powered Device

9.4 Watts if using an 802.3at Type2 PSE with LLDP advanced power management

Class 4 (25.5W) if using an 802.3at Type 2 PSE without LLDP

Class 0 (12.95W) if using an 802.3af (or 802.3at Type 1) PSE

However, the requested power says 15400 on the ilpower debug. So, I'm confused what is going on here. Regardless, the switch should be able to provide Class 0-4 without any issues, it is just not a UPOE capable switch.

One other thing, is that all PoE devices are reporting a short on pairs C and D, including the ones that are up and working. So if there is a LLDP negotiation taking places, I'm wondering if it is failing this due to the reported fault. In addition, the fault doesn't make sense, as the cables are working an communicating perfectly well.

Gi1/0/13 auto
Pair A 0 +/- 1 meters N/A Normal
Pair B 0 +/- 1 meters N/A Normal
Pair C 2 +/- 1 meters N/A Short
Pair D 3 +/- 1 meters N/A Short

Any help on this issue will be much appreciated.

Thank you,