PoE Issues with C9300L-24P and C9200CX-12P

[bgillies95]

Hi guys,

I have an issue with a customer's AP. It is a newly refreshed site with a mix of 9300L-24P and 9200CX-12P switches. At the moment, 3 of their APs are inconsistently powering up. These are brand new 9124AXE APs, the switches are brand new too. The symptom we are seeing is when an AP is plugged into a port on the switch, the AP is not detected by the switch and is therefore not powering on. Sometimes if we bounce the port a number of times it will eventually come up. In some cases we have left it for 30 minutes and then the AP powers on suddenly, however it may stop sending power after a few minutes or an hour. The best uptime we got from them was around 20+ days.

We organised an RMA for these 3 APs, and they have been installed. After replacement, we are still having the same issue. We got a 3rd party cabling team to perform end-to-end testing of the structured cabling (newly installed too - CAT6) and all tests were successful. Finally I managed to find a spare 9300L-24P switch and use that instead, however we have the same issue. The switch is currently running 17.6.4, however I have used 17.6.5 and 17.9.4 and all have the same issues. The best consistency we got with the APs, is when we ran an additional 40m patch on top of the structured cabling, bringing the total cable length to somewhere around 85-90 metres. After doing so, I could bounce the port 10 times in a row and the AP would come back up. Obviously this isn't a solution, but it does give some insight to the problem. I've tried literally everything from setting static power to 2-event mode and other PoE port settings. I got a 3rd party power injector and plugged it into the AP, which powers it on.

In a separate case at the same site, one of our 9124AXE APs does not power on the production C9200CX-12P switch as well as a new spare C9200CX-12P that we had. However, when I plug that same AP using the same cabling etc. into a C9300L-24P the AP comes up straight away.

Has anyone come across an issue like this???

[marce1000]

 - Make sure to enable LLDP on the switchport(s) where the 9124AXE APs are connected ; when an AP can boot up ; immediately after boot connect to it and issue the command show logging ; look for errors related to powering the AP , if any , 

 M.

[bgillies95]

Thanks, LLDP has been enabled previously just in case the AP was sending LLDP TLVs.

Here is the log output in the buffer of when the AP came up

1603612: Nov 11 08:39:17.158: %ILPOWER-5-DETECT: Interface Gi1/0/23: Power Device detected: IEEE PD
1603613: Nov 11 08:39:21.158: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/23: Power granted
1603616: Nov 11 08:39:38.497: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/23, changed state to up
1603618: Nov 11 08:39:40.500: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/23, changed state to up
1603630: Nov 11 08:40:47.114: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/23, changed state to down
1603633: Nov 11 08:40:48.116: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/23, changed state to down
1603635: Nov 11 08:40:50.691: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/23, changed state to up
1603636: Nov 11 08:40:52.694: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/23, changed state to up
1603640: Nov 11 08:41:22.460: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/23, changed state to down
1603641: Nov 11 08:41:23.466: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/23, changed state to down
1603642: Nov 11 08:41:25.990: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/23, changed state to up
1603643: Nov 11 08:41:27.992: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/23, changed state to up

All looks pretty standard. There are only a couple of PoE devices on the production switch. (CCTV cameras)

Sometimes we get the following error line on both the new spare 9300L and production 9300L

%ILPOWER-3-CONTROLLER_PORT_ERR: Controller port error, Interface Gi1/0/23: Power Controller reports power Imax error detected

[marce1000]

 - Checkout  https://bst.cloudapps.cisco.com/bugsearch?pf=prdNm&kw=Power%20Controller%20reports%20power%20Imax%20error&bt=custV&sb=anfr&prdNam=Cisco%20Catalyst%209300L-24P-4X-A%20Switch
               and or use latest advisory software version on the involved switch models (too) ; check if that can help , 

 M.

[bgillies95]

I found this bug, https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwd78924

However it seems to only be related to Class 1-3 PDs. I have seen similar described bugs as well, so I wouldn't be surprised if it extended to our 9124s which are a class 4. The affected versions are unfortunately the ones I have tried with! I will see if I can test with other software versions.

[Leo Laohoo]

Do the following: 

conf t
 interface Gi1/0/23
 power inline never
end
test cable tdr interface Gi1/0/23

Wait for approximately 20 seconds.  And then do the next command below: 

show cable tdr interface Gi1/0/23

Post the output to the command above.  

After concluding the test, re-enable PoE on the port. 

conf t
 interface Gi1/0/23
 no power inline never
end

[bgillies95]

TDR test last run on: November 12 12:44:22

Interface Speed Local pair Pair length Remote pair Pair status
--------- ----- ---------- ------------------ ----------- --------------------
Gi1/0/23 auto Pair A 0 +/- 1 meters N/A Normal
              Pair B 57 +/- 1 meters N/A Open
              Pair C 0 +/- 1 meters N/A Normal
              Pair D 0 +/- 1 meters N/A Normal

The length under Pair B is accurate. 3rd party cable testers picked up the cable length as 56 meters in their test.

Interestingly enough, when I looked on the switch before running the cable test above it was saying this AP is a class 0 PD. For whatever reason, the switch was not able to detect the PD class.

Gi1/0/23  auto   on         30.0    C9124AXE-Z          0     30.0
Gi1/0/24  auto   on         30.0    C9124AXE-Z          4     30.0 

Even though debug logs showed that the switch could pick up the AP

3169835: Nov 12 12:42:43.514: Ilpower interface (Gi1/0/23) process tlv from cdp INPUT:
3169836: Nov 12 12:42:43.514: (Gi1/0/23)power_consumption(mW) = 30000, power_request_id = 17664, power_man_id = 9,
3169837: Nov 12 12:42:43.514: (Gi1/0/23)power_request_level(mW) = 30000 15400
3169839: Nov 12 12:42:43.514: Ilpower interface (Gi1/0/23) power negotiation: consumption = 30000, alloc_power = 30000
3170152: Nov 12 12:42:51.995: Gi1/0/23 LLDP GET TLV
3170153: Nov 12 12:42:51.995: ILP get power-via-MDI TLV for IDB: Sending LLDP TLV on Gi1/0/23
3170154: Nov 12 12:42:51.995: (Gi1/0/23) Print LLDP power via-MDI TLV:
3170155: Nov 12 12:42:51.995: Gi1/0/23 LLDP GET TLV
3170156: Nov 12 12:42:51.995: ILP get power-via-MDI TLV for IDB: PSE Sending 4-wire LLDP TLV on Gi1/0/23
3170157: Nov 12 12:42:51.995: (Gi1/0/23)Print four wire power via-MDI TLV: FOUR WIRE via mdi class tlv: 

[Leo Laohoo]

---

@bgillies95 wrote:

              Pair B 57 +/- 1 meters N/A Open​

---

It is a cable issue nearest the AP.  Get the cablers to replace the cable first.  If the issue persist, inspect the keystone jack.  

---

@bgillies95 wrote:
The length under Pair B is accurate.

---

Run the TDR again with PoE turned on.  Observe the discrepancy.  

[bgillies95]

Cabling has been tested completely end-to-end with a Fluke cable tester. I saw the results for myself and they were good. Keystone jack was also replaced during this process and re-terminated. Cable tested prior to being terminated to the keystone jack. Tested after keystone jack was terminated and then finally tested with a new 1.5m patch lead running from the keystone jack. All 3 tests were successful.

TDR test last run on: November 12 18:01:52

Interface   Speed Local pair Pair length        Remote pair Pair status
---------   ----- ---------- ------------------ ----------- --------------------
Gi1/0/23    1000M Pair A     55   +/- 10 meters Pair A      Normal
                  Pair B     55   +/- 10 meters Pair B      Normal
                  Pair C     56   +/- 10 meters Pair C      Normal
                  Pair D     55   +/- 10 meters Pair D      Normal  

This is also the cable test for Gi1/0/24 which is having the same issue:

TDR test last run on: November 12 18:01:22

Interface   Speed Local pair Pair length        Remote pair Pair status
---------   ----- ---------- ------------------ ----------- --------------------
Gi1/0/24    1000M Pair A     62   +/- 10 meters Pair A      Normal
                  Pair B     62   +/- 10 meters Pair B      Normal
                  Pair C     62   +/- 10 meters Pair C      Normal
                  Pair D     62   +/- 10 meters Pair D      Normal  

[bgillies95]

In a similar issue with a 9200CX-12P at the same site, when we connect the 9124XE AP to the 9200CX it has the same issue. Switch doesn't detect the AP and it won't power on. However, when I take the same 9300L we were testing with as discussed above, the AP powers on straight away. Same patch cable and everything. Only difference in this scenario is the switch model being used.

Cable diagnostics for this test below.

TDR test last run on: November 12 18:26:02

Interface   Speed Local pair Pair length        Remote pair Pair status
---------   ----- ---------- ------------------ ----------- --------------------
Gi1/0/1     1000M Pair A     47   +/- 10 meters Pair A      Normal
                  Pair B     47   +/- 10 meters Pair B      Normal
                  Pair C     47   +/- 10 meters Pair C      Normal
                  Pair D     47   +/- 10 meters Pair D      Normal    

 This cable was also tested a few weeks ago and was successful.

[Leo Laohoo]

---

@bgillies95 wrote:
when we connect the 9124XE AP to the 9200CX it has the same issue. Switch doesn't detect the AP and it won't power on.

---

What firmware is the 9200CX on?  

Were there any PD devices directly adjacent to the AP port?  

Were there any error messages appearing for the AP port?

[bgillies95]

Tested with 17.9.1 and 17.12.1 with same results.

Not at the moment, but at the time of the wireless migration for the customer we had one of their old Cisco 1240APs on port 9 and the new 9124 was on port 12. At the moment they have a Polycom conference phone and a couple of IP phones.

We did get the following error message, but I have not been able to replicate it as of late.

%ILPOWER-3-CONTROLLER_PORT_ERR: Controller port error, Interface Gi1/0/12: Power given, but State Machine Power Good wait timer timed out

[Leo Laohoo]

---

@bgillies95 wrote:
Tested with 17.9.1 and 17.12.1 with same results.

---

Upgrade first to 17.9.4a.  If it does not work, downgrade to 17.9.3 to test.  Do not use the "reload" command.  Instead cold-reboot (pull the power cable).

---

@bgillies95 wrote:
Not at the moment, but at the time of the wireless migration for the customer we had one of their old Cisco 1240APs on port 9 and the new 9124 was on port 12. At the moment they have a Polycom conference phone and a couple of IP phones.

---

Try to plug the AP and only the AP.  

LLDP is turned on?

[Rich R]

Use CDP or LLDP - never both!

[Rich R]

Can you also share the complete switch port config?