Multiple 9166 APs intermittently running at 100MB

[Timtamarama]

Hey all 🙂

I have multiple CW9166I-Z's that are running at 100mb instead of the full speed 1000mb.
A port bounce only temporarily resolves the issue, and I'm looking for a longer term solution. 

The power they are drawing is 30.0 Watts which is correct.
The AP version is 17.12.6.203

This problem is occurring on various switch types including:
WS-C3560CG-8PC-S running software 15.2(2)E10
C9300LM-48UX running software 17.15.04  
WS-C2960S-48LPD-L running software 15.2(2)E9 

I have attended site to swap the patch lead at both field end and patch port with a CAT6 cable.

I have used the following commands:
conf t
int x/x
power inline never
do test cable-diagnostics tdr intx/x
exit
test cable-diagnostics tdr intx/x

this changes the result from

#show cable-diagnostics tdr int Gi0/8
TDR test last run on: March 05 14:33:08
Interface Speed Local pair Pair length Remote pair Pair status
--------- ----- ---------- ------------------ ----------- --------------------
Gi0/8 1000M Pair A N/A N/A Normal
Pair B N/A N/A Normal
Pair C N/A N/A Normal
Pair D N/A N/A Normal

to 

show cable-diagnostics tdr int Gi0/8
TDR test last run on: January 15 17:09:39
Interface Speed Local pair Pair length Remote pair Pair status
--------- ----- ---------- ------------------ ----------- --------------------
Gi0/8 auto Pair A N/A N/A Normal
Pair B N/A N/A Normal
Pair C 14 +/- 10 meters N/A Open
Pair D 15 +/- 10 meters N/A Short/Crosstalk

I have googled this to no avail...
What other troubleshooting steps could I take from here? Has anyone experienced this issue too? 

First time poster (so thank you in advance for being patient with me) 🙂

[Mark Elsen]

  - @Timtamarama      1)   Check port counters on the switchports for the interfaces  where the access
                                         point(s) are connected (zero the counters first , to have a clear analysis possible
                                         when an access point falls back to 100Mbps)

                                   2)   Check and or post the full boot process of an access point

                                   3)   Enable Syslog Messages in Access Points and Controller according to :
                                                https://www.cisco.com/c/en/us/td/docs/wireless/controller/ewc/17-12/config-guide/ewc_cg_17_12/enabling_syslog_messages_in_access_points_and_controller_for_syslog_server.html
                                       When an access points fall back to 100Mb, check the logs on the syslog server

  M.

[Flavio Miranda]

@Timtamarama 

Dont follow you here, sorry.

What happened for this change? And one more question. Are the interfaces in auto, righ?

#show cable-diagnostics tdr int Gi0/8
TDR test last run on: March 05 14:33:08
Interface Speed Local pair Pair length Remote pair Pair status
--------- ----- ---------- ------------------ ----------- --------------------
Gi0/8 1000M Pair A N/A N/A Normal
Pair B N/A N/A Normal
Pair C N/A N/A Normal
Pair D N/A N/A Normal

to

show cable-diagnostics tdr int Gi0/8
TDR test last run on: January 15 17:09:39
Interface Speed Local pair Pair length Remote pair Pair status
--------- ----- ---------- ------------------ ----------- --------------------
Gi0/8 auto Pair A N/A N/A Normal
Pair B N/A N/A Normal
Pair C 14 +/- 10 meters N/A Open
Pair D 15 +/- 10 meters N/A Short/Crosstalk

[Timtamarama]

Nothing 'happened' to cause these changes, that I am aware of. These were rolled out as a bulk deployment over the course of a year and I'm noticing this same problem occur at multiple sites that use varied types of switching. 
The interfaces are operating at 1000M when working correctly. An unknown variant is causing them to drop to 100M.

[Hafthor Hilmarsson O'Connor]

I have seen this also with a few partner and it has always been problems with cables, either length or quality, i.e. we do the Cable test and it shows up ok, but after that AP has gone to speed 1GB, which it was previous 100mb.

We have replaced these patches and or cables, and solved the problem.

[Timtamarama]

Patch lead has been replaced from the switch to the patch panel, and from the field outlet to the AP.
This has been replaced with the same category/length lead at both ends.

There was one switch where this affected 3x APs. Each AP was replaced with a different make/length of patch lead at both ends. Commscope 2m CAT6A , Kronix 1m CAT6,  and Panduit 0.5m CAT6A. 1day later it affected 2x different APs on the same switch. 
Changing the patch lead at either end hasn't fixed the problem, but I'm glad it worked for you. 

[Leo Laohoo · Accepted solution]

This is a known bug feature.  

This is not a hardware fault but a software bug which Cisco will not be fixing.  

[Rich R]

TDR cable test on IOS-XE (9300) is unreliable.  Cisco blame this on Broadcom and no indication that they intend to fix it.
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwd97177 was opened for us - you can see the date it was created!
There were a few other bugs that might be similarly implicated.

These problems are usually a cable or termination quality issue so that's what to focus on.  Also try using different cable testers - they won't all give the same result.

Make sure you do not have any unterminated console cable plugged in or any console cable longer than 1m - can act as antenna causing RF interference for AP and even prevent booting on reload.