Cisco 9166 access point_wired0 ethernet link up/link down constantly

[DennisNFA]

Running into an issue where about half of our 41 brand new Cisco CW9166I-B access points are not registering on our WLC. When consoled into the APs, we noticed after the initial boot phase, the wired0 ethernet port constantly goes link up/link down. Image is attached. Strangely this issue follows the AP around which leads us to believe it is NOT an issue with the WLC or switch stack. I can take a working AP down, plug in one of the problem ones, and still experience the same problem. The issue follows the APs. So the path the traffic takes is AP > Cisco Catalyst 9300UN switch > Cisco Catalyst 9800-L Controller. We have DHCP option 43 configured on our Bluecat DNS on the DHCP scope for the APs. Since around half of the APs worked no issue, we know that is not our problem.

The APs are on their own vlan and we've oddly been able to plug in a few APs to the user data ports at random desks and somehow, the AP will register with DHCP address (from the incorrect scope of course) and pop up in the WLC. If I move the AP then and plug it back into its respective port in the ceiling, it'll never rejoin into the WLC.

APs have been reset, attempted multiple different ports around the office designated for APs, removed switch interface configurations as well on a port by port basis. Nothing is working and I find it hard to believe I have 18 brand new access points right out of the box to be defective. Cisco support can't even make sense of this all. Has ANYONE ever run into a similar issue?

[bix99]

What sort of latency were you experiencing do you recall? as in ms ping etc.

[OthmanGAMI]

I have 10 APs (C9316I-ROW) with the same issue, and the link up/ down only works once I configure static IP on the AP. Another way is to use a fixed address from the DHCP server (Infoblox or Bluecat) by MAC-Address. However, the AP and WLC must be on the same VLAN.

[JoshBarrett]

This whole thread reads like something I would have tried. We have 35 new APs 9166's and we have serveral that are doing the same thing. Console to AP shows up/down until it reboots, Different switches 9300-48UN, and 9200L's. Different cables doesn't matter. Looks like we'll loading new firmware to the APs because with Cisco there's always a new firmware that MIGHT fix the issse.

[Leo Laohoo]

Is the switch port configured for Dot1x?

[JoshBarrett]

I solved my problem, I made a separate response. Hint DHCP

[JoshBarrett]

I just figured my issue out. TLDR: Windows DHCP server with failover. One of the servers were out of sync and had turned off their scopes. So when my switch with VLAN 34 interface and IP helper picked the broken DHCP server no IP for APs. But it picked the good one sometimes and that is what drove me nuts.

So for all my effort here is my checklist to do. Adjust to fit your configs.
My APs were on a different VLAN than the controller. APs were Vlan 34. Controller Vlan 1. And they are several switch hops away, All layer 2 stuff only the core did the routing and had the IPs with IP helper to the DHCP servers. So starting on the switch the APs were directly connected to I ran the commands 
access-list 100 permit udp any any eq 67
access-list 100 permit udp any any eq 68
then IP debug packet details 100 and waited for the AP to ask for DHCP. I'm all layer 2 so the part about routing failed at the end is fine. I did this to each switch until I got to the core. I also did a shutdown and no shutdown to the AP that worked and didn't work. It wasn't until I got to the core switch that I saw the debug give more info with the IP helpers listed. I also saw how the requests went to the different dhcp servers. I never thought to check the second dhcp server. I'd only ever opened the first one as they were always synced. When I did expand the second server most of the scopes were down and out of sync. I did a replicate failover scopes and it synced and cleaned up and the scopes all came back up. I did a shutdown to all the AP ports and brought them back up and everything is working. Also, I understand why when I put the APs on VLAN 1 they worked before. As VLAN 1 DHCP scopes on both servers were working. But it made me doubt my switch configs. sigh Also run show interfaces trunk to make sure spanning tree isn't cutting off your vlans to the next switch. Also sh vlan brief so you see the vlans as up.