Some APs dropping from new 9800

[jasonmeyer]

I have migrated to a new 9800-40 from an 8510. Recently, at approximately noon every day some APs drop from the 9800.  I did use an AP template to move the access points to the new controller, but I left the old controller in as a secondary controller. I have the old controller off so the APs don't drop for long. I have a case open with TAC but they haven't been the most responsive. I have include a few log files they had requested. AP models are 3702i and 3802i.

One odd thing I have noticed in the AP config, the ones that drop have a very long Controller Association Latency time, usually 4-5 minutes long. Not sure exactly what that means.

[marce1000]

 - Do you mean everything is stable if the old controller is of. It would be advisable to not have both new and old active in the same broadcast domain. I have let the output of show tech wireless processed by : https://cway.cisco.com/wireless-config-analyzer/ , you may want to run that again to have the complete report for yourself. Note that it takes some time , 5 to 10 minutes to complete. Below it the AP message summary report and in attachment the complete report. Take note of all advisories and errors , and as stated run again to have the complete picture :

AP Message Summary

Level	Message
20047	Configuration: AP has join profile name not found in controller, this is either incorrect config, or error in file. APS: AP70e4.22ba.f0a8,AP70e4.22be.afec,AP70e4.22b3.0aa4 Action: Check the AP join profile name is same as one configured in controller
20036	WCAE: Missing AP General configuration. APS: AP70e4.22ba.f0a8,AP70e4.22be.afec,AP70e4.22b3.0aa4 Action: Partially incomplete configuration file, try to capture again, optionally use transfer upload command
20037	WCAE: Missing AP RF slot 0 configuration. APS: AP70e4.22ba.f0a8,AP70e4.22be.afec,AP70e4.22b3.0aa4 Action: Partially incomplete configuration file, try to capture again, optionally use transfer upload command
20038	WCAE: Missing AP RF slot 1 configuration. APS: AP70e4.22ba.f0a8,AP70e4.22be.afec,AP70e4.22b3.0aa4 Action: Partially incomplete configuration file, try to capture again, optionally use transfer upload command
20042	Configuration: AP has Site name not found in controller, this is either incorrect config, or error in file. APS: AP70e4.22b3.0aa4 Action: Check the AP site name is same as one configured in controller
20007	CAPWAP: Invalid primary switch config. APS: More than 10 APs affected, use standalone tool for more details Action: The AP has configured a controller name which is not present in the analyzed config file. This may also indicate an error in the AP configuration.
20017	Syslog: Syslog to broadcast. APS: More than 10 APs affected, use standalone tool for more details Action: AP syslog is set to broadcast destination (default). It is recommended to configure unicast server, for security and ease of troubleshooting. Command: config ap syslog host global
120008	Security: AP local credentials have not been set (on default value). APS: More than 10 APs affected, use standalone tool for more details Action: It is strongly recommended to change the default AP credentials, to a custom username/password. This can be done globally under Wireless/Access Points/Global Configuration
60029	RF: AP shows low coverage (all neighbors < -75 dBm) on 5GHz band. This could affect roaming and be indication of poor RF design or NDP issues. APS: More than 10 APs affected, use standalone tool for more details Action: This message is intented to flag APs that don't have a smooth coverage transition to other APs. This may be result of AP physical placement
60028	RF: AP shows low coverage (all neighbors < -75 dBm) on 2.4GHz band. This could affect roaming and be indication of poor RF design or NDP issues. APS: HS-PressBox,HS-SkiRoom,FH-Gyms,HS-BlackBox,HS-WoodShop,HS-EwingPH,MS-CUSTN Action: None
60027	RF: AP is isolated (no neighbors) on 5GHz band. This could be expected on single AP scenarios, but could be indication of poor RF design or NDP issues. APS: HS-PressBox,HS-SkiRoom,HS-WoodShop Action: None
20048	Radio: AP has radio slots that are operational down. Validate if this is intentional. APS: PV-MediaCenterNorth Action: Confirm the reasons why the radio slot shows as operational down. This could be due to configuration, PoE limits, channel assignament, DFS, etc
60011	RF: AP Cochannel interference above threshold, 2.4 GHz Band. APS: More than 10 APs affected, use standalone tool for more details Action: None
60008	RF: Load Profile Failed, 5GHz Band. APS: More than 10 APs affected, use standalone tool for more details Action: None
60015	RF: AP high channel utilization. Band 5GHz. APS: HS-AdminEast,HS-UpperCommons1 Action: None
60013	RF: AP side channel interference above threshold. APS: More than 10 APs affected, use standalone tool for more details Action: None
60007	RF: Load Profile Failed, 2.4GHz Band. APS: More than 10 APs affected, use standalone tool for more details Action: None
60014	RF: AP high channel utilization. Band 2.4GHz. APS: HS-E103,HS-E105,HS-C129 Action: None
60005	RF: Interference Profile Failed, 2.4GHz Band. APS: More than 10 APs affected, use standalone tool for more details Action: None
60009	RF: Noise Profile Failed, 2.4GHz Band. APS: HS-B119N,HS-C219N,HS-Commons4,HS-A211H,HS-A103N,HS-C202N,HS-C215N,HS-C135.2,HS-B106N,MS-189N Action: None
60012	RF: AP Cochannel interference above threshold, 5 GHz Band. APS: More than 10 APs affected, use standalone tool for more details Action: None

[jasonmeyer]

No the old controller off line. However, it is still in the HA config of the APs. Was thinking of using an AP Template to remove it. I am reviewing your reports now. 

[marce1000]

 - Ok, remember to run https://cway.cisco.com/wireless-config-analyzer/  and use your show tech wireless.txt file as input.

 M.

[jasonmeyer]

Well this is weird. The time when the APs drop has come and gone with out one dropping. I will be sure to use that url to check my work when I can make the tweaks without affecting clients.

[Leo Laohoo]

What firmware is the controller running on?

[jasonmeyer]

17.3.4

[jasonmeyer]

What does this mean? CAPWAP DTLS session closed for AP, cause: DTLS server session error

[marce1000]

 - FYI : https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvw35698

 M.

[jasonmeyer]

Yeah, I saw that too. Mine never had a problem joining, they just drop from the controller for a minute then rejoin and are fine for the rest of the day. I've had both 3700 and 3800 drop. Its moved up to 11am my time now(probably due to Daylight savings). Its like there is some scheduled process running somewhere.

[jasonmeyer]

Gets interesting at line 14. Fails to get the proper site tag and ap join profile after that. 

[Arshad Safrulla]

Hi,

Below are some of my observations and recommendations.

1. Remove Native VLAN under the uplink. It is not recommended to have Native VLAN configured in the WLC side and also in the switch side. Configure a simple trunk port and make sure that you allow only Wireless management and wireless user VLAN's only.

2. Remove SVI's for VLAN's where mdns gateway is not in use.

3. Verify if the trustpoint is set for wireless management by using command "show wireless management trustpoint", if not there reassign it. 

c9800(config)#no wireless management trustpoint
c9800(config)#wireless management trustpoint CISCO_IDEVID_SUDI

Then you can validate the config by "wireless config validate" command in privilage exec mode. 

4. If you have your DHCP option 43 pointing to old WLC, remove it and add the new WLC AP management IP. There is only one dynamic AP manager interface in 9800.

Your AP filters are looking fine, I do not see any issues there. Next time when you post a show tech wireless in a public forum I would recommend you to think twice or sanitize certain information which can be used to exploit.

Also post a RA trace for one of the AP's impacted here. You may use AP MAC address to filter the logs.

[jasonmeyer]

Well after a few months of lack of TAC response, finally starting to get somewhere. My enviroment is a medium size school district. I had two site-tags earlier, one for elementary buildings, one for middle, highschool, and a community center. Both site-tags have less than 500 APs, but now only the APs for the secondary level building where the APs disjoining/rejoining. Interesting. I created another site tag and the high school APs to the new site tag, 181 APs. Now the problem only happens at the high school. The highschool can have anywhere from 3500-5000 clients at a time. The thing is, they all use the same AP Join profile. So not really sure why the difference in Sites.

[Scott Fella]

Wow... few months... that is a long time.  Have you tried to maybe test with 10-20 ap's at the HS using a different tag?  Also not setting the AireOS as a backup?  

[jasonmeyer]

I was thinking about adding 2 new sites, would be representative of 2 wings of the building, and putting APs in them.

I don't have an AireOS backup.