Can AP monitor mode help determine cause for over-active 5G channel changing?

[AllanGrah]

I am seeing in my AP logs and the controller SNMP logs that my 5Ghz channel is changing a lot the past few weeks. (with rising complaints of disconnects)   This is occurring on what seems to be almost every AP on this small campus of 6 buildings, using 2 WISM2 controllers supporting maybe 300 or so 3702 AP's at this campus.
I've disabled ED-RRM but the situation persisted.   The DCA is (was) at it's default setting of 10 minutes so I went ahead and changed that to 12 hours which I'm told is Cisco Best Practice.
Over the next 12 hours or so, I did not have a single 5Ghz radio reset or channel change.   BUT, I don't want a band-aid, I want to know what was causing the channel-changes.  (new radar in the area, or other 5Ghz interference, etc)

Can I use an AP in Monitor Mode to help determine whether something is blasting 5Ghz that might somehow be affecting us all of a sudden.   I'm at a static, 1 business campus with no recent infrastructure or code changes.  (8.2.166)    This campus has been very very static for 10+ yrs with no significant business neighbors.   Pretty much in a space by ourselves on good acreage.

Thanks in advance.......

[Haydn Andrews]

Hi Allan

Are you using DFS channels? Is it changing on all channels or just the DFS channels? Are there any logs in Prime or on the WLC showing DFS events?

You could use the AP in monitor mode, this will mean the AP is scanning continuously, where as the normal APs will be doing off channel scanning. So not going to get you more information, more going to give you a higher chance of catching the cause.

TAC recommends 8.2.170.0 as the recommended version for the 8.2.x train

https://www.cisco.com/c/en/us/support/docs/wireless/wireless-lan-controller-software/200046-tac-recommended-aireos.html 

The other thing you could do is look at a spectrum and passive survey conducted in the environment to see if its a non WiFi interfere causing the issues, or if neighbors have moved in and causing issues

[AllanGrah]

Thank you for your reply.
We do use the DFS channels 52-64, but not the UNI-II Extended channels 100-140.
Thanks for stating that the scanning is occurring on other channels that the AP isn't currently using, even during normal operation.    Sounds like I'd have no real gain in using Monitor Mode then since my channels are changing often. :-/

The output from the debug Airewave Radar and debug Airewave Channel indicates co-channel interference.  
I had approximately 80%-95% of the AP's on this 6 Building campus all change channels at the same time during the debug.   It is across 3 diff. Channel groups.

*RRM-CLNT-5_0: Mar 15 08:09:09.671: a8:9d:21:46:1a:20 Airewave Director: set channel on cell A8:9D:21:46:1A:20(1) to 36 from 40 to reduce co-channel interference from (-86,-120,127) to (-86,-121,-68) reason 1
*RRM-CLNT-5_0: Mar 15 08:09:09.683: 1c:6a:7a:8e:14:80 Airewave Director: set channel on cell 1C:6A:7A:8E:14:80(1) to 52 from 157 to reduce co-channel interference from (-86,-120,-45) to (-86,-120,-62) reason 1
*RRM-CLNT-5_0: Mar 15 08:09:09.715: 1c:6a:7a:5e:24:70 Airewave Director: set channel on cell 1C:6A:7A:5E:24:70(1) to 149 from 64 to reduce co-channel interference from (-128,-128,-65) to (-128,-128,-71) reason 1

It does seem that DCA is causing this, but I'd like to find the root cause since it is occurring across so many AP's at once, and because this remote location has been static for so long.

(PS. I had looked into that code level and noted it was primarily due to some FCC frequency adjustments.  I think in the EU.   Good eye though, good catch)

[Leo Laohoo]

If the APs are 3702 then the cause can be seen in the CleanAir report of the controller.

[AllanGrah]

I was hoping that was the case, but when I look at the AP, and controller and Prime CleanAir/interference reports I originally not really seeing anything significant.  FWIW - TAC told me that some 5G signal interference noted by DCA might not show up on these reports.   I have a tough time believing that, hence my thought to try Monitor Mode, and not contract a $3k-$5k site survey just to see if there's radar. :)

BUT deeper dives do seem to be showing something perhaps.
On the controller, I was checking under Monitor>Cisco CleanAir>802.11a/n/ac and 'Interference devices'.     Also selected a particular AP where I know the channel changes an radio resets were occurring, and found the AP under 'Monitor', then Access Points>Radios>802.11a/n/ac and used the little blue pulldown menu on the far right to select CleanAir and at least this time I am seeing significant non-wifi channel utilization of 35%-60%.but the Interference power is pretty low at -88db

We ran some controller 'debug airewave channel' and 'radar' last night and I did see this show up a lot which seemed concerning to me:

• *RRM-CLNT-5_0: Mar 14 16:49:56.391: 1c:6a:7a:8e:16:90 Checking Flex DFS channels (48/44/36/40/0/0/0/0) width/label 9/80
• *RRM-CLNT-5_0: Mar 14 16:49:56.391: 1c:6a:7a:8e:16:90 Flex DFS: radar in all sub channels (bit 0)
•  
• *RRM-CLNT-5_0: Mar 14 16:49:56.587: 1c:6a:7a:5b:d4:40 Checking Flex DFS channels (52/56/60/64/0/0/0/0) width/label 6/80
• *RRM-CLNT-5_0: Mar 14 16:49:56.587: 1c:6a:7a:5b:d4:40 Flex DFS: radar in all sub channels (bit 0)
•  
• *RRM-CLNT-5_0: Mar 14 16:50:01.787: a8:9d:21:54:a2:40 Checking Flex DFS channels (153/149/157/161/0/0/0/0) width/label 7/80
• *RRM-CLNT-5_0: Mar 14 16:50:01.787: a8:9d:21:54:a2:40 Flex DFS: radar in all sub channels (bit 0)

These 3 channel-sets are all that we are running.   The U-NII-2C channels had never been set for some reason.

[patoberli · Accepted solution]

You have interference on your 5 GHz network, that causes the AP to switch.
Also from your logs "to reduce co-channel interference", that means there is something creating an interference. So I suspect a spectrum survey would be the best to discover where this comes from.

[AllanGrah]

It seems that the interference is affecting most AP's at the same time. 
I dont think I can get approval for the $3k-$10k to site survey the entire campus of 6 buildings.
I am thinking I might open up the UNII-2 Extended channels since it seems we now have this signal interference on our standard UNII-2 channels. 
I pulled this from my local AP which is using channel 60/64.    I'll try to run the same on other AP's using other channels to see if they are cleaner.  I think these RF graphs are pretty neat, wish I had a more flexible tool for it though.
This is channel 60 on left/64 on right.   Note the spectrum color band.  I set to -75db to -100db.   There's some yellow/red showing up there in the -75db to -85db range. :-/.  Bunch of light Blue in the -90db range.
I was expecting the interference to be a little more constant but this is a bit more sporadic or peaky and not very well patterned IMO.

[Leo Laohoo]

Don't use UNII-2E unless you find out how dirty (or clean) UNII-3.

[AllanGrah]

Roger that!   Luckily I saw your reply with just minutes left in my Change Window here tonight. :)

[patoberli]

What is the channel load of those APs when it happens?
Also which channels do you have enabled and which channel width?

[AllanGrah]

In general, the load on the couple of AP's I've focused on has been very low,  like 10% or less.
Unfortunately the issue is very sporadic and I've not even seen spikes of load that correlate to the 5Ghz channel drops and 5Ghz radio reset.    Oddly,  there wasn't a single occurrence yesterday, despite previous days having multiple issues in a full 24hr day.

FWIW - my nearest building neighbors are 500+ feet away.

For many years now, we've used 5Ghz channels UNII-1, UNII-2 and UNII-3. (36, 40, 44, 48, 52, 56, 60, 64, 149, 153, 157, 161, 165)
We have 2 controllers for this campus  (3 buildings ea) and one was at 20Mhz width and the other was at 40Mhz width.    They both exhibited the issue initially.  The 20Mhz setting is an oversight, and I made it 40Mhz during the past week to make it Standard.  Yeah, I do realize that may sound like going against the grain to a degree but it is where we want to be ideally.  I know I may have to back them both down to 20Mhz if I have new interference in a bunch of channels which one would think is unexpected.   New interference to an established setup would 'generally' be expected to be just a small number of channels.

I'm trying to run Cisco Spectrum Analyzer on multiple AP's across campus, picking AP's using different channels.
I noticed a strong interference on Channel 36 on one AP, and as in the chart I posted, some in Channel 60/64 also so last night I removed those channels from our DCA list.    I did NOT open up the UNII-2 Extended channels yet per the previous advice given.   I guess that decision will rest up on what interference I find across these other channels?

[patoberli]

It is very weird that you have nearly no foreign devices on 5 GHz, but alarms and channel changes because of noise on those channels. That might mean you have somebody using devices like wireless cameras (which don't use Wi-Fi) or similar devices. You might want to physically inspect an area where this happens more often.
Oh, do you have inhouse mobile phone (mostly LTE) amplifiers? They also send near the 5 GHz range and if you don't have at least 1 meter separation between the two antennas you also have weird issues like this.
If you can spend a little money, look at these two notebook solutions:
https://www.metageek.com/products/wi-spy/
https://www.ekahau.com/products/spectrum-analyzer/overview/

Btw. I'm using a building with thick walls and using Best for the channels. I can use surprisingly often 80 Mhz, without a lot of co-channel interference.

[AllanGrah]

Thanks for your input, appreciate everyone's take on this issue.

We don't have any LTE amplifiers here or nearby but the wireless camera might be a factor.  There's a couple of thousand people here and you just never know what someone might do.  (like bring in their own switch/router or MiFi device!)
I will try to look at the logs to see which AP's were the first to do the channel-change.   (have about 155ish that logged it)

I want that Ekahau unit.  Saw that at Cisco Live and keep mentioning it to our management.   Tough sell here, but I won't give up.

Patoberly:  Did you open up all the UNII channels/groups to be able to run the 80Mhz bonding?  Just curious.

[patoberli]

Yes, I have all possible channels enabled here (36, 40, 44, 48, 52, 56, 60, 64, 100, 104, 108, 112, 116, 120, 124, 128, 132, 136, 140). Living in Europe, clients don't have any problem.

[AllanGrah]

Replying to my own post for others' future reference.

The debug snippet above about "Radar in all sub-channels" is NOT a concern.  Seems the "Bit 0" is indicative of a "False" reading.    If you see a 'Bit 128', then that would be a bad indication per Cisco.