NDP not working between 9136 APs on 2.4 and 5GHz, 17.9.4, 17.12.2

[jasonm002]

In both 17.9.4 and 17.12.2 NDP appears to be broken between only C9136I-B and only on 2.4 and 5ghz - 6ghz works fine. This happens whether I set ndp-mode auto, ndp-mode off-channel, or ndp-mode on-channel in the 2.4 and 5ghz RF profiles. The default setting for the RF profiles is ndp-mode auto, which results in all radios on C9136I-B getting set to on-channel NDP. The same is true for CW9166I-B which does not experience this problem. NDP type is currently set to transparent (default) for 2.4GHz, 5GHz, and 6GHz. 

The effect of this for me is that TPC is totally broken on 2.4 and 5GHz on only between neighboring C9136I-B APs. DCA appears to be assigning channels, but maybe not optimally. C9136I-B are discovering 9130AXI-B in a neighboring building as neighbors, but 9136s themselves cannot "hear" each other as neighbors on 2.4 and 5ghz and TPC is completely broken on those bands, resulting in max power levels on every radio. Workaround I'm using for that at the moment is to hard cap 2.4Ghz tx level in the RF profile.

The documentation in the 9800 guide for NDP is out of date, so you have to refer to https://video.cisco.com/detail/video/6249386397001 at around the 14 minute mark for how on-channel NDP is supposed to work. Appears that the 6E APs do support on-channel NDP, but the 9800 config guide is out of date: https://www.cisco.com/c/en/us/td/docs/wireless/controller/9800/17-12/config-guide/b_wl_17_12_cg/m_configuring_ndp-mode_on_access_point.html

Debug neighbor rx/tx/detail from an affected 9136 shows an apparent issue:

Nov 27 12:37:57 ap-p-commons6f28 kernel: [*11/27/2023 12:37:57.9767] chatter: cwx3/rrm_receive :: RRMReceive NDP RX: TA:[24:16:1B:18:DD:18] TLV:1 band 2 channel 37
Nov 27 12:37:57 ap-p-commons6f28 kernel: [*11/27/2023 12:37:57.9767] chatter: cwx3/rrm_receive :: RRMReceive::process_neighbor_pkt_check interface 3 band 2 rrm_band_id 3 rrm_context 0xffffff8bce75ad28 radio 0xffffff8bce540000
Nov 27 12:37:57 ap-p-commons6f28 kernel: [*11/27/2023 12:37:57.9767] chatter: cwx3/rrm_receive :: RRMReceive process_neighbor_pkt_check NDP Rx: [24:16:1B:18:DD:18] TLV length 104
Nov 27 12:37:57 ap-p-commons6f28 kernel: [*11/27/2023 12:37:57.9768] chatter: cwx3/rrm_receive :: RRMReceive Dropping an invalid NDP rx neighbor 24:16:1B:18:DD:18(3) packet

24:16:1B:18:DD:18 is a neighboring 9136.

I do have a TAC case open on this but curious if anyone else with 9136 deployment has noticed this. It does not necessarily cause a problem in all environments to have all the radios at max power but I would consider it a severe issue because it completely breaks TPC which is a foundational feature here.

[marce1000]

          - FYI  : https://bst.cisco.com/bugsearch/bug/CSCwe57613?rfs=qvlogin

 M.

[jasonm002]

Saw CSCwe57613, but no FIPS here so if it happens even when FIPS is not enabled, the bug either has to be rewritten or a new bug filed I suppose.

[eglinsky2012]

Are all the APs in the building/floor on the same controller?

I only have 1 building with 9136s. I have two 9800-80 HA pairs on 17.9.4 with no APSP yet. I realized the APs in the building are scattered across both controllers. There are a few in a large open room on the same pair and they're powering down as they should. But if I look at the neighbors on one that's on pair 1, I do not see the AP on pair 2 as a neighbor, or vice versa.

I'm not sure if this would make a difference, but I do not have both controllers in an RF group together (mobility group, yes, but not RF group, since we have more than 6,000 APs total and that's all a 9800-80 RF leader can handle). I normally make sure all APs in a building (or pair of joined buildings) are on the same controller, which is best practice in general.

[jasonm002]

Mine are all on the same controller. Also the 9136 can "hear" other APs in this situation, just not other 9136 so if you have for example 9130s in close enough proximity TPC would be adjusting levels based on those neighbors instead. 

In your setup it would be harder to diagnose this since you have a split-brain RF group scenario going on. Note that in the 9800 best practices guide the 9800-80 can handle 12,000 APs per RF group so in your case it would probably be a good idea to use the same RF group on both controller pairs and set one of them up to be a static leader, see:

https://www.cisco.com/c/en/us/products/collateral/wireless/catalyst-9800-series-wireless-controllers/guide-c07-743627.html#_Toc126338603

Once you've done that the APs should be able to "hear" each other regardless of what controller they are on, and if the issue exists it should be easier to see.

Also what is your RRM neighbor timeout factor set to? Are you using protected or transparent NDP? I'm at the defaults so, neighbor timeout factor 20 min, transparent NDP. 

I've noticed that sometimes I do have a single 9136 appear in the neighbor list of some other 9136 so it could be the case that a single packet is getting "through" sometimes for whatever reason, so one possible workaround for me may be to temporarily boost the NTF or .. perhaps the issue doesn't happen with Protected NTP mode - haven't tried those things yet.

[jasonm002]

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwj35416

[jasonm002 · Accepted solution]

The problem occurs when you have 6ghz monitor interval set >=3500 because for whatever reason when that is the case the 6ghz radio does on-channel NDP every 20ms and doesn't let the poor 2.4 and 5ghz radios send any due to (apparently) some kind of rate-limit involving on-channel NDP. They filed CSCwj84453 for this one but it's not public yet. For example I set "ap dot11 6ghz rrm monitor measurement 3000" and the issue seems to disappear.