C9136 XOR Sniffer mode crash

[mrTeigen]

Hi,

I am using 17.8.1 (9800CL) with two C9136 APs. I am creating this discussion in hopes to find other people with the same problem. My problem is that the sniffer mode works for a while, but then out of the blue it stops working and you have to reset the AP. I have yet to find a solution, but hopefully this thread will give us the answer in the future.

You can read more about XOR sniffer mode here: Information About XOR Radio Role Sniffer Support

This new solution is very cool, you do not have to reload the AP and put it into sniffer mode (you can, if you like). Now you can enable sniffer on one of the radios. For example 6Ghz like I do on the picture to the left. 

This works after you have upgraded the WLC, but if you boot the WLC it will stop working. Boot it again, same problem. If you bring up a new 9800CL with 17.8.1 it sometimes work. It is indeed very random.

I have added the logs from the WLC below, and some logs from the AP. This will go into a loop until you fix it.

Logs from the WLC

Apr 14 18:10:44.595: %CPPOSLIB-3-ERROR_NOTIFY: Chassis 1 F0/0: cpp_cp_svr: cpp_cp encountered an error -Traceback= 1#51a20a49236e2a39e512540b1a082814 errmsg:7FB1CDF4D000+E9A cpp_common_os_ledp:7FB1D3A23000+1E40C cpp_common_os_ledp:7FB1D3A23000+1419E cpp_wireless_svr_lib_ledp:7FB1D42E5000+59395 cpp_wireless_svr_lib_ledp:7FB1D42E5000+5FEB5 cpp_wireless_svr_smc_lib_ledp:7FB1D42BA000+1839D cpp_common_os_ledp:7FB1D3A23000+22290 cpp_common_os_ledp:7FB1D3A23000+22983 evlib:7FB1D1560000+8D37 evlib:7FB1D1560000+9A90 cpp_common_os_ledp:7FB1D3A23

Apr 14 18:10:44.595: %FMFP-3-OBJ_DWNLD_TO_DP_FAILED: Chassis 1 F0/0: fman_fp_image: WLClient 0x9000000d download to DP failed

Apr 14 18:10:44.596: %APMGR_TRACE_MESSAGE-3-WLC_GEN_ERR: Chassis 1 R0/0: wncd: Error in 687d.b460.33f0Ap will be disconnected. Sniffer Client data path plumb failed

 Logs from the AP

Apr 14 18:20:33 kernel: [*04/14/2022 18:20:33.7388] Flexconnect Switching to Standalone Mode!
Apr 14 18:20:33 kernel: [*04/14/2022 18:20:33.7953] GOING BACK TO DISCOVER MODE
Apr 14 18:20:33 kernel: [*04/14/2022 18:20:33.8153] OOBImageDnld: OOBImageDownloadTimer expired for image download..
Apr 14 18:20:33 kernel: [*04/14/2022 18:20:33.8153] OOBImageDnld: Do common error handler for OOB image download..
Apr 14 18:20:33 kernel: [*04/14/2022 18:20:33.8414] 
Apr 14 18:20:33 kernel: [*04/14/2022 18:20:33.8415] CAPWAP State: DTLS Teardown
Apr 14 18:20:33 kernel: [*04/14/2022 18:20:33.8704] OOBImageDnld: Do common error handler for OOB image download..
Apr 14 18:20:33 upgrade: Script called with args:[CANCEL]
Apr 14 18:20:33 kernel: [*04/14/2022 18:20:33.9715] status 'upgrade.sh: Script called with args:[CANCEL]'
Apr 14 18:20:34 kernel: [*04/14/2022 18:20:34.0056] do CANCEL, part1 is active part
Apr 14 18:20:34 upgrade: Cleanup tmp files ...
Apr 14 18:20:34 kernel: [*04/14/2022 18:20:34.0304] status 'upgrade.sh: Cleanup tmp files ...'
Apr 14 18:20:38 kernel: [*04/14/2022 18:20:38.5089] OOBImageDnld: OOBImageDownloadTimer expired for image download..
Apr 14 18:20:38 kernel: [*04/14/2022 18:20:38.5089] OOBImageDnld: Do common error handler for OOB image download..
Apr 14 18:20:38 kernel: [*04/14/2022 18:20:38.5302] dtls_queue_first: Nothing to extract!

How to fix

This is do not know yet, but I will update this discussion when I know more. But, to get your AP up and running again like a normal AP. You can reset it, or even better (easier). SSH to it (if you enabled ssh to the AP Join Profile before crashing it) and just type C9136-AP01#ap-type capwap

When this is fixed, I can create a discussion on how to do it, and I can create a fun little 6Ghz capture post instead.

Do someone know what all these logs mean?

[Rich R]

The best you can hope for in discussing this with others here is a workaround for recovering but this is clearly a software bug so the only fix for it will be in IOS-XE and the only way that will happen is if you open a TAC case with Cisco.  They'll open a bug for it and pass to dev team to work on a fix.  They should also be able to decode any relevant error logs.

17.8.1 is a limited support release so any fixes will only be in 17.9.x and that's at least a few months away from being released.  Getting things like this fixed can also take months/years sometimes.  If you work with TAC and are willing to test engineering special builds for them you could help speed that up.

[mrTeigen · Accepted solution]

Hi,

Thank you for replying. You are absolutely right regarding creating a TAC, but since this is my home lab I feel a bit weird creating a TAC. I had the 17.8.1 EFT to test, and the bug was there also. I am pretty excited about 17.9.x since that will most likely be a long-lived train with several MRs planned. I guess most C9136 fun stuff will be in that one, and 17.8.x is for testing stuff maybe. I am willing to test engineering special builds all day, since this is my home lab that I also use to connect my work computer (because I live dangerous, lol).

In other news, I found a fix. Reload the WLC

I have worked with troubleshooting stuff all my life, and I need to hit myself in the face. Reload worked. This time at least.

[Rich R]

Glad to hear you found a "workaround" LOL

Good luck with the TAC case ...

[mrTeigen]

Thank you! LOL. I also think I found out how I made this bug to trigger.

1. Enable sniffer on one of the radios
2. Done sniffing? set the radio role to Auto?
   1. well, Auto when the last you did was sniffing will still be sniffing. It also say so (Auto - Sniffer)
3. Crash or just reload the WLC. (I crashed it, different bug)
   1. If you reload the WLC when the AP is in sniffer mode, it will not work the next time.
4. Set the radio role to Client Serving, then to Auto. Then you can reload the WLC or crash it as much as you want.

[kittikhunw]

I have the same problem on wlc 9800-40 version 17.9.3 and access point 9164. We were able to fix it by installing ap service pack 17.9.3 from below without reload WLC.

https://software.cisco.com/download/home/286316412/type/286325254/release/17.9.3

Hope it can help someone who faces the same problem. Have a good day !

[Rich R]

Was that APSP1 or APSP3 or both?