topic Re: 9800-L-F: APs will not register in Wireless

9800-L-F: APs will not register

perrymcgrew — Wed, 19 Apr 2023 19:02:34 GMT

I have had case open for about a month on this and at least 3 different TAC engineers. I have 9800L-F set up in HA-SSO (RP+RMI). We have tried 17.9.x, 17.10,1 and now are on 17.11.1.

I have set the AP Mgt as VLAN 2 (10.0.0.0 /22) and the 9800L Management Interface at 10.0.3.253. I have 3560X switch set up in my office "lab" with 3 APs (two 9115AXi and 1 2802i) connected as well as the Primary 9800L chassis. The APs will not connect -- or if they do, it takes days. We have done numerous traces / debugs / packet captures and TAC still cannot explain why. I am hoping a fresh set of eyes can see what the problem is and how to fix.

The AP console session repeats this:

[*04/19/2023 14:36:12.8578] CAPWAP State: Discovery
[*04/19/2023 14:36:12.8808] Discovery Request sent to 10.0.3.253, discovery type STATIC_CONFIG(1)
[*04/19/2023 14:36:12.8818] Discovery Request sent to 10.0.3.253, discovery type STATIC_CONFIG(1)
[*04/19/2023 14:36:12.8828] Discovery Request sent to 255.255.255.255, discovery type UNKNOWN(0)
[*04/19/2023 14:36:12.9098] Discovery Response from 10.0.3.253
[*04/19/2023 14:36:22.2708] Started wait dtls timer (60 sec)
[*04/19/2023 14:36:22.2778]
[*04/19/2023 14:36:22.2778] CAPWAP State: DTLS Setup
[*04/19/2023 14:36:37.3428] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: DTLS Setup(3).
[*04/19/2023 14:37:19.3018] OOBImageDnld: OOBImageDownloadTimer expired for image download..
[*04/19/2023 14:37:19.3018] OOBImageDnld: Do common error handler for OOB image download..
[*04/19/2023 14:37:19.3288]
[*04/19/2023 14:37:19.3288] CAPWAP State: DTLS Teardown
[*04/19/2023 14:37:19.3778] OOBImageDnld: Do common error handler for OOB image download..
[*04/19/2023 14:37:19.4628] status 'upgrade.sh: Script called with args:[CANCEL]'
[*04/19/2023 14:37:19.5058] do CANCEL, part1 is active part
[*04/19/2023 14:37:19.5228] status 'upgrade.sh: Cleanup tmp files ...'
[*04/19/2023 14:37:19.5488] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: DTLS Teardown(4).
[*04/19/2023 14:37:19.5488] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: DTLS Teardown(4).
[*04/19/2023 14:37:24.0528] OOBImageDnld: OOBImageDownloadTimer expired for image download..
[*04/19/2023 14:37:24.0528] OOBImageDnld: Do common error handler for OOB image download..
[*04/19/2023 14:37:24.0728] No more AP manager addresses remain..
[*04/19/2023 14:37:24.0728] No valid AP manager found for controller 'CUN-WLC-9800LF' (ip: 10.0.3.253)
[*04/19/2023 14:37:24.0728] Failed to join controller CUN-WLC-9800LF.
[*04/19/2023 14:37:24.0728] Failed to join controller.

(TAC set a static IP on this AP of 10.0.2.1 /22 for a test. The other test APs use DHCP and have the same console messages)

The core where VLAN 2 is defined:

ip dhcp pool 9800_WLC_MGT
network 10.0.0.0 255.255.252.0
default-router 10.0.3.254
domain-name xxxxx.yyy
option 43 hex f104.0a00.03fd
dns-server 192.168.8.1 192.168.8.2

interface Vlan2
description 9800-WiFi_Mgt Subnet
ip address 10.0.3.254 255.255.252.0
no ip redirects
no ip unreachables
no ip proxy-arp

The 9800L Interfaces:

!
interface Port-channel10
description WLC AP MGMT PORTS
switchport mode trunk

!
interface TenGigabitEthernet0/1/0
description WLC AP MGMT PORT
switchport mode trunk
no negotiation auto
channel-group 10 mode on
service-policy output AutoQos-4.0-wlan-Port-Output-Policy

interface TenGigabitEthernet0/1/1
description WLC AP MGMT PORT
switchport mode trunk
no negotiation auto
channel-group 10 mode on
service-policy output AutoQos-4.0-wlan-Port-Output-Policy

CUN-WLC-9800LF#show wireless interface summ

Wireless Interface Summary

Interface Name Interface Type VLAN ID IP Address IP Netmask NAT-IP Address MAC Address
--------------------------------------------------------------------------------------------------
Vlan2 Management 2 10.0.3.253 255.255.252.0 0.0.0.0 8c1e.xxxx.yyyy

CUN-WLC-9800LF#show wireless management trustpoint
Trustpoint Name : ewlc-tp1
Certificate Info : Available
Certificate Type : SSC
Certificate Hash : 9a80a68f45b442770a4567d4xxxxxxxxxxxxxx
Private key Info : Available
FIPS suitability : Not Applicable

The 3560X switch that the 3 APs and Primary 9800L are connected to:

interface Port-channel10
description ** EtherChan to CUN-WLC-9800LF **
switchport trunk encapsulation dot1q
switchport mode trunk
spanning-tree portfast edge trunk

interface GigabitEthernet0/47
description CUN-WLC-9800LF LAG
switchport trunk encapsulation dot1q
switchport mode trunk
channel-group 10 mode on

interface GigabitEthernet0/48
description CUN-WLC-9800LF LAG
switchport trunk encapsulation dot1q
switchport mode trunk
channel-group 10 mode on

The AP connected ports:

interface GigabitEthernet0/1
description ** TEST AP PORTS **
switchport access vlan 2
switchport mode access
spanning-tree portfast edge

There is no VLAN pruning in place on trunk ports.

TIA - Perry

Re: 9800-L-F: APs will not register

Scott Fella — Wed, 19 Apr 2023 19:57:10 GMT

I have a 9800-L-C in my home lab using two 10G in LAG. The only difference is that I setup the management vlan as native vlan. Might as well give that a try and make sure your switch has the native vlan defined:

!
interface Port-channel10
description WLC AP MGMT PORTS
switchport trunk native vlan 2
switchport mode trunk
!
interface TenGigabitEthernet0/1/0
description WLC AP MGMT PORT
switchport trunk native vlan 2
switchport mode trunk
negotiation auto
channel-group 10 mode on
service-policy output AutoQos-4.0-wlan-Port-Output-Policy
!
interface TenGigabitEthernet0/1/1
description WLC AP MGMT PORT
switchport trunk native vlan 2
switchport mode trunk
negotiation auto
channel-group 10 mode on
service-policy output AutoQos-4.0-wlan-Port-Output-Policy

The 3560X switch that the 3 APs and Primary 9800L are connected to:

interface Port-channel10
description ** EtherChan to CUN-WLC-9800LF **
switchport trunk encapsulation dot1q
switchport trunk native vlan 2
switchport mode trunk
spanning-tree portfast edge trunk
!
interface GigabitEthernet0/47
description CUN-WLC-9800LF LAG
switchport trunk encapsulation dot1q
switchport trunk native vlan 2
switchport mode trunk
channel-group 10 mode on
!
interface GigabitEthernet0/48
description CUN-WLC-9800LF LAG
switchport trunk encapsulation dot1q
switchport trunk native vlan 2
switchport mode trunk
channel-group 10 mode on

Re: 9800-L-F: APs will not register

Scott Fella — Wed, 19 Apr 2023 20:03:09 GMT

Why are you not using the Gigabit ports? The fiber is for 10gig... just noticed that. You probably should try using these interfaces.

interface TwoGigabitEthernet0/0/0
interface TwoGigabitEthernet0/0/1

If the ports are not auto negotiating, that can be your issue using those ports. Here is the recommended SFP's also:

https://www.cisco.com/c/en/us/products/collateral/wireless/catalyst-9800-series-wireless-controllers/datasheet-c78-742434.html#SFPssupported

Re: 9800-L-F: APs will not register

Leo Laohoo — Wed, 19 Apr 2023 23:34:48 GMT

@perrymcgrew wrote:
[*04/19/2023 14:36:22.2778] CAPWAP State: DTLS Setup
[*04/19/2023 14:36:37.3428] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: DTLS Setup(3).
[*04/19/2023 14:37:19.3018] OOBImageDnld: OOBImageDownloadTimer expired for image download..
[*04/19/2023 14:37:19.3018] OOBImageDnld: Do common error handler for OOB image download..

Has the AP been factory-reset?

Re: 9800-L-F: APs will not register

Mark Elsen — Thu, 20 Apr 2023 06:48:48 GMT

- Have a checkup-review of your 9800 Controller current configuration with the CLI command : show tech wireless , have the output analyzed with : https://cway.cisco.com/wireless-config-analyzer
Checkout all advisories!

Re: 9800-L-F: APs will not register

perrymcgrew — Thu, 20 Apr 2023 10:46:02 GMT

I actually had the Native VLAN 2 specified on the Port-Channels and the Interfaces on the 9800L and the 3560X switch. The current TAC Sr Engineer on the case had me remove them. It has not helped at all.

Re: 9800-L-F: APs will not register

perrymcgrew — Thu, 20 Apr 2023 10:50:13 GMT

The SFP's are Cisco GLC-TE and are listed as supported. TAC has not stated that I can't use the Te0/1/0 & Te0/1/1 with 1 Gig Copper. The 3560X in my office does not support 10G. When I do move the 9800L to the datacenter, it will be connected to 2 Te Ports in our 4500X L3 switch.

Re: 9800-L-F: APs will not register

perrymcgrew — Thu, 20 Apr 2023 10:55:13 GMT

Yes. The APs have been factory reset as one of the tests. This AP's show ver displays: AP Running Image : 17.11.0.155 which matches the 9800L 17.11.1

Re: 9800-L-F: APs will not register

perrymcgrew — Thu, 20 Apr 2023 10:59:45 GMT

Yes, have run the config thru the tool several times. The last change TAC had me do was to remove the Native VLAN statements yesterday. I have not run the current configuration through the tool since, but I don't think it's going to provide any more insight.

Re: 9800-L-F: APs will not register

Leo Laohoo — Thu, 20 Apr 2023 11:25:51 GMT

I'm going to suggest downgrading to 17.9.3 and see if makes any difference.

Re: 9800-L-F: APs will not register

Mark Elsen — Thu, 20 Apr 2023 12:02:47 GMT

>...I have not run the current configuration through the tool since, but I don't think it's going to provide any more insight.
I would advise to always use WirelessAnalyzer after configuration changes , upgrades and downgrades . Especially when you are in a situation like this. Also make sure that for the 9800-L-F models these are installed and or up to date : https://software.cisco.com/download/home/286321399/type/283425232/release/17.11.1
https://software.cisco.com/download/home/286321399/type/282046486/release/16.12(3r)

AP join issues can further be tackled with : https://logadvisor.cisco.com/logadvisor/wireless/9800/9800APJoin
I also found : https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwb21594

Re: 9800-L-F: APs will not register

perrymcgrew — Thu, 20 Apr 2023 12:12:47 GMT

We upgraded to 17.9.3, then upgraded to 17.10.1 and then 17.11.1 per troubleshooting recommendation.

Re: 9800-L-F: APs will not register

perrymcgrew — Thu, 20 Apr 2023 12:15:00 GMT

This is the output from WCAE. There is nothing there we can see that explains why the APs are not joining the 9800L-F.

Re: 9800-L-F: APs will not register

Leo Laohoo — Thu, 20 Apr 2023 12:21:03 GMT

It is counter-productive to use Early Deployment (ED) releases like 17.10.1 or 17.11.1.

No idea what was in TAC's mindset to recommend upgrading to ED just to troubleshoot AP joining issue.

I would strongly recommend re-queuing the case to EMEA or NAM/LATAM.

Re: 9800-L-F: APs will not register

Mark Elsen — Thu, 20 Apr 2023 12:27:00 GMT

- Ok, remember the stuff from my second reply too ; you might try to spin up a virtual 9800 and check the APs can then join , in order to find out if there is some issue with the supporting intranet networking infrastructure or the access points OR a native problem with the 9800 box (note that virtual 9800 WLC is free for download ) ,

Re: 9800-L-F: APs will not register

perrymcgrew — Thu, 20 Apr 2023 13:00:44 GMT

17.11.1 provided some debugging enhancements. Probably not fair to say it was an official Cisco TAC suggestion -- more like trying to see if we may be hitting a new bug. We have had the same problems on all releases, including 17.9.3. I have asked current TAC if we should revert back from 17.11.1 and so far has not said we should need to do that.

Re: 9800-L-F: APs will not register

perrymcgrew — Thu, 20 Apr 2023 13:02:53 GMT

I just factory reset a 9115AXi. Console output attached. Does not resolve the issue.

Re: 9800-L-F: APs will not register

Scott Fella — Thu, 20 Apr 2023 13:50:12 GMT

Use the gigabit ports since you are not connecting it to a 4500x. What you need to validate is if that is the issue or not. You should really start with basic config and even only using 1 port. Then build it up and see when it breaks. You are also only testing with one controller right?

What I would do since this has been an issue for a while is to wipe the controllers, only use one and just configure the basics for network connectivity. Use only one gigabit port and just see if the ap joins or not. Then I would do the same for the other controller. This eliminates any issue with one or the other and gives you a good starting point. Also pick a code that you will use in production. Only once you get the ap's joined, is when you can go ahead and make small changes and or setup SSO if that is what you will be implementing.

Re: 9800-L-F: APs will not register

Mark Elsen — Thu, 20 Apr 2023 13:24:03 GMT

>...I just factory reset a 9115AXi. Console output attached. Does not resolve the issue.
I was wondering if there is full capwap connectivity from the access points subnet to the controller , put a machine with nmap installed in that subnet and try : nmap -sU -p5246-5247 controller-hostname ,

Re: 9800-L-F: APs will not register

Rich R — Thu, 20 Apr 2023 18:13:18 GMT

1. Why do you have "service-policy output AutoQos-4.0-wlan-Port-Output-Policy" on the WLC ports? Can you remove that please?

2. I agree with Leo - don't waste time with 17.10 and 17.11 - just stick with 17.9.3 and solve the problem on that. You're more likely to *add* new problems with 17.10 and 17.11

3. "OOB image download" - I've never seen that before - suspect it's related to the new http image download functionality - disable that for a start (or just go back to 17.9.3 as per above.

4. Presume your APs do not have EWC installed? Maybe to be sure you should re-flash them using the procedure in Repair C9120/C9115 Access Points from U-boot - Cisco
Erase the WLC config and install 17.9.3.
Then manually upgrade the APs to the 17.9.3 lightweight software https://software.cisco.com/download/home/286322352/type/286288051/release/15.3.3-JPN2 because 8.10.130.0 probably won't be able to join 17.9.3.

Then run the basic startup script on the WLC as Scott says - no fancy features, no SSO to start with. Get a basic working config.

This is about setting everything back to a simple, known, working state on a stable version of extended support software.