9800-L-F: APs will not register

[perrymcgrew]

I have had case open for about a month on this and at least 3 different TAC engineers.   I have 9800L-F set up in HA-SSO (RP+RMI).  We have tried 17.9.x, 17.10,1 and now are on 17.11.1.

I have set the AP Mgt as VLAN 2 (10.0.0.0 /22) and the 9800L Management Interface at 10.0.3.253.  I have 3560X switch set up in my office "lab" with 3 APs (two 9115AXi and 1 2802i) connected as well as the Primary 9800L chassis.   The APs will not connect -- or if they do, it takes days.  We have done numerous traces / debugs / packet captures and TAC still cannot explain why.   I am hoping a fresh set of eyes can see what the problem is and how to fix.

The AP console session repeats this: 

[*04/19/2023 14:36:12.8578] CAPWAP State: Discovery
[*04/19/2023 14:36:12.8808] Discovery Request sent to 10.0.3.253, discovery type STATIC_CONFIG(1)
[*04/19/2023 14:36:12.8818] Discovery Request sent to 10.0.3.253, discovery type STATIC_CONFIG(1)
[*04/19/2023 14:36:12.8828] Discovery Request sent to 255.255.255.255, discovery type UNKNOWN(0)
[*04/19/2023 14:36:12.9098] Discovery Response from 10.0.3.253
[*04/19/2023 14:36:22.2708] Started wait dtls timer (60 sec)
[*04/19/2023 14:36:22.2778]
[*04/19/2023 14:36:22.2778] CAPWAP State: DTLS Setup
[*04/19/2023 14:36:37.3428] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: DTLS Setup(3).
[*04/19/2023 14:37:19.3018] OOBImageDnld: OOBImageDownloadTimer expired for image download..
[*04/19/2023 14:37:19.3018] OOBImageDnld: Do common error handler for OOB image download..
[*04/19/2023 14:37:19.3288]
[*04/19/2023 14:37:19.3288] CAPWAP State: DTLS Teardown
[*04/19/2023 14:37:19.3778] OOBImageDnld: Do common error handler for OOB image download..
[*04/19/2023 14:37:19.4628] status 'upgrade.sh: Script called with args:[CANCEL]'
[*04/19/2023 14:37:19.5058] do CANCEL, part1 is active part
[*04/19/2023 14:37:19.5228] status 'upgrade.sh: Cleanup tmp files ...'
[*04/19/2023 14:37:19.5488] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: DTLS Teardown(4).
[*04/19/2023 14:37:19.5488] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: DTLS Teardown(4).
[*04/19/2023 14:37:24.0528] OOBImageDnld: OOBImageDownloadTimer expired for image download..
[*04/19/2023 14:37:24.0528] OOBImageDnld: Do common error handler for OOB image download..
[*04/19/2023 14:37:24.0728] No more AP manager addresses remain..
[*04/19/2023 14:37:24.0728] No valid AP manager found for controller 'CUN-WLC-9800LF' (ip: 10.0.3.253)
[*04/19/2023 14:37:24.0728] Failed to join controller CUN-WLC-9800LF.
[*04/19/2023 14:37:24.0728] Failed to join controller.

(TAC set a static IP on this AP of 10.0.2.1 /22 for a test.  The other test APs use DHCP and have the same console messages)  

The core where VLAN 2 is defined:

ip dhcp pool 9800_WLC_MGT
network 10.0.0.0 255.255.252.0
default-router 10.0.3.254
domain-name xxxxx.yyy
option 43 hex f104.0a00.03fd
dns-server 192.168.8.1 192.168.8.2

!

interface Vlan2
description 9800-WiFi_Mgt Subnet
ip address 10.0.3.254 255.255.252.0
no ip redirects
no ip unreachables
no ip proxy-arp

The 9800L Interfaces: 

!
interface Port-channel10
description WLC AP MGMT PORTS
switchport mode trunk

!
interface TenGigabitEthernet0/1/0
description WLC AP MGMT PORT
switchport mode trunk
no negotiation auto
channel-group 10 mode on
service-policy output AutoQos-4.0-wlan-Port-Output-Policy

!

interface TenGigabitEthernet0/1/1
description WLC AP MGMT PORT
switchport mode trunk
no negotiation auto
channel-group 10 mode on
service-policy output AutoQos-4.0-wlan-Port-Output-Policy

CUN-WLC-9800LF#show wireless interface summ

Wireless Interface Summary

Interface Name Interface Type VLAN ID IP Address IP Netmask          NAT-IP  Address MAC Address
--------------------------------------------------------------------------------------------------
Vlan2      Management                  2      10.0.3.253     255.255.252.0 0.0.0.0  8c1e.xxxx.yyyy

CUN-WLC-9800LF#show wireless management trustpoint
Trustpoint Name : ewlc-tp1
Certificate Info : Available
Certificate Type : SSC
Certificate Hash : 9a80a68f45b442770a4567d4xxxxxxxxxxxxxx
Private key Info : Available
FIPS suitability : Not Applicable

The 3560X switch that the 3 APs and Primary 9800L are connected to:

interface Port-channel10
description ** EtherChan to CUN-WLC-9800LF **
switchport trunk encapsulation dot1q
switchport mode trunk
spanning-tree portfast edge trunk

!

interface GigabitEthernet0/47
description CUN-WLC-9800LF LAG
switchport trunk encapsulation dot1q
switchport mode trunk
channel-group 10 mode on

!

interface GigabitEthernet0/48
description CUN-WLC-9800LF LAG
switchport trunk encapsulation dot1q
switchport mode trunk
channel-group 10 mode on

The AP connected ports:

interface GigabitEthernet0/1
description ** TEST AP PORTS **
switchport access vlan 2
switchport mode access
spanning-tree portfast edge

There is no VLAN pruning in place on trunk ports.  

TIA - Perry

[Rich R]

Good that the QOS is eliminated but can you do what we asked and go fully back to a bare basic config and start from scratch?

Also make sure the APs are using DHCP.

In the meantime get a packet capture spanned from the AP port and then the WLC port so we can see exactly what traffic is coming and going at both ends.  If you can't even ping things in the same subnet you may have a more fundamental problem there that has nothing to do with the WLC!

[perrymcgrew]

The only IP that an *unjoined* AP cannot ping is the WLC's Mgmt IP.  It *can* ping other devices on its /22 subnet as well as IPs that are not in the same subnet.  We (TAC)  have even statically set the AP's IP using the capwap ap ip syntax. 

[marce1000]

              - Are you talking about Wireless Management interface-ip here or Service port address ?

 M.

[perrymcgrew]

The SP IP is in VLAN 1 (192.168.0.1 /24) assigned to Gi0.   The 9800L WLC / AP Mgmt AP is in VLAN 2 (10.0.3.253 /22) which I just moved to Tw0/0/3 interface from the Port-Channel that used the 2 Te ports to eliminate any question posed here that the Te ports on the 9800-L-F need to be connected via Fiber SFPs -- which, according to TAC and the Docs is not true.  I have Cisco GLC-TE SFP's in Te0/1/0 & Te0/1/1 which are listed as supported.

I reset a 9115AXi AP to factory defaults and am posting the Boot process as well as the Radioactive Trace from the 9800L.  

[perrymcgrew]

The 9800L's log shows this message:

%IOSXE-4-PLATFORM: Chassis 1 R0/0: cpp_cp: QFP:0.0 Thread:003 TS:00000056155983588748 %SWPORT-4-MAC_CONFLICT: Dynamic mac 8C1E.806A.D2AB from TwoGigabitEthernet0/0/3 conflict with SVI, please check the network topology and make sure there is no loop.

I don't know if this is part of the AP Join issues.  The MAC listed here is the 9800L.  The SSIDs defined in the 9800L are Disabled.  As stated, our current production 5508 is in a completely different VLAN.  The Tw0/0/3 interface is assigned as a TRUNK.  We have tried assigning Native VLAN 2 to the 9800L Mgt ports & switch has not resolved the message or the AP Joins.

interface TwoGigabitEthernet0/0/3
description Test MGMT
switchport mode trunk
negotiation auto

!
interface Vlan2
description AP Management
ip address 10.0.3.252 255.255.252.0 secondary
ip address 10.0.3.253 255.255.252.0
no ip proxy-arp

Currently, the 9800L is in VTP Mode Transparent and below are the only VLANs it knows about.  

CUN-WLC-9800LF#sh vlan

VLAN Name Status Ports
---- -------------------------------- --------- -------------------------------
1 default active Tw0/0/1, Tw0/0/2, Te0/1/0, Te0/1/1
2 LMC_WiFi_Mgt active
12 LMC-R active
20 LOR-CORP active
30 TRCU active
49 guest_wired active Tw0/0/0
96 IT-Test active
100 LMC-D active
189 LOR-MGMT active
199 guest active
200 LMC-V active

[marce1000]

 - In general , and as a few of us remarked already : You have a  'complex brewery' going on with these setups you are trying , you should factory reset the controller and try a very simple setup with one access point on the most simple possible network topology , check if that can work for you , then try to configure the additional elements you need and find out where it goes wrong , 

 M.

[Rich R]

And do it with 17.9.3 not one of the new limited support releases.

[Leo Laohoo]

---

@perrymcgrew wrote:
%IOSXE-4-PLATFORM: Chassis 1 R0/0: cpp_cp: QFP:0.0 Thread:003 TS:00000056155983588748 %SWPORT-4-MAC_CONFLICT: Dynamic mac 8C1E.806A.D2AB from TwoGigabitEthernet0/0/3 conflict with SVI, please check the network topology and make sure there is no loop.

---

This is a spam "bug feature".  

I have a controller on 17.6.4 and my logs get spammed by this error message.  Different controller (same config) but on 17.9.3 and nadda.