9800LF - AP Join issues - MGT Interface is LAG

[perrymcgrew]

IOS-XE 17.09.02.  VTP mode is Client and I can see all our VLANs on switch and 9800LF.   I defined VLAN 2 with IP 10.0.3.254 /22 on Core and its routed.  DHCP is from the Core and has Option 43 pointing to the 9800LF IP assigned to its VLAN 2 interface

I have test lab setup in office.  3560X Switch connected to our Network.   2 APs connected to the 3560X Switch.  

3560X AP Port config

interface GigabitEthernet0/1
description ** TEST AP PORTS **
switchport trunk encapsulation dot1q
switchport trunk native vlan 2
switchport mode trunk

The 9800LF is connected to 3560X Switch using PortChannel  below:

interface Port-channel10
description ** EtherChan to CUN-WLC-9800LF **
switchport trunk encapsulation dot1q
switchport trunk native vlan 2
switchport mode trunk

interface GigabitEthernet0/47
description CUN-WLC-9800LF LAG
switchport trunk encapsulation dot1q
switchport trunk native vlan 2
switchport mode trunk
channel-group 10 mode on

interface GigabitEthernet0/48
description CUN-WLC-9800LF LAG
switchport trunk encapsulation dot1q
switchport trunk native vlan 2
switchport mode trunk
channel-group 10 mode on

The 9800LF ports are config'd as such:

interface Vlan2
description CUN-WLC-9800LF Mgt
ip address 10.0.3.252 255.255.252.0 secondary
ip address 10.0.3.253 255.255.252.0
mdns-sd gateway

interface Port-channel10
description 9800L MGT LAG
switchport trunk native vlan 2
switchport mode trunk

interface TenGigabitEthernet0/1/0
description PortChan 10
switchport trunk native vlan 2
switchport mode trunk
negotiation auto
channel-group 10 mode on
service-policy output AutoQos-4.0-wlan-Port-Output-Policy
!
interface TenGigabitEthernet0/1/1
description PortChan 10
switchport trunk native vlan 2
switchport mode trunk
negotiation auto
channel-group 10 mode on
service-policy output AutoQos-4.0-wlan-Port-Output-Policy

However, I don't see where Mgt Interface is set in the config

WLC-9800LF#show management-interface
No management interfaces configured

I can ping any IP address from the 9800LF.  Yet console from APs report no valid Controller found.  The Red Alarm LED is lit on the primary 9800LF.   TAC has been looking at this for a week and stated my config looks OK.   I am beginning to wonder if I need to move the IP off VLAN 2 and on to the 9800LF PortChannel as a L3 Etherchannel. If so, how does that affect my RMI+RP redundant Or is it that somehow need to set the Mgt Interface to VLAN 2 on the 9800?

Thx

[Flavio Miranda]

Hi

 You are not going to see anything with show management-interface.

I was wondering when you plug the AP on the 3560, does those APs got IP address just fine? Can you see it on show cdp nei detail?

[perrymcgrew]

Thx for the reply.   Yes, the APs get an IP from the DHCP.  See below:

SWitch# sh cdp neighbors gi0/3 det
-------------------------
Device ID: AP0845.D131.811C
Entry address(es):
IP address: 10.0.0.184
Platform: cisco AIR-AP2802I-B-K9, Capabilities: Router Trans-Bridge
Interface: GigabitEthernet0/3, Port ID (outgoing port): GigabitEthernet0
Holdtime : 112 sec

Version :
Cisco AP Software, ap3g3-k9w8 Version: 17.9.2.52
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 2014-2015 by Cisco Systems, Inc.

advertisement version: 2
Power drawn: 22.500 Watts
Power request id: 55169, Power management id: 2
Power request levels are:22500 15400 0 0 0
Management address(es):
IP address: 10.0.0.184

[marce1000]

 -  Have a checkup the 9800LF configuration with the CLI command : show  tech   wireless , have the output analyzed by  https://cway.cisco.com/tools/WirelessAnalyzer/  , please note do not use classical show tech-support (short version) , use the command denoted in green for Wireless Analyzer.               Checkout all advisories!
 If the Red Alarm Led is lid , check the outputs of : show  facility-alarm status        
                                                                                         show platform hardware slot R0  alarms visual      
                                                                                         show platform hardware slot R0  led status 
        also useful : 
           show platform software system all
           show platform resources
           show environment chassis active r0 
           show environment
           show environment summary
           show environment chassis active r0 
          show platform hardware slot R0  dram statistics 

          For debugging AP join issues use https://logadvisor.cisco.com/logadvisor/wireless/9800/9800APJoin
 If all keeps failing also update https://software.cisco.com/download/home/286321399/type/282046486/release/16.12(3r) and https://software.cisco.com/download/home/286321399/type/283425232/release/17.11.1 (if not yet done!)
          You can check the current firmware (version) with :
                                show platform hardware chassis active qfp datapath pmd ifdev | i FW
 

 M.

[perrymcgrew]

Thanks for the reply.   I just had to walk away from it this weekend! I had run the Sh Tech Wireless through WCAE.   The only "error" is mDNS (WLAN is using mDNS gateway functionality, but not corresponding SVI Interface detected. WLANs/Policies:...).   I have 6 "Best Practices" warnings.

CUN-WLC-9800LF#show facility-alarm status
System Totals Critical: 4 Major: 0 Minor: 0

Source Time Severity Description [Index]
------ ------ -------- -------------------

TwoGigabitEthernet0/0/0 Mar 10 2023 12:58:49 CRITICAL Physical Port Link Down [1]

TwoGigabitEthernet0/0/1 Mar 10 2023 12:58:49 CRITICAL Physical Port Link Down [1]

TwoGigabitEthernet0/0/2 Mar 10 2023 12:58:49 CRITICAL Physical Port Link Down [1]

TwoGigabitEthernet0/0/3 Mar 10 2023 12:58:49 CRITICAL Physical Port Link Down [1]

CUN-WLC-9800LF#show platform hardware slot R0 alarms visual
Current Visual Alarm States

Critical: On
Major : Off
Minor : Off

CUN-WLC-9800LF#show platform hardware slot R0 led status
Current LED States

LED State
----------------------------
System Green
Alarm Red
HA Green

[Rich R]

> However, I don't see where Mgt Interface is set in the config
wireless management interface Vlan2

You should read the best practice guide before doing anything on 9800 - in this case specifically https://www.cisco.com/c/en/us/products/collateral/wireless/catalyst-9800-series-wireless-controllers/guide-c07-743627.html#WirelessmanagementIPaddressing

"Wireless management interface VLAN tag

Cisco recommends using VLAN tagging for the wireless management interface of the WLC. To configure the wireless management traffic to be tagged, make sure there is no native VLAN command under the trunk configuration on the port/LAG. For example:
interface GigabitEthernet2
 switchport trunk allowed vlan 201,210,211
 switchport mode trunk

VLAN 201 is the wireless management interface VLAN and 210 and 211 are the client VLANs. Ensure that the corresponding VLAN is allowed on the switch port as well and is tagged by the trunk (non-native VLAN). In this sample configuration, the assumption is that the native VLAN (by default this is VLAN 1) is not used to carry any traffic."

[perrymcgrew]

Thanks for the reply Richard.   I seem to be getting closer.  The red Alarm light on the 9800L is off.  The AP is still not joining the 9800L (LED is flashing Green-Red).  BTW, a VLAN 1 IP is assigned to the SP.  I will see if I can figure out why the AP is still not joining the 9800L

9800L(config)#wireless management interface Vlan2

Removed the Switchport Trunk Native VLAN 2 from the 9800L's PortChannel (te0/1/0 and te0/1/1).   

interface Port-channel10
description 9800L MGT LAG
switchport mode trunk

interface TenGigabitEthernet0/1/0
description PortChan 10
switchport mode trunk
negotiation auto
channel-group 10 mode on
service-policy output AutoQos-4.0-wlan-Port-Output-Policy

interface TenGigabitEthernet0/1/1
description PortChan 10
switchport mode trunk
negotiation auto
channel-group 10 mode on
service-policy output AutoQos-4.0-wlan-Port-Output-Policy

interface Vlan2
description WLC-9800LF Mgt
ip address 10.0.3.250 255.255.252.0 secondary
ip address 10.0.3.253 255.255.252.0
mdns-sd gateway

On my 3560X Lab switch:

Port the 2802i AP is connected to:

interface GigabitEthernet0/3
description ** TEST AP PORTS **
switchport trunk encapsulation dot1q
switchport trunk native vlan 2
switchport mode trunk

The AP is pulling correct IP from our Core.  Option 43 is set to the 9800L Mgt IP 10.0.3.253:

Switch#sh cdp nei gi0/3 det
-------------------------
Device ID: AP0845.D131.811C
Entry address(es):
IP address: 10.0.0.185
Platform: cisco AIR-AP2802I-B-K9, Capabilities: Router Trans-Bridge
Interface: GigabitEthernet0/3, Port ID (outgoing port): GigabitEthernet0
Holdtime : 146 sec

The PortChan to the 9800L:

interface Port-channel10
description ** EtherChan to CUN-WLC-9800LF **
switchport trunk encapsulation dot1q
switchport mode trunk

The uplink trunk port to my Network

interface GigabitEthernet1/1
description TRUNK to Network 
switchport trunk encapsulation dot1q
switchport mode trunk
srr-queue bandwidth share 1 30 35 5
priority-queue out
mls qos trust cos
auto qos trust

From the 9800L, I can ping anything on the internet and inside my network including the Mgt subnet's default gateway on the core.  But I can't ping the IP address of the AP.   From the 3560X switch, I can ping the AP's IP address.  Going to review the 9800L routing.

[marce1000]

                                                >...(LED is flashing Green-Red).
   Scrutinize the AP boot process (trough an attached console) and watch for errors (if any)  , look at the controller logs when the AP tries to join (too). If POE is used then check if the AP is getting sufficient power.
                        Also note : https://www.cisco.com/c/en/us/support/docs/field-notices/722/fn72278.html , 'attached' = https://bst.cisco.com/bugsearch/bug/CSCwd80290

 M.

[perrymcgrew]

 I am aware of that bug. I'll have at least 180 2802i APs that will need to move off the 5508 (8.5.104) to the 9800L.   I actually have 2 APs -- the other is a 9115AXi.   It does the same thing as the 2802i.   Also, it seems I have regressed back to where the Red Alarm LED is again is RED.   On my core, i can ping the AP's DHCP assigned IP.  On the 9800L I can ping everything except the AP's assigned IP.   On the 3560X, I can ping all IPs including the AP's DHCP assigned IP.

Here is the console log from the 2802i

[*03/13/2023 13:25:00.0641] CAPWAP State: DTLS Teardown
[*03/13/2023 13:25:00.1354] OOBImageDnld: Do common error handler for OOB image download..
[*03/13/2023 13:25:00.2270] status 'upgrade.sh: Script called with args:[CANCEL]'
[*03/13/2023 13:25:00.2858] do CANCEL, part2 is active part
[*03/13/2023 13:25:00.3039] status 'upgrade.sh: Cleanup tmp files ...'
[*03/13/2023 13:25:00.3395] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: DTLS Teardown(4).
[*03/13/2023 13:25:00.3396] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: DTLS Teardown(4).
[*03/13/2023 13:25:04.7819] OOBImageDnld: OOBImageDownloadTimer expired for image download..
[*03/13/2023 13:25:04.7819] OOBImageDnld: Do common error handler for OOB image download..
[*03/13/2023 13:25:04.8110] No more AP manager addresses remain..
[*03/13/2023 13:25:04.8110] No valid AP manager found for controller 'CUN-WLC-9800LF' (ip: 10.0.3.253)
[*03/13/2023 13:25:04.8111] Failed to join controller CUN-WLC-9800LF.
[*03/13/2023 13:25:04.8111] Failed to join controller.
[*03/13/2023 13:25:24.8295]
[*03/13/2023 13:25:24.8295] CAPWAP State: Discovery
[*03/13/2023 13:25:24.8304] Got WLC address 10.0.3.253 from DHCP.
[*03/13/2023 13:25:24.8305] IP DNS query for CISCO-CAPWAP-CONTROLLER.lorettosystem.org
[*03/13/2023 13:25:24.8356] Discovery Request sent to 10.0.3.253, discovery type STATIC_CONFIG(1)
[*03/13/2023 13:25:24.8380] Discovery Request sent to 255.255.255.255, discovery type UNKNOWN(0)
[*03/13/2023 13:25:24.8388] Discovery Response from 10.0.3.253
[*03/13/2023 13:25:35.0000] Started wait dtls timer (60 sec)
[*03/13/2023 13:25:35.0099]
[*03/13/2023 13:25:35.0099] CAPWAP State: DTLS Setup
[*03/13/2023 13:26:32.0300] OOBImageDnld: OOBImageDownloadTimer expired for image download..
[*03/13/2023 13:26:32.0300] OOBImageDnld: Do common error handler for OOB image download..
[*03/13/2023 13:26:32.0666]
[*03/13/2023 13:26:32.0666] CAPWAP State: DTLS Teardown
[*03/13/2023 13:26:32.0983] OOBImageDnld: Do common error handler for OOB image download..
[*03/13/2023 13:26:32.1893] status 'upgrade.sh: Script called with args:[CANCEL]'
[*03/13/2023 13:26:32.2478] do CANCEL, part2 is active part
[*03/13/2023 13:26:32.2627] status 'upgrade.sh: Cleanup tmp files ...'
[*03/13/2023 13:26:32.2974] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: DTLS Teardown(4).
[*03/13/2023 13:26:32.2975] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: DTLS Teardown(4).
[*03/13/2023 13:26:36.7793] OOBImageDnld: OOBImageDownloadTimer expired for image download..
[*03/13/2023 13:26:36.7793] OOBImageDnld: Do common error handler for OOB image download..
[*03/13/2023 13:26:36.8031] No more AP manager addresses remain..
[*03/13/2023 13:26:36.8032] No valid AP manager found for controller 'CUN-WLC-9800LF' (ip: 10.0.3.253)
[*03/13/2023 13:26:36.8032] Failed to join controller CUN-WLC-9800LF.
[*03/13/2023 13:26:36.8032] Failed to join controller.

The 9800L does not "see" the join attempts.  

[marce1000]

                                      - Ref from ap boot and or console log :
                   >...No valid AP manager found for controller 'CUN-WLC-9800LF' (ip: 10.0.3.253)
       For me that indicates that the access point can not discover the controller completely and or correctly. Check controller logs when the AP tries to join and use these tools too : https://logadvisor.cisco.com/logadvisor/wireless/9800/9800APJoin
          You may for instance connect a laptop to an access point intended connection on the switch and put it in the same subnet/vlan intended for the access point(s) , giving it an address accordingly. Then check if you can ping the controller. Also run an nmap scan and check if the capwap ports are reachable (CAPWAP uses UDP ports 5246 (control channel) and 5247 (data channel).)

 M.
 

[perrymcgrew]

I reset the 2802i and have attached the console log.   I checked for that bug and there is no indication it's the cause why it won't join. 

[perrymcgrew]

So I believe I am now getting closer to root cause.   I believe the issue is with the 9800L configuration.   VLAN 2 is supposed to be my Mgt network for the 9800L and the APs.   VLAN 2 is 10.0.0.0 /22.   The 9800L VLAN 2 SVI is assigned 10.0.3.253 (Primary).  The SP is in VLAN 1 and the VLAN 1 SVI is assigned 192.168.0.1 /24

On my 3560X Lab switch, any port set up with Native VLAN will NOT join the 9800L

interface GigabitEthernet0/3
description ** TEST AP PORTS **
switchport trunk encapsulation dot1q
switchport trunk native vlan 2
switchport mode trunk

If I remove the Native VLAN 2 statement

interface GigabitEthernet0/3
description ** TEST AP PORTS **
switchport trunk encapsulation dot1q
switchport mode trunk

Both the 2802i and the 9115AXi *will* join.   However, they pull IP off the VLAN 1 subnet when it should be using VLAN 2 IPs.

This leads me back to 9800L and its VRF config.   

interface Port-channel10
description 9800L MGT LAG
switchport mode trunk

interface GigabitEthernet0
description WLC Service Port
vrf forwarding Mgmt-intf
no ip address
negotiation auto
!
interface Vlan1
description WLC Service Port
ip address 192.168.0.1 255.255.255.0
!
interface Vlan2
description WLC-9800LF Mgt
ip address 10.0.3.252 255.255.252.0 secondary
ip address 10.0.3.253 255.255.252.0
mdns-sd gateway
!

ip route 0.0.0.0 0.0.0.0 10.0.3.254
ip route vrf Mgmt-intf 0.0.0.0 0.0.0.0 192.168.0.254

I tried removing the vrf forwarding off the Gi0 to move it to VLAN 2 but it won't let me.   Got to do some more digging to find a way.  

[Rich R]

Agreed that is weird but also useful.  Tells us both vlans are working and your problem is something else.
You're not using Gig0 so your "ip route vrf Mgmt-intf 0.0.0.0 0.0.0.0 192.168.0.254" is doing absolutely nothing at all.  You have vlan 1 in your global routing table which already has a default route out of vlan 2.
Of course it's your choice but normally you would put G0 on an access port in vlan 1 and use that for your management independent of global routing table used by WLC and then that VRF route would come into play.

So possible reasons that APs aren't working when VLAN 2 is native ...

Did you maybe apply a flex profile with vlan 1 defined as native (which means the APs will be using vlan 2 tagged?
If so then correct your flex profile, put back native vlan 2 on the AP ports and then default the config on the APs and then they should join and work.

Otherwise you'll have to troubleshoot from first principles looking at VLANs, routing and ARP.  Packet captures on switch and WLC might help you understand where you've gone wrong.

[perrymcgrew]

I have not even gotten to look at FLEX profile yet.   I have several sites that have no Local WLC and use the current HQ 5508 as the WLC.  This is my "phase 1" when it comes to replacing the 5508 with the 9800L pair.  I have poured through lots of Cisco docs and have not found the "needle in the haystack" why the APs will not use my config'd VLAN 2 and only join using VLAN 1. 

vrf definition Mgmt-intf
!
address-family ipv4
exit-address-family
!
address-family ipv6
exit-address-family
vrf forwarding Mgmt-intf

ip route 0.0.0.0 0.0.0.0 10.0.3.254
ip route vrf Mgmt-intf 0.0.0.0 0.0.0.0 192.168.0.254

interface GigabitEthernet0
description WLC Service Port
vrf forwarding Mgmt-intf
no ip address
negotiation auto

interface Vlan1
description WLC Service Port
ip address 192.168.0.1 255.255.255.0

interface Port-channel10
description 9800L MGT LAG
switchport mode trunk

interface Vlan2
description WLC-9800LF Mgt
ip address 10.0.3.252 255.255.252.0 secondary
ip address 10.0.3.253 255.255.252.0
mdns-sd gateway

WLC-9800LF#sh ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2, m - OMP
n - NAT, Ni - NAT inside, No - NAT outside, Nd - NAT DIA
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
H - NHRP, G - NHRP registered, g - NHRP registration summary
o - ODR, P - periodic downloaded static route, l - LISP
a - application route
+ - replicated route, % - next hop override, p - overrides from PfR
& - replicated local route overrides by connected

Gateway of last resort is 10.0.3.254 to network 0.0.0.0

S* 0.0.0.0/0 [1/0] via 10.0.3.254
10.0.0.0/8 is variably subnetted, 3 subnets, 2 masks
C 10.0.0.0/22 is directly connected, Vlan2
L 10.0.3.252/32 is directly connected, Vlan2
L 10.0.3.253/32 is directly connected, Vlan2
192.168.0.0/24 is variably subnetted, 2 subnets, 2 masks
C 192.168.0.0/24 is directly connected, Vlan1
L 192.168.0.1/32 is directly connected, Vlan1

WLC-9800LF#sh vrf bri
Name        Default RD Protocols Interfaces
Mgmt-intf <not set>    ipv4,ipv6 Gi0

WLC-9800LF#sh ip route vrf Mgmt-intf

Routing Table: Mgmt-intf
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2, m - OMP
n - NAT, Ni - NAT inside, No - NAT outside, Nd - NAT DIA
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
H - NHRP, G - NHRP registered, g - NHRP registration summary
o - ODR, P - periodic downloaded static route, l - LISP
a - application route
+ - replicated route, % - next hop override, p - overrides from PfR
& - replicated local route overrides by connected

Gateway of last resort is not set

I will try moving the VLAN 1 IP up to the G0 interface

[perrymcgrew]

So feeling kind of dumb.   I changed the 3560 Lab switch ports the 2 APs are connected from Trunk Native VLAN 2 to Access VLAN 2 and the APs joined using the correct VLAN 2 subnet.   The Primary 9800L Alarm LED is RED again so I am not sure what that means - the Redundant 9800L Alarm LED is NOT lit. At least I feel I've made some progress!