2802 APs not joining back Primary Controller

[salamislimy]

HI 

We have two controllers - 5520 - primary(.5) and secondary(.7) in HA-SKU

Recently we upgraded the controller from 8.5.131.7 to 8.10.151.0

Had a total of 112 APs on the primary controller

to avoid disruptions to service, we first upgraded the secondary (.7) moved the APs in batches to secondary, predownloaded the image on the APs, upgraded the primary and moved the APs back

while moving  85 APs have joined back the primary controller

27 are on the secondary and wont join back the primary

I have tried changing the high availability settings on the GUI and also tried CLI commands on the controller but after the AP reboots its still comes back to the secondary controller

The AP shows that its primary controller is our primary controller (.5) but it is currently on the secondary controller (.7)

AP413#show configuration
Admin State : Enabled
AP Mode : Local
AP Submode : Not Configured
Location : default location
Primary controller name : Primary-controller-1
Primary controller IP : 10.120.5.5
Secondary controller name :Secondary-controller-1
Seocndary controller IP: 10.120.5.7
Tertiary controller name :
Controller from DHCP offer : 10.120.5.5, 10.120.5.7
AP join priority : 2
IP Prefer-mode : IPv4
CAPWAP UDP-Lite : Unconfigured
Last Joined Controller name: Secondary-controller-1
DTLS Encryption State : Disabled
Discovery Timer : 10
Heartbeat Timer : 30
CDP State : Enabled
Watchdog monitoring : Enabled
IOX : Disabled
RRM State : Enabled
LSC State : Disabled
SSH State : Enabled
Session Timeout : 300
Extlog Host : 0.0.0.0
Extlog Flags : 0
Extlog Status Interval : 0
Syslog Host : 255.255.255.255
Syslog Facility : 0
Syslog Level : informational
Core Dump File Compression : Disabled
Core Dump Filename :
Client Trace Status : Disabled(All)
Client Trace All Clients : Disabled
Client Trace Filter : 0x00000000
Client Trace Out ConsoleLog: Disabled
WLC Link LAG status : Disabled
AP Link LAG status : Disabled
AP WSA Mode : Disabled
Auxiliary-client Interface : Disabled

If you could let me know how to fix this or let me know of some debug commands to see what the issue is, that would be helpful

[Rich R]

1. You say the secondary with HA-SKU but if you have APs on them separately then they are not running as HA-SSO which is what the HA-SKU is for.  So assume you're actually doing N+1 HA for starters which means you need a valid RTU license on your secondary.

2. What debugging have you done?  What do the console logs on the AP show?  What do the logs on the WLCs show?  What do the join stats on the WLCs show?

[salamislimy]

Hi rruding

The Secondary is in HA-SKU, the only reason we moved the  APs on to the secondary was to upgrade the primary and not cause much disruption to the wireless network.

This is a remote location, so we havent been able to get the console logs yet.

Rebooted one of the APs and this is what I get

Primary Controller

show ap join stats summary 4c:71:0d:00:00:00
No join information found for AP: 4c:71:0d:00:00:00

Secondary Controller

>show ap join stats summary 4c:71:0d:00:00:00

Is the AP currently connected to controller................ Yes
Time at which the AP joined this controller last time...... Dec 09 08:15:52.701
Type of error that occurred last........................... AP got or has been disconnected
Reason for error that occurred last........................ The AP has been reset by the controller
Time at which the last join error occurred................. Dec 09 08:11:47.437

Cant find anything specific to the AP in the controller logs

[Rich R]

So you're saying you split your SSO pair into standalone WLCs?

If so I think you just made your upgrade a whole lot more difficult and disruptive.  That's usually only a method of last resort if the SSO pair upgrade goes wrong and the object is to get them back into HA not split the APs across them!

The point about the license remains - HA SKU is *only* for use in SSO pair where it inherits the primary license when it becomes active.

Next time just do a standard upgrade with pre-download.

If you're upgrading to 8.10 then it should be 8.10.162.0 anyway.

You can do packet captures remotely and on the WLC to see what's going on in the join/discovery packets - that might give you some clue as to what's going wrong.

Are the controller names configured on the APs identical to what is configured on the WLC?

Are both WLC running the same version of code now?

[salamislimy]

Sorry I think I dint explain the scenario correctly but anyways this is the error on the console of AP

[*12/09/2021 03:55:36.2661] upgrade.sh: Cleanup tmp files ...
[*12/09/2021 03:55:36.2968] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: DTLS Teardown(4).
[*12/09/2021 03:55:36.2969] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: DTLS Teardown(4).
[*12/09/2021 03:55:40.7880] No more AP manager addresses remain..
[*12/09/2021 03:55:40.7881] No valid AP manager found for controller 
[*12/09/2021 03:55:40.7881] Failed to join controller 
[*12/09/2021 03:55:40.7881] Failed to join controller.
[*12/09/2021 06:15:01.0004]

Seems to be similar to CSCvy37953 however the AP reboot doesnt fix the issue

going to TAC

Thanks again

[Rich R]

Can you post the results of show sysinfo, show license summary, show license statistics & show license in-use from both WLC?

[salamislimy]

Controller 1

>show sysinfo

Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 8.10.151.0
RTOS Version..................................... 8.10.151.0
Bootloader Version............................... 8.3.15.177
Emergency Image Version.......................... 8.3.143.0

OUI File Last Update Time........................ Tue Feb 06 10:44:07 UTC 2018

Build Type....................................... DATA + WPS

System Name......................................
System Location..................................
System Contact...................................
System ObjectID.................................. 1.3.6.1.4.1.9.1.2170
Redundancy Mode.................................. SSO

IPv6 Address..................................... ::
System Up Time................................... 3 days 22 hrs 59 mins 11 secs
System Timezone Location.........................
System Stats Realtime Interval................... 5

--More-- or (q)uit
System Stats Normal Interval..................... 180

Operating Environment............................ Commercial (10 to 35 C)
Internal Temp Alarm Limits....................... 10 to 38 C
Internal Temperature............................. +25 C
Fan Status....................................... OK

State of 802.11b Network......................... Enabled
State of 802.11a Network......................... Enabled
Number of WLANs.................................. 6
Number of Active Clients......................... 51

OUI Classification Failure Count................. 12841

Memory Current Usage............................. 11
Memory Average Usage............................. 11
CPU Current Usage................................ 0
CPU Average Usage................................ 0

Flash Type....................................... Compact Flash Card
Flash Size....................................... 1073741824

Power Supply 1................................... Present, OK
Power Supply 2................................... Present, OK
Maximum number of APs supported.................. 1500
System Nas-Id....................................
WLC MIC Certificate Types........................ SHA1/SHA2
Licensing Type................................... RTU

>show license summary

Feature name: ap_count
License type: Evaluation
License Eula: Not Accepted
Evaluation total period: 12 weeks 6 days
License state: Inactive, Not-In-Use
RTU License Count: 1500

Feature name: ap_count (adder)
License type: Permanent
License state: Active, In-use
RTU License Count: 170

>show license statistics

Total add operations:0
Total delete operations:0
Total eval activations:0
Total eval deactivations:0
Max ap count usage:155
Max eval ap count usage:0
Max base & adder ap count usage:155
Max Eval ap count usage:0
Total over usage:0
Eval license usage days:0
Max base & adder count attained:0
Max adder count added:0

>show licnese  =     ence  se u in-use

Feature name: ap_count (adder)
License type: Permanent
License state: Active, In-use
License Nodelocked: No
RTU License Count: 170

==================================
Total available count : 170
Total inuse count : 85

Controller 2

>show sysinfo

Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 8.10.151.0
RTOS Version..................................... 8.10.151.0
Bootloader Version............................... 8.3.15.177
Emergency Image Version.......................... 8.3.143.0

OUI File Last Update Time........................ Tue Feb 06 10:44:07 UTC 2018

Build Type....................................... DATA + WPS

System Location..................................
System Contact...................................
System ObjectID.................................. 1.3.6.1.4.1.9.1.2170
Redundancy Mode.................................. Disabled

IPv6 Address..................................... ::
System Up Time................................... 9 days 18 hrs 39 mins 20 secs
System Timezone Location.........................
System Stats Realtime Interval................... 5

--More-- or (q)uit
System Stats Normal Interval..................... 180

Operating Environment............................ Commercial (10 to 35 C)
Internal Temp Alarm Limits....................... 10 to 38 C
Internal Temperature............................. +22 C
Fan Status....................................... OK

State of 802.11b Network......................... Enabled
State of 802.11a Network......................... Enabled
Number of WLANs.................................. 6
Number of Active Clients......................... 21

OUI Classification Failure Count................. 3343

Memory Current Usage............................. 11
Memory Average Usage............................. 11
CPU Current Usage................................ 0
CPU Average Usage................................ 0

Flash Type....................................... Compact Flash Card

Flash Size....................................... 1073741824

--More-- or (q)uit

Power Supply 1................................... Present, OK
Power Supply 2................................... Present, OK
Maximum number of APs supported.................. 1500
System Nas-Id....................................
WLC MIC Certificate Types........................ SHA1/SHA2
Licensing Type................................... RTU

>show license summary

Feature name: ap_count

License type: Evaluation

License Eula: Not Accepted

Evaluation total period: 12 weeks 6 days

License state: Active, Not-In-Use

RTU License Count: 1500

>show license statistics

Total add operations:0

Total delete operations:0

Total eval activations:0

Total eval deactivations:0

Max ap count usage:0

Max eval ap count usage:0

Max base & adder ap count usage:0

Max Eval ap count usage:0

Total over usage:0

Eval license usage days:0

Max base & adder count attained:0

Max adder count added:0

>show license in-use

There are no licenses

>

[Jaccobbchoi]

May I know if you pointed your management gateway to an HSRP virtual IP Address? Or if there's HSRP configured on your Management VLAN?

[Rich R]

Post above alerted me to this and no updates since reply back in December.

So your output confirmed you have NO LICENSE on your 2nd WLC (only eval).

That is because, as I explained twice already, that the HA SKU is for SSO HA not N+1 HA.  Read the documentation on the difference.

In order to run N+1 HA you must buy a license for each WLC.

Regardless of that your problem is that the APs won't join your primary.

Did you get packet captures (compare a working and non-working example) or has TAC solved the problem for you meanwhile?

If you still have the problem then next steps:

- upgrade to 8.10.171.0 - there are some relevant fixes.

- factory reset the troublesome APs.

- get the packet captures

[salamislimy]

So we did a factory reset on the APs - No luck

TAC spent a lot of time on this and dint get a resolution

We couldnt proceed with the TAC case any further as a lot of time was spent on this

we found a work around where changing the vlan on the switchport where the AP is connected worked, so we went ahead with the work around.