Cisco WLC 5520 Upgrade 8.10.190.0. 2 Out of 6 Aps Flashing Green/Red

[Barry Landon]

Ive upgraded my Cisco WLC 5520 from 8.10.151.0 to 8.10.190.0.  I have 4 access points that are online and working however 2 access points are flashing green and red continuously.

Layer 2 i can see a mac address but at our L3 core i have no arp entry.  

DHCP server has no cached reservation entries.  It only sees the 4 active access points.

Ive done a factory reset through CLI and straight after performed a hard reset using the mode button.

Management interface and access points subnet are on the same vlan.

All our APs are AP2802I-E-K9.

The WLC has sent the correct code to the 2 down access points but for some reason they have not come online.

Below is the output when the device reloaded after cli / mode button reset.

U-Boot 2013.01-ge1c49d93a (Jul 28 2023 - 09:08:14) SDK version: 2015_T2.0p10

Board: Barbados-2K
SoC: MV88F6920 Rev A1
running 2 CPUs
CPU: ARM Cortex A9 MPCore (Rev 1) LE
CPU 0
CPU @ 1800 [MHz]
L2 @ 900 [MHz]
TClock @ 250 [MHz]
DDR4 @ 900 [MHz]
DDR4 32 Bit Width,FastPath Memory Access, DLB Enabled, ECC Disabled
DRAM: 1 GiB

RST I2C0
NAND: ECC 4bits, 256 MiB
SF: Detected MX25L3205D with page size 64 KiB, total 4 MiB
PCI-e 1 (IF 0 - bus 0) Root Complex Interface, Detected Link X1, GEN 2.0
PCI-e 2 (IF 1 - bus 1) Root Complex Interface, Detected Link X1, GEN 2.0

Map: Code: 0x3fed7000:0x3ffad848
BSS: 0x3ffef314
Stack: 0x3f9c6f20
Heap: 0x3f9c7000:0x3fed7000
U-Boot Environment: 0x00100000:0x00110000 (SPI)

Board configuration:
| port | Interface | PHY address |
|--------|------------|--------------|
| egiga1 | SGMII | 0x01 |
| egiga2 | SGMII | 0x00 |

Button is pressed. Configuration reset activated..
Keep the button pressed for > 20 seconds for full factory reset

Wait for the button to be released ....
Button pressed for 27 seconds

Performing full factory reset..

Setting env for full reset..
SF: Detected MX25L3205D with page size 64 KiB, total 4 MiB
Erasing SPI flash....Writing to SPI flash.....done
do_resetb
do_clear fips env

Setting env for configuration reset..
Erasing SPI flash....Writing to SPI flash.....done
do_format_storage
Creating 1 MTD partitions on "nand0":
0x000000200000-0x000010000000 : "mtd=2"
UBI: attaching mtd1 to ubi0
Remove UBI volume storage (id 5)
No size specified -> Using max size (68694016)
Creating dynamic volume storage of size 68694016
reset u-boot
resetting ...

BootROM - 1.78
Booting from SPI flash, Secure mode
BootROM: RSA Public key verification PASSED
BootROM: CSK block signature verification PASSED
BootROM: Boot header signature verification PASSED
BootROM: Box ID verification PASSED
BootROM: JTAG is disabled

General initialization - Version: 1.0.0
Detected Device ID 6920
Master bootloder version 1.4a
High speed PHY - Version: 2.0
BoardId = 0x21board SerDes lanes topology details:
| Lane # | Speed| Type |
------------------------------|
| 1 | 0 | SGMII1 |
| 2 | 5 | PCIe1 |
| 4 | 5 | PCIe2 |
| 5 | 0 | SGMII2 |
-------------------------------
:** Link is Gen1, check the EP capability
PCIe, Idx 1: Link upgraded to Gen2 based on client cpabilities
:** Link is Gen1, check the EP capability
PCIe, Idx 2: Link upgraded to Gen2 based on client cpabilities
High speed PHY - Ended Successfully
DDR4 Training Sequence - Ver TIP-0.23.(Sublib 0.8)0
DDR4 Training Sequence - Switching XBAR Window to FastPath Window
DDR4 Training Sequence - Ended Successfully
BootROM: Image checksum verification PASSED
BootROM: Boot image signature verification PASSED

____ _
/ ___|(_) ___ ___ ___
| | | |/ __| / __|/ _ \
| |___ | |\__ \| (__| (_) |
\____||_||___/ \___|\___/
_ _ ____ _
| | | | | __ ) ___ ___ | |_
| | | |___| _ \ / _ \ / _ \| __|
| |_| |___| |_) | (_) | (_) | |_
\___/ |____/ \___/ \___/ \__|
** LOADER **

U-Boot 2013.01-ge1c49d93a (Jul 28 2023 - 09:08:14) SDK version: 2015_T2.0p10

Board: Barbados-2K
SoC: MV88F6920 Rev A1
running 2 CPUs
CPU: ARM Cortex A9 MPCore (Rev 1) LE
CPU 0
CPU @ 1800 [MHz]
L2 @ 900 [MHz]
TClock @ 250 [MHz]
DDR4 @ 900 [MHz]
DDR4 32 Bit Width,FastPath Memory Access, DLB Enabled, ECC Disabled
DRAM: 1 GiB

RST I2C0
NAND: ECC 4bits, 256 MiB
SF: Detected MX25L3205D with page size 64 KiB, total 4 MiB
PCI-e 1 (IF 0 - bus 0) Root Complex Interface, Detected Link X1, GEN 2.0
PCI-e 2 (IF 1 - bus 1) Root Complex Interface, Detected Link X1, GEN 2.0

Map: Code: 0x3fed7000:0x3ffad848
BSS: 0x3ffef314
Stack: 0x3f9c6f20
Heap: 0x3f9c7000:0x3fed7000
U-Boot Environment: 0x00100000:0x00110000 (SPI)

Board configuration:
| port | Interface | PHY address |
|--------|------------|--------------|
| egiga1 | SGMII | 0x01 |
| egiga2 | SGMII | 0x00 |
Net: , egiga1, egiga2 [PRIME]
Hit ESC key to stop autoboot: 0
Creating 1 MTD partitions on "nand0":
0x000000200000-0x000010000000 : "mtd=2"
UBI: attaching mtd1 to ubi0
UBIFS: mounted UBI device 0, volume 1, name "part2"
Loading file 'part2/info.ver' to addr 0x02000000 with size 84 (0x00000054)...
Done
Loading file 'part2/part.bin' to addr 0x02000000 with size 6387129 (0x006175b9)...
Done
SF: Detected MX25L3205D with page size 64 KiB, total 4 MiB

Checking image signing.
Image signing verification success, continue to run...
Preferred boot heartbeat count: 3
SF: Detected MX25L3205D with page size 64 KiB, total 4 MiB
Erasing SPI flash....Writing to SPI flash.....done
## Booting kernel from Legacy Image at 02000000 ...
Image Name: Barbados Firmware Image
Created: 2023-09-28 0:25:17 UTC
Image Type: ARM Linux Multi-File Image (uncompressed)
Data Size: 6386121 Bytes = 6.1 MiB
Load Address: 00008000
Entry Point: 00008000
Contents:
Image 0: 4204432 Bytes = 4 MiB
Image 1: 2165533 Bytes = 2.1 MiB
Image 2: 16137 Bytes = 15.8 KiB
Verifying Checksum ... OK
## Loading init Ramdisk from multi component Legacy Image at 02000000 ...
## Flattened Device Tree from multi component Image at 02000000
Booting using the fdt at 0x02613300
Loading Multi-File Image ... OK
OK
reserving fdt memory region: addr=0 size=4000
Loading Device Tree to 01ff9000, end 01ffff08 ... OK

Starting Device Tree update ('fdt_skip_update' = no)

Limit DDR size at 3GB due to power of 2 requirement of Address decoding
Updating device tree successful

Starting kernel ...

[01/01/1970 00:00:00.0000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 260096
[01/01/1970 00:00:00.0000] Memory: 1025732K/1048576K available (5743K kernel code, 411K rwdata, 2504K rodata, 359K init, 496K bss, 22844K reserved, 0K highmem)
[01/01/1970 00:00:00.1200] CPU1: Booted secondary processor
[01/01/1970 00:00:01.4600] buginf tty flushing thread started, ttyport=bf192400
[01/01/1970 00:00:01.5800] m25p80 spi1.0: found mx25l3206, expected n25q032
[*01/01/1970 00:00:02.7338] buginf() enabled.
[*01/01/1970 00:00:02.7435] Made it into bootsh: Sep 28 2023 00:18:16 T-eea26d9aec ddbce8a3d538e497d7baad0ab582d0-geea26d9a-aut
[*01/01/1970 00:00:10.8683] bootsh mini ramfs booted /bootpart/part2/ramfs_data_ci sco.cpio.lzma

Welcome to Cisco.

Usage of this device is governed by Cisco's End User License Agreement,
available at:
http://www.cisco.com/c/en/us/td/docs/general/warranty/English/EU1KEN_.html.

Restricted Rights Legend

Use, duplication, or disclosure by the Government is subject to
restrictions as set forth in subparagraph (c) of the Commercial
Computer Software - Restricted Rights clause at FAR sec. 52.227-19 and
subparagraph (c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.

Cisco Systems, Inc.
170 West Tasman Drive
San Jose, California 95134-1706

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

This product contains some software licensed under the
"GNU General Public License, version 2" provided with
ABSOLUTELY NO WARRANTY under the terms of
"GNU General Public License, version 2", available here:
http://www.gnu.org/licenses/old-licenses/gpl-2.0.html

This product contains some software licensed under the
"GNU Library General Public License, version 2" provided
with ABSOLUTELY NO WARRANTY under the terms of "GNU Library
General Public License, version 2", available here:
http://www.gnu.org/licenses/old-licenses/lgpl-2.0.html

This product contains some software licensed under the
"GNU Lesser General Public License, version 2.1" provided
with ABSOLUTELY NO WARRANTY under the terms of "GNU Lesser
General Public License, version 2.1", available here:
http://www.gnu.org/licenses/old-licenses/lgpl-2.1.html

This product contains some software licensed under the
"GNU General Public License, version 3" provided with
ABSOLUTELY NO WARRANTY under the terms of
"GNU General Public License, Version 3", available here:
http://www.gnu.org/licenses/gpl.html.

This product contains some software licensed under the
"GNU Affero General Public License, version 3" provided
with ABSOLUTELY NO WARRANTY under the terms of
"GNU Affero General Public License, version 3", available here:
http://www.gnu.org/licenses/agpl-3.0.html.

Factory full reset detected...
Full Factory Reset triggered: clear all files from storage..
[*01/01/1970 00:00:33.6682] /init: ioctl(SIOCGIFHWADDR): wired0: No such device
[*01/01/1970 00:00:33.6713] /init: ioctl(SIOCGIFHWADDR): wired0: No such device
[ OK ] Reached target Cisco File Systems (Pre).
[ OK ] Reached target Timers.
[ OK ] Created slice -.slice.
[ OK ] Created slice system.slice.
[ OK ] Created slice system-sshd\x2dkeygen.slice.
[ OK ] Created slice system-serial\x2dgetty.slice.
Starting Cisco UBIFS reformat/mount...
[ OK ] Listening on udev Kernel Socket.
[ OK ] Listening on udev Control Socket.
Starting udev Coldplug all Devices...
Starting udev Kernel Device Manager...
[ OK ] Reached target Slices.
[ OK ] Listening on Delayed Shutdown Socket.
[ OK ] Reached target Sockets.
[ OK ] Started udev Kernel Device Manager.
[ OK ] Started Cisco UBIFS reformat/mount.
Starting Cisco early mount...
[ OK ] Started Cisco early mount.
[ OK ] Reached target Local File Systems (Pre).
Starting Cisco system time setup...
Starting Cisco fips check...
Starting Cisco seed generation...
[*05/11/2018 23:59:59.0495] Manufacturing time is: May 11 23:59:59 2018
[*05/11/2018 23:59:59.0496] Storing the manufacturing time in reload time as the r eload time stamp was not present
[*05/11/2018 23:59:59.0084] Setting system time Fri May 11 23:59:59 UTC 2018
Starting Cisco platform file generation...
[ OK ] Started Cisco system time setup.
[ OK ] Started Cisco seed generation.
[ OK ] Started Cisco fips check.
[ OK ] Started udev Coldplug all Devices.
[ OK ] Started Cisco platform file generation.
Starting Cisco btldr service...
[ OK ] Started Cisco btldr service.
[ OK ] Reached target Local File Systems.
Starting Cisco pkg install service...
[ OK ] Started Cisco system time saving.
Starting Cisco system time saving...
[*05/12/2018 00:00:00.3266] GCM-128 POST passed
[*05/12/2018 00:00:00.3267] GCM-256 POST passed
Starting Cisco act2 check...
Starting Cisco S10 boot service...
Starting Security Auditing Service...
[*05/12/2018 00:00:03.8011] parameter passed without an option given
[ OK ] Started Security Auditing Service.
[05/12/2018 00:00:04.7400] PCI: enabling device 0000:00:02.0 (0140 -> 0143)
[05/12/2018 00:00:04.8300] PCI: enabling device 0000:02:00.0 (0140 -> 0142)
[ OK ] Started Cisco pkg install service.
[05/12/2018 00:00:11.4600] PCI: enabling device 0000:00:03.0 (0140 -> 0143)
[05/12/2018 00:00:11.5400] PCI: enabling device 0000:03:00.0 (0140 -> 0142)
[ OK ] Started Cisco act2 check.
[*** ] A start job is running for Cisco S10 boot service (21s / 1min 37s)[*05/12 /2018 00:00:04.0098] 0
[*05/12/2018 00:00:04.1960] Active version: 8.10.190.0
[*05/12/2018 00:00:04.1989] Backup version: 0.0.0.0
[*05/12/2018 00:00:04.4920] device wired0 entered promiscuous mode
[*05/12/2018 00:00:04.9111] wifi0: Initializing TX ring
[*05/12/2018 00:00:04.9186] wifi0: prepare firmware downloading
[*05/12/2018 00:00:05.7432] wifi0: Firmware downloading
[*05/12/2018 00:00:06.5536]
[*05/12/2018 00:00:06.5536]
[*05/12/2018 00:00:06.5536]
[*05/12/2018 00:00:06.5536]
[*05/12/2018 00:00:06.5536] Authenticating ACT2 ID:AP2800
[*05/12/2018 00:00:06.5536] Authenticating SN:FDO22190K07
[*05/12/2018 00:00:06.5541]
[*05/12/2018 00:00:06.5542] (ACT2Boot) Hardware is Cisco (c) authentic
[*05/12/2018 00:00:06.5542]
[*05/12/2018 00:00:06.5730] Seeding /dev/urandom from ACT2
[*05/12/2018 00:00:07.3589] 0;<30>systemd[1]: Started Cisco pkg install service.
[*05/12/2018 00:00:07.9968] 0;Reading ACT2 SUDI certificates
[*05/12/2018 00:00:08.5068] 0;0;0;0;0;
[*05/12/2018 00:00:10.9748] Firmware download complete
[*05/12/2018 00:00:11.1728] wifi0: Getting HW specs. Retrying for 3
[*05/12/2018 00:00:11.1731] Mac_Init
[*05/12/2018 00:00:11.1734] Preparing to set HW specs
[*05/12/2018 00:00:11.1737] FIPS enc aes-ccmp Verified
[*05/12/2018 00:00:11.1738] FIPS dec aes-ccmp Verified
[*05/12/2018 00:00:11.1739] FIPS enc aes-ccmp-256 Verified
[*05/12/2018 00:00:11.1740] FIPS dec aes-ccmp-256 Verified
[*05/12/2018 00:00:11.1741] FIPS enc aes-gcmp Verified
[*05/12/2018 00:00:11.1742] FIPS dec aes-gcmp Verified
[*05/12/2018 00:00:11.1743] FIPS enc aes-gcmp-256 Verified
[*05/12/2018 00:00:11.1744] FIPS dec aes-gcmp-256 Verified
[*05/12/2018 00:00:11.6273] wifi1: Initializing TX ring
[*05/12/2018 00:00:11.6349] wifi1: prepare firmware downloading
[*05/12/2018 00:00:12.4555] wifi1: Firmware downloading
[*05/12/2018 00:00:13.6768] 0;0;0;0;0;0;0;
[*05/12/2018 00:00:16.8188] Firmware download complete
[*05/12/2018 00:00:17.0128] wifi1: Getting HW specs. Retrying for 3
[*05/12/2018 00:00:17.0131] Mac_Init
[*05/12/2018 00:00:17.0134] Preparing to set HW specs
[*05/12/2018 00:00:17.0137] FIPS enc aes-ccmp Verified
[*05/12/2018 00:00:17.0138] FIPS dec aes-ccmp Verified
[*05/12/2018 00:00:17.0139] FIPS enc aes-ccmp-256 Verified
[*05/12/2018 00:00:17.0140] FIPS dec aes-ccmp-256 Verified
[*05/12/2018 00:00:17.0141] FIPS enc aes-gcmp Verified
[*05/12/2018 00:00:17.0142] FIPS dec aes-gcmp Verified
[*05/12/2018 00:00:17.0143] FIPS enc aes-gcmp-256 Verified
[*05/12/2018 00:00:17.0144] FIPS dec aes-gcmp-256 Verified
[*05/12/2018 00:00:18.0897] Last reload reason : 0: unknown
[ OK ] Started Cisco S10 boot service.
Starting Cisco nss service...
[*05/12/2018 00:00:21.0987] Initializing NSS Firmware
Starting Cisco watchdog...
Starting Cisco nolan service...
[ OK ] Started Cisco watchdog.
[ OK ] Started Cisco nolan service.
Starting Cisco nolan USB service...
[ OK ] Started Cisco nolan USB service.
[ OK ] Started Cisco pre-PoE negotiation.
Starting Cisco pre-PoE negotiation...
[ OK ] Started Cisco nss service.
Starting Cisco Kclick...
Starting Cisco S15platform...
[05/12/2018 00:00:24.6100] Disabling lock debugging due to kernel taint
[ OK ] Started Cisco S15platform.
[ OK ] Started Cisco Kclick.
Starting Cisco certificate create service...
Starting Cisco ME service...
Starting Cisco Kclick...
Starting Cisco Sku service...
[*05/12/2018 00:00:24.8237] stile module dp init successfully
[*05/12/2018 00:00:24.8237] NBAR Label:BLD_POLARIS_DEV_LATEST_20200414_053707 Modu le timestamp: Sep 28 2023 at 00:11:19 User: ttaichol
[*05/12/2018 00:00:27.6568] SKU to radio FW
[*05/12/2018 00:00:27.6615] grep: /storage/config.mobexp: No such file or director y
[*05/12/2018 00:00:27.7529] wifi2 no private ioctls.
[*05/12/2018 00:00:27.7530]
[ OK ] Started Cisco ME service.
[*05/12/2018 00:00:28.1451] Create symlink /dev/mic_cert result 0
[*05/12/2018 00:00:28.1451] Loading certificates from storage...
[*05/12/2018 00:00:28.1786] barbados /tmp/SC4_1dbm.txt
[ OK ] Started Cisco certificate create service.
[ OK ] Started Cisco Sku service.
Starting Cisco certtime service...
[ OK ] Started Cisco Tam service.
Starting Cisco Tam service...
[ OK ] Started Cisco Kclick.
Starting Cisco syslog service...
[*05/12/2018 00:00:29.4173] /etc/init.d/S16mod_load: line 98: can't create /storag e/syslogs/1.modlog: nonexistent directory
[*05/12/2018 00:00:29.4725] Creating default base capwap config.
Starting rsyslog client...
Starting Cisco S16platfomm...
[ OK ] Started Cisco syslog service.
[ OK ] Started rsyslog client.
[ OK ] Started Cisco S16platfomm.
Starting Cisco synclogd...
[ OK ] Started Cisco synclogd.
[ *** ] A start job is running for Cisco certtime service (39s / 1min 32s)[*05/12 /2018 00:00:34.5828]
[*05/12/2018 00:00:34.5828] Click sched monitor: schedulers = 1
[*05/12/2018 00:00:34.7493] starting tam services using ACT2...
[*05/12/2018 00:00:34.7493]
[*05/12/2018 00:00:34.7493] TAM init done
[*05/12/2018 00:00:35.0730] tams_init: INIT app Logging into TAM Server
[*05/12/2018 00:00:35.3715] tams_init: Init App login SUCCESS
[ OK ] Started Cisco certtime service.
[ OK ] Reached target System Initialization.
[ OK ] Reached target Paths.
[ OK ] Reached target Basic System.
Starting Cisco printkd...
Starting Cisco led service...
Starting AP Trace daemon...
Starting WCPD process...
[ OK ] Started NTP_PROC daemon.
Starting NTP_PROC daemon...
Starting OpenSSH ecdsa Server Key Generation...
Starting Cisco klogd...
[ OK ] Started Serial Getty on ttyS0.
Starting Serial Getty on ttyS0...
[ OK ] Reached target Login Prompts.
Starting OpenSSH rsa Server Key Generation...
Starting Clean Air daemon...
Starting Cisco kexec...
Starting Cisco rtd service...
Starting RSYNC User Key Generation...
Starting Fast CGI daemon...
Starting Cisco image/firmware updater service...
[ OK ] Started System Monitor service.
Starting System Monitor service...
Starting Marvell Forwarding daemon...
Starting OpenSSH ed25519 Server Key Generation...
Starting capwapd...
Starting Cisco brain service...
Starting gRPC server daemon...
Starting Hostapd process...
[ OK ] Started Cisco led service.
[ OK ] Started OpenSSH ecdsa Server Key Generation.
[ OK ] Started Cisco kexec.
[ OK ] Started Cisco image/firmware updater service.
[ OK ] Started OpenSSH ed25519 Server Key Generation.
[ OK ] Started Cisco printkd.
[ OK ] Started Cisco klogd.
[ OK ] Started Cisco rtd service.
[ OK ] Started AP Trace daemo [05/12/2018 00:00:49.3200] wired0 emac 2: link up
[05/12/2018 00:00:49.3700] wired0: link up
[*05/12/2018 00:00:50.1321] ip6_port srcr2, ip6local fe80::4201:7aff:fe4b:70dc, ip 6 ::, plen 0, gw6 ::, gw6_mac 00:00:00:00:00:00, mtu 1500, vid 0, mode6 2(slaac)
[*05/12/2018 00:00:50.2619] device aptrace0 entered promiscuous mode
[*05/12/2018 00:00:50.3330] device to_container entered promiscuous mode
[*05/12/2018 00:00:50.6475] pid 7934's current affinity mask: 3
[*05/12/2018 00:00:50.6475] pid 7934's new affinity mask: 1
[*05/12/2018 00:00:51.7784] chpasswd: password for user changed
[*05/12/2018 00:00:52.4630] DOT11_TXP[0]:Domain configured: 12 class:E
[*05/12/2018 00:00:52.9203] DOT11_TXP[0]:Domain configured: 1 class:E
[*05/12/2018 00:00:53.0015] DOT11_TXP[0]:Domain configured: 12 class:E
[*05/12/2018 00:00:53.4830] DOT11_TXP[0]:Domain configured: 1 class:E
[*05/12/2018 00:00:53.5664] DOT11_TXP[0]:Domain configured: 12 class:E
[*05/12/2018 00:00:54.7783] DOT11_DRV[0]: Stopped Radio 0
[*05/12/2018 00:00:54.8035] DOT11_DRV[0]: DOT11_DRV[0]: set_channel Channel set to 6
[*05/12/2018 00:00:55.7079] DOT11_DRV[1]: Stopped Radio 1
[*05/12/2018 00:00:55.7314] DOT11_DRV[1]: DOT11_DRV[1]: set_channel Channel set to 36
[*05/12/2018 00:00:56.4838] DOT11_DRV[0]: DOT11_DRV[0]: set_channel Channel set to 1
[*05/12/2018 00:00:57.2113] DOT11_DRV[0]: DOT11_DRV[0]: set_channel Channel set to 1
[*05/12/2018 00:00:57.2397] DOT11_DRV[0]: DOT11_DRV[0]: set_channel Channel set to 1
[*05/12/2018 00:00:58.0934] bftype is 6
[*05/12/2018 00:00:58.1165] DOT11_DRV[0]: DOT11_DRV[0]: set_channel Channel set to 1
[*05/12/2018 00:00:58.7710] DOT11_DRV[0]: DOT11_DRV[0]: set_channel Channel set to 1
[*05/12/2018 00:00:58.9170] failed wlconfig_commit netdev = wifi1
[*05/12/2018 00:00:58.9170] Error for wireless request "Commit changes" (8B00) :
[*05/12/2018 00:00:58.9171] SET failed on device wifi1 ; Operation not permitt ed.
[*05/12/2018 00:00:59.2134] DOT11_DRV[1]: DOT11_DRV[1]: set_channel Channel set to 36
[*05/12/2018 00:00:59.7675] DOT11_DRV[1]: DOT11_DRV[1]: set_channel Channel set to 36
[*05/12/2018 00:01:00.5063] DOT11_DRV[1]: DOT11_DRV[1]: set_channel Channel set to 36
[*05/12/2018 00:01:00.5239] DOT11_DRV[1]: DOT11_DRV[1]: set_channel Channel set to 36
[*05/12/2018 00:01:01.4806] bftype is 6
[*05/12/2018 00:01:01.5038] DOT11_DRV[1]: DOT11_DRV[1]: set_channel Channel set to 36
[*05/12/2018 00:01:02.2539] DOT11_DRV[1]: DOT11_DRV[1]: set_channel Channel set to 36
[*05/12/2018 00:01:03.0115] DOT11_DRV[1]: DOT11_DRV[1]: set_channel Channel set to 36
[05/12/2018 00:01:29.9800] wired0 emac 2: link up
[05/12/2018 00:01:30.0300] wired0: link up

[marce1000]

           - Are these APs (still) getting enough power ?
           - Check controller logs too when they try to join.

 M.

[balaji.bandi]

If they are working before Upgrade and only failing After Upgrade.  ( so i do not suspect anything related to Power - hope your switch have enough power to boot up the AP full). - worth checking on switch show power inline

Layer 2 i can see a mac address but at our L3 core i have no arp entry.   - This sounds like you are not getting IP address from your DHCP ?

where your DHCP see is the AP MAC address ask for IP address request ?

After AP boot do you see the any Prompt ? then check did the AP got any IP address ?

if the AP got IP address you see you can try manual joining the WLC - check is that works ?

[Leo Laohoo]

Do this: 

configure boot path 1

And then reboot the AP.

[Rich R]

Your logs only go as far as "wired0: link up" - after that AP should get IP via DHCP and then attempt to discover and join the WLC.

Capture the complete logs showing the discovery and join attempts and attach to your reply as a .txt file rather than pasting hundreds of lines of logs into your message.