Cisco AIR-AP2802I-E-K9 Boot loop Issue

[Patrick716]

Hello im have this issue where it completely boot looping every time. I've boot into part2 and boot fine connects to my other 2802I which is setup up as WLC. Then begins to download the image from my tftp after this is done reboots and get stuck in the boot loop of part1.

Here is my boot log.

BootROM - 1.78
Booting from SPI flash, Secure mode
BootROM: RSA Public key verification PASSED
BootROM: CSK block signature verification PASSED
BootROM: Boot header signature verification PASSED
BootROM: Box ID verification PASSED
BootROM: JTAG is disabled

General initialization - Version: 1.0.0
Detected Device ID 6920
Master bootloder version 1.40
High speed PHY - Version: 2.0
BoardId = 0x21board SerDes lanes topology details:
| Lane # | Speed| Type |
------------------------------|
| 1 | 0 | SGMII1 |
| 2 | 5 | PCIe1 |
| 4 | 5 | PCIe2 |
| 5 | 0 | SGMII2 |
-------------------------------
:** Link is Gen1, check the EP capability
PCIe, Idx 1: Link upgraded to Gen2 based on client cpabilities
:** Link is Gen1, check the EP capability
PCIe, Idx 2: Link upgraded to Gen2 based on client cpabilities
High speed PHY - Ended Successfully
DDR4 Training Sequence - Ver TIP-0.23.(Sublib 0.8)0
DDR4 Training Sequence - Switching XBAR Window to FastPath Window
DDR4 Training Sequence - Ended Successfully
BootROM: Image checksum verification PASSED
BootROM: Boot image signature verification PASSED

____ _
/ ___|(_) ___ ___ ___
| | | |/ __| / __|/ _ \
| |___ | |\__ \| (__| (_) |
\____||_||___/ \___|\___/
_ _ ____ _
| | | | | __ ) ___ ___ | |_
| | | |___| _ \ / _ \ / _ \| __|
| |_| |___| |_) | (_) | (_) | |_
\___/ |____/ \___/ \___/ \__|
** LOADER **

U-Boot 2013.01-gcb01933d4 (Nov 10 2020 - 23:41:12) SDK version: 2015_T2.0p10

Board: Barbados-2K
SoC: MV88F6920 Rev A1
running 2 CPUs
CPU: ARM Cortex A9 MPCore (Rev 1) LE
CPU 0
CPU @ 1800 [MHz]
L2 @ 900 [MHz]
TClock @ 250 [MHz]
DDR4 @ 900 [MHz]
DDR4 32 Bit Width,FastPath Memory Access, DLB Enabled, ECC Disabled
DRAM: 1 GiB

RST I2C0
NAND: ECC 4bits, 256 MiB
SF: Detected MX25L3205D with page size 64 KiB, total 4 MiB
*** Warning - bad CRC, using default environment

PCI-e 1 (IF 0 - bus 0) Root Complex Interface, Detected Link X1, GEN 2.0
PCI-e 2 (IF 1 - bus 1) Root Complex Interface, Detected Link X1, GEN 2.0

Map: Code: 0x3fed8000:0x3ffae360
BSS: 0x3ffefe24
Stack: 0x3f9c7f20
Heap: 0x3f9c8000:0x3fed8000
U-Boot Environment: 0x00100000:0x00110000 (SPI)

Board configuration:
| port | Interface | PHY address |
|--------|------------|--------------|
| egiga1 | SGMII | 0x01 |
| egiga2 | SGMII | 0x00 |
Net: , egiga1, egiga2 [PRIME]
Hit ESC key to stop autoboot: 0
Creating 1 MTD partitions on "nand0":
0x000000200000-0x000010000000 : "mtd=2"
UBI: attaching mtd1 to ubi0
UBIFS: recovery needed
UBIFS: recovery deferred
UBIFS: mounted UBI device 0, volume 0, name "part1"
Loading file 'part1/part.bin' to addr 0x02000000 with size 6273845 (0x005fbb35)...
Done
SF: Detected MX25L3205D with page size 64 KiB, total 4 MiB

Checking image signing.
Image signing verification success, continue to run...
## Booting kernel from Legacy Image at 02000000 ...
Image Name: Barbados Firmware Image
Created: 2021-03-04 11:15:15 UTC
Image Type: ARM Linux Multi-File Image (uncompressed)
Data Size: 6272837 Bytes = 6 MiB
Load Address: 00008000
Entry Point: 00008000
Contents:
Image 0: 4202632 Bytes = 4 MiB
Image 1: 2054051 Bytes = 2 MiB
Image 2: 16137 Bytes = 15.8 KiB
Verifying Checksum ... OK
## Loading init Ramdisk from multi component Legacy Image at 02000000 ...
## Flattened Device Tree from multi component Image at 02000000
Booting using the fdt at 0x025f787c
Loading Multi-File Image ... OK
OK
reserving fdt memory region: addr=0 size=4000
Loading Device Tree to 00ff9000, end 00ffff08 ... OK

Starting Device Tree update ('fdt_skip_update' = no)

Limit DDR size at 3GB due to power of 2 requirement of Address decoding
Updating device tree successful

Starting kernel ...

[01/01/1970 00:00:00.0000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 260096
[01/01/1970 00:00:00.0000] Memory: 1025868K/1048576K available (5743K kernel code, 411K rwdata, 2476K rodata, 359K init, 496K bss, 22708K reserved, 0K highmem)
[01/01/1970 00:00:00.1200] CPU1: Booted secondary processor
[01/01/1970 00:00:01.4300] buginf tty flushing thread started, ttyport=bf13bc00
[01/01/1970 00:00:01.5500] m25p80 spi1.0: found mx25l3206, expected n25q032
[*01/01/1970 00:00:02.6602] buginf() enabled.
[*01/01/1970 00:00:02.6698] Made it into bootsh: Mar 4 2021 10:13:22 T-0b7bc0faa97ae8a26ea605c7622aac357397184c-g0b7bc0fa-aut
[*01/01/1970 00:00:03.8512] verify signature failed for /bootpart/part1/ramfs_data_cisco.cpio.lzma
[*01/01/1970 00:00:03.8512] bootsh mini ramfs booted /bootpart/part1/ramfs_data_cisco.cpio.lzma
[*01/01/1970 00:00:21.5243] lzma: unexpected EOF
[*01/01/1970 00:00:21.5263] Uncompressing lzma file: /bootpart/part1/ramfs_data_cisco.cpio.lzma: File exists
[*01/01/1970 00:00:21.5263] Fatal error: failed to start the image. Please fall back to alternate partition...
[01/01/1970 00:00:21.8100] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100
[01/01/1970 00:00:21.8100]
[01/01/1970 00:00:21.8100] CPU1: stopping
[01/01/1970 00:00:21.8100] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 3.14.33 #1
[01/01/1970 00:00:21.8100] Backtrace:
[01/01/1970 00:00:21.8100] [<80109fd8>] (dump_backtrace) from [<8010a2a4>] (show_stack+0x18/0x1c)
[01/01/1970 00:00:21.8100] r6:00000000 r5:809c9174 r4:00000000 r3:00200040
[01/01/1970 00:00:21.8100] [<8010a28c>] (show_stack) from [<8068d0c8>] (dump_stack+0x70/0x8c)
[01/01/1970 00:00:21.8100] [<8068d058>] (dump_stack) from [<8010d0b0>] (handle_IPI+0xe0/0x238)
[01/01/1970 00:00:21.8100] r4:00000001 r3:80973a58
[01/01/1970 00:00:21.8100] [<8010cfd0>] (handle_IPI) from [<801004d4>] (gic_handle_irq+0x58/0x60)
[01/01/1970 00:00:21.8100] r7:bf0adfac r6:8096b434 r5:bf0adf78 r4:00000005
[01/01/1970 00:00:21.8100] [<8010047c>] (gic_handle_irq) from [<80694100>] (__irq_svc+0x40/0x50)
[01/01/1970 00:00:21.8100] Exception stack(0xbf0adf78 to 0xbf0adfc0)
[01/01/1970 00:00:21.8100] df60: bf7e66e0 00000000
[01/01/1970 00:00:21.8100] df80: 0000ba68 801161a0 bf0ac000 bf0ac000 10c03c7d 809c9194 0000406a 414fc091
[01/01/1970 00:00:21.8100] SMP: failed to stop secondary CPUs
[01/01/1970 00:00:21.8100] Rebooting in 5 seconds..
[01/01/1970 00:00:21.8100] dfa0: 00000000 bf0adfcc bf0adfd0 bf0adfc0 80106eb8 80106ebc 60000013 ffffffff
[01/01/1970 00:00:21.8100] r6:ffffffff r5:60000013 r4:80106ebc r3:80106eb8
[01/01/1970 00:00:21.8100] [<80106e88>] (arch_cpu_idle) from [<80160e68>] (cpu_startup_entry+0x138/0x1cc)
[01/01/1970 00:00:21.8100] [<80160d30>] (cpu_startup_entry) from [<8010cd54>] (secondary_start_kernel+0x124/0x144)
[01/01/1970 00:00:21.8100] [<8010cc30>] (secondary_start_kernel) from [<00100564>] (0x100564)
[01/01/1970 00:00:21.8100] r4:3f09406a r3:8010054c

[Rich R]

Clearly the software on part1 is corrupted because it all goes wrong after "verify signature failed for /bootpart/part1/ramfs_data_cisco.cpio.lzma".

What version is installed in part2?
What version is your ME WLC running (which is what it should be installing from TFTP)?

Have you tried manually installing the correct code version on the AP?

If you are trying to upgrade from a very old version (eg 8.3) to a very new version (eg 8.10) then try updating it to a 8.5.182.0 - 15.3(3)JF15 - version first manually.
https://software.cisco.com/download/home/286304510/type/286288051/release/15.3.3-JF15

[Patrick716]

The version of the ME WLC is 8.10.151.0 then I am using the corresponding image package. 

Then the version of part2 is 8.5.120.0

memory. -AP2802I-E-K9 ARMv7 Processor rev 1 (v7l) with 1028520/672736K bytes of--More--
Processor board ID FCW2315P3RB
AP Running Image : 8.5.120.0
Primary Boot Image : 8.5.120.0
Backup Boot Image : 0.0.0.0
2 Gigabit Ethernet interfaces
2 802.11 Radios
Radio Driver version : 9.0.5.5-W8964
Radio FW version : 9.1.8.1
NSS FW version : 2.4.18

Base ethernet MAC Address : 78:BC:1A:1C:FF:DC
Part Number : 0-000000-00
PCA Assembly Number : 800-105674-01
PCA Revision Number : A0
PCB Serial Number : FOC23137V11
Top Assembly Part Number : 800-105674-01
Top Assembly Serial Number : FCW2315P3RB
Top Revision Number : A0
Product/Model Number : AIR-AP2802I-E-K9

This is what happens when it trys to install the update from the ME WLC.

[*07/19/2022 08:13:18.2548] CAPWAP State: Image Data
[*07/19/2022 08:13:18.2922] do PRECHECK, part2 is active part
[*07/19/2022 08:13:18.5069] Image Data Request sent to 192.168.3.150
[*07/19/2022 08:13:26.6406] ..................................................
[*07/19/2022 08:13:36.3574] ..................................................
[*07/19/2022 08:13:44.8355] ..................................................
[*07/19/2022 08:13:53.2421] ..................................................
[*07/19/2022 08:14:01.8172] ..................................................
[*07/19/2022 08:14:10.5093] .........
[*07/19/2022 08:14:11.9851] Successfully downloaded image.
[*07/19/2022 08:14:12.0216] do PREDOWNLOAD, part2 is active part
[*07/19/2022 08:14:12.7892] Image signing verify success.
[*07/19/2022 08:14:14.3042] upgrade.sh: part to upgrade is part1
[*07/19/2022 08:14:30.4935] tar: write error: No space left on device
[*07/19/2022 08:14:30.4954] upgrade.sh: Command "tar -xf /tmp/part.tar -C /bootpart/part1" failed.
[*07/19/2022 08:14:30.4970] upgrade.sh: df /bootpart/part1
[*07/19/2022 08:14:30.5002] upgrade.sh: du -h /bootpart/part1
[*07/19/2022 08:14:30.5048] upgrade.sh: Writing image to disk...
[*07/19/2022 08:14:30.5692] sh: 1K-blocks: unknown operand
[*07/19/2022 08:14:30.6678] upgrade.sh: AP backup version: 8.10.151.0
[*07/19/2022 08:14:30.7306] do ACTIVATE, part2 is active part
[*07/19/2022 08:14:30.8326] upgrade.sh: activate part1, set BOOT to part1
[*07/19/2022 08:14:31.4569] upgrade.sh: AP primary version: 8.10.151.0
[*07/19/2022 08:14:31.4584] upgrade.sh: AP backup version: 8.5.120.0
[*07/19/2022 08:14:34.4640] AP Rebooting: Reset Reason - Image Upgrade

Will their be a way of recovering/reflashing the part1?

[Rich R · Accepted solution]

That free space problem is a well known problem with ME upgrades.

Just install the standard CAPWAP image I provided the link to - that should then go into part1 then once it's booted on part1 you should then be able to convert it back to ME using the correct ME version.

[Patrick716]

Okay thank you for your help i will try this and get back to you if it worked or not

[Patrick716]

rruddling this fixed the issue for part1. But I had to use this version ap3g3-k9w8-tar.153-3.JPK2 because I don't have access to that file on my account. My next problem now is it not automatically connecting to my ME WLC sadly. 

[matemat]

Can you describe exactly the CAPWAP image installations when the device does not boot?

[Rich R]

what model of AP are you referring to @matemat ?

[matemat]

It is AIR-AP2802E-E-K9

[Rich R]

Well first it would be useful to see why the AP isn't booting.
Capture the complete console log from the AP from power on, save to a text file (.txt) and attach the file here (do not paste the logs here please).
It could be that the AP is faulty and needs RMA.

[Rich R]

That's the 17.6.1 image which is too new (change in CAPWAP protocol) - so you either need to use an older CAPWAP image like the one I recommended or upgrade your ME to 8.10.171.0 https://software.cisco.com/download/home/286304510/type/286289839/release/8.10.171.0

[Patrick716]

Thank you a lot for you help much appreciated i have found a version that has worked now 

[Rich R]

Great - glad we could help.