2802 Stuck trying to update to ME Image

[uniqueandshort]

I have an issue with one out of a bunch of other 2802's when converting to ME via the WLC. 

I believe the issue is related to other posts about the version and the lack of space available but none of the other posts get stuck where i do at trying to erase the storage and or putting on a recovery image (capwap or ME).

It's connecting to the WLC but i think the below is the source of the issue:

[*04/30/2025 10:24:58.3007] CAPWAP State: Image Data
[*04/30/2025 10:24:58.3011] AP image version 8.10.196.0 backup 0.0.0.0, Controller 8.10.196.0
[*04/30/2025 10:24:58.3012] Version is the same, but need to update to ME Image.

Any help or pointers very appreciated, thanks. Here is the rest of the output if that helps (then it boot loops):

[*04/30/2025 10:24:58.3007] CAPWAP State: Image Data
[*04/30/2025 10:24:58.3011] AP image version 8.10.196.0 backup 0.0.0.0, Controller 8.10.196.0
[*04/30/2025 10:24:58.3012] Version is the same, but need to update to ME Image.
[*04/30/2025 10:24:58.3012] wtpImgFileReadRequest: request ap3g3, local /tmp/part.tar
[*04/30/2025 10:24:58.3025] Image Data Request sent to 192.168.xxx.xxx, fileName [ap3g3], slaveStatus 0
[*04/30/2025 10:24:58.3225] Image Data Response from 192.168.xxx.xxx
[*04/30/2025 10:24:58.3226] AC accepted join request with result code: 0
[*04/30/2025 10:25:06.3780] <..........................................
[*04/30/2025 10:25:06.3780] <..................................................
[*04/30/2025 10:25:17.8858] .................Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Image Data(10).
[*04/30/2025 10:25:21.7525] .................................
[*04/30/2025 10:25:29.1040] ..................................................
[*04/30/2025 10:25:40.2916] ..................................................
[*04/30/2025 10:25:51.4289] ..................<30>systemd[1]: Starting Pnp Watcher...
[*04/30/2025 10:25:55.3675] Warning: Stopping dbg_day0_bundle.service, but it can still be activated by:
[*04/30/2025 10:25:55.3676] dbg_day0_bundle.timer
[*04/30/2025 10:25:55.3914] ...............<30>systemd[1]: Stopped Pnp agent.
[*04/30/2025 10:25:59.2856] .................
[*04/30/2025 10:26:03.0401] ..........> 68933120 bytes, 52065 msgs, 384 last
[*04/30/2025 10:26:05.2171] Last block stored, IsPre 0, WriteTaskId 0
[*04/30/2025 10:26:05.2172] Image transfer completed from WLC, last 1
[*04/30/2025 10:26:05.2174] wtpProcessImageDataRequest(10): fileName ap3g3, pre 0
[*04/30/2025 10:26:05.2472] upgrade.sh: Script called with args:[PREDOWNLOAD]
[*04/30/2025 10:26:05.3113] do PREDOWNLOAD, part2 is active part
[*04/30/2025 10:26:05.3204] upgrade.sh: Creating before-upgrade.log
[*04/30/2025 10:26:05.3391] upgrade.sh: Start doing upgrade arg1=PREDOWNLOAD arg2= arg3= ...
[*04/30/2025 10:26:05.4156] upgrade.sh: Using image /tmp/part.tar on barbados ...
[*04/30/2025 10:26:08.7126] Image signing verify success.
[*04/30/2025 10:26:08.7395] upgrade.sh: part to upgrade is part1
[*04/30/2025 10:26:08.7542] upgrade.sh: AP version1: part1 , img 8.10.196.0
[*04/30/2025 10:26:08.8529] sh: SC4_1dbm.txt: unknown operand
[*04/30/2025 10:26:09.0492] root: Adjusting boot partition "part1" size
[04/30/2025 10:26:13.4600] UBI error: ubi_create_volume: not enough PEBs, only 493 available
[04/30/2025 10:26:13.5600] UBI error: ubi_create_volume: cannot create volume 0, error -28
[*04/30/2025 10:26:13.6856] Set volume size to 62599168
[*04/30/2025 10:26:13.6856] Volume ID 0, size 493 LEBs (62599168 bytes, 59.7 MiB), LEB size 126976 bytes (124.0 KiB), dynamic, name "part1", alignment 1
[*04/30/2025 10:26:14.7058] mknod: /dev/ubivol/part1: File exists
[*04/30/2025 10:26:15.7142] mknod: /dev/ubivol/part1: File exists
[*04/30/2025 10:26:16.7232] mknod: /dev/ubivol/part1: File exists
[*04/30/2025 10:26:17.7348] mknod: /dev/ubidev/part1: File exists
[*04/30/2025 10:26:18.7431] mknod: /dev/ubidev/part1: File exists
[*04/30/2025 10:26:19.7515] mknod: /dev/ubidev/part1: File exists
[*04/30/2025 10:26:20.8280] /tmp
[*04/30/2025 10:26:20.8993] sh: SC4_1dbm.txt: unknown operand
[*04/30/2025 10:26:21.0297] upgrade.sh: Extracting and verifying image in part1...
[*04/30/2025 10:26:21.0384] upgrade.sh: BOARD generic case execute
[*04/30/2025 10:26:21.1142] upgrade.sh: mount ubifs /dev/ubivol/part1 as /bootpart, status=0
[*04/30/2025 10:26:21.1257] upgrade.sh: Untar /tmp/part.tar to /bootpart/part1...
[*04/30/2025 10:26:36.2264] tar: write error: No space left on device
[*04/30/2025 10:26:36.2356] upgrade.sh: Failed to untar: 1
[*04/30/2025 10:26:36.2392] Filesystem 1K-blocks Used Available Use% Mounted on
[*04/30/2025 10:26:36.2392] /dev/ubivol/part1 54704 54700 0 100% /bootpart
[*04/30/2025 10:26:36.2434] 54.8M /bootpart/part1
[*04/30/2025 10:26:36.2523] upgrade.sh: Sync image to disk...
[*04/30/2025 10:26:36.3229] upgrade.sh: Image extraction and verification failed. Retry count 1
[*04/30/2025 10:26:39.5608] Image signing verify success.

[marce1000]

  - The partition (part1) is full ; check if this or a similar procedure can help you :
     https://community.cisco.com/t5/wireless/recovering-an-air-ap2802i-no-space-left-on-device/m-p/3055110/highlight/true#M89939

   M.

[Rich R]

Download the ME tar file and put it on your TFTP server:
https://software.cisco.com/download/home/286304510/type/286289839/release/8.10.196.0
Then convert the AP from CAPWAP to ME with the file on the the TFTP server:
https://www.cisco.com/c/en/us/td/docs/wireless/controller/technotes/8-2/b_Mobility_Express_Deployment_guide/b_Mobility_Express_Deployment_guide_chapter_01100.html#task_7B18649C99D94D2BB551EDDEB16B6601
https://www.cisco.com/c/en/us/support/docs/wireless/mobility-express-aironet-access-points/215410-convert-capwap-1852-access-point-ap-to.html

Hopefully this way the install should work.

[uniqueandshort]

Thanks to you both. I will try this and let you know

[uniqueandshort]

Forgive me for trying a new WLC first before trying your suggestions above, but this is interesting now...

[*05/01/2025 10:58:25.1117] CAPWAP State: Image Data
[*05/01/2025 10:58:25.1122] AP image version 8.10.196.0 backup 0.0.0.0, Controller 8.10.196.0
[*05/01/2025 10:58:25.1123] Version is the same, do not need update.

 - notice the "Version is the same, but need to update to ME Image." has changed to "Version is the same, do not need update."

The AP now successfully connects to the new WLC and runs. BUT going back to what I intend to do is make this AP its own controller by hitting the "Convert to ME" button in the WLC which is how i ended up in the mess in the first place. 

Having read some other articles, would anyone agree that this sounds like a memory problem in part1 and so whenever I try a re-image or transfer image etc involving Part1, it throws a bunch of errors?

######### EDIT / UPDATE ##########

After further testing I followed suggested article from marce1000 and used as similar version as suggested and this brought the image version in backup from 0.0.0.0 to 8.5.161.0 The AP booted ok and connected to the WLC ok

From the WLC, i hit "Convert to ME", i notice a console output that said it cant do this as its not a ME image. So i tried this from cli:

ap-type mobility-express tftp://192.168.7.4/AIR-AP2800-K9-ME-8-5-140-0.tar but get the usual disk full error. So I tried AIR-AP2800-K9-ME-8-10-151-0.tar with the same error then i tried AIR-AP2800-K9-ME-8-10-196-0.tar but all have the same error:

It may take a few minutes. If longer, please abort command, check network and try again.
######################################################################## 100.0%
Image download completed.
Upgrading ...
upgrade.sh: Script called with args:[NO_UPGRADE]
do NO_UPGRADE, part2 is active part
upgrade.sh: Script called with args:[-c CHECK_ME]
do CHECK_ME, part2 is active part
upgrade.sh: Creating before-upgrade.log
upgrade.sh: Start doing upgrade arg1=CHECK_ME arg2=,from_cli arg3= ...
upgrade.sh: Using image /tmp/cli_part.tar on barbados ...
Image signing verify success.
upgrade.sh: part to upgrade is part1
upgrade.sh: AP version1: part1 , img 8.10.196.0
sh: SC4_1dbm.txt: unknown operand
root: Adjusting boot partition "part1" size
[05/01/2025 12:18:33.6600] UBI error: ubi_create_volume: not enough PEBs, only 445 available
[05/01/2025 12:18:33.7700] UBI error: ubi_create_volume: cannot create volume 0, error -28
Set volume size to 56504320
Volume ID 0, size 445 LEBs (56504320 bytes, 53.9 MiB), LEB size 126976 bytes (124.0 KiB), dynamic, name "part1", alignment 1
mknod: /dev/ubivol/part1: File exists
mknod: /dev/ubivol/part1: File exists
mknod: /dev/ubivol/part1: File exists
mknod: /dev/ubidev/part1: File exists
mknod: /dev/ubidev/part1: File exists
mknod: /dev/ubidev/part1: File exists
/tmp
sh: SC4_1dbm.txt: unknown operand
upgrade.sh: Extracting and verifying image in part1...
upgrade.sh: BOARD generic case execute
upgrade.sh: mount ubifs /dev/ubivol/part1 as /bootpart, status=0
upgrade.sh: Untar /tmp/cli_part.tar to /bootpart/part1...
tar: write error: No space left on device
upgrade.sh: Failed to untar: 1
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/ubivol/part1 49200 49192 0 100% /bootpart
49.3M /bootpart/part1
upgrade.sh: Sync image to disk...
upgrade.sh: Image extraction and verification failed. Retry count 1

[Rich R]

What version was it running when you tried to do the conversions?
8.5.140.0 and 8.5.161.0 are both older versions. 
Make sure you have 8.5.182.0 - 15.3(3)JF15 - installed and running before you try to do the conversion.

[uniqueandshort]

Thanks Rich. I feel I am getting closer and now it's just my own confusion over how the CAPWAP and ME versions interact. Tonight I installed these in order (without any issues):

archive download-sw /reload tftp://192.168.x.x/ap3g3-k9w8-tar.153-3.JD16.tar
archive download-sw /reload tftp://192.168.x.x/ap3g3-k9w8-tar.153-3.JF12.tar
archive download-sw /reload tftp://192.168.x.x/ap3g3-k9w8-tar.153-3.JK1.tar

Then i installed a ME image:

ap-type mobility-express tftp://192.168.x.x/AIR-AP2800-K9-ME-8-5-140-0.tar

Then i connected it to the WLC which runs 8.10.196. While it did connect to the controller to check versions, it  .... and it started boot looping again. 

What am I doing wrong? Is this still because I am trying to jump up too many versions? Even worse, I tried this all again from the top, now it wont install JK1 and boot loops there too now. I have to manually setenv BOOT part2 to get it to stop. 

[uniqueandshort]

After many attempts, I found this article:

https://www.cisco.com/c/en/us/support/docs/wireless/catalyst-9800-series-wireless-controllers/220443-how-to-avoid-boot-loop-due-to-corrupted.html

I removed the part 1 (ubi remove part1) and now it will not write anything (even a lightweight image) to that part. I've since read elsewhere that this renders the part unusable thereafter! Surely Cisco haven't published an article that tells you how to brick your device - Does anyone know how I get an image back on this part? I can boot part 2 ok but cannot install new versions etc because they involve writing to the part 1 in every case? I don't have a service contract as these are second hand

[Rich R]

The AP should only be bricked if you delete both partitions.
As long as it can boot one partition then you should be able to update the other partition.
If it's not allowing you to do that then you could have faulty hardware in which case RMA/replacement is your only option.
However you were previously able to update both partitions so unless something has gone faulty this is more likely a software problem.

Going back to your previous reply you're jumping around different, old versions which is just playing with fire!
archive download-sw /reload tftp://192.168.x.x/ap3g3-k9w8-tar.153-3.JD16.tar = 8.3.143.0
archive download-sw /reload tftp://192.168.x.x/ap3g3-k9w8-tar.153-3.JF12.tar = 8.5.161.0
archive download-sw /reload tftp://192.168.x.x/ap3g3-k9w8-tar.153-3.JK1.tar = 8.10.112.0
Then i installed a ME image: ap-type mobility-express tftp://192.168.x.x/AIR-AP2800-K9-ME-8-5-140-0.tar
Then i connected it to the WLC which runs 8.10.196 - why?

Install 8.5.182.0 CAPWAP - 15.3(3)JF15 then try the conversion to 8.10.196.0 ME.  Do not join it to a WLC again.

You might be better off building yourself a 9800-CL WLC instead of spending all this time trying to get the AP to run ME ...