C9130AXI_E stuck downloading image

[HVO-cisco]

WLC 9800-40
Software 17.9.2
Hello, we have a problem with ap's not joing the controller.
Tried a factory reset, and we log the following:

Any suggestions ?

[*02/10/2023 11:04:27.9330] Reset chip in progess, not resetting
[*02/10/2023 11:04:28.0865] DOT11_TXP[1]:Domain configured: 12 class:E
[*02/10/2023 11:04:29.4183] DOT11_DRV[2]: Init Radio2
[*02/10/2023 11:04:29.4493] DOT11_DRV[1]: Init Radio1
[*02/10/2023 11:04:29.4771] DOT11_DRV[1]: Stop Radio1 - Begin
[*02/10/2023 11:04:29.4803] DOT11_DRV[1]: Stop Radio1 - End
[*02/10/2023 11:04:29.4971] wlan: [5582:I:ANY] acfg_set_chan: 988: VAP is not ready. Saving channel [36] to configure later
[*02/10/2023 11:04:29.4972] DOT11_DRV[1]: set_channel Channel set to 36
[*02/10/2023 11:04:29.5051] DOT11_DRV[0]: Init Radio0
[*02/10/2023 11:04:29.7012] wlan: [5582:I:ANY] acfg_set_chan: 988: VAP is not ready. Saving channel [6] to configure later
[*02/10/2023 11:04:29.7016] DOT11_DRV[0]: set_channel Channel set to 6
[*02/10/2023 11:04:29.7283] wlan: [5582:I:ANY] acfg_set_chan: 988: VAP is not ready. Saving channel [1] to configure later
[*02/10/2023 11:04:29.7288] DOT11_DRV[0]: set_channel Channel set to 1
[*02/10/2023 11:04:29.7313] wlan: [5582:I:ANY] acfg_set_chan: 988: VAP is not ready. Saving channel [1] to configure later
[*02/10/2023 11:04:29.7317] DOT11_DRV[0]: set_channel Channel set to 1
[*02/10/2023 11:04:29.7345] DOT11_DRV[0]: Channel set to 1, width 20
[*02/10/2023 11:04:29.7372] wlan: [0:I:ANY] ol_ath_twt_disable_complete_event_handler: 14954: twt disabled.
[*02/10/2023 11:04:29.7372]
[*02/10/2023 11:04:29.7425] DOT11_DRV[0]: Channel set to 1, width 20
[*02/10/2023 11:04:29.7515] wlan: [5582:I:ANY] acfg_set_chan: 988: VAP is not ready. Saving channel [36] to configure later
[*02/10/2023 11:04:29.7520] DOT11_DRV[1]: set_channel Channel set to 36
[*02/10/2023 11:04:29.7600] wlan: [5582:I:ANY] acfg_set_chan: 988: VAP is not ready. Saving channel [36] to configure later
[*02/10/2023 11:04:29.7604] DOT11_DRV[1]: set_channel Channel set to 36
[*02/10/2023 11:04:29.7633] DOT11_DRV[1]: Channel set to 36, width 20
[*02/10/2023 11:04:29.7647] wlan: [0:I:ANY] ol_ath_twt_disable_complete_event_handler: 14954: twt disabled.
[*02/10/2023 11:04:29.7647]
[*02/10/2023 11:04:29.7691] DOT11_DRV[1]: Channel set to 36, width 20
[*02/10/2023 11:04:31.0377] AP is in good condition, BLE is off
[*02/10/2023 11:04:31.0377] Set radio wifi0 power 4 antenna chain 4 15
[*02/10/2023 11:04:31.0377] Set radio wifi1 power 4 antenna chain 8 255
[*02/10/2023 11:04:32.2903] IOT Bootloader Flash Tool 1.4
[*02/10/2023 11:04:32.2904] Firmware upgrade initiated by ble_management
[*02/10/2023 11:04:32.2904] Opening serial console to.. /dev/ttyMSM1
[*02/10/2023 11:04:32.2904] bundled_fw_ver = ['Lodestar', '2.7.16']
[*02/10/2023 11:04:32.2904] RSP: 04FF10FF070002150AFF020710320000000000
[*02/10/2023 11:04:32.2904] ls_ver = 2.7.16
[*02/10/2023 11:04:32.2904] ls_fw_ver 2.7.16
[*02/10/2023 11:04:32.2904] upgrade false
[*02/10/2023 11:04:32.2904] FW version is latest. No need to upgrade
[*02/10/2023 11:04:33.9916] ethernet_port wired0, ip 192.168.251.77, netmask 255.255.255.0, gw 192.168.251.1, mtu 1500, bcast 192.168.251.255, dns1 X.X.X.X, dns2 XX.XX.XX.16, domain XXX.XX<30>systemd[1]: Starting ntp watcher...
[*02/10/2023 11:04:35.3480] AP IPv4 Address updated from 0.0.0.0 to 192.168.251.77
[*02/10/2023 11:04:46.4362] dtls_init: Use SUDI certificate
[*02/10/2023 11:04:46.4393]
[*02/10/2023 11:04:46.4393] CAPWAP State: Init
[*02/10/2023 11:04:52.0954] Start: RPC thread 2680357776 created.
[*02/10/2023 11:05:43.4148]
[*02/10/2023 11:05:43.4148] CAPWAP State: Discovery
[*02/10/2023 11:05:43.4164] Got WLC address X.X.X.140 from DHCP.
[*02/10/2023 11:05:43.4165] IP DNS query for CISCO-CAPWAP-CONTROLLER.XXX.XX
[*02/10/2023 11:06:03.4412] Discovery Request sent to X.X.X.140, discovery type DHCP(2)
[*02/10/2023 11:06:03.4425] Discovery Response from X.X.X.140
[*02/10/2023 11:06:03.4428] Discovery Request sent to 255.255.255.255, discovery type UNKNOWN(0)
[*02/10/2023 11:06:03.4432]
[*02/10/2023 11:06:03.4432] CAPWAP State: Discovery
[*02/10/2023 11:07:32.0002]
[*02/10/2023 11:07:32.0003] CAPWAP State: DTLS Setup
[*02/10/2023 11:07:32.3662] sudi99_request_check_and_load: Use HARSA SUDI certificate
[*02/10/2023 11:07:32.9994]
[*02/10/2023 11:07:32.9994] CAPWAP State: Join
[*02/10/2023 11:07:33.0082] Sending Join request to X.X.X.140 through port 5272
[*02/10/2023 11:07:33.0287] Join Response from X.X.X.140
[*02/10/2023 11:07:33.0288] AC accepted join request with result code: 0
[*02/10/2023 11:07:33.0295] Received wlcType 0, timer 30
[*02/10/2023 11:07:33.0295] TLV ID 2216 not found
[*02/10/2023 11:07:33.0295] TLV-DEC-ERR-1: No proc for 2216
[*02/10/2023 11:07:33.1196]
[*02/10/2023 11:07:33.1196] CAPWAP State: Image Data
[*02/10/2023 11:07:33.1201] AP image version 8.10.130.0 backup 0.0.0.0, Controller 17.9.2.52
[*02/10/2023 11:07:33.1201] Version does not match.
[*02/10/2023 11:07:33.1452] upgrade.sh: Script called with args:[PRECHECK]
[*02/10/2023 11:07:33.1783] do PRECHECK, part1 is active part
[*02/10/2023 11:07:33.2009] upgrade.sh: /tmp space: OK available 80668, required 40000
[*02/10/2023 11:07:33.2013] wtpImgFileReadRequest: request ap1g6a, local /tmp/part.tar
[*02/10/2023 11:07:33.2026] Image Data Request sent to X.X.X.140, fileName [ap1g6a], slaveStatus 0
[*02/10/2023 11:07:33.2040] Image Data Response from X.X.X.140
[*02/10/2023 11:07:33.2040] AC accepted join request with result code: 0
[*02/10/2023 11:07:33.2082] <.....................Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Image Data(10).
[*02/10/2023 11:07:40.1031] ..........Set PnP NTP Server pnpntpserver.XX.no.
[*02/10/2023 11:07:45.1034] ...................
[*02/10/2023 11:07:55.1778] ..................................................
[*02/10/2023 11:08:20.2025] ...............<30>systemd[1]: Starting Pnp Watcher...
[*02/10/2023 11:08:30.0739] Warning: Stopping dbg_day0_bundle.service, but it can still be activated by:
[*02/10/2023 11:08:30.0739] dbg_day0_bundle.timer
[*02/10/2023 11:08:30.3404] ..........<30>systemd[1]: Stopped Pnp agent.
[*02/10/2023 11:08:34.1157] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Image Data(10).
[*02/10/2023 11:08:36.1378] ..............Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Image Data(10).
[*02/10/2023 11:08:48.5273] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Image Data(10).
[*02/10/2023 11:08:48.7317] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Image Data(10).
[*02/10/2023 11:08:52.0475] ...........
[*02/10/2023 11:08:58.0077] ..................................................
[*02/10/2023 11:09:14.7405] ........................Invalid event 56 & state 10 combination.
[*02/10/2023 11:09:26.9926] Failed to handle timer message.
[*02/10/2023 11:09:27.2703] ......Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Image Data(10).
[*02/10/2023 11:09:28.7518] ....................
[*02/10/2023 11:09:37.9582] ..................................................
[*02/10/2023 11:10:00.5021] .............> 84377600 bytes, 62735 msgs, 370 last
[*02/10/2023 11:10:03.0973] Last block stored, IsPre 0, WriteTaskId 0
[*02/10/2023 11:10:03.0974] Image transfer completed from WLC, last 1
[*02/10/2023 11:10:03.0976] wtpProcessImageDataRequest(10): fileName ap1g6a, pre 0
[*02/10/2023 11:10:03.1220] upgrade.sh: Script called with args:[PREDOWNLOAD]
[*02/10/2023 11:10:03.1535] do PREDOWNLOAD, part1 is active part
[*02/10/2023 11:10:03.1685] upgrade.sh: Start doing upgrade arg1=PREDOWNLOAD arg2= arg3= ...
[*02/10/2023 11:10:03.2211] upgrade.sh: Using image /tmp/part.tar on axel-qca ...
[*02/10/2023 11:10:03.2213] sh: write error: No space left on device
[*02/10/2023 11:10:03.2258] tar: write error: No space left on device
[*02/10/2023 11:10:03.2537] upgrade.sh: ERROR: Image type mismatch. Expected:ap1g6a Got:
[*02/10/2023 11:10:03.2700] upgrade.sh: Cleanup for do_upgrade...
[*02/10/2023 11:10:03.2864] upgrade.sh: /tmp/upgrade_in_progress cleaned
[*02/10/2023 11:10:03.3009] upgrade.sh: Cleanup tmp files ...
[*02/10/2023 11:10:03.3414] capwap-upgrade script returned failure when calling PREDOWNLOAD.
[*02/10/2023 11:10:03.3417] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Image Data(10).
[*02/10/2023 11:10:03.3419] CAPWAP SM handler: Failed to process message type 15 state 10.
[*02/10/2023 11:10:03.3419] Failed to handle capwap control message from controller - status 2
[*02/10/2023 11:10:03.3420] Failed to process encrypted capwap packet 0x5560029000 from X.X.X.140
[*02/10/2023 11:10:03.3420] Failed to send capwap message 0 to the state machine. Packet already freed.
[*02/10/2023 11:10:19.4299] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Image Data(10).
[*02/10/2023 11:10:27.0528] Re-Tx Count=1, Max Re-Tx Value=5, SendSeqNum=7, NumofPendingMsgs=1
[*02/10/2023 11:10:27.0528]
[*02/10/2023 11:10:29.9037] Re-Tx Count=2, Max Re-Tx Value=5, SendSeqNum=7, NumofPendingMsgs=1
[*02/10/2023 11:10:29.9037]
[*02/10/2023 11:10:32.7546] Re-Tx Count=3, Max Re-Tx Value=5, SendSeqNum=7, NumofPendingMsgs=1
[*02/10/2023 11:10:32.7546]
[*02/10/2023 11:10:35.6056] Re-Tx Count=4, Max Re-Tx Value=5, SendSeqNum=7, NumofPendingMsgs=1
[*02/10/2023 11:10:35.6056]
[*02/10/2023 11:10:38.4565] Re-Tx Count=5, Max Re-Tx Value=5, SendSeqNum=7, NumofPendingMsgs=1
[*02/10/2023 11:10:38.4565]
[*02/10/2023 11:10:41.3074] Max retransmission count exceeded, going back to DISCOVER mode.
[*02/10/2023 11:10:41.3075] Dropping msg CAPWAP_ECHO_REQUEST, type = 1, len = 0, eleLen = 8, sendSeqNum = 7
[*02/10/2023 11:10:41.3280]
[*02/10/2023 11:10:41.3280] CAPWAP State: DTLS Teardown
[*02/10/2023 11:10:41.3426] Aborting image download(0x0): Dtls cleanup, ap1g6a
[*02/10/2023 11:10:41.4175] upgrade.sh: Script called with args:[ABORT]
[*02/10/2023 11:10:41.4502] do ABORT, part1 is active part
[*02/10/2023 11:10:41.4744] upgrade.sh: Cleanup tmp files ...
[*02/10/2023 11:10:41.4976] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: DTLS Teardown(4).
[*02/10/2023 11:10:41.4976] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: DTLS Teardown(4).
[*02/10/2023 11:10:56.0797]
[*02/10/2023 11:10:56.0797] CAPWAP State: Discovery
[*02/10/2023 11:10:56.0804] Got WLC address X.X.X.140 from DHCP.
[*02/10/2023 11:10:56.0805] IP DNS query for CISCO-CAPWAP-CONTROLLER.XXX.XX
[*02/10/2023 11:11:16.1022] Discovery Request sent to X.X.X.140, discovery type DHCP(2)
[*02/10/2023 11:11:16.1037] Discovery Request sent to 255.255.255.255, discovery type UNKNOWN(0)
[*02/10/2023 11:11:16.1038] Discovery Response from X.X.X.140
[*02/10/2023 11:11:16.1045] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Discovery(2).
[*02/10/2023 11:11:25.0003]
[*02/10/2023 11:11:25.0003] CAPWAP State: DTLS Setup
[*02/10/2023 11:11:25.6318]
[*02/10/2023 11:11:25.6318] CAPWAP State: Join
[*02/10/2023 11:11:25.6340] Sending Join request to X.X.X.140 through port 5272
[*02/10/2023 11:11:25.6379] Join Response from X.X.X.140
[*02/10/2023 11:11:25.6379] AC accepted join request with result code: 0
[*02/10/2023 11:11:25.6424] Received wlcType 0, timer 30
[*02/10/2023 11:11:25.6424] TLV ID 2216 not found
[*02/10/2023 11:11:25.6424] TLV-DEC-ERR-1: No proc for 2216
[*02/10/2023 11:11:25.6655]
[*02/10/2023 11:11:25.6655] CAPWAP State: Image Data
[*02/10/2023 11:11:25.6658] AP image version 8.10.130.0 backup 0.0.0.0, Controller 17.9.2.52
[*02/10/2023 11:11:25.6658] Version does not match.
[*02/10/2023 11:11:25.6908] upgrade.sh: Script called with args:[PRECHECK]
[*02/10/2023 11:11:25.7238] do PRECHECK, part1 is active part
[*02/10/2023 11:11:25.7459] upgrade.sh: /tmp space: OK available 80600, required 40000
[*02/10/2023 11:11:25.7463] wtpImgFileReadRequest: request ap1g6a, local /tmp/part.tar
[*02/10/2023 11:11:25.7474] Image Data Request sent to X.X.X.140, fileName [ap1g6a], slaveStatus 0
[*02/10/2023 11:11:25.7488] Image Data Response from X.X.X.140
[*02/10/2023 11:11:25.7488] AC accepted join request with result code: 0
[*02/10/2023 11:11:25.7537] <.......<30>systemd[1]: Starting Lighttpd Watcher...
[*02/10/2023 11:11:31.1014] ...........................................
[*02/10/2023 11:11:40.6688] ......................................Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Image Data(10).
[*02/10/2023 11:11:47.8905] .Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Image Data(10).
[*02/10/2023 11:11:48.1390] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Image Data(10).
[*02/10/2023 11:11:53.5490] ...........
[*02/10/2023 11:11:55.8135] ................................................Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Image Data(10).
[*02/10/2023 11:12:09.6671] ..
[*02/10/2023 11:12:10.0406] ........................

[marce1000]

 - Check this thread for hints : https://community.cisco.com/t5/other-network-architecture-subjects/9130axi-can-t-join-9800-80-wlc-write-error-no-space-left-on/m-p/4737454

 M.

[Rich R · Accepted solution]

This was actually covered in the 9800 troubleshooting session at Cisco Live 2 days ago! 
"17.3.5 works as upgrade hop towards 17.9 especially for 9130.  Problem : APs that you kept in stock or got through RMA might not join a 17.9 WLC anymore" 
The new image won't fit in the existing flash partition size so upgrade will never succeed.  You are running very old code on that AP!

The guidance for AP running 17.2 or 17.3.1 to 17.3.4 is to first upgrade the AP to 17.3.5 (or later) or 17.6.x (which will increase the partition size) and then upgrade to 17.9.x
As your AP is running 8.10.130.0 you might even need to upgrade it to 8.10.183.0 before trying the above step.
You can download the upgrade images directly and upgrade via console with TFTP.

[HVO-cisco]

@Rich R 
Thank you very much !!

Regards
Johann Folkestad

[dal]

I have this problem

Any guide on how to upgrade the C9130AXI via TFTP?

Thanks

[Rich R]

AP#archive download-sw /reload ?
WORD URL of AP image <tftp|sftp>://<server_ip>/<file_path>

Just remember you need to upgrade it to an intermediate version first to get the partition sizes adjusted before you upgrade to the latest versions.

This is also covered in the TAC recommended versions page (link below) and the release notes https://www.cisco.com/c/en/us/td/docs/wireless/controller/9800/17-9/release-notes/rn-17-9-9800.html#Cisco_Concept.dita_59a2987f-2633-4630-8c7b-a8e8aecdeaf7 both of which you should always consult before starting upgrades.

[Thomas Lindell]

I mean it's less that obvious

This will work

!if you issue a show version on your ap and it tells you it's AP Running IMage is 8.10.130.0

!ex:

Processor board ID FJC25411PKG
AP Running Image : 8.10.130.0
Primary Boot Image : 8.10.130.0
Backup Boot Image : 0.0.0.0
Primary Boot Image Hash:
Backup Boot Image Hash:

Hold down the Reset button for 30 Seconds during power up while serial console is attached
once at the login prompt login with
Username = Cisco
Password = Cisco
en
Cisco

!Assign a Required IP so you can reach your TFTP Server
!First download the Legacy LWAPP Firmware it's necessary to Extract it from C9800-AP-universalk9.17.03.08a.zip
!in the case of the 9130Axi you need ap1g6a
!put that image on your TFTP Server

archive download-sw /no-reload tftp://192.168.1.1/ap1g6a
reboot
archive download-sw /no-reload tftp://192.168.1.1/ap1g6a-k9w8-tar.153-3.JPS.tar

After this you can connect it to a controller and it will work properly

[Rich R]

Did you mean "it's less than obvious" @Thomas Lindell ?

> !First download the Legacy LWAPP Firmware it's necessary to Extract it from C9800-AP-universalk9.17.03.08a.zip
Or you can just download it directly from
https://software.cisco.com/download/home/286323627/type/286288051/release/15.3.3-JK11

> archive download-sw /no-reload tftp://192.168.1.1/ap1g6a-k9w8-tar.153-3.JPS.tar
The original poster was trying to join the AP to a WLC running 17.9.2 so the correct image for that would be:
https://software.cisco.com/download/home/286323627/type/286288051/release/15.3.3-JPN1
15.3(3)JPS is for 17.14.1.