topic C9117AXI stuck in EWC-AP Recovery mode in Wireless

C9117AXI stuck in EWC-AP Recovery mode

peat — Fri, 04 Feb 2022 14:26:40 GMT

Hi,

I was given an 9117ax to setup with a SSID to use for a wireless survey. But I think i got confused about what models were available and I thought i had a lightweight mode AP when in fact it was already the embedded WLC version.

The long story short is I followed the cisco guide on converting a lightweight AP to a standalone AP and even though everything seem to go fine after the tftp upload of the AP image and the WLC bin file on the reboot the AP has gone into EWC-AP recovery mode and keeps saying

"EWC-AP Recovery Mode", Run archive download-sw "

So I tried that archive download command and it looked like it went through fine till it got to 72.6% and then it said error .

I tried again and it stopped at 72.6% again.

I did a show flash and it showed that there was only 187mb left but the wlc.bin is 280mb so im guessing thats my problem?

But after googling how to make space it doesnt look like a switches/routers output and the list of files in the flash doesnt have filesize or are recognisable to me.

I seen somewhere mention show flash cores but that said there was nothing there.

I've tried a factory reset but that didn't change anything.

Can anyone help get this AP back to working ?

The model is C9117AXI-E which after reading up is the EWC version so it looks like I really messed up and have tried to put the WLC image on a WAP that already had it on it.

Re: C9117AXI stuck in EWC-AP Recovery mode

Scott Fella — Fri, 04 Feb 2022 14:32:12 GMT

Did you use the ap1g6 image from the tar file you downloaded? You can always try to delete some of the files on flash to free up some space is that is the reason. Maybe post what you see so we have a better idea.

Here is a guide I used back in the day's for site survey.

Cisco 802.11ax site survey (single AP method) | SC-WiFi

Re: C9117AXI stuck in EWC-AP Recovery mode

peat — Fri, 04 Feb 2022 16:03:47 GMT

Hi,

Yes used the ap1g6 image.

Yes I would like to delete some files from the flash. Im sure that would fix the problem.

This is the list

AP1#show flash

Directory of /storage/
total 56K
-rw-r--r-- 1 root root 2 Dec 4 08:12 BOOT_COUNT
-rw-r--r-- 1 root root 2 Dec 4 08:12 BOOT_COUNT.reserve
-rw-r--r-- 1 root root 28 Dec 4 08:12 RELOADED_AT_UTC
drwxr-xr-x 3 rsyncuse root 224 Dec 4 08:12 application
-rw-r--r-- 1 root root 394 Dec 4 08:15 base_capwap_cfg_info
-rw-r--r-- 1 root root 2 Dec 4 07:19 boot_debug
-rw-r--r-- 1 root root 0 Dec 4 07:20 bootloader.log
-rw-r--r-- 1 root root 18 Dec 4 08:11 config
-rw-r--r-- 1 root root 0 Dec 4 08:12 config.local
-rw-r--r-- 1 root root 0 Dec 4 08:12 config.mobexp
-rw-r--r-- 1 root root 0 Dec 4 08:11 config.mobexp.tmp_5793
-rw-r--r-- 1 root root 0 Dec 4 08:12 config.oeap
-rw-r--r-- 1 root root 0 Dec 4 08:11 config.tls_client.tmp_5793
drwxr-xr-x 2 root root 160 Aug 20 2020 cores
drwxr-xr-x 2 root root 160 Jan 1 1970 dropbear
lrwxrwxrwx 1 root root 31 Dec 4 07:20 iot_status_iot0.txt -> /storage/iot_status_ttyMSM1.txt
-rw-r--r-- 1 root root 153 Dec 4 07:20 iot_status_ttyMSM1.txt
-rw-r--r-- 1 root root 213 Dec 4 07:49 last_good_uplink_config
drwxr-xr-x 2 root root 160 Aug 20 2020 lists
-rw-r--r-- 1 root root 63 Aug 20 2020 part1_info.ver
-rw-r--r-- 1 root root 219 Aug 20 2020 part2_info.ver
-rw-r--r-- 1 root root 4096 Dec 4 08:17 random_seed
-rw-r--r-- 1 root root 3 Dec 4 07:19 rxtx_mode
drwxr-xr-x 2 root root 648 Dec 4 07:20 ssh
-rw-r--r-- 1 root root 40 Dec 4 07:19 stable_secret_seed
drwxr-xr-x 3 1 root 224 Aug 20 2020 support
drwxr-xr-x 2 root root 1432 Dec 4 08:12 syslogs
-rw-r--r-- 1 root root 1 Dec 4 08:12 wips_adminstate_capwap
---------------------------------------------------------------------------
Filesystem Size Used Available Use% Mounted on
/dev/ubivol/part2 519.4M 519.4M 0 100% /part2

I dont know why its saying 0mb free at the moment though. When I checked earlier it was 187mb free.

I had found your guide before but it didnt help me get past the EWC-AP in Recovery mode part. As I tried the archive download-sw command but that always fails at 72.6%

Re: C9117AXI stuck in EWC-AP Recovery mode

peat — Fri, 04 Feb 2022 16:09:28 GMT

Also just seen this

AP1#show filesystems
Filesystem Size Used Available Use% Mounted on
devtmpfs 938.8M 0 938.8M 0% /dev
/sysroot 940.8M 135.6M 805.2M 14% /
tmpfs 1.0M 44.0K 980.0K 4% /dev/shm
tmpfs 940.8M 0 940.8M 0% /run
tmpfs 940.8M 0 940.8M 0% /sys/fs/cgroup
/dev/ubivol/part1 371.7M 29.9M 337.1M 8% /part1
/dev/ubivol/part2 519.4M 519.4M 0 100% /part2
none 100.0M 1.5M 98.5M 1% /tmp
none 1.0M 44.0K 980.0K 4% /dev/shm
none 100.0M 1.5M 98.5M 1% /tmp/var/run/netns

Can I get into those tmpfs folders and delete things in there?

Re: C9117AXI stuck in EWC-AP Recovery mode

Scott Fella — Fri, 04 Feb 2022 18:38:50 GMT

The only files I typically delete are log files and crash files. I don't know what else you can do, what free space does the release notes tell you. Maybe attach a screen shot of the failure or the command you are using.

Re: C9117AXI stuck in EWC-AP Recovery mode

Scott Fella — Fri, 04 Feb 2022 18:44:25 GMT

My bad... you should be using this file C9800-AP-iosxe-wlc.bin for the EWC. On the console, try to issue capwap erase all

Re: C9117AXI stuck in EWC-AP Recovery mode

Prince.O — Fri, 04 Feb 2022 20:47:36 GMT

Hi Peat,

I have a few questions to better understand the issue..

1. What version of code is currently running on your 9117 AP?

- You can verify this with the following command from the AP CLI "show version".

- Please provide this output for review

2. What EWC software version are you trying to load onto the AP ?

Re: C9117AXI stuck in EWC-AP Recovery mode

peat — Sat, 05 Feb 2022 18:17:48 GMT

Right im not sure how ive done it but ive managed to get the AP to boot back into the original firmware so the EWC-AP recovery mode has stopped.

So it looks like its back to how it was when i first plugged it in. Its constantly in discovery by the look of it.

[*08/20/2020 00:18:38.2756] CAPWAP State: Discovery
[*08/20/2020 00:18:38.2782] Discovery Request sent to 255.255.255.255, discovery type UNKNOWN(0)

I have set a static IP on it which i was using for the TFTP so im not sure what its doing, it also keeps rebooting after a few minutes which is exactly what it was doing when i got it out of the box. thats why I thought (which i think was wrong) it was a lightweight version.

the model is a 9117axi-e so is that the one with the embedded WLC already? If so how do I access the WLC?

Ive googled a lot but keep coming up with the convert capwap to embedded WLC and that is what got my into this mess in the first place.

To answer the what version question. this is what that shows now i have got it to boot into the original firmware.

cisco C9117AXI-E with 1928016/1410056K bytes of memory.
Processor board ID KWC24320052
AP Running Image : 8.9.100.0
Primary Boot Image : 8.9.100.0
Backup Boot Image : 17.7.1.11
Primary Boot Image Hash:
Backup Boot Image Hash: 5f0aa7bb03c338adaf25b4a937a9e92c9b96a2b1e0979dd6c5754b246fe43ad8e2a8fc3d88413ddd96bd84920f01f03ea6f6a21a6d269fb390fc9d168379009e
1 Gigabit Ethernet interfaces
2 802.11 Radios
Radio FW version : QC_IMAGE_VERSION_STRING=WLAN.HK.1.0-03095-QCAHKSWPL_SILICONZ-1.196094.2.198916.1
NSS FW version : NSS.HK.H.CS.q_len512-E_custC

Do I just need some sort of simple Command that makes it boot into EWC mode?

Re: C9117AXI stuck in EWC-AP Recovery mode

peat — Sat, 05 Feb 2022 18:40:24 GMT

More info.

I have ran the capwap erase all command and rebooted. I have connected it up to my home network and it has picked up a dhcp IP address but the AP is flashing green blue red.

I tried the cisco mobile app just in case to see if I could set it up that way but it cant join it.

It still looks to me like its in lightweight mode since it keeps saying

[*08/20/2020 00:39:28.6355] CAPWAP State: Discovery
[*08/20/2020 00:39:28.6375] IP DNS query for CISCO-CAPWAP-CONTROLLER.lan
[*08/20/2020 00:39:28.6722] Discovery Request sent to 255.255.255.255, discovery type UNKNOWN(0)

but havent i got the embedded wlc model rather than the lightweight version?

Re: C9117AXI stuck in EWC-AP Recovery mode

Scott Fella — Sat, 05 Feb 2022 19:37:24 GMT

Your output shows that the ap is running capwap. You need to extract the tat file and upload the bin file. That is the EWC controller image. Follow the link I have posted.

Re: C9117AXI stuck in EWC-AP Recovery mode

peat — Sun, 06 Feb 2022 12:17:33 GMT

Thats what I did in the first place which put me in recovery mode.

I will give it another go and see if it works better the second time.

So I am guessing CAPWAP = lightweight mode? So my first assumption was right and i didnt mess the AP up?.

That will be helpful when i explain to the company who supplied the AP why I cant do the wifi survey with it on monday.

Re: C9117AXI stuck in EWC-AP Recovery mode

peat — Sun, 06 Feb 2022 18:44:49 GMT

Re: C9117AXI stuck in EWC-AP Recovery mode

peat — Sun, 06 Feb 2022 13:59:31 GMT

I tried again and it was going through fine then half way through the reboot I saw this

[*12/04/2021 07:20:37.1485] [error][verify_package] Fail to verify image: /part2/wlc/images/C9800-AP-iosxe-wlc.bin
[*12/04/2021 07:20:37.1485] FAILED
[ OK ] Started ME Recovery Service.
Starting ME Recovery Service...
[FAILED] Failed to start ME Install Script.
See 'systemctl status me-install.service' for details.
[*12/04/2021 07:20:37.2476] systemd[1]: Failed to start ME Install Script.
[*12/04/2021 07:20:37.2478] systemd[1]: me-install.service failed.
[ OK ] Reached target Multi-User System.
[*12/04/2021 07:20:37.2521] cat: can't open '/part2/wlc/harddisk/stable_ap_version': No such file or directory
[*12/04/2021 07:20:37.2554] cat: can't open '/part2/wlc/harddisk/stable_remote_ap_version': No such file or directory
[*12/04/2021 07:20:37.2690] cat: can't open '/part2/wlc/harddisk/stable_controller_version': No such file or directory
[*12/04/2021 07:20:37.2717] cat: can't open '/part2/wlc/harddisk/stable_remote_controller_version': No such file or directory
[*12/04/2021 07:20:37.3080] Sat Dec 4 07:20:37 UTC 2021 In Auto Recovery Mode
[*12/04/2021 07:20:37.3117] Sat Dec 4 07:20:37 UTC 2021 Checking if stable AP and WLC versions exist
[*12/04/2021 07:20:37.3154] Sat Dec 4 07:20:37 UTC 2021 No stable AP and WLC versions exist.
[*12/04/2021 07:20:37.3190] Sat Dec 4 07:20:37 UTC 2021 Checking if stable remote AP and WLC versions exist
[*12/04/2021 07:20:37.3225] Sat Dec 4 07:20:37 UTC 2021 No stable remote AP and WLC versions exist.
[*12/04/2021 07:20:37.3265] Sat Dec 4 07:20:37 UTC 2021 Attempted auto recovery, but there is no better version to switch to
EWC-AP in Recovery Mode......
Please use 'archive download-sw' to upgrade AP and Controller image.
Please use 'show flexconnect ewc-ap launch-log' to check EWC-AP launch log.

So I tried the archive download-sw command and got this

AP1#archive download-sw ewc-ap tftp://192.168.1.2/C9800-AP-iosxe-wlc.bin
Starting download eWLC image tftp://192.168.1.2/C9800-AP-iosxe-wlc.bin ...
It may take a few minutes. If longer, please abort command, check network and try again.
############## 19.7%EWC-AP in Recovery Mode......
Please use 'archive download-sw' to upgrade AP and Controller image.
Please use 'show flexconnect ewc-ap launch-log' to check EWC-AP launch log.
################################################ 67.1%EWC-AP in Recovery Mode......
Please use 'archive download-sw' to upgrade AP and Controller image.
Please use 'show flexconnect ewc-ap launch-log' to check EWC-AP launch log.
#################################################### 72.6%
curl: (23) Error
transfer command failed (Error 23)

I also tried this command

AP1#ap-type ewc-ap tftp://192.168.1.2/ap1g6 tftp://192.168.1.2/C9800-AP-iosxe-wlc.bin
Image upgrade not needed, changing AP Type to EWC-AP

So I am back to where i was at the first post. Stuck in EWC-AP recovery mode.

Re: C9117AXI stuck in EWC-AP Recovery mode

Rich R — Mon, 07 Feb 2022 11:12:24 GMT

A few things to clarify:

- Like Scott said that is a CAPWAP (lightweight) image.

- The AP boot partitions (which you can't manually delete files from) are part1 and part2.

- The model number you quoted (9117axi-e) is not an actual PID - you probably meant C9117AXI-E which is simply the EU regulatory domain (-E) version of the 9117 AP (CAPWAP). The equivalent PID for EWC version would have been C9117AXI-EWC-E.

- It's normal for a CAPWAP AP to keep rebooting when it can't find a WLC to join

You're trying to go from a very old deferred 8.9 AireOS CAPWAP version to the very latest 17.7.1 IOS-XE version - that might be a problem (just guessing, never tried it).

I suggest upgrading the AP to the latest 8.10.162.0 AireOS CAPWAP image first https://software.cisco.com/download/home/286322356/type/286288051/release/15.3.3-JK6

Then once it is running that you could try loading the 17.7.1 EWC.

In fact if you read the release notes (which you should ALWAYS do) you'd have seen this:

https://www.cisco.com/c/en/us/td/docs/wireless/controller/ewc/17-7/rel-notes/ewc-rn-17-7-x.html#id_88368

Note Before converting from CAPWAP to embedded wireless controller (EWC), ensure that you upgrade the corresponding AP with the CAPWAP image in Cisco AireOS Release 8.10.105.0. If this upgrade is not performed, the conversion will fail.

So 8.10.162.0 should work just as well in my opinion. But if it doesn't then use 8.10.105.0 https://software.cisco.com/download/home/286322356/type/286288051/release/15.3.3-JK

Re: C9117AXI stuck in EWC-AP Recovery mode

peat — Mon, 07 Feb 2022 12:02:19 GMT

Thanks that's very helpful. Yes I think i misunderstood the joint tftp command with the ap1g6 part. I thought that was the APs firmware being updated unfortunately.

Re: C9117AXI stuck in EWC-AP Recovery mode

peat — Mon, 07 Feb 2022 11:29:37 GMT

I was a bit more optimistic there but unfortunately we dont have a the right entitlement to download the AP updates.

I was able to get the EWC bundle fine so that's a bit strange.

Re: C9117AXI stuck in EWC-AP Recovery mode

Rich R — Mon, 07 Feb 2022 11:32:32 GMT

Sometimes if you search you can find the same file elsewhere.

Re: C9117AXI stuck in EWC-AP Recovery mode

Rich R — Mon, 07 Feb 2022 12:15:12 GMT

The TFTP command is upgrading the AP firmware but when you upgrade to EWC you're actually configuring it to run 2 separate components - the IOS-XE WLC (EWC) firmware and the ClickOS (COS) AP image. So once it is installed it is running both together on the same box and the AP still "talks" CAPWAP to the WLC.

Re: C9117AXI stuck in EWC-AP Recovery mode

Scott Fella — Mon, 07 Feb 2022 16:08:58 GMT

If you have the bundle (tar file), you should have the EWC image (.bin) and the AP image ap1g6. That should get you started.

If you have Hyper-V or ESXi, you can try to spin up a 9800-CL. It's free!

Re: C9117AXI stuck in EWC-AP Recovery mode

peat — Tue, 08 Feb 2022 15:06:51 GMT

I received the firmware update and have just applied it.

The AP rebooted and went into ewc-ap recovery mode which I'm assuming should not happen when its just updating the firmware from 8.9.100.0 to 8.10.162.0.

Even so i tried the archive download-sw ewc-ap tftp://192.168.1.1/C9800-AP-iosxe-wlc.bin command and that just failed again on 72.6%