topic Re: Embedded controller problems after Update on 9115 ap in Wireless

Embedded controller problems after Update on 9115 ap

netlevelag — Mon, 05 Jul 2021 20:13:48 GMT

Hello Everyone

I have a C9115 that i converted to EWC Using ap-type mobility-express tftp://10.1.202.111/9100/ap1g7 tftp://10.1.202.111/9100/C9800-AP-iosxe-wlc.bin

The device came back up in recovery mode and i tried installing the binary using this command: archive download-sw ewc-ap tftp://10.1.202.111/9100/C9800-AP-iosxe-wlc.bin

The download stops dead at 78%

I think that the flash is full and I found this bug: https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvt44832/?rfs=iqvred

When I try and list the cores I get this:

#show flash cores
Directory of /storage/cores/
total 0K

Directory of EWC-AP core files
Directory does not exist!
---------------------------------------------------------------------------
Filesystem Size Used Available Use% Mounted on
/dev/ubivol/part2 520.1M 497.8M 22.3M 96% /part2

Any help would be welcome. Thank you

Re: Embedded controller problems after Update on 9115 ap

Mark Elsen — Thu, 11 Feb 2021 15:29:24 GMT

- I have been following up on this for a while , more and more I think the tftp-client on this particular ap-model is buggy. Could you use FTP instead ?

Re: Embedded controller problems after Update on 9115 ap

netlevelag — Thu, 11 Feb 2021 15:35:52 GMT

I have seen that in some threads and I tried. This is the output:

#archive download-sw ewc-ap ftp://cisco:Welcome99!@10.1.10.11/tftp/9100/C9800-AP-iosxe-wlc.bin
Error: ftp to be implemented in image_download.sh.
Checking .../part2/wlc/images/C9800-AP-iosxe-wlc.bin doesn't exist!

Re: Embedded controller problems after Update on 9115 ap

Mark Elsen — Thu, 11 Feb 2021 15:50:29 GMT

Tx, for the feedback , are there any other download protocols available ? Usually can be show with command completion request ? when you arrived at the relevant part of the command.

Re: Embedded controller problems after Update on 9115 ap

netlevelag — Fri, 12 Feb 2021 10:11:39 GMT

Only tftp and

AP34ED.1BDA.81E4#archive download-sw ewc-ap
WORD URL of eWLC image <tftp|sftp>://<server_ip>/<file_path>, or 'rollback'
to load previously-used eWLC image

So I changed it to: archive download-sw ewc-ap sftp://cisco:Welcome99!@10.1.10.11/tftp/9100/C9800-AP-iosxe-wlc.bin

and it got to:

AP34ED.1BDA.XXXX############################################################### 87.4%
AP34ED.1BDA.XXXX#ransfer command failed (Error 23)

I am sure that it has to do with the bug above. This is the result from show flash:

AP34ED.1BDA.XXXX#show flash
Directory of /storage/
total 84K
-rw-r--r-- 1 root root 2 Nov 25 00:10 BOOT_COUNT
-rw-r--r-- 1 root root 2 Nov 25 00:10 BOOT_COUNT.reserve
-rw-r--r-- 1 root root 29 Nov 25 00:10 RELOADED_AT_UTC
drwxr-xr-x 3 rsyncuse root 224 Nov 25 00:10 application
-rw-r--r-- 1 root root 303 Nov 25 15:54 base_capwap_cfg_info
-rw-r--r-- 1 root root 2 Nov 25 00:10 boot_debug
-rw-r--r-- 1 root root 10366 Nov 25 00:11 bootloader.log
-rw-r--r-- 1 root root 5 Nov 25 00:10 bootloader_verify.shadow
-rw-r--r-- 1 root root 18 Nov 25 00:10 config
-rw-r--r-- 1 root root 0 Nov 25 00:10 config.local
-rw-r--r-- 1 root root 0 Nov 25 00:11 config.mobexp
-rw-r--r-- 1 root root 0 Nov 25 00:11 config.oeap
drwxr-xr-x 2 root root 160 Sep 27 2019 cores
drwxr-xr-x 2 root root 160 Jan 1 1970 dropbear
drwxr-xr-x 2 root root 160 Nov 25 00:10 firmware
-rw-r--r-- 1 root root 251 Nov 25 00:17 last_good_uplink_config
drwxr-xr-x 2 root root 160 Sep 27 2019 lists
-rw-r--r-- 1 root root 18837 Nov 25 00:10 nvram_setting
-rw-r--r-- 1 root root 63 Sep 27 2019 part1_info.ver
-rw-r--r-- 1 root root 218 Sep 27 2019 part2_info.ver
-rw-r--r-- 1 root root 4096 Nov 25 00:16 random_seed
-rw-r--r-- 1 root root 3 Nov 25 00:10 rxtx_mode
drwxr-xr-x 2 root root 648 Nov 25 00:10 ssh
drwxr-xr-x 3 1 root 224 Sep 27 2019 support
drwxr-xr-x 2 root root 4144 Nov 25 00:41 syslogs
-rw-r--r-- 1 root root 1 Nov 25 00:11 wips_adminstate_capwap
---------------------------------------------------------------------------
Filesystem Size Used Available Use% Mounted on
/dev/ubivol/part2 520.1M 520.1M 12.0K 100% /part2

The Flash is full and can't download the image.

Re: Embedded controller problems after Update on 9115 ap

merlin76 — Fri, 12 Feb 2021 11:40:04 GMT

IOS XE uses "ip tftp blocksize 8192" by default to speed up file transfers, might be there are incompatibilties with the TFTP server. Normal TFTP blocksize is 512. So perhaps lowering the blocksize to a value of 1456 or even 512 fixes the issue.

OpenTFTP for example uses 512 by default and if you like to use other rates you have to set the blocksize in the ini file.

Re: Embedded controller problems after Update on 9115 ap

Mark Elsen — Fri, 12 Feb 2021 11:54:19 GMT

- What's in the logs of the sftp-server when this happens ?

Re: Embedded controller problems after Update on 9115 ap

netlevelag — Fri, 12 Feb 2021 12:17:53 GMT

Not much. Only: SFTP client [admin] from [10.1.202.105] logged out the server with totally [0 Bytes] uploaded and [216.24 MB] downloaded.

Re: Embedded controller problems after Update on 9115 ap

netlevelag — Fri, 12 Feb 2021 15:07:22 GMT

Thank you.

I have tried with Solarwinds and Synology TFTP. Could not find any settings where I could change the blocksize.

I have never had any problems with TFTP before and have also tried SFTP and it stops at 87%. I am pretty sure that the flash is full.

Any other ideas welcome!

Re: Embedded controller problems after Update on 9115 ap

netlevelag — Fri, 12 Feb 2021 15:08:17 GMT

I am getting this on the console:

[*11/25/2020 00:24:10.1040] Going to restart CAPWAP (reason : WTP IP address changed)...
[*11/25/2020 00:24:10.1040]
[*11/25/2020 00:24:10.1050] Restarting CAPWAP State Machine.
[*11/25/2020 00:24:10.1050] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Discovery(2).
[*11/25/2020 00:24:10.1220]
[*11/25/2020 00:24:10.1220] CAPWAP State: DTLS Teardown
[*11/25/2020 00:24:10.2090] status 'upgrade.sh: Script called with args:[ABORT]'
[*11/25/2020 00:24:10.2510] do ABORT, part2 is active part
[*11/25/2020 00:24:10.2830] status 'upgrade.sh: Cleanup tmp files ...'
[*11/25/2020 00:24:10.3030] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: DTLS Teardown(4).
[*11/25/2020 00:24:10.3030] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: DTLS Teardown(4).

Is there any way I can free up space in the flash?

Re: Embedded controller problems after Update on 9115 ap

merlin76 — Fri, 12 Feb 2021 15:30:28 GMT

Don't know about the one from Solarwinds but Synology uses OpenTFTP.

You need to SSH into the NAS, then:

sudo vi /etc/opentftp.ini

Press ESC

Insert "blksize=8192" into "[TFTP-OPTIONS]" section

Press ESC again

Save and quit with ":wq!"

/usr/syno/sbin/synoservicectl --restart opentftp

You can also change blocksize on IOS XE:

enable

conf t

ip tftp blocksize 512

end

That also works on IOS switches, but you should have at least LAN base image.

Re: Embedded controller problems after Update on 9115 ap

Mark Elsen — Fri, 12 Feb 2021 17:28:12 GMT

- Try to run the sftp (/ssh-server) in debugging or verbose mode, check if that can give more info's

Re: Embedded controller problems after Update on 9115 ap

netlevelag — Mon, 15 Feb 2021 08:41:02 GMT

Thank you!

I tried that and get the same error.

curl: (71) Error
transfer command failed (Error 71)

I am going to RMA the device.

Re: Embedded controller problems after Update on 9115 ap

Mark Elsen — Mon, 15 Feb 2021 09:01:11 GMT

- I have been struggling to understand which models do EWC , yours may not :

https://www.cisco.com/c/en/us/products/collateral/wireless/embedded-wireless-controller-catalyst-access-points/white-paper-c11-743398.html

Re: Embedded controller problems after Update on 9115 ap

netlevelag — Mon, 15 Feb 2021 09:04:13 GMT

Yes it does. That is why there is a EWC Firmware for the device. You can easyly look it up by googling the product with the EWC Part in it.

Re: Embedded controller problems after Update on 9115 ap

Mark Elsen — Mon, 15 Feb 2021 09:08:35 GMT

- Yes, I know but look at the full part numbers as given in the document, and or check if that matches you ap part number. I have been at this point before, you may have to finally check it our or have it confirmed by TAC.

Re: Embedded controller problems after Update on 9115 ap

netlevelag — Mon, 15 Feb 2021 09:41:51 GMT

I will send it back to the reseller. The device is obviously affected by the bug I have posted before. There are many many threads about people succesfuly converting the device.

Re: Embedded controller problems after Update on 9115 ap

Mark Elsen — Wed, 17 Feb 2021 08:10:00 GMT

- Also have a look at this document :

https://www.wifireference.com/2019/12/07/cisco-embedded-wireless-controller-on-catalyst-access-points/

(take a look at the checklist at : Installing the EWC Software)