Cisco AP1832I not starting correctly

[morabusa]

Hi, I am trying to add a new AP to an existing WLC Mobility Express but while I can see that the AP joins "corretly", then I am getting an error and the AP does not upgrade the IOS:

*** AP:

cisco AIR-AP1832I-E-K9 ARMv7 Processor rev 0 (v7l) with 966404/744100K bytes of memory.
AP Running Image : 8.2.151.0
Primary Boot Image : 8.2.151.0
Backup Boot Image : 8.3.112.0
1 Gigabit Ethernet interfaces
2 802.11 Radios

*** 

WLC:

(Cisco Controller) >show run-config

Press Enter to continue...

System Inventory
NAME: "Mobility Express" , DESCR: "Cisco Aironet 1830 Series Mobility Express"

System Information
Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 8.5.105.0

(Cisco Controller) >show ap image all

Total number of APs.............................. 9
Number of APs
Initiated....................................... 0
Downloading..................................... 0
Predownloading.................................. 0
Completed predownloading........................ 0
Not Supported................................... 0
Failed/Waiting to Predownload................... 0

Predownload Predownload Flexconnect Failure Download
AP Name Primary Image Backup Image Status Version Next Retry Time Retry Count Predownload Reason Progress
------------------ -------------- -------------- --------------- -------------- ---------------- ------------ -------------- -------------------- ----------
Almacen 8.5.105.0 8.3.122.0 None None NA NA NA
Fabricacion 8.5.105.0 8.3.122.0 None None NA NA NA
Pasillo 8.5.105.0 8.3.122.0 None None NA NA NA
Fabricacion2 8.5.105.0 8.3.122.0 None None NA NA NA
Direccion 8.5.105.0 8.3.122.0 None None NA NA NA
FC_Planta_Baja 8.5.105.0 8.3.122.0 None None NA NA NA
Almacen_Escalera 8.5.105.0 8.4.100.0 None None NA NA NA
Archivo 8.5.105.0 8.4.100.0 None None NA NA NA
AP2C31.24BE.1658 8.2.151.0 0.0.0.0 None None NA NA NA  <---- This is the new AP

--More-- or (q)uit

(Cisco Controller) >show ap join stats summary all  (output trunkated but there are more APs)

2c:31:24:be:c5:a0 2c:31:24:be:c5:a0 AP2C31.24BE.1658 10.34.211.13 Joined

(Cisco Controller) >show ap join stats detailed 2c:31:24:be:c5:a0

Sync phase statistics
- Time at sync request received............................ Not applicable
- Time at sync completed................................... Not applicable

Discovery phase statistics
- Discovery requests received.............................. 95
- Successful discovery responses sent...................... 95
- Unsuccessful discovery request processing................ 0
- Reason for last unsuccessful discovery attempt........... Not applicable
- Time at last successful discovery attempt................ Jun 15 12:35:02.304
- Time at last unsuccessful discovery attempt.............. Not applicable

Join phase statistics
- Join requests received................................... 91
- Successful join responses sent........................... 91
- Unsuccessful join request processing..................... 0
- Reason for last unsuccessful join attempt................ Not applicable
- Time at last successful join attempt..................... Jun 15 12:35:11.932
- Time at last unsuccessful join attempt................... Not applicable

Configuration phase statistics

--More-- or (q)uit
- Configuration requests received.......................... 0
- Successful configuration responses sent.................. 0
- Unsuccessful configuration request processing............ 0
- Reason for last unsuccessful configuration attempt....... Not applicable
- Time at last successful configuration attempt............ Not applicable
- Time at last unsuccessful configuration attempt.......... Not applicable

Last AP message decryption failure details
- Reason for last message decryption failure............... Not applicable

Last AP disconnect details
- Reason for last AP connection failure.................... Timed out while waiting for ECHO repsonse from the AP
- Last AP disconnect reason................................ Not applicable

Last join error summary
- Type of error that occurred last......................... AP got or has been disconnected
- Reason for error that occurred last...................... Timed out while waiting for ECHO repsonse from the AP
- Time at which the last join error occurred............... Jun 15 12:02:10.286

AP disconnect details
- Reason for last AP connection failure.................... Timed out while waiting for ECHO repsonse from the AP
Ethernet Mac : 2c:31:24:be:c5:a0 Ip Address : 10.34.211.13

(ICMP for this IP is working well and not packet loss and not duplicated IP)

In the AP Side I can see this:

[*06/15/2020 11:23:35.5089] ethernet_port wired0, ip 10.34.211.13, netmask 255.255.255.0, gw 10.34.211.254, mtu 1500, bcast 10.34.211.255, dns1 10.34.210.115, dns2 10.34.210.8, vid 0, static_ip_failover false, dhcp_vlan_failover false
[*06/15/2020 11:23:35.5389] ethernet_port wired0, ip 10.34.211.13, netmask 255.255.255.0, gw 10.34.211.254, mtu 1500, bcast 10.34.211.255, dns1 10.34.210.115, dns2 10.34.210.8, vid 0, static_ip_failover false, dhcp_vlan_failover false
[*06/15/2020 11:23:35.8388] CAPWAP State: Discovery
[*06/15/2020 11:23:35.8688] Discovery Request sent to 255.255.255.255, discovery type UNKNOWN(0)
[*06/15/2020 11:23:35.8688] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Discovery(2).
[*06/15/2020 11:23:35.8688] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Discovery(2).
[*06/15/2020 11:23:35.8688] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Discovery(2).
[*06/15/2020 11:23:35.8688] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Discovery(2).
[*06/15/2020 11:23:35.8688] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Discovery(2).
[*06/15/2020 11:23:35.8688] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Discovery(2).
[*06/15/2020 11:23:35.8688] Discovery Response from 10.34.211.1
[*06/15/2020 11:23:45.0000] CAPWAP State: DTLS Setup
[*06/15/2020 11:23:45.0699] CAPWAP State: Join
[*06/15/2020 11:23:45.0799] Sending Join request to 10.34.211.1 through port 5256
[*06/15/2020 11:23:45.0799] Join Response from 10.34.211.1
[*06/15/2020 11:23:45.0799] TLV-DEC-PROC: Info missing for 1363
[*06/15/2020 11:23:45.1799] CAPWAP State: Image Data
[*06/15/2020 11:23:45.2199] do PRECHECK, part1 is active part

[*06/15/2020 11:24:09.2624] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Image Data(10).
[*06/15/2020 11:24:13.6910]
[*06/15/2020 11:24:13.6910] Lost connection to the controller, going to restart CAPWAP...
[*06/15/2020 11:24:13.6910]
[*06/15/2020 11:24:13.6910] Restarting CAPWAP State Machine.
[*06/15/2020 11:24:13.6910] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Image Data(10).
[*06/15/2020 11:24:13.7010] CAPWAP State: DTLS Teardown
[*06/15/2020 11:24:13.7010] Dropping dtls packet since session is not established. a22d301, 147e, a22d30d, 1488, (nil)
[*06/15/2020 11:24:13.7010] Dropping dtls packet since session is not established. a22d301, 147e, a22d30d, 1488, (nil)
[*06/15/2020 11:24:18.4495] CAPWAP State: Discovery
[*06/15/2020 11:24:18.4595] Discovery Request sent to 255.255.255.255, discovery type UNKNOWN(0)
[*06/15/2020 11:24:18.4695] Discovery Response from 10.34.211.1
[*06/15/2020 11:24:28.0000] CAPWAP State: DTLS Setup
[*06/15/2020 11:24:28.0799] CAPWAP State: Join
[*06/15/2020 11:24:28.0799] Sending Join request to 10.34.211.1 through port 5256
[*06/15/2020 11:24:28.0799] Join Response from 10.34.211.1
[*06/15/2020 11:24:28.0799] TLV-DEC-PROC: Info missing for 1363
[*06/15/2020 11:24:28.1799] CAPWAP State: Image Data
[*06/15/2020 11:24:28.2199] do PRECHECK, part1 is active part
[*06/15/2020 11:24:56.7010]
[*06/15/2020 11:24:56.7010] Lost connection to the controller, going to restart CAPWAP...
[*06/15/2020 11:24:56.7010]
[*06/15/2020 11:24:56.7010] Restarting CAPWAP State Machine.
[*06/15/2020 11:24:56.7010] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Image Data(10).
[*06/15/2020 11:24:56.7010] CAPWAP State: DTLS Teardown
[*06/15/2020 11:24:56.7110] Dropping dtls packet since session is not established. a22d301, 147e, a22d30d, 1488, (nil)
[*06/15/2020 11:24:56.7110] Dropping dtls packet since session is not established. a22d301, 147e, a22d30d, 1488, (nil)
[*06/15/2020 11:25:01.4595] CAPWAP State: Discovery
[*06/15/2020 11:25:01.4795] Discovery Request sent to 255.255.255.255, discovery type UNKNOWN(0)
[*06/15/2020 11:25:01.4795] Discovery Response from 10.34.211.1
[*06/15/2020 11:25:18.8276] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Discovery(2).
[*06/15/2020 11:25:11.0000] CAPWAP State: DTLS Setup
[*06/15/2020 11:25:11.0799] CAPWAP State: Join
[*06/15/2020 11:25:11.0799] Sending Join request to 10.34.211.1 through port 5256
[*06/15/2020 11:25:11.0799] Join Response from 10.34.211.1
[*06/15/2020 11:25:11.0799] TLV-DEC-PROC: Info missing for 1363
[*06/15/2020 11:25:11.1799] CAPWAP State: Image Data
[*06/15/2020 11:25:11.2199] do PRECHECK, part1 is active part

Any idea about what could be happening? Thanks!

[Rafael E · Accepted solution]

In ME, it does not have the AP images, you need to point AP to the TFTP server where they can download their image. 

You can do that in the upgrade configuration for ME.

You will need the uncompress the .zip file (download it from the cisco.com) on the root path for your TFTP server. 

[Rich R]

And 8.5.105.0 is *very* old so while you're downloading the AP image bundle why not update to ME 8.5.161.0 at the same time ;)
https://software.cisco.com/download/home/286288035/type/286289839/release/8.5.161.0

[morabusa]

Hi, thanks for the answers. And yes, I need to upgrade everything to 8.5.161.0 because 8.5.105.0 has been retired. Is there any way to force the APs to upgrade firmware? I have tried this command but it does not work anymore:

debug capwap console cli
(previous step before running this: archive download-sw tftp://192.168.1.250/ap3g2-k9w8-tar.153-3.JF.tar)

Thanks!

[Rich R · Accepted solution]

Going by this discussion https://community.cisco.com/t5/other-wireless-mobility-subjects/how-to-manually-update-software-on-cos-ap/td-p/3794296 it will be easier to let them download automatically from your TFTP server. So upgrade your ME then the rest will download automatically.
And I'm pretty sure that is the wrong image you're trying to use there. You should be using the images from inside the AP bundle AIR-AP1830-K9-ME-8-5-161-0.zip on your TFTP server. From apname_decoder.inc in the zip file:
/*AP Models and their Associated Image Names*/
AP1850(ap1g4)
AP1830(ap1g4)
AP3800(ap3g3)
AP2800(ap3g3)
AP1560(ap3g3)
AP1815i(ap1g5)
AP1815w(ap1g5)
AP1815m(ap1g5)
AP1540(ap1g5)

[morabusa]

Ok thanks! Just one last question, the current ME device is and AIR-AP1832I-E-K9 and most of other APs are the same model. I have downloaded the AIR-AP1830-K9-ME-8-5-161-0.zip IOS and I will install it in the ME and then all other 1832i APs will go to get upgraded automatically according to what I have read.

On the other hand, I also have two AIR-CAP702I-E-K9 devices but I cannot find the IOS 8.5.161.0 for this model. I just can find the lightweight or autonomous firmware for them which are versions 15.X.X. It looks like there is not version 8.X.X for this AP but somehow, I am currently seeing that they have the 8.5.105.0 IOS, as you can see in the attached file. So, how can I install 8.X.X firmwares on AIR-CAP702I-E-K9? Thanks.

[Rich R]

According to the release notes https://www.cisco.com/c/en/us/td/docs/wireless/access_point/mob_exp/85/release_notes/b_ME_RN_85.html#me-supp-ap the 700i series is supported and it uses the ap1g1 image which is part of the bundle so should work fine. Just load all the images on your TFTP server and the APs will download the correct image.
https://www.cisco.com/c/en/us/td/docs/wireless/controller/technotes/8-5/b_cisco_mobility_express_8_5/b_cisco_mobility_express_8_5_chapter_0111.html#concept_4E1DCD5CC7A74E9FA52AF3B393D071D8
https://dannyda.com/2020/02/11/how-to-update-cisco-mobility-express-with-web-gui-and-tftp-without-using-terminal/