Unable to get new APs addedd to the Mobility Express Controller

[Miguel Angel Alvarez Rodriguez]

I've been trying to setup a WiFi network since a few days ago. I posted several messages in the community previously but I come to a point 'm totalyy blocked. I have tried to open a support case with Cisco, but although the Smarnets we're bought with the APs, Cisco still has not activated them, so I don't hace support from them, and I need to install this Tuesday so If anybody can help I will be very grateful.

This is the situation:

I have a 1815i with latest Mobility image working (I have to tftp this image). The WLC web is reachable and the underlying AP is working fine. Previously I have do the same with one 1542i, but as long as we've got the same problem we tried with the 1815, so it not seems to be a problem with one concrete AP or one concrete OS version.

The WLC has been configured with the default options, I tried the two ways: using console wizard and using web wizard. Same problem in both cases.

The problem is that it is not possible to add more APs. They boot and start discover, but discover fails with different mesasages. Actually the mesasage is:

CAPWAP State: Discovery
[*10/01/2021 09:36:10.1611] Discovery Response from 192.168.1.100
[*10/01/2021 09:36:10.2273] Mesh Reconfiguring DHCP.
[*10/01/2021 09:36:11.8681] Start: RPC thread 1968032752 created.
[*10/01/2021 09:36:17.5974] CRIT-MeshWiredAdj[0][A4:88:73:1C:96:F8]: Blacklisting Adjacency due to GW UNREACHABLE
[*10/01/2021 09:36:17.5976] CRIT-MeshWiredAdj[0][A4:88:73:1C:96:F8]: Remove as Parent
[*10/01/2021 09:36:17.5979] CRIT-MeshLink: Link Down Block Root port Mac: A4:88:73:1C:96:F8 BH Id: 0 Port:0 Device:DEVNO_WIRED0
[*10/01/2021 09:36:17.5984] CRIT-MeshWiredBackhaul[0]: Remove as uplink
[*10/01/2021 09:36:17.5990] DOT11_DRV[0]: Stop Radio0 - Begin
[*10/01/2021 09:36:17.5999] Mesh Reconfiguring DHCP.
[*10/01/2021 09:36:17.6196] DOT11_DRV[1]: Stop Radio1 - Begin

Controller is in the 192.168.1.100 and its underlying AP is the 192.168.1.101

Previously I have other messages, but the was derivated from other problems I have been solving (DHCP problems, versions problems, bas WLC configuration problems...)

All the APs are into a isolated and dedicated network (no more APs, routers...). There'are no VLANs or any other kind of complex. All IPs are being assiganed manually for avoiding DHCP problems. All APs have been factory reseted each time I try something new.

I don't know what else to do, and as I mentioned before I can not open a case till Cisco manage my Smartnets, maybe too late for my customer, because this APs are going to be installed in height and all the installation is going to start next Tuesday. After been installed I will not have physicall access to them.

Mobility-Express must run right out-of-the-box, this is the theory, but it doesn't. Please, help.

[Miguel Angel Alvarez Rodriguez]

More information: after reboting one AP I found a new error message prior to the one showed in the original post. This is:

*10/01/2021 09:53:27.4132] DOT11_DRV[1]: Stop Radio1 - Begin
[*10/01/2021 09:53:29.2594] DOT11_DRV[0]: Stop Radio0 - Begin
[*10/01/2021 09:53:29.6731] DOT11_DRV[1]: Stop Radio1 - Begin
[*10/01/2021 09:53:31.4926] DOT11_DRV[0]: Stop Radio0 - Begin
[*10/01/2021 09:53:31.9231] DOT11_DRV[1]: Stop Radio1 - Begin
[*10/01/2021 09:53:33.7327] DOT11_DRV[0]: Stop Radio0 - Begin
[*10/01/2021 09:53:34.1732] DOT11_DRV[1]: Stop Radio1 - Begin
[*10/01/2021 09:53:35.9827] DOT11_DRV[0]: Stop Radio0 - Begin
[*10/01/2021 09:53:36.4232] DOT11_DRV[1]: Stop Radio1 - Begin
[*10/01/2021 09:53:38.2327] DOT11_DRV[0]: Stop Radio0 - Begin
[*10/01/2021 09:53:38.6731] DOT11_DRV[1]: Stop Radio1 - Begin
[*10/01/2021 09:53:40.4827] DOT11_DRV[0]: Stop Radio0 - Begin
[*10/01/2021 09:53:40.9232] DOT11_DRV[1]: Stop Radio1 - Begin
[*10/01/2021 09:53:41.0830]
[*10/01/2021 09:53:41.0830] CAPWAP State: DTLS Teardown
[*10/01/2021 09:53:41.5221] upgrade.sh: Script called with args:[ABORT]
[*10/01/2021 09:53:41.6167] do ABORT, part2 is active part
[*10/01/2021 09:53:41.6628] upgrade.sh: Cleanup tmp files ...
[*10/01/2021 09:53:41.7491] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: DTLS Teardown(4).
[*10/01/2021 09:53:41.7495] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: DTLS Teardown(4).
[*10/01/2021 09:53:42.7835] DOT11_DRV[0]: Stop Radio0 - Begin
[*10/01/2021 09:53:43.1731] DOT11_DRV[1]: Stop Radio1 - Begin
[*10/01/2021 09:53:45.0330] DOT11_DRV[0]: Stop Radio0 - Begin
[*10/01/2021 09:53:45.4931] DOT11_DRV[1]: Stop Radio1 - Begin
[*10/01/2021 09:53:46.0214] No more AP manager addresses remain..
[*10/01/2021 09:53:46.0216] No valid AP manager found for controller 'AP1*****' (ip: 192.168.1.100)
[*10/01/2021 09:53:46.0216] Failed to join controller AP1*****.
[*10/01/2021 09:53:46.0217] Failed to join controller.
[*10/01/2021 09:53:47.3381] DOT11_DRV[0]: Stop Radio0 - Begin
[*10/01/2021 09:53:47.7432] DOT11_DRV[1]: Stop Radio1 - Begin
[*10/01/2021 09:53:49.5733] DOT11_DRV[0]: Stop Radio0 - Begin
[*10/01/2021 09:53:50.0388] DOT11_DRV[1]: Stop Radio1 - Begin
[*10/01/2021 09:53:51.8227] DOT11_DRV[0]: Stop Radio0 - Begin
[*10/01/2021 09:53:52.2933] DOT11_DRV[1]: Stop Radio1 - Begin

[Miguel Angel Alvarez Rodriguez]

Other kind of messaged I'm receiving prior to the one in the original message states as the only messaged showed is this one. This error is related to a mismath version in the OS of the APs, but they are different APs (one is 1815i and the other 1842i), and the latest versions of them are different in the web. Also yesterday I was told by a community member that this is not important, because the controller itself would do the wotk of upgrading.

[*10/01/2021 09:57:32.4252] CAPWAP State: Discovery
[*10/01/2021 09:57:32.4368] Discovery Request sent to 255.255.255.255, discovery type UNKNOWN(0)

[*10/01/2021 09:57:41.9900] CRIT-MeshControl: Could not select an uplink for last 5 minutes. Restarting backhauls
[*10/01/2021 09:57:41.9901] CRIT-MeshControl: Stopping Mesh

[*10/01/2021 09:57:42.0948] CRIT-MeshControl: Starting Mesh

[*10/01/2021 09:57:42.7516] CRIT-MeshWiredBackhaul[0]: Set as uplink
[*10/01/2021 09:57:42.7517] CRIT-MeshWiredAdj[0][A4:88:73:1C:96:F8]: Set as Parent(new)

[*10/01/2021 09:57:42.8002] CRIT-MeshLink: Set Root port Mac: A4:88:73:1C:96:F8 BH Id: 0 Port:0 Device:DEVNO_WIRED0
[*10/01/2021 09:57:42.8016] CRIT-MeshLink: Notify Capwap Link Up: mac: A4:88:73:1C:96:F8 BH Id: 0 GW_reachable: No Roam: false
[*10/01/2021 09:58:01.8819] CAPWAP State: Discovery
[*10/01/2021 09:58:01.8928] Discovery Request sent to 255.255.255.255, discovery type UNKNOWN(0)
[*10/01/2021 09:58:01.8945] Discovery Response from 192.168.1.100
[*10/01/2021 09:58:13.0011]
[*10/01/2021 09:58:13.0011] CAPWAP State: DTLS Setup
[*10/01/2021 09:58:13.9351]
[*10/01/2021 09:58:13.9351] CAPWAP State: Join
[*10/01/2021 09:58:13.9459] Sending Join request to 192.168.1.100 through port 5272
[*10/01/2021 09:58:13.9570] Join Response from 192.168.1.100
[*10/01/2021 09:58:13.9571] AC rejected join request[10], try again later.
[*10/01/2021 09:58:13.9572] Received wlcType 1, timer 120
[*10/01/2021 09:58:14.0160]
[*10/01/2021 09:58:14.0161] CAPWAP State: Image Data
[*10/01/2021 09:58:14.0190] AP image version 8.10.130.0 backup 8.10.162.0, Controller 8.10.151.0
[*10/01/2021 09:58:14.0191] Version does not match.
[*10/01/2021 09:58:14.0865] upgrade.sh: Script called with args:[PRECHECK]
[*10/01/2021 09:58:14.1824] do PRECHECK, part2 is active part
[*10/01/2021 09:58:14.2316] upgrade.sh: /tmp space: OK available 81240, required 40000
[*10/01/2021 09:58:14.2328] wtpImgFileReadRequest: request ap1g5, local /tmp/part.tar
[*10/01/2021 09:58:14.2489] Image Data Request sent to 192.168.1.100, fileName [ap1g5], slaveStatus 0
[*10/01/2021 09:58:14.9434] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Image Data(10).
[*10/01/2021 09:58:14.9438] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Image Data(10).
[*10/01/2021 09:58:16.9917] Re-Tx Count=1, Max Re-Tx Value=5, SendSeqNum=1, NumofPendingMsgs=1
[*10/01/2021 09:58:16.9917]
[*10/01/2021 09:58:19.8429] Re-Tx Count=2, Max Re-Tx Value=5, SendSeqNum=1, NumofPendingMsgs=1
[*10/01/2021 09:58:19.8429]
[*10/01/2021 09:58:20.7547] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Image Data(10).
[*10/01/2021 09:58:20.8608] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Image Data(10).
[*10/01/2021 09:58:22.6941] Re-Tx Count=3, Max Re-Tx Value=5, SendSeqNum=1, NumofPendingMsgs=1
[*10/01/2021 09:58:22.6942]
[*10/01/2021 09:58:25.5454] Re-Tx Count=4, Max Re-Tx Value=5, SendSeqNum=1, NumofPendingMsgs=1
[*10/01/2021 09:58:25.5454]
[*10/01/2021 09:58:28.3968] Re-Tx Count=5, Max Re-Tx Value=5, SendSeqNum=1, NumofPendingMsgs=1
[*10/01/2021 09:58:28.3968]
[*10/01/2021 09:58:31.2480] Max retransmission count exceeded, going back to DISCOVER mode.
[*10/01/2021 09:58:31.2481] Dropping msg CAPWAP_IMAGE_DATA_REQUEST, type = 5, len = 84, eleLen = 92, sendSeqNum = 1
[*10/01/2021 09:58:31.3058]
[*10/01/2021 09:58:31.3059] CAPWAP State: DTLS Teardown
[*10/01/2021 09:58:31.4260] Aborting image download(0x0): Dtls cleanup, ap1g5
[*10/01/2021 09:58:31.6310] upgrade.sh: Script called with args:[ABORT]
[*10/01/2021 09:58:31.7271] do ABORT, part2 is active part
[*10/01/2021 09:58:31.7693] upgrade.sh: Cleanup tmp files ...
[*10/01/2021 09:58:31.8506] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: DTLS Teardown(4).
[*10/01/2021 09:58:31.8508] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: DTLS Teardown(4).
[*10/01/2021 09:58:34.3765] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: DTLS Teardown(4).
/2021 09:57:04.7794] DOT11_DRV[1]: Stop Radio1 - Begin

[Rich R · Accepted solution]

What AP model are you trying to join?

And what mode is that AP set to? (I see mesh in those logs - are you planning to run them as mesh?)

If you're not using mesh then set the AP to local mode.

Even though it's an isolated subnet you might need to set a "default gateway" that the AP can ping/ARP - maybe use the WLC for that if you don't have an actual router (see previous discussions about bugs affecting AP behaviour with default gateway reachability)

What debugs do you see for the AP join on the controller? And

"show ap join stats summary all"

"show ap join stats detailed <base wireless MAC>"

Not applicable till the AP can join - but have you set up a TFTP server for the APs to download the image from? 

[Miguel Angel Alvarez Rodriguez]

Hello. Thanks for answering.

WLC AP is a 1815i. APs Í'm tryind to add are 1542i (there will be also some other 1815i)

Don't know exactly WhatsApp Mesh neand. I want all of them beign controles by the WLC controle as if it worl be a physical one WLAN controler.

The only chage I made to the APs has been upgrading their firmware to a latest versión of ME and to add them static IP address un the world interface.

How do I set local model?

I try to entable the tftp service un the controler using web interface and a PC with tftpd32 AMD the light me imagen downloaded, but i' ve not been able to make it work. That's why I final upgradecthe APs one by one manual.

[Miguel Angel Alvarez Rodriguez]

I'm reaching to the office. When arribe I'll send you the debut output. Thanks.

[Miguel Angel Alvarez Rodriguez]

Hi rrdudling:

Change the APs mode to local made the trick!!

Thanks a lot. You save me

Note to Cisco: all mobility express videos shows a two minutes installation procees without any kind of manual intervention. I don 't understand why I need to do all these stuff to make it works.

[Rich R]

Glad to hear you got it working @Miguel Angel Alvarez Rodriguez 

The problem is that some of the outdoor APs are shipped pre-configured as mesh and that can cause a whole lot of problems!

It's actually a selectable option when you order:

So whoever ordered your APs must have selected the wrong option!