Converted CAPWAP AP (1831i) reboots into ME Provision mode

[Marcel Tempelman]

Hello all,

we have a customer who has several locations with a Mobility Express setup. At one location we have nothing but issues. We have fixed several issues but one remains. One access point functions as a static ME controller but the other APs do not join this controller. They keep on rebooting and booting into ME Provisioning mode.

What we did:

- Upgrade from 8.4.101 to 8.6.101

- Downgrade to recommended version 8.5.131

- Upgrade to 8.7.101

- factory reset all aps and rebuilt the whole setup.

All AP have been factory reset manually and changed to AP-Type CAPWAP (this is one suggested fix in a bug). I saw that the APs stayed at CAPWAP so I converted them in the ME GUI to ME. What happens then is that the AP reboots, skips the controller discovery and starts the ME Wizard. What also happens is that it claims the priority in VRRP which results in being unable to access the working ME Controller. This location has 5 1832i in ME mode. 1 is still working. When I startup the other APs the Virtual MAC address of the controller travels around on every AP. Every 2-3 seconds the vMAC changes ports.

I have never encountered this problem at the other locations. I could safely factory reset an AP. Till now I have not been able to find a bug regarding this. We're going to pull the APs back to our office for testing and being able to walk this through with TAC but in the meantime I was wondering if someone has seen this same behavior.

With regards,

Marcel Tempelman.

[Jhosbell Verdesca De Aguiar]

Is the Master AP - or the AP running as WLC - able to ping its configured default gateway?

Otherwise it won't respond to any discovery request. His own AP won't even join the ME WLC. So make sure it can ping the gateway.

[Marcel Tempelman]

The master AP is managable. The WLC IP is reachable from the management server whic resides on a complete different subnet at another location. All APs are in the same subnet. The APs that are ME Capable are just skipping the discovery. It seems the discovery works if the APs stay in CAPWAP mode.

Regards,

Marcel.

[Marcel Tempelman]

The APs are connected to a 2960L switch. I disabled ip igmp snooping:

see:

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvf15791/?rfs=iqvred

This way I was able to stay connected to the WLC interface.

The AP gets an IP address from the DHCP server (a local 891 router)

Sep  3 12:51:35 kernel: [*09/03/2018 12:51:35.3512] ethernet_port wired0, ip 10.43.10.126, netmask 255.255.255.0, gw 10.43.10.254, mtu 1500, bcast 10.43.10.255, dns1 192.168.250.151, domain wlan.company.local, vid 0, static_ip_failover false, dhcp_vlan_f

Here is the discovery:

Sep  3 12:52:26 kernel: [*09/03/2018 12:52:26.3752] dtls_init: Use MIC certificate
Sep  3 12:52:26 kernel: [*09/03/2018 12:52:26.3752]
Sep  3 12:52:26 kernel: [*09/03/2018 12:52:26.3752] CAPWAP State: Init
Sep  3 12:52:26 kernel: [*09/03/2018 12:52:26.3752]
Sep  3 12:52:26 kernel: [*09/03/2018 12:52:26.3752] PNP is not required, Starting CAPWAP discovery
Sep  3 12:52:26 kernel: [*09/03/2018 12:52:26.3752]
Sep  3 12:52:26 kernel: [*09/03/2018 12:52:26.3752]
Sep  3 12:52:26 kernel: [*09/03/2018 12:52:26.3752] CAPWAP State: Discovery
Sep  3 12:52:26 kernel: [*09/03/2018 12:52:26.3852] Discovery Request sent to 10.43.10.124, discovery type STATIC_CONFIG(1)
Sep  3 12:52:26 kernel: [*09/03/2018 12:52:26.3852]
Sep  3 12:52:26 kernel: [*09/03/2018 12:52:26.3852] CAPWAP State: Discovery
Sep  3 12:52:27 kernel: [*09/03/2018 12:52:27.3849] Discovery Response from 10.43.10.124
Sep  3 12:52:30 kernel: [*09/03/2018 12:52:30.1741] chatter: tohost_virtual :: ToHost: device 'virtual' came up
Sep  3 12:52:35 kernel: [*09/03/2018 12:52:35.8823]
Sep  3 12:52:35 kernel: [*09/03/2018 12:52:35.8823] CAPWAP State: DTLS Setup

The 10.43.10.124 is an IP address obtained from the DHCP server. The Master AP is at .240.

In earlier troubleshooting sessions we saw that the DHCP servers of the AP became active and then it gave itself a 192.168 address.

I have converted the APs back to CAPWAP and now they have joined the master AP again.

Regards,

Marcel Tempelman

[Marcel Tempelman]

I do not have 100% proof but his seems to be the fix:

The router at the location had multicast routing enabled en ip pim spare mode enable on the management VLAN (in which the APs reside). If I disable both and disable all APs but one. If the controller page is available again on this AP, I enable the other APs again (I must make sure the controller has a working TFTP-server configured with the right firmware). This way all APs come back, join, load the right firmware and everybody is happy again.

I had a lab withut multicast routing and it worked without a glitch, so I disabled it a location which had problems and got that one working again. Want to lab that but you know time....

with kind regards,

Marcel Tempelman.