All Flexconnect AP admin status disabled after WAN failure

[Kanan Huseynli]

Hi,

we have two different sites which are connected via cisco SD-WAN. HQ site has WLCs (5508 in HA), branch site has APs in flexconnect mode (2602E and 3702I). When ISP failure happens and we lose branch site. We see that these APs are shown in WLC with admin staus "disabled" after site is restored (AP rejoins to WLC). We have to do manually enable admin status.

What could be the reason for this? How to do troubleshooting properly?

Thanks!

[marce1000]

 - Not a direct response but since both 5508 and aireos are getting older and according to https://www.cisco.com/c/en/us/support/docs/wireless/wireless-lan-controller-software/200046-tac-recommended-aireos.html  use https://software.cisco.com/download/specialrelease/8f166c6d88b9f77aabb63f78affa9749 on the 5508 (strongly recommended)  , 

 M.

[Kanan Huseynli]

Hi thanks,

we are ordering WLC9800 , but since 5508 EoS/EoL we dont want to touch it globally (like image upgrade).

[marce1000]

                >.... we dont want to touch it globally (like image upgrade).
  Well sorry , but for these kind of issues  that is usually a vital step to try , consider that  a showstopper , 
a topic that is even more important for the 5508 because of the EoS/EOL as explained

 M.

[Rich R]

That sounds like buggy behaviour so as Marce says your only option is to update the software and hope it's fixed.  We've never seen that happen with our APs though.

What version are you currently using?

[Kanan Huseynli]

Hi,

thanks for you reply too.

AP 2602E:

Software version: 8.5.135.0
Boot version: 12.4.25.0
IOS version: 15.3(3)JF8$

IQ_BAGHDAD_AP04# sh boot
BOOT path-list: flash:/ap3g2-k9w8-mx.153-3.JF8/ap3g2-k9w8-mx.153-3.JF8

WLC5508:

Software version: 8.5.135.0

By the way, in environment we have actually three pair of WLCs. WLC5508 ; another WLC5508 and WLC5760 (w/ 03.06.07E ct5760-ipservicesk9)

What I noticed, in DHCP option 43 one of the dynamic interface of WLC 5760 is configured (before our team, honestly don't know details and I'm not good at wireless). So, basically DHCP gives IP of 5760. However, in AP configuration it is mentioned 5508WLC two IP addresses: primary is WLC5508 secondary is another WLC5508.

I checked tshoot guide, device should try to connect all WLC (whatever it knows) and if all answers it will connect based on config order (in order, it is mentioned primarily 5508 WLC).

To simulate WAN failure , I blocked traffic from one of the AP to WLC and after some time removed, it worked normal.

[Rich R]

So it sounds like it actually worked normally as expected?

That version of code is rather old.  If you want to avoid problem listed in field notice below suggest to update to 8.5.182.7 (link below).

[Kanan Huseynli]

Sorry for the delay.

Yesterday I did one more test. I blocked connection (via regular access-list on gateway SVI) to WLC 5508 and allowed only to WLC 5760.

AP has connected to 5760 and I have seen below logs:

*May 24 15:03:14.275: %EVT-4-WRN: Write of flash:/event.capwap done
*May 24 15:03:14.295: %LWAPP-3-CLIENTERRORLOG: Switching to Standalone mode
*May 24 15:03:14.307: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 5760_WLC_IP :5246
*May 24 15:03:14.315: %CLEANAIR-6-STATE: Slot 0 down
*May 24 15:03:14.315: %CLEANAIR-6-STATE: Slot 1 down
*May 24 15:01:40.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 5760_WLC_IP peer_port: 5246
*May 24 15:01:40.319: %CAPWAP-5-DTLSREQSUCC: DTLS connection created sucessfully peer_ip: 5760_WLC_IP peer_port: 5246
*May 24 15:01:40.319: %CAPWAP-5-SENDJOIN: sending Join Request to 5760_WLC_IP
*May 24 15:01:45.319: %CAPWAP-5-SENDJOIN: sending Join Request to 5760_WLC_IP .3perform archive download capwap:/ap3g2 tar file
*May 24 15:01:45.375: %CAPWAP-6-AP_IMG_DWNLD: Required image not found on AP. Downloading image from Controller.

After AP downloaded image and registered to WLC5760, I removed access-list from gateway and AP began to connect to WLC 5508.

*May 24 15:13:28.419: %DTLS-5-ALERT: Received WARNING : Close notify alert from 5760_WLC_IP
*May 24 15:13:28.419: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 5760_WLC_IP :5246
*May 24 15:13:33.387: AP image integrity check PASSED

*May 24 15:15:23.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 5508_WLC_IP peer_port: 5246
*May 24 15:15:24.207: %CAPWAP-5-DTLSREQSUCC: DTLS connection created sucessfully peer_ip: 5508_WLC_IPpeer_port: 5246
*May 24 15:15:24.207: %CAPWAP-5-SENDJOIN: sending Join Request to 5508_WLC_IP

It joined to 5508 (downloaded image from 5508 in this case) and registered. After successful download reboot&register I see admin status down.

Can it be related to image change?

[Rich R]

> Can it be related to image change?
Very likely.

Also - it's really bad to have an AP switching between WLCs with completely different software versions like that - if you use N+1 then the WLCs should be running the same version of software so that no download is required when an AP switches.
Moreover 5760 is end of support and the last software release was in 2017 so you must be using very old software!

[Kanan Huseynli · Accepted solution]

Yes, team before us did such design maybe because of just to track AP.

After downloading image from 5760 and registering status is normal

After switching back up 5508 (when it is available) and downloading&registering image from 5508, admin status is disabled.

Here I think there are three options:

a) it is bug

b) it is expected behavior (like when UP upgraded 5508 makes it admin status down so manual enable is needed)

3) it is expected behavior via configuration (but I didn't find any option regarding this behavior)

is there such configuration like default status is down after upgrade?

P.S we are at the stage of ordering C9800, so should work with these old guys for some time, unfortunately..