Flexconnect 1261 APs boot loop after upgrade to 8.0 MR3

Francesco Pettene · ‎04-10-2016

Hello

after upgrading a cluster of 5508s to 8.0MR3 from 7.6.130, 6 Flexconnect 1261 APs
started looping, losing their registration after like 10 seconds and then power
cycling themselves.

In this WLC cluster i have lots of 1262 (dual band) working fine without any kind of loop.

This is the output of "debug capwap event enable" grepping out all unrelevant lines:
have you got any ideas?

Thank you
fp


(Cisco Controller) >*spamApTask6: Apr 09 17:23:38.524: [PA] 54:78:1a:a0:41:c0 Echo Timer Expiry: Did not receive heartbeat reply from AP 54:78:1a:a0:41:c0 (10:178:31:140/14658)
*spamApTask5: Apr 09 17:28:01.924: [PA] 54:78:1a:a0:41:c0 Echo Timer Expiry: Did not receive heartbeat reply from AP 54:78:1a:a0:41:c0 (10:178:31:140/14657)
*spamApTask6: Apr 09 17:32:25.524: [PA] 54:78:1a:a0:41:c0 Echo Timer Expiry: Did not receive heartbeat reply from AP 54:78:1a:a0:41:c0 (10:178:31:140/14658)
*spamApTask5: Apr 09 17:36:49.528: [PA] 54:78:1a:a0:41:c0 Echo Timer Expiry: Did not receive heartbeat reply from AP 54:78:1a:a0:41:c0 (10:178:31:140/14657)
(Cisco Controller) >*spamApTask6: Apr 09 17:41:14.336: [PA] 54:78:1a:a0:41:c0 Echo Timer Expiry: Did not receive heartbeat reply from AP 54:78:1a:a0:41:c0 (10:178:31:140/14658)
*spamApTask5: Apr 09 17:45:38.736: [PA] 54:78:1a:a0:41:c0 Echo Timer Expiry: Did not receive heartbeat reply from AP 54:78:1a:a0:41:c0 (10:178:31:140/14657)
*spamApTask6: Apr 09 17:50:02.736: [PA] 54:78:1a:a0:41:c0 Echo Timer Expiry: Did not receive heartbeat reply from AP 54:78:1a:a0:41:c0 (10:178:31:140/14658)
*spamApTask5: Apr 09 17:54:28.136: [PA] 54:78:1a:a0:41:c0 Echo Timer Expiry: Did not receive heartbeat reply from AP 54:78:1a:a0:41:c0 (10:178:31:140/14657)
*spamApTask6: Apr 09 17:57:12.123: [PA] 54:78:1a:a0:41:c0 Discovery Request from 10.178.31.140:14658
*spamApTask6: Apr 09 17:57:12.123: [PA] 54:78:1a:a0:41:c0 Discovery Response sent to 10.178.31.140 port 14658
*spamApTask6: Apr 09 17:57:12.123: [PA] 54:78:1a:a0:41:c0 Discovery Response sent to 10.178.31.140:14658
*spamApTask6: Apr 09 17:57:22.059: [PA] 00:1e:4a:a6:d4:7f DTLS connection not found, creating new connection for 10:178:31:140 (14658) 10:177:159:253 (5246)
*spamApTask6: Apr 09 17:57:22.609: [PA] acDtlsPlumbControlPlaneKeys: lrad:10.178.31.140(14658) mwar:10.177.159.253(5246)
*spamApTask6: Apr 09 17:57:22.609: [PA] 00:1e:4a:a6:d4:7f DTLS keys for Control Plane are plumbed successfully for AP 10.178.31.140. Index 332
*spamApTask3: Apr 09 17:57:22.609: [PA] 00:1e:4a:a6:d4:7f DTLS Session established server (10.177.159.253:5246), client (10.178.31.140:14658)
*spamApTask3: Apr 09 17:57:22.609: [PA] 00:1e:4a:a6:d4:7f Starting wait join timer for AP: 10.178.31.140:14658
*spamApTask6: Apr 09 17:57:22.675: [PA] 54:78:1a:a0:41:c0 Join Request from 10.178.31.140:14658
*spamApTask6: Apr 09 17:57:22.676: [PA] 00:1e:4a:a6:d4:7f Deleting AP entry 10.178.31.140:14658 from temporary database.
*spamApTask6: Apr 09 17:57:22.677: [PA] 54:78:1a:a0:41:c0 Dropping request or response packet to AP :10.178.31.140 (14658)  by Controller: 10.177.159.253 (5246), message Capwap_wtp_event_response, state Capwap_no_state
*spamApTask6: Apr 09 17:57:22.677: [PA] 54:78:1a:a0:41:c0 Message type Capwap_wtp_event_response is not allowed to send in state Capwap_no_state for AP 10.178.31.140
*spamApTask6: Apr 09 17:57:22.678: [PA] 54:78:1a:a0:41:c0 Join Response sent to 10.178.31.140:14658
*spamApTask6: Apr 09 17:57:23.050: [PA] 54:78:1a:a0:41:c0 Configuration Status from 10.178.31.140:14658
*spamApTask6: Apr 09 17:57:23.051: [PA] 54:78:1a:a0:41:c0 Updating IP info for AP 54:78:1a:a0:41:c0 -- static 1, 10.178.31.140/255.255.255.128, gtw 10.178.31.129
*spamApTask6: Apr 09 17:57:23.110: [PA] 54:78:1a:a0:41:c0 Configuration Status Response sent to 10:178:31:140

Ric Beeching · ‎04-10-2016

Can you do a show ap join stats detailed 54:78:1a:a0:41:c0

This may provide a reason for the disconnect.

-----------------------------
Please rate helpful / correct posts

Francesco Pettene · ‎04-10-2016

Hello Ric

here it is:

(Cisco Controller) >show ap join stats detailed 54:78:1a:a0:41:c0

Sync phase statistics
- Time at sync request received............................ Not applicable
- Time at sync completed................................... Not applicable

Discovery phase statistics
- Discovery requests received.............................. 118
- Successful discovery responses sent...................... 118
- Unsuccessful discovery request processing................ 0
- Reason for last unsuccessful discovery attempt........... Not applicable
- Time at last successful discovery attempt................ Apr 11 08:23:38.318
- Time at last unsuccessful discovery attempt.............. Not applicable

Join phase statistics
- Join requests received................................... 59
- Successful join responses sent........................... 59
- Unsuccessful join request processing..................... 0
- Reason for last unsuccessful join attempt................ Not applicable
- Time at last successful join attempt..................... Apr 11 08:23:48.870
- Time at last unsuccessful join attempt................... Not applicable

Configuration phase statistics

--More or (q)uit current module or <ctrl-z> to abort
- Configuration requests received.......................... 119
- Successful configuration responses sent.................. 60
- Unsuccessful configuration request processing............ 0
- Reason for last unsuccessful configuration attempt....... Not applicable
- Time at last successful configuration attempt............ Apr 11 08:23:49.304
- Time at last unsuccessful configuration attempt.......... Not applicable

Last AP message decryption failure details
- Reason for last message decryption failure............... Not applicable

Last AP disconnect details
- Reason for last AP connection failure.................... Image data request received from an unsupported AP
- Last AP disconnect reason................................ AP is set to static IP

Last join error summary
- Type of error that occurred last......................... AP got or has been disconnected
- Reason for error that occurred last...................... Image data request received from an unsupported AP
- Time at which the last join error occurred............... Apr 11 08:25:18.804

AP disconnect details
- Reason for last AP connection failure.................... Image data request received from an unsupported AP
Ethernet Mac : fc:99:47:f3:94:0f Ip Address : 10.178.31.140

Ric Beeching · ‎04-10-2016

Interesting - I did check the compatibility matrix first but 1260s are listed as supported for 7.6.130.0.

Compatibility Matrix

You could try downloading a recovery image that is lower than previous, manually upload this to one of the affected APs and see if booting on to that allows the WLC to recognise it. Otherwise I'd log a TAC case if you have support with Cisco or if not change to a different code version.

Ric

-----------------------------
Please rate helpful / correct posts

Francesco Pettene · ‎04-12-2016

Hello Ric

1260s are listed as supported also under 8.0.132.0:

8.0.132.0

15.3(3)JA7/
12.4(25e)JAP7

Lightweight APs: 1040, 1130, 1140, 1240, 1250, 1260, 1600, 1700, 2600, 2700, 3500e, 3500i, 3500p, 3600e, 3600i, 3600p, 3702e, 3702i, 3702p, 600 OEAP, 700, 700W, AP801, and AP802

Outdoor and Industrial APs: 1522, 1524PS, 1524SB, 1532E, 1532I, 1552E, 1552H, 1552I, 1552C, 1552EU, 1552CU, 1552S, 1570, and IW3702

TAC is currently requesting crashinfos and AP console output during the issue, which

is pretty difficult because all devices are installed in a warehouse on high walls.