AP3702 often rejoin to WLC3504

[slyfox]

Hello Community, 

4 of 45 AIR-CAP3702I-E-K9 very often rejoin WLC AIR-CT3504-K9 Version: 8.8.120.0
debug showed the following errors:

*spamApTask3: Aug 14 14:15:53.936: [PA] 00:c8:8b:ec:58:90 Max retransmissions reached on AP (CAPWAP_CONFIGURATION_UPDATE_REQUEST, 2)
*spamApTask3: Aug 14 14:15:53.937: [PA] 00:c8:8b:ec:58:90 AP Message Timeout: Max retransmissions reached on AP 00:C8:8B:EC:58:90
*spamApTask3: Aug 14 14:15:53.938: [PA] 00:c8:8b:ec:58:90 Event = Capwap_msg_timer_expiry State = Capwap_run
*spamApTask3: Aug 14 14:15:53.938: [PA] Failed to process timer message 1
*spamApTask3: Aug 14 14:18:36.879: [PA] 00:c8:8b:ec:58:90 Unable to get Ap mode in Join request
*spamApTask3: Aug 14 14:18:36.977: [PA] 00:c8:8b:ec:58:90 LWAPP message validation failed for SPAM_VENDOR_SPECIFIC_PAYLOAD(104) in message of len=9 from AP 00:c8:8b:ec:58:90
*spamApTask3: Aug 14 14:18:36.977: [PA] 00:c8:8b:ec:58:90 Failed to validate vendor specific message element
*spamApTask3: Aug 14 14:18:36.978: [PA] 00:c8:8b:ec:58:90 Recieved Invalid Fragmentation Threshold From AP = 0
*spamApTask3: Aug 14 14:18:36.978: [PA] 00:c8:8b:ec:58:90 MAC_OPERATION : InvalidFragmentation Threshold = 0, Resetting it toDefault: 2346
*spamApTask3: Aug 14 14:18:36.979: [PA] 00:c8:8b:ec:58:90 [Slot 2] Corrected infeasible channel (36,0,20) -> (112,0,20)
*spamApTask3: Aug 14 14:18:36.982: [PA] 00:c8:8b:ec:58:90 Channel 140 is invalid in the domain
*spamApTask3: Aug 14 14:18:36.982: [PA] 00:c8:8b:ec:58:90 [Slot 2] Corrected infeasible channel (112,0,20) -> (44,0,20)
*spamApTask3: Aug 14 14:18:36.983: [PA] 00:c8:8b:ec:58:90 MAC_OPERATION : InvalidFragmentation Threshold = 0, Resetting it toDefault: 2346
*spamApTask3: Aug 14 14:18:37.818: [PA] ---begin send 11ax related vap msg to ap
*spamApTask3: Aug 14 14:18:37.818: [PA] ---begin send 11ax related vap msg to ap
*spamApTask3: Aug 14 14:18:37.818: [PA] ---begin send 11ax related vap msg to ap
*spamApTask3: Aug 14 14:18:37.818: [PA] ---begin send 11ax related vap msg to ap
*spamApTask3: Aug 14 14:18:37.818: [PA] ---begin send 11ax related vap msg to ap
*spamApTask3: Aug 14 14:18:37.819: [PA] ---begin send 11ax related vap msg to ap
*spamApTask3: Aug 14 14:18:37.819: [PA] ---begin send 11ax related vap msg to ap
*spamApTask3: Aug 14 14:18:37.819: [PA] ---begin send 11ax related vap msg to ap
*spamApTask3: Aug 14 14:18:37.819: [PA] ---begin send 11ax related vap msg to ap
*spamApTask3: Aug 14 14:18:37.819: [PA] 00:c8:8b:ec:58:90 AP 00:c8:8b:ec:58:90 has default credentials and so invalid to support Telnet
*spamApTask3: Aug 14 14:18:37.819: [PA] 00:c8:8b:ec:58:90 AP 00:c8:8b:ec:58:90 has default credentials and so invalid to support SSH
*spamApTask3: Aug 14 14:18:40.837: [PA] 00:c8:8b:ec:58:90 Channel 0 is invalid in the domain
*spamApTask3: Aug 14 14:18:40.838: [PA] 00:c8:8b:ec:58:90 [Slot 2] Corrected infeasible channel (44,0,20) -> (116,0,20)


What could be a known reason for this?

[Scott Fella]

Since 4 out of 45 AP’s have this, a few things to look at when troubleshooting. Did you verify the hardware of these 4 are the same with working AP’s. You would see this from the console or you can ssh to the ap and verify the hardware. Are these connected to a switch that has known good working AP’s? Have you tried to factory reset the ap to see if that helps? Are these 4 in the same area or mixed in with known good AP’s? Always also test the cable, make sure the ports are clean, if you have a module connected, re-seat it and or remove it to see if that fixed the issue.

[slyfox]

Hello @Scott Fella 
All 4 trouble AP are connected to the different switches and located in different areas.
Switches model all the same everywhere IE-4010-4S24P (Version: 15.2(7)E2 IE4010-UNIVERSALK9-M)
Rest access points are connected to the same switches and are working normally.
No, I did not perform a factory reset.
Cable was replaced. Switches are new, the port also was changed. 
What do you mean by hardware verification?

[Scott Fella]

Look at the show ap inventory or ssh into the ap and verify the hardware. Make sure the radios match, etc. you don’t want a manufacturing issue (wrong radio or regulatory domain) making you chase your tail. Do factory reset a few of the aps and monitor. I’m guessing there are not module connected to these AP’s.

[Leo Laohoo]

Console into one of the four APs and post the logs of the times when the AP left and joined the WLC.

[slyfox]

@Leo Laohoo 

*Aug 17 01:36:48.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.84.10.5 peer_port: 5246
*Aug 17 01:36:48.223: %CAPWAP-5-DTLSREQSUCC: DTLS connection created sucessfully peer_ip: 10.84.10.5 peer_port: 5246
*Aug 17 01:36:48.223: %CAPWAP-5-SENDJOIN: sending Join Request to 10.84.10.5
*Aug 17 01:36:53.223: %CAPWAP-5-SENDJOIN: sending Join Request to 10.84.10.5
*Aug 17 01:36:53.423: %DOT11-5-EXPECTED_RADIO_RESET: Restarting Radio interface Dot11Radio0 due to the reason code 56
*Aug 17 01:36:53.427: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to down
*Aug 17 01:36:53.435: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to reset
*Aug 17 01:36:54.055: %DOT11-5-EXPECTED_RADIO_RESET: Restarting Radio interface Dot11Radio1 due to the reason code 56
*Aug 17 01:36:54.071: %DOT11-5-EXPECTED_RADIO_RESET: Restarting Radio interface Dot11Radio0 due to the reason code 10
*Aug 17 01:36:54.079: %CAPWAP-5-JOINEDCONTROLLER: AP has joined controller wlc1
*Aug 17 01:36:54.163: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to up
*Aug 17 01:36:54.259: %WIDS-6-ENABLED: IDS Signature is loaded and enabled
*Aug 17 01:36:54.427: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to down
*Aug 17 01:36:54.475: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to down
*Aug 17 01:36:54.483: %LINK-5-CHANGED: Interface Dot11Radio1, changed state to reset
*Aug 17 01:36:55.163: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to down
*Aug 17 01:36:55.291: %LINEPROTO-5-UPDOWN: Line protocol on Interface NVI0, changed state to up
*Aug 17 01:36:55.463: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to up
*Aug 17 01:36:55.519: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to up
*Aug 17 01:36:55.527: %LINK-6-UPDOWN: Interface Dot11Radio2, changed state to down
*Aug 17 01:36:55.535: %LINK-5-CHANGED: Interface Dot11Radio2, changed state to reset
*Aug 17 01:36:56.519: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to up
*Aug 17 01:36:56.527: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio2, changed state to down
*Aug 17 01:36:56.555: %LINK-6-UPDOWN: Interface Dot11Radio2, changed state to up
*Aug 17 01:36:56.563: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to down
*Aug 17 01:36:56.571: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to reset
*Aug 17 01:36:57.555: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio2, changed state to up
*Aug 17 01:36:57.563: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to down
*Aug 17 01:36:57.603: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to up
*Aug 17 01:36:58.603: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to up
*Aug 17 01:37:29.603: %CLEANAIR-6-STATE: Slot 0 enabled
*Aug 17 01:37:30.911: %CLEANAIR-6-STATE: Slot 1 enabled
*Aug 17 01:37:32.247: %CLEANAIR-6-STATE: Slot 2 enabled

[Leo Laohoo]

I wouldn't be using 8.8.120.0.
I've used that version and ran into lots of problem.

[slyfox]

@Leo Laohoo 
well, after the replacement of one of the affected AP I would say this could be a case of ios version.
advise which version to take a look on

[patoberli]

My 2700 and 3700 APs also reboot regularly because of a software crash, but I'm running 8.5.161.0.
There were a few crash bugs fixed in 8.8.130.0, you might want to give that version a try:
https://www.cisco.com/c/en/us/td/docs/wireless/controller/release/notes/crn88mr3.html#resolved-caveats

[marce1000]

 - Are you sure these ap's belong to the same regulatory domain as the working ones ?

 M.

[slyfox]

@marce1000 
Yes, they all belong to
Regulatory Domains: 802.11bg:-E 802.11a:-E