Randomly wireless APs getting reboot. Last reboot reason showing as "Power loss". But there is no any power issues.

[SMK91]

Hi Guys,

I have 42 wireless APs (cisco AIR-CAP3702E-E-K9). All the APs getting rebooted time to time without any pattern. The last rebooted reason showing as "Power loss".

LKKURXA157 uptime is 1 day, 10 hours, 53 minutes
System returned to ROM by power-on
System image file is "flash:/ap3g2-k9w8-mx.153-3.JF10/ap3g2-k9w8-xx.153-3.JF10"
Last reload reason: Power Loss

Until now, we have done several things to resolve this issue. Until no any improvement.

1. Upgraded to latest recommended version.

2. To isolate the power issue we have Installed a PoE switch to provide the PoE power to one AP (earlier all the APs powered up with power injector).

2. Monitor the states of the selected POE powered AP , Observed that AP also showing last reboot reason as "Power Loss". But PoE switch has good uptime.

3. Enabled the AP core dump feature. But there is no any crash logs.

These are the logs generated from APs.

---------------Show msglog---------------

Message Log Severity Level ...................... ERROR
*spamApTask0: Jan 02 15:03:37.983: %LWAPP-3-VENDOR_PLD_VALIDATE_ERR: spam_lrad.c:11913 Validation of SPAM_VENDOR_SPECIFIC_PAYLOAD(185) with length=9 failed - AP 84:b2:61:b9:d5:10
*spamApTask5: Jan 02 15:03:27.282: %CAPWAP-3-DTLS_CLOSED_ERR: capwap_ac_sm.c:7076 84:b2:61:b9:d5:10: DTLS connection closed forAP 10:110:16:121 (55340), Controller: 10:110:16:23 (5246) Echo Timer Expiry
*spamApTask5: Jan 02 15:03:27.281: %CAPWAP-3-ECHO_ERR: capwap_ac_sm.c:7821 Did not receive heartbeat reply; AP: 84:b2:61:b9:d5:10
*Dot1x_NW_MsgTask_2: Jan 02 15:01:42.485: %DOT1X-3-ABORT_AUTH: 1x_bauth_sm.c:487 Authentication Aborted for client 78:61:7c:28:c8:4a Abort Reason:DOT1X RESTARTED DUE TO EAPOL-START/CLIENT ROAM

Please advise.

[marce1000]

 - Followup on the switch logs too; preferably configure a syslog server. if this is a controller based solution the configure syslogging for all AP's with the command seen below. Watch for error-like correlation events between the switch and the AP's on the syslog server.

 M.

config ap syslog host global.

[SMK91]

Hi marce1000,

Thanks for your reply. I'll check the capability of arranging a syslog server. Do you have any experience with this kind of issue? Please advice.

[marce1000]

 - No experience and I think it is for 99% certainty external. By that I mean not related to the native Cisco wireless environment it's capabilities or lack thereof. One thing to try is , to power some AP's manually (no POE) and others with POE , see if that makes any difference as which category of AP's is going down.

 M.

[SMK91]

- There is only on AP with the PoE power (Arrange to monitor separately). All other APs are with direct power. There are no any differences. All the APs getting down randomly and showing last reboot reason as "Power loss".

[marce1000]

 - I state my case as to being external as or really meaning Power loss. But now my analysis becomes more exotic : monitor for instance an AP with a webcam have a video-recording for several hours. At time of power loss is standard power-up sequence of the AP seen in terms of led pattern for startup. If so it really is power loss....

 M.

[SMK91]

- Thanks for the update. I'll checked the that one also.

[Scott Fella]

Do you have any modules connected to these AP’s by chance, because I know that the modules can cause ap’s to reload.

[SMK91]

-Scott-
These APs worked fine more than one year without any issue. We haven't change the setup. Suddenly started to face this issue.

[Scott Fella]

Let me ask… do you have any modules in the ap’s?

[SMK91]

Hi Scott-
Which kind of modules?

[Scott Fella]

Any of the modules that gets installed on the underneath side of the AP’s. If you don’t install a module, you are better off.

[SMK91]

-Hi Scott-
We haven't install any modules..

[Scott Fella]

Then I don’t know. Could be code, port issues, or other things. Tough to say without actually having all the data. You have a TAC case open?

[SMK91]

Hi Scott;
Already opened TAC case, until now requeue the case within 4 engineers. All are not aware about this type of case. Time to time they collecting same logs. Sorry to say there is no help from TAC side.