topic Re: 2702 AP logs in Wireless

2702 AP logs

Moudar — Tue, 13 Jun 2023 10:09:05 GMT

I got this AP who did restarted CAPWAP on its own. That was very disruptive to clients!

uptime is 29 weeks, 1 day, 21 hours, 14 minutes.

What can the problem be?

AP 2702, WLC 5520 running 8.10.181.3

Logs:

*Jun 13 06:57:56.431: Failed to read from HW TRNG *Jun 13 06:57:56.431: HW TRNG failed! *Jun 13 07:26:04.995: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 10.0.0.148:5246 *Jun 13 07:26:04.999: %DOT11-5-EXPECTED_RADIO_RESET: Restarting Radio interface Dot11Radio0 due to the reason code 27 *Jun 13 07:26:04.999: %DOT11-5-EXPECTED_RADIO_RESET: Restarting Radio interface Dot11Radio1 due to the reason code 27 *Jun 13 07:26:04.999: %WIDS-6-DISABLED: IDS Signature is removed and disabled. *Jun 13 07:26:05.063: %CAPWAP-5-AP_EASYADMIN_INFO: AP Easy Admin information - EASY_ADMIN is not set, turn off easy admin service! *Jun 13 07:26:05.063: %CAPWAP-5-AP_EASYADMIN_INFO: AP Easy Admin information - Easy Admin is not enabled, turn it off! *Jun 13 07:26:05.087: %DOT11-5-EXPECTED_RADIO_RESET: Restarting Radio interface Dot11Radio0 due to the reason code 39 *Jun 13 07:26:05.087: %DOT11-5-EXPECTED_RADIO_RESET: Restarting Radio interface Dot11Radio1 due to the reason code 39 *Jun 13 07:26:05.099: %CLEANAIR-6-STATE: Slot 0 down *Jun 13 07:26:05.103: %CLEANAIR-6-STATE: Slot 1 down *Jun 13 07:26:05.103: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to administratively down *Jun 13 07:26:05.103: %LINK-5-CHANGED: Interface Dot11Radio1, changed state to administratively down *Jun 13 07:26:05.103: %DOT11-5-EXPECTED_RADIO_RESET: Restarting Radio interface Dot11Radio0 due to the reason code 10 *Jun 13 07:26:05.107: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to up *Jun 13 07:26:05.711: %DOT11-5-EXPECTED_RADIO_RESET: Restarting Radio interface Dot11Radio1 due to the reason code 10 *Jun 13 07:26:05.739: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to up *Jun 13 07:26:06.103: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to down *Jun 13 07:26:06.131: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to down *Jun 13 07:26:06.139: %LINK-5-CHANGED: Interface Dot11Radio1, changed state to reset *Jun 13 07:26:07.123: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to up *Jun 13 07:26:07.131: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to down *Jun 13 07:26:07.167: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to up *Jun 13 07:26:07.179: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to down *Jun 13 07:26:07.187: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to reset *Jun 13 07:26:08.167: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to up *Jun 13 07:26:08.179: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to down *Jun 13 07:26:08.207: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to up *Jun 13 07:26:09.207: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to up *Jun 13 07:26:26.211: %LWAPP-3-CLIENTERRORLOG: Not sending the Delete mobile payload for 7C:50:79:E1:F9:24. CAPWAP State is not RUN *Jun 13 07:26:55.743: %CAPWAP-3-DHCP_RENEW: Could not discover WLC. Either IP address is not assigned or assigned IP is wrong. Renewing DHCP IP. *Jun 13 07:26:58.815: %DHCP-6-ADDRESS_ASSIGN: Interface BVI1 assigned DHCP address 10.2.7.14, mask 255.255.255.0, hostname MST-Ullervad-Hus-A-Trapp-201-Ny *Jun 13 07:27:06.743: %CAPWAP-5-DHCP_OPTION_43: Controller address 10.0.0.147 obtained through DHCP *Jun 13 07:27:06.743: %CAPWAP-5-DHCP_OPTION_43: Controller address 10.0.0.148 obtained through DHCP *Jun 13 07:27:16.747: AP has SHA2 MIC certificate - Using SHA2 MIC certificate for DTLS. *Jun 13 07:27:17.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.0.0.148 peer_port: 5246 *Jun 13 07:27:17.223: %CAPWAP-5-DTLSREQSUCC: DTLS connection created sucessfully peer_ip: 10.0.0.148 peer_port: 5246 *Jun 13 07:27:17.223: %CAPWAP-5-SENDJOIN: sending Join Request to 10.0.0.148 *Jun 13 07:27:22.223: %CAPWAP-5-SENDJOIN: sending Join Request to 10.0.0.148 *Jun 13 07:27:22.307: %DOT11-5-EXPECTED_RADIO_RESET: Restarting Radio interface Dot11Radio0 due to the reason code 56 *Jun 13 07:27:22.311: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to down *Jun 13 07:27:22.319: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to reset *Jun 13 07:27:22.939: %DOT11-5-EXPECTED_RADIO_RESET: Restarting Radio interface Dot11Radio1 due to the reason code 56 *Jun 13 07:27:22.943: %DOT11-5-EXPECTED_RADIO_RESET: Restarting Radio interface Dot11Radio0 due to the reason code 10 *Jun 13 07:27:22.947: %CAPWAP-5-JOINEDCONTROLLER: AP has joined controller 5520New *Jun 13 07:27:22.955: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to up *Jun 13 07:27:23.119: %WIDS-6-ENABLED: IDS Signature is loaded and enabled%CRYPTO_PKI: Cert not yet valid or is expired - start date: 21:36:50 UTC Nov 5 2013 end date: 21:36:50 UTC May 20 2022 *Jun 13 07:27:23.127: %DOT11-3-NA_SENSOR_CERT_ERROR: Certificate installation error: Error in saving WSA certificate. *Jun 13 07:27:23.311: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to down *Jun 13 07:27:23.347: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to down *Jun 13 07:27:23.355: %LINK-5-CHANGED: Interface Dot11Radio1, changed state to reset *Jun 13 07:27:23.955: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to down *Jun 13 07:27:24.339: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to up *Jun 13 07:27:24.391: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to up *Jun 13 07:27:24.399: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to down *Jun 13 07:27:24.407: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to reset *Jun 13 07:27:25.391: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to up *Jun 13 07:27:25.399: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to down *Jun 13 07:27:25.427: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to up *Jun 13 07:27:26.427: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to up *Jun 13 07:27:48.499: %CLEANAIR-6-STATE: Slot 0 enabled *Jun 13 07:27:50.555: %CLEANAIR-6-STATE: Slot 1 enabled *Jun 13 07:57:56.419: Failed to read from HW TRNG *Jun 13 07:57:56.419: HW TRNG failed! *Jun 13 08:29:47.991: %WIDS-4-SIG_ALARM: Attack is detected on Sig:Standard Id:5 Channel:11 Source MAC:089d.f400.f492 *Jun 13 08:29:48.995: %WIDS-4-SIG_ALARM: Attack is detected on Sig:Standard Id:5 Channel:11 Source MAC:089d.f400.f492 *Jun 13 08:49:48.731: %WIDS-6-SIG_ALARM_OFF: Attack is cleared on Sig:Standard Id:5 Channel:11 *Jun 13 08:49:49.735: %WIDS-6-SIG_ALARM_OFF: Attack is cleared on Sig:Standard Id:5 Channel:11 *Jun 13 08:57:56.779: Failed to read from HW TRNG *Jun 13 08:57:56.779: HW TRNG failed! *Jun 13 09:28:39.995: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 10.0.0.148:5246 *Jun 13 09:28:39.999: %DOT11-5-EXPECTED_RADIO_RESET: Restarting Radio interface Dot11Radio0 due to the reason code 27 *Jun 13 09:28:39.999: %DOT11-5-EXPECTED_RADIO_RESET: Restarting Radio interface Dot11Radio1 due to the reason code 27 *Jun 13 09:28:39.999: %WIDS-6-DISABLED: IDS Signature is removed and disabled. *Jun 13 09:28:40.063: %CAPWAP-5-AP_EASYADMIN_INFO: AP Easy Admin information - EASY_ADMIN is not set, turn off easy admin service! *Jun 13 09:28:40.063: %CAPWAP-5-AP_EASYADMIN_INFO: AP Easy Admin information - Easy Admin is not enabled, turn it off! *Jun 13 09:28:40.087: %DOT11-5-EXPECTED_RADIO_RESET: Restarting Radio interface Dot11Radio0 due to the reason code 39 *Jun 13 09:28:40.087: %DOT11-5-EXPECTED_RADIO_RESET: Restarting Radio interface Dot11Radio1 due to the reason code 39 *Jun 13 09:28:40.099: %CLEANAIR-6-STATE: Slot 0 down *Jun 13 09:28:40.103: %CLEANAIR-6-STATE: Slot 1 down *Jun 13 09:28:40.103: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to administratively down *Jun 13 09:28:40.103: %LINK-5-CHANGED: Interface Dot11Radio1, changed state to administratively down *Jun 13 09:28:40.103: %DOT11-5-EXPECTED_RADIO_RESET: Restarting Radio interface Dot11Radio0 due to the reason code 10 *Jun 13 09:28:40.107: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to up *Jun 13 09:28:40.711: %DOT11-5-EXPECTED_RADIO_RESET: Restarting Radio interface Dot11Radio1 due to the reason code 10 *Jun 13 09:28:40.739: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to up *Jun 13 09:28:41.103: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to down *Jun 13 09:28:41.131: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to down *Jun 13 09:28:41.139: %LINK-5-CHANGED: Interface Dot11Radio1, changed state to reset *Jun 13 09:28:42.123: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to up *Jun 13 09:28:42.131: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to down *Jun 13 09:28:42.167: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to up *Jun 13 09:28:42.179: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to down *Jun 13 09:28:42.187: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to reset *Jun 13 09:28:43.167: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to up *Jun 13 09:28:43.179: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to down *Jun 13 09:28:43.207: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to up *Jun 13 09:28:44.207: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to up *Jun 13 09:29:01.215: %LWAPP-3-CLIENTERRORLOG: Not sending the Delete mobile payload for 7C:50:79:E1:F9:24. CAPWAP State is not RUN *Jun 13 09:29:30.743: %CAPWAP-3-DHCP_RENEW: Could not discover WLC. Either IP address is not assigned or assigned IP is wrong. Renewing DHCP IP. *Jun 13 09:29:33.815: %DHCP-6-ADDRESS_ASSIGN: Interface BVI1 assigned DHCP address 10.2.7.14, mask 255.255.255.0, hostname MST-Ullervad-Hus-A-Trapp-201-Ny *Jun 13 09:29:41.743: %CAPWAP-5-DHCP_OPTION_43: Controller address 10.0.0.147 obtained through DHCP *Jun 13 09:29:41.743: %CAPWAP-5-DHCP_OPTION_43: Controller address 10.0.0.148 obtained through DHCP *Jun 13 09:29:51.747: AP has SHA2 MIC certificate - Using SHA2 MIC certificate for DTLS. *Jun 13 09:29:52.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.0.0.148 peer_port: 5246 *Jun 13 09:29:52.219: %CAPWAP-5-DTLSREQSUCC: DTLS connection created sucessfully peer_ip: 10.0.0.148 peer_port: 5246 *Jun 13 09:29:52.223: %CAPWAP-5-SENDJOIN: sending Join Request to 10.0.0.148 *Jun 13 09:29:57.219: %CAPWAP-5-SENDJOIN: sending Join Request to 10.0.0.148 *Jun 13 09:29:57.763: %DOT11-5-EXPECTED_RADIO_RESET: Restarting Radio interface Dot11Radio0 due to the reason code 56 *Jun 13 09:29:57.771: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to down *Jun 13 09:29:57.779: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to reset *Jun 13 09:29:58.395: %DOT11-5-EXPECTED_RADIO_RESET: Restarting Radio interface Dot11Radio1 due to the reason code 56 *Jun 13 09:29:58.399: %DOT11-5-EXPECTED_RADIO_RESET: Restarting Radio interface Dot11Radio0 due to the reason code 10 *Jun 13 09:29:58.403: %CAPWAP-5-JOINEDCONTROLLER: AP has joined controller 5520New *Jun 13 09:29:58.411: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to up *Jun 13 09:29:58.571: %WIDS-6-ENABLED: IDS Signature is loaded and enabled%CRYPTO_PKI: Cert not yet valid or is expired - start date: 21:36:50 UTC Nov 5 2013 end date: 21:36:50 UTC May 20 2022 *Jun 13 09:29:58.579: %DOT11-3-NA_SENSOR_CERT_ERROR: Certificate installation error: Error in saving WSA certificate. *Jun 13 09:29:58.771: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to down *Jun 13 09:29:58.807: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to down *Jun 13 09:29:58.815: %LINK-5-CHANGED: Interface Dot11Radio1, changed state to reset *Jun 13 09:29:59.411: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to down *Jun 13 09:29:59.799: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to up *Jun 13 09:29:59.851: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to up *Jun 13 09:29:59.859: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to down *Jun 13 09:29:59.867: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to reset *Jun 13 09:30:00.851: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to up *Jun 13 09:30:00.859: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to down *Jun 13 09:30:00.887: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to up *Jun 13 09:30:01.887: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to up *Jun 13 09:30:21.483: %CLEANAIR-6-STATE: Slot 0 enabled *Jun 13 09:30:23.539: %CLEANAIR-6-STATE: Slot 1 enabled *Jun 13 09:57:56.571: Failed to read from HW TRNG *Jun 13 09:57:56.571: HW TRNG failed!

Re: 2702 AP logs

M02@rt37 — Tue, 13 Jun 2023 10:58:36 GMT

Hello @Moudar,

Thanks for the output.

It appears that there are a few potential issues that could be causing the AP to restart CAPWAP and disrupt clients.

--Radio Resets: The log entries show that the radio interfaces (Dot11Radio0 and Dot11Radio1) are being restarted due to reason codes 27, 39, and 10. These reason codes usually indicate radio interface errors or issues. Check for any interference sources, environmental factors, or configuration problems that could be causing the radio interfaces to reset.

--Easy Admin Service and IDS Signature: The log entries mention the Easy Admin service and IDS signature being disabled or not enabled. Review the AP's configuration related to these features to ensure they are correctly configured and not causing any conflicts or issues.

--DHCP IP Renewal and Controller Discovery: The AP attempts to renew its DHCP IP address and discovers the WLC addresses through DHCP options. Verify that the DHCP server is functioning properly and providing the correct WLC IP addresses. Ensure that the WLCs are reachable and properly configured.

--Certificate Issues: There are some certificate-related errors in the logs, such as certificate expiration and installation errors. Check the AP's certificate configuration and ensure that the certificates are valid and properly installed.

--CleanAir State Changes: The CleanAir feature on slot 0 and slot 1 undergoes state changes. This could indicate interference or environmental factors affecting the AP's performance. Investigate any potential sources of interference or environmental changes that might be impacting the AP's operation.

--WIDS (Wireless Intrusion Detection System) Alarms: The logs show WIDS alarms related to detected attacks on specific signatures and channels. These alarms could be a result of actual attacks or false positives. Analyze the WIDS alarms further to determine their significance and whether they could be triggering the AP restarts.

Not the least- --Hardware Random Number Generator ('HW TRNG') Failure: This failure could be a hardware issue with the AP, which might require further investigation.

Please, check all these points.

Re: 2702 AP logs

Mark Elsen — Tue, 13 Jun 2023 11:30:42 GMT

- FYI : https://www.cisco.com/c/en/us/support/docs/wireless/aironet-3700-series/117869-technote-ap-00.html
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvs56322

But in general and since both aireos and aireos based controllers are getting older consider using https://software.cisco.com/download/home/286284738/type/280926587/release/8.10.185.0 as adviced per
https://www.cisco.com/c/en/us/support/docs/wireless/wireless-lan-controller-software/200046-tac-recommended-aireos.html

Meaning : better is to use the last or latest release that they can run .

Re: 2702 AP logs

Moudar — Tue, 13 Jun 2023 12:11:01 GMT

I will restart the AP and hope for the best 🙂

Re: 2702 AP logs

Moudar — Tue, 13 Jun 2023 13:30:59 GMT

--Radio Resets:

If there is no indication of interference (as Prime shows) what environmental factors, or configuration problems could be causing the radio interfaces to reset?

Easy Admin Service

AP EasyAdmin ............................... Disable

So I don't know why an AP should have a problem from AP Easy Admin if it is disabled

-DHCP IP Renewal and Controller Discovery

We are using DNS as a discovery mechanism. What i know is: no other AP had a like problem

--Certificate Issues:

I think this is about 2702 APs that can't join the WLC after the 4th of December

--CleanAir State Changes:

If there is no indication of interference, what could environmental factors affecting the AP's performance be?

--WIDS (Wireless Intrusion Detection System) Alarms:

We have about 1000 APs and if i am not wrong all of them are sending logs about these "Attacks". How to decide what is an attack and what is a false positive?

--Hardware Random Number Generator ('HW TRNG') Failure:

I did restart the AP and will have an eye on it to see if these logs come again!

Re: 2702 AP logs

Rich R — Wed, 14 Jun 2023 01:45:08 GMT

And like Marce said - update the software to at least eliminate known bugs.

AP and radio crashes happen from time to time, and they certainly aren't going to get fixed in AireOS or 2702 now, so upgrading to the latest software and hoping for the best is about all you can do.