AIR 3502i ap's will not join Controller

[HVO-cisco]

Hi, after a power failure, a lot of our 3502i would not reconnect to the controller.
Has taken factory reset, but receives the following error messages:

*Mar 1 00:14:00.475: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 255.255.255.255 port 0 CLI Request Triggered
*Mar 1 00:14:01.475: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 255.255.255.255 port 514 started - CLI initiated%No matching route to delete
Translating "CISCO-CAPWAP-CONTROLLER.xxxxx.no"...domain server (x.x.x.x)

*Mar 1 00:14:11.503: %CAPWAP-5-DHCP_OPTION_43: Controller address x.x.x.x obtained through DHCP
*Mar 1 00:14:11.503: %CAPWAP-5-DHCP_OPTION_43: Controller address x.x.x.x obtained through DHCP
*Mar 1 00:14:12.365: %DOT11-5-EXPECTED_RADIO_RESET: Restarting Radio interface Dot11Radio0 due to the reason code 10
*Mar 1 00:14:12.365: %DOT11-5-EXPECTED_RADIO_RESET: Restarting Radio interface Dot11Radio1 due to the reason code 10
*Mar 1 00:14:12.365: %CDP_PD-4-POWER_OK: Full power - NEGOTIATED inline power source
*Mar 1 00:14:13.457: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to up
*Mar 1 00:14:14.457: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to up
*Mar 1 00:14:14.552: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to up
*Mar 1 00:14:15.552: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to up
*Jan 3 14:41:13.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: x.x.x.x peer_port: 5246
*Jan 3 14:41:13.207: %DTLS-5-ALERT: Received FATAL : Certificate unknown alert from x.x.x.x
*Jan 3 14:41:13.207: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to x.x.x.x:5246

Then it seems to go into a loop.
Any tips that can help ?
Controller is 8510 with software 8.5.10.0

Johann Folkestad

[marce1000]

                                           >...due to the reason code 10

 - Ref  : https://www.cisco.com/c/en/us/support/docs/wireless/aironet-3700-series/117869-technote-ap-00.html

  >...

10

RADIO_FC_TX_INPROG

The driver attempts to free a packet that the radio still has in progress.

     Though I have my doubts if this is  all very useful in this case -> you may want to have a cold start of a trouble some access point (again) , and check if that can help for the particular ap. If not I would do a cold start of everything , beginning with the controller (first).

 M.

[Sandeep Choudhary]

Please paste the output of these commands:

from WLC:

show time

show sysinfo

from AP:

sh version

Make sure that WLC has correct time / NTP server configured.

Regards

Dont forget to rate helpful posts

[HVO-cisco]

Hi, I want to emphasize that only the 3502i models are problematic.
We have many other models such as 3700, 2802i etc. that work well against the same controller.
Regards
Johann Folkestad

(Cisco Controller) >show time
Here are the output:

Time............................................. Tue Jan 4 05:59:23 2022
Timezone delta................................... 0:0
Timezone location................................ (GMT +1:00) Amsterdam, Berlin, Rome, Vienna
NTP Servers
NTP Version.................................. 3
NTP Polling Interval......................... 4000
Index NTP Key Index NTP Server Status NTP Msg Auth Status
------- ------------------------------------------------------------------- --
1 0 1xx.3x.1xx.1 In Sync

(Cisco Controller) >show sysinfo

Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 8.5.151.0
RTOS Version..................................... 8.5.151.0
Bootloader Version............................... 8.1.133.7
Emergency Image Version.......................... 8.1.133.7

OUI File Last Update Time........................ Sun Sep 07 10:44:07 IST 2014
Build Type....................................... DATA + WPS
System Name...................................... wlan-controller-1
System Location.................................. Datarom BK
System Contact................................... xxxxxxxxxxxx
System ObjectID.................................. 1.3.6.1.4.1.9.1.1615
Redundancy Mode.................................. SSO
IP Address....................................... xxx.xx.xxx.x
IPv6 Address..................................... ::
System Up Time................................... 357 days 12 hrs 49 mins 52 secs
System Timezone Location......................... (GMT +1:00) Amsterdam, Berlin, Rome, Vienna
System Stats Realtime Interval................... 5
System Stats Normal Interval..................... 180
Configured Country............................... NO - Norway
Operating Environment............................ Commercial (10 to 35 C)
Internal Temp Alarm Limits....................... 10 to 38 C
Internal Temperature............................. +23 C
Fan Status....................................... OK

RAID Volume Status
Drive 0.......................................... Good
Drive 1.......................................... Good

State of 802.11b Network......................... Enabled
State of 802.11a Network......................... Enabled
Number of WLANs.................................. 6
Number of Active Clients......................... 183

OUI Classification Failure Count................. 1973098

Memory Current Usage............................. 35
Memory Average Usage............................. 35
CPU Current Usage................................ 0
CPU Average Usage................................ 0

Flash Type....................................... Compact Flash Card
Flash Size....................................... 1073741824

Burned-in MAC Address............................ 2C:33:11:D0:91:80
Power Supply 1................................... Present, OK
Power Supply 2................................... Present, OK
Maximum number of APs supported.................. 6000
System Nas-Id.................................... wlan-controller-1
WLC MIC Certificate Types........................ SHA1/SHA2
Licensing Type................................... RTU

AP70ca.9b25.4382>show version
Cisco IOS Software, C3500 Software (AP3G1-K9W8-M), Version 15.3(3)JF10, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2019 by Cisco Systems, Inc.
Compiled Thu 13-Jun-19 16:00 by prod_rel_team

ROM: Bootstrap program is C3500 boot loader
BOOTLDR: C3500 Boot Loader (AP3G1-BOOT-M), Version 15.3 [vtoky-imagetype 106]

AP70ca.9b25.4382 uptime is 14 hours, 30 minutes
System returned to ROM by power-on
System image file is "flash:/ap3g1-k9w8-mx.153-3.JF10/ap3g1-k9w8-xx.153-3.JF10"
Last reload reason:

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco AIR-CAP3502I-E-K9 (PowerPC460exr) processor (revision A0) with 98294K/32768K bytes of memory.
Processor board ID FCZ1545W5BF
PowerPC460exr CPU at 666Mhz, revision number 0x18A8
Last reset from power-on
LWAPP image version 8.5.151.0
1 Gigabit Ethernet interface
2 802.11 Radios

32K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address: 70:CA:9B:25:43:82
Part Number : 73-12175-05
PCB Serial Number : FOC153722J7
Top Assembly Part Number : 800-32891-01
Top Assembly Serial Number : FCZ1545W5BF
Top Revision Number : A0
Product/Model Number : AIR-CAP3502I-E-K9

[marce1000]

                 - What's in the controller-logs when these AP's try to join ?

 M.

[Leo Laohoo]

---

@HVO-cisco wrote:

Top Assembly Serial Number : FCZ1545W5BF

---

AP is more than 10 years old. 

---

@HVO-cisco wrote:

Product Version.................................. 8.5.151.0

---

The WLC firmware is affected by this bug.
FN 63942 - Wireless Lightweight Access Points and WLAN Controllers Fail to Create CAPWAP Connections Due to Certificate Expiration

[HVO-cisco]

Hello, I think this is the issue. Thank you.

Regards
Johann Folkestad

[Leo Laohoo]

Post the complete output to the AP command "sh version".

[HVO-cisco]

Hi Leo.

Here is the output:

AP70ca.9b25.4382>show version
Cisco IOS Software, C3500 Software (AP3G1-K9W8-M), Version 15.3(3)JF10, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2019 by Cisco Systems, Inc.
Compiled Thu 13-Jun-19 16:00 by prod_rel_team

ROM: Bootstrap program is C3500 boot loader
BOOTLDR: C3500 Boot Loader (AP3G1-BOOT-M), Version 15.3 [vtoky-imagetype 106]

AP70ca.9b25.4382 uptime is 14 hours, 30 minutes
System returned to ROM by power-on
System image file is "flash:/ap3g1-k9w8-mx.153-3.JF10/ap3g1-k9w8-xx.153-3.JF10"
Last reload reason:

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco AIR-CAP3502I-E-K9 (PowerPC460exr) processor (revision A0) with 98294K/32768K bytes of memory.
Processor board ID FCZ1545W5BF
PowerPC460exr CPU at 666Mhz, revision number 0x18A8
Last reset from power-on
LWAPP image version 8.5.151.0
1 Gigabit Ethernet interface
2 802.11 Radios

32K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address: 70:CA:9B:25:43:82
Part Number : 73-12175-05
PCB Serial Number : FOC153722J7
Top Assembly Part Number : 800-32891-01
Top Assembly Serial Number : FCZ1545W5BF
Top Revision Number : A0
Product/Model Number : AIR-CAP3502I-E-K9

[marce1000]

                         - For completeness also check : 

(Cisco Controller)> show certificate all

 M.