One of our Airlap 1142N AP is joining the WLC 5500

[manly009]

Hi Guys, 

Suddenly I have been reported one of our AP is not connecting to WLC. Looks like the AP is getting an IP, but it cannot associate with WLC. Previously I have config ap cert-expiry-ignore {mic|ssc} enable, it fixed all old APs joining issues. But this one AP I am not sure what to do......was thinking to tick "Accept Self Signed Certificate (SSC)" under Security AAA - AP policies, but I am not sure if it will cause other connection issues? 

The log I got from AP: 

*Mar 1 00:00:09.122: %LINK-3-UPDOWN: Interface GigabitEthernet0, changed state to up
*Mar 1 00:00:09.168: %SYS-5-RESTART: System restarted --
Cisco IOS Software, C1140 Software (C1140-RCVK9W8-M), Version 12.4(18a)JA, RELEASE SOFTWARE (fc4)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2008 by Cisco Systems, Inc.
Compiled Fri 21-Nov-08 01:28 by prod_rel_team
*Mar 1 00:00:09.196: %CAPWAP-5-CHANGED: CAPWAP changed state to DISCOVERY
*Mar 1 00:00:10.122: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0, changed state to up
*Mar 1 00:00:18.265: %DHCP-6-ADDRESS_ASSIGN: Interface GigabitEthernet0 assigned DHCP address 10.8.44.57, mask 255.255.254.0, hostname Gym

*Mar 1 00:00:28.101: %CAPWAP-3-ERRORLOG: Not sending discovery request AP does not have an Ip !!
*Mar 1 00:00:28.223: Logging LWAPP message to 255.255.255.255.

*Mar 1 00:00:32.233: %CDP_PD-2-POWER_LOW: All radios disabled - NEGOTIATED WS-C3750X-24P (2894.0f34.ed2e)
Translating "CISCO-LWAPP-CONTROLLER.school.com"...domain server (10.8.2.42) [OK]

*Mar 1 00:00:38.173: %CAPWAP-3-ERRORLOG: Did not get log server settings from DHCP.
*Jan 22 04:01:52.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.8.46.2 peer_port: 5246
*Jan 22 04:01:52.821: %DTLS-5-ALERT: Received FATAL : Certificate unknown alert from 10.8.46.2
*Jan 22 04:01:52.821: %CAPWAP-3-ERRORLOG: Bad certificate alert received from peer.
*Jan 22 04:01:52.821: %DTLS-5-PEER_DISCONNECT: Peer 10.8.46.2 has closed connection.
*Jan 22 04:01:52.822: %CAPWAP-3-ERRORLOG: Invalid event 38 & state 3 combination.

Thanks a lot for help. 

Mang

[manly009]

Also I can see it was trying to join WLC:

Reason For Last Unsuccessful Attempt: Maximum number of AP supported has already joined.

We definitely got 7 more available licenses.

The log from AP after the Recovery Image Applied:

E1: VC0 is active
PCIEx: initialization done
flashfs[0]: 46 files, 9 directories
flashfs[0]: 0 orphaned files, 0 orphaned directories
flashfs[0]: Total bytes: 32385024
flashfs[0]: Bytes used: 14600192
flashfs[0]: Bytes available: 17784832
flashfs[0]: flashfs fsck took 18 seconds.
Reading cookie from system serial eeprom...Done
Base Ethernet MAC address: 00:22:bd:1a:39:50
Ethernet speed is 1000 Mb - FULL duplex
Loading "flash:/c1140-k9w8-mx.153-3.JA10/c1140-k9w8-mx.153-3.JA10"...############

File "flash:/c1140-k9w8-mx.153-3.JA10/c1140-k9w8-mx.153-3.JA10" uncompressed and installed, entry point: 0x4000
executing...
enet halted

Secondary Bootloader - Starting system.
Xmodem file system is available.
flashfs[0]: 46 files, 9 directories
flashfs[0]: 0 orphaned files, 0 orphaned directories
flashfs[0]: Total bytes: 32385024
flashfs[0]: Bytes used: 14600192
flashfs[0]: Bytes available: 17784832
flashfs[0]: flashfs fsck took 6 seconds.
Reading cookie from system serial eeprom...Done
Base Ethernet MAC address: 00:22:bd:1a:39:50

Secondary bootloader Ethernet not enabled, skip ether_init
Boot CMD: 'boot flash:/c1140-k9w8-mx.153-3.JA10/c1140-k9w8-xx.153-3.JA10;flash:/c1140-rcvk9w8-mx/c1140-rcvk9w8-xx'
Loading "flash:/c1140-k9w8-mx.153-3.JA10/c1140-k9w8-xx.153-3.JA10"...###################################
File "flash:/c1140-k9w8-mx.153-3.JA10/c1140-k9w8-xx.153-3.JA10" uncompressed and installed, entry point: 0x4000
executing...

Restricted Rights Legend

Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
(c) of the Commercial Computer Software - Restricted
Rights clause at FAR sec. 52.227-19 and subparagraph
(c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.

cisco Systems, Inc.
170 West Tasman Drive
San Jose, California 95134-1706



Cisco IOS Software, C1140 Software (C1140-K9W8-M), Version 15.3(3)JA10, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2016 by Cisco Systems, Inc.
Compiled Tue 23-Aug-16 03:15 by prod_rel_team

Initializing flashfs...

flashfs[2]: 46 files, 9 directories
flashfs[2]: 0 orphaned files, 0 orphaned directories
flashfs[2]: Total bytes: 32126976
flashfs[2]: Bytes used: 14600192
flashfs[2]: Bytes available: 17526784
flashfs[2]: flashfs fsck took 14 seconds.
flashfs[2]: Initialization complete.
flashfs[4]: 0 files, 1 directories
flashfs[4]: 0 orphaned files, 0 orphaned directories
flashfs[4]: Total bytes: 11999232
flashfs[4]: Bytes used: 1024
flashfs[4]: Bytes available: 11998208
flashfs[4]: flashfs fsck took 1 seconds.
flashfs[4]: Initialization complete....done Initializing flashfs.

Ethernet speed is 1000 Mb - FULL duplex

Radio0 present 8363 8000 90020000 0 90030000 B
Rate table has 300 entries (16 legacy/64 11n/220 11ac)

POWER TABLE FILENAME = flash:/c1140-k9w8-mx.153-3.JA10/T2.bin

Radio1 present 8363 8000 98020000 0 98030000 0
POWER TABLE FILENAME = flash:/c1140-k9w8-mx.153-3.JA10/T5.bin

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco AIR-LAP1142N-N-K9 (PowerPC405ex) processor (revision A0) with 98294K/32768K bytes of memory.
Processor board ID FCW1336S029
PowerPC405ex CPU at 586Mhz, revision number 0x147E
Last reset from power-on
LWAPP image version 8.0.140.0
1 Gigabit Ethernet interface
2 802.11 Radios

32K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address: 00:22:BD:1A:39:50
Part Number : 73-11451-06
PCA Assembly Number : 800-30554-03
PCA Revision Number : A0
PCB Serial Number : FOC13331VKX
Top Assembly Part Number : 800-31273-01
Top Assembly Serial Number : FCW1336S029
Top Revision Number : A0
Product/Model Number : AIR-LAP1142N-N-K9
% Please define a domain-name first.


Press RETURN to get started!


*Mar 1 00:00:16.650: %SOAP_FIPS-2-SELF_TEST_IOS_SUCCESS: IOS crypto FIPS self test passed (15)
*Mar 1 00:00:16.652: *** CRASH_LOG = YES

*Mar 1 00:00:17.762: %SOAP_FIPS-2-SELF_TEST_HW_SUCCESS: HW crypto FIPS self test passed (1-6)
*Mar 1 00:00:17.763: Security Core found.

*Mar 1 00:00:17.775: Registering HW DTLS
Base Ethernet MAC address: 00:22:BD:1A:39:50

*Mar 1 00:00:19.708: %SOAP_FIPS-2-SELF_TEST_RAD_SUCCESS: RADIO crypto FIPS self test passed on interface Dot11Radio 0 (4)
*Mar 1 00:00:19.881: loading Power Tables from flash:/c1140-k9w8-mx.153-3.JA10/T2.bin. Class = A
*Mar 1 00:00:19.882: record size of 2ss: 404 read_ptr: 2576028

*Mar 1 00:00:19.957: %LINK-6-UPDOWN: Interface GigabitEthernet0, changed state to up
*Mar 1 00:00:20.754: %SOAP_FIPS-2-SELF_TEST_RAD_SUCCESS: RADIO crypto FIPS self test passed on interface Dot11Radio 1 (4)
*Mar 1 00:00:20.840: loading Power Tables from flash:/c1140-k9w8-mx.153-3.JA10/T5.bin. Class = N
*Mar 1 00:00:20.840: record size of 2ss: 404 read_ptr: 2576028

*Mar 1 00:00:21.002: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0, changed state to upcapwap_read_version_info: Info file flash:/c1140-k9w8-mx.153-3.JA8/info not find
*Mar 1 00:11:19.167: %SYS-5-RESTART: System restarted --
Cisco IOS Software, C1140 Software (C1140-K9W8-M), Version 15.3(3)JA10, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2016 by Cisco Systems, Inc.
Compiled Tue 23-Aug-16 03:15 by prod_rel_team
*Mar 1 00:11:19.167: %SNMP-5-COLDSTART: SNMP agent on host Gym is undergoing a cold start
*Mar 1 00:11:19.851: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to up
*Mar 1 00:11:20.245: %LINEPROTO-5-UPDOWN: Line protocol on Interface BVI1, changed state to up
*Mar 1 00:11:20.554: %SSH-5-ENABLED: SSH 2.0 has been enabled
*Mar 1 00:11:20.554: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to uplwapp_crypto_init: MIC Present and Parsed Successfully

*Mar 1 00:11:27.477: %DHCP-6-ADDRESS_ASSIGN: Interface BVI1 assigned DHCP address 10.8.44.53, mask 255.255.254.0, hostname Gym

*Mar 1 00:11:36.234: Currently running a Release Image
validate_sha2_block: Failed to get certificate chain
*Mar 1 00:11:36.267: Using SHA-1 signed certificate for image signing validation.%Default route without gateway, if not a point-to-point interface, may impact performance
*Mar 1 00:11:42.927: AP image integrity check PASSED

*Mar 1 00:11:43.045: validate_sha2_block:No SHA2 Block present on this AP.

*Mar 1 00:11:43.080: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to reset
*Mar 1 00:11:43.080: %LINK-5-CHANGED: Interface Dot11Radio1, changed state to reset
*Mar 1 00:11:53.111: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 255.255.255.255 port 0 CLI Request Triggered

*Jan 30 01:10:54.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.8.46.2 peer_port: 5246
*Jan 30 01:10:54.545: %DTLS-5-ALERT: Received FATAL : Certificate unknown alert from 10.8.46.2
*Jan 30 01:10:54.545: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 10.8.46.2:5246
*Jan 30 01:11:05.535: %CDP_PD-4-POWER_OK: Full power - NEGOTIATED inline power source
*Jan 30 01:11:06.558: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to up
*Jan 30 01:11:07.558: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to up
*Jan 30 01:11:07.584: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to up
*Jan 30 01:11:08.584: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to up

Any other thing I can try?

[Leo Laohoo]

Look at the firmware the AP is booting up on.
The AP is booting the firmware for 8.0.140.0.
Post the complete output to the WLC command "show ap join stats detailed <AP NAME>".

[manly009]

Does this help?

Base MAC Address
00:22:bd:1a:39:50
AP Name
Gym
Ethernet MAC Address
00:22:bd:1a:39:50
IP Address(Ipv4/Ipv6)
10.8.44.53
Status
Not joined
Last AP Join

Timestamp Message
Jan 29 14:53:10.079
Received Discovery request and sent response
Jan 29 14:55:30.628
Received Join request and sent response
Discovery Phase Statistics

Requests Received
78
Responses Sent
74
Unsuccessful Request Processed
2
Reason For Last Unsuccessful Attempt
Maximum number of AP supported has already joined
Last Successful Attempt Time
Jan 29 14:53:10.079
Last Unsuccessful Attempt Time
Jan 23 12:04:22.789
Join Phase Statistics

Requests Received
1
Responses Sent
1
Unsuccessful Request Processed
0
Reason For Last Unsuccessful Attempt
-
Last Successful Attempt Time
Jan 29 14:55:30.628
Last Unsuccessful Attempt Time
-
Configuration Phase Statistics

Requests Received
0
Responses Sent
0
Unsuccessful Request Processed
0
Reason For Last Unsuccessful Attempt
-
Last Successful Attempt Time
-
Last Unsuccessful Attempt Time
-
Last Error Summary

Last AP Message Decryption Failure
-
Last AP Connection Failure
Image data request received from an unsupported AP
Last AP Disconnect Reason
-
Last Error Occurred
AP got or has been disconnected
Last Error Occurred Reason
Image data request received from an unsupported AP
Last Join Error Timestamp
Jan 29 14:58:23.363

[Leo Laohoo]

---

@manly009 wrote:
Maximum number of AP supported has already joined

---

Can you post the complete output to the following commands: 

1.  WLC:  sh sysinfo; 

2.  WLC:  sh ap summary (first page only)

[manly009]

AP:

Gym>en
Password:
Gym#en
Gym#show ver
Cisco IOS Software, C1140 Software (C1140-K9W8-M), Version 15.3(3)JA10, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2016 by Cisco Systems, Inc.
Compiled Tue 23-Aug-16 03:15 by prod_rel_team

ROM: Bootstrap program is C1140 boot loader
BOOTLDR: C1140 Boot Loader (C1140-BOOT-M) Version 12.4(18a)JA, RELEASE SOFTWARE (fc4)

Gym uptime is 1 hour, 39 minutes
System returned to ROM by power-on
System image file is "flash:/c1140-k9w8-mx.153-3.JA10/c1140-k9w8-xx.153-3.JA10"
Last reload reason:



This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco AIR-LAP1142N-N-K9 (PowerPC405ex) processor (revision A0) with 98294K/32768K bytes of memory.
Processor board ID FCW1336S029
PowerPC405ex CPU at 586Mhz, revision number 0x147E
Last reset from power-on
LWAPP image version 8.0.140.0
1 Gigabit Ethernet interface
2 802.11 Radios

32K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address: 00:22:BD:1A:39:50
Part Number : 73-11451-06
PCA Assembly Number : 800-30554-03
PCA Revision Number : A0
PCB Serial Number : FOC13331VKX
Top Assembly Part Number : 800-31273-01
Top Assembly Serial Number : FCW1336S029
Top Revision Number : A0
Product/Model Number : AIR-LAP1142N-N-K9



Configuration register is 0xF

WLC:

Cannot see this AP joined or summary as it cannot join WLC.

[Rich R]

And as Leo requested:
1. WLC: sh sysinfo;
2. WLC: sh ap summary (first page only)

[Leo Laohoo]

WTF ...
Exactly where are the APs installed?
You've got two APs with different Regulatory Domain, -A and -N.
Please post the complete output to the WLC command "sh time".

[manly009]

Sorry about the Previous confusion. 

The issue has been fixed after I update Recovery image on faulty AP and

Run:

AP config-certficate-expiry enabled on Controller.

Thanks a lot,

[manly009]

Can we please delete this post since not many useful info on it. 

Thanks

Manly009

[patoberli]

Please just mark your solution or one of the posts with the correct as the answer, which will help people finding the right info faster. No need to delete stuff.

[Leo Laohoo]

Post the complete output to the AP command "sh version".

[patoberli]

I guess you are hitting this bug:
https://www.cisco.com/c/en/us/support/docs/field-notices/639/fn63942.html
Did you enable the workaround command on the WLC:
config ap cert-expiry-ignore mic enable
config ap cert-expiry-ignore ssc enable

Not sure if that will solve your issue, but give it a go.

[Leo Laohoo]

He's not hitting the certificate bug. Look at the firmware the AP is booting on: 8.0.140.0.
It is something else and I feel that certain information is being held back.

[Rich R]

>He's not hitting the certificate bug. Look at the firmware the AP is booting on: 8.0.140.0.
As long as the fix config was applied *with* the fixed code after the APs had managed to join.
Setting the WLC clock back to let them join would prove that.
Agreed it feels like there's an incomplete picture here.