I am currently trouble shooting a issue with my 1140 access points. Recently they have been "freezing" or "locking up", not allowing client to connect or to pass traffic. As a fix I have to reboot individual APs, which is getting a little old. I am running a vWLC, with 53 access points connecting to it. Here is the software information:
Primary Software Version 7.5.102.0
Backup Software Version 0.0.0.0
Predownload Status None
Predownloaded Version None
Predownload Next Retry Time NA
Predownload Retry Count NA
Boot Version 12.4.23.3
IOS Version 15.2(4)JA1
Mini IOS Version 7.5.1.73
I have examined the log information from the vWLC web interface, Mangment>SNMP>Trap Logs, but have been unable to pinpoint anything to narrow down the problem.
Here are the logs from the biggest offender:
MS-AP10#sh run | in log
service timestamps log datetime msec
logging rate-limit console 9
aaa authentication login default local
logging trap emergencies
logging origin-id string AP:a493.4cf3.1088
logging facility kern
logging snmp-trap notifications
logging snmp-trap informational
logging snmp-trap debugging
logging host 255.255.255.255
MS-AP10#sh log
Syslog logging: enabled (0 messages dropped, 5 messages rate-limited, 0 flushes, 0 overruns, xml disabled, filtering disabled)
No Active Message Discriminator.
No Inactive Message Discriminator.
Console logging: level debugging, 517 messages logged, xml disabled,
filtering disabled
Monitor logging: level debugging, 0 messages logged, xml disabled,
filtering disabled
Buffer logging: level debugging, 517 messages logged, xml disabled,
filtering disabled
Exception Logging: size (4096 bytes)
Count and timestamp logging messages: disabled
Persistent logging: disabled
Trap logging: level emergencies, 0 message lines logged
Logging to 255.255.255.255 (udp port 514, audit disabled,
link down),
0 message lines logged,
0 message lines rate-limited,
0 message lines dropped-by-MD,
xml disabled, sequence number disabled
filtering disabled
Logging Source-Interface: VRF Name:
Log Buffer (1048576 bytes):
*Mar 1 00:00:11.059: FIPS IOS test Image Checksum successful
*Mar 1 00:00:11.063: FIPS IOS test Crypto RNG DEK Key Test successful
*Mar 1 00:00:11.063: FIPS IOS test SHA-1 successful
*Mar 1 00:00:11.063: FIPS IOS test HMAC-SHA1 successful
*Mar 1 00:00:11.063: FIPS IOS test AES CBC 128-bit Encrypt successful
*Mar 1 00:00:11.063: FIPS IOS test AES CBC 128-bit Decrypt successful
*Mar 1 00:00:11.063: FIPS IOS test IOS AES CMAC Encrypt successful
*Mar 1 00:00:11.063: FIPS IOS test IOS CCM Encrypt successful
*Mar 1 00:00:11.063: FIPS IOS test IOS CCM Decrypt successful
*Mar 1 00:00:11.166: FIPS IOS test RSA Signature Generation successful
*Mar 1 00:00:11.169: FIPS IOS test RSA Signature Verification successful
*Mar 1 00:00:11.169: %SOAP_FIPS-2-SELF_TEST_IOS_SUCCESS: IOS crypto FIPS self test passed
*Mar 1 00:00:11.171: *** CRASH_LOG = YES
*Mar 1 00:00:12.281: FIPS HW test SHA-1 successful
*Mar 1 00:00:12.281: FIPS HW test HMAC-SHA1 successful
*Mar 1 00:00:12.281: FIPS HW test AES CBC 128-bit Encrypt successful
*Mar 1 00:00:12.281: FIPS HW test AES CBC 128-bit Decrypt successful
*Mar 1 00:00:12.282: %SOAP_FIPS-2-SELF_TEST_HW_SUCCESS: HW crypto FIPS self test passed
*Mar 1 00:00:12.282: Security Core found.
*Mar 1 00:00:12.298: Registering HW DTLS
Base Ethernet MAC address: A4:93:4C:F3:10:88
*Mar 1 00:00:14.218: FIPS RADIO test AES 128-bit encrypt for TX on Dot11Radio 0 successful
*Mar 1 00:00:14.218: FIPS RADIO test AES 128-bit CCM encrypt on Dot11Radio 0 successful
*Mar 1 00:00:14.219: FIPS RADIO test AES 128-bit CCM decrypt on Dot11Radio 0 successful
*Mar 1 00:00:14.219: FIPS RADIO test AMAC AES 128-bit CMAC encrypt on Dot11Radio 0 successful
*Mar 1 00:00:14.219: %SOAP_FIPS-2-SELF_TEST_RAD_SUCCESS: RADIO crypto FIPS self test passed on interface Dot11Radio 0
*Mar 1 00:00:14.357: %LINK-6-UPDOWN: Interface GigabitEthernet0, changed state to up
*Mar 1 00:00:15.108: FIPS RADIO test AES 128-bit encrypt for TX on Dot11Radio 1 successful
*Mar 1 00:00:15.108: FIPS RADIO test AES 128-bit CCM encrypt on Dot11Radio 1 successful
*Mar 1 00:00:15.108: FIPS RADIO test AES 128-bit CCM decrypt on Dot11Radio 1 successful
*Mar 1 00:00:15.108: FIPS RADIO test AMAC AES 128-bit CMAC encrypt on Dot11Radio 1 successful
*Mar 1 00:00:15.108: %SOAP_FIPS-2-SELF_TEST_RAD_SUCCESS: RADIO crypto FIPS self test passed on interface Dot11Radio 1
*Mar 1 00:00:15.468: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0, changed state to up
*Mar 1 00:09:17.133: %SYS-5-RESTART: System restarted --
Cisco IOS Software, C1140 Software (C1140-K9W8-M), Version 15.2(4)JA1, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2013 by Cisco Systems, Inc.
Compiled Tue 30-Jul-13 23:25 by prod_rel_team
*Mar 1 00:09:17.133: %SNMP-5-COLDSTART: SNMP agent on host MS-AP10 is undergoing a cold start
*Mar 1 00:09:17.227: %PARSER-4-BADCFG: Unexpected end of configuration file.
*Mar 1 00:09:17.249: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to reset
*Mar 1 00:09:17.250: %LINK-5-CHANGED: Interface Dot11Radio1, changed state to reset
*Mar 1 00:09:18.189: %LINEPROTO-5-UPDOWN: Line protocol on Interface BVI1, changed state to up
*Mar 1 00:09:18.280: Starting Ethernet promiscuous mode
*Mar 1 00:09:18.432: %SSH-5-ENABLED: SSH 2.0 has been enabledlwapp_crypto_init: MIC Present and Parsed Successfully
*Mar 1 00:09:45.635: Logging LWAPP message to 255.255.255.255.
*Mar 1 00:09:50.391: %LWAPP-3-LWAPP_INTERFACE_GOT_IP_ADDRESS: Interface BVI1 obtained IP from DHCP...
*Mar 1 00:09:50.511: %DHCP-6-ADDRESS_ASSIGN: Interface BVI1 assigned DHCP address 172.17.1.128, mask 255.255.255.0, hostname MS-AP10
*Mar 1 00:09:50.512: %LWAPP-3-LWAPP_INTERFACE_GOT_IP_ADDRESS: Interface BVI1 obtained IP from DHCP...
*Mar 1 00:10:01.395: %CAPWAP-5-DHCP_OPTION_43: Controller address 172.17.1.4 obtained through DHCP
*Mar 1 00:10:01.395: %CAPWAP-3-ERRORLOG: Did not get log server settings from DHCP.
*Mar 1 00:10:10.395: %CAPWAP-3-ERRORLOG: Could Not resolve CISCO-CAPWAP-CONTROLLER
*Mar 1 00:10:20.397: %CAPWAP-3-ERRORLOG: Go join a capwap controller
*Oct 25 16:04:26.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 172.17.1.4 peer_port: 5246
*Oct 25 16:04:26.521: %CAPWAP-5-DTLSREQSUCC: DTLS connection created sucessfully peer_ip: 172.17.1.4 peer_port: 5246
*Oct 25 16:04:26.522: %CAPWAP-5-SENDJOIN: sending Join Request to 172.17.1.4
*Oct 25 16:04:26.524: %CAPWAP-3-ERRORLOG: Invalid event 10 & state 5 combination.
*Oct 25 16:04:26.524: %CAPWAP-3-ERRORLOG: CAPWAP SM handler: Failed to process message type 10 state 5.
*Oct 25 16:04:26.524: %CAPWAP-3-ERRORLOG: Failed to handle capwap control message from controller
*Oct 25 16:04:26.524: %CAPWAP-3-ERRORLOG: Failed to process encrypted capwap packet from 172.17.1.4
*Oct 25 16:04:26.602: Starting Ethernet promiscuous mode
*Oct 25 16:04:29.768: ac_first_hop_mac - IP:172.17.1.4 Hop IP:172.17.1.4 IDB:BVI1
*Oct 25 16:04:29.913: %LWAPP-4-CLIENTEVENTLOG: OfficeExtend Localssid saved in AP flash
*Oct 25 16:04:30.942: %CAPWAP-5-JOINEDCONTROLLER: AP has joined controller Town_vWLC
*Oct 25 16:04:30.993: %LWAPP-4-CLIENTEVENTLOG: No Flex ACL map configuration file to load. Connect to controller to get configuration file
*Oct 25 16:04:30.994: %LWAPP-4-CLIENTEVENTLOG: No Flex ACL map configuration file to load. Connect to controller to get configuration file
*Oct 25 16:04:30.994: %LWAPP-4-CLIENTEVENTLOG: No LS Flex ACL map configuration file to load. Connect to controller to get configuration file
*Oct 25 16:04:30.995: %LWAPP-4-CLIENTEVENTLOG: No Central Dhcp map configuration file to load. Connect to controller to get configuration fileWLAN id 1, SSID building_wireless, L2ACL , L2ACL AP
WLAN id 2, SSID building_guest, L2ACL , L2ACL AP
*Oct 25 16:04:35.708: %WIDS-6-ENABLED: IDS Signature is loaded and enabled
*Oct 25 16:04:38.980: %CDP_PD-4-POWER_OK: Full power - NEGOTIATED inline power source
*Oct 25 16:04:40.005: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to up
*Oct 25 16:04:41.092: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to up
*Oct 25 16:04:41.117: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to up
*Oct 25 16:04:42.117: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to up
*Oct 25 16:31:25.654: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to down
*Oct 25 16:31:25.656: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to reset
*Oct 25 16:31:26.654: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to down
------CLIPPED---------
*Oct 28 13:41:23.131: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to reset
*Oct 28 13:41:24.128: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to down
*Oct 28 13:41:24.152: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to up
*Oct 28 13:41:25.152: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to up
MS-AP10#
To me this is pointing at a power problem, power is fluctuating causing the interface to bounce up and down then eventually locking up the device. This building has had power issues in the past, (I had to replace all of their PoE switches after an outage) but I cannot find any concrete evidence to support my theory.
