Access Points Lock up

Joe Liberi — Sun, 04 Jul 2021 08:10:13 GMT

I am currently trouble shooting a issue with my 1140 access points. Recently they have been "freezing" or "locking up", not allowing client to connect or to pass traffic. As a fix I have to reboot individual APs, which is getting a little old. I am running a vWLC, with 53 access points connecting to it. Here is the software information:

Primary Software Version 7.5.102.0

Backup Software Version 0.0.0.0

Predownload Status None

Predownloaded Version None

Predownload Next Retry Time NA

Predownload Retry Count NA

Boot Version 12.4.23.3

IOS Version 15.2(4)JA1

Mini IOS Version 7.5.1.73

I have examined the log information from the vWLC web interface, Mangment>SNMP>Trap Logs, but have been unable to pinpoint anything to narrow down the problem.

Here are the logs from the biggest offender:

MS-AP10#sh run | in log

service timestamps log datetime msec

logging rate-limit console 9

aaa authentication login default local

logging trap emergencies

logging origin-id string AP:a493.4cf3.1088

logging facility kern

logging snmp-trap notifications

logging snmp-trap informational

logging snmp-trap debugging

logging host 255.255.255.255

MS-AP10#sh log

Syslog logging: enabled (0 messages dropped, 5 messages rate-limited, 0 flushes, 0 overruns, xml disabled, filtering disabled)

No Active Message Discriminator.

No Inactive Message Discriminator.

Console logging: level debugging, 517 messages logged, xml disabled,

filtering disabled

Monitor logging: level debugging, 0 messages logged, xml disabled,

filtering disabled

Buffer logging: level debugging, 517 messages logged, xml disabled,

filtering disabled

Exception Logging: size (4096 bytes)

Count and timestamp logging messages: disabled

Persistent logging: disabled

Trap logging: level emergencies, 0 message lines logged

Logging to 255.255.255.255 (udp port 514, audit disabled,

link down),

0 message lines logged,

0 message lines rate-limited,

0 message lines dropped-by-MD,

xml disabled, sequence number disabled

filtering disabled

Logging Source-Interface: VRF Name:

Log Buffer (1048576 bytes):

*Mar 1 00:00:11.059: FIPS IOS test Image Checksum successful

*Mar 1 00:00:11.063: FIPS IOS test Crypto RNG DEK Key Test successful

*Mar 1 00:00:11.063: FIPS IOS test SHA-1 successful

*Mar 1 00:00:11.063: FIPS IOS test HMAC-SHA1 successful

*Mar 1 00:00:11.063: FIPS IOS test AES CBC 128-bit Encrypt successful

*Mar 1 00:00:11.063: FIPS IOS test AES CBC 128-bit Decrypt successful

*Mar 1 00:00:11.063: FIPS IOS test IOS AES CMAC Encrypt successful

*Mar 1 00:00:11.063: FIPS IOS test IOS CCM Encrypt successful

*Mar 1 00:00:11.063: FIPS IOS test IOS CCM Decrypt successful

*Mar 1 00:00:11.166: FIPS IOS test RSA Signature Generation successful

*Mar 1 00:00:11.169: FIPS IOS test RSA Signature Verification successful

*Mar 1 00:00:11.169: %SOAP_FIPS-2-SELF_TEST_IOS_SUCCESS: IOS crypto FIPS self test passed

*Mar 1 00:00:11.171: *** CRASH_LOG = YES

*Mar 1 00:00:12.281: FIPS HW test SHA-1 successful

*Mar 1 00:00:12.281: FIPS HW test HMAC-SHA1 successful

*Mar 1 00:00:12.281: FIPS HW test AES CBC 128-bit Encrypt successful

*Mar 1 00:00:12.281: FIPS HW test AES CBC 128-bit Decrypt successful

*Mar 1 00:00:12.282: %SOAP_FIPS-2-SELF_TEST_HW_SUCCESS: HW crypto FIPS self test passed

*Mar 1 00:00:12.282: Security Core found.

*Mar 1 00:00:12.298: Registering HW DTLS

Base Ethernet MAC address: A4:93:4C:F3:10:88

*Mar 1 00:00:14.218: FIPS RADIO test AES 128-bit encrypt for TX on Dot11Radio 0 successful

*Mar 1 00:00:14.218: FIPS RADIO test AES 128-bit CCM encrypt on Dot11Radio 0 successful

*Mar 1 00:00:14.219: FIPS RADIO test AES 128-bit CCM decrypt on Dot11Radio 0 successful

*Mar 1 00:00:14.219: FIPS RADIO test AMAC AES 128-bit CMAC encrypt on Dot11Radio 0 successful

*Mar 1 00:00:14.219: %SOAP_FIPS-2-SELF_TEST_RAD_SUCCESS: RADIO crypto FIPS self test passed on interface Dot11Radio 0

*Mar 1 00:00:14.357: %LINK-6-UPDOWN: Interface GigabitEthernet0, changed state to up

*Mar 1 00:00:15.108: FIPS RADIO test AES 128-bit encrypt for TX on Dot11Radio 1 successful

*Mar 1 00:00:15.108: FIPS RADIO test AES 128-bit CCM encrypt on Dot11Radio 1 successful

*Mar 1 00:00:15.108: FIPS RADIO test AES 128-bit CCM decrypt on Dot11Radio 1 successful

*Mar 1 00:00:15.108: FIPS RADIO test AMAC AES 128-bit CMAC encrypt on Dot11Radio 1 successful

*Mar 1 00:00:15.108: %SOAP_FIPS-2-SELF_TEST_RAD_SUCCESS: RADIO crypto FIPS self test passed on interface Dot11Radio 1

*Mar 1 00:00:15.468: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0, changed state to up

*Mar 1 00:09:17.133: %SYS-5-RESTART: System restarted --

Cisco IOS Software, C1140 Software (C1140-K9W8-M), Version 15.2(4)JA1, RELEASE SOFTWARE (fc2)

Technical Support: http://www.cisco.com/techsupport

Compiled Tue 30-Jul-13 23:25 by prod_rel_team

*Mar 1 00:09:17.133: %SNMP-5-COLDSTART: SNMP agent on host MS-AP10 is undergoing a cold start

*Mar 1 00:09:17.227: %PARSER-4-BADCFG: Unexpected end of configuration file.

*Mar 1 00:09:17.249: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to reset

*Mar 1 00:09:17.250: %LINK-5-CHANGED: Interface Dot11Radio1, changed state to reset

*Mar 1 00:09:18.189: %LINEPROTO-5-UPDOWN: Line protocol on Interface BVI1, changed state to up

*Mar 1 00:09:18.280: Starting Ethernet promiscuous mode

*Mar 1 00:09:18.432: %SSH-5-ENABLED: SSH 2.0 has been enabledlwapp_crypto_init: MIC Present and Parsed Successfully

*Mar 1 00:09:45.635: Logging LWAPP message to 255.255.255.255.

*Mar 1 00:09:50.391: %LWAPP-3-LWAPP_INTERFACE_GOT_IP_ADDRESS: Interface BVI1 obtained IP from DHCP...

*Mar 1 00:09:50.511: %DHCP-6-ADDRESS_ASSIGN: Interface BVI1 assigned DHCP address 172.17.1.128, mask 255.255.255.0, hostname MS-AP10

*Mar 1 00:09:50.512: %LWAPP-3-LWAPP_INTERFACE_GOT_IP_ADDRESS: Interface BVI1 obtained IP from DHCP...

*Mar 1 00:10:01.395: %CAPWAP-5-DHCP_OPTION_43: Controller address 172.17.1.4 obtained through DHCP

*Mar 1 00:10:01.395: %CAPWAP-3-ERRORLOG: Did not get log server settings from DHCP.

*Mar 1 00:10:10.395: %CAPWAP-3-ERRORLOG: Could Not resolve CISCO-CAPWAP-CONTROLLER

*Mar 1 00:10:20.397: %CAPWAP-3-ERRORLOG: Go join a capwap controller

*Oct 25 16:04:26.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 172.17.1.4 peer_port: 5246

*Oct 25 16:04:26.521: %CAPWAP-5-DTLSREQSUCC: DTLS connection created sucessfully peer_ip: 172.17.1.4 peer_port: 5246

*Oct 25 16:04:26.522: %CAPWAP-5-SENDJOIN: sending Join Request to 172.17.1.4

*Oct 25 16:04:26.524: %CAPWAP-3-ERRORLOG: Invalid event 10 & state 5 combination.

*Oct 25 16:04:26.524: %CAPWAP-3-ERRORLOG: CAPWAP SM handler: Failed to process message type 10 state 5.

*Oct 25 16:04:26.524: %CAPWAP-3-ERRORLOG: Failed to handle capwap control message from controller

*Oct 25 16:04:26.524: %CAPWAP-3-ERRORLOG: Failed to process encrypted capwap packet from 172.17.1.4

*Oct 25 16:04:26.602: Starting Ethernet promiscuous mode

*Oct 25 16:04:29.768: ac_first_hop_mac - IP:172.17.1.4 Hop IP:172.17.1.4 IDB:BVI1

*Oct 25 16:04:29.913: %LWAPP-4-CLIENTEVENTLOG: OfficeExtend Localssid saved in AP flash

*Oct 25 16:04:30.942: %CAPWAP-5-JOINEDCONTROLLER: AP has joined controller Town_vWLC

*Oct 25 16:04:30.993: %LWAPP-4-CLIENTEVENTLOG: No Flex ACL map configuration file to load. Connect to controller to get configuration file

*Oct 25 16:04:30.994: %LWAPP-4-CLIENTEVENTLOG: No Flex ACL map configuration file to load. Connect to controller to get configuration file

*Oct 25 16:04:30.994: %LWAPP-4-CLIENTEVENTLOG: No LS Flex ACL map configuration file to load. Connect to controller to get configuration file

*Oct 25 16:04:30.995: %LWAPP-4-CLIENTEVENTLOG: No Central Dhcp map configuration file to load. Connect to controller to get configuration fileWLAN id 1, SSID building_wireless, L2ACL , L2ACL AP

WLAN id 2, SSID building_guest, L2ACL , L2ACL AP

*Oct 25 16:04:35.708: %WIDS-6-ENABLED: IDS Signature is loaded and enabled

*Oct 25 16:04:38.980: %CDP_PD-4-POWER_OK: Full power - NEGOTIATED inline power source

*Oct 25 16:04:40.005: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to up

*Oct 25 16:04:41.092: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to up

*Oct 25 16:04:41.117: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to up

*Oct 25 16:04:42.117: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to up

*Oct 25 16:31:25.654: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to down

*Oct 25 16:31:25.656: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to reset

*Oct 25 16:31:26.654: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to down

------CLIPPED---------

*Oct 28 13:41:23.131: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to reset

*Oct 28 13:41:24.128: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to down

*Oct 28 13:41:24.152: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to up

*Oct 28 13:41:25.152: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to up

MS-AP10#

To me this is pointing at a power problem, power is fluctuating causing the interface to bounce up and down then eventually locking up the device. This building has had power issues in the past, (I had to replace all of their PoE switches after an outage) but I cannot find any concrete evidence to support my theory.

Access Points Lock up

victorhdss — Thu, 05 Dec 2013 17:31:08 GMT

I'm currently having the same problem. The APs 1042 just stops associating new users and all the users that are already associated cannot send and/or receive traffic. If you check AP status into the WLC it shows operational and the number of clients associated.

We just start experiencing this problem after we upgraded (about 3 weeks ago) our WLC / AP to 7.5.102.0 . Recently we acquired some 1602 wich still didnt locked up.

I will let you know if I figure something out. If you find something new, please, let me know.

Regards,

Victor Silva

Access Points Lock up

Stephen Rodriguez — Thu, 05 Dec 2013 18:33:31 GMT

You need to open a TAC case and try to get them to open an Escalation Case so that you can get the radio buffers checked.

HTH,
Steve

------------------------------------------------------------------------------------------------
Please remember to rate useful posts, and mark questions as answered

Access Points Lock up

Rasika Nayanajith — Thu, 05 Dec 2013 19:42:15 GMT

This sounds like a bug, possibly could be this CSCuj15277. As steve said, open a TAC case & get a code which has fix for this.

Symptom:

AP1140 on 7.5.102.0 stops accepting new clients on the 2.4 GHz interface.

When the AP is in the failed state the radio shows as UP and beaconing, but there's no data transmission (just beacons).

The Transmit queues seen on the radio do0 shows the following (note the global Current value and the active cnt on the voice queue):

Workaround:

Reload the AP

Known Fixed Releases:(5)

7.6(1.117)

10.1(11.6)

10.1(102.7)

15.2(4)JB1

7.5(102.11)

HTH

Rasika

*** Pls rate all useful responses***

topic Access Points Lock up in Wireless

Access Points Lock up

Access Points Lock up

Access Points Lock up

Access Points Lock up