topic WLC Heartbeat/timeout errors in Wireless

WLC Heartbeat/timeout errors

Kyle Gatlin — Sun, 04 Jul 2021 04:54:09 GMT

Had various issues with WLC recently. Still not sure if it's WLC or LAP that's causing issue.

Here are the logs:

5	Thu Mar 29 16:20:10 2012	AP Disassociated. Base Radio MAC:c4:71:fe:42:15:ac
6	Thu Mar 29 16:20:10 2012	AP's Interface:1(unknown type) Operation State Down: Base Radio MAC:c4:71:fe:42:15:ac Cause=Max Retransmission Status:NA
7	Thu Mar 29 16:20:10 2012	AP's Interface:0(unknown type) Operation State Down: Base Radio MAC:c4:71:fe:42:15:ac Cause=Max Retransmission Status:NA

0 Thu Mar 29 05:16:31 2012 AP Disassociated. Base Radio MAC:c4:71:fe:42:15:ac

1 Thu Mar 29 05:16:31 2012 AP's Interface:1(unknown type) Operation State Down: Base Radio MAC:c4:71:fe:42:15:ac Cause=Heartbeat Timeout

2 Thu Mar 29 05:16:31 2012 AP's Interface:0(unknown type) Operation State Down: Base Radio MAC:c4:71:fe:42:15:ac Cause=Heartbeat Timeout

3 Thu Mar 29 05:09:12 2012 Client Excluded: MACAddress:00:20:00:9a:32:7d Base Radio MAC :00:3a:98:98:fb:b0 Slot: 0 User Name: unknown Ip Address: unknown Reason:802.1x Authentication failed 3 times. ReasonCode: 4

4 Thu Mar 29 05:06:45 2012 Client Excluded: MACAddress:00:20:00:9a:32:7d Base Radio MAC :00:3a:98:98:fb:b0 Slot: 0 User Name: unknown Ip Address: unknown Reason:802.1x Authentication failed 3 times. ReasonCode: 4

5 Thu Mar 29 05:05:09 2012 Client Excluded: MACAddress:00:20:00:9a:32:7d Base Radio MAC :00:3a:98:98:fb:b0 Slot: 0 User Name: unknown Ip Address: unknown Reason:802.1x Authentication failed 3 times. ReasonCode: 4

6 Thu Mar 29 05:02:49 2012 Client Excluded: MACAddress:00:20:00:9a:32:7d Base Radio MAC :00:3a:98:98:fb:b0 Slot: 0 User Name: unknown Ip Address: unknown Reason:802.1x Authentication failed 3 times. ReasonCode: 4

7 Thu Mar 29 05:01:01 2012 Client Excluded: MACAddress:00:20:00:9a:32:7d Base Radio MAC :00:3a:98:98:fb:b0 Slot: 0 User Name: unknown Ip Address: unknown Reason:802.1x Authentication failed 3 times. ReasonCode: 4

8 Thu Mar 29 04:59:24 2012 Client Excluded: MACAddress:00:20:00:9a:32:7d Base Radio MAC :00:3a:98:98:fb:b0 Slot: 0 User Name: unknown Ip Address: unknown Reason:802.1x Authentication failed 3 times. ReasonCode: 4

9 Thu Mar 29 04:56:46 2012 Client Excluded: MACAddress:00:20:00:9a:32:7d Base Radio MAC :00:3a:98:98:fb:b0 Slot: 0 User Name: unknown Ip Address: unknown Reason:802.1x Authentication failed 3 times. ReasonCode: 4

10 Thu Mar 29 04:55:11 2012 Client Excluded: MACAddress:00:20:00:9a:32:7d Base Radio MAC :00:3a:98:98:fb:b0 Slot: 0 User Name: unknown Ip Address: unknown Reason:802.1x Authentication failed 3 times. ReasonCode: 4

11 Thu Mar 29 04:53:59 2012 Coverage hole pre alarm for client[1] 70:de:e2:79:b1:16 on 802.11b/g interface of AP 00:3a:98:98:fb:b0 (AP5475.d09b.baca). Hist: 0 1 7 19 24 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

12 Thu Mar 29 04:53:36 2012 Client Excluded: MACAddress:00:20:00:9a:32:7d Base Radio MAC :00:3a:98:98:fb:b0 Slot: 0 User Name: unknown Ip Address: unknown Reason:802.1x Authentication failed 3 times. ReasonCode: 4

13 Thu Mar 29 04:51:26 2012 Client Excluded: MACAddress:00:20:00:9a:32:7d Base Radio MAC :00:3a:98:98:fb:b0 Slot: 0 User Name: unknown Ip Address: unknown Reason:802.1x Authentication failed 3 times. ReasonCode: 4

14 Thu Mar 29 04:48:36 2012 Rogue AP : 00:24:b2:80:a8:aa detected on Base Radio MAC : 00:3a:98:98:f9:c0 Interface no:0(802.11b/g) with RSSI: -58 and SNR: 34 and Classification: unclassified

15 Thu Mar 29 04:48:36 2012 Rogue AP : 00:18:39:d8:91:77 detected on Base Radio MAC : 00:3a:98:98:f9:c0 Interface no:0(802.11b/g) with RSSI: -82 and SNR: 10 and Classification: unclassified

16 Thu Mar 29 04:48:36 2012 Rogue AP : 00:20:a6:a5:18:b5 detected on Base Radio MAC : 00:3a:98:98:f9:c0 Interface no:0(802.11b/g) with RSSI: -82 and SNR: 13 and Classification: unclassified

Log System Time Trap

3 Thu Mar 29 05:31:03 2012 AP Disassociated. Base Radio MAC:c4:71:fe:42:15:ac

4 Thu Mar 29 05:31:03 2012 AP's Interface:1(unknown type) Operation State Down: Base Radio MAC:c4:71:fe:42:15:ac Cause=Heartbeat Timeout

5 Thu Mar 29 05:31:03 2012 AP's Interface:0(unknown type) Operation State Down: Base Radio MAC:c4:71:fe:42:15:ac Cause=Heartbeat Timeout

Issue occurs across two different APs. One I think may actually be bad? the one with MAC 15:AC. No lights come on this device when plugged in. Keep getting these errors.

The other is working on the LAN at work but not working at remote location. Gonna head back out there with it and see if it works.

15:AC is a brand new out of the box LAP.

Any ideas of where to start would be appreciated!

WLC Heartbeat/timeout errors

George Stefanick — Thu, 29 Mar 2012 18:02:06 GMT

Hi Kyle,

Have you looked at the switch port status this ap is connected to, if not lets take a peek and see if there are any issues on the port. Also do a show log on the switch and see if there is any activity on the ap ports as well.

How many WLCs and APs do you have and all the other aps are fine?

WLC Heartbeat/timeout errors

Kyle Gatlin — Thu, 29 Mar 2012 18:15:01 GMT

The issue has followed the AP across multiple interfaces at location. AP also generates same error message on WLC at main office as well.

The WLC and two APs are currently working at remote location. Two APs were sent back. The non functioning AP and the one that was to replace it but didn't work. I got one to work at the office here, but the other still generates those errors. The 15:AC device.

Gonna take the other one out Monday and see if I can get it to work. But when in remote location, AP would not work on any interface. Just joins the WLC for a few seconds and then disappears.

WLC Heartbeat/timeout errors

George Stefanick — Thu, 29 Mar 2012 18:27:33 GMT

What is the connection between the remote locations and the location the WLC is at..?

The AP creates a CAPWAP tunnel between the AP and the WLC. Inside this tunnel there are 2 paths. A control path and a data path. The control path is what keeps your ap connected to the WLC. There are speciifc latecy requirements for this control path to stay up. If you see her up and down. I suspect the latecny could be your issue. What do these WAN links look like ?

Also if they are remote could might consider HREAP local switching perhaps to keep that traffic local.

WLC Heartbeat/timeout errors

Kyle Gatlin — Thu, 29 Mar 2012 18:31:57 GMT

There's no connection. They're both networks that I manage that happen to have a similar WLC setup. Both are independent from each other.

At the remote location it's three 3750s fiber trunked together. Vlan 130 is the Vlan for wireless. This vlan is the vlan for the WLC and AP interfaces.

The 3750 that the WLC is connected to is the switch I grabbed some of those logs from. The AP I was testing with was also hanging off this same switch (15:AC).

The APs that were working were hanging off of the trunked switches.

WLC Heartbeat/timeout errors

George Stefanick — Thu, 29 Mar 2012 18:39:25 GMT

Sorry I like pictures and crayons sometimes.. Helps me visualize ... So your remote location(s) has a WLC and acces points. There is no WAN invloved. Understood...

Now knowing this a new round of questions ...

What WLC model and code are on

What is your AP model

How is your switch port configured at your WLC

How is your switch port configured at your access points

WLC Heartbeat/timeout errors

Kyle Gatlin — Thu, 29 Mar 2012 18:47:09 GMT

NP. Thanks for your help so far! No WAN. Remote has a cable modem that hooks up to a Cisco router-on-a-stick with sub interfaces and all that. I'd also like to note that everythign was working just fine until last week.

Here's the interface pointing to WLC.

interface FastEthernet0/2

description Trunk wlc

switchport trunk allowed vlan 130

switchport mode trunk

end

Here's a switchport for an AP.

interface FastEthernet0/7

switchport access vlan 130

end

I am not at location so I can't get exact WLC info at the moment. Here's what I got via cdp neigh detail

Device ID: Cisco_d6:a2:e0

Entry address(es):

IP address: 192.168.130.5

Platform: AIR-WLC2106-K9, Capabilities: Host

Interface: FastEthernet0/2, Port ID (outgoing port): Fastethernet0/0/1

Holdtime : 134 sec

Version :

Manufacturer's Name: Cisco Systems Inc. Product Name: Cisco Controller Product Version: 6.0.196.0 RTOS Version: 6.0.196.0 Bootloader Version: 4.0.191.0 Build Type: DATA + WPS

advertisement version: 1

Management address(es):

WLC Heartbeat/timeout errors

George Stefanick — Thu, 29 Mar 2012 19:09:55 GMT

I might recommend upgrading the code on the WLC. The 6 train is very buggy.. I posted this on my blog, that code had some bugs labeled as "catastrophic", not my words, rather what cisco labeled the bug severity.

http://www.my80211.com/cisco-field-alerts/2010/6/24/bugs-csctf34858-severity-1-catastrophic-wlc-code-levels-6018.html

So this was working fine and then started up all of a sudden ... Any chnages to the network at all ?

WLC Heartbeat/timeout errors

Kyle Gatlin — Thu, 29 Mar 2012 19:16:13 GMT

Thank you very much! I'll look up on how to upgrade the WLC to that code base! Hopefully that does the trick!

WLC Heartbeat/timeout errors

Kyle Gatlin — Mon, 02 Apr 2012 14:27:30 GMT

Correct. NO network changes as far as I am aware.

WLC Heartbeat/timeout errors

Kyle Gatlin — Mon, 02 Apr 2012 16:01:31 GMT

http://www.cisco.com/cisco/software/release.html?mdfid=280954465&release=7.0.230.0&relind=AVAILABLE&flowid=7009&softwareid=280926587&rellifecycle=ED&reltype=latest

Do I need to download and install both or just the main OS?

WLC Heartbeat/timeout errors

Kyle Gatlin — Wed, 04 Apr 2012 14:27:42 GMT

Is the AP currently connected to controller................ No

Time at which the AP joined this controller last time...... Apr 02 12:34:01.451

Type of error that occurred last........................... AP got or has been disconnected

Reason for error that occurred last........................ Timed out while waiting for ECHO repsonse from the AP

Time at which the last join error occurred................. Apr 02 12:38:34.546

(Cisco Controller) show>ap join stats summary c4:71:fe:42:15:ac

Is the AP currently connected to controller................ No

Time at which the AP joined this controller last time...... Not applicable

Type of error that occurred last........................... AP got or has been disconnected

Reason for error that occurred last........................ Timed out while waiting for ECHO repsonse from the AP

Time at which the last join error occurred................. Apr 02 11:33:12.444

(Cisco Controller) show>ap join stats summary 54:75:d0:9b:bf:f2

Is the AP currently connected to controller................ No

Time at which the AP joined this controller last time...... Not applicable

Type of error that occurred last........................... AP got or has been disconnected

Reason for error that occurred last........................ Timed out while waiting for ECHO repsonse from the AP

Time at which the last join error occurred................. Apr 02 11:58:09.217