8510 WLC Flexconnect AP Association Issues

dperowne · ‎01-17-2014

Seeing a host of issues on a large multi-site flexconnect network. It could be a number of things, such as the APs having issues with the L3 routing over WAN DSL, or WLC issues including bugs in the 7.5 code or potentially AP management issues (what is the per interface AP management limit on these new WLCs?)

There are a number of small remote sites that run 2602 APs in flexconnect mode back to the main DC with an 8510 controller (v7.5.102.0). DTLS is disabled.

Prime registers many alarms for AP association/deassociation, and when searching for a site of APs on the WLC and re-searching, I never get the same number with a few APs ghosting in and out due to this issue.

Investigation on the APs themselves shows random de-assoc and assoc events lasting 15secs to 5mins or many more minutes. No obvious pattern.

The problem it's causing is clients on the WiFi end are cut off and their authentication session is left open on the central server not allowing reconnection.

I'm still trying to figure out if this is a bug, WLC set up issue in regards to AP registration, L3 AP operation or asymmetrical routing issue, or a combination of all of the above --so I thought I'd post it on here to see if anyone had any ideas while I continue trawling through configuration and logs...

Example AP log output:

AP_04#sh log | b 13:27

*Jan 17 13:13:27.203: %CAPWAP-3-ERRORLOG: Retransmission count for packet exceeded max(CAPWAP_WTP_EVENT_REQUEST

., 123)

*Jan 17 13:13:27.207: %LWAPP-3-CLIENTERRORLOG: Switching to Standalone mode

*Jan 17 13:13:27.207: %CAPWAP-3-ERRORLOG: GOING BACK TO DISCOVER MODE

*Jan 17 13:13:27.207: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 10.90.100.100:5246

*Jan 17 13:13:27.271: %WIDS-6-DISABLED: IDS Signature is removed and disabled.

*Jan 17 13:13:31.031: %CLEANAIR-6-STATE: Slot 0 down

*Jan 17 13:13:31.031: %CLEANAIR-6-STATE: Slot 1 down

*Jan 17 13:13:37.275: %CAPWAP-3-ERRORLOG: Go join a capwap controller

*Jan 17 13:14:37.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.90.100.100 peer_port: 5246

*Jan 17 13:14:39.463: %CAPWAP-5-DTLSREQSUCC: DTLS connection created sucessfully peer_ip: 10.90.100.100 peer_port: 5246

*Jan 17 13:14:39.463: %CAPWAP-5-SENDJOIN: sending Join Request to 10.90.100.100

*Jan 17 13:14:44.463: %CAPWAP-5-SENDJOIN: sending Join Request to 10.90.100.100

*Jan 17 13:14:47.859: Starting Ethernet promiscuous mode

*Jan 17 13:14:54.743: ac_first_hop_mac - IP:10.3.122.254 Hop IP:10.3.122.254 IDB:BVI1

*Jan 17 13:14:54.759: %LWAPP-4-CLIENTEVENTLOG: OfficeExtend Localssid saved in AP flash

*Jan 17 13:14:59.155: %CAPWAP-5-JOINEDCONTROLLER: AP has joined controller SOV-1H2-WLC-02P

*Jan 17 13:15:01.327: %LWAPP-4-CLIENTEVENTLOG: No Flex ACL map configuration file to load. Connect to controller to get configuration file

*Jan 17 13:15:01.327: %LWAPP-4-CLIENTEVENTLOG: No LS Flex ACL map configuration file to load. Connect to controller to get configuration file

*Jan 17 13:15:01.327: %LWAPP-4-CLIENTEVENTLOG: No Central Dhcp map configuration file to load. Connect to controller to get configuration file

*Jan 17 13:15:01.327: %LWAPP-3-CLIENTERRORLOG: Switching to Connected mode

*Jan 17 13:15:14.935: %WIDS-6-ENABLED: IDS Signature is loaded and enabled

*Jan 17 13:16:03.467: %CLEANAIR-6-STATE: Slot 0 enabled

*Jan 17 13:16:05.523: %CLEANAIR-6-STATE: Slot 1 enabled

*Jan 17 13:17:53.771: %CAPWAP-3-ERRORLOG: Retransmission count for packet exceeded max(CAPWAP_WTP_EVENT_REQUEST

., 56)

*Jan 17 13:17:53.775: %LWAPP-3-CLIENTERRORLOG: Switching to Standalone mode

*Jan 17 13:17:53.775: %CAPWAP-3-ERRORLOG: GOING BACK TO DISCOVER MODE

*Jan 17 13:17:53.775: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 10.90.100.100:5246

*Jan 17 13:17:55.583: %CLEANAIR-6-STATE: Slot 0 down

*Jan 17 13:17:55.583: %CLEANAIR-6-STATE: Slot 1 down

*Jan 17 13:17:55.903: %WIDS-6-DISABLED: IDS Signature is removed and disabled.

*Jan 17 13:18:05.907: %CAPWAP-3-ERRORLOG: Go join a capwap controller

*Jan 17 13:18:06.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.90.100.100 peer_port: 5246

*Jan 17 13:18:36.715: DTLS_CLIENT_ERROR: ../capwap/base_capwap/dtls/base_capwap_dtls_connection_db.c:2134 Max retransmission count reached!

*Jan 17 13:19:05.999: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 10.90.100.100:5246

*Jan 17 13:19:05.999: %CAPWAP-3-ERRORLOG: Go join a capwap controller

*Jan 17 13:18:06.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.90.100.100 peer_port: 5246

*Jan 17 13:18:12.791: %CAPWAP-5-DTLSREQSUCC: DTLS connection created sucessfully peer_ip: 10.90.100.100 peer_port: 5246

*Jan 17 13:18:12.791: %CAPWAP-5-SENDJOIN: sending Join Request to 10.90.100.100

*Jan 17 13:18:17.791: %CAPWAP-5-SENDJOIN: sending Join Request to 10.90.100.100

*Jan 17 13:18:20.663: Starting Ethernet promiscuous mode

*Jan 17 13:18:25.659: ac_first_hop_mac - IP:10.3.122.254 Hop IP:10.3.122.254 IDB:BVI1

*Jan 17 13:18:25.675: %LWAPP-4-CLIENTEVENTLOG: OfficeExtend Localssid saved in AP flash

*Jan 17 13:18:31.971: %CAPWAP-5-JOINEDCONTROLLER: AP has joined controller SOV-1H2-WLC-02P

*Jan 17 13:18:32.035: %LWAPP-4-CLIENTEVENTLOG: No Flex ACL map configuration file to load. Connect to controller to get configuration file

*Jan 17 13:18:32.035: %LWAPP-4-CLIENTEVENTLOG: No LS Flex ACL map configuration file to load. Connect to controller to get configuration file

*Jan 17 13:18:32.035: %LWAPP-4-CLIENTEVENTLOG: No Central Dhcp map configuration file to load. Connect to controller to get configuration file

*Jan 17 13:18:32.035: %LWAPP-3-CLIENTERRORLOG: Switching to Connected mode

*Jan 17 13:18:52.251: %WIDS-6-ENABLED: IDS Signature is loaded and enabled

*Jan 17 13:19:23.167: %CLEANAIR-6-STATE: Slot 0 enabled

*Jan 17 13:19:25.227: %CLEANAIR-6-STATE: Slot 1 enabled

*Jan 17 13:21:05.763: %CAPWAP-3-ERRORLOG: Retransmission count for packet exceeded max(CAPWAP_WTP_EVENT_REQUEST

., 32)

*Jan 17 13:21:05.767: %LWAPP-3-CLIENTERRORLOG: Switching to Standalone mode

*Jan 17 13:21:05.767: %CAPWAP-3-ERRORLOG: GOING BACK TO DISCOVER MODE

*Jan 17 13:21:05.767: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 10.90.100.100:5246

*Jan 17 13:21:07.983: %WIDS-6-DISABLED: IDS Signature is removed and disabled.

*Jan 17 13:21:10.227: %CLEANAIR-6-STATE: Slot 0 down

*Jan 17 13:21:10.227: %CLEANAIR-6-STATE: Slot 1 down

*Jan 17 13:21:17.987: %CAPWAP-3-ERRORLOG: Go join a capwap controller

*Jan 17 13:22:18.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.90.100.100 peer_port: 5246

*Jan 17 13:22:32.319: %CAPWAP-5-DTLSREQSUCC: DTLS connection created sucessfully peer_ip: 10.90.100.100 peer_port: 5246

*Jan 17 13:22:32.319: %CAPWAP-5-SENDJOIN: sending Join Request to 10.90.100.100

*Jan 17 13:22:37.319: %CAPWAP-5-SENDJOIN: sending Join Request to 10.90.100.100

*Jan 17 13:22:40.851: Starting Ethernet promiscuous mode

*Jan 17 13:22:46.075: ac_first_hop_mac - IP:10.3.122.254 Hop IP:10.3.122.254 IDB:BVI1

*Jan 17 13:22:46.095: %LWAPP-4-CLIENTEVENTLOG: OfficeExtend Localssid saved in AP flash

*Jan 17 13:22:52.335: %CAPWAP-5-JOINEDCONTROLLER: AP has joined controller SOV-1H2-WLC-02P

*Jan 17 13:22:52.399: %LWAPP-4-CLIENTEVENTLOG: No Flex ACL map configuration file to load. Connect to controller to get configuration file

*Jan 17 13:22:52.399: %LWAPP-4-CLIENTEVENTLOG: No LS Flex ACL map configuration file to load. Connect to controller to get configuration file

*Jan 17 13:22:52.399: %LWAPP-4-CLIENTEVENTLOG: No Central Dhcp map configuration file to load. Connect to controller to get configuration file

*Jan 17 13:22:52.399: %LWAPP-3-CLIENTERRORLOG: Switching to Connected mode

*Jan 17 13:23:10.835: %WIDS-6-ENABLED: IDS Signature is loaded and enabled

*Jan 17 13:23:57.883: %CLEANAIR-6-STATE: Slot 0 enabled

*Jan 17 13:23:59.943: %CLEANAIR-6-STATE: Slot 1 enabled

*Jan 17 13:25:25.531: %CAPWAP-3-ERRORLOG: Received packet with invalid sequence number: got 73 expected 74.

*Jan 17 13:25:25.531: %CAPWAP-3-ERRORLOG: Failed to handle capwap control message from controller

*Jan 17 13:25:25.531: %CAPWAP-3-ERRORLOG: Failed to process encrypted capwap packet from 10.90.100.100

*Jan 17 13:25:55.271: %CAPWAP-3-ERRORLOG: Retransmission count for packet exceeded max(CAPWAP_WTP_EVENT_REQUEST

., 42)

*Jan 17 13:25:55.275: %LWAPP-3-CLIENTERRORLOG: Switching to Standalone mode

*Jan 17 13:25:55.275: %CAPWAP-3-ERRORLOG: GOING BACK TO DISCOVER MODE

*Jan 17 13:25:55.275: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 10.90.100.100:5246

*Jan 17 13:25:57.427: %WIDS-6-DISABLED: IDS Signature is removed and disabled.

*Jan 17 13:25:59.943: %CLEANAIR-6-STATE: Slot 0 down

*Jan 17 13:25:59.943: %CLEANAIR-6-STATE: Slot 1 down

*Jan 17 13:26:07.431: %CAPWAP-3-ERRORLOG: Go join a capwap controller

*Jan 17 13:26:07.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.90.100.100 peer_port: 5246

*Jan 17 13:26:10.103: %CAPWAP-5-DTLSREQSUCC: DTLS connection created sucessfully peer_ip: 10.90.100.100 peer_port: 5246

*Jan 17 13:26:10.103: %CAPWAP-5-SENDJOIN: sending Join Request to 10.90.100.100

*Jan 17 13:26:15.103: %CAPWAP-5-SENDJOIN: sending Join Request to 10.90.100.100

*Jan 17 13:26:17.719: Starting Ethernet promiscuous mode

*Jan 17 13:26:22.715: ac_first_hop_mac - IP:10.3.122.254 Hop IP:10.3.122.254 IDB:BVI1

*Jan 17 13:26:22.735: %LWAPP-4-CLIENTEVENTLOG: OfficeExtend Localssid saved in AP flash

*Jan 17 13:26:29.047: %CAPWAP-5-JOINEDCONTROLLER: AP has joined controller SOV-1H2-WLC-02P

*Jan 17 13:26:31.255: %LWAPP-4-CLIENTEVENTLOG: No Flex ACL map configuration file to load. Connect to controller to get configuration file

*Jan 17 13:26:31.255: %LWAPP-4-CLIENTEVENTLOG: No LS Flex ACL map configuration file to load. Connect to controller to get configuration file

*Jan 17 13:26:31.255: %LWAPP-4-CLIENTEVENTLOG: No Central Dhcp map configuration file to load. Connect to controller to get configuration file

*Jan 17 13:26:31.255: %LWAPP-3-CLIENTERRORLOG: Switching to Connected mode

*Jan 17 13:26:46.731: %WIDS-6-ENABLED: IDS Signature is loaded and enabled

*Jan 17 13:27:41.555: %CLEANAIR-6-STATE: Slot 0 enabled

*Jan 17 13:27:43.615: %CLEANAIR-6-STATE: Slot 1 enabled

AP_04#

AP Event Log from WLC:

Jan 17 11:12:06.995: %CAPWAP-5-JOINEDCONTROLLER: AP has joined controller WLC-02

Jan 17 11:12:09.111: %LWAPP-4-CLIENTEVENTLOG: No Flex ACL map configuration file to load. Connect to controller to get configur

Jan 17 11:12:09.111: %EVT-5-NTC: CAPWAP Received DELETE_BLACKLIST_ENTRY

., 16) 11:12:24.995: %CAPWAP-3-ERRORLOG: Retransmission count for packet exceeded max(UNKNOWN_MESSAGE_TYPE (11)

Jan 17 11:12:24.995: %LWAPP-3-CLIENTERRORLOG: Switching to Standalone mode

Jan 17 11:12:24.995: %CAPWAP-3-ERRORLOG: GOING BACK TO DISCOVER MODE

Jan 17 11:12:24.995: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 10.90.100.100:5246

Jan 17 11:12:35.059: %CAPWAP-3-ERRORLOG: Go join a capwap controller

Jan 17 11:12:35.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.90.100.100 peer_port: 5246

Jan 17 11:12:35.399: %CAPWAP-5-DTLSREQSUCC: DTLS connection created sucessfully peer_ip: 10.90.100.100 peer_port: 5246

Jan 17 11:12:35.403: %CAPWAP-5-SENDJOIN: sending Join Request to 10.90.100.100

Jan 17 11:12:40.399: %CAPWAP-5-SENDJOIN: sending Join Request to 10.90.100.100

Jan 17 11:12:47.251: %EVT-5-NTC: CAPWAP Received DECRYPT_ERR_REPORT_PERIOD

Jan 17 11:12:47.251: %EVT-5-NTC: CAPWAP Received SPAM_DOMAIN_SECRET_PAYLOAD

Jan 17 11:12:47.251: %EVT-5-NTC: CAPWAP Received STATION_CFG_PAYLOAD

Jan 17 11:12:47.251: %EVT-5-CFG: Slot0: beacon period 100

Jan 17 11:12:47.251: %EVT-5-CFG: Slot0: dtim interval 1

Jan 17 11:12:47.251: %EVT-5-CFG: Slot0: country code GB

Jan 17 11:12:47.251: %EVT-5-CFG: Slot0: world mode 1

Jan 17 11:12:47.251: %EVT-5-CFG: Slot0: GPR period 10

Jan 17 11:12:47.251: %EVT-5-CFG: Slot0: max clients 200

Jan 17 11:12:47.251: %EVT-5-CFG: Slot0: dfs pkdetect 0

Jan 17 11:12:47.251: %EVT-5-CFG: Slot0: Vocera Fix 0

Jan 17 11:12:47.251: %EVT-5-NTC: CAPWAP Received OPERATION_RATE_SET_PAYLOAD

Jan 17 11:12:47.251: %EVT-5-CFG: Slot0: rates 8C129824 3048606C 00000000 00000000

Jan 17 11:12:47.251: %EVT-5-NTC: CAPWAP Received MULTI_DOMAIN_CAPABILITY_PAYLOAD

Jan 17 11:12:47.255: %EVT-5-NTC: CAPWAP Received PHY_TX_POWER_PAYLOAD

Jan 17 11:12:47.255: %EVT-5-NTC: slot=0 radio state 6 change

Jan 17 11:12:47.291: %EVT-5-CFG: Slot0: tx power level 1

Jan 17 11:12:47.291: %EVT-5-NTC: CAPWAP Received MAC_OPERATION_PAYLOAD

Jan 17 11:12:47.291: %EVT-5-CFG: Slot0: rts threshhold 2347

Jan 17 11:12:47.291: %EVT-5-CFG: Slot0: fragmentation threshold 2346

Jan 17 11:12:47.291: %EVT-5-NTC: CAPWAP Received SPAM_VENDOR_ID_PAYLOAD

Jan 17 11:12:47.291: %EVT-5-NTC: Vendor subtype Unknown Payload(111)

Jan 17 11:12:47.351: %EVT-5-CFG: d0: Set Extended Config: period 100, range 0,mcast_buf 0, mcast_rate 0, rxsens 0, cca 0

Jan 17 11:12:47.351: %EVT-5-NTC: CAPWAP Received ANTENNA_PAYLOAD

Jan 17 11:12:47.351: %EVT-5-CFG: Slot0: Antenna Diversity FF, Mode 3, Num antennas 4, Gain 0

Jan 17 11:12:47.351: %EVT-5-CFG: Slot0: HT Antenna Selection: Rx 0F Tx 0F

Jan 17 11:12:47.351: %EVT-5-NTC: CAPWAP Received AP_QOS_PAYLOAD

Jan 17 11:12:47.351: %EVT-5-CFG: Slot0: Change max/uni/multi priority 3/3/3 for WLAN 1

Jan 17 11:12:47.351: %EVT-5-NTC: CAPWAP Received AIRESPACE_CAPABILITY_PAYLOAD

Jan 17 11:12:47.351: %EVT-5-NTC: CAPWAP Received CHANGE_STATE_EVENT_PAYLOAD

Jan 17 11:12:55.559: %EVT-5-NTC: CAPWAP Received AP_DOT11H_PAYLOAD

Jan 17 11:12:55.559: %EVT-5-NTC: CAPWAP Received AP_DTPC_PAYLOAD

Jan 17 11:12:55.559: %EVT-5-NTC: CAPWAP Received AP_TSM_CONFIG_PAYLOAD

Jan 17 11:12:55.559: %EVT-5-CFG: Slot0: Voice metrics OFF

Jan 17 11:12:55.559: %EVT-5-NTC: CAPWAP Received AP_CAC_CONFIG_PAYLOAD

Jan 17 11:12:55.559: %EVT-5-NTC: CAPWAP Received SPAM_VENDOR_ID_PAYLOAD

Jan 17 11:12:55.559: %EVT-5-NTC: Vendor subtype AP_ENHANCED_CHD_CONFIG_PAYLOAD

Jan 17 11:12:55.559: %EVT-5-CFG: CHD voice Thresholds: Rssi = 80 dBm , Total = 100 pkts, Pct = 50

Jan 17 11:12:55.559: %EVT-5-CFG: CHD data Thresholds: Rssi = 80 dBm , Total = 50 pkts, Pct = 50

Jan 17 11:12:55.559: %EVT-5-CFG: CHD alarm count = 5

Jan 17 11:12:55.559: %EVT-5-NTC: CAPWAP Received SPAM_VENDOR_ID_PAYLOAD

Jan 17 11:12:55.559: %EVT-5-NTC: Vendor subtype AP_VOICE_MAC_OPT_PAYLOAD

Jan 17 11:12:55.559: %EVT-5-NTC: CAPWAP Received AP_CISCO_7920_VSIE_PAYLOAD

Jan 17 11:12:55.559: %EVT-5-NTC: CAPWAP Received SPAM_VENDOR_ID_PAYLOAD

Jan 17 11:12:55.559: %EVT-5-NTC: Vendor subtype PHY_HT_CONTROL_PAYLOAD

Jan 17 11:12:55.559: %EVT-5-NTC: CAPWAP Received SPAM_VENDOR_ID_PAYLOAD

Jan 17 11:12:55.559: %EVT-5-NTC: Vendor subtype PHY_HT_CAPABILITIES_PAYLOAD

Jan 17 11:12:55.559: %EVT-5-CFG: Slot0: mcsrates FFFFFF00 00000000 00000000 00000000

Jan 17 11:12:55.559: %EVT-5-CFG: Slot0: VHT mcsrates 00000000 00000000

Jan 17 11:12:55.559: %EVT-5-NTC: CAPWAP Received SPAM_VENDOR_ID_PAYLOAD

Jan 17 11:12:55.559: %EVT-5-NTC: Vendor subtype PHY_HT_ADDITIONAL_INFO_PAYLOAD

Jan 17 11:12:55.559: %EVT-5-CFG: Slot0: tx ampdu 0x31, tx amsdu 0x3F, guard interval 1, aggr sched rt timeout 10, rifs enable 1

Jan 17 11:12:55.559: %EVT-5-CFG: Slot1: beacon period 100

Jan 17 11:12:55.559: %EVT-5-CFG: Slot1: dtim interval 1

Jan 17 11:12:55.559: %EVT-5-CFG: Slot1: country code GB

Jan 17 11:12:55.559: %EVT-5-CFG: Slot1: world mode 1

Jan 17 11:12:55.559: %EVT-5-CFG: Slot1: GPR period 10

Jan 17 11:12:55.559: %EVT-5-CFG: Slot1: max clients 200

Jan 17 11:12:55.559: %EVT-5-CFG: Slot1: dfs pkdetect 1

Jan 17 11:12:55.559: %EVT-5-CFG: Slot1: Vocera Fix 0

Jan 17 11:12:55.559: %EVT-5-NTC: CAPWAP Received OPERATION_RATE_SET_PAYLOAD

Jan 17 11:12:55.559: %EVT-5-CFG: Slot1: rates 8C129824 3048606C 00000000 00000000

Jan 17 11:12:55.559: %EVT-5-NTC: CAPWAP Received MULTI_DOMAIN_CAPABILITY_PAYLOAD

Jan 17 11:12:55.559: %EVT-5-NTC: CAPWAP Received PHY_TX_POWER_PAYLOAD

Jan 17 11:12:55.559: %EVT-5-NTC: slot=1 radio state 6 change

Jan 17 11:12:55.571: %EVT-5-CFG: Slot1: tx power level 1

Jan 17 11:12:55.571: %EVT-5-NTC: CAPWAP Received MAC_OPERATION_PAYLOAD

Jan 17 11:12:55.571: %EVT-5-CFG: Slot1: rts threshhold 2347

Jan 17 11:12:55.571: %EVT-5-CFG: Slot1: fragmentation threshold 2346

Jan 17 11:12:55.571: %EVT-5-NTC: CAPWAP Received SPAM_VENDOR_ID_PAYLOAD

Jan 17 11:12:55.571: %EVT-5-NTC: Vendor subtype Unknown Payload(111)

Jan 17 11:12:57.771: %EVT-5-CFG: d1: Set Extended Config: period 100, range 0,mcast_buf 0, mcast_rate 0, rxsens 0, cca 0

Jan 17 11:12:57.771: %EVT-5-NTC: CAPWAP Received ANTENNA_PAYLOAD

Jan 17 11:12:57.771: %EVT-5-CFG: Slot1: Antenna Diversity FF, Mode 3, Num antennas 4, Gain 0

Jan 17 11:12:57.771: %EVT-5-CFG: Slot1: HT Antenna Selection: Rx 0F Tx 0F

Jan 17 11:12:57.771: %EVT-5-NTC: CAPWAP Received AP_QOS_PAYLOAD

Jan 17 11:12:57.771: %EVT-5-CFG: Slot1: Change max/uni/multi priority 3/3/3 for WLAN 1

Jan 17 11:12:57.771: %EVT-5-NTC: CAPWAP Received AIRESPACE_CAPABILITY_PAYLOAD

Jan 17 11:12:57.771: %EVT-5-NTC: CAPWAP Received CHANGE_STATE_EVENT_PAYLOAD

Jan 17 11:13:06.031: %EVT-5-NTC: CAPWAP Received AP_DOT11H_PAYLOAD

Jan 17 11:13:06.031: %EVT-5-NTC: CAPWAP Received AP_DTPC_PAYLOAD

Jan 17 11:13:06.031: %EVT-5-NTC: CAPWAP Received AP_TSM_CONFIG_PAYLOAD

Jan 17 11:13:06.031: %EVT-5-CFG: Slot1: Voice metrics OFF

Jan 17 11:13:06.031: %EVT-5-NTC: CAPWAP Received AP_CAC_CONFIG_PAYLOAD

Jan 17 11:13:06.031: %EVT-5-NTC: CAPWAP Received SPAM_VENDOR_ID_PAYLOAD

Jan 17 11:13:06.031: %EVT-5-NTC: Vendor subtype AP_ENHANCED_CHD_CONFIG_PAYLOAD

Jan 17 11:13:06.031: %EVT-5-CFG: CHD voice Thresholds: Rssi = 80 dBm , Total = 100 pkts, Pct = 50

Jan 17 11:13:06.031: %EVT-5-CFG: CHD data Thresholds: Rssi = 80 dBm , Total = 50 pkts, Pct = 50

Jan 17 11:13:06.031: %EVT-5-CFG: CHD alarm count = 5

Jan 17 11:13:06.031: %EVT-5-NTC: CAPWAP Received SPAM_VENDOR_ID_PAYLOAD

Jan 17 11:13:06.031: %EVT-5-NTC: Vendor subtype AP_VOICE_MAC_OPT_PAYLOAD

Jan 17 11:13:06.031: %EVT-5-NTC: CAPWAP Received SPAM_VENDOR_ID_PAYLOAD

Jan 17 11:13:06.031: %EVT-5-NTC: Vendor subtype PHY_HT_CONTROL_PAYLOAD

Jan 17 11:13:06.031: %EVT-5-NTC: CAPWAP Received SPAM_VENDOR_ID_PAYLOAD

Jan 17 11:13:06.031: %EVT-5-NTC: Vendor subtype PHY_HT_CAPABILITIES_PAYLOAD

Jan 17 11:13:06.031: %EVT-5-CFG: Slot1: mcsrates FFFFFF00 00000000 00000000 00000000

Jan 17 11:13:06.031: %EVT-5-CFG: Slot1: VHT mcsrates 3F000000 00000000

Jan 17 11:13:06.031: %EVT-5-NTC: CAPWAP Received SPAM_VENDOR_ID_PAYLOAD

Jan 17 11:13:06.031: %EVT-5-NTC: Vendor subtype PHY_HT_ADDITIONAL_INFO_PAYLOAD

Jan 17 11:13:06.031: %EVT-5-CFG: Slot1: tx ampdu 0x31, tx amsdu 0x3F, guard interval 1, aggr sched rt timeout 10, rifs enable 1

Jan 17 11:13:10.099: %EVT-5-NTC: Radio d0 RST 54 Flags 60109 BCN 0

Jan 17 11:13:10.191: %EVT-5-NTC: disabled beacons: d0 disc start 2 60109 0 0 0 0 0 512

Jan 17 11:13:14.091: %EVT-5-NTC: Radio d1 RST 54 Flags 60109 BCN 0

Jan 17 11:13:14.091: %EVT-5-NTC: CAPWAP Received BCAST_SSID_MODE_PAYLOAD

Jan 17 11:13:14.091: %EVT-5-NTC: CAPWAP Received AP_TIMER_PAYLOAD

Jan 17 11:13:14.091: %EVT-5-NTC: CAPWAP Received CLIENT_AUTO_HANDOFF_PAYLOAD

Jan 17 11:13:14.091: %EVT-5-NTC: CAPWAP Received SPAM_80211_MANAGEMENT_TRAFFIC_LIMIT_PAYLOAD

Jan 17 11:13:14.091: %EVT-5-NTC: disabled beacons: d1 disc start 2 60109 0 0 0 0 0 512

Jan 17 11:13:14.095: %EVT-5-NTC: CAPWAP Received MM_SWITCH_INFO_PAYLOAD

Jan 17 11:13:14.095: %EVT-5-NTC: CAPWAP Received SPAM_VENDOR_ID_PAYLOAD

Jan 17 11:13:14.095: %EVT-5-NTC: Vendor subtype AP_HOMEAP_CONFIG_PAYLOAD

Jan 17 11:13:16.351: %LWAPP-4-CLIENTEVENTLOG: OfficeExtend Localssid saved in AP flash

Jan 17 11:13:16.351: %EVT-5-NTC: CAPWAP Received RFID_TAG_STATUS_PAYLOAD

Jan 17 11:13:16.351: %EVT-5-NTC: CAPWAP Received SPAM_VENDOR_ID_PAYLOAD

Jan 17 11:13:16.351: %EVT-5-NTC: Vendor subtype MANAGER_IP_ADDR_PAYLOAD

Jan 17 11:13:16.351: %EVT-4-WRN: Unsupported Vendor SubType 19

Jan 17 11:13:16.351: %EVT-5-NTC: CAPWAP Received MESH_LSC_ENABLE_PAYLOAD

Jan 17 11:13:16.351: %EVT-5-NTC: CAPWAP Received SPAM_VENDOR_ID_PAYLOAD

Jan 17 11:13:16.351: %EVT-5-NTC: Vendor subtype AP_MFP_CONFIG_SUBTYPE

Jan 17 11:13:16.351: %EVT-5-NTC: CAPWAP Received SPAM_VENDOR_ID_PAYLOAD

Jan 17 11:13:16.351: %EVT-5-NTC: Vendor subtype AP_TCP_STATE_PAYLOAD

Jan 17 11:13:16.351: %EVT-4-WRN: Unsupported Vendor SubType 77

Jan 17 11:13:16.351: %EVT-5-NTC: CAPWAP Received E2E_DSCP_MAP_PAYLOAD

Jan 17 11:13:16.351: %EVT-5-NTC: CAPWAP Received SPAM_AP_LOG_FACILITY_PAYLOAD

Jan 17 11:13:16.351: %EVT-5-NTC: CAPWAP Received SPAM_VENDOR_ID_PAYLOAD

Jan 17 11:13:16.351: %EVT-5-NTC: Vendor subtype AGGREGATE_PROBE_INFO_PAYLOAD

Jan 17 11:13:16.351: %EVT-5-CFG: AGGREGATE_PROBE_INFO_PAYLOAD: aggrProbeDelay 62465

Jan 17 11:13:16.351: %EVT-5-NTC: CAPWAP Received SPAM_VENDOR_ID_PAYLOAD

Jan 17 11:13:16.351: %EVT-5-NTC: Vendor subtype SPAM_DOT11_PADDING_PAYLOAD

Jan 17 11:13:16.351: %EVT-5-CFG: SPAM_DOT11_PADDING: dot11PaddingStatus 0

Jan 17 11:13:16.351: %EVT-5-NTC: CAPWAP Received SPAM_VENDOR_ID_PAYLOAD

Jan 17 11:13:16.351: %EVT-5-NTC: Vendor subtype ROGUE_CONFIG_PAYLOAD

Jan 17 11:13:16.351: %EVT-5-CFG: ROGUE_CONFIG_PAYLOAD: mode: 1 report-interval: 10 min-rssi: -128 transient-interval 0

Jan 17 11:13:16.351: %EVT-5-NTC: CAPWAP Received SPAM_VENDOR_ID_PAYLOAD

Jan 17 11:13:16.351: %EVT-5-NTC: Vendor subtype AP_WMMAC_TEST_CONFIG_PAYLOAD

Jan 17 11:13:16.351: %EVT-5-NTC: CAPWAP Received AP_FIPS_PAYLOAD

Jan 17 11:13:16.351: %EVT-5-NTC: CAPWAP Received SPAM_VENDOR_ID_PAYLOAD

Jan 17 11:13:16.351: %EVT-5-NTC: Vendor subtype Unknown Payload(119)

Jan 17 11:13:16.351: %EVT-5-NTC: CAPWAP Received SPAM_VENDOR_ID_PAYLOAD

Jan 17 11:13:16.351: %EVT-5-NTC: Vendor subtype Unknown Payload(113)

Jan 17 11:13:16.351: %EVT-5-NTC: CAPWAP Received RA_FILTERING_CFG_PAYLOAD

Jan 17 11:13:16.351: %EVT-5-CFG: Slot0: Router Advertisement filtering ENABLE

Jan 17 11:13:16.351: %EVT-5-CFG: Slot1: Router Advertisement filtering ENABLE

Jan 17 11:13:16.351: %EVT-5-NTC: CAPWAP Received SPAM_AP_LED_STATE_CONFIG_PAYLOAD

Jan 17 11:13:16.351: %EVT-5-NTC: CAPWAP Received MCAST_MGID_RANGE_PAYLOAD

Jan 17 11:13:16.351: %EVT-5-NTC: CAPWAP Received SPAM_VENDOR_ID_PAYLOAD

Jan 17 11:13:16.351: %EVT-5-NTC: Vendor subtype Unknown Payload(133)

Jan 17 11:13:16.415: %EVT-5-NTC: CAPWAP Received SPAM_VENDOR_ID_PAYLOAD

Jan 17 11:13:16.415: %EVT-5-NTC: Vendor subtype Unknown Payload(135)

Jan 17 11:13:20.471: %CAPWAP-5-JOINEDCONTROLLER: AP has joined controller WLC-02

Jan 17 11:13:22.631: %LWAPP-4-CLIENTEVENTLOG: No Flex ACL map configuration file to load. Connect to controller to get configur

WLC msglog:

Cisco Controller) >show msglog

Message Log Severity Level ...................... VERBOSE

*rmgrTrasport: Jan 17 12:15:37.146: #RMGR-3-INVALID_PING_RESPONSE: rmgr_utils.c:259 Ping response from 10.128.10.128 is invalid. Incorrect checksum.

*spamApTask3: Jan 17 12:15:37.146: #OSAPI-4-MSGQ_SEND_FAILED: osapi_msgq.c:520 Failed to send a message to the message queue object: CDP-Q . enqueue failed.

-Traceback: 0xbd440d 0x5bb353 0x6ecad7 0x6eeed9 0x77bfa3 0xee4a74 0x75dba7 0x75f8ab 0x760b49 0xecd88e 0xbe2d6e 0x1a341d2 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1

*spamApTask3: Jan 17 12:15:37.146: #LOG-3-Q_IND: rmgr_utils.c:259 Ping response from 10.128.10.128 is invalid. Incorrect checksum.[...It occurred 3 times.!]

*rmgrTrasport: Jan 17 12:15:37.145: #RMGR-3-INVALID_PING_RESPONSE: rmgr_utils.c:259 Ping response from 10.128.10.128 is invalid. Incorrect checksum.

*rmgrMain: Jan 17 12:15:37.139: #OSAPI-5-MUTEX_UNLOCK_FAILED: osapi_sem.c:1036 Failed to release a mutual exclusion object. invalid(NULL) pointer passed.

-Traceback: 0xbd9a5d 0xc12468 0xc1ecb2 0xc17cf4 0xbe2d6e 0x1a341d2 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1

*rmgrMain: Jan 17 12:15:37.139: #OSAPI-4-MUTEX_LOCK_FAILED: osapi_sem.c:1179 Failed to acquire a mutual exclusion object. invalid(NULL) pointer passed.

-Traceback: 0xbd9dd8 0xc123ad 0xc1ecb2 0xc17cf4 0xbe2d6e 0x1a341d2 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1

*rmgrMain: Jan 17 12:15:37.139: #LOG-4-Q_IND: osapi_msgq.c:520 Failed to send a message to the message queue object: CDP-Q . enqueue failed.[...It occurred 8 times.!]

*spamApTask1: Jan 17 12:15:37.107: #OSAPI-4-MSGQ_SEND_FAILED: osapi_msgq.c:520 Failed to send a message to the message queue object: CDP-Q . enqueue failed.

-Traceback: 0xbd440d 0x5bb353 0x6ecad7 0x6eeed9 0x77bfa3 0xee4a74 0x75dba7 0x75f8ab 0x760b49 0xecd88e 0xbe2d6e 0x1a341d2 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1

*rmgrTrasport: Jan 17 12:15:37.016: #RMGR-3-INVALID_PING_RESPONSE: rmgr_utils.c:259 Ping response from 10.128.10.128 is invalid. Incorrect checksum.

*rmgrTrasport: Jan 17 12:15:37.016: #RMGR-3-INVALID_PING_RESPONSE: rmgr_utils.c:259 Ping response from 10.128.133.72 is invalid. Incorrect checksum.

*rmgrMain: Jan 17 12:15:37.011: #OSAPI-5-MUTEX_UNLOCK_FAILED: osapi_sem.c:1036 Failed to release a mutual exclusion object. invalid(NULL) pointer passed.

-Traceback: 0xbd9a5d 0xc12468 0xc1ecb2 0xc17cf4 0xbe2d6e 0x1a341d2 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1

*rmgrMain: Jan 17 12:15:37.011: #OSAPI-4-MUTEX_LOCK_FAILED: osapi_sem.c:1179 Failed to acquire a mutual exclusion object. invalid(NULL) pointer passed.

-Traceback: 0xbd9dd8 0xc123ad 0xc1ecb2 0xc17cf4 0xbe2d6e 0x1a341d2 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1

*rmgrMain: Jan 17 12:15:36.974: #OSAPI-5-MUTEX_UNLOCK_FAILED: osapi_sem.c:1036 Failed to release a mutual exclusion object. invalid(NULL) pointer passed.

-Traceback: 0xbd9a5d 0xc12468 0xc1ee7c 0xc197eb 0xbe2d6e 0x1a341d2 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1

Warning: Output is too long and has been truncated

--More-- or (q)uit

*rmgrMain: Jan 17 12:15:36.974: #OSAPI-4-MUTEX_LOCK_FAILED: osapi_sem.c:1179 Failed to acquire a mutual exclusion object. invalid(NULL) pointer passed.

-Traceback: 0xbd9dd8 0xc123ad 0xc1ee7c 0xc197eb 0xbe2d6e 0x1a341d2 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1

*rmgrMain: Jan 17 12:15:36.883: #OSAPI-5-MUTEX_UNLOCK_FAILED: osapi_sem.c:1036 Failed to release a mutual exclusion object. invalid(NULL) pointer passed.

-Traceback: 0xbd9a5d 0xc12468 0xc1ecb2 0xc17cf4 0xbe2d6e 0x1a341d2 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1

*rmgrMain: Jan 17 12:15:36.883: #OSAPI-4-MUTEX_LOCK_FAILED: osapi_sem.c:1179 Failed to acquire a mutual exclusion object. invalid(NULL) pointer passed.

-Traceback: 0xbd9dd8 0xc123ad 0xc1ecb2 0xc17cf4 0xbe2d6e 0x1a341d2 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1

*rmgrMain: Jan 17 12:15:36.755: #OSAPI-5-MUTEX_UNLOCK_FAILED: osapi_sem.c:1036 Failed to release a mutual exclusion object. invalid(NULL) pointer passed.

-Traceback: 0xbd9a5d 0xc12468 0xc1ecb2 0xc17cf4 0xbe2d6e 0x1a341d2 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1

*rmgrMain: Jan 17 12:15:36.755: #OSAPI-4-MUTEX_LOCK_FAILED: osapi_sem.c:1179 Failed to acquire a mutual exclusion object. invalid(NULL) pointer passed.

-Traceback: 0xbd9dd8 0xc123ad 0xc1ecb2 0xc17cf4 0xbe2d6e 0x1a341d2 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1

*rmgrMain: Jan 17 12:15:36.627: #OSAPI-5-MUTEX_UNLOCK_FAILED: osapi_sem.c:1036 Failed to release a mutual exclusion object. invalid(NULL) pointer passed.

-Traceback: 0xbd9a5d 0xc12468 0xc1ecb2 0xc17cf4 0xbe2d6e 0x1a341d2 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1

*rmgrMain: Jan 17 12:15:36.627: #OSAPI-4-MUTEX_LOCK_FAILED: osapi_sem.c:1179 Failed to acquire a mutual exclusion object. invalid(NULL) pointer passed.

-Traceback: 0xbd9dd8 0xc123ad 0xc1ecb2 0xc17cf4 0xbe2d6e 0x1a341d2 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1

*rmgrTrasport: Jan 17 12:15:36.608: #RMGR-3-INVALID_PING_RESPONSE: rmgr_utils.c:259 Ping response from 10.128.10.128 is invalid. Incorrect checksum.

*spamApTask6: Jan 17 12:15:36.500: #OSAPI-4-MSGQ_SEND_FAILED: osapi_msgq.c:520 Failed to send a message to the message queue object: CDP-Q . enqueue failed.

-Traceback: 0xbd440d 0x5bb353 0x6ecad7 0x6eeed9 0x77bfa3 0xee4a74 0x75dba7 0x75f8ab 0x760b49 0xecd88e 0xbe2d6e 0x1a341d2 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1

*rmgrMain: Jan 17 12:15:36.499: #OSAPI-5-MUTEX_UNLOCK_FAILED: osapi_sem.c:1036 Failed to release a mutual exclusion object. invalid(NULL) pointer passed.

-Traceback: 0xbd9a5d 0xc12468 0xc1ecb2 0xc17cf4 0xbe2d6e 0x1a341d2 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1

*rmgrMain: Jan 17 12:15:36.499: #OSAPI-4-MUTEX_LOCK_FAILED: osapi_sem.c:1179 Failed to acquire a mutual exclusion object. invalid(NULL) pointer passed.

Scott Fella · ‎01-17-2014

There are a few thing I always check:

? Are there any logs on the switch or errors on the port the FlexConnect AP's are connected to

? Is the WAN latency less than 150msec

? I take a look at the up times and the ap join time vary as this can tell you a lot.

? there is a WAN latency feature available on the AP in the advanced tab.

Sent from Cisco Technical Support iPhone App

-Scott
*** Please rate helpful posts ***

dperowne · ‎01-20-2014

Thanks for the reply Scott.

I haven't checked the switches yet, but they aren't Cisco so the CLI feature set is somewhat restricted. I will do this though as I need to confirm PoE as well, although the majority of APs don't have an uptime issue, only join issue which for this bad site is as little as 16s sometimes, but often achieves 5mins and some over an hour.

The WAN latency is probably an issue. I haven't done mass assessments yet, but I ran some quick ping tests from the local authentication proxy and saw 28/50/412ms and 28/159/1296ms for 1000 ping response. I was aware of the link latency check and I will be enabling shortly on the WLCs to various APs to see how the CAPWAP is seeing things.

Will post more info soon.

**EDIT**

Here is a snippet of a report I ran on Prime:

AP Up Time CAPWAP Up Time CAPWAP Join Taken Time

2 mins 19 secs 53 secs 1 mins 25 secs

7 mins 4 secs 5 mins 38 secs 1 mins 25 secs

12 mins 29 secs 10 mins 24 secs 2 mins 4 secs

12 mins 34 secs 10 mins 22 secs 2 mins 11 secs

12 mins 35 secs 11 mins 9 secs 1 mins 25 secs

12 mins 38 secs 11 mins 8 secs 1 mins 29 secs

12 mins 40 secs 11 mins 11 secs 1 mins 28 secs

12 mins 42 secs 9 mins 6 secs 21 secs

12 mins 43 secs 7 mins 41 secs 17 secs

12 mins 44 secs 11 mins 17 secs 1 mins 26 secs

12 mins 46 secs 11 mins 19 secs 1 mins 26 secs

12 mins 48 secs 11 mins 22 secs 1 mins 25 secs

12 mins 51 secs 11 mins 24 secs 1 mins 26 secs

12 mins 51 secs 11 mins 23 secs 1 mins 27 secs

13 mins 8 secs 10 mins 24 secs 2 mins 43 secs

19 mins 19 secs 17 mins 53 secs 1 mins 25 secs

20 mins 1 secs 18 mins 29 secs 1 mins 31 secs

25 mins 0 secs 22 mins 33 secs 2 mins 26 secs

29 mins 10 secs 26 mins 45 secs 2 mins 24 secs

35 mins 27 secs 31 mins 48 secs 3 mins 38 secs

36 mins 45 secs 34 mins 21 secs 2 mins 23 secs

38 mins 17 secs 36 mins 51 secs 1 mins 25 secs

40 mins 57 secs 39 mins 29 secs 1 mins 27 secs

42 mins 28 secs 41 mins 2 secs 1 mins 25 secs

42 mins 58 secs 41 mins 29 secs 1 mins 28 secs

43 mins 29 secs 42 mins 0 secs 1 mins 28 secs

43 mins 58 secs 42 mins 32 secs 1 mins 25 secs

44 mins 42 secs 43 mins 16 secs 1 mins 25 secs

45 mins 37 secs 44 mins 5 secs 1 mins 31 secs

48 mins 9 secs 46 mins 40 secs 1 mins 28 secs

52 mins 7 secs 50 mins 38 secs 1 mins 28 secs

54 mins 0 secs 27 secs 3 mins 39 secs

54 mins 44 secs 53 mins 17 secs 1 mins 26 secs

1 hrs 9 mins 17 secs 1 hrs 6 mins 50 secs 2 mins 26 secs

1 hrs 12 mins 16 secs 1 hrs 9 mins 45 secs 2 mins 30 secs

1 hrs 18 mins 10 secs 1 hrs 16 mins 41 secs 1 mins 28 secs

1 hrs 31 mins 39 secs 1 hrs 30 mins 13 secs 1 mins 25 secs

1 hrs 37 mins 50 secs 1 hrs 36 mins 24 secs 1 mins 25 secs

2 hrs 5 mins 14 secs 2 hrs 2 mins 48 secs 2 mins 25 secs

2 hrs 5 mins 15 secs 2 hrs 1 mins 34 secs 3 mins 40 secs

2 hrs 5 mins 17 secs 2 hrs 3 mins 47 secs 1 mins 29 secs

2 hrs 5 mins 19 secs 2 hrs 3 mins 52 secs 1 mins 26 secs

2 hrs 5 mins 20 secs 2 hrs 3 mins 53 secs 1 mins 26 secs

2 hrs 5 mins 25 secs 2 hrs 1 mins 46 secs 3 mins 38 secs

2 hrs 5 mins 27 secs 2 hrs 4 mins 1 secs 1 mins 25 secs

2 hrs 5 mins 31 secs 2 hrs 4 mins 5 secs 1 mins 25 secs

8 hrs 49 mins 8 secs 8 hrs 45 mins 3 secs 17 secs

8 hrs 49 mins 11 secs 8 hrs 47 mins 43 secs 1 mins 27 secs

8 hrs 49 mins 13 secs 8 hrs 47 mins 46 secs 1 mins 26 secs

8 hrs 49 mins 16 secs 8 hrs 45 mins 41 secs 3 mins 34 secs

8 hrs 49 mins 18 secs 8 hrs 47 mins 53 secs 1 mins 24 secs

8 hrs 49 mins 19 secs 8 hrs 47 mins 54 secs 1 mins 24 secs

33 days 23 hrs 23 mins 55 secs 5 hrs 14 mins 27 secs 17 secs

33 days 23 hrs 31 mins 33 secs 11 hrs 9 mins 40 secs 1 mins 16 secs

33 days 23 hrs 39 mins 4 secs 11 hrs 8 mins 22 secs 2 mins 42 secs

33 days 23 hrs 40 mins 15 secs 11 hrs 10 mins 42 secs 20 secs

33 days 23 hrs 42 mins 16 secs 5 hrs 39 mins 26 secs 15 secs

33 days 23 hrs 47 mins 21 secs 11 hrs 9 mins 40 secs 1 mins 22 secs

33 days 23 hrs 48 mins 24 secs 11 hrs 19 mins 58 secs 4 mins 43 secs

33 days 23 hrs 51 mins 43 secs 5 hrs 14 mins 19 secs 28 secs

33 days 23 hrs 53 mins 29 secs 11 hrs 10 mins 39 secs 30 secs

33 days 23 hrs 54 mins 44 secs 5 hrs 31 mins 33 secs 1 mins 29 secs

33 days 23 hrs 57 mins 49 secs 11 hrs 10 mins 36 secs 32 secs

34 days 16 hrs 22 mins 53 secs 11 hrs 20 mins 8 secs 4 mins 26 secs

34 days 16 hrs 22 mins 59 secs 11 hrs 20 mins 17 secs 4 mins 18 secs

34 days 16 hrs 23 mins 31 secs 11 hrs 10 mins 43 secs 26 secs

34 days 16 hrs 25 mins 22 secs 11 hrs 10 mins 41 secs 25 secs

Investigating further, but it's different behaviour at different sites...

Scott Fella · ‎01-20-2014

The latency is what's the issue. Have you tried to enable link latency on these AP's. If you click on the FlexConnect AP and in the advanced tab, enable link latency and see if that helps. Your RRT times are very high and I don't know if you will ever get it working. Your only other option is to put a WLC at each site that has high RRT times. The WLC 2504 is pretty cheap and supports up to 75 AP's.

Here are the WAN requirements for FlexConnect

http://www.cisco.com/en/US/products/ps11635/products_tech_note09186a0080b7f141.shtml#wan

Sent from Cisco Technical Support iPhone App

-Scott
*** Please rate helpful posts ***

dperowne · ‎01-20-2014

Yes, I enabled link latency when I posted earlier. I'm not convinced it's running right though as while I did see 2 of the 21 APs at this site with realistic results (eg: 10/1/1200) it seems they are mostly <1ms now which I am not sure is possible. Does the counter reset when they re-associate?

So we discovered the client auth issue. There were some remote sites with the same client DHCP scope (they are local scopes NAT'd and not routed over the WAN). The WLC was seeing multiple client devices with the same IP and inducing client exclusion via IP Theft, which booted off random clients at different sites. I disabled that to ensure client authentication can continue in those instances.

So now I guess the question is, other than local gateway connectivity / DHCP loss, is there anything that would affect user traffic? ie: frequent Flex state changes? These APs are configured for local switching, and they flip to standalone mode and continue to serve clients OK because the authentication in this case is managed by a local proxy and its sends auth traffic over the Internet, rather than through the WLC, so even new clients can connect when the CAPWAP tunnel is down. Testing I've done in the lab doesn't appear to affect anything, cell phone continues to browse and download fine, so appears to be more of a management headache than service affecting?...

Scott Fella · ‎01-20-2014

Frequent joins can cause issue as a typical failure wouldn't cause flapping as much as a slow WAN. You would have to baseline this with sites that don't have such a high latency as these other sites you have. Again, Cisco recommends something like less than 300ms, anything past that, they will not support and will tell that's the reason.

Thanks,

Scott

Help out other by using the rating system and marking answered questions as "Answered"

-Scott
*** Please rate helpful posts ***

Scott Fella · ‎01-20-2014

Clients should still work fine as long as nothing happens when the AP joins and disconnects. I have seen AP's reboot when they have lost connectivity to the WLC and they were in FlexConnect. So it can be a nightmare especially with low latency WAN.

Thanks,

Scott

Help out other by using the rating system and marking answered questions as "Answered"

-Scott
*** Please rate helpful posts ***

Saravanan Lakshmanan · ‎02-20-2014

Filed below bug for this issue, please open TAC case for more info.

CSCum63628 CDP -Traceback: 0xbd440d 0x5bb353 0x6ecad7 0x6eeed9 0x77bfa3 0xee4a74..