01-17-2014 05:44 AM - edited 07-04-2021 11:59 PM
Seeing a host of issues on a large multi-site flexconnect network. It could be a number of things, such as the APs having issues with the L3 routing over WAN DSL, or WLC issues including bugs in the 7.5 code or potentially AP management issues (what is the per interface AP management limit on these new WLCs?)
There are a number of small remote sites that run 2602 APs in flexconnect mode back to the main DC with an 8510 controller (v7.5.102.0). DTLS is disabled.
Prime registers many alarms for AP association/deassociation, and when searching for a site of APs on the WLC and re-searching, I never get the same number with a few APs ghosting in and out due to this issue.
Investigation on the APs themselves shows random de-assoc and assoc events lasting 15secs to 5mins or many more minutes. No obvious pattern.
The problem it's causing is clients on the WiFi end are cut off and their authentication session is left open on the central server not allowing reconnection.
I'm still trying to figure out if this is a bug, WLC set up issue in regards to AP registration, L3 AP operation or asymmetrical routing issue, or a combination of all of the above --so I thought I'd post it on here to see if anyone had any ideas while I continue trawling through configuration and logs...
Example AP log output:
AP_04#sh log | b 13:27
*Jan 17 13:13:27.203: %CAPWAP-3-ERRORLOG: Retransmission count for packet exceeded max(CAPWAP_WTP_EVENT_REQUEST
., 123)
*Jan 17 13:13:27.207: %LWAPP-3-CLIENTERRORLOG: Switching to Standalone mode
*Jan 17 13:13:27.207: %CAPWAP-3-ERRORLOG: GOING BACK TO DISCOVER MODE
*Jan 17 13:13:27.207: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 10.90.100.100:5246
*Jan 17 13:13:27.271: %WIDS-6-DISABLED: IDS Signature is removed and disabled.
*Jan 17 13:13:31.031: %CLEANAIR-6-STATE: Slot 0 down
*Jan 17 13:13:31.031: %CLEANAIR-6-STATE: Slot 1 down
*Jan 17 13:13:37.275: %CAPWAP-3-ERRORLOG: Go join a capwap controller
*Jan 17 13:14:37.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.90.100.100 peer_port: 5246
*Jan 17 13:14:39.463: %CAPWAP-5-DTLSREQSUCC: DTLS connection created sucessfully peer_ip: 10.90.100.100 peer_port: 5246
*Jan 17 13:14:39.463: %CAPWAP-5-SENDJOIN: sending Join Request to 10.90.100.100
*Jan 17 13:14:44.463: %CAPWAP-5-SENDJOIN: sending Join Request to 10.90.100.100
*Jan 17 13:14:47.859: Starting Ethernet promiscuous mode
*Jan 17 13:14:54.743: ac_first_hop_mac - IP:10.3.122.254 Hop IP:10.3.122.254 IDB:BVI1
*Jan 17 13:14:54.759: %LWAPP-4-CLIENTEVENTLOG: OfficeExtend Localssid saved in AP flash
*Jan 17 13:14:59.155: %CAPWAP-5-JOINEDCONTROLLER: AP has joined controller SOV-1H2-WLC-02P
*Jan 17 13:15:01.327: %LWAPP-4-CLIENTEVENTLOG: No Flex ACL map configuration file to load. Connect to controller to get configuration file
*Jan 17 13:15:01.327: %LWAPP-4-CLIENTEVENTLOG: No Flex ACL map configuration file to load. Connect to controller to get configuration file
*Jan 17 13:15:01.327: %LWAPP-4-CLIENTEVENTLOG: No LS Flex ACL map configuration file to load. Connect to controller to get configuration file
*Jan 17 13:15:01.327: %LWAPP-4-CLIENTEVENTLOG: No Central Dhcp map configuration file to load. Connect to controller to get configuration file
*Jan 17 13:15:01.327: %LWAPP-3-CLIENTERRORLOG: Switching to Connected mode
*Jan 17 13:15:14.935: %WIDS-6-ENABLED: IDS Signature is loaded and enabled
*Jan 17 13:16:03.467: %CLEANAIR-6-STATE: Slot 0 enabled
*Jan 17 13:16:05.523: %CLEANAIR-6-STATE: Slot 1 enabled
*Jan 17 13:17:53.771: %CAPWAP-3-ERRORLOG: Retransmission count for packet exceeded max(CAPWAP_WTP_EVENT_REQUEST
., 56)
*Jan 17 13:17:53.775: %LWAPP-3-CLIENTERRORLOG: Switching to Standalone mode
*Jan 17 13:17:53.775: %CAPWAP-3-ERRORLOG: GOING BACK TO DISCOVER MODE
*Jan 17 13:17:53.775: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 10.90.100.100:5246
*Jan 17 13:17:55.583: %CLEANAIR-6-STATE: Slot 0 down
*Jan 17 13:17:55.583: %CLEANAIR-6-STATE: Slot 1 down
*Jan 17 13:17:55.903: %WIDS-6-DISABLED: IDS Signature is removed and disabled.
*Jan 17 13:18:05.907: %CAPWAP-3-ERRORLOG: Go join a capwap controller
*Jan 17 13:18:06.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.90.100.100 peer_port: 5246
*Jan 17 13:18:36.715: DTLS_CLIENT_ERROR: ../capwap/base_capwap/dtls/base_capwap_dtls_connection_db.c:2134 Max retransmission count reached!
*Jan 17 13:19:05.999: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 10.90.100.100:5246
*Jan 17 13:19:05.999: %CAPWAP-3-ERRORLOG: Go join a capwap controller
*Jan 17 13:18:06.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.90.100.100 peer_port: 5246
*Jan 17 13:18:12.791: %CAPWAP-5-DTLSREQSUCC: DTLS connection created sucessfully peer_ip: 10.90.100.100 peer_port: 5246
*Jan 17 13:18:12.791: %CAPWAP-5-SENDJOIN: sending Join Request to 10.90.100.100
*Jan 17 13:18:17.791: %CAPWAP-5-SENDJOIN: sending Join Request to 10.90.100.100
*Jan 17 13:18:20.663: Starting Ethernet promiscuous mode
*Jan 17 13:18:25.659: ac_first_hop_mac - IP:10.3.122.254 Hop IP:10.3.122.254 IDB:BVI1
*Jan 17 13:18:25.675: %LWAPP-4-CLIENTEVENTLOG: OfficeExtend Localssid saved in AP flash
*Jan 17 13:18:31.971: %CAPWAP-5-JOINEDCONTROLLER: AP has joined controller SOV-1H2-WLC-02P
*Jan 17 13:18:32.035: %LWAPP-4-CLIENTEVENTLOG: No Flex ACL map configuration file to load. Connect to controller to get configuration file
*Jan 17 13:18:32.035: %LWAPP-4-CLIENTEVENTLOG: No Flex ACL map configuration file to load. Connect to controller to get configuration file
*Jan 17 13:18:32.035: %LWAPP-4-CLIENTEVENTLOG: No LS Flex ACL map configuration file to load. Connect to controller to get configuration file
*Jan 17 13:18:32.035: %LWAPP-4-CLIENTEVENTLOG: No Central Dhcp map configuration file to load. Connect to controller to get configuration file
*Jan 17 13:18:32.035: %LWAPP-3-CLIENTERRORLOG: Switching to Connected mode
*Jan 17 13:18:52.251: %WIDS-6-ENABLED: IDS Signature is loaded and enabled
*Jan 17 13:19:23.167: %CLEANAIR-6-STATE: Slot 0 enabled
*Jan 17 13:19:25.227: %CLEANAIR-6-STATE: Slot 1 enabled
*Jan 17 13:21:05.763: %CAPWAP-3-ERRORLOG: Retransmission count for packet exceeded max(CAPWAP_WTP_EVENT_REQUEST
., 32)
*Jan 17 13:21:05.767: %LWAPP-3-CLIENTERRORLOG: Switching to Standalone mode
*Jan 17 13:21:05.767: %CAPWAP-3-ERRORLOG: GOING BACK TO DISCOVER MODE
*Jan 17 13:21:05.767: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 10.90.100.100:5246
*Jan 17 13:21:07.983: %WIDS-6-DISABLED: IDS Signature is removed and disabled.
*Jan 17 13:21:10.227: %CLEANAIR-6-STATE: Slot 0 down
*Jan 17 13:21:10.227: %CLEANAIR-6-STATE: Slot 1 down
*Jan 17 13:21:17.987: %CAPWAP-3-ERRORLOG: Go join a capwap controller
*Jan 17 13:22:18.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.90.100.100 peer_port: 5246
*Jan 17 13:22:32.319: %CAPWAP-5-DTLSREQSUCC: DTLS connection created sucessfully peer_ip: 10.90.100.100 peer_port: 5246
*Jan 17 13:22:32.319: %CAPWAP-5-SENDJOIN: sending Join Request to 10.90.100.100
*Jan 17 13:22:37.319: %CAPWAP-5-SENDJOIN: sending Join Request to 10.90.100.100
*Jan 17 13:22:40.851: Starting Ethernet promiscuous mode
*Jan 17 13:22:46.075: ac_first_hop_mac - IP:10.3.122.254 Hop IP:10.3.122.254 IDB:BVI1
*Jan 17 13:22:46.095: %LWAPP-4-CLIENTEVENTLOG: OfficeExtend Localssid saved in AP flash
*Jan 17 13:22:52.335: %CAPWAP-5-JOINEDCONTROLLER: AP has joined controller SOV-1H2-WLC-02P
*Jan 17 13:22:52.399: %LWAPP-4-CLIENTEVENTLOG: No Flex ACL map configuration file to load. Connect to controller to get configuration file
*Jan 17 13:22:52.399: %LWAPP-4-CLIENTEVENTLOG: No Flex ACL map configuration file to load. Connect to controller to get configuration file
*Jan 17 13:22:52.399: %LWAPP-4-CLIENTEVENTLOG: No LS Flex ACL map configuration file to load. Connect to controller to get configuration file
*Jan 17 13:22:52.399: %LWAPP-4-CLIENTEVENTLOG: No Central Dhcp map configuration file to load. Connect to controller to get configuration file
*Jan 17 13:22:52.399: %LWAPP-3-CLIENTERRORLOG: Switching to Connected mode
*Jan 17 13:23:10.835: %WIDS-6-ENABLED: IDS Signature is loaded and enabled
*Jan 17 13:23:57.883: %CLEANAIR-6-STATE: Slot 0 enabled
*Jan 17 13:23:59.943: %CLEANAIR-6-STATE: Slot 1 enabled
*Jan 17 13:25:25.531: %CAPWAP-3-ERRORLOG: Received packet with invalid sequence number: got 73 expected 74.
*Jan 17 13:25:25.531: %CAPWAP-3-ERRORLOG: Failed to handle capwap control message from controller
*Jan 17 13:25:25.531: %CAPWAP-3-ERRORLOG: Failed to process encrypted capwap packet from 10.90.100.100
*Jan 17 13:25:55.271: %CAPWAP-3-ERRORLOG: Retransmission count for packet exceeded max(CAPWAP_WTP_EVENT_REQUEST
., 42)
*Jan 17 13:25:55.275: %LWAPP-3-CLIENTERRORLOG: Switching to Standalone mode
*Jan 17 13:25:55.275: %CAPWAP-3-ERRORLOG: GOING BACK TO DISCOVER MODE
*Jan 17 13:25:55.275: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 10.90.100.100:5246
*Jan 17 13:25:57.427: %WIDS-6-DISABLED: IDS Signature is removed and disabled.
*Jan 17 13:25:59.943: %CLEANAIR-6-STATE: Slot 0 down
*Jan 17 13:25:59.943: %CLEANAIR-6-STATE: Slot 1 down
*Jan 17 13:26:07.431: %CAPWAP-3-ERRORLOG: Go join a capwap controller
*Jan 17 13:26:07.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.90.100.100 peer_port: 5246
*Jan 17 13:26:10.103: %CAPWAP-5-DTLSREQSUCC: DTLS connection created sucessfully peer_ip: 10.90.100.100 peer_port: 5246
*Jan 17 13:26:10.103: %CAPWAP-5-SENDJOIN: sending Join Request to 10.90.100.100
*Jan 17 13:26:15.103: %CAPWAP-5-SENDJOIN: sending Join Request to 10.90.100.100
*Jan 17 13:26:17.719: Starting Ethernet promiscuous mode
*Jan 17 13:26:22.715: ac_first_hop_mac - IP:10.3.122.254 Hop IP:10.3.122.254 IDB:BVI1
*Jan 17 13:26:22.735: %LWAPP-4-CLIENTEVENTLOG: OfficeExtend Localssid saved in AP flash
*Jan 17 13:26:29.047: %CAPWAP-5-JOINEDCONTROLLER: AP has joined controller SOV-1H2-WLC-02P
*Jan 17 13:26:31.255: %LWAPP-4-CLIENTEVENTLOG: No Flex ACL map configuration file to load. Connect to controller to get configuration file
*Jan 17 13:26:31.255: %LWAPP-4-CLIENTEVENTLOG: No Flex ACL map configuration file to load. Connect to controller to get configuration file
*Jan 17 13:26:31.255: %LWAPP-4-CLIENTEVENTLOG: No LS Flex ACL map configuration file to load. Connect to controller to get configuration file
*Jan 17 13:26:31.255: %LWAPP-4-CLIENTEVENTLOG: No Central Dhcp map configuration file to load. Connect to controller to get configuration file
*Jan 17 13:26:31.255: %LWAPP-3-CLIENTERRORLOG: Switching to Connected mode
*Jan 17 13:26:46.731: %WIDS-6-ENABLED: IDS Signature is loaded and enabled
*Jan 17 13:27:41.555: %CLEANAIR-6-STATE: Slot 0 enabled
*Jan 17 13:27:43.615: %CLEANAIR-6-STATE: Slot 1 enabled
AP_04#
AP Event Log from WLC:
Jan 17 11:12:06.995: %CAPWAP-5-JOINEDCONTROLLER: AP has joined controller WLC-02
Jan 17 11:12:09.111: %LWAPP-4-CLIENTEVENTLOG: No Flex ACL map configuration file to load. Connect to controller to get configur
Jan 17 11:12:09.111: %EVT-5-NTC: CAPWAP Received DELETE_BLACKLIST_ENTRY
., 16) 11:12:24.995: %CAPWAP-3-ERRORLOG: Retransmission count for packet exceeded max(UNKNOWN_MESSAGE_TYPE (11)
Jan 17 11:12:24.995: %LWAPP-3-CLIENTERRORLOG: Switching to Standalone mode
Jan 17 11:12:24.995: %CAPWAP-3-ERRORLOG: GOING BACK TO DISCOVER MODE
Jan 17 11:12:24.995: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 10.90.100.100:5246
Jan 17 11:12:35.059: %CAPWAP-3-ERRORLOG: Go join a capwap controller
Jan 17 11:12:35.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.90.100.100 peer_port: 5246
Jan 17 11:12:35.399: %CAPWAP-5-DTLSREQSUCC: DTLS connection created sucessfully peer_ip: 10.90.100.100 peer_port: 5246
Jan 17 11:12:35.403: %CAPWAP-5-SENDJOIN: sending Join Request to 10.90.100.100
Jan 17 11:12:40.399: %CAPWAP-5-SENDJOIN: sending Join Request to 10.90.100.100
Jan 17 11:12:47.251: %EVT-5-NTC: CAPWAP Received DECRYPT_ERR_REPORT_PERIOD
Jan 17 11:12:47.251: %EVT-5-NTC: CAPWAP Received DECRYPT_ERR_REPORT_PERIOD
Jan 17 11:12:47.251: %EVT-5-NTC: CAPWAP Received SPAM_DOMAIN_SECRET_PAYLOAD
Jan 17 11:12:47.251: %EVT-5-NTC: CAPWAP Received STATION_CFG_PAYLOAD
Jan 17 11:12:47.251: %EVT-5-CFG: Slot0: beacon period 100
Jan 17 11:12:47.251: %EVT-5-CFG: Slot0: dtim interval 1
Jan 17 11:12:47.251: %EVT-5-CFG: Slot0: country code GB
Jan 17 11:12:47.251: %EVT-5-CFG: Slot0: world mode 1
Jan 17 11:12:47.251: %EVT-5-CFG: Slot0: GPR period 10
Jan 17 11:12:47.251: %EVT-5-CFG: Slot0: max clients 200
Jan 17 11:12:47.251: %EVT-5-CFG: Slot0: dfs pkdetect 0
Jan 17 11:12:47.251: %EVT-5-CFG: Slot0: Vocera Fix 0
Jan 17 11:12:47.251: %EVT-5-NTC: CAPWAP Received OPERATION_RATE_SET_PAYLOAD
Jan 17 11:12:47.251: %EVT-5-CFG: Slot0: rates 8C129824 3048606C 00000000 00000000
Jan 17 11:12:47.251: %EVT-5-NTC: CAPWAP Received MULTI_DOMAIN_CAPABILITY_PAYLOAD
Jan 17 11:12:47.255: %EVT-5-NTC: CAPWAP Received PHY_TX_POWER_PAYLOAD
Jan 17 11:12:47.255: %EVT-5-NTC: slot=0 radio state 6 change
Jan 17 11:12:47.291: %EVT-5-CFG: Slot0: tx power level 1
Jan 17 11:12:47.291: %EVT-5-NTC: CAPWAP Received MAC_OPERATION_PAYLOAD
Jan 17 11:12:47.291: %EVT-5-CFG: Slot0: rts threshhold 2347
Jan 17 11:12:47.291: %EVT-5-CFG: Slot0: fragmentation threshold 2346
Jan 17 11:12:47.291: %EVT-5-NTC: CAPWAP Received SPAM_VENDOR_ID_PAYLOAD
Jan 17 11:12:47.291: %EVT-5-NTC: Vendor subtype Unknown Payload(111)
Jan 17 11:12:47.351: %EVT-5-CFG: d0: Set Extended Config: period 100, range 0,mcast_buf 0, mcast_rate 0, rxsens 0, cca 0
Jan 17 11:12:47.351: %EVT-5-NTC: CAPWAP Received ANTENNA_PAYLOAD
Jan 17 11:12:47.351: %EVT-5-CFG: Slot0: Antenna Diversity FF, Mode 3, Num antennas 4, Gain 0
Jan 17 11:12:47.351: %EVT-5-CFG: Slot0: HT Antenna Selection: Rx 0F Tx 0F
Jan 17 11:12:47.351: %EVT-5-NTC: CAPWAP Received AP_QOS_PAYLOAD
Jan 17 11:12:47.351: %EVT-5-CFG: Slot0: Change max/uni/multi priority 3/3/3 for WLAN 1
Jan 17 11:12:47.351: %EVT-5-NTC: CAPWAP Received AIRESPACE_CAPABILITY_PAYLOAD
Jan 17 11:12:47.351: %EVT-5-NTC: CAPWAP Received CHANGE_STATE_EVENT_PAYLOAD
Jan 17 11:12:55.559: %EVT-5-NTC: CAPWAP Received AP_DOT11H_PAYLOAD
Jan 17 11:12:55.559: %EVT-5-NTC: CAPWAP Received AP_DTPC_PAYLOAD
Jan 17 11:12:55.559: %EVT-5-NTC: CAPWAP Received AP_TSM_CONFIG_PAYLOAD
Jan 17 11:12:55.559: %EVT-5-CFG: Slot0: Voice metrics OFF
Jan 17 11:12:55.559: %EVT-5-NTC: CAPWAP Received AP_CAC_CONFIG_PAYLOAD
Jan 17 11:12:55.559: %EVT-5-NTC: CAPWAP Received AP_CAC_CONFIG_PAYLOAD
Jan 17 11:12:55.559: %EVT-5-NTC: CAPWAP Received SPAM_VENDOR_ID_PAYLOAD
Jan 17 11:12:55.559: %EVT-5-NTC: Vendor subtype AP_ENHANCED_CHD_CONFIG_PAYLOAD
Jan 17 11:12:55.559: %EVT-5-CFG: CHD voice Thresholds: Rssi = 80 dBm , Total = 100 pkts, Pct = 50
Jan 17 11:12:55.559: %EVT-5-CFG: CHD data Thresholds: Rssi = 80 dBm , Total = 50 pkts, Pct = 50
Jan 17 11:12:55.559: %EVT-5-CFG: CHD alarm count = 5
Jan 17 11:12:55.559: %EVT-5-NTC: CAPWAP Received SPAM_VENDOR_ID_PAYLOAD
Jan 17 11:12:55.559: %EVT-5-NTC: Vendor subtype AP_VOICE_MAC_OPT_PAYLOAD
Jan 17 11:12:55.559: %EVT-5-NTC: CAPWAP Received AP_CISCO_7920_VSIE_PAYLOAD
Jan 17 11:12:55.559: %EVT-5-NTC: CAPWAP Received SPAM_VENDOR_ID_PAYLOAD
Jan 17 11:12:55.559: %EVT-5-NTC: Vendor subtype PHY_HT_CONTROL_PAYLOAD
Jan 17 11:12:55.559: %EVT-5-NTC: CAPWAP Received SPAM_VENDOR_ID_PAYLOAD
Jan 17 11:12:55.559: %EVT-5-NTC: Vendor subtype PHY_HT_CAPABILITIES_PAYLOAD
Jan 17 11:12:55.559: %EVT-5-CFG: Slot0: mcsrates FFFFFF00 00000000 00000000 00000000
Jan 17 11:12:55.559: %EVT-5-CFG: Slot0: VHT mcsrates 00000000 00000000
Jan 17 11:12:55.559: %EVT-5-NTC: CAPWAP Received SPAM_VENDOR_ID_PAYLOAD
Jan 17 11:12:55.559: %EVT-5-NTC: Vendor subtype PHY_HT_ADDITIONAL_INFO_PAYLOAD
Jan 17 11:12:55.559: %EVT-5-CFG: Slot0: tx ampdu 0x31, tx amsdu 0x3F, guard interval 1, aggr sched rt timeout 10, rifs enable 1
Jan 17 11:12:55.559: %EVT-5-CFG: Slot1: beacon period 100
Jan 17 11:12:55.559: %EVT-5-CFG: Slot1: dtim interval 1
Jan 17 11:12:55.559: %EVT-5-CFG: Slot1: country code GB
Jan 17 11:12:55.559: %EVT-5-CFG: Slot1: world mode 1
Jan 17 11:12:55.559: %EVT-5-CFG: Slot1: GPR period 10
Jan 17 11:12:55.559: %EVT-5-CFG: Slot1: max clients 200
Jan 17 11:12:55.559: %EVT-5-CFG: Slot1: dfs pkdetect 1
Jan 17 11:12:55.559: %EVT-5-CFG: Slot1: Vocera Fix 0
Jan 17 11:12:55.559: %EVT-5-NTC: CAPWAP Received OPERATION_RATE_SET_PAYLOAD
Jan 17 11:12:55.559: %EVT-5-CFG: Slot1: rates 8C129824 3048606C 00000000 00000000
Jan 17 11:12:55.559: %EVT-5-NTC: CAPWAP Received MULTI_DOMAIN_CAPABILITY_PAYLOAD
Jan 17 11:12:55.559: %EVT-5-NTC: CAPWAP Received PHY_TX_POWER_PAYLOAD
Jan 17 11:12:55.559: %EVT-5-NTC: slot=1 radio state 6 change
Jan 17 11:12:55.571: %EVT-5-CFG: Slot1: tx power level 1
Jan 17 11:12:55.571: %EVT-5-NTC: CAPWAP Received MAC_OPERATION_PAYLOAD
Jan 17 11:12:55.571: %EVT-5-CFG: Slot1: rts threshhold 2347
Jan 17 11:12:55.571: %EVT-5-CFG: Slot1: fragmentation threshold 2346
Jan 17 11:12:55.571: %EVT-5-NTC: CAPWAP Received SPAM_VENDOR_ID_PAYLOAD
Jan 17 11:12:55.571: %EVT-5-NTC: Vendor subtype Unknown Payload(111)
Jan 17 11:12:57.771: %EVT-5-CFG: d1: Set Extended Config: period 100, range 0,mcast_buf 0, mcast_rate 0, rxsens 0, cca 0
Jan 17 11:12:57.771: %EVT-5-NTC: CAPWAP Received ANTENNA_PAYLOAD
Jan 17 11:12:57.771: %EVT-5-CFG: Slot1: Antenna Diversity FF, Mode 3, Num antennas 4, Gain 0
Jan 17 11:12:57.771: %EVT-5-CFG: Slot1: HT Antenna Selection: Rx 0F Tx 0F
Jan 17 11:12:57.771: %EVT-5-NTC: CAPWAP Received AP_QOS_PAYLOAD
Jan 17 11:12:57.771: %EVT-5-CFG: Slot1: Change max/uni/multi priority 3/3/3 for WLAN 1
Jan 17 11:12:57.771: %EVT-5-NTC: CAPWAP Received AIRESPACE_CAPABILITY_PAYLOAD
Jan 17 11:12:57.771: %EVT-5-NTC: CAPWAP Received CHANGE_STATE_EVENT_PAYLOAD
Jan 17 11:13:06.031: %EVT-5-NTC: CAPWAP Received AP_DOT11H_PAYLOAD
Jan 17 11:13:06.031: %EVT-5-NTC: CAPWAP Received AP_DTPC_PAYLOAD
Jan 17 11:13:06.031: %EVT-5-NTC: CAPWAP Received AP_TSM_CONFIG_PAYLOAD
Jan 17 11:13:06.031: %EVT-5-CFG: Slot1: Voice metrics OFF
Jan 17 11:13:06.031: %EVT-5-NTC: CAPWAP Received AP_CAC_CONFIG_PAYLOAD
Jan 17 11:13:06.031: %EVT-5-NTC: CAPWAP Received AP_CAC_CONFIG_PAYLOAD
Jan 17 11:13:06.031: %EVT-5-NTC: CAPWAP Received SPAM_VENDOR_ID_PAYLOAD
Jan 17 11:13:06.031: %EVT-5-NTC: Vendor subtype AP_ENHANCED_CHD_CONFIG_PAYLOAD
Jan 17 11:13:06.031: %EVT-5-CFG: CHD voice Thresholds: Rssi = 80 dBm , Total = 100 pkts, Pct = 50
Jan 17 11:13:06.031: %EVT-5-CFG: CHD data Thresholds: Rssi = 80 dBm , Total = 50 pkts, Pct = 50
Jan 17 11:13:06.031: %EVT-5-CFG: CHD alarm count = 5
Jan 17 11:13:06.031: %EVT-5-NTC: CAPWAP Received SPAM_VENDOR_ID_PAYLOAD
Jan 17 11:13:06.031: %EVT-5-NTC: Vendor subtype AP_VOICE_MAC_OPT_PAYLOAD
Jan 17 11:13:06.031: %EVT-5-NTC: CAPWAP Received SPAM_VENDOR_ID_PAYLOAD
Jan 17 11:13:06.031: %EVT-5-NTC: Vendor subtype PHY_HT_CONTROL_PAYLOAD
Jan 17 11:13:06.031: %EVT-5-NTC: CAPWAP Received SPAM_VENDOR_ID_PAYLOAD
Jan 17 11:13:06.031: %EVT-5-NTC: Vendor subtype PHY_HT_CAPABILITIES_PAYLOAD
Jan 17 11:13:06.031: %EVT-5-CFG: Slot1: mcsrates FFFFFF00 00000000 00000000 00000000
Jan 17 11:13:06.031: %EVT-5-CFG: Slot1: VHT mcsrates 3F000000 00000000
Jan 17 11:13:06.031: %EVT-5-NTC: CAPWAP Received SPAM_VENDOR_ID_PAYLOAD
Jan 17 11:13:06.031: %EVT-5-NTC: Vendor subtype PHY_HT_ADDITIONAL_INFO_PAYLOAD
Jan 17 11:13:06.031: %EVT-5-CFG: Slot1: tx ampdu 0x31, tx amsdu 0x3F, guard interval 1, aggr sched rt timeout 10, rifs enable 1
Jan 17 11:13:10.099: %EVT-5-NTC: Radio d0 RST 54 Flags 60109 BCN 0
Jan 17 11:13:10.191: %EVT-5-NTC: disabled beacons: d0 disc start 2 60109 0 0 0 0 0 512
Jan 17 11:13:14.091: %EVT-5-NTC: Radio d1 RST 54 Flags 60109 BCN 0
Jan 17 11:13:14.091: %EVT-5-NTC: CAPWAP Received BCAST_SSID_MODE_PAYLOAD
Jan 17 11:13:14.091: %EVT-5-NTC: CAPWAP Received AP_TIMER_PAYLOAD
Jan 17 11:13:14.091: %EVT-5-NTC: CAPWAP Received CLIENT_AUTO_HANDOFF_PAYLOAD
Jan 17 11:13:14.091: %EVT-5-NTC: CAPWAP Received SPAM_80211_MANAGEMENT_TRAFFIC_LIMIT_PAYLOAD
Jan 17 11:13:14.091: %EVT-5-NTC: disabled beacons: d1 disc start 2 60109 0 0 0 0 0 512
Jan 17 11:13:14.095: %EVT-5-NTC: CAPWAP Received MM_SWITCH_INFO_PAYLOAD
Jan 17 11:13:14.095: %EVT-5-NTC: CAPWAP Received SPAM_VENDOR_ID_PAYLOAD
Jan 17 11:13:14.095: %EVT-5-NTC: Vendor subtype AP_HOMEAP_CONFIG_PAYLOAD
Jan 17 11:13:16.351: %LWAPP-4-CLIENTEVENTLOG: OfficeExtend Localssid saved in AP flash
Jan 17 11:13:16.351: %EVT-5-NTC: CAPWAP Received RFID_TAG_STATUS_PAYLOAD
Jan 17 11:13:16.351: %EVT-5-NTC: CAPWAP Received SPAM_VENDOR_ID_PAYLOAD
Jan 17 11:13:16.351: %EVT-5-NTC: Vendor subtype MANAGER_IP_ADDR_PAYLOAD
Jan 17 11:13:16.351: %EVT-4-WRN: Unsupported Vendor SubType 19
Jan 17 11:13:16.351: %EVT-5-NTC: CAPWAP Received MESH_LSC_ENABLE_PAYLOAD
Jan 17 11:13:16.351: %EVT-5-NTC: CAPWAP Received SPAM_VENDOR_ID_PAYLOAD
Jan 17 11:13:16.351: %EVT-5-NTC: Vendor subtype AP_MFP_CONFIG_SUBTYPE
Jan 17 11:13:16.351: %EVT-5-NTC: CAPWAP Received SPAM_VENDOR_ID_PAYLOAD
Jan 17 11:13:16.351: %EVT-5-NTC: Vendor subtype AP_TCP_STATE_PAYLOAD
Jan 17 11:13:16.351: %EVT-4-WRN: Unsupported Vendor SubType 77
Jan 17 11:13:16.351: %EVT-5-NTC: CAPWAP Received E2E_DSCP_MAP_PAYLOAD
Jan 17 11:13:16.351: %EVT-5-NTC: CAPWAP Received SPAM_AP_LOG_FACILITY_PAYLOAD
Jan 17 11:13:16.351: %EVT-5-NTC: CAPWAP Received SPAM_VENDOR_ID_PAYLOAD
Jan 17 11:13:16.351: %EVT-5-NTC: Vendor subtype AGGREGATE_PROBE_INFO_PAYLOAD
Jan 17 11:13:16.351: %EVT-5-CFG: AGGREGATE_PROBE_INFO_PAYLOAD: aggrProbeDelay 62465
Jan 17 11:13:16.351: %EVT-5-NTC: CAPWAP Received SPAM_VENDOR_ID_PAYLOAD
Jan 17 11:13:16.351: %EVT-5-NTC: Vendor subtype SPAM_DOT11_PADDING_PAYLOAD
Jan 17 11:13:16.351: %EVT-5-CFG: SPAM_DOT11_PADDING: dot11PaddingStatus 0
Jan 17 11:13:16.351: %EVT-5-NTC: CAPWAP Received SPAM_VENDOR_ID_PAYLOAD
Jan 17 11:13:16.351: %EVT-5-NTC: Vendor subtype ROGUE_CONFIG_PAYLOAD
Jan 17 11:13:16.351: %EVT-5-CFG: ROGUE_CONFIG_PAYLOAD: mode: 1 report-interval: 10 min-rssi: -128 transient-interval 0
Jan 17 11:13:16.351: %EVT-5-NTC: CAPWAP Received SPAM_VENDOR_ID_PAYLOAD
Jan 17 11:13:16.351: %EVT-5-NTC: Vendor subtype AP_WMMAC_TEST_CONFIG_PAYLOAD
Jan 17 11:13:16.351: %EVT-5-NTC: CAPWAP Received AP_FIPS_PAYLOAD
Jan 17 11:13:16.351: %EVT-5-NTC: CAPWAP Received AP_FIPS_PAYLOAD
Jan 17 11:13:16.351: %EVT-5-NTC: CAPWAP Received SPAM_VENDOR_ID_PAYLOAD
Jan 17 11:13:16.351: %EVT-5-NTC: Vendor subtype Unknown Payload(119)
Jan 17 11:13:16.351: %EVT-5-NTC: CAPWAP Received SPAM_VENDOR_ID_PAYLOAD
Jan 17 11:13:16.351: %EVT-5-NTC: Vendor subtype Unknown Payload(113)
Jan 17 11:13:16.351: %EVT-5-NTC: CAPWAP Received RA_FILTERING_CFG_PAYLOAD
Jan 17 11:13:16.351: %EVT-5-CFG: Slot0: Router Advertisement filtering ENABLE
Jan 17 11:13:16.351: %EVT-5-CFG: Slot1: Router Advertisement filtering ENABLE
Jan 17 11:13:16.351: %EVT-5-NTC: CAPWAP Received SPAM_AP_LED_STATE_CONFIG_PAYLOAD
Jan 17 11:13:16.351: %EVT-5-NTC: CAPWAP Received MCAST_MGID_RANGE_PAYLOAD
Jan 17 11:13:16.351: %EVT-5-NTC: CAPWAP Received SPAM_VENDOR_ID_PAYLOAD
Jan 17 11:13:16.351: %EVT-5-NTC: Vendor subtype Unknown Payload(133)
Jan 17 11:13:16.415: %EVT-5-NTC: CAPWAP Received SPAM_VENDOR_ID_PAYLOAD
Jan 17 11:13:16.415: %EVT-5-NTC: Vendor subtype Unknown Payload(135)
Jan 17 11:13:20.471: %CAPWAP-5-JOINEDCONTROLLER: AP has joined controller WLC-02
Jan 17 11:13:22.631: %LWAPP-4-CLIENTEVENTLOG: No Flex ACL map configuration file to load. Connect to controller to get configur
WLC msglog:
Cisco Controller) >show msglog
Message Log Severity Level ...................... VERBOSE
*rmgrTrasport: Jan 17 12:15:37.146: #RMGR-3-INVALID_PING_RESPONSE: rmgr_utils.c:259 Ping response from 10.128.10.128 is invalid. Incorrect checksum.
*spamApTask3: Jan 17 12:15:37.146: #OSAPI-4-MSGQ_SEND_FAILED: osapi_msgq.c:520 Failed to send a message to the message queue object: CDP-Q . enqueue failed.
-Traceback: 0xbd440d 0x5bb353 0x6ecad7 0x6eeed9 0x77bfa3 0xee4a74 0x75dba7 0x75f8ab 0x760b49 0xecd88e 0xbe2d6e 0x1a341d2 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1
*spamApTask3: Jan 17 12:15:37.146: #LOG-3-Q_IND: rmgr_utils.c:259 Ping response from 10.128.10.128 is invalid. Incorrect checksum.[...It occurred 3 times.!]
*rmgrTrasport: Jan 17 12:15:37.145: #RMGR-3-INVALID_PING_RESPONSE: rmgr_utils.c:259 Ping response from 10.128.10.128 is invalid. Incorrect checksum.
*rmgrMain: Jan 17 12:15:37.139: #OSAPI-5-MUTEX_UNLOCK_FAILED: osapi_sem.c:1036 Failed to release a mutual exclusion object. invalid(NULL) pointer passed.
-Traceback: 0xbd9a5d 0xc12468 0xc1ecb2 0xc17cf4 0xbe2d6e 0x1a341d2 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1
*rmgrMain: Jan 17 12:15:37.139: #OSAPI-4-MUTEX_LOCK_FAILED: osapi_sem.c:1179 Failed to acquire a mutual exclusion object. invalid(NULL) pointer passed.
-Traceback: 0xbd9dd8 0xc123ad 0xc1ecb2 0xc17cf4 0xbe2d6e 0x1a341d2 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1
*rmgrMain: Jan 17 12:15:37.139: #LOG-4-Q_IND: osapi_msgq.c:520 Failed to send a message to the message queue object: CDP-Q . enqueue failed.[...It occurred 8 times.!]
*spamApTask1: Jan 17 12:15:37.107: #OSAPI-4-MSGQ_SEND_FAILED: osapi_msgq.c:520 Failed to send a message to the message queue object: CDP-Q . enqueue failed.
-Traceback: 0xbd440d 0x5bb353 0x6ecad7 0x6eeed9 0x77bfa3 0xee4a74 0x75dba7 0x75f8ab 0x760b49 0xecd88e 0xbe2d6e 0x1a341d2 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1
*rmgrTrasport: Jan 17 12:15:37.016: #RMGR-3-INVALID_PING_RESPONSE: rmgr_utils.c:259 Ping response from 10.128.10.128 is invalid. Incorrect checksum.
*rmgrTrasport: Jan 17 12:15:37.016: #RMGR-3-INVALID_PING_RESPONSE: rmgr_utils.c:259 Ping response from 10.128.133.72 is invalid. Incorrect checksum.
*rmgrMain: Jan 17 12:15:37.011: #OSAPI-5-MUTEX_UNLOCK_FAILED: osapi_sem.c:1036 Failed to release a mutual exclusion object. invalid(NULL) pointer passed.
-Traceback: 0xbd9a5d 0xc12468 0xc1ecb2 0xc17cf4 0xbe2d6e 0x1a341d2 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1
*rmgrMain: Jan 17 12:15:37.011: #OSAPI-4-MUTEX_LOCK_FAILED: osapi_sem.c:1179 Failed to acquire a mutual exclusion object. invalid(NULL) pointer passed.
-Traceback: 0xbd9dd8 0xc123ad 0xc1ecb2 0xc17cf4 0xbe2d6e 0x1a341d2 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1
*rmgrMain: Jan 17 12:15:36.974: #OSAPI-5-MUTEX_UNLOCK_FAILED: osapi_sem.c:1036 Failed to release a mutual exclusion object. invalid(NULL) pointer passed.
-Traceback: 0xbd9a5d 0xc12468 0xc1ee7c 0xc197eb 0xbe2d6e 0x1a341d2 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1
Warning: Output is too long and has been truncated
--More-- or (q)uit
*rmgrMain: Jan 17 12:15:36.974: #OSAPI-4-MUTEX_LOCK_FAILED: osapi_sem.c:1179 Failed to acquire a mutual exclusion object. invalid(NULL) pointer passed.
-Traceback: 0xbd9dd8 0xc123ad 0xc1ee7c 0xc197eb 0xbe2d6e 0x1a341d2 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1
*rmgrMain: Jan 17 12:15:36.883: #OSAPI-5-MUTEX_UNLOCK_FAILED: osapi_sem.c:1036 Failed to release a mutual exclusion object. invalid(NULL) pointer passed.
-Traceback: 0xbd9a5d 0xc12468 0xc1ecb2 0xc17cf4 0xbe2d6e 0x1a341d2 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1
*rmgrMain: Jan 17 12:15:36.883: #OSAPI-4-MUTEX_LOCK_FAILED: osapi_sem.c:1179 Failed to acquire a mutual exclusion object. invalid(NULL) pointer passed.
-Traceback: 0xbd9dd8 0xc123ad 0xc1ecb2 0xc17cf4 0xbe2d6e 0x1a341d2 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1
*rmgrMain: Jan 17 12:15:36.755: #OSAPI-5-MUTEX_UNLOCK_FAILED: osapi_sem.c:1036 Failed to release a mutual exclusion object. invalid(NULL) pointer passed.
-Traceback: 0xbd9a5d 0xc12468 0xc1ecb2 0xc17cf4 0xbe2d6e 0x1a341d2 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1
*rmgrMain: Jan 17 12:15:36.755: #OSAPI-4-MUTEX_LOCK_FAILED: osapi_sem.c:1179 Failed to acquire a mutual exclusion object. invalid(NULL) pointer passed.
-Traceback: 0xbd9dd8 0xc123ad 0xc1ecb2 0xc17cf4 0xbe2d6e 0x1a341d2 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1
*rmgrMain: Jan 17 12:15:36.627: #OSAPI-5-MUTEX_UNLOCK_FAILED: osapi_sem.c:1036 Failed to release a mutual exclusion object. invalid(NULL) pointer passed.
-Traceback: 0xbd9a5d 0xc12468 0xc1ecb2 0xc17cf4 0xbe2d6e 0x1a341d2 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1
*rmgrMain: Jan 17 12:15:36.627: #OSAPI-4-MUTEX_LOCK_FAILED: osapi_sem.c:1179 Failed to acquire a mutual exclusion object. invalid(NULL) pointer passed.
-Traceback: 0xbd9dd8 0xc123ad 0xc1ecb2 0xc17cf4 0xbe2d6e 0x1a341d2 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1
*rmgrTrasport: Jan 17 12:15:36.608: #RMGR-3-INVALID_PING_RESPONSE: rmgr_utils.c:259 Ping response from 10.128.10.128 is invalid. Incorrect checksum.
*spamApTask6: Jan 17 12:15:36.500: #OSAPI-4-MSGQ_SEND_FAILED: osapi_msgq.c:520 Failed to send a message to the message queue object: CDP-Q . enqueue failed.
-Traceback: 0xbd440d 0x5bb353 0x6ecad7 0x6eeed9 0x77bfa3 0xee4a74 0x75dba7 0x75f8ab 0x760b49 0xecd88e 0xbe2d6e 0x1a341d2 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1
*rmgrMain: Jan 17 12:15:36.499: #OSAPI-5-MUTEX_UNLOCK_FAILED: osapi_sem.c:1036 Failed to release a mutual exclusion object. invalid(NULL) pointer passed.
-Traceback: 0xbd9a5d 0xc12468 0xc1ecb2 0xc17cf4 0xbe2d6e 0x1a341d2 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1c9 0x1a8b1
*rmgrMain: Jan 17 12:15:36.499: #OSAPI-4-MUTEX_LOCK_FAILED: osapi_sem.c:1179 Failed to acquire a mutual exclusion object. invalid(NULL) pointer passed.
01-17-2014 06:09 AM
There are a few thing I always check:
? Are there any logs on the switch or errors on the port the FlexConnect AP's are connected to
? Is the WAN latency less than 150msec
? I take a look at the up times and the ap join time vary as this can tell you a lot.
? there is a WAN latency feature available on the AP in the advanced tab.
Sent from Cisco Technical Support iPhone App
01-20-2014 01:51 AM
Thanks for the reply Scott.
I haven't checked the switches yet, but they aren't Cisco so the CLI feature set is somewhat restricted. I will do this though as I need to confirm PoE as well, although the majority of APs don't have an uptime issue, only join issue which for this bad site is as little as 16s sometimes, but often achieves 5mins and some over an hour.
The WAN latency is probably an issue. I haven't done mass assessments yet, but I ran some quick ping tests from the local authentication proxy and saw 28/50/412ms and 28/159/1296ms for 1000 ping response. I was aware of the link latency check and I will be enabling shortly on the WLCs to various APs to see how the CAPWAP is seeing things.
Will post more info soon.
**EDIT**
Here is a snippet of a report I ran on Prime:
AP Up Time CAPWAP Up Time CAPWAP Join Taken Time
2 mins 19 secs 53 secs 1 mins 25 secs
7 mins 4 secs 5 mins 38 secs 1 mins 25 secs
12 mins 29 secs 10 mins 24 secs 2 mins 4 secs
12 mins 34 secs 10 mins 22 secs 2 mins 11 secs
12 mins 35 secs 11 mins 9 secs 1 mins 25 secs
12 mins 38 secs 11 mins 8 secs 1 mins 29 secs
12 mins 40 secs 11 mins 11 secs 1 mins 28 secs
12 mins 42 secs 9 mins 6 secs 21 secs
12 mins 43 secs 7 mins 41 secs 17 secs
12 mins 44 secs 11 mins 17 secs 1 mins 26 secs
12 mins 46 secs 11 mins 19 secs 1 mins 26 secs
12 mins 48 secs 11 mins 22 secs 1 mins 25 secs
12 mins 51 secs 11 mins 24 secs 1 mins 26 secs
12 mins 51 secs 11 mins 23 secs 1 mins 27 secs
13 mins 8 secs 10 mins 24 secs 2 mins 43 secs
19 mins 19 secs 17 mins 53 secs 1 mins 25 secs
20 mins 1 secs 18 mins 29 secs 1 mins 31 secs
25 mins 0 secs 22 mins 33 secs 2 mins 26 secs
29 mins 10 secs 26 mins 45 secs 2 mins 24 secs
35 mins 27 secs 31 mins 48 secs 3 mins 38 secs
36 mins 45 secs 34 mins 21 secs 2 mins 23 secs
38 mins 17 secs 36 mins 51 secs 1 mins 25 secs
40 mins 57 secs 39 mins 29 secs 1 mins 27 secs
42 mins 28 secs 41 mins 2 secs 1 mins 25 secs
42 mins 58 secs 41 mins 29 secs 1 mins 28 secs
43 mins 29 secs 42 mins 0 secs 1 mins 28 secs
43 mins 58 secs 42 mins 32 secs 1 mins 25 secs
44 mins 42 secs 43 mins 16 secs 1 mins 25 secs
45 mins 37 secs 44 mins 5 secs 1 mins 31 secs
48 mins 9 secs 46 mins 40 secs 1 mins 28 secs
52 mins 7 secs 50 mins 38 secs 1 mins 28 secs
54 mins 0 secs 27 secs 3 mins 39 secs
54 mins 44 secs 53 mins 17 secs 1 mins 26 secs
1 hrs 9 mins 17 secs 1 hrs 6 mins 50 secs 2 mins 26 secs
1 hrs 12 mins 16 secs 1 hrs 9 mins 45 secs 2 mins 30 secs
1 hrs 18 mins 10 secs 1 hrs 16 mins 41 secs 1 mins 28 secs
1 hrs 31 mins 39 secs 1 hrs 30 mins 13 secs 1 mins 25 secs
1 hrs 37 mins 50 secs 1 hrs 36 mins 24 secs 1 mins 25 secs
2 hrs 5 mins 14 secs 2 hrs 2 mins 48 secs 2 mins 25 secs
2 hrs 5 mins 15 secs 2 hrs 1 mins 34 secs 3 mins 40 secs
2 hrs 5 mins 17 secs 2 hrs 3 mins 47 secs 1 mins 29 secs
2 hrs 5 mins 19 secs 2 hrs 3 mins 52 secs 1 mins 26 secs
2 hrs 5 mins 20 secs 2 hrs 3 mins 53 secs 1 mins 26 secs
2 hrs 5 mins 20 secs 2 hrs 3 mins 53 secs 1 mins 26 secs
2 hrs 5 mins 25 secs 2 hrs 1 mins 46 secs 3 mins 38 secs
2 hrs 5 mins 27 secs 2 hrs 4 mins 1 secs 1 mins 25 secs
2 hrs 5 mins 31 secs 2 hrs 4 mins 5 secs 1 mins 25 secs
8 hrs 49 mins 8 secs 8 hrs 45 mins 3 secs 17 secs
8 hrs 49 mins 11 secs 8 hrs 47 mins 43 secs 1 mins 27 secs
8 hrs 49 mins 13 secs 8 hrs 47 mins 46 secs 1 mins 26 secs
8 hrs 49 mins 16 secs 8 hrs 45 mins 41 secs 3 mins 34 secs
8 hrs 49 mins 18 secs 8 hrs 47 mins 53 secs 1 mins 24 secs
8 hrs 49 mins 19 secs 8 hrs 47 mins 54 secs 1 mins 24 secs
33 days 23 hrs 23 mins 55 secs 5 hrs 14 mins 27 secs 17 secs
33 days 23 hrs 31 mins 33 secs 11 hrs 9 mins 40 secs 1 mins 16 secs
33 days 23 hrs 39 mins 4 secs 11 hrs 8 mins 22 secs 2 mins 42 secs
33 days 23 hrs 40 mins 15 secs 11 hrs 10 mins 42 secs 20 secs
33 days 23 hrs 42 mins 16 secs 5 hrs 39 mins 26 secs 15 secs
33 days 23 hrs 47 mins 21 secs 11 hrs 9 mins 40 secs 1 mins 22 secs
33 days 23 hrs 48 mins 24 secs 11 hrs 19 mins 58 secs 4 mins 43 secs
33 days 23 hrs 51 mins 43 secs 5 hrs 14 mins 19 secs 28 secs
33 days 23 hrs 53 mins 29 secs 11 hrs 10 mins 39 secs 30 secs
33 days 23 hrs 54 mins 44 secs 5 hrs 31 mins 33 secs 1 mins 29 secs
33 days 23 hrs 57 mins 49 secs 11 hrs 10 mins 36 secs 32 secs
34 days 16 hrs 22 mins 53 secs 11 hrs 20 mins 8 secs 4 mins 26 secs
34 days 16 hrs 22 mins 59 secs 11 hrs 20 mins 17 secs 4 mins 18 secs
34 days 16 hrs 23 mins 31 secs 11 hrs 10 mins 43 secs 26 secs
34 days 16 hrs 25 mins 22 secs 11 hrs 10 mins 41 secs 25 secs
Investigating further, but it's different behaviour at different sites...
01-20-2014 06:26 AM
The latency is what's the issue. Have you tried to enable link latency on these AP's. If you click on the FlexConnect AP and in the advanced tab, enable link latency and see if that helps. Your RRT times are very high and I don't know if you will ever get it working. Your only other option is to put a WLC at each site that has high RRT times. The WLC 2504 is pretty cheap and supports up to 75 AP's.
Here are the WAN requirements for FlexConnect
http://www.cisco.com/en/US/products/ps11635/products_tech_note09186a0080b7f141.shtml#wan
Sent from Cisco Technical Support iPhone App
01-20-2014 09:53 AM
Yes, I enabled link latency when I posted earlier. I'm not convinced it's running right though as while I did see 2 of the 21 APs at this site with realistic results (eg: 10/1/1200) it seems they are mostly <1ms now which I am not sure is possible. Does the counter reset when they re-associate?
So we discovered the client auth issue. There were some remote sites with the same client DHCP scope (they are local scopes NAT'd and not routed over the WAN). The WLC was seeing multiple client devices with the same IP and inducing client exclusion via IP Theft, which booted off random clients at different sites. I disabled that to ensure client authentication can continue in those instances.
So now I guess the question is, other than local gateway connectivity / DHCP loss, is there anything that would affect user traffic? ie: frequent Flex state changes? These APs are configured for local switching, and they flip to standalone mode and continue to serve clients OK because the authentication in this case is managed by a local proxy and its sends auth traffic over the Internet, rather than through the WLC, so even new clients can connect when the CAPWAP tunnel is down. Testing I've done in the lab doesn't appear to affect anything, cell phone continues to browse and download fine, so appears to be more of a management headache than service affecting?...
01-20-2014 06:49 PM
Frequent joins can cause issue as a typical failure wouldn't cause flapping as much as a slow WAN. You would have to baseline this with sites that don't have such a high latency as these other sites you have. Again, Cisco recommends something like less than 300ms, anything past that, they will not support and will tell that's the reason.
Thanks,
Scott
Help out other by using the rating system and marking answered questions as "Answered"
01-20-2014 09:58 AM
Clients should still work fine as long as nothing happens when the AP joins and disconnects. I have seen AP's reboot when they have lost connectivity to the WLC and they were in FlexConnect. So it can be a nightmare especially with low latency WAN.
Thanks,
Scott
Help out other by using the rating system and marking answered questions as "Answered"
02-20-2014 10:58 AM
Filed below bug for this issue, please open TAC case for more info.
CSCum63628 CDP -Traceback: 0xbd440d 0x5bb353 0x6ecad7 0x6eeed9 0x77bfa3 0xee4a74..
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide