troubleshooting wireless AP drop outs

q-le · ‎01-31-2016

hi ALL,

We had a 5508 WLC and over 100 x 3502 APs. It ibeen working well for us but recently we have complain from some users that

at some place the wifi had become unstable( drop outs) even though there are no changes occurs.

where do I start troubleshooting ie at the WLC or the AP itself ?

I have reboot the AP but does not help.

Any suggestions are much appreciated.

Thanks

Peter

Leo Laohoo · ‎01-31-2016

where do I start troubleshooting ie at the WLC or the AP itself ?

Version of the WLC firmware?

Are the dropouts happening at 802.11a or 802.11b radios?

What kind of clients? Laptops? Tablets? Smartphones? What kind?

q-le · ‎01-31-2016

hi Leo,

Version 7.6.130.0

The clients are the laptops so i assume they use 802.11n radios ie not sure what radios they use .

P

Leo Laohoo · ‎01-31-2016

The clients are the laptops so i assume they use 802.11n radios ie not sure what radios they use

Please find out more information about the clients.

And then, pick one of the laptops, put it close to an AP (about 10 metres away) and post the output to the following commands:

1. sh client detail <laptop wireless NIC MAC address>;

Find out which AP does this wireless client associate/authenticate to and post the outputs to the next following commands:

2. sh ap auto-rf 802.11a <AP name>;

3. sh ap auto-rf 802.11b <AP name>

q-le · ‎02-02-2016

Please find outputs as below:

(Cisco Controller) >show client detail 28:b2:bd:e3:2f:1f
Client MAC Address............................... 28:b2:bd:e3:2f:1f
Client Username ................................. QLe@test.com
AP MAC Address................................... f4:0f:1b:a0:ad:50
AP Name.......................................... LAP-OTSoundControl-R
AP radio slot Id................................. 1
Client State..................................... Associated
Client NAC OOB State............................. Access
Wireless LAN Id.................................. 2
Hotspot (802.11u)................................ Not Supported
BSSID............................................ f4:0f:1b:a0:ad:5f
Connected For ................................... 630 secs
Channel.......................................... 48
IP Address....................................... 172.18.152.132
Gateway Address.................................. 172.18.152.3
Netmask.......................................... 255.255.255.0
IPv6 Address..................................... fe80::a9a4:46d8:6fe6:c012
Association Id................................... 14
Authentication Algorithm......................... Open System
Reason Code...................................... 1
Status Code...................................... 0
Client CCX version............................... 4
Client E2E version............................... 1

--More-- or (q)uit
Re-Authentication Timeout........................ 85742
QoS Level........................................ Silver
Avg data Rate.................................... 0
Burst data Rate.................................. 0
Avg Real time data Rate.......................... 0
Burst Real Time data Rate........................ 0
802.1P Priority Tag.............................. disabled
CTS Security Group Tag........................... Not Applicable
KTS CAC Capability............................... No
WMM Support...................................... Enabled
APSD ACs....................................... BK(T/D) BE(T/D) VI(T/D) VO(T/D)
Power Save....................................... ON
Current Rate..................................... m12
Supported Rates.................................. 6.0,9.0,12.0,18.0,24.0,36.0,
............................................. 48.0,54.0
Mobility State................................... Local
Mobility Move Count.............................. 0
Security Policy Completed........................ Yes
Policy Manager State............................. RUN
Policy Manager Rule Created...................... Yes
Audit Session ID................................. ac126f3c00089f4f56b14c88
AAA Role Type.................................... none
Local Policy Applied............................. none

IPv4 ACL Name.................................... none
FlexConnect ACL Applied Status................... Unavailable
IPv4 ACL Applied Status.......................... Unavailable
IPv6 ACL Name.................................... none
IPv6 ACL Applied Status.......................... Unavailable
Layer2 ACL Name.................................. none
Layer2 ACL Applied Status........................ Unavailable
Client Type...................................... SimpleIP
mDNS Status...................................... Enabled
mDNS Profile Name................................ default-mdns-profile
No. of mDNS Services Advertised.................. 0
Policy Type...................................... WPA2
Authentication Key Management.................... 802.1x
Encryption Cipher................................ CCMP (AES)
Protected Management Frame ...................... No
Management Frame Protection...................... No
EAP Type......................................... EAP-TLS
Interface........................................ wireless_soh
VLAN............................................. 152
Quarantine VLAN.................................. 0
Access VLAN...................................... 152
Client Capabilities:
CF Pollable................................ Not implemented

CF Poll Request............................ Not implemented
Short Preamble............................. Not implemented
PBCC....................................... Not implemented
Channel Agility............................ Not implemented
Listen Interval............................ 225
Fast BSS Transition........................ Not implemented
Client Wifi Direct Capabilities:
WFD capable................................ No
Manged WFD capable......................... No
Cross Connection Capable................... No
Support Concurrent Operation............... No
Fast BSS Transition Details:
Client Statistics:
Number of Bytes Received................... 2198799
Number of Bytes Sent....................... 3768125
Total Number of Bytes Sent................. 3768125
Total Number of Bytes Recv................. 2198799
Number of Bytes Sent (last 90s)............ 4932
Number of Bytes Recv (last 90s)............ 22394
Number of Packets Received................. 8713
Number of Packets Sent..................... 8481
Number of Interim-Update Sent.............. 0
Number of EAP Id Request Msg Timeouts...... 0

Number of EAP Id Request Msg Failures...... 0
Number of EAP Request Msg Timeouts......... 0
Number of EAP Request Msg Failures......... 0
Number of EAP Key Msg Timeouts............. 0
Number of EAP Key Msg Failures............. 0
Number of Data Retries..................... 3293
Number of RTS Retries...................... 0
Number of Duplicate Received Packets....... 89
Number of Decrypt Failed Packets........... 0
Number of Mic Failured Packets............. 0
Number of Mic Missing Packets.............. 0
Number of RA Packets Dropped............... 0
Number of Policy Errors.................... 0
Radio Signal Strength Indicator............ -62 dBm
Signal to Noise Ratio...................... 35 dB
Client Rate Limiting Statistics:
Number of Data Packets Recieved............ 0
Number of Data Rx Packets Dropped.......... 0
Number of Data Bytes Recieved.............. 0
Number of Data Rx Bytes Dropped............ 0
Number of Realtime Packets Recieved........ 0
Number of Realtime Rx Packets Dropped...... 0
Number of Realtime Bytes Recieved.......... 0

Number of Realtime Rx Bytes Dropped........ 0
Number of Data Packets Sent................ 0
Number of Data Tx Packets Dropped.......... 0
Number of Data Bytes Sent.................. 0
Number of Data Tx Bytes Dropped............ 0
Number of Realtime Packets Sent............ 0
Number of Realtime Tx Packets Dropped...... 0
Number of Realtime Bytes Sent.............. 0
Number of Realtime Tx Bytes Dropped........ 0
Nearby AP Statistics:
SOH-LAP-OTPromptC(slot 1)
antenna0: 42 secs ago.................... -82 dBm
antenna1: 42 secs ago.................... -88 dBm
SOH-LAP-Event3(slot 1)
antenna0: 43 secs ago.................... -90 dBm
antenna1: 43 secs ago.................... -92 dBm
SOH-LAP-OTSoundCo(slot 1)
antenna0: 43 secs ago.................... -60 dBm
antenna1: 43 secs ago.................... -58 dBm
DNS Server details:
DNS server IP ............................. 172.18.100.62
DNS server IP ............................. 172.18.100.63
Assisted Roaming Prediction List details:

(Cisco Controller) >show ap auto-rf 802.11a LAP-OTSoundControl-R

Number Of Slots.................................. 2
AP Name.......................................... LAP-OTSoundControl-R
MAC Address...................................... f4:0f:1b:9d:9c:c8
Slot ID........................................ 1
Radio Type..................................... RADIO_TYPE_80211a
Sub-band Type.................................. All
Noise Information
Noise Profile................................ PASSED
Channel 36................................... -98 dBm
Channel 40................................... -96 dBm
Channel 44................................... -98 dBm
Channel 48................................... -96 dBm
Channel 52................................... -97 dBm
Channel 56................................... -98 dBm
Channel 60................................... -96 dBm
Channel 64................................... -97 dBm
Channel 100.................................. -97 dBm
Channel 104.................................. -96 dBm
Channel 108.................................. -96 dBm
Channel 112.................................. -96 dBm
Channel 116.................................. -96 dBm
Channel 132.................................. -96 dBm

Channel 136.................................. -96 dBm
Channel 140.................................. -96 dBm
Channel 149.................................. -96 dBm
Channel 153.................................. -96 dBm
Channel 157.................................. -95 dBm
Channel 161.................................. -94 dBm
Channel 165.................................. -96 dBm
Interference Information
Interference Profile......................... PASSED
Channel 36................................... -128 dBm @ 0 % busy
Channel 40................................... -128 dBm @ 0 % busy
Channel 44................................... -128 dBm @ 0 % busy
Channel 48................................... -128 dBm @ 0 % busy
Channel 52................................... -128 dBm @ 0 % busy
Channel 56................................... -128 dBm @ 0 % busy
Channel 60................................... -128 dBm @ 0 % busy
Channel 64................................... -128 dBm @ 0 % busy
Channel 100.................................. -128 dBm @ 0 % busy
Channel 104.................................. -128 dBm @ 0 % busy
Channel 108.................................. -128 dBm @ 0 % busy
Channel 112.................................. -128 dBm @ 0 % busy
Channel 116.................................. -128 dBm @ 0 % busy
Channel 132.................................. -128 dBm @ 0 % busy

Channel 136.................................. -128 dBm @ 0 % busy
Channel 140.................................. -128 dBm @ 0 % busy
Channel 149.................................. -128 dBm @ 0 % busy
Channel 153.................................. -74 dBm @ 65 % busy
Channel 157.................................. -70 dBm @ 89 % busy
Channel 161.................................. -128 dBm @ 0 % busy
Channel 165.................................. -128 dBm @ 0 % busy
Rogue Histogram (20/40/80)
.............................................
Channel 36................................... 0/ 0/ 0
Channel 40................................... 0/ 0/ 0
Channel 44................................... 0/ 0/ 0
Channel 48................................... 0/ 1/ 0
Channel 52................................... 0/ 0/ 0
Channel 56................................... 0/ 0/ 0
Channel 60................................... 0/ 0/ 0
Channel 64................................... 0/ 0/ 0
Channel 100.................................. 0/ 0/ 0
Channel 104.................................. 0/ 0/ 0
Channel 108.................................. 0/ 0/ 0
Channel 112.................................. 0/ 0/ 0
Channel 116.................................. 0/ 0/ 0
Channel 132.................................. 0/ 0/ 0

Channel 136.................................. 0/ 0/ 0
Channel 140.................................. 0/ 0/ 0
Channel 149.................................. 0/ 0/ 1
Channel 153.................................. 0/ 1/ 1
Channel 157.................................. 0/ 1/ 1
Channel 161.................................. 0/ 0/ 1
Channel 165.................................. 0/ 0/ 0
Load Information
Load Profile................................. PASSED
Receive Utilization.......................... 0 %
Transmit Utilization......................... 0 %
Channel Utilization.......................... 2 %
Attached Clients............................. 4 clients
Coverage Information
Coverage Profile............................. PASSED
Failed Clients............................... 0 clients
Client Signal Strengths
RSSI -100 dbm................................ 0 clients
RSSI -92 dbm................................ 0 clients
RSSI -84 dbm................................ 0 clients
RSSI -76 dbm................................ 0 clients
RSSI -68 dbm................................ 2 clients
RSSI -60 dbm................................ 2 clients

RSSI -52 dbm................................ 0 clients
Client Signal To Noise Ratios
SNR 0 dB.................................. 0 clients
SNR 5 dB.................................. 0 clients
SNR 10 dB.................................. 0 clients
SNR 15 dB.................................. 0 clients
SNR 20 dB.................................. 0 clients
SNR 25 dB.................................. 0 clients
SNR 30 dB.................................. 2 clients
SNR 35 dB.................................. 2 clients
SNR 40 dB.................................. 0 clients
SNR 45 dB.................................. 0 clients
Nearby APs
AP 04:c5:a4:8d:6f:20 slot 1.................. -78 dBm on 64 (172.18.111.60)
AP 04:c5:a4:8d:7c:50 slot 1.................. -83 dBm on 36 (172.18.111.60)
Radar Information
Channel Assignment Information
Current Channel Average Energy............... -72 dBm
Previous Channel Average Energy.............. 127 dBm
Channel Change Count......................... 1
Last Channel Change Time..................... Wed Feb 3 10:49:42 2016
Recommended Best Channel..................... 48
RF Parameter Recommendations

Power Level.................................. 1
RTS/CTS Threshold............................ 2347
Fragmentation Threshold...................... 2346
Antenna Pattern.............................. 0

Persistent Interference Devices
Class Type Channel DC (%%) RSSI (dBm) Last Update Time
------------------------- ------- ------ ---------- ------------------------
All third party trademarks are the property of their respective owners.

(Cisco Controller) >show ap auto-rf 802.11a LAP-OTSoundControl-R LAP-OTSoundControl-R b LAP-OTSoundControl-R

Number Of Slots.................................. 2
AP Name..........................................LAP-OTSoundControl-R
MAC Address...................................... f4:0f:1b:9d:9c:c8
Slot ID........................................ 0
Radio Type..................................... RADIO_TYPE_80211b/g
Sub-band Type.................................. All
Noise Information
Noise Profile................................ PASSED
Channel 1.................................... -97 dBm
Channel 2.................................... -94 dBm
Channel 3.................................... -89 dBm
Channel 4.................................... -70 dBm
Channel 5.................................... -80 dBm
Channel 6.................................... -84 dBm
Channel 7.................................... -86 dBm
Channel 8.................................... -73 dBm
Channel 9.................................... -86 dBm
Channel 10................................... -91 dBm
Channel 11................................... -90 dBm
Interference Information
Interference Profile......................... PASSED

--More-- or (q)uit
Channel 1.................................... -128 dBm @ 0 % busy
Channel 2.................................... -128 dBm @ 0 % busy
Channel 3.................................... -128 dBm @ 0 % busy
Channel 4.................................... -128 dBm @ 0 % busy
Channel 5.................................... -67 dBm @ 35 % busy
Channel 6.................................... -67 dBm @ 60 % busy
Channel 7.................................... -68 dBm @ 10 % busy
Channel 8.................................... -128 dBm @ 0 % busy
Channel 9.................................... -128 dBm @ 0 % busy
Channel 10................................... -63 dBm @ 1 % busy
Channel 11................................... -128 dBm @ 0 % busy
Load Information
Load Profile................................. PASSED
Receive Utilization.......................... 0 %
Transmit Utilization......................... 1 %
Channel Utilization.......................... 17 %
Attached Clients............................. 3 clients
Coverage Information
Coverage Profile............................. PASSED
Failed Clients............................... 0 clients
Client Signal Strengths
RSSI -100 dbm................................ 0 clients
RSSI -92 dbm................................ 0 clients

--More-- or (q)uit
RSSI -84 dbm................................ 0 clients
RSSI -76 dbm................................ 1 clients
RSSI -68 dbm................................ 0 clients
RSSI -60 dbm................................ 1 clients
RSSI -52 dbm................................ 1 clients
Client Signal To Noise Ratios
SNR 0 dB.................................. 0 clients
SNR 5 dB.................................. 0 clients
SNR 10 dB.................................. 0 clients
SNR 15 dB.................................. 1 clients
SNR 20 dB.................................. 0 clients
SNR 25 dB.................................. 0 clients
SNR 30 dB.................................. 0 clients
SNR 35 dB.................................. 2 clients
SNR 40 dB.................................. 0 clients
SNR 45 dB.................................. 0 clients
Nearby APs
AP 04:c5:a4:8d:6f:20 slot 0.................. -64 dBm on 11 (172.18.111.60)
AP 04:c5:a4:8d:7b:50 slot 0.................. -80 dBm on 1 (172.18.111.60)
AP 04:c5:a4:8d:7c:50 slot 0.................. -69 dBm on 1 (172.18.111.60)
AP 10:8c:cf:11:a1:b0 slot 0.................. -63 dBm on 6 (172.18.111.60)
AP 58:f3:9c:aa:89:20 slot 0.................. -92 dBm on 6 (172.18.111.60)
AP 58:f3:9c:ad:3d:80 slot 0.................. -90 dBm on 1 (172.18.111.60)

--More-- or (q)uit
AP d0:c2:82:2d:9b:40 slot 0.................. -84 dBm on 11 (172.18.111.60)
AP f4:4e:05:c5:49:90 slot 0.................. -85 dBm on 1 (172.18.111.60)
Radar Information
Channel Assignment Information
Current Channel Average Energy............... unknown
Previous Channel Average Energy.............. unknown
Channel Change Count......................... 0
Last Channel Change Time..................... Mon Feb 1 12:08:38 2016
Recommended Best Channel..................... 1
RF Parameter Recommendations
Power Level.................................. 4
RTS/CTS Threshold............................ 2347
Fragmentation Threshold...................... 2346
Antenna Pattern.............................. 0

Persistent Interference Devices
Class Type Channel DC (%%) RSSI (dBm) Last Update Time
------------------------- ------- ------ ---------- ------------------------
All third party trademarks are the property of their respective owners.

(Cisco Controller) >show ap auto-rf 802.11b LAP-OTSoundControl-R show ap auto-rf 802.11a LAP-OTSoundControl-R show client detail 28:b2:bd:e3:2f:1f
Client MAC Address............................... 28:b2:bd:e3:2f:1f
Client Username ................................. QLe@test.com
AP MAC Address................................... f4:0f:1b:a0:ad:50
AP Name..........................................LAP-OTSoundControl-R
AP radio slot Id................................. 1
Client State..................................... Associated
Client NAC OOB State............................. Access
Wireless LAN Id.................................. 2
Hotspot (802.11u)................................ Not Supported
BSSID............................................ f4:0f:1b:a0:ad:5f
Connected For ................................... 621 secs
Channel.......................................... 48
IP Address....................................... 172.18.152.132
Gateway Address.................................. 172.18.152.3
Netmask.......................................... 255.255.255.0
IPv6 Address..................................... fe80::a9a4:46d8:6fe6:c012
Association Id................................... 14
Authentication Algorithm......................... Open System
Reason Code...................................... 1
Status Code...................................... 0
Client CCX version............................... 4
Client E2E version............................... 1

--More-- or (q)uit
Re-Authentication Timeout........................ 85752
QoS Level........................................ Silver
Avg data Rate.................................... 0
Burst data Rate.................................. 0
Avg Real time data Rate.......................... 0
Burst Real Time data Rate........................ 0
802.1P Priority Tag.............................. disabled
CTS Security Group Tag........................... Not Applicable
KTS CAC Capability............................... No
WMM Support...................................... Enabled
APSD ACs....................................... BK(T/D) BE(T/D) VI(T/D) VO(T/D)
Power Save....................................... ON
Current Rate..................................... m12
Supported Rates.................................. 6.0,9.0,12.0,18.0,24.0,36.0,
............................................. 48.0,54.0
Mobility State................................... Local
Mobility Move Count.............................. 0
Security Policy Completed........................ Yes
Policy Manager State............................. RUN
Policy Manager Rule Created...................... Yes
Audit Session ID................................. ac126f3c00089f4f56b14c88
AAA Role Type.................................... none
Local Policy Applied............................. none

--More-- or (q)uit
IPv4 ACL Name.................................... none
FlexConnect ACL Applied Status................... Unavailable
IPv4 ACL Applied Status.......................... Unavailable
IPv6 ACL Name.................................... none
IPv6 ACL Applied Status.......................... Unavailable
Layer2 ACL Name.................................. none
Layer2 ACL Applied Status........................ Unavailable
Client Type...................................... SimpleIP
mDNS Status...................................... Enabled
mDNS Profile Name................................ default-mdns-profile
No. of mDNS Services Advertised.................. 0
Policy Type...................................... WPA2
Authentication Key Management.................... 802.1x
Encryption Cipher................................ CCMP (AES)
Protected Management Frame ...................... No
Management Frame Protection...................... No
EAP Type......................................... EAP-TLS
Interface........................................ wireless_soh
VLAN............................................. 152
Quarantine VLAN.................................. 0
Access VLAN...................................... 152
Client Capabilities:
CF Pollable................................ Not implemented

--More-- or (q)uit
CF Poll Request............................ Not implemented
Short Preamble............................. Not implemented
PBCC....................................... Not implemented
Channel Agility............................ Not implemented
Listen Interval............................ 225
Fast BSS Transition........................ Not implemented
Client Wifi Direct Capabilities:
WFD capable................................ No
Manged WFD capable......................... No
Cross Connection Capable................... No
Support Concurrent Operation............... No
Fast BSS Transition Details:
Client Statistics:
Number of Bytes Received................... 2198799
Number of Bytes Sent....................... 3768125
Total Number of Bytes Sent................. 3768125
Total Number of Bytes Recv................. 2198799
Number of Bytes Sent (last 90s)............ 4932
Number of Bytes Recv (last 90s)............ 22394
Number of Packets Received................. 8713
Number of Packets Sent..................... 8481
Number of Interim-Update Sent.............. 0
Number of EAP Id Request Msg Timeouts...... 0

--More-- or (q)uit
Number of EAP Id Request Msg Failures...... 0
Number of EAP Request Msg Timeouts......... 0
Number of EAP Request Msg Failures......... 0
Number of EAP Key Msg Timeouts............. 0
Number of EAP Key Msg Failures............. 0
Number of Data Retries..................... 3293
Number of RTS Retries...................... 0
Number of Duplicate Received Packets....... 89
Number of Decrypt Failed Packets........... 0
Number of Mic Failured Packets............. 0
Number of Mic Missing Packets.............. 0
Number of RA Packets Dropped............... 0
Number of Policy Errors.................... 0
Radio Signal Strength Indicator............ -62 dBm
Signal to Noise Ratio...................... 35 dB
Client Rate Limiting Statistics:
Number of Data Packets Recieved............ 0
Number of Data Rx Packets Dropped.......... 0
Number of Data Bytes Recieved.............. 0
Number of Data Rx Bytes Dropped............ 0
Number of Realtime Packets Recieved........ 0
Number of Realtime Rx Packets Dropped...... 0
Number of Realtime Bytes Recieved.......... 0

--More-- or (q)uit
Number of Realtime Rx Bytes Dropped........ 0
Number of Data Packets Sent................ 0
Number of Data Tx Packets Dropped.......... 0
Number of Data Bytes Sent.................. 0
Number of Data Tx Bytes Dropped............ 0
Number of Realtime Packets Sent............ 0
Number of Realtime Tx Packets Dropped...... 0
Number of Realtime Bytes Sent.............. 0
Number of Realtime Tx Bytes Dropped........ 0
Nearby AP Statistics:
SOH-LAP-OTPromptC(slot 1)
antenna0: 33 secs ago.................... -82 dBm
antenna1: 33 secs ago.................... -88 dBm
SOH-LAP-Event3(slot 1)
antenna0: 34 secs ago.................... -90 dBm
antenna1: 34 secs ago.................... -92 dBm
SOH-LAP-OTSoundCo(slot 1)
antenna0: 34 secs ago.................... -60 dBm
antenna1: 34 secs ago.................... -58 dBm
DNS Server details:
DNS server IP ............................. 172.18.100.62
DNS server IP ............................. 172.18.100.63
Assisted Roaming Prediction List details:

Thanks

P

Leo Laohoo · ‎02-02-2016

I don't see anything wrong here.

RSSI is good. SNR is good. No co-channel interference.

q-le · ‎02-04-2016

Thank you.

I run the test after the event had finished so it look ok as you said.

but apparently the users had one of the Airport Express wifi running same SSID

for extra coverage - not sure if that caused issues and also how many users can

a 3702i supports ie 50 users?

I also captured the SSID ie vivace which the users used on that day and it picked up BSSID from Cisco and another one from "unknown" vendor.

Can not make sense of all this - Can you help please.

Thanks

P

Leo Laohoo · ‎02-04-2016

but apparently the users had one of the Airport Express wifi running same SSID

Can you please make sure the users complaining isn't happening when they've joined to the honeypot?

for extra coverage - not sure if that caused issues and also how many users cana 3702i supports ie 50 users?

Let's take things one step at a time.