High RSSI, High SNR, but high Data Retries rate (Multipath effect?)

Tymofii Dmytrenko · ‎02-08-2017

Hello gents,

Trying to troubleshoot a very strange issue. We are seeing voice quality issues when calls are made using wireless connection (from a corporate laptop) - sound is choppy one way, or both ways. Remote end is wired. We don't see such issues in our other offices, where we use the same type of APs (it's 3702i), or other models, such as 1142s/2600s.

I have noticed that RSSI is reported at -55dBm and SNR is at 40dBm, which is quite an awesome. However, at the same time Data Retries counter is increasing consistently.

I am trying to understand if Data Retries corresponds to Packets Received or Packets Sent. In any case, at the moment it says 875746 packets received and 1021059 packets sent, with 280174 data retries. It is very high re-transmission rate in my humble opinion. It's about 27.5% if data reties correspond to outbound direction, and even high if it's inbound - 32%.

I know it is very hard to troubleshoot multipath issues - there's no way to capture it (correct me if I am wrong). We have metal furniture here, and APs are installed on metal tiles (i.e. it's like a metal screen behind them). I believe this is causing issues to us, but I am not an expert yet to make such conclusions.

How do you identify the root cause for data retries having that RSSI and SNR values are high, there's no co-channel interference and it's 5GHz band only with 40MHz channel. Spectrum analyzer doesn't show/report anything unusual either. Feel a bit lost at the moment. Shouldn't multipath have a constructive effect on MIMO systems (or is it not always the case?).

Scott Fella · ‎02-08-2017

You always try to place AP's in areas that will give you the best coverage and not impact wireless. There are best practices in regards to metal tiles or near beams, but you can't always avoid them. MIMO helps but it's not a feature that will eliminate multi paths issues. One thing to look at is your QoS and or the channel utilization from the AP: show AP auto-RF 802.11a <AP name> and see what the utilization is. the signal you have is indeed good and retries is normal on wireless:) That information is from the AP and if you wanted to see retries on the client end, you would need some tool that can gather that from the device.

What type of voice are you using that is choppy and is it all the time, even after hours? What code are you running, controller model, AP? Test also with a basic SSID with no features enabled for comparison.

-Scott

*** Please rate helpful posts ***

-Scott
*** Please rate helpful posts ***

Tymofii Dmytrenko · ‎02-08-2017

Hi Scott

Appreciate your prompt replies. I know MIMO is not a panacea but when you read about Spatial Streams, MRC, Beamforming and stuff like that you pretty much begin to believe it's all magic.

I am 100% sure QoS is configured properly, we use classic QoS model (Bronze/Silver/Gold/Platinum). WMM set to Allowed on this WLAN and level is set to Platinum. This profile is configured to mark voice using dot1q CoS using values up to 5. APs are FlexConnect and this WLAN is locally switched - hence APs trust CoS (and not DSCP). When I check the switchport on Cat3850, I see the following

Ingress COS0 171833495     0
Ingress COS1 0             0
Ingress COS2 0             0
Ingress COS3 1270506       0
Ingress COS4 21025         0
Ingress COS5 4595053       0
Ingress COS6 0             0
Ingress COS7 0             0

All three correspond to Signaling, Video and Voice traffic.

It is Lync voice, so we had to adjust QoS policy using GPO and we apply non-standard DSCP values, such as 49,41 and 33 for voice,video and signaling. This is because the way Windows maps DSCP into WMM (that is, it only uses IP Precedence and directly maps it to WMM). Again, I am more than confident QoS works as expected because my routers report EF for voice packets and CS3 for Signaling and it's pretty much trust based model (if wireless frame has invalid WMM, then it won't be correct DSCP).

It is something else. Here's what my current AP reports from the RF perspective:

(Cisco Controller) >show AP auto-RF 802.11a GBMAN1APF09-10

Number Of Slots.................................. 2
AP Name.......................................... GBMAN1APF09-10
MAC Address...................................... 00:d7:8f:1c:68:a4
  Slot ID........................................ 1
  Radio Type..................................... RADIO_TYPE_80211a
  Sub-band Type.................................. All
  Noise Information
    Noise Profile................................ PASSED
    Channel 36...................................  -92 dBm
    Channel 40...................................  -95 dBm
    Channel 44...................................  -96 dBm
    Channel 48...................................  -97 dBm
    Channel 52...................................  -97 dBm
    Channel 56...................................  -95 dBm
    Channel 60...................................  -95 dBm
    Channel 64...................................  -96 dBm
    Channel 100..................................  -95 dBm
    Channel 104..................................  -97 dBm
    Channel 108..................................  -98 dBm
    Channel 112..................................  -96 dBm
    Channel 116..................................  -97 dBm
    Channel 132..................................  -95 dBm
    Channel 136..................................  -95 dBm
    Channel 140..................................  -95 dBm
  Interference Information
    Interference Profile......................... PASSED
    Channel 36................................... -128 dBm @  0 % busy
    Channel 40................................... -128 dBm @  0 % busy
    Channel 44................................... -128 dBm @  0 % busy
    Channel 48................................... -128 dBm @  0 % busy
    Channel 52................................... -128 dBm @  0 % busy
    Channel 56................................... -128 dBm @  0 % busy
    Channel 60................................... -128 dBm @  0 % busy
    Channel 64................................... -128 dBm @  0 % busy
    Channel 100.................................. -128 dBm @  0 % busy
    Channel 104.................................. -128 dBm @  0 % busy
    Channel 108.................................. -128 dBm @  0 % busy
    Channel 112.................................. -128 dBm @  0 % busy
    Channel 116.................................. -128 dBm @  0 % busy
    Channel 132.................................. -128 dBm @  0 % busy
    Channel 136.................................. -128 dBm @  0 % busy
    Channel 140.................................. -128 dBm @  0 % busy
    Rogue Histogram (20/40/80/160)
    .............................................
    Channel 36...................................  1/ 0/ 0/ 0
    Channel 40...................................  0/ 0/ 0/ 0
    Channel 44...................................  0/ 0/ 0/ 0
    Channel 48...................................  0/ 1/ 0/ 0
    Channel 52...................................  0/ 0/ 0/ 0
    Channel 56...................................  0/ 1/ 0/ 0
    Channel 60...................................  0/ 0/ 0/ 0
    Channel 64...................................  0/ 0/ 0/ 0
    Channel 100..................................  0/ 0/ 0/ 0
    Channel 104..................................  0/ 0/ 0/ 0
    Channel 108..................................  0/ 0/ 0/ 0
    Channel 112..................................  0/ 0/ 0/ 0
    Channel 116..................................  0/ 0/ 0/ 0
    Channel 132..................................  0/ 0/ 0/ 0
    Channel 136..................................  0/ 0/ 0/ 0
    Channel 140..................................  0/ 0/ 0/ 0
  Load Information
    Load Profile................................. FAILED
    Receive Utilization.......................... 0 %
    Transmit Utilization......................... 0 %
    Channel Utilization.......................... 1 %
    Attached Clients............................. 17 clients
  Coverage Information
    Coverage Profile............................. PASSED
    Failed Clients............................... 0 clients
  Client Signal Strengths
    RSSI -100 dbm................................ 0 clients
    RSSI  -92 dbm................................ 0 clients
    RSSI  -84 dbm................................ 0 clients
    RSSI  -76 dbm................................ 0 clients
    RSSI  -68 dbm................................ 3 clients
    RSSI  -60 dbm................................ 6 clients
    RSSI  -52 dbm................................ 7 clients
  Client Signal To Noise Ratios
    SNR    0 dB.................................. 0 clients
    SNR    5 dB.................................. 0 clients
    SNR   10 dB.................................. 0 clients
    SNR   15 dB.................................. 0 clients
    SNR   20 dB.................................. 0 clients
    SNR   25 dB.................................. 1 clients
    SNR   30 dB.................................. 3 clients
    SNR   35 dB.................................. 2 clients
    SNR   40 dB.................................. 6 clients
    SNR   45 dB.................................. 4 clients
  Nearby APs
    AP 00:a2:ee:b4:5d:40 slot 1..................  -84 dBm on  36  40MHz (10.132.10.199)  GBMAN1APF09-15
    AP 00:a2:ee:b4:5e:00 slot 1..................  -44 dBm on 132  40MHz (10.132.10.199)  GBMAN1APF09-11
    AP 00:a2:ee:b4:6d:d0 slot 1..................  -52 dBm on 112  40MHz (10.132.10.199)  GBMAN1APF09-12
    AP 00:a2:ee:b4:6f:a0 slot 1..................  -55 dBm on  56  40MHz (10.132.10.199)  GBMAN1APF09-08
    AP 00:a2:ee:b4:73:50 slot 1..................  -68 dBm on  64  40MHz (10.132.10.199)  GBMAN1APF09-07
    AP 00:a2:ee:b5:f3:b0 slot 1..................  -78 dBm on 112  40MHz (10.132.10.199)  GBMAN1APF09-06
    AP 00:a2:ee:b6:59:d0 slot 1..................  -84 dBm on  56  40MHz (10.132.10.199)  GBMAN1APF09-04
    AP 00:a2:ee:b6:f7:d0 slot 1..................  -60 dBm on 108  20MHz (10.132.10.199)  GBMAN1APF09-09
    AP 00:a2:ee:b6:fc:20 slot 1..................  -88 dBm on  52  40MHz (10.132.10.199)  GBMAN1APF09-16
    AP 00:a2:ee:b6:fd:80 slot 1..................  -70 dBm on  64  40MHz (10.132.10.199)  GBMAN1APF09-14
    AP 00:d7:8f:28:8f:f0 slot 1..................  -81 dBm on  44  40MHz (10.132.10.199)  GBMAN1APF09-05
    AP 00:d7:8f:2e:35:e0 slot 1..................  -59 dBm on 100  40MHz (10.132.10.199)  GBMAN1APF09-13
  Radar Information
  Channel Assignment Information
    Current Channel Average Energy...............  -70 dBm
    Previous Channel Average Energy..............  -70 dBm
    Channel Change Count......................... 170
    Last Channel Change Time..................... Tue Feb  7 11:43:34 2017
    Recommended Best Channel..................... 36
  RF Parameter Recommendations
    Power Level.................................. 4
    RTS/CTS Threshold............................ 2347
    Fragmentation Threshold...................... 2346
    Antenna Pattern.............................. 0
  Persistent Interference Devices
  Class Type                 Channel  DC (%%)  RSSI (dBm)  Last Update Time
  -------------------------  -------  ------  ----------  ------------------------
  All third party trademarks are the property of their respective owners.

Even though it says load profile has failed, I believe it's only informational because the real load on AP is very low. We had to move the majority of clients to wired connections purely because of this. So, connected clients are phones and tablets with little traffic.

I know retries are normal on wireless... but not at 30% rate? Also, do you know if Data Retries counter applies to Retries from the client, or to the client (is it TX or RX from AP perspective)? According to WLC help, it is from client. But how does it know? If client has to resend the packet X times, AP will be unaware about it (hence it doesn't send ACK).

Here's the client details, just in case:

(Cisco Controller) >show client detail 4c:34:88:41:28:41
Client MAC Address............................... 4c:34:88:41:28:41
Client Username ................................. <myusername>
AP MAC Address................................... 00:d7:8f:1f:a9:b0
AP Name.......................................... GBMAN1APF09-10
AP radio slot Id................................. 1
Client State..................................... Associated
Client User Group................................ <myusername>
Client NAC OOB State............................. Access
Wireless LAN Id.................................. 1
Wireless LAN Network Name (SSID)................. Internal-Wireless
Wireless LAN Profile Name........................ Internal-Wireless
Hotspot (802.11u)................................ Not Supported
BSSID............................................ 00:d7:8f:1f:a9:bf
Connected For ................................... 1694 secs
Channel.......................................... 36
IP Address....................................... X.X.X.X
Gateway Address.................................. X.X.X.X
Netmask.......................................... 255.255.255.0
Association Id................................... 10
Authentication Algorithm......................... Open System
Reason Code...................................... 1
Status Code...................................... 0
Client CCX version............................... 4
Client E2E version............................... 1
Re-Authentication Timeout........................ 27082
QoS Level........................................ Platinum
Avg data Rate.................................... 0
Burst data Rate.................................. 0
Avg Real time data Rate.......................... 0
Burst Real Time data Rate........................ 0
802.1P Priority Tag.............................. 5
CTS Security Group Tag........................... Not Applicable
KTS CAC Capability............................... No
Qos Map Capability............................... No
WMM Support...................................... Enabled
  APSD ACs.......................................  BK  BE  VI  VO
Current Rate..................................... m9 ss2
Supported Rates.................................. 12.0,18.0,24.0,36.0,48.0,
    ............................................. 54.0
Mobility State................................... Local
Mobility Move Count.............................. 0
Security Policy Completed........................ Yes
Policy Manager State............................. RUN
Audit Session ID................................. 0a840ac700002c67589addbb
AAA Role Type.................................... none
Local Policy Applied............................. none
IPv4 ACL Name.................................... none
FlexConnect ACL Applied Status................... Unavailable
IPv4 ACL Applied Status.......................... Unavailable
IPv6 ACL Name.................................... none
IPv6 ACL Applied Status.......................... Unavailable
Layer2 ACL Name.................................. none
Layer2 ACL Applied Status........................ Unavailable
Client Type...................................... SimpleIP
mDNS Status...................................... Disabled
mDNS Profile Name................................ none
No. of mDNS Services Advertised.................. 0
Policy Type...................................... WPA2
Authentication Key Management.................... 802.1x
Encryption Cipher................................ CCMP (AES)
Protected Management Frame ...................... No
Management Frame Protection...................... No
EAP Type......................................... PEAP
FlexConnect Data Switching....................... Local
FlexConnect Dhcp Status.......................... Local
FlexConnect Vlan Based Central Switching......... No
FlexConnect Authentication....................... Central
FlexConnect Central Association.................. No
FlexConnect VLAN NAME............................ corporate
Quarantine VLAN.................................. 0
Access VLAN...................................... 170
Local Bridging VLAN.............................. 170
Client Capabilities:
      CF Pollable................................ Not implemented
      CF Poll Request............................ Not implemented
      Short Preamble............................. Not implemented
      PBCC....................................... Not implemented
      Channel Agility............................ Not implemented
      Listen Interval............................ 225
      Fast BSS Transition........................ Not implemented
      11v BSS Transition......................... Not implemented
Client Wifi Direct Capabilities:
      WFD capable................................ No
      Manged WFD capable......................... No
      Cross Connection Capable................... No
      Support Concurrent Operation............... No
Fast BSS Transition Details:
Client Statistics:
      Number of Bytes Received................... 125793263
      Number of Bytes Sent....................... 321844304
      Total Number of Bytes Sent................. 321844304
      Total Number of Bytes Recv................. 125793263
      Number of Bytes Sent (last 90s)............ 458787
      Number of Bytes Recv (last 90s)............ 76035
      Number of Packets Received................. 461756
      Number of Packets Sent..................... 798942
      Number of Interim-Update Sent.............. 0
      Number of EAP Id Request Msg Timeouts...... 0
      Number of EAP Id Request Msg Failures...... 0
      Number of EAP Request Msg Timeouts......... 0
      Number of EAP Request Msg Failures......... 0
      Number of EAP Key Msg Timeouts............. 0
      Number of EAP Key Msg Failures............. 0
      Number of Data Retries..................... 141892
      Number of RTS Retries...................... 0
      Number of Duplicate Received Packets....... 445
      Number of Decrypt Failed Packets........... 0
      Number of Mic Failured Packets............. 0
      Number of Mic Missing Packets.............. 0
      Number of RA Packets Dropped............... 0
      Number of Policy Errors.................... 0
      Radio Signal Strength Indicator............ -60 dBm
      Signal to Noise Ratio...................... 33 dB
Client Rate Limiting Statistics:
      Number of Data Packets Received............ 0
      Number of Data Rx Packets Dropped.......... 0
      Number of Data Bytes Received.............. 0
      Number of Data Rx Bytes Dropped............ 0
      Number of Realtime Packets Received........ 0
      Number of Realtime Rx Packets Dropped...... 0
      Number of Realtime Bytes Received.......... 0
      Number of Realtime Rx Bytes Dropped........ 0
      Number of Data Packets Sent................ 0
      Number of Data Tx Packets Dropped.......... 0
      Number of Data Bytes Sent.................. 0
      Number of Data Tx Bytes Dropped............ 0
      Number of Realtime Packets Sent............ 0
      Number of Realtime Tx Packets Dropped...... 0
      Number of Realtime Bytes Sent.............. 0
      Number of Realtime Tx Bytes Dropped........ 0
Nearby AP Statistics:
      GBMAN1APF09-15(slot 1)
        antenna0: 21639 secs ago................. -84 dBm
        antenna1: 21639 secs ago................. -83 dBm
      GBMAN1APF09-11(slot 1)
        antenna0: 0 secs ago..................... -72 dBm
        antenna1: 0 secs ago..................... -75 dBm
      GBMAN1APF09-12(slot 1)
        antenna0: 0 secs ago..................... -91 dBm
        antenna1: 0 secs ago..................... -87 dBm
      GBMAN1APF09-08(slot 1)
        antenna0: 1 secs ago..................... -83 dBm
        antenna1: 1 secs ago..................... -83 dBm
      GBMAN1APF09-07(slot 1)
        antenna0: 3061 secs ago.................. -90 dBm
        antenna1: 3061 secs ago.................. -88 dBm
      GBMAN1APF09-06(slot 1)
        antenna0: 9156 secs ago.................. -90 dBm
        antenna1: 9156 secs ago.................. -91 dBm
      GBMAN1APF09-04(slot 1)
        antenna0: 21652 secs ago................. -72 dBm
        antenna1: 21652 secs ago................. -78 dBm
      GBMAN1APF09-09(slot 1)
        antenna0: 0 secs ago..................... -91 dBm
        antenna1: 0 secs ago..................... -90 dBm
      GBMAN1APF09-14(slot 0)
        antenna0: 3 secs ago..................... -89 dBm
        antenna1: 3 secs ago..................... -88 dBm
      GBMAN1APF09-14(slot 1)
        antenna0: 3061 secs ago.................. -92 dBm
        antenna1: 3061 secs ago.................. -95 dBm
      GBMAN1APF09-10(slot 0)
        antenna0: 2 secs ago..................... -55 dBm
        antenna1: 2 secs ago..................... -54 dBm
      GBMAN1APF09-10(slot 1)
        antenna0: 1 secs ago..................... -66 dBm
        antenna1: 1 secs ago..................... -61 dBm
      GBMAN1APF09-13(slot 1)
        antenna0: 10 secs ago.................... -91 dBm
        antenna1: 10 secs ago.................... -82 dBm
DNS Server details:
      DNS server IP ............................. X.X.X.X
      DNS server IP ............................. X.X.X.X
Assisted Roaming Prediction List details:


 Client Dhcp Required:     True
Allowed (URL)IP Addresses
-------------------------

AVC Profile Name: ............................... none

Let me know if you need WLAN details.

The call can be choppy, sometimes it's not. I have spectrum analyzer running all the time and it doesn't report any problems. Channel doesn't seem to be busy.

What I've noticed is that I probably have an asymmetric power issue. My laptop's driver was set to Highest Transmit Power (Intel) and when I check Statistics using Intel PROSet Wireless Tool, I can see that all Transmit packets go out using the highest possible MCS rate with both available SS, i.e. 2x80, but on Receiving end packets were spread across multiple MCS rates and highest one is never used... so I changed power on the driver to Medium-low (God knows what is this in dBm) and now I see that Transmit packets are also using different MCS. In my understanding, asymmetric power is BAD because basically... my laptop is capable to send all frames using MAX MCS ad AP will receive them, but not only this AP, but other APs as well should they end up on the same channel. But, I am not an expert to decide what value is the right one. I understand it has to match AP settings... but probably have to go to Intel to find out what are corresponding dBm values.

Although, not sure if asymmetric power can cause quality issues. In any case - AP and client report high quality of connection. I am literally 2.5m away from AP - it sits above me :) But, all tiles are metal - hence... it's like a screen/mirror at the back of it, and my laptop sits in a metal locker (where PMs decided to install docking stations) - I have that one open, but I have a mirror just below my laptop :)

I will try to do a number of tests tonight when office will be empty and I'll post it here.

Again, thank you very much for your time. Highly appreciate it.

Code is 8.2.141.0, WLC is 5508, APs are 3702i, FlexConnect, locally switched WLAN (WPA2/dot1x) with session timeout set to 28800s and idle timeout 10m. In fact, here's WLAN config

WLAN Identifier.................................. 1
Profile Name..................................... Internal-Wireless
Network Name (SSID).............................. Internal-Wireless
Status........................................... Enabled
MAC Filtering.................................... Disabled
Broadcast SSID................................... Enabled
AAA Policy Override.............................. Disabled
Network Admission Control
Client Profiling Status
Radius Profiling ............................ Enabled
DHCP ....................................... Enabled
HTTP ....................................... Enabled
Local Profiling ............................. Enabled
DHCP ....................................... Enabled
HTTP ....................................... Enabled
Radius-NAC State............................... Disabled
SNMP-NAC State................................. Disabled
Quarantine VLAN................................ 0
Maximum number of Associated Clients............. 0
Maximum number of Clients per AP Radio........... 200
ATF Policy....................................... 0
Number of Active Clients......................... 32
Exclusionlist Timeout............................ 60 seconds
Session Timeout.................................. 28800 seconds
User Idle Timeout................................ Disabled
Sleep Client..................................... disable
Sleep Client Timeout............................. 720 minutes
User Idle Threshold.............................. 0 Bytes
NAS-identifier................................... none
CHD per WLAN..................................... Enabled
Webauth DHCP exclusion........................... Disabled
Interface........................................ corporate
Multicast Interface.............................. Not Configured
WLAN IPv4 ACL.................................... unconfigured
WLAN IPv6 ACL.................................... unconfigured
WLAN Layer2 ACL.................................. unconfigured
mDNS Status...................................... Disabled
mDNS Profile Name................................ unconfigured
DHCP Server...................................... Default
DHCP Address Assignment Required................. Enabled
Static IP client tunneling....................... Disabled
Tunnel Profile................................... Unconfigured
PMIPv6 Mobility Type............................. none
PMIPv6 MAG Profile........................... Unconfigured
PMIPv6 Default Realm......................... Unconfigured
PMIPv6 NAI Type.............................. Hexadecimal
PMIPv6 MAG location.......................... AP
Quality of Service............................... Platinum
Per-SSID Rate Limits............................. Upstream Downstream
Average Data Rate................................ 0 0
Average Realtime Data Rate....................... 0 0
Burst Data Rate.................................. 0 0
Burst Realtime Data Rate......................... 0 0
Per-Client Rate Limits........................... Upstream Downstream
Average Data Rate................................ 0 0
Average Realtime Data Rate....................... 0 0
Burst Data Rate.................................. 0 0
Burst Realtime Data Rate......................... 0 0
Scan Defer Priority.............................. 4,5,6
Scan Defer Time.................................. 100 milliseconds
WMM.............................................. Allowed
WMM UAPSD Compliant Client Support............... Disabled
Media Stream Multicast-direct.................... Disabled
CCX - AironetIe Support.......................... Disabled
CCX - Gratuitous ProbeResponse (GPR)............. Disabled
CCX - Diagnostics Channel Capability............. Disabled
Dot11-Phone Mode (7920).......................... Disabled
Wired Protocol................................... 802.1P (Tag=5)
Passive Client Feature........................... Disabled
Peer-to-Peer Blocking Action..................... Disabled
Radio Policy..................................... 802.11a only
DTIM period for 802.11a radio.................... 1
DTIM period for 802.11b radio.................... 1
Radius Servers
Authentication................................ X.X.X.X *
Authentication................................ X.X.X.X *
Accounting.................................... X.X.X.X *
Accounting.................................... X.X.X.X *
Interim Update............................. Enabled
Interim Update Interval.................... 0
Framed IPv6 Acct AVP ...................... Prefix
Dynamic Interface............................. Disabled
Dynamic Interface Priority.................... wlan
Local EAP Authentication......................... Disabled
Radius NAI-Realm................................. Disabled
Mu-Mimo.......................................... Enabled
Security

802.11 Authentication:........................ Open System
FT Support.................................... Disabled
Static WEP Keys............................... Disabled
802.1X........................................ Disabled
Wi-Fi Protected Access (WPA/WPA2)............. Enabled
WPA (SSN IE)............................... Disabled
WPA2 (RSN IE).............................. Enabled
TKIP Cipher............................. Disabled
AES Cipher.............................. Enabled
OSEN IE.................................... Disabled
Auth Key Management
802.1x.................................. Enabled
PSK..................................... Disabled
CCKM.................................... Disabled
FT-1X(802.11r).......................... Disabled
FT-PSK(802.11r)......................... Disabled
PMF-1X(802.11w)......................... Disabled
PMF-PSK(802.11w)........................ Disabled
OSEN-1X................................. Disabled
FT Reassociation Timeout................... 20
FT Over-The-DS mode........................ Enabled
GTK Randomization.......................... Disabled
SKC Cache Support.......................... Disabled
CCKM TSF Tolerance......................... 1000
Wi-Fi Direct policy configured................ Disabled
EAP-Passthrough............................... Disabled
CKIP ......................................... Disabled
Web Based Authentication...................... Disabled
Web Authentication Timeout.................... 300
Web-Passthrough............................... Disabled
Mac-auth-server............................... 0.0.0.0
Web-portal-server............................. 0.0.0.0
Conditional Web Redirect...................... Disabled
Splash-Page Web Redirect...................... Disabled
Auto Anchor................................... Disabled
FlexConnect Local Switching................... Enabled
FlexConnect Central Association............... Disabled
flexconnect Central Dhcp Flag................. Disabled
flexconnect nat-pat Flag...................... Disabled
flexconnect Dns Override Flag................. Disabled
flexconnect PPPoE pass-through................ Disabled
flexconnect local-switching IP-source-guar.... Disabled
FlexConnect Vlan based Central Switching ..... Disabled
FlexConnect Local Authentication.............. Disabled
FlexConnect Learn IP Address.................. Enabled
Client MFP.................................... Optional
PMF........................................... Disabled
PMF Association Comeback Time................. 1
PMF SA Query RetryTimeout..................... 200
Tkip MIC Countermeasure Hold-down Timer....... 60
Eap-params.................................... Disabled
AVC Visibilty.................................... Enabled
AVC Profile Name................................. None
Flex Avc Profile Name............................ None
Flow Monitor Name................................ None
Split Tunnel Configuration
Split Tunnel................................. Disabled
Call Snooping.................................... Disabled
Roamed Call Re-Anchor Policy..................... Disabled
SIP CAC Fail Send-486-Busy Policy................ Enabled
SIP CAC Fail Send Dis-Association Policy......... Disabled
KTS based CAC Policy............................. Disabled
Assisted Roaming Prediction Optimization......... Disabled
802.11k Neighbor List............................ Disabled
802.11k Neighbor List Dual Band.................. Disabled
802.11v Directed Multicast Service............... Disabled
802.11v BSS Max Idle Service..................... Enabled
802.11v BSS Transition Service................... Disabled
802.11v BSS Transition Disassoc Imminent......... Disabled
802.11v BSS Transition Disassoc Timer............ 200
802.11v BSS Transition OpRoam Disassoc Timer..... 40
DMS DB is empty
Band Select...................................... Disabled
Load Balancing................................... Disabled
Multicast Buffer................................. Disabled
Universal Ap Admin............................... Disabled

Mobility Anchor List
WLAN ID IP Address Status Priority
------- --------------- ------ --------

802.11u........................................ Disabled

MSAP Services.................................. Disabled

Local Policy
----------------
Priority Policy Name
-------- ---------------

Lync State ...................................... Disabled
Audio QoS Policy................................. Silver
Video QoS Policy................................. Silver
App-Share QoS Policy............................. Silver
File Transfer QoS Policy......................... Silver

As you can see it's pretty simple

APs are grouped into AP group and use RF Profiles.

Scott Fella · ‎02-08-2017

On your wlan I see interim update enabled, verify that this is not checked. Also I do see that your retried are high, have you updated the drivers on the domain machines? What you see from the client detail is what the AP sees not what the client sees also.

-Scott

*** Please rate helpful posts ***

-Scott
*** Please rate helpful posts ***

Tymofii Dmytrenko · ‎02-08-2017

Thanks Scott

I am struggling to find any detailed information on this interim update feature? Is it simple a RADIUS accounting messages that are being sent periodically? Why do you think they may impact the voice quality?

I haven't upgraded my drivers, in fact, I have downgraded them. When I was using the latest driver I had significantly more Limited Connectivity issues (with yellow triangle on the connection bar). I have started to look at different drivers and after watching George's "lessons learned" video where he has mentioned exactly same problem and that they didn't have issues on 18.20.0.9 I've installed it ans since then I am on this version. It works OK. I can try to re-install the latest driver to see how it goes.

My colleague has a more recent version of drivers, but he also has a high retry rate in this office.

So, the data retries on the AP (client statistics) is how many times AP had to retransmit packets. And if I want to see a re-transmission rate, then I simply divide data retries by packet transferred, is it correct?

Scott Fella · ‎02-09-2017

Glad things are better!

Interim update can cause issue with client dropping when using 802.1x due to the value that was set by default. It's like a best practice thing for me and one of the things I look for in the wlan settings.

The ping latency is what I also look at when users complain. When it jumps around like that and then at times you get request timeouts, it's a driver issue. In regards to user complaints that it's slow and you also notice ping latency, check if the connection is on the 2.4ghz vs 5ghz. Driver issues can also be when you are channel bonding and when connected on the 2.4ghz everything is stable but not on the 5ghz if using channel bonding. I then would set the client adapter to 20mhz only and only use 5ghz to see if that fixed the issue. This way you are only troubleshooting one band at a time and if a driver upgrade/downgrade doesn't help, maybe setting the 5ghz on the client or wireless environment to 20mhz is your temp fix.

Latest drivers are not always the best also so you need to play around with different ones. I just ran into user with a new Yoga laptop with an Intel 8620 and came with the latest driver 19.x. They couldn't connect at all to any of our SSID's even an open one. But they can connect in sites that had Aruba and also from my cell phone which I tested. Downgrading to 18.40.x fixed their issues in our Cisco sites.

Also, the DCA, I typically set that to 24hours and anchor that at 0 midnight. EDRRM I will leave at default also as I want to prevent channel changes from happening. Always look at the logs also in case of radar events as that will cause issues if your also using DFS channels.

Band Select is something I use more now due to all the RF noise in the 2.4ghz even for guests. All it takes is one user who knows someone and things get escalated:) The one thing I play with is the cycle count. I have set this as high as 7 to get the distribution of devices more on the 5ghz than 2.4ghz because users will indeed have a better experience. I have played around with the other settings but in isolated sites, cycle count of 7 is being used in many of our high density sites with good results.

All this stuff to be honest is lessons learned. When supporting operations one thing to understand is what the user are saying. It's not always the truth, but when you figure out the issue and then later down the road another user(s) from another site complains about the same thing, then troubleshooting is easier as you know where to start. There are things that are stuck in my head from deployment of wireless with Apple devices, Surface laptops, iPhones and iPads, users complaint about having to log onto guest all the time, etc. Keep good notes for yourself and always ask users for feedback. Don't always assume things are working fine. When this happens, users will connect to the wired network and or tether off their phone. I have seen this happen a lot.

-Scott

*** Please rate helpful posts ***

-Scott
*** Please rate helpful posts ***

Tymofii Dmytrenko · ‎02-15-2017

Hi Scott

Thank you very much for a detailed response. Apologies for becoming silent - spent few days away from this cursed site, but now I am back here... we have our pilot group doubled now and so far, wireless network is stable.

I have my corporate WLAN advertised on 5GHz only and this, I believe, removed some stress. We have very low interference on our bonded 40MHz channels and very low utilization (but that's because everyone's wired).

Remember I told you about fluctuated latency? Well... I had this again and it was pain in the ass. Spent two days to get to the bottom of that, only to realize it was Ekahau Site Survey that was causing this! I had it running in background (to analyze surveys that we've done), but apparently, even if you use it for planning only, it scans on background - that was the source of massive latency fluctuations, massive amount of full scans of my NIC and root cause for the voice quality issues on my PC in particular. Once I close it (or disable Passive scanning) everything comes back to normal instantly. Crap... I was about to quit and look for another job :) I have twitted on Ekahau page and this ended up to be a feature request.

From what you say, I do agree that the whole wireless subject is based on lessons learned and personal perception. Theory and practice don't work same way as they do for wired networks. I am going to meet Peter Mackenzie (CWNE#33) in a couple of weeks to discuss wireless troubleshooting topic in general and see how Omnipliance can help us with similar cases.

Going to apply some changes tonight, such as disable interim update on (guest and byod networks - even though those are not dot1x we still use RADIUS for accounting); will enable DCA to run every 24h and will enable ban select on Guest and BYOD networks... want to see how clients will become distributed tomorrow.

You said you enable Band Select. How does it work with network where you have Voice? Config guide says it is not recommended to use Band Select on VoIP WLANs.

Highly appreciate all your input so far.

Scott Fella · ‎02-15-2017

Well if you are using only 5Ghz for your corporate SSID, then band select doesn't do anything for you. Here at MS, we need to keep 2.4ghz enabled because we have locations around the world in which there might be 2.4ghz only devices. In this scenario we use band select to "persuade" the clients that are dual band to associate to the 5ghz. This isn't fool proof as there are devices that are dual band that will always try to connect to the 2.4ghz. The more devices we can get on the 5ghz, the less issues or complaints from users. We also use band select of guest as this also eliminates complaints which is typically slowness when on the 2.4ghz.

Glad you found the issue with Ekahau:) We use both Ekahau and AirMagnet as its preference on the engineer but that is one thing I have learned in the past, shut down the tool before doing any other testing even if you run multiple NICs.

Your doing a good job at discovering the issue and making changes to determine if the change helps or not. This will always be the ongoing process:) One thing to take out of this is that from my consulting days and having deployed wireless in many different environments, you learn what devices require certain configs or RF tweaks to get them to work. We just purchased a bunch of AirCheck G2's for all our IT field engineer around the world so that we can see real time data collection when any issues arise. We are also looking at some tools and possible in house tools to pull data from certain controllers/AP's to help look at real time data (maybe a little delay) so during an event we can look at utilization, clients per radio/AP so we can make changes on the fly. Having feet on the ground to observe user experience is also important as tools only give you environmental details as user experience varies.

-Scott

*** Please rate helpful posts ***

-Scott
*** Please rate helpful posts ***

Tymofii Dmytrenko · ‎02-15-2017

I was planning to enable Band Select on Dual Band WLANs of course :) Not on the corporate one. Haven't heard anything about AirCheck G2, will check it later.

I was trialing Omnipeek and surprised what it can do as compared to Wireshark. It's an awesome product.

One more question I wanted to ask... do you configure a lower limit for TCP? What I found is... if I leave it to default (-10dBm), it decreases AP's power to very low level - i.e. Level 6.. which is equal to 5dBm maybe. Users directly below the AP don't get high RSSI, maybe -60dBm. I know some people do limit it to make sure TPC doesn't shrink cells too much, let's say 7dBm (so the lowest will be 8dBm in my case).

Scott Fella · ‎02-15-2017

I always configure the max and min tx power along with data rates, never do I leave it default. I tweak both these to determine where devices can associate and to help roam. The G2 is a nice handheld to give to engineers whom maybe don't have wireless experience because you can view data saved at a later time. It also allows you to VNC to the device as long as the device management port is active on a wired connection. I have used a small Ethernet bridge to make it portable that connects to guest network:)

ommnipeek is a good tool. We have that along with the NETSCOUT OneTouch G2 which also is a larger handheld that captures wired/wireless. I know some have used the Cisco AP's as sniffer and spectrum analyzer so there are tools already available but having additional tools can help. The only reason I did go with these tools is because how many sites we have local in Redmond that I can just jump in a car and troubleshoot. The OneTouch we have modules to also test cable and sniff wired traffic inline. Basically the OneTouch has all the bells and whistles of the AirCheck but does a lot more.

I know some have also used InSSIDer and various other tools that are free or low cost.

Download the CiscoWireless here is the one for iPhone/iPad

Cisco Wireless by Cisco
https://appsto.re/us/x7P87.i

There is one for android also. I use this to locate AP's I'm on and you also have he RF Dashboard that is on v8.0 and later.

-Scott

*** Please rate helpful posts ***

-Scott
*** Please rate helpful posts ***

Tymofii Dmytrenko · ‎02-15-2017

When you say you always fine tune Min/Max power, I suppose you are still referring to TPC limits? You don't use static Tx Power assignment, don't you? Tell me you don't... :) I can't imagine assigning power manually. Have you had to assign static Tx power to a selection of APs previously?

Thanks

Scott Fella · ‎02-15-2017

Max/Min can only be set on the TPC. The only time I would hard set a power would be outdoor.

-Scott

*** Please rate helpful posts ***

-Scott
*** Please rate helpful posts ***

Freerk Terpstra · ‎03-15-2017

Hi Tymofii,

In most of my designs I work with 25mW as output power on the access-point while aiming for a -65 dBm RSSI on the client for both the primary and secondary access-point. Based on this I expect the WLC to come up with power-levels somewhere around "3" to match up with the design. Due to the nature of the RRM algorithm (aiming for the third strongest neighbor) I had to use static power planning multiple times in warehouses and (large) open areas. In all of the cases you have to perform a post-deployment site survey and sometimes multiple times to get it right.

It is best practice to restart the RRM algorithm when access-points are added to your deployment or just every 6 months or so. Currently the only way to do this is on the cli with "config 802.11a/b channel global restart"; reloading the WLC does not do the trick (except for the first boot after a software upgrade).

I have to say that you ask good questions and I hope a lot of people will find this topic and learn from all the valuable information which Scot and you have discussed!

Please rate useful posts... :-)

Tymofii Dmytrenko · ‎03-16-2017

Thanks Freerk for the feedback

We've done a lot of changes recently and I was aggressively learning about everything in the wireless world :) Scott has helped massively and I think we've got to the point where our network is now stable. Although we do receive some complaints, now we think it is client related (not network).

On a side note... due to all these issues and lack of any troubleshooting tools, our management has arranged everything we need now - Omnipeek Enterprise, IxChariot, Omnipliance WiFi, Chanalyzer w/ Cisco Spectrum Expert integration, ESS PRO.

So, that was positive from a negative....

I've been told many people still assign channels statically and don't trust RRM... From your post, I see you are trying to follow the same model? RRM every 6 months is pretty much static?

I also see that WLC doesn't use all channels in 5GHz band, we have overlapping APs on Ch44 for example, while Ch132 is not in use at all (and many others). Not even used by neighbors... So, in this case static assignment can give me a better plan.

Scott Fella · ‎03-16-2017

I don't static any AP's except for outdoor and MESH. The issue you might be running into is the driver version on the Intel. v19 is very buggy in our environment and v18.40 is very stable. I wouldn't worry about overlapping channels as much is it's not all over the place. Look at channel utilization and noise floor to verify if that's an issue.

-Scott

*** Please rate helpful posts ***

-Scott
*** Please rate helpful posts ***