7.0.250.0 massively drops connections

[Bruno Dinis]

Hello.

The other day I've been surprised with massevely dropped connections in my 802.11g network 

http://s18.postimg.org/6hyju1ig9/image.png and seems to me it would be a controller problem

I've got 2 WLC, one running 7.6.130.0(wlc1) and another running legacy AP's with 7.0.250.0 (wlc2).

I've got 300 AP's in the WLC2, and these are all 802.11g enabled.

When I was trying to figure out what's wrong between 13:30 (1.30PM) and 14:00 (2PM) in my Syslog I was flooded with following messages.

[root@syslog 2015]# (summary of syslog entries for the day)
      6 APF-1-CONFLICT_IN_ASS_REQ: apf_80211.c
    149 APF-1-PROC_RSN_WARP_IE_FAILED: apf_80211.c
     52 APF-1-UNABLE_TO_CONTAIN_ROGUE: apf_rogue.c
  75871 APF-3-CHECK_SUPP_RATES_FAILED: apf_utils.c
     10 APF-3-MOBILE_ANNOUNCE_CLSN: apf_mm.c
     34 APF-3-USER_DEL_FAILED: apf_ms.c
      4 CAPWAP-3-ECHO_ERR: capwap_ac_sm.c
    108 DOT1X-3-AAA_AUTH_SEND_FAIL: 1x_aaa.c
    908 DOT1X-3-ABORT_AUTH: 1x_bauth_sm.c
    293 DOT1X-3-AUTHKEY_TX_TRANS_ERR: 1x_kxsm.c
    549 DOT1X-3-INVALID_REPLAY_CTR: 1x_eapkey.c
     21 DOT1X-3-INVALID_WPA_KEY_MSG: 1x_eapkey.c
    344 DOT1X-3-INVALID_WPA_KEY_MSG_STATE: 1x_eapkey.c
      3 DOT1X-3-INVALID_WPA_KEY_STATE: 1x_eapkey.c
   3132 DOT1X-3-MAX_EAPOL_KEY_RETRANS: 1x_ptsm.c
   3468 DOT1X-3-MAX_EAP_RETRANS: 1x_ptsm.c
  25826 DOT1X-3-MAX_EAP_RETRIES: 1x_auth_pae.c
    514 DOT1X-3-WPA_SEND_STATE_ERR: 1x_kxsm.c
      6 Feb 10 14:07
     55 LOG-1-Q_IND: apf_80211.c
      8 LOG-3-Q_IND: 1x_eapkey.c
  49001 LOG-3-Q_IND: rrmTables.c
     43 LOG-3-Q_IND: spam_lrad.c
      2 LWAPP-1-AP_CONTAINED: spam_lrad.c
   2339 LWAPP-3-REPLAY_ERR: spam_lrad.c
      2 OSAPI-2-TASK_PRIORITY_INFO: osapi_task.c
      5 RRM-3-RRM_LOGMSG: rrmChanUtils.c
  49172 RRM-3-RRM_LOGMSG: rrmTables.c
     36 SIM-3-DHCP_SERVER_NO_REPLY: sim_interface.

these messages are only for the day that occured the error, and seems to me that the RRM are flooding my syslog message, because I've got almost 100k messages about RRM.

I've been comparing with my WLC1, and the RRM settings are the same for both controllers.

I didn't have power problems in the WLC2 and I'm not sure what's causing the problem..

[George Stefanick]

When I have a failure like this I normally rush to check the PEM state of the clients that failed. Do you happen to see what that was ? 

[Bruno Dinis]

I didn't check PEM state of any client because I was away from work. But was approximately 1500 clients dropped.

[George Stefanick]

The drop happen at 1:30 and 30 minutes later they all came back online? Was anything done to get them back online? Was it just the 802.11g network, not 802.11a ?

[Bruno Dinis]

nothing was done, because I was away and I've noticed that by night.

It was just the 802.11g network (almost my entire LAPs in WLC2 7.0.250.0) the 802.11a/n/ac are on the main WLC1 and nothing was wrong with it...

It's odd i've been checking the WLC2 and 50k RRM messages on logging, but I'm not sure how to workaround the issue or if these logs have something to do with the network break down.