Client connection Issue with 702W AP on 8.5.131.0

[charlietweet]

We are seeing clients having random and unpredictable connection issues in housing facilities where we have Cisco 702Ws.  We have debugs and a TAC case open, but it's been slow trying to determine root cause via those methods.  We are running on 8540s and WiSM2s with Cisco controller code:  8.5.131.0.  This sounds buggy, but we are not seeing this as a bug from Cisco and TAC has not been able to provide any analysis.  Here is the "debug mac" of one client seeing the issue (the specific debugs immediately after successful authentication completes and the problem begins):

*apfReceiveTask: Sep 07 14:39:02.898: 0c:54:15:49:34:9b 0.0.0.0
DHCP_REQD (7) DHCP Policy timeout. Number of DHCP Discover 0, DHCP Request 0 from client
*apfReceiveTask: Sep 07 14:39:02.898: 0c:54:15:49:34:9b Interface Group was NULL.Number of DHCP Discovery 0 from client
*apfReceiveTask: Sep 07 14:39:02.898: 0c:54:15:49:34:9b 0.0.0.0 DHCP_REQD (7) Pem timed out, Try to delete client in 10 secs.
*apfReceiveTask: Sep 07 14:39:02.898: 0c:54:15:49:34:9b Scheduling deletion of Mobile Station: (callerId: 12) in 10 seconds

[mikevinson]

We have been trying to deal with this issue as student have been calling in.  Have been hesitant of pre-emptive rebooting, but this starting to sound like a viable option now.

I'm curious if see if you upgrade to 8.5.135 resolves this issue... but as you stated you will at least get a reboot of the AP's!  Upgrading to 8.5.135 without knowing if this fixes the issue is a tough one for us to justify.  I hope if you upgrade, to hear good news and it resolves this problem.

[albertfandango]

---

@mikevinson wrote:

I'm curious if see if you upgrade to 8.5.135 resolves this issue... but as you stated you will at least get a reboot of the AP's!  Upgrading to 8.5.135 without knowing if this fixes the issue is a tough one for us to justify.  I hope if you upgrade, to hear good news and it resolves this problem.

---

Unfortunately have had to postpone the software upgrade due to more pressing issues elsewhere which need addressed, but I'll let you know as soon as I've completed the work. 

[albertfandango]

Some additional information on our own experience with timings: in one specific instance where resetting the AP (19th September) resolved the fault for the student's devices connecting to a 702W in his room, the student reported the problem reoccurred on September 30th.

Resetting the specific AP again resolved the problem.

[mikevinson]

We upgraded to 8.5.135.0 on the morning of Sept 30th based on TACs recommendation.  We were receiving more and more complaints of connectivity issues and needed to show our residence hall management that something was being done.  I'm not holding my breath that this will be the fix but at least we had a complete reboot of all 702W APs in our student housing, as this seems to fix the issue temporarily. 

I will update later with how this code train behaves, but it may be a few weeks before we might hear of anything from students.  I'm not planning on not closing my TAC case until I feel comfortable that this is our solution.

[mikevinson]

After our upgrade, we seemed to have some minor relief, but the connectivity issues have been coming back.  Again a reboot of the AP seems to fix the issue (at least temporarily).

Sending more information to TAC and asking how to proceed.

[albertfandango]

We've had a few more pressing issues to deal with so this problem has been sidelined, however our current workaround seems to be the least worst option; Prime template scheduling a reload of all our 702Ws at 5:30am once a week. This will do pending a fix for CSCvi84734.

[d.friday]

I don't know your exact issue, but Cisco has a bug fix for the 702w APs and it will be release with the next version. it should be released in the next couple of weeks.

CSCvi42919 Cisco 702w AP - Radio resets with ath_ACIF_radio_dead message

[Tristan G]

Hi,

We've hit several issues with the 702W relating to key corruption that results in these symptoms.   These fixes are being integrated for us into an engineering release. These only affect 802.1X/WPA2 enterprise networks.  We first saw this when moving to an 8.3.x release and was not present on 8.0.x which we moved from.

Have you logged a TAC case and if so, could you please share the SR so we can work together to ensure that we're hitting the same issue?

Cheers,

Tristan

[charlietweet]

Thanks Tristan.  FYI, we escalated our TAC case, but we still have not received any official update with a root cause or recommendation.  Our case number is:  685097396

[Ben_Idaho]

The University of Idaho is also having this exact issue.  We have 700+ 702w access points.  Our case number is 685307840.

[rusca]

I can confirm that this issue also happens on 8.5.120.0. We ran this version before moving to 8.5.131 and both versions had the same issue on the 702W.

[Ben_Idaho]

The symptoms of bug CSCvi84734 seem to mostly fit what we are seeing.  We have provided a significant amount of information and debugs to TAC and they are still analyzing it.  We do not have a confirmed bug assigned to the case yet.

[Nik_TU]

We are seeing the exact same issues at our university campus as well. We have about 1K 702Ws all in residence halls. We are running 8.5.131.0 code on 8540 controllers. Opened a TAC case last week and we have been referred to https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvi84734/?reffering_site=dumpcr but still need to provide logs for confirmation.

[charlietweet]

We were told by our SE to escalate the TAC case to Sev2 in an effort to get a new Cisco employee to look at the issue.  He confirmed the bug-id is CSCvi84734, but still was not able to provide more detail as to when a code fix will be released.  At this point, we are considering whether or not to downgrade or keep rebooting the 702's on a daily basis as each complaint comes in.  We have found the reboot of the AP seems to resolve all issues for clients on that AP for at least a week or so.   

[paddy.grattansmith]

Can confirm that University of New South Wales is having issues with 700(702i/702w) series and 8.5.135.0.