Wireless guests unable to pass traffic

[kev-matthews]

Hi Folks,

This is a bit of an odd one that got me scratching my head, so hopefully the collective can help!

We have a wireless deployment that spans multiple sites, the WLC's are located in Central Data Centres and we provide two SSIDs, one for our corporate users who breakout via H-REAP into a local VLAN and a guest one, which breaks out via a mobility Anchor in one of the Data Centres.  Auth for the guests is done via Web on the Mobility Anchor and due to some issues we had with DHCP allocation we allocate Guest IP addresses from a Router located in the DMZ that the Anchor sits in before it hits the internet.

No AP's are registered to the Anchor Controller, they are distributed between the other 6 WLC's (5 of which not shown in the diagram)

I've attached a diagram to give a very high level of our setup.

The problem is as follows:

Guests authenticate fine, they're passing traffic, but after a period of time they suddenly can't get internet access anymore.  They are not prompted for re-auth, nor are they disconnected from the AP itself.  The client retains all of its DHCP settings.  The user has to disconnect and reconnect to the Guest Wireless in order to continue being able to access the internet.

I managed to replicate this on my iPad yesterday at a different site, and I found that sometimes the connection would 'hang' and then come back to life and at other times I would need to reconnect to the guest network, though I was not prompted to re-auth.

The software version that we're using on all the 5508 controllers is 7.0.116.0 which we upgraded to last week, as we suffered from the loss of vlan tag issues in 7.0.98.0, but from the reports I've been getting - it would appear that this particular issue was occurring pre-upgrade.

All the 5508s are anchored to the DMZ Controller and they are all members of the same mobility group.

The Timeouts for the guest wireless is set as follows:

Session Timeout: 32400 Seconds

Timeout Value (assume this is for exclusion): 60 Seconds

Are there any debugs that I can run to show what's happening to the client?  If so would these be run on one of the normal WLC's or on the Anchor?  IS there anything that I can look for from a client side?

Any pointers that you can provide would be really appreciated!

Thanks

Kev

[Surendra BG]

"sh run-config" from the WLC (Please let us know the WLAN to which we are connecting and facing the issue), Run the debug.. "debug client "

and paste it here..

Regards

Surendra

[kev-matthews]

Hi Surendra,

Which WLC should I run this from, the Anchor or the WLC that the AP is attached to (or both?)

Thanks

Kev

[Surendra BG]

Hi Kev,

Nice question!!

I prefer running on both.. however  anchor is the one which is required for sure.. in short.. please run on both..

Regards

Surendra

[kev-matthews]

Hi Surendra

I've been running some tests and I have attached the Running config from both controlers - WLC1.txt is the controller labeled WLC1 in the Diagram and WLC-MA.txt is the DMZ controller.

Debug logs were generated on both WLCs for the Client MAC (attached as debug-wlc1.txt and debug-wlc-ma.txt).  It only seemed to generate logs for the initial connection, not later on during the time of the issues.

During the tests I noticed the following client behaviour:

1. The client can be surfing the web, then all of a sudden, gets 'page cannot be found' errors, this is for any site and also for the login URL.  This initially indicated a DNS issue to me however point 3 should cover reasons why I don't think this is the case.

2. I can ping the router (by IP) in the DMZ the whole time that the problem occurs and no packets are lost.

3.  I can telnet to the router in the DMZ by IP, however when the issue occurs my connection to the router is lost.  Since it is running directly to the IP of the router (in the same IP subnet that gets allocated to the Guest Client) this is why I have ruled out DNS for the time being.

4. After a period of upto 60 seconds, the connectivity would then be restored and I would be able to carry on as normal.  This behaviour occured quite regularly (approx 10 > 15 minute intervals) during the test.

DNS is served by public servers on the internet side of the firewall (for information)

Hope that helps

Kev

[Surendra BG]

ok! i need the debug when the issue happens.. and another thing.. when the issue happens, whats the client PEM state on the ANchor WLC??

Regards

Surendra

[kev-matthews]

Hi Surendra

I was running the debug for the whole session but that was the only log that got generated (on the initial connection) - please can you advise how I gather the PEM state from the controller?

Thanks

Kev

[Surendra BG]

Hi,

PEM - On the DMZ WLC >> Monitor >> Clients >> select the client >> Here you see the PEM state.

It will like (DHCP-RQRD, 802.1X required etc)

Regards

Surendra

[Surendra BG]

btw.. i checked ur config.. just curious to know if you have configues PER USER BANDWIDTH (QOS) on the WLC??

Regards

Surendra

[kev-matthews]

Thanks - I'll setup some more tests to get the PEM state.

We've setup QOS for per-user bandwidth in the 'Bronze' class (for the guest SSID only)

Config for this QOS policy is:

All Data-Rates set to 5120 kbps

[Surendra BG · Accepted solution]

Hi,

Thanks for the update!!

This is a great case to work on if you could raise a TAC ticket!! Either you can raise one and inform then to assign this to me or i can open one and take the ownership of the SR!! lemme know how this works out for you!!

I work for APAC time zone and my working hours are from 8 PM to 2 AM US EST.

Regards

Surendra

[kev-matthews]

Thanks Surendra,

I think I'll need to raise the request through our service partners, but will let you know if I have any problems with this.

Kind Regards

Kev

[Surendra BG]

ok!! lemme know how this goes..

Regards

Surendra

[Surendra BG]

Hi,

Hope you are doing great!!

Just curious to know if i have any update? Will it be fine if you could share your email address here??

Regards

Surendra

[kev-matthews]

Hi Surendra,

Just dropped you a message with my email address

We're currently waiting for our support provider to raise the case, I'll let you know the case ref as soon as I have it (I will chase them now)

Kind regards

Kev