09-04-2015 08:33 PM - edited 07-05-2021 03:52 AM
One of my first 8510 deployments went horribly wrong over last couple of days. For 6 weeks, had stability. This week, suddenly ping times to the controller- any interface- is on the order of hundreds of milliseconds and user traffic reflects the same latency. Failing over to SSO partner makes no difference. Removing all but one uplink makes no difference. If I shut down uplinks momentarily and restore them, ping times go back to back to 1ms as they should be, over hours as box pair gets busier the latency comes back, and 5508s on same networks with identical uplinks trunks dont have anything like this happening. 8510s are coasting easy on CPU and memory.
I DO NOT have lag set up on these, but TAC said it shouldn't matter. I have another pair with LAG configured that doesn't show same behavior. I plan on doing LAG, but the behavior just seems buggy. An 8.0.110.9- escalation build for 5508s random reboot.
Any ever seen this sort of condition? Is definitely only between our firectly connected Nexus 7K and the 8510 pair, and not a matter of traffic as the 5508s on same subnet have no similar problems. USing Cisco's own SFPs.
Solved! Go to Solution.
09-05-2015 08:40 PM
I haven't see it so far and also have a pair of 8510s in HA-SKU connected to a Nexus 7K. Currently they only have single 10Gb interface each connected.
That setup is running 8.0.110.0.Have you monitored the 7k ports to see if they are being saturated?
09-05-2015 08:40 PM
I haven't see it so far and also have a pair of 8510s in HA-SKU connected to a Nexus 7K. Currently they only have single 10Gb interface each connected.
That setup is running 8.0.110.0.Have you monitored the 7k ports to see if they are being saturated?
09-10-2015 07:38 AM
Ignore that Correct Answer thing.
The 7K ports are nowhere near saturation. And- msglogs on 8510 show nothing of interest. No memory or CPU issues shown on 8510 during condition. After pulling a few hundred APs off to the 5508s, the 8510 seemed to settle right down. So the mystery remains, and I will be adding LAG early next week.
09-10-2015 07:47 AM
Also working with TAC now- they are leaning toward an AVC bug with high client counts
09-10-2015 08:26 AM
Have you enabled AVC, Does the issue go away once AVC disabled on WLC.
09-10-2015 08:32 AM
We are a university and absolutely rely on AVC, so yes- it is enabled. I can't simply disable it until an outage window because it is where we police P2P traffic.
09-10-2015 08:40 AM
#AVC is done at WLC on AireOS until 8.0 unlike 8.1 and Converged access is on AP(not MA).
http://www.cisco.com/c/en/us/td/docs/wireless/controller/release/notes/crn81.html
This release extends the AVC functionality from Cisco WLC to the AP.
09-10-2015 08:43 AM
Thanks, Saravana. I'm not following the difference between 8.0 and 8.1 with AVC- if it's not done on WLC, where is it done? (AP?) Also not understanding the converged access statement.
-Lee
09-10-2015 08:58 AM
On 8.1, They've moved the AVC filter/policy to the edge ie., on AP so that WLC doesn't have to handle it, Basically not putting pressure on WLC and let the individual APs to handle the AVC stuff.
Converged Access controllers handles AVC filter/policy at AP from day 1.
09-10-2015 12:59 PM
We have growing circumstantial evidence that AVC is impacting performance on both 5508s and 8510s running 8.0.110.9. This code stuff is utterly maddening on big networks.
09-10-2015 01:53 PM
Please touch base with TAC/BU for fix/workaround/code update suggestions.
Also, to stay on top, signup with BU/TME to receive pre-fcs image on that code family to pilot/test the features you mostly use to avoid similar situation when expected to migrate. If no time/resource/infrastructure available then try forward the top feature usage and how it is used on your setup to BU to include this in their test bed.
09-07-2016 03:19 AM
I know this is an old tread.
But is there a BugID for this bug ?
I have the same issue at a customer with 25K to 30K wireless clients on a 8510, when AVC is enabled the performance drops to below half in certain periods.
They are running 8.0.133
09-18-2017 12:13 AM
Ping ?
09-18-2017 02:55 AM
09-18-2017 03:04 AM
Right now we have the issue on 8.2.151 around 3K APs, and about 22K Clients on 8510.
When AVC is enabled, and when we reach about >2Gbit bandwith through the WLC, we start to get very high latency from Wireless clients to the rest of the net, and from wired management station to WLC management interface (200ms+). performance tests from wireless clients at this point also drops to around 5 to 10Mbit.
When we disable AVC, bandwith increases to what we would expect, 4Gbit+, and latency from wired management station to WLC goes down to >1ms. Wireless performance tests increases to a more expected 200Mbit+.
Im really setting my eyes on BugID: CSCuz10099
I have an open TAC case, and is looking forward to more from them.
/Thomas
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide