cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
3483
Views
35
Helpful
21
Replies

8510s that slow down all L2 networks they connect to

lbadman
Level 1
Level 1

One of my first 8510 deployments went horribly wrong over last couple of days. For 6 weeks, had stability. This week, suddenly ping times to the controller- any interface- is on the order of hundreds of milliseconds and user traffic reflects the same latency.  Failing over to SSO partner makes no difference. Removing all but one uplink makes no difference. If I shut down uplinks momentarily and restore them, ping times go back to back to 1ms as they should be, over hours as box pair gets busier the latency comes back, and 5508s on same networks with identical uplinks trunks dont have anything like this happening. 8510s are coasting easy on CPU and memory.

 

I DO NOT have lag set up on these, but TAC said it shouldn't matter. I have another pair with LAG configured that doesn't show same behavior. I plan on doing LAG, but the behavior just seems buggy. An 8.0.110.9- escalation build for 5508s random reboot.

 

Any ever seen this sort of condition? Is definitely only between our firectly connected Nexus 7K and the 8510 pair, and not a matter of traffic as the 5508s on same subnet have no similar problems. USing Cisco's own SFPs. 

1 Accepted Solution

Accepted Solutions

Ric Beeching
Level 7
Level 7

I haven't see it so far and also have a pair of 8510s in HA-SKU connected to a Nexus 7K. Currently they only have single 10Gb interface each connected.

That setup is running 8.0.110.0.Have you monitored the 7k ports to see if they are being saturated?

 

-----------------------------
Please rate helpful / correct posts

View solution in original post

21 Replies 21

Ric Beeching
Level 7
Level 7

I haven't see it so far and also have a pair of 8510s in HA-SKU connected to a Nexus 7K. Currently they only have single 10Gb interface each connected.

That setup is running 8.0.110.0.Have you monitored the 7k ports to see if they are being saturated?

 

-----------------------------
Please rate helpful / correct posts

Ignore that Correct Answer thing.

The 7K ports are nowhere near saturation. And- msglogs on 8510 show nothing of interest. No memory or CPU issues shown on 8510 during condition. After pulling a few hundred APs off to the 5508s, the 8510 seemed to settle right down. So the mystery remains, and I will be adding LAG early next week.

 

Also working with TAC now- they are leaning toward an AVC bug with high client counts

 

Have you enabled AVC, Does the issue go away once AVC disabled on WLC.

 

We are a university and absolutely rely on AVC, so yes- it is enabled. I can't simply disable it until an outage window because it is where we police P2P traffic.

#AVC is done at WLC on AireOS until 8.0 unlike 8.1 and Converged access is on AP(not MA).

http://www.cisco.com/c/en/us/td/docs/wireless/controller/release/notes/crn81.html
This release extends the AVC functionality from Cisco WLC to the AP. 

Thanks, Saravana. I'm not following the difference between 8.0 and 8.1 with AVC- if it's not done on WLC, where is it done? (AP?) Also not understanding the converged access statement.

 

-Lee

On 8.1, They've moved the AVC filter/policy to the edge ie., on AP so that WLC doesn't have to handle it, Basically not putting pressure on WLC and let the individual APs to handle the AVC stuff.

Converged Access controllers handles AVC filter/policy at AP from day 1.

We have growing circumstantial evidence that AVC is impacting performance on both 5508s and 8510s running 8.0.110.9. This code stuff is utterly maddening on big networks. 

Please touch base with TAC/BU for fix/workaround/code update suggestions.

Also, to stay on top, signup with BU/TME to receive pre-fcs image on that code family to pilot/test the features you mostly use to avoid similar situation when expected to migrate. If no time/resource/infrastructure available then try forward the top feature usage and how it is used on your setup to BU to include this in their test bed.

I know this is an old tread.

But is there a BugID for this bug ?

I have the same issue at a customer with 25K to 30K wireless clients on a 8510, when AVC is enabled the performance drops to below half in certain periods.

They are running 8.0.133

We ran into an issue with v8.0.x with teredo traffic and AVC, I have to dig up the bug. However, AVC inspected every traffic and since teredo traffic was fragmented and sending keep alive, this bogged down the controller and caused packets to drop.

Now we ran into other issues with v8.0.x that also caused us to move to v8.2.110.0 to fix this. You should be able to find open bug list for ex-sop, NDP issues and 5ghz not responding to clients are some. RRM wasn't working well because of the NDP issue. My suggestion is to get off that code and go with the latest v8.2. v8.0.x was a pain and we had to provide TAC a lot of packet captures.
-Scott
*** Please rate helpful posts ***

Right now we have the issue on 8.2.151 around 3K APs, and about 22K Clients on 8510.

 

When AVC is enabled, and when we reach about >2Gbit bandwith through the WLC, we start to get very high latency from Wireless clients to the rest of the net, and from wired management station to WLC management interface (200ms+). performance tests from wireless clients at this point also drops to around 5 to 10Mbit.

 

When we disable AVC, bandwith increases to what we would expect, 4Gbit+, and latency from wired management station to WLC goes down to >1ms. Wireless performance tests increases to a more expected 200Mbit+.

 

Im really setting my eyes on BugID: CSCuz10099

 

I have an open TAC case, and is looking forward to more from them.

 

/Thomas

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: