8510s that slow down all L2 networks they connect to

[lbadman]

One of my first 8510 deployments went horribly wrong over last couple of days. For 6 weeks, had stability. This week, suddenly ping times to the controller- any interface- is on the order of hundreds of milliseconds and user traffic reflects the same latency.  Failing over to SSO partner makes no difference. Removing all but one uplink makes no difference. If I shut down uplinks momentarily and restore them, ping times go back to back to 1ms as they should be, over hours as box pair gets busier the latency comes back, and 5508s on same networks with identical uplinks trunks dont have anything like this happening. 8510s are coasting easy on CPU and memory.

I DO NOT have lag set up on these, but TAC said it shouldn't matter. I have another pair with LAG configured that doesn't show same behavior. I plan on doing LAG, but the behavior just seems buggy. An 8.0.110.9- escalation build for 5508s random reboot.

Any ever seen this sort of condition? Is definitely only between our firectly connected Nexus 7K and the 8510 pair, and not a matter of traffic as the 5508s on same subnet have no similar problems. USing Cisco's own SFPs. 

[Thomas Obbekaer Thomsen]

Guess when we disabled AVC ?

[Thomas Obbekaer Thomsen]

Ping from wired management station to WLC mgmt interface. Again ... guess when we disabled AVC.

[Thomas Obbekaer Thomsen]

The big problem here, I think, is that it is hard to see any load problem regarding AVC in the WLC.

There does not seem to be any CPU load on the Control Plane, and the DP does not seem (and I might just overlook something) to have any load either, when looking at : debug fastpath dump fpapool all

Where should one look for AVC performance issues on the WLC ?

What process, or CPU is loaded ?

/Thomas

[Michael Burk]

Same thing in our environment, we had 17k clients and were capping out around 1.5GB per 10GB link with latency peaking as high as 400ms and dropped packets. Immediately after disabling AVC the performance is back to stable. 

Downtime note: Disabling AVC is a WLAN setting, when you change WLAN settings it requires an WLAN reset. In our case; there were 17k clients on one SSID, it took about 5 minutes for all clients to come back online because all of the authentication requests were hammering the controller at once, and when the clients was authenticated, now all of the DHCP requests where hitting the DHCP server at once. So plan appropriately for your scale. 

[Freerk Terpstra]

There is no reference to a specific platform within the conditions of the bug, did you receive any information from TAC regarding other platforms like the 5520 or 8540?

[Thomas Obbekaer Thomsen]

Sorry for the very late reply.

Lets just say I did not "receive any information from TAC". 

But I have on "good authority" (ssshhhh its a secret) that these other platforms (5520 and 8540) have a a marginal hit when AVC is enabled. - I have no info on the old Cavium platforms (5508, WiSM2, 2504).

[Saravanan Lakshmanan]

show port stats from wlc and 7k. 

does wlc msglog shows specific error message associated to the behavior.

check memory on wlc.

it seem not flushing the buffer, need to verify if they're port-level or memory issue or datapath loop.