ASR1002 throughput degradation when wccp redirect-list is changed

[steve22076]

We have two ASR 1002's going to 2 different WAN service providers, and two 7371 WAE load balanced by mask assignment. When we change the ACL (adding or removing lines) from our wccp redirect-list, the throughput on interfaces applied to the wccp service-groups is degraded to almost no traffic passing, until we completely remove wccp service group from the global configuration and then reapply. Then traffic throughput on the interface goes back to normal.

Our ACL defined in the redirect list specifies our specific networks on our WAN that have WAE's and need the redirection. All other networks are denied implicitly. We need to regularly change this ACL, and this service interruption is a major issue. This was not an issue before moving to the ASR platform from 7206's.

At TAC's request we have upgraded our IOS version to 15.1(3)S4 and that did not make any difference. Does anyone know why this occurs and if there is a way to work around this other than removing wccp configuration and adding back, every time the ACL needs to be modified?

As a side note to this... We have recently added riverbed appliances, and created separate service groups with separate redirect-lists. The exact same behavior occurs on the ASR 1002 when the ACL for the riverbed's redirect list is altered.

[kenny.kerns]

I'm not sure if this is directly related but I had issues with WCCP on both an ASR1001 and Nexus 7010 when trying to redirect to Bluecoat devices.

My issue was very similar to yours in that we would see terrible performance several hours after making a change to the WCCP configuration on the router.

After much troubleshooting with both TAC and Bluecoat I found that the default mask assignment method creates 64 buckets per service group, which chews up tons of memory. Are you seeing any log entries indicating TCAM resource contention?

In the end I changed the mask assignemt value to 4 buckets (0x3). This change was performed on the Bluecoat device, so in your case I would imagine that you need to make a similar change on the WAE/Riverbed devices.

Kenny

[steve22076]

Thank you very much for sharing that information.  It is great to hear verification that the mask assignment change did resolve your problem.   That is the latest resolution that TAC has recommended, but we have to restart the WCCP service on all redundant edge routers to be able to implement this, so planning the outage window is taking some time.   We've been told that TAC will set this up in a lab and test for us by our Cisco SE.  We're hoping to get verfication that this actually resolves the problem before we take the outage.   

     If you could, can you tell me if this resolved the issue 100% or do you still have any performance issues when making a change to your WCCP ACL going to your bluecoat equipment?    We may also need to implement this in our redirects to BlueCoat from our Nexus.  Do you happen to have a link to how to make this change in Bluecoat?   Thanks again!