WCCP imposed limitations on WSA-S370

[rikherlaar]

Hi all

Transaprent proxy'ing design - 

Running WCCP on ASA5585X - all configured correctly (client and server same vlan - and all working fine under low load)

2x S370's assuming load for about 4000 users -

The moment we migrate as-is (legacy DMZ) to the new build DMZ (injecting default route attracts outbound traffic -) we see that around 8000 sessions are getting established and after that it's becoming dead slow - users complain about the fact that loading a web page takes ages.

So our initial thinking was that the S370's just didn't cut it and we had underestimated the load (Cisco expressed load in terms of "users" which is very coarse since a user can obviously trigger hundreds of HTTP(S) sessions - hard to dimension it properly w/o having forensic data about the "as-is" state. 

Now - I read before that the expected bandwidth that a single S-370 will be able to fill (and we do an awful lot of inspection and filtering as it concerns a highly secure environment) a 100Mbps max. Our aggregate outbound internet access capacity is about 1Gbps to offer some perspective .

Now here' s the kicker - while the performance problems endured - we enabled explicit proxy - and these pages loaded very fast. So my initial theory that the S-370 was fully swamped was apparently not accurate.

The only diff I do recognize here is that explicit proxy doesn't bank on GRE encapsulation (as it's not even hitting the WCCP redirect interface - but routed to one of the proxies directly).

Two possibilities -

• ASA cannot cope with the load - (ASA is performing WCCP in SW - and is not the best platform to perform WCCP in a relatively large environment - but 8000 sessions is not enormous neither did we see high CPU load or any other evidence this guy was running out of steam)

- or - 

S-370 taking a serious hit due to GRE decapsulation at a certain moment - 

Is there anything I can get from either ASA and S-370 to identify the actual issue - 

The attached graphs (for a single S370) are attached - note that day-1 (Nov 19th) - the migration started around 10:00 AM - 

The second day (20th) - are not live traffic related (customer did not want any further attempts before we could identify the issue) - we did what we could to simulate substantial load with Load Runner on a  few laptops but I realize the ramp-up time and the IO horse power cannot get close to the total amount of actual users that would blast in the moment we'd change the default-gateway again

We have an open TAC case- which I had hoped would learn us that we have either underestimated the # required WSA's for the design or to get away from the ASA as WCCP server - so far nothing conclusive though.

Kind regards

Rik

[Ken Stieers]

1. what version of code on the S370s? Get to 8.0...

2. Does your WCCP ACL have a deny for both WSAs (keeps traffic from one WSA from being redirected to the other)?

access-list WCCP_Redirect extended deny ip any4 object-group CompanyInternal
access-list WCCP_Redirect extended deny ip host 172.16.x.21 any4                     
access-list WCCP_Redirect extended deny ip host 172.16.x.22 any4                    
access-list WCCP_Redirect remark redirect traffic from internal sources to the WSA(s)
access-list WCCP_Redirect extended permit ip object-group CompanyInternal any4

[rikherlaar]

Hi Ken - thx-

So for now we cannot move up to version 8 yet courtesy to ESA's being managed as well from same platform today and that capability changes drastically apparenty. So unless there's hard evidence that the perfornabce numbers in version 8 are significantly better we're a bit hesitant. Right now we're at highest 7.x release.

As for the ACL - good one , I thought this had only impact on client athentication but will slam it on to be on the safe side.

In general - I'm still a bit uncertain what causes the performance hit here - ASA or S370 and would it be significant less stressful for the WSA to deal with L2 redirect (introducing an ASR1K) versus GRE .

I admit that the ASA is not the best possible platform for larger environments when it comes down to WCCP - but I was trying to get a feel for breakpoints.

/R

[Ken Stieers · Accepted solution]

That ACL goes on the "redirect list" in the gui.  It coontrols who's traffic gets thown at the WSAs, you can be redirected twice without it.

7.7 is WAY slow.  TAC's previous advice was to revert to 7.5...  8.x has a lot of performance fixes.

[rikherlaar]

Thx Ken

Good call on 7.7 - I will pass the message onwards -

Updated to 8.0.6 in the meantime and this is day and night - performance increased  four fold -

WCCP on ASA proved to be a non-issue - the high CPU load on 4 cores (SSP10) had nothing to do with WCCP as such.

RPS seems still to be at times spiking the CPU's of the WSA's - we added 4 additonal WSAv's - and that turned out to spread the load acceptably. For a 7500 "user" environment - clearly the two initial WSA S370's were not enough - the S300's seem to handle the load a tad better than the original 370's - could have to to with better X86 (UCS vs HP) ;-)

Regards

/R