Slow Proxy Throughput - Web Proxy CPU at 100% - S670 - AsyncOS 7.7.0-761

[jasonnickel]

We are seeing throughput and CPU issues with our pair of S670 proxies.

Both S670s are running AsyncOS 7.7.0-761 and their overall CPU usage is around 20-30%, however the Web Proxy CPU usage is showing 100% during the day and around 40% after hours when most people are no longer in the office.

Here is a "status" snapshot for a typical day.

Enter "status detail" for more information.

Status as of:                  Tue Feb 10 11:17:01 2015 CST
Up since:                      Thu Dec 11 22:12:41 2014 CST (60d 13h 4m 19s)
System Resource Utilization:
  CPU                                    21.8%
  RAM                                     0.0%
  Reporting/Logging Disk                 24.0%
Transactions per Second:
  Average in last minute                   844
Bandwidth (Mbps):
  Average in last minute                86.489
Response Time (ms):
  Average in last minute                   296
Connections:
  Total connections                      16055

However here is the proxystat output from the same time period.

Press Ctrl-C to stop.
  %proxy  reqs                         client    server    %bw  disk  disk
     CPU  /sec   hits blocks misses    kb/sec    kb/sec  saved   wrs   rds
 98.00   608    611   2532   2944     66128     57309   13.3  2210   544
 98.00   676    498   2617   3645     95540     89996    5.8  2911   567
 99.00   777    770   3038   3960     89891     76332   15.1  2948  1093
 98.00   710    555   2988   3548     83362     78107    6.3  2567   595
 99.00   696    871   2516   3588     81584     71525   12.3  2634   875
 99.00   731    850   2958   3500     96219     83045   13.7  2197   917
 98.00   700    586   3106   3318     72434     61207   15.5  2024   478
 99.00   786    852   3236   3793    107520     93851   12.7  2228  1083

We are trying to figure out what exactly is causing the high CPU usage, but everything in terms of the content being served seems normal.

Our Internet connection is 500 Mbps and between the two S670s we are only able to hit around 300 Mbps.

[Ken Stieers · Accepted solution]

Get off of 7.7 as soon as you can...

Seriously.  There were performance issues... they fixed a bunch of them in 8.0...

[Handy Putra]

There are multiple factors that can contribute to high usage of proxy CPU.

From the 'rate' output provided, the requests per seconds are quite high as well and if the configuration in the appliance is very expensive, this will definitely causing proxy CPU to be very high.

Configuration been expensive if you have more than 25 on each Identities, Access Policies and Custom URL categories and regular expression(this part is the most resource intensive).

Since the way the appliance work, it will check every single traffics that come to the appliance from top to buttom for each Identities, Access Polcies, etc until the condition is met and if you have massive requests/traffics at the same time or per seconds and the appliance need to perform this for every single requests/traffics, it will then use up all the resources that it has and the CPU tend to be spiking.

Hope this helps.

[jasonnickel]

Upgraded to 8.0.6 and the CPU issues went away.

The proxy CPU time went from 100% down to 55%.