topic Re: Poor Performance in Network Security

Poor Performance

Rosa Ladeira — Mon, 11 Mar 2019 18:30:05 GMT

I have been working with Pix Firewall and ASA 5550.

I am using the default policy configuration including inspect http.

I got throughput 10 times biger without using inspect http (on both pix and asa) when moving files :

wget http://averybigfile

Re: Poor Performance

terrygwazdosky — Tue, 24 Aug 2010 17:57:43 GMT

Could you post the output of the following commands?:

sh run service-policy

sh run class-map

sh run policy-map

Re: Poor Performance

Rosa Ladeira — Tue, 24 Aug 2010 18:34:09 GMT

on pix firewall:

pix# show running-config policy-map
!
policy-map global_policy
class inspection_default
inspect dns maximum-length 4096
inspect ftp
inspect h323 ras
inspect netbios
inspect rtsp
inspect skinny
inspect esmtp
inspect sqlnet
inspect sunrpc
inspect tftp
inspect sip
inspect xdmcp
inspect rsh
inspect icmp
inspect http

pix# sh running-config class-map
!
class-map inspection_default
match default-inspection-traffic

on ASA :

asa# show running-config policy-map

policy-map type inspect dns preset_dns_map

parameters

message-length maximum 512

policy-map global_policy

class inspection_default

inspect dns preset_dns_map

inspect ftp

inspect h323 h225

inspect h323 ras

inspect rsh

inspect rtsp

inspect esmtp

inspect sqlnet

inspect skinny

inspect sunrpc

inspect xdmcp

inspect sip

inspect netbios

inspect tftp

inspect icmp

asa# show running-config class-map

class-map inspection_default

match default-inspection-traffic

asa# show running-config policy-map

policy-map type inspect dns preset_dns_map

parameters

message-length maximum 512

policy-map global_policy

class inspection_default

inspect dns preset_dns_map

inspect ftp

inspect h323 h225

inspect h323 ras

inspect rsh

inspect rtsp

inspect esmtp

inspect sqlnet

inspect skinny

inspect sunrpc

inspect xdmcp

inspect sip

inspect netbios

inspect tftp

inspect icmp

Re: Poor Performance

terrygwazdosky — Tue, 24 Aug 2010 19:20:01 GMT

Everything looks pretty baisc. I didn't see the service-policy, but since everything else is default I'm assuming you are just using a global policy and not interface specific policies.

A few other questions:

Is it just http file transfers that are slow? How is browsing in general?
Are you doing any URL filtering? I've had occasional issues with Websense that caused slow web traffic.
What version of software are the PIX and ASA running?

Also, try this:

Run this command which will clear your service-policy statistics: "clear service-policy global" (unless you are using interface specific policies)
Enable http inspection with defaults
Run this command to outline what the traffic flow matches: "sh service-policy flow tcp host eq 1025 host eq http". It will most likely just hit the defaults.
Perform testing
Run this command: "show service-policy inspect http", and look to see if there are any drops or resets that may indicate protocol violations and the like
If nothing show up with the above it might be worth setting up a capture for traffic to and from the website you use for testing and then looking at the results in Wireshark to look for wierdness.

Let us know how it goes.

Re: Poor Performance

Scott Nishimura — Tue, 24 Aug 2010 19:52:01 GMT

Hi Rosa,

Is there a reason you are running the http inspection.. It will do strict http checking so it can slow down the traffic. The ASA will already be looking at the tcp traffic so its more like double checks that are going on. If you are transfering data using port 80, then the inspection will definitely be analyzing the traffic.

http://www.cisco.com/en/US/docs/security/asa/asa82/command/reference/i2.html#wp1735782

Alot of people are not running with http inspection unless you need the strict checks that it does.

show perfmon will show you the packets per sec that http is looking at along with tcp fixups, etc.

regards,

scott

Re: Poor Performance

Rosa Ladeira — Tue, 24 Aug 2010 20:11:58 GMT

Thanks for helping.

Answering your questions:
* Is it just http file transfers that are slow?  
Yes. Many user had questioned about.

* How is browsing in general?
It is fine.

* Are you doing any URL filtering?
No. I have done ASA's factory reset before testing in order to use only 2 interfaces. 
Each ASA's interface has a host. One of them wget's.

* What version of software are the PIX and ASA running?
ASA Version 8.2(1) 
Cisco Adaptive Security Appliance Software Version 8.2(1) 
Device Manager Version 6.2(3)
Hardware:   ASA5550, 4096 MB RAM, CPU Pentium 4 3000 MHz
Internal ATA Compact Flash, 256MB
BIOS Flash M50FW080 @ 0xffe00000, 1024KB

Encryption hardware device : Cisco ASA-55x0 on-board accelerator (revision 0x0)
                             Boot microcode   : CN1000-MC-BOOT-2.00 
                             SSL/IKE microcode: CNLite-MC-SSLm-PLUS-2.03
                             IPSec microcode  : CNlite-MC-IPSECm-MAIN-2.04
 

* Run this command which will clear your service-policy statistics: 
"clear service-policy global" (unless you are using interface specific policies)
Done

* Enable http inspection with defaults
Done

* Run this command to outline what the traffic flow matches: 
"sh service-policy flow tcp host  eq 1025 host  eq http".  
It will most likely just hit the defaults.

show service-policy flow tcp host 147.65.32.25 eq 1025 host 147.65.1.48 
eq http

Global policy: 
  Service-policy: global_policy
    Class-map: class-default
      Match: any
      Action:
asa#    Output flow:

 
* Perform testing
Now it runs fine & fast

* Run this command: "show service-policy inspect http", and look to see if there are any drops 
or resets that may indicate protocol violations and the like

asa#             show service-policy inspect http

Global policy: 
  Service-policy: global_policy
    Class-map: inspection_default
      Inspect: http, packet 1035582, drop 0, reset-drop 0


* If nothing show up with the above it might be worth setting up a capture for traffic to and 
from the website you use for testing and then looking at the results in Wireshark to look for wierdness.
 
Let us know how it goes.

Re: Poor Performance

Rosa Ladeira — Tue, 24 Aug 2010 20:33:37 GMT

Hy Scott, thanks for your answer.

I was not sure that not running inspect http would be a correct choice.

According to your answer, running inspect a protocol will "allways" slow down performance ?

I had configured :

inspect dns preset_dns_map
inspect ftp
inspect h323 h225
inspect h323 ras
inspect rsh
inspect rtsp
inspect esmtp
inspect sqlnet
inspect skinny
inspect sunrpc
inspect xdmcp
inspect sip
inspect netbios
inspect tftp
inspect icmp
inspect http

sip, h family rtsp skiny will cause VoIP degradation ?

Re: Poor Performance

Scott Nishimura — Tue, 24 Aug 2010 20:39:11 GMT

Hi Rosa,

its up to you whether you need the extra strict http checks. A lot of sites do not adhere to standards. As for it always causing performance problems-- not really, but it does add extra inspection and when you have the firewall doing inspections, it is sent to the cpu for further processing. So if you are having http file transfer, it can slow down the traffic as it has to look at every packet.

Sip inspection is to open up additional secondary pinhole conns so that is what that inspection is doing and is different from the http which is looking at all port 80 traffic.

regards,

scott

Re: Poor Performance

Rosa Ladeira — Wed, 25 Aug 2010 14:20:37 GMT

Hi Terry.

Yesterday I have made tests using a 100Mbs network interface client host.

Results were mascarade.

Today I have used a 1000Mbs network interface's client.

I have done ASA's factory reset before testing.

Following your suggested configuration on ASA, I got:

global policy with no inspect http : throughput -> 450Mbs

global policy with inspect http : throughput -> 200Mbs

Moving client host to http server subnet (no ASA between them) throughput scales to 900Mbs.

As you can see above Scott has suggested not using inspect http.

What do you think about ?

Re: Poor Performance

Jitendriya Athavale — Wed, 25 Aug 2010 14:45:30 GMT

i belive this could be because of out of order packets...

can u please apply captures on outside and inside and see if you see any out of order packets

Re: Poor Performance

terrygwazdosky — Wed, 25 Aug 2010 14:48:33 GMT

Wow, that is a big difference. I don't notice anything approaching that level of slow down with 1000+ users on an ASA 5520 and that's with http inspeection and Websense filtering.

If you don't need the http filtering, at least in the short term, you may want to leave it off for until you can get to the bottom of this issue. I'm wondering if maybe you've hit a bug with the version of code you have. It might be worth opening a TAC case to get some further assistance with troubleshooting.

I'd also reccommend setting up a capture of the traffic and reviewing the results in Wireshark or whichever program you use for packet analysis.

Re: Poor Performance

Kureli Sankar — Wed, 25 Aug 2010 14:58:32 GMT

Enabling http inspection expects the packets to arrive in order (so we can inspect). If they don't arrive in order then, the ASA has to hold them until all the packets arrive. The hold buffer or queue is very small so, there are chances that the packets may be just dropped. Packets arriving out of order is the nature of the internet and may be you can reach out the ISP and ask them why we see (if you really see out of order packet via captures) out of order packets and ask if they can do anything about this.

Http inspection also sends syslogs about the URL requested by each host on the inside.

So, leave http inspection turned off unless there is a requirement that you have to have that on due to some Sarbanes Oxley regulation or some thing of that nature.

https://supportforums.cisco.com/docs/DOC-8982#http_inspection_enabled

-KS