cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
17307
Views
25
Helpful
17
Replies

IPS in ASA 5510 killing upload speed

Jake Pratt
Level 1
Level 1

I recently upgraded from a 20 Mb metro ethernet circuit to a 100 Mb connection.  My ASA 5510 is seriously throttling my upload speed.  I have narrowed it down to the IPS module.  If I stop sending traffic to the IPS, I get upload speeds between 50-85 Mbps.  If I start sending it through again, my upload speeds are between 3-7 Mbps.  In both cases, my download speeds range between 70-92 Mbps, so it's really only affecting my upload speeds.  Is there anything I can do to my IPS traffic, so I can still use my modules, and still take advantage of the huge upload speeds we pay for?

Here is some info from my ASA:

I am matching all traffic:

access-list traffic_for_ips extended permit ip any any

Here are my policy and class settings:

class-map inspection_default
match default-inspection-traffic
class-map botnet-DNS
match port udp eq domain
class-map ips_class_map
match access-list traffic_for_ips
!
!
policy-map type inspect dns preset_dns_map
parameters
  message-length maximum 512
policy-map global_policy
class inspection_default
  inspect ftp
  inspect h323 ras
  inspect rsh
  inspect rtsp
  inspect sqlnet
  inspect skinny 
  inspect sunrpc
  inspect xdmcp
  inspect sip 
  inspect netbios
  inspect tftp
  inspect dns preset_dns_map
class ips_class_map
  ips inline fail-open
policy-map botnet-policy
class botnet-DNS
  inspect dns dynamic-filter-snoop
!
service-policy global_policy global
service-policy botnet-policy interface Outside

If anyone has any ideas, I would love to hear them.  Thanks.

1 Accepted Solution

Accepted Solutions

Aastha Chaudhary
Cisco Employee
Cisco Employee

Date Created: 13-MAY-2011 06:49 PM Created By: Chaudhary, Aastha(AACHAUDH,265429) Customer was experiencing slow upload speeds(3-7 MBPS) on IPS module in ASA 5510. Download speeds range between 70-92 Mbps

Used the workaround for bug id CSCsv69844 i.e. set the Regex Depth Setting to 800000(Please note that this workaround should only be used with TAC's recommendation and approval.)

View solution in original post

17 Replies 17

rhermes
Level 7
Level 7

You have a few choices.

One is to place your IPS into promiscuous IDS mode so that it will not have any effect on traffic yet still report on intrusions.

If you want to keep the in-line protection in place you need to figure out what signature is getting hit and dropping your traffic. There are a few normalizer engine signatures that will drop traffic and not generate an alert. Go turn on alerting for those signatures and see what signature is getting hit. You can then try to correct the source of the problem (our last reader had an out-of-order packet problem caused a similar issue) or remove the drop action from the signature.

- Bob

Thanks for the reply Bob.  I used to be running the IPS in promiscuous mode, but I had to change it to inline to enable something specific... I think it was my botnet filtering.  Either that or to allow some type of reporting to MARS.

Anyway, I'm not familiar with how to turn on alerting to the specific sigs you are talking about.  Could you give me a quick run down of how to do that?

Thanks again!

I tried changing my IPS policy from  inline to promiscuous, and my up speeds are now ranging from 7-15 Mbps,  but still nowhere near what my down speeds are.

Jake -

If you're using the free Cisco IME to manage your IPS sensor;

hit the "Configure" button on the top menu bar

select "Policies" in the lower left frame

select "Active Signatures" in the upper left frame

Once the active signatures load (it takes a while) look in the "Engine" column for the Normalizer engine

You'll see several signatures set to drop but not alert. You can edit them from there.

- Bob

Thanks.  I'm using the ASA's ASDM to monitor the IPS.  From what I've seen of IME, it looks very similar, but a little different.  In the ASDM, you go to configure the IPS, then go to Policies > Signatur Definitions > sig0 > All signatures, and look for the normalizer engine signatures.  I'll go through there and look at which ones are set deny the packet, but not alert, and set them to alert.

If you're using the free Cisco IME to manage your IPS sensor;

hit the "Configure" button on the top menu bar

select "Policies" in the lower left frame

select "Active Signatures" in the upper left frame

Once the active signatures load (it takes a while) look in the "Engine" column for the Normalizer engine

You'll see several signatures set to drop but not alert. You can edit them from there.

Thanks again, Bob.  I've gone through, and added alerting to the following sigs.  These sigs were set to drop packets and not alert.

1330/9 TCP Drop - Data in SYNACK
1330/1 TCP Drop - Bad Flags
1330/0 TCP Drop - Bad Checksum
1330/18 TCP Drop - Segment out of window
1330/17 TCP Drop - Segment out state order
1330/12 TCP Drop - Segment out of order
1330/10 TCP Drop - Data past FIN
1250/0 Packet Bad Length

I am now running in promiscuous mode, and it seems these are all set to drop inline only.  After setting all these sigs to produce an alert, I have been trying to see any evidence of these sigs being triggered.  I have been looking at both the sensor monitoring events in the ASDM, and the ASA logging events, while running another speed test.  While I see my up speeds are a little faster in promiscuous mode (about 14 Mbps), I don't see any obvious signs of these sigs triggering and dropping packets.  I could be looking in the wrong place, though.  And from what I've seen, there isn't a way to have an SSM sensor report directly to a syslog server.

Thanks

That is very strange that you see a decrease in traffic throughput when in IDS mode. The sensor should be in the traffic stream when in promiscuous mode.

You should eb able to see alerts on teh CLI console of your sensor with the command:

show event alert past 00:01 (shows the last min of events plus any that occur as long as you leave that command active)

You can filter the output looking for alerts in the past with the usual "| inc xxxxxx" filter.

- Bob

It's weird.  I am just not showing anything useful when I monitor the events.  Do I need to be producing verbose alerts rather than just alerts?

No, verbose alerts only capture a larger portion of the offending packet for inspection. With the "sh ev pa 23:59" command you should be seeing all the IPS event alerts your sensor has generated in the past 24 hours or so.


- Bob

Hello Jake,

Throughput speeds will vary greatly depending on the method you are using to measure. What tool or method are you using to derive your throughput speeds?

Thank you,

Blayne Dreier

Cisco TAC Escalation Team

**Please check out our Podcasts**

TAC Security Show: http://www.cisco.com/go/tacsecuritypodcast

TAC IPS Media Series: https://supportforums.cisco.com/docs/DOC-12758

Throughput speeds will vary greatly depending on the method you are using to measure. What tool or method are you using to derive your throughput speeds?

This is a Qwest metro circuit, so I am using the Qwest speed test tool (speedtest.qwest.net), for the closest server to my location.  That is the tool the recommend.  I seem to get more accurate readings on that, than with speedtest.net.  I am quite confident that I am getting accurate readings from this speed test.  Sure there is some variance from test to test, but the amount of change in my upload speeds when I'm performing IPS packet inspection, and when I'm not is like 45-80 Mbps.  And that is pretty consistant.  I'm am sure the IPS modules are creating the choke point.

Hello Jake,

I have no doubt that the reading is accurate for the test that is being performed. I'm more interested in how the test is being performed. We really need to break the test down into two different components: Download and Upload. The two actions are never equivalent in speedtests due to the programatic requirements surrounding the fact that your PC is always the client. If you ran the Download test through the sensor with your client on the left and your server on the right and then swapped the location of the server and client, you would receive the same reading in both scenarios (assuming the same security and traffic policy in both directions). The same would be true if you ran the Upload test in the same manner. However, the Upload and Download readings will vary greatly and to different degrees, given variations in configuration and software packet handling.

Regarding the speedtest at http://sunnyvale.speedtest.qwest.net, the entire process appears to always consist of 8 connections that are initiated by the client. Each connection transmits approximately 7MB.

Connection 1:

Download

10 GETs for /speedtest/latency.txt?x=Z, where Z is a very large number. latency.txt is 10 bytes.

1 GET for /speedtest/random350x350.jpg?x=Z. random350x350.jpg is 245388 bytes.

1 GET for /speedtest/random1000x1000.jpg?x=Z. random1000x1000.jpg is 1986284 bytes.

latency.txt is a small text file. Visiting the file directly from the browser returns a page with "test=test."

Both random* images are static (not dynamically generated) and appear to be randomly colored pixels.

Upload

7 POSTs to /speedtest/upload.php?x=0.W, where W is a string of integers.

Visiting http://sunnyvale.speedtest.qwest.net/speedtest/upload.php?x=0.W via a browser returns a page with "size=Y," where Y changes given different values of W. Very small and very large values of W return very large Y's.

POST 1 and 2 are approximately 110000 bytes.

POST 3-7 are approximately 850000 bytes.

The data uploaded appears to be randomly generated text.

Connections 2-8:

Identical to Connection 1 except that the 10 GETs for latency.txt are not present.

If very long strings of random text are a good depiction of your normal outbound data traffic, this test accurately gauges the true throughput of the IPS on your network during normal operation. Most of the time, speedtests do not accurately depict normal traffic profiles and can be quite misleading.

If you are certain that this test accurately depicts your normal traffic profile, we can tune the IPS to perform better on this test (and therefore perform better on your normal traffic). However, if your normal traffic profile differs from this test, I suggest tuning for your normal traffic profile instead.

In either case, if you would like assistance tuning the sensor, please open a TAC case and we'll be glad to help.

Thank you,

Blayne Dreier

Cisco TAC Escalation Team

**Please check out our Podcasts**

TAC Security Show: http://www.cisco.com/go/tacsecuritypodcast

TAC IPS Media Series: https://supportforums.cisco.com/docs/DOC-12758

Jake,

I see that you've opened a TAC case. I'll keep an eye on it to make sure we're headed in the right direction. Please let me know your response to my previous post so that we know how to proceed.

Thank you,

Blayne Dreier

Cisco TAC Escalation Team

**Please check out our Podcasts**

TAC Security Show: http://www.cisco.com/go/tacsecuritypodcast

TAC IPS Media Series: https://supportforums.cisco.com/docs/DOC-12758

Jake Pratt
Level 1
Level 1

After getting some unusual runaround with TAC, and over a week of not hearing anything, they finally got me assigned to a tech that knew what he was talking about.  There is a bug fix for this issue, which can be found here: http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCsv69844

Basically, this is what I had to do (as stated in the article):

Once the IPS is upgraded to the code that contains this enhancement,
the further configuration needs to be done as follows:

1. Log into the sensor via the service account and run the following commands:

bash-2.05b# su
bash-2.05b# /etc/init.d/cids stop
bash-2.05b# cd /usr/cids/idsRoot/etc
bash-2.05b# vi sensorApp.conf

2. Add the following lines:

[Process]
RegexDepth=800000

3. Save the file and exit the editor:  "ZZ" or :wq

4. Reboot the sensor:

bash-2.05b# /etc/init.d/cids reboot

NOTE: 800000 is a suggested starting value, this might need to be lowered
depending on the specific network. The exact value needs to be adjusted
experimentally based on the performance monitoring.

A few helpful notes if you're not extremely familiar with this process:

1. You'll want to make sure you are SSH'd directly into the sensor.  I was initially sessioned in from the ASA, and as soon as I ran "/etc/init.d/cids stop", it killed my connection to the sensor.  You won't lose it if you are connected directly via SSH.

2. To get to the "bash-2.05b#" prompt, you must connect with a user that has "service" access.  If you don't have one, log in with admin creds, and create a new one.

3. You'll need to know how to use the vi editor.  If you don't, just Google it.  There is a lot of information out there.

After I tuned my sensor like this, my up speeds jumped up to 60-80 Mbps.

Thanks to my tech, Aastha, for the help!

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: