IPS slows down internet access

[ahamadfaiz]

Hi All,

We have an IPS4270-20-K9 appliance monitoring our DMZ network and INSIDE segment. There are two virutal sensors with different Signature Definiton and Event Action Rules policies for each segment.

We recently updated software version from 7.0(2)E4 to version 7.1(7)E4. Ever since we have issues with internet access. We are unable to access sites like Google, Youtube etc (yeah these are allowed through our network). Youtube page opens but the streaming does not happen. We were suspecting our proxy or ISP as it was a few sites that does not work. However, we tested the sites by directly connecting to our internet router and it worked fine.

Then we tested by bypassing IPS inspection by changin Bypass Merode in to Off. Everything works just fine then. We did this for a few times while the sites were not accessible and it gave the same result.

When I check the instpection-load it is always below 25. The CPU shows close to 100% all the time, but the Cisco says it is not the correct measure.

Have anyone faced similar issue. Please assist with this.

Regards,

Faiz

[ahamadfaiz]

Hi,

Additional investigation shows that the signature TCP Drop - RST or SYN in Window is getting triggered in huge numbers. The traffic is from external IP addresses to Proxy IP, which is obviously for return traffic from internet.

The signature description says that "If a packet in a stream causes this signature to produce an alert, processing will cease for that stream".

I suspect that is causing this issue. Please let me know.

Regards,

Faiz

[TM13]

Hi Faiz,

  I have almost same issue, but looks like symptoms are different, our download speed is not so bad, but upload speed is incredibly slow, even not uploading and dropping everything, we having 4240 and it was happend 2 weeks ago after updated latest signature but still find the real cause, in my case if we put Bypass mode it having same result, but looks like you having different situation, you need to check the Signature ID and maybe can disable them and see. Any TAC created?

[ahamadfaiz]

Hi Tulgabat,

Thank you for the reply.

I am yet to dig deep in to the event and signature. I hope I can get more info then.

However, we have identifed a common user ID that has been misused. This user was accesging youtube and such streaming sites extensively. We have now disabled this user and we have not faced this issue after that. However, we cannot conclude anything yet because this issue is intermittend and we need wait for a few days to see if that has helped.

Regards,

Faiz

[largenb]

Hi Faiz,

It sounds like you may have been able to isolate the issue.

For future reference, if you would like to keep this signature (1330-14 in this case) enabled on the IPS for all of your other hosts but want it tuned to not alert on the particular proxy host, you could add an event action rule for the internal proxy for this particular signature and subtract the produce alert from the action.

Via IDM

Configuration -> Event Action Rules -> rules0 -> Even Action Filters

+ Add

Name your filter. Add the proxy as the destination. Fill in the other needed fields. -> Action to Subtract -> remove any of the alert variables.

This is really helpful when you are first placing an IPS in place and/or when you are adding new networks.

[ahamadfaiz]

Hi William,

Thank you.

I am aware of event action filters and have a few created already.

However, I wanted to know why this signature is getting triggered in the first place. Moreover, it does not process the traffic if the signature triggers for a stream. I am trying to undestand what causes this. Especially for streaming traffic.

Please assist.

Regards,

Faiz

[ahamadfaiz]

Hi All,

We still have this issue in our network. I see the following errors in IPS this time:

evError: eventId=6822257491706  vendor=Cisco  severity=error 

  originator:  

    hostId:  

    appName: cidwebserver 

    appInstanceId: 1458 

  time: Aug 31, 2013 09:08:36 UTC  offset=180  timeZone=GMT+03:00 

  errorMessage: Throttled connect timed out [ClientPipe::connect]

Messages, like this one, in the category - Connect timeout - were logged 12 times in the last 72492 seconds.  name=errSystemError 

evError: eventId=6822257491707  vendor=Cisco  severity=error 

  originator:  

    hostId:

    appName: cidwebserver 

    appInstanceId: 1458 

  time: Aug 31, 2013 09:08:36 UTC  offset=180  timeZone=GMT+03:00 

  errorMessage: - ct-sensorApp.1475 not responding, please check system processes - The connect to the specified Io::ClientPipe failed.

Messages, like this one, in the category - ctlTrans Timeout - were logged 12 times in the last 72492 seconds.  name=errSystemError 

It appears that the sensorApp is not responding. Can anyone assist.

Regards,

Faiz