PIX 525 Firewall - Performance problem

[fullerms]

Hi all,

We have a performance issue with our Pix 525. CPU utilization has suddenly shot up to 80-90%, and during peak traffic loads, the pix even crashes.The normal utilization is around 30-40%.

The firewall was originally running IOS version 6.2(2) and upgraded to V6.3(3) and then to 6.3(4) after being advised by Cisco TAC.

A TAC case has been running for the past one week, and a resolution is still not available. Could someone give out a few pointers as to where I should start looking?

[a.awan]

Are you using this firewall for terminating any VPN tunnels? Which process seems to be eating up the cpu? I think you can use the show processes command to determine the runtime for each process.

A bug might be causing it but it might be worth while making sure that there is no worm lose on your network. The following link will not solve your problem but it will provide you with some basic pix perfomance monitoring guidelines which in turn might lead you to the source of the problem.

http://www.cisco.com/en/US/products/hw/vpndevc/ps2030/products_tech_note09186a008009491c.shtml

[fullerms]

All the outputs listed in the url have been sent to TAC. I think we can eliminate bugs also, since we have upgraded the IOS twice. Any inputs on how to track down a worm?

[a.awan]

A clear indication of a worm is a lot of half-open connections (embroynic) originating from the inside of your network. A recent worm i had encountered was initiating TCP sessions to port 135, 137 and a few others. This usually results in the firewall reaching its connection limit very quickly and then starves out legitimate users but i would think this can also drive the CPU usage to high levels if enough insiders are affected. I guess while you wait for TAC to respond you might as well check this option out too.

[mhaverstock]

We have had similar issues with our PIX before, and as indicated in other people's responses, the cause is probably a worm or virus in your network, sending traffic out through the PIX.

Do a "sh conn count" Do you have a "huge" number of connections open. (say more than 200,000) If so, you can do a "sh conn" There is a lot to scroll through, but just look for repeat offenders. (An IP address that has hundreds of screens of connections open) Create an access-list to block that conversation and it should reduce the load.

Another thing to check that could make CPU go high, is the number of new connections per second. Do a "sh perfmon". How many new TCP and UDP connections per second are being set up? If this number is too high, the only way to stop it is to find out who ("sh conn" usually works) and then block that IP address.

If you have access to a sniffer, look for unsual traffic pattens in and out of the PIX.