PIX Firewall - Possible Denial-of-Service Attack?

[davelockerby]

I recently discovered an issue with PIX firewalls running FOS version 6.2(2).

When a PIX Firewall is setup to send trap logs to a log server and the syslog daemon is not running on the log server, a denial-of-service attack takes place between the log server and the PIX firewall.

It seems that the PIX and syslog server begin to send very high volumes of ICMP traffic to one another. After about 2 hours of the syslog daeman being stopped on the log server, I looked at the number of sent and received packets on the syslog server's NIC. Both the sent and received packets were just shy of 2 billion packets!

How to stop DOS Attack:

Option 1: Restart the syslog daeman on the log server; the denial-of -service-attack ends.

Option 2: Remove the syslog server that has the daeman stopped from the PIX configuration.

Note: A 443Mhz Celeron processor on a PIX 515E experiences constant processor utilization between 90-93%!

Is this a documented problem? If it is documented, please post a link to the fix/patch.

I recently discovered an issue with PIX firewalls running FOS version 6.2(2).

When a PIX Firewall is setup to send trap logs to a log server and the syslog daemon is not running on the log server, a denial-of-service attack takes place between the log server and the PIX firewall.

It seems that the PIX and syslog server begin to send very high volumes of ICMP traffic to one another. After about 2 hours of the syslog daeman being stopped on the log server, I looked at the number of sent and received packets on the syslog server's NIC. Both the sent and received packets were just shy of 2 billion packets!

How to stop DOS Attack:

Option 1: Restart the syslog daeman on the log server; the denial-of -service-attack ends.

Option 2: Remove the syslog server that has the daeman stopped from the PIX configuration.

Note: A 443Mhz Celeron processor on a PIX 515E experiences constant processor utilization between 90-93%!

Is this a documented problem? If it is documented, please post a link to the fix/patch.

Also...I contacted Cisco Tech Support and they were somewhat puzzled with the issue? Interesting.

[gfullage]

I fail to see how this "behaviour" can be categorized as a DOS attack. A DOS attack implies someone is doing something to your PIX/network that stops other legitimate users from getting through. In your case a daemon has stopped and you see increased traffic in the PIX. Unless someone actually got into your netowrk and killed the syslog daemon on your syslog server, this is not a DOS attack.

As for what's going on, what kind of ICMP packets are these? What level of logging do you have defined on the PIX, and how busy is your PIX? The PIX is going to send a UDP syslog packet (unless you have TCP syslog configured) to your syslog server. If the syslog server isn't listening on that port because the daemon has died, then it's probably going to send back an ICMP Unreachable message to the PIX. Did you actually see ICMP packets going in both directions, or did you see UDP from PIX -> syslog server and ICMP from syslog server -> PIX?

Depending on how busy your PIX is and what level of syslogging you have configured, the PIX can certainly send quite a lot of syslog traffic over the network. Did this actually cause the PIX to stop passing other traffic?

[davelockerby]

Please re-read the title of the post: " PIX Firewall - Possible Denial-of-Service Attack?" The "title" of the post was in the form of a question.

I will re-phrase my question.

Q: Can the scenario that I described in my previous post create a "Very-High-Limitation-of-Service condition" on the PIX?

Thanks for the reply. I will answer your questions in the order that they were typed in your reply to my post.

Q: What kind of ICMP packets are these?

A: ICMP Type 3, or ICMP unreachable packets.

Q: What level of logging do you have defined on the PIX?

A: The level of logging is trap errors (logging trap 3) and timestamp logging. (logging on, logging timestamp, logging trap errors).

Q: How busy is your PIX?

A-1: Before trap error logging is enable on the PIX 515E, the sh cpu usage command displays the following CPU usage: CPU utilization for 5 seconds = 0%; 1 minute: 0%; 5 minutes: 2% (Note: syslog daemon not running on log server and the logging traps errors was removed from the PIX configuration).

A-2: When "logging trap 3" is configured on the PIX, CPU utilization on the PIX was elevated to 90-93% utilization: CPU utilization for 5 seconds = 93%; 1 minute: 93%; 5 minutes: 91%. (Note: syslog daemon not running on log server).

Q: Did you actually see ICMP packets going in both directions, or did you see UDP from PIX -> syslog server and ICMP from syslog server -> PIX?

A: I guess I am only reading ICMP unreachable messages being sent from the log server to the console (313001: Denied ICMP type=3, code=3 from X.X.X.X on interface 1 - where X.X.X.X equals the IP address of the log server). Note: This message repeats over and over again, until the no logging console command is issued via an ssh session.

Q: Did this actually cause the PIX to stop passing other traffic?

A: Yes, some traffic. Legitimate traffic destined to the Internet came to a crawl, and in some instances users could not access websites. SMTP email relay came to a crawl. VPN connections through the PIX could not be established at all.

My company's 10Mb Ethernet connection to the Internet seemed more like a 64Kb frame-relay connection or dial-up connection to the Internet while the PIX CPU was elevated between 90-93% utilization.

Please note the below sh proc command ran on the PIX while the processor utilization was running at a constant 90-93%: (review the Logger process).

PC SP STATE Runtime SBASE Stack Process

Hsi 800b0e09 807d38c8 8052ddd8 0 807d2940 3716/4096 arp_timer

Lsi 800b5271 808369d0 8052ddd8 20 80835a58 3800/4096 FragDBGC

Cwe 8000a945 80b51638 80375d90 0 80b506d0 3944/4096 CryptIC PDR poll

Lwe 8000f9fe 80b525d8 80531508 0 80b51760 3704/4096 dbgtrace

Lrd 8021b441 80b54548 8052e288 884880 80b527f0 6384/8192 Logger

Hwe 8020a550 80b57800 805075b0 0 80b55888 7772/8192 tcp_fast

_process

Lsi 80256f4d 8112bf10 8052ddd8 10 8112af88 3736/4096 route_process

I hope the additional information provides answers to the posted questions.

If the above situation is not a form of Denial-of -Service attack, how should it be defined?

-An administration setup/maintenance error? Perhaps.

-A daemon issue? Perhaps.

-An extraordinary limitation of services where Internet resources cannot be reached at times? Absolutely.

[mostiguy]

I don't see what the big deal is here. When logging is enabled, it logs events. When the logging service is down, I'd bet that the number of events increased geometrically due to the ICMP errors, as each ICMP error generates another one.

Don't let your logging service go down. If your logging service goes down, it isn't worth anything, as 99% of logs won't prove anything, so don't bother logging if you can't keep it up.

[davelockerby]

I whole-heartedly agree that it is absolutely critical to keep log servers available for logging system events and messages at all times. I also agree that the condition created might not fit a text-book definition of an DOS attack, however the scenario comes just shy of denying all traffic from passing through the PIX.

The prevailing reason to keep a syslog server available at all times should be based on the importance of logging system messages and not on preventing the PIX from becoming swampped with IMCP unreachable messages.

Instead of taking the "MS approach" on this issue and blame the problem on another technology, Cisco should research adding features in the PIX FOS to prevent the PIX from being inundated with ICMP unreachable messages in the event that a syslog server becomes uuavailable.

Cisco is admitting, via building certain functionality into the PIX, that systems do experience problems and do become unavailable at times. Why else would the url-server command have a syntax option to re-direct web requests to additional Websense servers?

At the very least, this ongoing discussion brings forward the extreme importance of keeping log servers available at all times. It is obvious that log servers should remain available to log crucial system events and messages. Given the scenario where a syslog server becomes unavailable and a PIX firewall becomes inundated with IMCP unreachable messages provides another reason for keeping log servers available at all times.

[mpalardy]

I agree with everybody syslog server should never gone 'out to coffee'. Tell this to your server management team.

Here's the command to use to prevent high CPU on pix when syslog server comes down. So no everlasting replies are sent from PIX to server and vice-versa.

icmp permit host your-ip-syslog-server unreachable INSIDE

Also, you may dedicate an interface on your PIX where you will place your syslog server and send PIX logs via this interface. So in case your inside network becomes overloaded, all your syslog will continue to flow to this interface.

Mike