Is my Pix 501 overloaded?

jeefskiii · ‎12-10-2004

Hello,

Have a Cisco Pix firewall in front of a co-located webserver and occasionaly we are dropping connections when connecting to http/s, ftp and pop3. I am trying to determine if the device is overloaded and can't handle the incoming trafic and we need to upgrade or change our configuration. I have output some show command info below in case it helps. show interface doesn't seem to be reporting any errors but maybe the memory is a bit high.

Any help much appreciated.

Can't get the attachment/uploader to work so will plit this in two.

_________________________________________

show cpu usage

CPU utilization for 5 seconds = 10%; 1 minute: 6%; 5 minutes: 5%

This is the highest I have seen it.

_________________________________________

show traffic

outside:

received (in 84285.100 secs):

4439775 packets 746363379 bytes

1 pkts/sec 8039 bytes/sec

transmitted (in 84285.100 secs):

5449513 packets 236389169 bytes

13 pkts/sec 2040 bytes/sec

inside:

received (in 84285.110 secs):

5485816 packets 241820288 bytes

14 pkts/sec 2002 bytes/sec

transmitted (in 84285.110 secs):

4302825 packets 737645278 bytes

0 pkts/sec 8038 bytes/sec

_________________________________________

show perfmon

PERFMON STATS: Current Average

Xlates 0/s 0/s

Connections 10/s 10/s

TCP Conns 2/s 2/s

UDP Conns 7/s 7/s

URL Access 1/s 1/s

URL Server Req 0/s 0/s

TCP Fixup 131/s 48/s

TCPIntercept 0/s 0/s

HTTP Fixup 98/s 31/s

FTP Fixup 0/s 0/s

AAA Authen 0/s 0/s

AAA Author 0/s 0/s

AAA Account 0/s 0/s

_________________________________________

show blocks

SIZE MAX LOW CNT

4 600 577 599

80 400 398 399

256 100 90 100

1550 933 584 673

_________________________________________

show memory

Free memory: 70200 bytes

Used memory: 16707016 bytes

------------- ----------------

Total memory: 16777216 bytes

_________________________________________

show xlate

26 in use, 26 most used

_________________________________________

show conn count

16123 in use, 16128 most used

jeefskiii · ‎12-10-2004

Part two, forgot to mention in the question - its a PIX 501.

_________________________________________

show interface

interface ethernet0 "outside" is up, line protocol is up

Hardware is i82559 ethernet, address is 0011.93f6.6c9a

IP address 111.222.333.444, subnet mask 255.255.255.192

MTU 1500 bytes, BW 100000 Kbit full duplex

4461785 packets input, 749900741 bytes, 0 no buffer

Received 20721 broadcasts, 0 runts, 0 giants

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

5476880 packets output, 259665243 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 babbles, 0 late collisions, 0 deferred

0 lost carrier, 0 no carrier

input queue (curr/max blocks): hardware (128/128) software (0/59)

output queue (curr/max blocks): hardware (0/50) software (0/1)

interface ethernet1 "inside" is up, line protocol is up

Hardware is i82559 ethernet, address is 0011.93f6.6c9b

IP address 192.168.1.1, subnet mask 255.255.255.0

MTU 1500 bytes, BW 100000 Kbit full duplex

5514534 packets input, 265259986 bytes, 0 no buffer

Received 35 broadcasts, 0 runts, 0 giants

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

4323851 packets output, 741095276 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 babbles, 0 late collisions, 0 deferred

0 lost carrier, 0 no carrier

input queue (curr/max blocks): hardware (128/128) software (0/50)

output queue (curr/max blocks): hardware (1/59) software (0/1)

_________________________________________

jeefskiii · ‎12-10-2004

Part 3 of 3.

show processes

PC SP STATE Runtime SBASE Stack Process

Hsi 001eaa09 007c0ccc 00555860 0 007bfd44 3628/4096 arp_timer

Lsi 001effad 007e3e04 00555860 0 007e2e8c 3816/4096 FragDBGC

Lwe 00119abf 00838844 00558fc0 0 008379dc 3688/4096 dbgtrace

Lwe 003e3f55 0083a9d4 0054e188 68400 00838a8c 7624/8192 Logger

Hsi 003e806d 0083dacc 00555860 10 0083bb54 7864/8192 tcp_fast

Hsi 003e7f0d 0083fb7c 00555860 0 0083dc04 7924/8192 tcp_slow

Lsi 003006f9 008bf054 00555860 0 008be0cc 3944/4096 xlate clean

Lsi 00300607 008c00f4 00555860 0 008bf17c 3548/4096 uxlate clean

Mwe 002f82d3 008e8cf4 00555860 10 008e6d5c 7776/8192 tcp_intercept_timer_process

Lsi 0043a545 008f95f4 00555860 0 008f866c 3900/4096 route_process

Hsi 002e80f4 008fa684 00555860 7720 008f971c 2456/4096 PIX Garbage Collector

Hwe 00217101 008feb74 00555860 0 008fac0c 16048/16384 isakmp_time_keeper

Lsi 002e5e74 0090f94c 00555860 0 0090e9c4 3944/4096 perfmon

Mwe 0020e719 0091b41c 00555860 0 009194a4 7860/8192 IPsec timer handler

Mwe 00261395 0094a98c 00555860 10 00946a24 15592/16384 IP Background

Lwe 002f8f4a 009fd09c 0056bc98 0 009fc224 3704/4096 pix/trace

Lwe 002f9182 009fe14c 0056c3c8 0 009fd2d4 3704/4096 pix/tconsole

Hwe 0011f217 00a0802c 00502bc0 150 00a04564 13408/16384 ci/console

Csi 002f0fd3 00a0956c 00555860 10 00a08614 3432/4096 update_cpu_usage

Hwe 002dcba1 00a2e0fc 00534c00 0 00a2a274 15884/16384 uauth_in

Hwe 003e6b5d 00a301fc 007fb6a0 0 00a2e324 7896/8192 uauth_thread

Hwe 003fce0a 00a3134c 0054e788 0 00a303d4 3960/4096 udp_timer

Hsi 001e2636 00a3300c 00555860 0 00a32094 3928/4096 557mcfix

Crd 001e25eb 00a340cc 00555cd8 50084100 00a33144 3640/4096 557poll

Lsi 001e26a5 00a3516c 00555860 0 00a341f4 3848/4096 557timer

Cwe 001e4229 00a4b244 006cf448 989650 00a4934c 6136/8192 pix/intf0

Mwe 003fcb7a 00a4c354 00835fd0 0 00a4b41c 3896/4096 riprx/0

Msi 003a3999 00a4d464 00555860 0 00a4c4ec 3888/4096 riptx/0

Cwe 001e4229 00a535fc 007449b8 1299520 00a51704 6136/8192 pix/intf1

Mwe 003fcb7a 00a5470c 00835f88 0 00a537d4 3896/4096 riprx/1

Msi 003a3999 00a5581c 00555860 0 00a548a4 3888/4096 riptx/1

Hwe 003e6df1 00a7dbfc 007e6aa8 0 00a7d954 284/1024 listen/http0

Hwe 003e6df1 00a7e0dc 007e6ba0 0 00a7de34 284/1024 listen/http1

Hwe 003e6df1 00a7e604 007e6c98 0 00a7e3bc 172/1024 listen/pfm

Hwe 003e6df1 00a7eeb4 007e6f80 0 00a7e86c 1196/2048 listen/telnet_1

Hwe 003e6df1 00a7f764 007e6e88 0 00a7f11c 1196/2048 listen/ssh_0

Hwe 003e6df1 00a800d4 007e7170 0 00a7fa8c 1196/2048 listen/ssh_1

Mwe 00370852 00a8265c 00555860 780 00a806e4 5476/8192 Crypto CA

Mwe 003e0b11 00925b44 00555860 0 00923bcc 6440/8192 ssh/timer

H* 003e77c7 0009ff2c 00555848 3550 00fdc4e4 3844/8192 telnet/ci

scottmac · ‎12-10-2004

The quick & easy visual verification would be to bring up the PDM and check the utilization graphs over time.

PDM is a Java applet and can be reached by browsing the internal address of the PIX. You must use "https" (note, the "s' at the end).

There are a number of graphs presented including traffic and utilization.

At 5% - 10% processor utilization, it probably not overloaded.

What flavor of license does this Pix have? The standard unit will only allow 10 concurrent users (a fifty user license is available for upgrade).

Is it possible you are hitting the max concurrent user limit?

Check it out and let us know ....

Scott

jeefskiii · ‎12-10-2004

Thanks for that, I will check the graph stuff out just now.

One thing I never quite got my head around was the user licenses. I am not sure whether this is relevant to they way we use the firewall. We basicaly use it to protect our web server and all the ports that are not in use and for NAT. Any one can connect to the server via HTTP for example and I suspect that at anyone time there are more than ten people - 20 to 30 I would estimate.

Am I right in thinking the licence is for VPN connections - we don't use this feature, but I think we just have the standard ten licences.

The firewall specs say that it allows 7000 concurrent connections, I thought maybe this is the figure we are more interested in. What counts as a connection and how do we know how many we have?

Still a bit confused.

jeefskiii · ‎12-10-2004

How do I connect to the PDM remotely via a web browser? I can't connect to the internal network from the outside the way I have things set up. I can only connect via ssh. I only have a pix in front of the server not localy in my location.

scottmac · ‎12-10-2004

To allow a PDM connection from the outside interface could be a security risk.

The command would be "pdm location outside" (or inside with an inside address).

Can you bring up a graphical session on the web server? If you can do a remote desktop or X on the server, then initiate the browser session from the inside (like: ssh to the server, bring up X, start a browser and aim it at the inside interface of the Pix - don't forget it's SSL - you need httpS://...).

Good Luck

Scott

thomam04 · ‎12-11-2004

Hello,

the memory usage concerns me a bit. There is only 70KB of memory left...but I think the real problem is the amount of connections in use. 16123 connections is use? That's a lot for a PIX 501. Do a "sh conn detail" and verify that the connections in use are legitimate. One thing I would definitively recommend is upgrading the memory to at least 32MB/64MB. The PIX 501 has an AMD 133Mhz CPU so I would assume it's using standard PS/2 72pin memory...not quite sure though.

Hope this helps.

Regards,

-Markus

jeefskiii · ‎12-12-2004

sh conn detail shows thousands of UDP connections from the same IP address, looks a bit like this:

UDP outside:111.222.333.444/53 inside:192.168.1.2/53 flags -

...

There are about three IP addresses in total repeated thousands of times.

There are also about 30 other connections to TCP and UDP from various IPs which looks normal. I would estimate that we have around 30 simultaneous users looking at http at any one time.

scottmac · ‎12-12-2004

Port 53 is DNS. The three "users" are probably your local web servers.

You might want to investigate a cache for your site.... even if it's something like "squid" (public domain / free / *nix) or an internal caching DNS box.

FWIW

Scott

jeefskiii · ‎12-13-2004

I found this -

https://puck.nether.net/pipermail/cisco-nsp/2003-September/005888.html

I think there is some wierdness going on with the 6.3(3) OS. This forum suggested adding to the config:

no fixup protocol dns

then

clear xlate

I did this about five hours ago and since then connections have remained at around 120-140 and am now using about 10MB of memory instead of all of it. Normal service appears to be resumed.

Thanks everyone for your help, a windy path to get a fix but would probably never have gone in this particular direction without your input. Learnt a bit about PIX admin too.

thomam04 · ‎12-13-2004

Hi!

I did some more research on Cisco's site and there is a known bug : CSCec45748 (New DNS conns reset the idle timer of previous DNS conns) in Pix OS 6.3.3. This has been fixed in 6.3.4.

Regards,

-Markus