12-10-2004 04:05 AM - edited 02-20-2020 11:47 PM
Hello,
Have a Cisco Pix firewall in front of a co-located webserver and occasionaly we are dropping connections when connecting to http/s, ftp and pop3. I am trying to determine if the device is overloaded and can't handle the incoming trafic and we need to upgrade or change our configuration. I have output some show command info below in case it helps. show interface doesn't seem to be reporting any errors but maybe the memory is a bit high.
Any help much appreciated.
Can't get the attachment/uploader to work so will plit this in two.
_________________________________________
show cpu usage
CPU utilization for 5 seconds = 10%; 1 minute: 6%; 5 minutes: 5%
This is the highest I have seen it.
_________________________________________
show traffic
outside:
received (in 84285.100 secs):
4439775 packets 746363379 bytes
1 pkts/sec 8039 bytes/sec
transmitted (in 84285.100 secs):
5449513 packets 236389169 bytes
13 pkts/sec 2040 bytes/sec
inside:
received (in 84285.110 secs):
5485816 packets 241820288 bytes
14 pkts/sec 2002 bytes/sec
transmitted (in 84285.110 secs):
4302825 packets 737645278 bytes
0 pkts/sec 8038 bytes/sec
_________________________________________
show perfmon
PERFMON STATS: Current Average
Xlates 0/s 0/s
Connections 10/s 10/s
TCP Conns 2/s 2/s
UDP Conns 7/s 7/s
URL Access 1/s 1/s
URL Server Req 0/s 0/s
TCP Fixup 131/s 48/s
TCPIntercept 0/s 0/s
HTTP Fixup 98/s 31/s
FTP Fixup 0/s 0/s
AAA Authen 0/s 0/s
AAA Author 0/s 0/s
AAA Account 0/s 0/s
_________________________________________
show blocks
SIZE MAX LOW CNT
4 600 577 599
80 400 398 399
256 100 90 100
1550 933 584 673
_________________________________________
show memory
Free memory: 70200 bytes
Used memory: 16707016 bytes
------------- ----------------
Total memory: 16777216 bytes
_________________________________________
show xlate
26 in use, 26 most used
_________________________________________
show conn count
16123 in use, 16128 most used
12-10-2004 04:09 AM
Part two, forgot to mention in the question - its a PIX 501.
_________________________________________
show interface
interface ethernet0 "outside" is up, line protocol is up
Hardware is i82559 ethernet, address is 0011.93f6.6c9a
IP address 111.222.333.444, subnet mask 255.255.255.192
MTU 1500 bytes, BW 100000 Kbit full duplex
4461785 packets input, 749900741 bytes, 0 no buffer
Received 20721 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
5476880 packets output, 259665243 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collisions, 0 deferred
0 lost carrier, 0 no carrier
input queue (curr/max blocks): hardware (128/128) software (0/59)
output queue (curr/max blocks): hardware (0/50) software (0/1)
interface ethernet1 "inside" is up, line protocol is up
Hardware is i82559 ethernet, address is 0011.93f6.6c9b
IP address 192.168.1.1, subnet mask 255.255.255.0
MTU 1500 bytes, BW 100000 Kbit full duplex
5514534 packets input, 265259986 bytes, 0 no buffer
Received 35 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
4323851 packets output, 741095276 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collisions, 0 deferred
0 lost carrier, 0 no carrier
input queue (curr/max blocks): hardware (128/128) software (0/50)
output queue (curr/max blocks): hardware (1/59) software (0/1)
_________________________________________
12-10-2004 04:10 AM
Part 3 of 3.
show processes
PC SP STATE Runtime SBASE Stack Process
Hsi 001eaa09 007c0ccc 00555860 0 007bfd44 3628/4096 arp_timer
Lsi 001effad 007e3e04 00555860 0 007e2e8c 3816/4096 FragDBGC
Lwe 00119abf 00838844 00558fc0 0 008379dc 3688/4096 dbgtrace
Lwe 003e3f55 0083a9d4 0054e188 68400 00838a8c 7624/8192 Logger
Hsi 003e806d 0083dacc 00555860 10 0083bb54 7864/8192 tcp_fast
Hsi 003e7f0d 0083fb7c 00555860 0 0083dc04 7924/8192 tcp_slow
Lsi 003006f9 008bf054 00555860 0 008be0cc 3944/4096 xlate clean
Lsi 00300607 008c00f4 00555860 0 008bf17c 3548/4096 uxlate clean
Mwe 002f82d3 008e8cf4 00555860 10 008e6d5c 7776/8192 tcp_intercept_timer_process
Lsi 0043a545 008f95f4 00555860 0 008f866c 3900/4096 route_process
Hsi 002e80f4 008fa684 00555860 7720 008f971c 2456/4096 PIX Garbage Collector
Hwe 00217101 008feb74 00555860 0 008fac0c 16048/16384 isakmp_time_keeper
Lsi 002e5e74 0090f94c 00555860 0 0090e9c4 3944/4096 perfmon
Mwe 0020e719 0091b41c 00555860 0 009194a4 7860/8192 IPsec timer handler
Mwe 00261395 0094a98c 00555860 10 00946a24 15592/16384 IP Background
Lwe 002f8f4a 009fd09c 0056bc98 0 009fc224 3704/4096 pix/trace
Lwe 002f9182 009fe14c 0056c3c8 0 009fd2d4 3704/4096 pix/tconsole
Hwe 0011f217 00a0802c 00502bc0 150 00a04564 13408/16384 ci/console
Csi 002f0fd3 00a0956c 00555860 10 00a08614 3432/4096 update_cpu_usage
Hwe 002dcba1 00a2e0fc 00534c00 0 00a2a274 15884/16384 uauth_in
Hwe 003e6b5d 00a301fc 007fb6a0 0 00a2e324 7896/8192 uauth_thread
Hwe 003fce0a 00a3134c 0054e788 0 00a303d4 3960/4096 udp_timer
Hsi 001e2636 00a3300c 00555860 0 00a32094 3928/4096 557mcfix
Crd 001e25eb 00a340cc 00555cd8 50084100 00a33144 3640/4096 557poll
Lsi 001e26a5 00a3516c 00555860 0 00a341f4 3848/4096 557timer
Cwe 001e4229 00a4b244 006cf448 989650 00a4934c 6136/8192 pix/intf0
Mwe 003fcb7a 00a4c354 00835fd0 0 00a4b41c 3896/4096 riprx/0
Msi 003a3999 00a4d464 00555860 0 00a4c4ec 3888/4096 riptx/0
Cwe 001e4229 00a535fc 007449b8 1299520 00a51704 6136/8192 pix/intf1
Mwe 003fcb7a 00a5470c 00835f88 0 00a537d4 3896/4096 riprx/1
Msi 003a3999 00a5581c 00555860 0 00a548a4 3888/4096 riptx/1
Hwe 003e6df1 00a7dbfc 007e6aa8 0 00a7d954 284/1024 listen/http0
Hwe 003e6df1 00a7e0dc 007e6ba0 0 00a7de34 284/1024 listen/http1
Hwe 003e6df1 00a7e604 007e6c98 0 00a7e3bc 172/1024 listen/pfm
Hwe 003e6df1 00a7eeb4 007e6f80 0 00a7e86c 1196/2048 listen/telnet_1
Hwe 003e6df1 00a7f764 007e6e88 0 00a7f11c 1196/2048 listen/ssh_0
Hwe 003e6df1 00a800d4 007e7170 0 00a7fa8c 1196/2048 listen/ssh_1
Mwe 00370852 00a8265c 00555860 780 00a806e4 5476/8192 Crypto CA
Mwe 003e0b11 00925b44 00555860 0 00923bcc 6440/8192 ssh/timer
H* 003e77c7 0009ff2c 00555848 3550 00fdc4e4 3844/8192 telnet/ci
12-10-2004 09:05 AM
The quick & easy visual verification would be to bring up the PDM and check the utilization graphs over time.
PDM is a Java applet and can be reached by browsing the internal address of the PIX. You must use "https" (note, the "s' at the end).
There are a number of graphs presented including traffic and utilization.
At 5% - 10% processor utilization, it probably not overloaded.
What flavor of license does this Pix have? The standard unit will only allow 10 concurrent users (a fifty user license is available for upgrade).
Is it possible you are hitting the max concurrent user limit?
Check it out and let us know ....
Scott
12-10-2004 10:21 AM
Thanks for that, I will check the graph stuff out just now.
One thing I never quite got my head around was the user licenses. I am not sure whether this is relevant to they way we use the firewall. We basicaly use it to protect our web server and all the ports that are not in use and for NAT. Any one can connect to the server via HTTP for example and I suspect that at anyone time there are more than ten people - 20 to 30 I would estimate.
Am I right in thinking the licence is for VPN connections - we don't use this feature, but I think we just have the standard ten licences.
The firewall specs say that it allows 7000 concurrent connections, I thought maybe this is the figure we are more interested in. What counts as a connection and how do we know how many we have?
Still a bit confused.
12-10-2004 10:55 AM
How do I connect to the PDM remotely via a web browser? I can't connect to the internal network from the outside the way I have things set up. I can only connect via ssh. I only have a pix in front of the server not localy in my location.
12-10-2004 11:38 AM
To allow a PDM connection from the outside interface could be a security risk.
The command would be "pdm location
Can you bring up a graphical session on the web server? If you can do a remote desktop or X on the server, then initiate the browser session from the inside (like: ssh to the server, bring up X, start a browser and aim it at the inside interface of the Pix - don't forget it's SSL - you need httpS://...).
Good Luck
Scott
12-11-2004 02:09 PM
Hello,
the memory usage concerns me a bit. There is only 70KB of memory left...but I think the real problem is the amount of connections in use. 16123 connections is use? That's a lot for a PIX 501. Do a "sh conn detail" and verify that the connections in use are legitimate. One thing I would definitively recommend is upgrading the memory to at least 32MB/64MB. The PIX 501 has an AMD 133Mhz CPU so I would assume it's using standard PS/2 72pin memory...not quite sure though.
Hope this helps.
Regards,
-Markus
12-12-2004 07:32 AM
sh conn detail shows thousands of UDP connections from the same IP address, looks a bit like this:
UDP outside:111.222.333.444/53 inside:192.168.1.2/53 flags -
UDP outside:111.222.333.444/53 inside:192.168.1.2/53 flags -
UDP outside:111.222.333.444/53 inside:192.168.1.2/53 flags -
UDP outside:111.222.333.444/53 inside:192.168.1.2/53 flags -
...
There are about three IP addresses in total repeated thousands of times.
There are also about 30 other connections to TCP and UDP from various IPs which looks normal. I would estimate that we have around 30 simultaneous users looking at http at any one time.
12-12-2004 04:27 PM
Port 53 is DNS. The three "users" are probably your local web servers.
You might want to investigate a cache for your site.... even if it's something like "squid" (public domain / free / *nix) or an internal caching DNS box.
FWIW
Scott
12-13-2004 07:06 AM
I found this -
https://puck.nether.net/pipermail/cisco-nsp/2003-September/005888.html
I think there is some wierdness going on with the 6.3(3) OS. This forum suggested adding to the config:
no fixup protocol dns
then
clear xlate
I did this about five hours ago and since then connections have remained at around 120-140 and am now using about 10MB of memory instead of all of it. Normal service appears to be resumed.
Thanks everyone for your help, a windy path to get a fix but would probably never have gone in this particular direction without your input. Learnt a bit about PIX admin too.
12-13-2004 07:25 AM
Hi!
I did some more research on Cisco's site and there is a known bug : CSCec45748 (New DNS conns reset the idle timer of previous DNS conns) in Pix OS 6.3.3. This has been fixed in 6.3.4.
Regards,
-Markus
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide