PIX 501 performance slow down 1-3 hours after reboot

lpaster · ‎03-10-2003

the configuration:

I have a PIX 501.

The PIX external interface is on a DSL with fixed IP.

The PIX internal interface is on LAN with 4 PC's. There is a lot of traffic on the LAN between 3 of the PC's (digital video recorders) and the 4th PC that is used to view the video.

the problem:

after PIX is rebooted, it works fast and I can access the LAN through the PIX from the internet and get data (web, ftp, video) on internal boxes using port forwarding and/or PPTP VPN.

after some time (minutes to hours) the connection speed goes very very slow, but the connection is still there: I can get the data but it takes forever, very slow - basically un-usable.

problem is gone when PIX is rebooted again...

I have similar systems and configs in other locations and NO PROBLEMS at all.

anyone has any ideas???

gfullage · ‎03-10-2003

Difficult to say what's going on here.

Do normal ping packet responses slow down at the same time, or are they still OK? What is the CPU util on the PIX when it slows down? Are new connections still able to be made inbound? And are they also slow?

Try and do as much testing as you can and let us know the results, maybe that'll point us in the right direction.

lpaster · ‎03-11-2003

the PIX is in the "slow" state.

here is the ping results from my computer --> PPTP VPN --> Internet --> PIX --> a box behind PIX:

C:\Documents and Settings\lpaster>ping 10.100.135.100

Pinging 10.100.135.100 with 32 bytes of data:

Reply from 10.100.135.100: bytes=32 time=221ms TTL=128

Reply from 10.100.135.100: bytes=32 time=220ms TTL=128

Reply from 10.100.135.100: bytes=32 time=221ms TTL=128

Reply from 10.100.135.100: bytes=32 time=220ms TTL=128

Ping statistics for 10.100.135.100:

Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:

Minimum = 220ms, Maximum = 221ms, Average = 220ms

the inbound connections can still go through but they are very very slow.

getting to the PIX itself (for example opening PDM from over the internet) is still quick and respomnsive.

Result of PIX command: "show cpu usage" -

CPU utilization for 5 seconds = 36%; 1 minute: 36%; 5 minutes: 37%

Result of PIX command: "show conn" -

4 in use, 11 most used

TCP out 10.100.135.200:3953 in DP-9511-035:80 idle 0:06:30 Bytes 12726 flags UIOB

TCP out 198.100.100.29:4021 in DP-9511-036:1999 idle 0:00:03 Bytes 0 flags SaAB

TCP out 10.100.135.200:3954 in DP-9511-035:80 idle 0:00:21 Bytes 9437 flags UIOB

TCP out 198.100.100.29:4019 in DP-9511-035:1999 idle 0:00:07 Bytes 0 flags UFRB

Result of PIX command: "show xlate" -

0 in use, 2 most used

Result of PIX command: "show interface" -

interface ethernet0 "outside" is up, line protocol is up

Hardware is i82559 ethernet, address is 000b.be1c.fc9b

IP address XXXXXXXXXXXX, subnet mask 255.255.255.248

MTU 1500 bytes, BW 10000 Kbit half duplex

15981682 packets input, 1125348876 bytes, 0 no buffer

Received 18941 broadcasts, 0 runts, 0 giants

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

16015855 packets output, 2386941351 bytes, 0 underruns

0 output errors, 5916 collisions, 0 interface resets

0 babbles, 0 late collisions, 11517 deferred

1 lost carrier, 0 no carrier

input queue (curr/max blocks): hardware (128/128) software (0/30)

output queue (curr/max blocks): hardware (1/32) software (0/3)

interface ethernet1 "inside" is up, line protocol is up

Hardware is i82559 ethernet, address is 000b.be1c.fc9c

IP address 10.100.135.1, subnet mask 255.255.255.0

MTU 1500 bytes, BW 10000 Kbit full duplex

193509 packets input, 61111339 bytes, 0 no buffer

Received 2270 broadcasts, 0 runts, 0 giants

2600 input errors, 2600 CRC, 0 frame, 0 overrun, 2600 ignored, 0 abort

165872 packets output, 10366259 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 babbles, 0 late collisions, 0 deferred

0 lost carrier, 0 no carrier

input queue (curr/max blocks): hardware (128/128) software (0/21)

output queue (curr/max blocks): hardware (0/6) software (0/1)

lpaster · ‎03-11-2003

after reboot here are the command results:

Result of PIX command: "show cpu usage"

CPU utilization for 5 seconds = 15%; 1 minute: 17%; 5 minutes: 17%

Result of PIX command: "show conn"

2 in use, 8 most used

TCP out 198.100.100.29:4860 in DP-9511-035:1999 idle 0:00:00 Bytes 0 flags UFRB

TCP out 10.100.135.200:4337 in DP-9511-035:1999 idle 0:00:00 Bytes 7351943 flags UIOB

Result of PIX command: "show xlate"

0 in use, 0 most used

Result of PIX command: "show interface"

interface ethernet0 "outside" is up, line protocol is up

Hardware is i82559 ethernet, address is 000b.be1c.fc9b

IP address XXXXXXXXXXXXXXX, subnet mask 255.255.255.248

MTU 1500 bytes, BW 10000 Kbit half duplex

203269 packets input, 14406295 bytes, 0 no buffer

Received 351 broadcasts, 0 runts, 0 giants

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

205586 packets output, 38981566 bytes, 0 underruns

0 output errors, 229 collisions, 0 interface resets

0 babbles, 0 late collisions, 140 deferred

1 lost carrier, 0 no carrier

input queue (curr/max blocks): hardware (128/128) software (0/26)

output queue (curr/max blocks): hardware (1/43) software (0/3)

interface ethernet1 "inside" is up, line protocol is up

Hardware is i82559 ethernet, address is 000b.be1c.fc9c

IP address 10.100.135.1, subnet mask 255.255.255.0

MTU 1500 bytes, BW 10000 Kbit full duplex

6013 packets input, 7950990 bytes, 0 no buffer

Received 50 broadcasts, 0 runts, 0 giants

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

3911 packets output, 248989 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 babbles, 0 late collisions, 0 deferred

0 lost carrier, 0 no carrier

input queue (curr/max blocks): hardware (128/128) software (0/7)

output queue (curr/max blocks): hardware (0/4) software (0/1)

gfullage · ‎03-11-2003

Still difficult to pinpoint what's going wrong. The xlates and conns seem fine, although the internal host 10.100.135.200 seems fairly busy after the reboot:

TCP out 10.100.135.200:4337 in DP-9511-035:1999 idle 0:00:00 Bytes 7351943 flags UIOB

After the reboot there's a lot more traffic hitting the outside interface of the PIX than seems to be going inside, but I'm not sure if you ran PDM or telnet to the outside interface after the reboot which would explain it:

interface ethernet0 "outside" is up, line protocol is up

203269 packets input, 14406295 bytes, 0 no buffer

205586 packets output, 38981566 bytes, 0 underruns

interface ethernet1 "inside" is up, line protocol is up

6013 packets input, 7950990 bytes, 0 no buffer

3911 packets output, 248989 bytes, 0 underruns

This is also seen before the reboot

:

interface ethernet0 "outside" is up, line protocol is up

15981682 packets input, 1125348876 bytes, 0 no buffer

16015855 packets output, 2386941351 bytes, 0 underruns

interface ethernet1 "inside" is up, line protocol is up

193509 packets input, 61111339 bytes, 0 no buffer

165872 packets output, 10366259 bytes, 0 underruns

Why is there around 3.4Gig of traffic hitting the outside, but only 71Meg hitting the inside? What's going on the outside of this PIX? What type of traffic are you using here, I see you have PPTP enabled, is that most of your traffic? Or would you expect more traffic to be outbound than inbound?

shannong · ‎03-11-2003

I agree with Mr GF. Look at it this way:

Before reboot:

In the outside Interface: 1125348876 bytes

Out the inside Interface: 10366259 bytes

If all traffic is valid and passed along, these would be about equal. Instead, we see a 100:1 ratio traffic. Either the firewall is denying alot of traffic...... Or you are syslogging, PDM monitoring, or other mechanism that causes the Pix to generate traffic on it's own? The Pix working hard for it's own processes can cause connection problems and latency.

Please check and verify this for us.... The problem might be the Pix working hard on something or receiving alot of invalid traffic.

lpaster · ‎03-12-2003

1. I was doing all tests from PDM interface so that is a reason why so much traffic on outside compared to inside.

BTW the PDM is never slow: the traffic is only slow when talking from outside to inside (either with VPN or port forwarding).

anyway, I wanted to run same tests today using SSH (so there's less traffic on external interface) but the system is not getting into "slow" mode for the last few hours, I'll have to wait until it slows down again probably soon.

2. host 10.100.135.200 is my box at remote location where I test from. this is the IP assigned to me after I connected with PPTP VPN.

3. the traffic through the PIX is mostly heartbit application from remote loction, once every few seconds pinging the internal systems.

sometimes we also move some files from internal to remote computer but not too often maybe once every 1-2 days.

4. one other thought I had - maybe some internal host can cause the problem somehow? I'm using 5 internal computers and they are connected through netgear hub that connects to PIX.

also just wanted to thank you all for working with me on this issue.

shannong · ‎03-10-2003

When "it" is happening, do a "show conn" and "show xlate" do see what your traffic levels are like. Also, a "show interface" to get an idea of throughput.

If you're using 6.2+, do a "show cpu usage" to see what the processor is doing at the time.

It's possible you have an infected host creating/attempting connections out to the Internet at a high rate.