PIX 515 hangs

malcorn · ‎03-30-2005

I have a 515E firewall that about every two weeks just hangs. I heard through our T-1 provider that there was a field notice or recall out on this issue, but can't find anything that is later than May 2002. Is there something newer to review?

Thanks

fedrodri · ‎04-01-2005

Hi,

You might be refering to the following Field Notice:

-- Field Notice: PIX 515 and 506 Hang:

http://www.cisco.com/en/US/products/hw/vpndevc/ps2030/products_field_notice09186a00800949c7.shtml

The problem has been resolved on the production line and units manufactured as of October, 2001 does not present this problem (traffic related!).

Now, please note that the FN applies to PIX-515, and not to PIX-515Es... If your PIX is hanging every two weeks or so, how do you resolve the problem? I guess that simply by re-loading it, right? When it does hang, do you have console access to it? If you do, could you enter the commands "clear xlate" and "clear local-host", and see if traffic resumes? Before doing so, capture a 'show tech' from the PIX (I'm wondering if the PIX memory is just filled out!).

Thanks,

Federico Rodriguez

malcorn · ‎04-01-2005

I have verified that the PIX 515E is not related to the field notice. When it does hang, there is no access except for rebooting since I am not at the location. It is sitting in a co-lo environment.

here is a sample of the "show tech"while PIX is OK

Hardware: PIX-515E, 32 MB RAM, CPU Pentium II 433 MHz

Flash E28F128J3 @ 0x300, 16MB

BIOS Flash AM29F400B @ 0xfffd8000, 32KB

0: ethernet0: address is 000c.ce7d.698d, irq 10

1: ethernet1: address is 000c.ce7d.698e, irq 11

2: ethernet2: address is 0002.b3cd.98f1, irq 11

Licensed Features:

Failover: Disabled

VPN-DES: Enabled

VPN-3DES-AES: Disabled

Maximum Physical Interfaces: 3

Maximum Interfaces: 5

Cut-through Proxy: Enabled

Guards: Enabled

URL-filtering: Enabled

Inside Hosts: Unlimited

Throughput: Unlimited

IKE peers: Unlimited

This PIX has a Restricted (R) license.

Serial Number: 807182641 (0x301ca131)

Running Activation Key:

Configuration last modified by enable_15 at 21:13:26.738 UTC Fri Apr 1 2005

------------------ show clock ------------------

21:15:23.710 UTC Fri Apr 1 2005

------------------ show memory ------------------

Free memory: 14753664 bytes

Used memory: 18800768 bytes

------------- ----------------

Total memory: 33554432 bytes

------------------ show conn count ------------------

79 in use, 3638 most used

------------------ show xlate count ------------------

78 in use, 675 most used

------------------ show blocks ------------------

SIZE MAX LOW CNT

4 1600 1596 1599

80 400 397 400

256 2036 1708 2036

1550 1189 620 695

2560 200 193 198

------------------ show interface ------------------

fedrodri · ‎04-01-2005

Hi,

It would have been nice to get a 'show tech' from when the PIX hangs; what is the version that you're running; it was cut-off from the 'show tech' you sent... Now, what about if you do a 'show local-host'? What I would be looking for is for unusual entries on this table (it holds both xlate and conn table, on a per host basis), like per example connections or xlates that have been idle for more time than what you have specified on the 'timeout xlate' and 'timeout conn'... I have a hintch it is an xlate/conn related problem, and not with your box (more like a sofware bug)...

Are the hangs related to certain heavy traffic conditions? Keep monitoring the 1550 blocks (see if they reach 0); those memory blocks are for Ethernet packets storage before sending them to the PIX OS for processing! It's bad if they reach zero :0(, or could be that you're PIX is overwhelmed with traffic!

Hope that helps! And definitely, try getting console access when the problem happens and capture a 'show tech' and 'show local-host'; this will help a lot.

Best regards,

Federico Rodriguez

malcorn · ‎04-01-2005

PIX version 6.3

The only problem is that something on my network is using an old set of IP addresses that we used from a previous ISP 209.152.196.0/128

These were some printers:

local host: <192.168.10.61>,

TCP connection count/limit = 0/unlimited

TCP embryonic count = 0

TCP intercept watermark = unlimited

UDP connection count/limit = 1/unlimited

AAA:

Xlate(s):

Global 146.145.??.?? Local 192.168.10.61

Conn(s):

UDP out 209.152.196.30:161 in 192.168.10.61:1032 idle 0:01:41 flags -

fedrodri · ‎04-02-2005

It would seem that printer is communicating with that outside host via SNMP... but, the translation is made correctly; it is not being translated to something on the old IP address space given by the previous ISP... So this 'local-host' entry seems to be fine! Are there any local hosts being translated to something on the old IP address space?

Is that 146.145.x.x an IP address from a global Pool? Have you verify that you have at least one PAT address for once the global pool is exhausted? Could you confirm whether all traffic stops or if it is only for certain hosts that inbound/outbound traffic will not work? When the PIX hangs, I mean... Then PIX 6.3.? is the version? I was thinking of this bug, perhaps (?):

-- CSCdy58717 Bug Details: xlate table does not timeout entries.Need clear xlate to work:

http://www.cisco.com/cgi-bin/Support/Bugtool/onebug.pl?bugid=CSCdy58717

I've seen some behaviors similar to this on version 6.3.1, or look into this one (it could eventually fill out the memory if the DNS connections are not cleared):

-- CSCec45748 Bug Details: New DNS conns reset the idle timer of previous DNS conns.

So far, this is all that I can tell you. There is not enough information to tell exactly what's going on. You may want to open a TAC case as well, but you're gonna be asked for the same information: show tech from the time the problem happens and other things! See if you have anything saved in flash: 'show crashinfo', there might be something there that could help.

Best regards,

Federico Rodriguez

malcorn · ‎04-03-2005

I have tried to get to both of the bug details, but the links don't work. I am attaching a crashinfo to this message. Maybe this will help.

fedrodri · ‎04-04-2005

Hi! Thanks for the information... I'm looking at it and let you know as soon as I know...

Thanks!

Federico Rodriguez