12-09-2010 07:23 AM - edited 03-11-2019 12:20 PM
I am running a Cisco PIX Firewall Version 6.3(5) that is randomly rebooting on me and looking to work out the root cause. Any help would be much apprecated.
------------------------------
Traceback:
0: 002fb6b8
1: 00103b6d
2: 00000000
vector 0x0000000e (page fault)
edi 0x00000011
esi 0x00fa12e4
ebp 0x00dab378
esp 0x00dab350
ebx 0x00fa3714
edx 0x031d6560
ecx 0x00fa37ec
eax 0x00120c0a
error code 0x00000000
eip 0x002fb43f
cs 0x00000008
eflags 0x00dab378
CR2 0x036f9170
Stack dump: base:0x00daa3f4 size:4096, active:292
0x00dab3f0: 0x00000000
0x00dab3ec: 0x002fb4a8
0x00dab3e8-0x00dab3e4: 0x00000000
0x00dab3e0-0x00dab3c4: 0x12345678
0x00dab3c0: 0x00000000
0x00dab3bc: 0x00103b6d
0x00dab3b8: 0x00dab3e4
0x00dab3b4: 0x0000000a
0x00dab3b0-0x00dab39c: 0x12345678
0x00dab398-0x00dab390: 0x00000000
0x00dab38c: 0x002fb4bf
0x00dab388: 0x00120c0a
0x00dab384: 0x00fa37ec
0x00dab380: 0x00000000
0x00dab37c: 0x002fb6b8
0x00dab378: 0x00dab3b8
0x00dab374-0x00dab370: 0x12345678
0x00dab36c: 0x0056ed50
0x00dab368: 0x00000066
0x00dab364: 0x00120c0a
0x00dab360: 0x00fa37ec
0x00dab35c: 0x00010216
0x00dab358: 0x00000008
0x00dab354: 0x002fb43f
0x00dab350: 0x00000000 *
0x00dab34c: 0x00120c0a
0x00dab348: 0x00fa37ec
0x00dab344: 0x031d6560
0x00dab340: 0x00fa3714
0x00dab33c: 0x00dab350
0x00dab338: 0x00dab378
0x00dab334: 0x00fa12e4
0x00dab330: 0x00000011
0x00dab32c: 0x0000000e
0x00dab328: 0x00105369
0x00dab324: 0x00dab378
0x00dab320: 0x00000008
0x00dab31c: 0x002fb43f
0x00dab318: 0x00000000
0x00dab314: 0x00120c0a
0x00dab310: 0x00fa37ec
0x00dab30c: 0x031d6560
0x00dab308: 0x00fa3714
0x00dab304: 0x00dab350
0x00dab300: 0x00dab378
0x00dab2fc: 0x00fa12e4
0x00dab2f8: 0x00000011
0x00dab2f4: 0x0000000e
0x00dab2f0: 0x00dab2f4
0x00dab2ec: 0x002f20a8
0x00dab2e8: 0x00dab328
0x00dab2e4: 0x00000011
0x00dab2e0: 0x00fa12e4
0x00dab2dc: 0x00fa3714
0x00dab2d8: 0x00dab378
0x00dab2d4: 0x00000000
0x00dab2d0: 0x00000002
Cisco PIX Firewall Version 6.3(5)
Cisco PIX Device Manager Version 1.1(2)
Compiled on Thu 04-Aug-05 21:40 by morlee
PIX515 up 19 mins 45 secs
Hardware: PIX-515, 32 MB RAM, CPU Pentium 200 MHz
Flash i28F640J5 @ 0x300, 16MB
BIOS Flash AT29C257 @ 0xfffd8000, 32KB
0: ethernet0: address is 0003.6bf7.5420, irq 11
1: ethernet1: address is 0003.6bf7.5421, irq 10
2: ethernet2: address is 0002.b397.2937, irq 9
Licensed Features:
Failover: Disabled
VPN-DES: Enabled
VPN-3DES-AES: Disabled
Maximum Physical Interfaces: 3
Maximum Interfaces: 5
Cut-through Proxy: Enabled
Guards: Enabled
URL-filtering: Enabled
Inside Hosts: Unlimited
Throughput: Unlimited
IKE peers: Unlimited
This PIX has a Restricted (R) license.
Serial Number: 406050023 (0x1833d4e7)
Running Activation Key: 0x6bea8a4e 0x96a90c43 0x90ee5890 0x07dcf802
Configuration has not been modified since last system restart.
------------------ show clock ------------------
11:38:20.587 UTC Thu Dec 9 2010
------------------ show memory ------------------
Free memory: 14429944 bytes
Used memory: 19124488 bytes
------------- ----------------
Total memory: 33554432 bytes
------------------ show conn count ------------------
1341 in use, 1993 most used
------------------ show xlate count ------------------
279 in use, 493 most used
------------------ show blocks ------------------
SIZE MAX LOW CNT
4 1600 1598 1600
80 400 398 400
256 2036 2024 2036
1550 1189 756 804
------------------ show interface ------------------
interface ethernet0 "outside" is up, line protocol is up
Hardware is i82559 ethernet, address is 0003.6bf7.5420
IP address X.X.X.X, subnet mask 255.255.255.192
MTU 1500 bytes, BW 100000 Kbit full duplex
173858 packets input, 123603899 bytes, 0 no buffer
Received 262 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
141930 packets output, 42506330 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collisions, 0 deferred
0 lost carrier, 0 no carrier
input queue (curr/max blocks): hardware (128/128) software (0/15)
output queue (curr/max blocks): hardware (0/16) software (0/1)
interface ethernet1 "inside" is up, line protocol is up
Hardware is i82559 ethernet, address is 0003.6bf7.5421
IP address X.X.X.X, subnet mask 255.255.255.252
MTU 1500 bytes, BW 100000 Kbit full duplex
243966 packets input, 154178549 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
239119 packets output, 130112713 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collisions, 0 deferred
0 lost carrier, 0 no carrier
input queue (curr/max blocks): hardware (128/128) software (0/16)
output queue (curr/max blocks): hardware (0/16) software (0/1)
interface ethernet2 "dmz" is up, line protocol is up
Hardware is i82559 ethernet, address is 0002.b397.2937
IP address X.X.X.X, subnet mask 255.255.255.0
MTU 1500 bytes, BW 10000 Kbit full duplex
68788 packets input, 10833900 bytes, 0 no buffer
Received 615 broadcasts, 103 runts, 0 giants
1032 input errors, 452 CRC, 477 frame, 0 overrun, 452 ignored, 0 abort
104108 packets output, 115769523 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collisions, 0 deferred
0 lost carrier, 0 no carrier
input queue (curr/max blocks): hardware (128/128) software (0/5)
output queue (curr/max blocks): hardware (0/48) software (0/1)
0 aggregate VLAN packets input, 0 bytes
1 aggregate VLAN packets output, 46 bytes
68772 native VLAN packets input, 10833220 bytes
104096 native VLAN packets output, 115735326 bytes
0 invalid VLAN ID errors
interface vlan165 "vlan165" is up, line protocol is up
Hardware is i82559 ethernet, address is 0002.b397.2937
IP address X.X.X.X, subnet mask 255.255.255.0
MTU 1500 bytes, BW 10000 Kbit full duplex
0 packets input, 0 bytes
1 packets output, 46 bytes
------------------ show cpu usage ------------------
CPU utilization for 5 seconds = 2%; 1 minute: 2%; 5 minutes: 2%
------------------ show process ------------------
PC SP STATE Runtime SBASE Stack Process
Hrd 001f02c9 0096f5fc 0056ed38 0 0096e674 3628/4096 arp_timer
Lsi 001f5a95 00a127f4 0056ed50 0 00a1187c 3928/4096 FragDBGC
Lwe 0011a13f 00a1e99c 005724b8 0 00a1db34 3688/4096 dbgtrace
Lwe 003fb2fd 00a20b2c 0056f218 1650 00a1ebe4 6600/8192 Logger
Hrd 003ff455 00a23c24 0056ed38 0 00a21cac 8024/8192 tcp_fast
Hrd 003ff2f5 00a25cd4 0056ed38 0 00a23d5c 8020/8192 tcp_slow
Lsi 00314885 00b5c454 0056ed50 0 00b5b4cc 3916/4096 xlate clean
Lsi 00314793 00b5d4f4 0056ed50 0 00b5c57c 3548/4096 uxlate clean
Mwe 0030be5f 00cfd8f4 0056ed50 0 00cfb95c 7908/8192 tcp_intercept_timer
_process
Lsi 00452ee5 00daa2cc 0056ed50 0 00da9344 3900/4096 route_process
H* 002fb6fc 0009ff2c 0056ed38 0 00daa3f4 1400/4096 PIX Garbage Collect
or
Hwe 0021e529 00db588c 0056ed50 0 00db1924 16048/16384 isakmp_time_keepe
r
Lsi 002f929c 00dcf1cc 0056ed50 0 00dce244 3944/4096 perfmon
Mwe 00214d39 00df95fc 0056ed50 0 00df7684 7860/8192 IPsec timer handler
Hrd 003b105b 00e0d944 0056ed38 0 00e0b9fc 7048/8192 qos_metric_daemon
Mwe 0026d0dd 00e2849c 0056ed50 0 00e24534 15592/16384 IP Background
Lwe 0030cad6 00edadec 00585368 0 00ed9f74 3704/4096 pix/trace
Lwe 0030cd0e 00edbe9c 00585a98 0 00edb024 3704/4096 pix/tconsole
Hrd 0011fa67 00ee7d74 0056ed38 0 00ee438c 14508/16384 ci/console
Crd 003048fb 00ee9394 0056f1c8 0 00ee843c 3540/4096 update_cpu_usage
Hwe 002ef791 00f9a13c 0054e100 0 00f962b4 15884/16384 uauth_in
Hwe 003fdf05 00f9c23c 008e6ac0 0 00f9a364 7896/8192 uauth_thread
Hwe 0041553a 00f9d38c 00567c88 0 00f9c414 3960/4096 udp_timer
Hrd 001e7d4e 00f9f04c 0056ed38 0 00f9e0d4 3928/4096 557mcfix
Crd 001e7d03 00fa010c 0056f1c8 784100 00f9f184 3584/4096 557poll
Lrd 001e7dbd 00fa11ac 0056f218 0 00fa0234 3848/4096 557timer
Cwe 001e99a9 00fb7284 00756ae0 6980 00fb538c 6264/8192 pix/intf0
Mwe 004152aa 00fb8394 00930cb0 0 00fb745c 3896/4096 riprx/0
Msi 003ba8a1 00fb94a4 0056ed50 0 00fb852c 3888/4096 riptx/0
Cwe 001e99a9 00fbf6ac 007ce078 9260 00fbd7b4 5952/8192 pix/intf1
Mwe 004152aa 00fc07bc 00930c68 0 00fbf884 3896/4096 riprx/1
Msi 003ba8a1 00fc18cc 0056ed50 0 00fc0954 3888/4096 riptx/1
Cwe 001e99a9 00fc7ad4 00845618 1780 00fc5bdc 6208/8192 pix/intf2
Mwe 004152aa 00fc8be4 00930c20 0 00fc7cac 3896/4096 riprx/2
Msi 003ba8a1 00fc9cf4 0056ed50 0 00fc8d7c 3888/4096 riptx/2
Mwe 004152aa 00fe6974 00930bd8 0 00fe5a3c 3896/4096 riprx/3
Msi 003ba8a1 00fe7a84 0056ed50 0 00fe6b0c 3888/4096 riptx/3
Hwe 003fe199 010ce034 008bd248 0 010cd98c 1308/2048 listen/http1
Hwe 003e4e45 010d0ba4 010d1054 0 010ced7c 7468/8192 isakmp_receiver
Hwe 003fe199 010d52f4 008bd628 0 010d4cac 1196/2048 listen/telnet_1
Hwe 003fe199 010d5c5c 008bd438 0 010d5614 1196/2048 listen/ssh_1
Hrd 00422ce7 010d7024 0056ed38 0 010d68cc 1844/2048 ppp_timer_thread
Hwe 00445fc4 011c82cc 0056ed50 0 011c7354 3684/4096 pptp_mgmt
Hwe 003e4e45 011ca214 011eecd4 0 011c842c 7100/8192 pptp_control/0
Hwe 00431aa5 011cb434 0056ed50 0 011ca4dc 3832/4096 L2TP data daemon
Hwe 00431895 011cc4f4 0056ed50 0 011cb58c 3848/4096 L2TP mgmt daemon
Mwe 0012d869 011f0c6c 0056ed50 0 011eecf4 7888/8192 DHCPD Timer
Mwe 004152aa 011f617c 00930b00 0 011f4244 7992/8192 dhcpd_recv/3
Mwe 0038707e 011f826c 0056ed50 0 011f62f4 7960/8192 Crypto CA
Hwe 003feba7 01218ba4 008bd340 20 0121726c 3908/8192 telnet/ci
------------------ show failover ------------------
No license for Failover
------------------ show traffic ------------------
outside:
received (in 1194.200 secs):
173858 packets 123603899 bytes
145 pkts/sec 103503 bytes/sec
transmitted (in 1194.200 secs):
141930 packets 42506330 bytes
118 pkts/sec 35593 bytes/sec
inside:
received (in 1194.410 secs):
243966 packets 154178549 bytes
204 pkts/sec 129083 bytes/sec
transmitted (in 1194.410 secs):
239119 packets 130112713 bytes
200 pkts/sec 108934 bytes/sec
dmz:
received (in 1194.630 secs):
68791 packets 10833900 bytes
57 pkts/sec 9068 bytes/sec
transmitted (in 1194.630 secs):
104108 packets 115769523 bytes
87 pkts/sec 96908 bytes/sec
vlan165:
received (in 1194.830 secs):
0 packets 0 bytes
0 pkts/sec 0 bytes/sec
transmitted (in 1194.830 secs):
1 packets 46 bytes
0 pkts/sec 0 bytes/sec
------------------ show perfmon ------------------
PERFMON STATS: Current Average
Xlates 0/s 2/s
Connections 7/s 9/s
TCP Conns 0/s 2/s
UDP Conns 6/s 7/s
URL Access 0/s 1/s
URL Server Req 0/s 0/s
TCP Fixup 335/s 375/s
TCPIntercept 0/s 0/s
HTTP Fixup 27/s 89/s
FTP Fixup 0/s 0/s
AAA Authen 0/s 0/s
AAA Author 0/s 0/s
AAA Account 0/s 0/s
---------------------
Thanks in advance.
Mark K~
12-09-2010 07:34 AM
Hi Mark,
I believe the tracebacks are better analized by cisco (TAC case).
Federico.
12-09-2010 07:39 AM
Federico,
I understand that, with this unit being old and having no active service contract was looking to reach out to the next best thing in hopes that this looks familiar to anyone else who may have stumbled accross a simiar case.
Mark K~
12-09-2010 10:52 AM
Hi Mark,
Federico is definitely correct, this would be best looked at under a TAC case. That being said, I don't see any bugs that match this specifically but the crash seems to be related to the DHCP client configured for one of your interfaces. A workaround would be to use static IP addresses on the interfaces.
Given the age of the PIX 6.3 code, there's a good chance that an upgrade to a newer code version would stop the crashes. However, since PIX code is no longer being developed, any bugs that are not already fixed would not be fixed in the future.
Hope that helps.
-Mike
12-09-2010 12:06 PM
Thanks,
Thsi is running the latest and greatest IOS for what we currently have H/W wise in this system. Anythign above 6.x requires more RAM.
After looking again I noticed the following on one of my interfaces;
1032 input errors, 452 CRC, 477 frame, 0 overrun, 452 ignored, 0 abort
This was caused by a duplex missmatch that has been since corrected. Cleared counters and have not noticed this errors increase or a reboot since. Going to cross fingers and wait and see.
Thanks again for your assistance with this.
Mark K~
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide