PIX High CPU %85

heang.sokdet · ‎07-12-2011

Hello all,

I have one problem with PIX high CPU loading %85 (PIX Version is 8.0(3)). please see the below log:

pixfirewall# show cpu

CPU utilization for 5 seconds = 86%; 1 minute: 86%; 5 minutes: 86%

pixfirewall# show processes

PC SP STATE Runtime SBASE Stack Process

Lwe 0010772c 01964aa4 01593568 0 01962b90 7940/8192 block_diag

Mrd 0026db04 01aaa274 01593668 3030040 01aa2400 30776/32768 Dispatch Unit

Mwe 0011dcf5 01afdd5c 015934e8 0 01afbeb8 7672/8192 Reload Control Thread

Mwe 001262e6 01b0306c 01595da0 0 01b014b8 7076/8192 aaa

Lwe 0026c552 01b0b83c 0159fd28 0 01b09938 7216/8192 dbgtrace

Msi 00581224 01b13e04 015934e8 10 01b11ef0 7792/8192 557mcfix

Msi 00581076 01b15f24 015934e8 0 01b14010 7760/8192 557statspoll

Mwe 00ef7ee5 01ff162c 015934e8 4 01b2ec58 7120/8192 netfs_thread_init

Mwe 01247d75 01b3d074 015934e8 0 01b3b1f0 7640/8192 Chunk Manager

Msi 00a5f07e 01b480c4 015934e8 11 01b461d0 7696/8192 PIX Garbage Collector

Mwe 00a4b564 01b5cc1c 01508b78 0 01b5ad18 7924/8192 IP Address Assign

Mwe 00cda36e 01b5f9b4 01516ce8 0 01b5dab0 7924/8192 QoS Support Module

Mwe 00ae2b7f 01b61b0c 01509ca8 0 01b5fc08 7924/8192 Client Update Task

Lwe 0126a79d 01b6437c 015934e8 9365 01b624e8 7680/8192 Checkheaps

Mwe 00d4159a 01b6ae9c 015934e8 1 01b69028 7152/8192 Session Manager

Mwe 00e9608d 01b757f4 02b49498 5 01b71da0 14312/16384 uauth

Mwe 00e1531e 01b77d7c 015235c8 0 01b75e78 7216/8192 Uauth_Proxy

Mwe 00e93e6e 01fc4fec 01528e18 0 01fc3118 7240/8192 SMTP

Mwe 00e80c92 01fc6fac 01528db8 28 01fc5238 5544/8192 Logger

Mwe 00e7cac6 01fc91ec 015934e8 0 01fc7358 7184/8192 Thread Logger

Mwe 01125e92 01fd9b3c 0155beb8 0 01fd7c58 6880/8192 vpnlb_thread

Mwe 00387b75 01fddd1c 015934e8 0 01fdbe98 7640/8192 TLS Proxy Inspector

Msi 00d6295b 02053484 015934e8 10 02051580 7776/8192 emweb/cifs_timer

Msi 006996d8 01b3afa4 015934e8 12 01b390d0 7168/8192 arp_timer

Mwe 006a4b8c 021184cc 015aded8 0 02116618 7828/8192 arp_forward_thread

Mwe 00e99969 021655a4 01529088 0 021636b0 7760/8192 tcp_fast

Mwe 00e9ce18 021675bc 01529088 0 021656d8 7744/8192 tcp_slow

Mwe 00ecd661 021752ac 0152fb48 0 021733b8 7908/8192 udp_timer

Mwe 001cd7d8 01b339e4 015934e8 0 01b31b50 7760/8192 CTCP Timer process

Mwe 010b8233 01b35bd4 015934e8 0 01b33d60 7728/8192 L2TP data daemon

Mwe 010b9393 026e8834 015934e8 0 026e69a0 7760/8192 L2TP mgmt daemon

Mwe 010a0ceb 0272095c 01555438 3 0271caa8 16052/16384 ppp_timer_thread

Msi 01126347 027229a4 015934e8 15 02720ad0 7636/8192 vpnlb_timer_thread

Mwe 00213753 029018dc 01b061d8 1 028ff9d8 7788/8192 VAC+ random IV generator handler

Mwe 001ecb07 029038a4 01b06208 6 02901a00 7592/8192 IPsec message handler

Msi 00200c3c 029058ec 015934e8 49 02903a28 7376/8192 CTM message handler

Mwe 00d10918 02adeae4 015934e8 0 02adcc40 7776/8192 ICMP event handler

Mwe 00977ec3 02ae2c54 015934e8 3 02adedb0 15792/16384 IP Background

Mwe 00264747 02b3596c 014dd358 48 02b15ab8 122396/131072 tmatch compile thread

Mwe 00bbe2c5 02b9b5bc 015934e8 0 02b97708 15880/16384 Crypto PKI RECV

Mwe 00bc0faa 02b9f6bc 015934e8 0 02b9b828 15848/16384 Crypto CA

Lsi 00a77c28 02bce42c 015934e8 0 02bcc518 7792/8192 uauth_urlb clean

Lwe 00a565df 02c1d43c 015934e8 26 02c1b5b8 6372/8192 pm_timer_thread

Mwe 006035e5 02c1fd24 015934e8 45 02c1de90 7668/8192 IKE Timekeeper

Mwe 005f41bb 02c25224 014fb418 21 02c21650 12824/16384 IKE Daemon

Mwe 00e2a832 02c28ebc 01527cd8 0 02c26fd8 7892/8192 RADIUS Proxy Event Daemon

Mwe 00df1da6 02c2ae9c 01b2db70 0 02c290f8 7032/8192 RADIUS Proxy Listener

Mwe 00e2905f 02c2d0ac 015934e8 0 02c2b218 7760/8192 RADIUS Proxy Time Keeper

Mwe 007367dc 02c2f804 0196da7c 0 02c2da30 7184/8192 Integrity FW Task

M* 009eeb0d 0009feec 01593668 4348 02c3f590 121784/131072 ci/console

Msi 00522cd9 02c61dc4 015934e8 14 02c5ff00 6388/8192 fover_thread

Mwe 01083c35 02c63eb4 016d3990 17 02c62020 7504/8192 lu_ctl

Msi 00a9ddcc 02c65e44 015934e8 206 02c64140 6088/8192 update_cpu_usage

Msi 00a98263 02c701ec 015934e8 521 02c6e3b8 6804/8192 NIC status poll

Mwe 007367dc 02c7448c 0196dac8 0 02c725f8 7536/8192 System Ready

Mwe 00517194 01fdbb1c 015a7b38 0 01fd9d78 7572/8192 fover_rx

Mwe 005134c8 01b66c9c 015a7b9c 0 01b64de8 7844/8192 fover_tx

Mwe 00512343 01fcd464 015adf78 0 01fcb580 7848/8192 fover_ip

Mwe 00529599 02c88424 015a7bb0 0 02c84740 15556/16384 fover_rep

Mwe 00520895 02c9031c 015a7bb8 6 02c88768 31652/32768 fover_parse

Mwe 004ffd73 01b7bf1c 015a59f8 3 01b7a098 7764/8192 fover_ifc_test

Mwe 00503b9d 02c92624 015934e8 0 02c90790 7760/8192 fover_health_monitoring_thread

Mwe 0028641e 02c9410c 01330db0 12 02c927b8 6484/8192 fover_serial_rx

Mwe 00540684 02c966d4 015a67c8 0 02c947e0 7908/8192 fover_serial_tx

Mwe 0053d738 02c9a8c4 015934e8 0 02c98a30 7760/8192 ha_trans_ctl_tx

Mwe 0053d738 02cad914 015934e8 0 02caba80 7760/8192 ha_trans_data_tx

Mwe 00533e7f 02caf94c 015934e8 0 02cadaa8 7520/8192 fover_FSM_thread

Mwe 0108353b 02cb19d4 015ae078 0 02cafad0 7832/8192 lu_rx

Lwe 0108345c 02cb3a0c 016d38b0 0 02cb1af8 7940/8192 lu_dynamic_sync

Mwe 00ef7ee5 02e06b1c 015934e8 32 02da1c38 240/8192 rtcli async executor process

Mwe 006940d6 02eb3fd4 015ae198 0 02eb2100 7860/8192 IP Thread

Mwe 0069b6fe 02eb6114 015ae138 0 02eb4220 7892/8192 ARP Thread

Mwe 00587588 02eb81c4 015adf68 0 02eb6370 7732/8192 icmp_thread

Mwe 00ece70e 02eba324 015934e8 0 02eb8490 7656/8192 udp_thread

Mwe 00e9f086 02ebc304 015ae188 0 02eba5b0 7476/8192 tcp_thread

Mwe 00eaa450 02ebe564 015934e8 0 02ebc6d0 7656/8192 npshim_thread

Mwe 00df1da6 02f8f6e4 02ea7430 0 02f8d930 7368/8192 EAPoUDP-sock

Mwe 002b7605 02f914bc 015934e8 0 02f8f958 6840/8192 EAPoUDP

Mwe 00ca74e1 02fba3dc 0166c9e0 122 02fb8518 5844/8192 qos_metric_daemon

Mwe 00cecccc 030c2e4c 0151754c 0 030c0fc8 7796/8192 radius_snd

Mwe 00ecd710 030c47b4 021730fc 0 030c30a0 5336/8192 radius_rcv_auth

Mwe 00ecd710 030c68d4 02173044 0 030c51c0 5336/8192 radius_rcv_acct

Mwe 00ef7ee5 03137f1c 015934e8 4045 0312ad00 4040/8192 Unicorn Admin Thread

Mwe 00ecd710 0320d0f4 02172ed4 0 0320b8d0 6164/8192 snmp

Mwe 00df1da6 0325aeac 032720c8 0 03259108 7352/8192 IKE Receiver

Mwe 00eaaca9 03278484 02ed6b58 0 032767d0 7332/8192 listen/ssh

Mwe 010f13dd 0341b15c 0155b300 0 03413268 32484/32768 vpnfol_thread_msg

Msi 010f7d72 0341d174 015934e8 8 0341b290 7744/8192 vpnfol_thread_timer

Mwe 010f6292 0341f18c 0155b478 0 0341d2b8 7876/8192 vpnfol_thread_sync

Msi 010f78dc 034211d4 015934e8 40 0341f2e0 7760/8192 vpnfol_thread_unsent

- - - - 553966 - - scheduler

- - - - 3624780 - - total elapsed

as we can see in the process, only Dispatch Unit is running too much. Could you please tell me what cause such a problem? how to fix it?

any help is appreciated.

thanks in advance.

Sokdet

varrao · ‎07-12-2011

Hi Sokdet,

Dispatch unit is the thread which is responsible for handling the traffic from the firewall interfaces, so high cpu cycle utilization is expected for it, I woudl suggest you to open a TAC case for it, so that this can be uinvestigated in detail.

Hope this helps,

Thanks,

Varun

Thanks,
Varun Rao

heang.sokdet · ‎07-12-2011

Hi Varun,

on this firewall interfaces there is no traffic. it is not connected to anywhere. even I turn the firewall off and on. the CPU is still keep at %85.

pixfirewall# sh traffic

Outside:

received (in 277.480 secs):

0 packets 0 bytes

0 pkts/sec 0 bytes/sec

transmitted (in 277.480 secs):

0 packets 0 bytes

0 pkts/sec 0 bytes/sec

1 minute input rate 0 pkts/sec, 0 bytes/sec

1 minute output rate 0 pkts/sec, 0 bytes/sec

1 minute drop rate, 0 pkts/sec

5 minute input rate 0 pkts/sec, 0 bytes/sec

5 minute output rate 0 pkts/sec, 0 bytes/sec

5 minute drop rate, 0 pkts/sec

Inside:

received (in 277.480 secs):

0 packets 0 bytes

0 pkts/sec 0 bytes/sec

transmitted (in 277.480 secs):

0 packets 0 bytes

0 pkts/sec 0 bytes/sec

1 minute input rate 0 pkts/sec, 0 bytes/sec

1 minute output rate 0 pkts/sec, 0 bytes/sec

1 minute drop rate, 0 pkts/sec

5 minute input rate 0 pkts/sec, 0 bytes/sec

5 minute output rate 0 pkts/sec, 0 bytes/sec

5 minute drop rate, 0 pkts/sec

----------------------------------------

Aggregated Traffic on Physical Interface

----------------------------------------

Ethernet0:

received (in 280.560 secs):

0 packets 0 bytes

0 pkts/sec 0 bytes/sec

transmitted (in 280.560 secs):

0 packets 0 bytes

0 pkts/sec 0 bytes/sec

1 minute input rate 0 pkts/sec, 0 bytes/sec

1 minute output rate 0 pkts/sec, 0 bytes/sec

1 minute drop rate, 0 pkts/sec

5 minute input rate 0 pkts/sec, 0 bytes/sec

5 minute output rate 0 pkts/sec, 0 bytes/sec

5 minute drop rate, 0 pkts/sec

Ethernet1:

received (in 280.560 secs):

0 packets 0 bytes

0 pkts/sec 0 bytes/sec

transmitted (in 280.560 secs):

0 packets 0 bytes

0 pkts/sec 0 bytes/sec

1 minute input rate 0 pkts/sec, 0 bytes/sec

1 minute output rate 0 pkts/sec, 0 bytes/sec

1 minute drop rate, 0 pkts/sec

5 minute input rate 0 pkts/sec, 0 bytes/sec

5 minute output rate 0 pkts/sec, 0 bytes/sec

5 minute drop rate, 0 pkts/sec

this firewall is out of warranty. so I cannot open a TAC case. do you have any idea?

thanks

Sokdet

varrao · ‎07-12-2011

Hi Sokdet,

If there is no traffic on the PIX at all, can you do a "clear conn count" and "clear xlate" on the PIX. and then check cpu usage..

Varun

Thanks,
Varun Rao

heang.sokdet · ‎07-12-2011

Hi Varun,

sorry for late reply. I have tried those commands, but still not help.

thanks,

Sokdet