High Cpu Utilisation on Pix 515E

[vinod.mali]

         Hi Guys,

I am facing high CPU util on my pix 515 E which is in failover mode.During peak hours the util is see rising to 60% where as in off peak hours it is normally12%.

During normal operation the average utilisation was observed to be 30% but suddenly from 2/3 days it is constantly 60% doule the value as earlier.

Have gone through the logs and traffic but not able to tarce anything particular

require your help

below is the o/p of some command taken for analysis

IOS version 8.0(4)

sh cpu usage

CPU utilization for 5 seconds = 51%; 1 minute: 61%; 5 minutes: 58%

sh cpu usage

CPU utilization for 5 seconds = 51%; 1 minute: 61%; 5 minutes: 58%

sh blocks

SIZE MAX LOW CNT

0 400 371 400

4 100 100 100

80 270 12 270

256 1612 1574 1612

1550 2871 321 2094

2048 100 100 100

2560 164 164 164

4096 100 100 100

8192 100 100 100

16384 104 104 104

65536 16 16 16

# sh processes

PC SP STATE Runtime SBASE Stack Process

Lwe 001076ec 01a43e8c 016a6b10 0 01a41f78 7940/8192 block_diag

Mrd 00285d14 01e0553c 016a6c10 74343075 01de56c8 123788/131072 Dispatch Unit

Mwe 0011e185 01e591d4 016a6a90 0 01e57330 7704/8192 Reload Control Thread

Mwe 00126a06 01e60534 016a9328 14 01e5c980 12688/16384 aaa

Lwe 00284d31 01e68e44 016b4d60 0 01e66f40 7216/8192 dbgtrace

Msi 005f47cc 01e71554 016a6a90 2058 01e6f640 7644/8192 557mcfix

Msi 005f461e 01e736c4 016a6a90 1 01e717b0 7760/8192 557statspoll

Mwe 00fcb6a5 024f84ac 016a6a90 4 01e8c718 7120/8192 netfs_thread_init

Mwe 01335495 01e9acc4 016a6a90 0 01e98e40 7672/8192 Chunk Manager

Msi 00b0516e 01ea622c 016a6a90 3956 01ea4338 7604/8192 PIX Garbage Collector

Mwe 00af0854 01ebbdd4 01617ea0 0 01eb9ed0 7924/8192 IP Address Assign

Mwe 00d9aa6e 0204b9bc 016276d0 0 02049ab8 7924/8192 QoS Support Module

Mwe 00b8c68f 0204db64 01618fd0 0 0204bc60 7924/8192 Client Update Task

Lwe 01357e3d 02050424 016a6a90 1180666 0204e590 7612/8192 Checkheaps

Mwe 00e03ffa 0205c93c 016a6a90 584 02054ac8 30712/32768 Session Manager

Mwe 00f6383d 0206737c 02fb17f8 5 02063928 14312/16384 uauth

Mwe 00ed94ed 02069954 01634240 0 02067a50 7216/8192 Uauth_Proxy

Msp 00f1f0e4 024c6c64 016a6a90 131 024c4d50 7700/8192 SSL

Mwe 00f6161e 024c8d94 01639aa0 0 024c6ec0 7240/8192 SMTP

Mwe 00f4d560 024cada4 01639a40 46268 024c9030 3900/8192 Logger

Mwe 00f49856 024cd034 016a6a90 0 024cb1a0 7184/8192 Thread Logger

Mwe 01211372 024ddc04 0166ea40 0 024dbd20 6880/8192 vpnlb_thread

Mwe 003a867d 024e1e84 016a6a90 0 024e0000 7672/8192 TLS Proxy Inspector

<--- More --->

Msi 00e23f5b 0256a524 016a6a90 3861 02568620 7628/8192 emweb/cifs_timer

Msi 007136f0 01e2a6d4 016a6a90 10880 01e28800 6304/8192 arp_timer

Mwe 0071ee04 01e98b84 016c8dd0 0 01e96cd0 7828/8192 arp_forward_thread

Msi 00f66fb9 01e629e4 016a6a90 118265 01e60af0 5148/8192 tcp_fast

Msi 00f6a3d9 0268e66c 016a6a90 20997 0268c788 4800/8192 tcp_slow

Mwe 00f9fe61 0269c35c 01641bf0 0 0269a468 7908/8192 udp_timer

Mwe 001d7968 01e914f4 016a6a90 0 01e8f660 7760/8192 CTCP Timer process

Mwe 011a24b3 01e93734 016a6a90 0 01e918c0 7728/8192 L2TP data daemon

Mwe 011a3613 02c3493c 016a6a90 0 02c32aa8 7760/8192 L2TP mgmt daemon

Mwe 0118a61b 02c6ca64 01667f90 609 02c68bb0 15900/16384 ppp_timer_thread

Msi 01211827 02c6eaac 016a6a90 3556 02c6cbd8 7580/8192 vpnlb_timer_thread

Mwe 001f79f7 01e64af4 01e943c8 828 01e62c60 4076/8192 IPsec message handler

Msi 0020cf6c 0206ba84 016a6a90 9700 02069bc0 7564/8192 CTM message handler

Mwe 00bc3719 01e66c34 016a6a90 0 01e64dd0 7640/8192 NAT security-level reconfiguration

Mwe 00dd1d28 02e267fc 016a6a90 0 02e24958 7776/8192 ICMP event handler

Mwe 00a192a3 02e2a9bc 016a6a90 97854 02e26b18 14412/16384 IP Background

Mwe 0027a439 02e7d8ec 015e9030 9396 02e5daa8 120700/131072 tmatch compile thread

Mwe 00c775f5 030049fc 016a6a90 0 03000b48 15912/16384 Crypto PKI RECV

Mwe 00c7a39a 03008b4c 016a6a90 0 03004cb8 15880/16384 Crypto CA

Lsi 00b1f6e8 03036b8c 016a6a90 2291 03034c78 7644/8192 uauth_urlb clean

Lwe 00afc01f 030875bc 016a6a90 8042 03085748 6284/8192 pm_timer_thread

Mwe 0067a26d 03089fa4 016a6a90 12703 03088110 7580/8192 IKE Timekeeper

Mwe 0066abe3 0308f4f4 01607750 58591 0308b920 9588/16384 IKE Daemon

Mwe 00eef1d2 03093214 01638950 0 03091330 7892/8192 RADIUS Proxy Event Daemon

<--- More --->

Mwe 00eb5596 03095244 0206cd68 5 030934a0 7032/8192 RADIUS Proxy Listener

Mwe 00eed9ff 030974a4 016a6a90 0 03095610 7760/8192 RADIUS Proxy Time Keeper

Mwe 006ffa54 0309a2fc 016c8d00 0 030984b8 7024/8192 Integrity FW Task

Mwe 0029ea0e 030dbb1c 0142bf50 3582 030bc338 123092/131072 ci/console

Mwe 00577355 030deb8c 016a6a90 536 030dccf8 5576/8192 fover_thread

Mwe 0116cbc5 030e0cfc 0184a0f8 3041 030dee68 5556/8192 lu_ctl

Msi 00b45f5c 030e2ccc 016a6a90 46244 030e0fd8 5928/8192 update_cpu_usage

Msi 00b3fd4c 030ed124 016a6a90 167479 030eb2f0 5548/8192 NIC status poll

Mwe 0056524c 024d1224 016bcd10 6788 024cf480 5348/8192 fover_rx

Mwe 00561330 024dfd44 016bcd74 1950 024dde90 7436/8192 fover_tx

Mwe 005601ab 020526cc 016c8e70 2312 020507e8 7596/8192 fover_ip

Mwe 00577d31 0310544c 016bcd88 114 03101768 10008/16384 fover_rep

Mwe 0056ee0d 0310d344 016bcd90 9557 03105790 22252/32768 fover_parse

Mwe 0054e1d4 0310f65c 016baab0 17 0310d7b8 4132/8192 fover_ifc_test

Mwe 00551475 03111674 016a6a90 141524 0310f7e0 4476/8192 fover_health_monitoring_thread

Mwe 0029ea0e 0311315c 0142bfa8 21351 03111808 6256/8192 fover_serial_rx

Mwe 0058ee0c 03115724 016bb9a0 5834 03113830 6284/8192 fover_serial_tx

Mwe 0058bec0 03119914 016a6a90 2466 03117a80 7548/8192 ha_trans_ctl_tx

Mwe 0058bec0 0312c964 016a6a90 14103 0312aad0 5572/8192 ha_trans_data_tx

Mwe 00582607 0312e99c 016a6a90 109 0312caf8 4764/8192 fover_FSM_thread

Mwe 0116c4cb 03130a24 016c8f70 7648 0312eb20 5300/8192 lu_rx

Lwe 0116c3ec 03132a5c 0184a018 2 03130b48 6368/8192 lu_dynamic_sync

Mwe 00e688db 0322a914 0162af80 72 03228a20 4196/8192 SNMP Notify Thread

Mwe 00fcb6a5 032961fc 016a6a90 41 0322aea8 24816/32768 rtcli async executor process

<--- More --->

Mwe 0070dc3e 033a460c 016c9090 81614 033a0738 13480/16384 IP Thread

Mwe 00715726 033a679c 016c9030 8662 033a48a8 4292/8192 ARP Thread

Mwe 005fab40 033a889c 016c8e60 1860 033a6a48 4692/8192 icmp_thread

Mwe 00fa0ede 033aaa4c 016a6a90 1404 033a8bb8 7508/8192 udp_thread

Mwe 00f6c726 033aca7c 016c9080 78517 033aad28 5472/8192 tcp_thread

Mwe 00f77d2b 033aed2c 016a6a90 2 033ace98 7304/8192 npshim_thread

Mwe 00b7f5c5 03424bc4 016a6a90 173 03420d20 15764/16384 Time Range Process

Mwe 00eb5596 0342d73c 01e690c0 5 0342b988 7368/8192 EAPoUDP-sock

Mwe 002d08b5 0342f65c 016a6a90 0 0342daf8 6872/8192 EAPoUDP

Mwe 00dadb43 035943fc 01627f34 0 03592578 7796/8192 radius_snd

Mwe 00f9ff10 03595dfc 0269a1ac 0 035946e8 5336/8192 radius_rcv_auth

Mwe 00f9ff10 03597f6c 0269a0f4 0 03596858 5336/8192 radius_rcv_acct

Lsi 010af6cc 035f2e8c 016a6a90 61 035f0f78 7792/8192 websns_clean_cache

Msi 010af057 035f4fac 016a6a90 9688 035f30e8 3388/8192 websns_keepalive

Mwe 00f6383d 035f70bc 02fb1820 29416 035f5258 7472/8192 websns_rcv_tcp

Mwe 010ae3f5 035f92bc 01645680 23091 035f73c8 5912/8192 websns_snd

Mwe 00f9ff10 035fb39c 0269a03c 0 035f9538 7208/8192 websns_rcv_udp

Mwe 00f9ff10 035fd50c 02699f84 0 035fb6a8 7208/8192 n2h2_rcv_udp

Mwe 00f9ff10 0374a144 02699e14 60273 03748920 3740/8192 snmp

Mwe 00eb5596 03799b04 037b1278 1256 03797d60 7268/8192 IKE Receiver

Mwe 002aa031 037b70dc 016a6a90 69 037b5238 6972/8192 DHCPD Timer

Mwe 002ac3be 037cdc04 016a6a90 187 037cbdc0 1976/8192 dhcp_daemon

Mwe 011dbe3d 0394508c 0166de68 3140 0393d198 29112/32768 vpnfol_thread_msg

Msi 011e28f2 0384a664 016a6a90 87377 03848780 5396/8192 vpnfol_thread_timer

<--- More --->

Mwe 011e0cf2 03947054 0166dfe0 0 039451c0 7060/8192 vpnfol_thread_sync

Msi 011e245c 039490dc 016a6a90 22729 039471e8 7608/8192 vpnfol_thread_unsent

Mwe 006fb040 024d8624 016a6a90 0 024d6790 7724/8192 Integrity Fw Timer Thread

Mwe 00fcb6a5 0339435c 016a6a90 2872 033870c8 4024/8192 Unicorn Admin Thread

Mwe 00f0bfd3 039c193c 016a6a90 4 039bfaa8 6304/8192 ssh/timer

Lwe 00b7b253 030f1434 016a6a90 28 030ef5d0 7564/8192 vPif_stats_cleaner

Mwe 00f7923c 01a4f1dc 039d54e8 4 01a4d528 6760/8192 listen/ssh

M* 00f047a4 0009feec 016a6c10 199 03ba6578 25840/32768 ssh

- - - - 307733344 - - scheduler

- - - - 384627935 - - total elapsed

sh traffic

outside:

received (in 386140.890 secs):

446338717 packets151701600892 bytes

1000 pkts/sec392009 bytes/sec

transmitted (in 386140.890 secs):

416905030 packets43029087866 bytes

1001 pkts/sec111010 bytes/sec

1 minute input rate 2644 pkts/sec, 690466 bytes/sec

1 minute output rate 2719 pkts/sec, 297688 bytes/sec

1 minute drop rate, 9 pkts/sec

5 minute input rate 2681 pkts/sec, 748731 bytes/sec

5 minute output rate 2701 pkts/sec, 279464 bytes/sec

5 minute drop rate, 14 pkts/sec

inside:

received (in 386140.880 secs):

415868926 packets36734994462 bytes

1010 pkts/sec95000 bytes/sec

transmitted (in 386140.880 secs):

448348003 packets142393792212 bytes

1005 pkts/sec368004 bytes/sec

1 minute input rate 2725 pkts/sec, 240490 bytes/sec

1 minute output rate 2659 pkts/sec, 640514 bytes/sec

1 minute drop rate, 5 pkts/sec

5 minute input rate 2696 pkts/sec, 225510 bytes/sec

5 minute output rate 2689 pkts/sec, 693534 bytes/sec

<--- More --->

5 minute drop rate, 6 pkts/sec

dmz:

received (in 386152.200 secs):

9115765 packets3824758993 bytes

1 pkts/sec9003 bytes/sec

transmitted (in 386152.200 secs):

9006788 packets6056538317 bytes

1 pkts/sec15005 bytes/sec

1 minute input rate 10 pkts/sec, 1110 bytes/sec

1 minute output rate 5 pkts/sec, 289 bytes/sec

1 minute drop rate, 0 pkts/sec

5 minute input rate 15 pkts/sec, 1605 bytes/sec

5 minute output rate 12 pkts/sec, 7607 bytes/sec

5 minute drop rate, 0 pkts/sec

state:

received (in 386192.070 secs):

329095 packets168447602 bytes

0 pkts/sec2 bytes/sec

transmitted (in 386192.070 secs):

2564316 packets2897621540 bytes

6 pkts/sec7002 bytes/sec

1 minute input rate 0 pkts/sec, 23 bytes/sec

1 minute output rate 11 pkts/sec, 14550 bytes/sec

1 minute drop rate, 0 pkts/sec

<--- More --->

5 minute input rate 0 pkts/sec, 23 bytes/sec

5 minute output rate 10 pkts/sec, 12984 bytes/sec

5 minute drop rate, 0 pkts/sec

----------------------------------------

Aggregated Traffic on Physical Interface

----------------------------------------

Ethernet0:

received (in 386193.930 secs):

446076991 packets159127473834 bytes

1010 pkts/sec412006 bytes/sec

transmitted (in 386193.930 secs):

416940387 packets49497797587 bytes

1001 pkts/sec128001 bytes/sec

1 minute input rate 2772 pkts/sec, 734256 bytes/sec

1 minute output rate 2851 pkts/sec, 338262 bytes/sec

1 minute drop rate, 0 pkts/sec

5 minute input rate 2681 pkts/sec, 793197 bytes/sec

5 minute output rate 2701 pkts/sec, 320368 bytes/sec

5 minute drop rate, 0 pkts/sec

Ethernet1:

received (in 386193.950 secs):

415573532 packets43094626952 bytes

1009 pkts/sec111009 bytes/sec

<--- More --->

transmitted (in 386193.950 secs):

448018366 packets151059749497 bytes

1004 pkts/sec391005 bytes/sec

1 minute input rate 2725 pkts/sec, 281477 bytes/sec

1 minute output rate 2659 pkts/sec, 692714 bytes/sec

1 minute drop rate, 0 pkts/sec

5 minute input rate 2662 pkts/sec, 266251 bytes/sec

5 minute output rate 2655 pkts/sec, 746174 bytes/sec

5 minute drop rate, 0 pkts/sec

Ethernet2:

received (in 386194.830 secs):

328683 packets177176804 bytes

0 pkts/sec2 bytes/sec

transmitted (in 386194.830 secs):

2562183 packets2935659864 bytes

6 pkts/sec7000 bytes/sec

1 minute input rate 0 pkts/sec, 34 bytes/sec

1 minute output rate 12 pkts/sec, 14721 bytes/sec

1 minute drop rate, 0 pkts/sec

5 minute input rate 0 pkts/sec, 34 bytes/sec

5 minute output rate 10 pkts/sec, 13139 bytes/sec

5 minute drop rate, 0 pkts/sec

Ethernet3:

received (in 386194.850 secs):

<--- More --->

0 packets0 bytes

0 pkts/sec0 bytes/sec

transmitted (in 386194.850 secs):

0 packets0 bytes

0 pkts/sec0 bytes/sec

1 minute input rate 0 pkts/sec, 0 bytes/sec

1 minute output rate 0 pkts/sec, 0 bytes/sec

1 minute drop rate, 0 pkts/sec

5 minute input rate 0 pkts/sec, 0 bytes/sec

5 minute output rate 0 pkts/sec, 0 bytes/sec

5 minute drop rate, 0 pkts/sec

Ethernet4:

received (in 386195.510 secs):

0 packets0 bytes

0 pkts/sec0 bytes/sec

transmitted (in 386195.510 secs):

0 packets0 bytes

0 pkts/sec0 bytes/sec

1 minute input rate 0 pkts/sec, 0 bytes/sec

1 minute output rate 0 pkts/sec, 0 bytes/sec

1 minute drop rate, 0 pkts/sec

5 minute input rate 0 pkts/sec, 0 bytes/sec

5 minute output rate 0 pkts/sec, 0 bytes/sec

5 minute drop rate, 0 pkts/sec

<--- More --->

Ethernet5:

received (in 386195.540 secs):

9111822 packets3974053334 bytes

1 pkts/sec10001 bytes/sec

transmitted (in 386195.540 secs):

9003293 packets6210259782 bytes

1 pkts/sec16002 bytes/sec

1 minute input rate 11 pkts/sec, 1264 bytes/sec

1 minute output rate 6 pkts/sec, 421 bytes/sec

1 minute drop rate, 0 pkts/sec

5 minute input rate 15 pkts/sec, 1851 bytes/sec

5 minute output rate 12 pkts/sec, 7848 bytes/sec

5 minute drop rate, 0 pkts/sec

KBPO-PRI# sh perf

KBPO-PRI# sh perfmon

PERFMON STATS: Current Average

Xlates 1/s 0/s

Connections 14/s 7/s

TCP Conns 6/s 3/s

UDP Conns 4/s 2/s

URL Access 4/s 2/s

URL Server Req 6/s 4/s

TCP Fixup 69/s 1/s

TCP Intercept Established Conns 0/s 0/s

TCP Intercept Attempts 0/s 0/s

TCP Embryonic Conns Timeout 0/s 0/s

HTTP Fixup 69/s 1/s

FTP Fixup 0/s 0/s

AAA Authen 0/s 0/s

AAA Author 0/s 0/s

AAA Account 0/s 0/s

VALID CONNS RATE in TCP INTERCEPT: Current Average

N/A 100.00%

[Maykol Rojas]

Hi,

Do you have Failover replication for http traffic enable? Do you see errors on the interfaces when the Spikes goes up?

Mike