cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
37078
Views
10
Helpful
28
Replies

Cisco ASA 5520 CPU Usage is ranging 87%- 93%

mparas_04
Level 1
Level 1

Hi,

I know this topic was already discussed before, and I already tried their solution but nothing happened. Bear with me if I'll post this again.

Our company’s Cisco ASA 5520 CPU usage drastically increased up to  93% after installing the antivirus our company purchased.

Upon entering the show commands, which I will post the result later, it shows that the “Dispatch Unit is very high.

I tried to clear the conn of each IP address that has very high bytes, but nothing happened.

I’ll post all the result, and please help me solve this issue. I’m not really familiar with Firewall or security.

INTFW(config)# show proc cpu-usage sorted non-zero

PC         Thread       5Sec     1Min     5Min   Process

081aa324   6bdaf870    81.3%    81.5%    81.4%   Dispatch Unit

08bd08d6   6bda9210     5.7%     5.7%     5.7%   Logger

INTFW(config)# show proc cpu-usage sorted non-zero

PC         Thread       5Sec     1Min     5Min   Process

081aa324   6bdaf870    81.3%    81.5%    81.4%   Dispatch Unit

08bd08d6   6bda9210     5.7%     5.7%     5.7%   Logger

INTFW(config)# show proc cpu-hog

Process:      vpnfol_sync/Bulk Sync - Import , PROC_PC_TOTAL: 23, MAXHOG: 5, LASTHOG: 5

LASTHOG At:   11:27:17 PHST Aug 8 2011

PC:           8da1592 (suspend)

Process:      vpnfol_sync/Bulk Sync - Import , NUMHOG: 23, MAXHOG: 5, LASTHOG: 5

LASTHOG At:   11:27:17 PHST Aug 8 2011

PC:           8da1592 (suspend)

Traceback:    8da1c7e  8d9ff8f  8062413

Process:      ssh_init, PROC_PC_TOTAL: 4, MAXHOG: 4, LASTHOG: 3

LASTHOG At:   07:41:20 PHST Aug 18 2011

PC:           806dcd5 (suspend)

Process:      ssh_init, NUMHOG: 4, MAXHOG: 4, LASTHOG: 3

LASTHOG At:   07:41:20 PHST Aug 18 2011

PC:           806dcd5 (suspend)

Traceback:    8b9d3e6  8bab837  8ba024a  8062413

Process:      ssh_init, PROC_PC_TOTAL: 90801, MAXHOG: 5, LASTHOG: 2

LASTHOG At:   04:47:28 PHST Apr 5 2012

PC:           8b9ac8c (suspend)

Process:      ssh_init, NUMHOG: 90801, MAXHOG: 5, LASTHOG: 2

LASTHOG At:   04:47:28 PHST Apr 5 2012

PC:           8b9ac8c (suspend)

Traceback:    8b9ac8c  8ba77ed  8ba573e  8ba58e8  8ba6971  8ba02b4  8062413

Process:      telnet/ci, PROC_PC_TOTAL: 1, MAXHOG: 3, LASTHOG: 3

LASTHOG At:   08:43:18 PHST Apr 16 2012

PC:           8870ba5 (suspend)

Process:      telnet/ci, NUMHOG: 1, MAXHOG: 3, LASTHOG: 3

LASTHOG At:   08:43:18 PHST Apr 16 2012

PC:           8870ba5 (suspend)

Traceback:    8870ba5  9298bf1  92789fe  9279191  80ca7e7  80cacbb  80c14b5

               80c1c5f  80c2da6  80c3850  8062413

Process:      Unicorn Proxy Thread, PROC_PC_TOTAL: 5, MAXHOG: 3, LASTHOG: 2

LASTHOG At:   20:23:09 PHST Apr 27 2012

PC:           8c0e8e5 (suspend)

Process:      Unicorn Proxy Thread, NUMHOG: 5, MAXHOG: 3, LASTHOG: 2

LASTHOG At:   20:23:09 PHST Apr 27 2012

PC:           8c0e8e5 (suspend)

Traceback:    8c0e8e5  8c23428  8c24561  8cff99d  8cfdb0c  8cf9f81  8cf9ef5

               8cfa9b0  8cec6c9  8cebf7b  8cec22c  8ce5e2f  8d00cfb  8d01d67

Process:      Unicorn Proxy Thread, PROC_PC_TOTAL: 12, MAXHOG: 5, LASTHOG: 4

LASTHOG At:   20:23:09 PHST Apr 27 2012

PC:           8c2bb4d (suspend)

Process:      Unicorn Proxy Thread, NUMHOG: 12, MAXHOG: 5, LASTHOG: 4

LASTHOG At:   20:23:09 PHST Apr 27 2012

PC:           8c2bb4d (suspend)

Traceback:    8c2bb4d  8c0ef7a  8c11576  8c11625  8c12748  8c140f8  8c0f074

               8c23bae  8f2f1f1  8062413

Process:      vpnfol_sync/Bulk Sync - Import , PROC_PC_TOTAL: 488, MAXHOG: 100, LASTHOG: 2

LASTHOG At:   02:44:29 PHST May 6 2012

PC:           80635a5 (suspend)

Process:      ssh_init, NUMHOG: 461, MAXHOG: 3, LASTHOG: 2

LASTHOG At:   02:44:29 PHST May 6 2012

PC:           80635a5 (suspend)

Traceback:    80635a5  8133d0b  9224474  923d3c8  9239045  9238e95  9226f50

               92263d8  92158bf  920530c  922564a  92254c1  9214606  92050bc

Process:      telnet/ci, PROC_PC_TOTAL: 1, MAXHOG: 5, LASTHOG: 5

LASTHOG At:   17:46:33 PHST May 9 2012

PC:           8beab4b (suspend)

Process:      telnet/ci, NUMHOG: 1, MAXHOG: 5, LASTHOG: 5

LASTHOG At:   17:46:33 PHST May 9 2012

PC:           8beab4b (suspend)

Traceback:    8beb37e  8bf5961  8870405  92861be  80cf185  80c2c3f  80c3850

               8062413

Process:      snmp, PROC_PC_TOTAL: 65, MAXHOG: 3, LASTHOG: 3

LASTHOG At:   07:51:40 PHST May 10 2012

PC:           8b37300 (suspend)

Process:      snmp, NUMHOG: 65, MAXHOG: 3, LASTHOG: 3

LASTHOG At:   07:51:40 PHST May 10 2012

PC:           8b37300 (suspend)

Traceback:    8b37300  8b35d27  8b32e39  8b358c8  8b10b5e  8b0f7bc  8062413

Process:      ssh_init, PROC_PC_TOTAL: 43490, MAXHOG: 4, LASTHOG: 2

LASTHOG At:   08:03:59 PHST May 10 2012

PC:           83cf301 (suspend)

Process:      ssh_init, NUMHOG: 43490, MAXHOG: 4, LASTHOG: 2

LASTHOG At:   08:03:59 PHST May 10 2012

PC:           83cf301 (suspend)

Traceback:    83cfb25  83c9883  812ea45  89e51b2  89b8dda  8ba0e44  8ba0278

               8062413

Process:      Dispatch Unit, PROC_PC_TOTAL: 50959, MAXHOG: 46, LASTHOG: 2

LASTHOG At:   08:16:30 PHST May 10 2012

PC:           81aa324 (suspend)

Process:      Dispatch Unit, NUMHOG: 50959, MAXHOG: 46, LASTHOG: 2

LASTHOG At:   08:16:30 PHST May 10 2012

PC:           81aa324 (suspend)

Traceback:    81aa324  8062413

Process:      Dispatch Unit, PROC_PC_TOTAL: 4912632, MAXHOG: 1010, LASTHOG: 3

LASTHOG At:   08:16:40 PHST May 10 2012

PC:           81aa50f (suspend)

Process:      Dispatch Unit, NUMHOG: 4502524, MAXHOG: 1010, LASTHOG: 3

LASTHOG At:   08:16:40 PHST May 10 2012

PC:           81aa50f (suspend)

Traceback:    81aa50f  8062413

Process:      snmp, PROC_PC_TOTAL: 85863, MAXHOG: 4, LASTHOG: 3

LASTHOG At:   08:16:40 PHST May 10 2012

PC:           8c09598 (suspend)

Process:      snmp, NUMHOG: 85863, MAXHOG: 4, LASTHOG: 3

LASTHOG At:   08:16:40 PHST May 10 2012

PC:           8c09598 (suspend)

Traceback:    8b300cd  8b1086d  8b0f7bc  8062413

Process:      snmp, PROC_PC_TOTAL: 43522, MAXHOG: 4, LASTHOG: 3

LASTHOG At:   08:16:40 PHST May 10 2012

PC:           8b3709e (suspend)

Process:      snmp, NUMHOG: 43522, MAXHOG: 4, LASTHOG: 3

LASTHOG At:   08:16:40 PHST May 10 2012

PC:           8b3709e (suspend)

Traceback:    8b3709e  8b35dcb  8b32e39  8b358c8  8b10b5e  8b0f7bc  8062413

Process:      Dispatch Unit, NUMHOG: 14404267, MAXHOG: 1012, LASTHOG: 3

LASTHOG At:   08:17:07 PHST May 10 2012

PC:           81aa5f9 (suspend)

Traceback:    81aa5f9  8062413

Process:      Dispatch Unit, PROC_PC_TOTAL: 20260397, MAXHOG: 1012, LASTHOG: 3

LASTHOG At:   08:17:08 PHST May 10 2012

PC:           81aa5f9 (suspend)

CPU hog threshold (msec):  2.844

Last cleared: None

INTFW(config)# show int | in error

        1762 input errors, 0 CRC, 0 frame, 1762 overrun, 0 ignored, 0 abort

        0 output errors, 0 collisions, 0 interface resets

        38632851 input errors, 0 CRC, 0 frame, 38632851 overrun, 0 ignored, 0 abort

        0 output errors, 0 collisions, 0 interface resets

        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

        0 output errors, 0 collisions, 7 interface resets

        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

        0 output errors, 0 collisions, 0 interface resets

        1 input errors, 0 CRC, 0 frame, 1 overrun, 0 ignored, 0 abort

        0 output errors, 0 collisions, 0 interface resets

INTFW(config)# show int

Interface GigabitEthernet0/0 "outside", is up, line protocol is up

  Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec

        Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)

        MAC address d0d0.fd3f.0ff4, MTU 1500

        IP address x.x.x.6, subnet mask 255.255.255.248

        30015960429 packets input, 26267024403964 bytes, 0 no buffer

        Received 9057 broadcasts, 0 runts, 0 giants

        1762 input errors, 0 CRC, 0 frame, 1762 overrun, 0 ignored, 0 abort

        0 L2 decode drops

        199746407478 packets output, 25119852006560 bytes, 0 underruns

        0 output errors, 0 collisions, 0 interface resets

        0 late collisions, 0 deferred

        0 input reset drops, 0 output reset drops, 0 tx hangs

        input queue (blocks free curr/low): hardware (255/230)

        output queue (blocks free curr/low): hardware (255/24)

  Traffic Statistics for "outside":

        30002303388 packets input, 25691387461881 bytes

        199746407478 packets output, 21463867385699 bytes

        629259354 packets dropped

      1 minute input rate 1754 pkts/sec,  1668152 bytes/sec

      1 minute output rate 11769 pkts/sec,  944305 bytes/sec

      1 minute drop rate, 20 pkts/sec

      5 minute input rate 1646 pkts/sec,  1415643 bytes/sec

      5 minute output rate 11907 pkts/sec,  1263071 bytes/sec

      5 minute drop rate, 19 pkts/sec

Interface GigabitEthernet0/1 "inside", is up, line protocol is up

  Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec

        Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)

        MAC address d0d0.fd3f.0ff5, MTU 1500

        IP address x.x.x.9, subnet mask 255.255.255.248

        197887766666 packets input, 24998369433168 bytes, 0 no buffer

        Received 278288 broadcasts, 0 runts, 0 giants

        38632921 input errors, 0 CRC, 0 frame, 38632921 overrun, 0 ignored, 0 abort

        0 L2 decode drops

        29089991932 packets output, 26007238507372 bytes, 79 underruns

        0 output errors, 0 collisions, 0 interface resets

        0 late collisions, 0 deferred

        0 input reset drops, 0 output reset drops, 0 tx hangs

        input queue (blocks free curr/low): hardware (255/230)

        output queue (blocks free curr/low): hardware (255/0)

  Traffic Statistics for "inside":

        197875091433 packets input, 21381545513997 bytes

        29089992011 packets output, 25452507365233 bytes

        47959890 packets dropped

      1 minute input rate 11609 pkts/sec,  926890 bytes/sec

      1 minute output rate 1731 pkts/sec,  1703914 bytes/sec

      1 minute drop rate, 3 pkts/sec

      5 minute input rate 11612 pkts/sec,  988624 bytes/sec

      5 minute output rate 1615 pkts/

INTFW(config)# show conn

----partial result of show conn. Some of the results have an higher bytes but I think this will be enough.

158026 in use, 165954 most used

TCP outside x.x.x.138:1522 inside x.x.x.106:3609, idle 0:00:24, bytes 1231922, flags UIO

TCP outside x.x.x.138:1522 inside x.x.x.106:4583, idle 0:00:05, bytes 108207477, flags UIO

INTFW(config)# show traffic

folink:

        received (in 1922566.370 secs):

                62152861 packets        4669911582 bytes

                1 pkts/sec      2000 bytes/sec

        transmitted (in 1922566.370 secs):

                1215835634 packets      1396053558570 bytes

                0 pkts/sec      726002 bytes/sec

      1 minute input rate 1 pkts/sec,  117 bytes/sec

      1 minute output rate 55 pkts/sec,  65230 bytes/sec

      1 minute drop rate, 0 pkts/sec

      5 minute input rate 1 pkts/sec,  117 bytes/sec

      5 minute output rate 51 pkts/sec,  59983 bytes/sec

      5 minute drop rate, 0 pkts/sec

outside:

        received (in 1922872.370 secs):

                30003574779 packets     25692551618468 bytes

                15000 pkts/sec  13361000 bytes/sec

        transmitted (in 1922872.370 secs):

                199756000629 packets    21464645138678 bytes

                103001 pkts/sec 11162000 bytes/sec

      1 minute input rate 1496 pkts/sec,  1370318 bytes/sec

      1 minute output rate 11724 pkts/sec,  1001443 bytes/sec

      1 minute drop rate, 23 pkts/sec

      5 minute input rate 1518 pkts/sec,  1369006 bytes/sec

      5 minute output rate 11644 pkts/sec,  992991 bytes/sec

      5 minute drop rate, 25 pkts/sec

inside:

        received (in 1922876.630 secs):

                197884596127 packets    21382322027279 bytes

                102001 pkts/sec 11119000 bytes/sec

        transmitted (in 1922876.630 secs):

                29091209527 packets     25453660568576 bytes

                15001 pkts/sec  13237000 bytes/sec

      1 minute input rate 11607 pkts/sec,  996877 bytes/sec

      1 minute output rate 1476 pkts/sec,  1352799 bytes/sec

      1 minute drop rate, 14 pkts/sec

      5 minute input rate 11487 pkts/sec,  986769 bytes/sec

      5 minute output rate 1453 pkts/sec,  1345452 bytes/sec

      5 minute drop rate, 5 pkts/sec

Thanks,

Mark

28 Replies 28

what is the command?

when I entered the show logging, this result shows:

INTFW(config)# show logging

Syslog logging: enabled

    Facility: 16

    Timestamp logging: disabled

    Standby logging: disabled

    Debug-trace logging: disabled

    Console logging: disabled

    Monitor logging: disabled

    Buffer logging: disabled

    Trap logging: disabled

    History logging: disabled

    Device ID: disabled

    Mail logging: disabled

    ASDM logging: level informational, 2153425037 messages logged

I think this is not the one you are asking.

I see you have asdm, you can go to monitoring--->Logging and grab the logs from there, or do, logging buffered 6 and then show log.

Mike

Mike

oh yes we are, and I don't even know how to use it yet.

here is the result, I just did it on CLI instead.

INTFW(config)# show log

Syslog logging: enabled

    Facility: 16

    Timestamp logging: disabled

    Standby logging: disabled

    Debug-trace logging: disabled

    Console logging: disabled

    Monitor logging: disabled

    Buffer logging: level informational, 189920 messages logged

    Trap logging: disabled

    History logging: disabled

    Device ID: disabled

    Mail logging: disabled

    ASDM logging: level informational, 2163529896 messages logged

12.84/3306)

<134>:%ASA-session-6-302013: Built outboection 3204397854 for outside:156.99.135.115/445 (156.99.135.115/445) to inside:x.x.211.122/4070 (x.x.211.122/4070)

<134>:%ASA-session-6-302013: Built outbound TCP connection 3204397855 for outside:48.29.51.119/445 (48.29.51.119/445) to inside:x.x.212.168/4095 (x.x.212.168/4095)

<134>:%ASA-session-6-302013: Built outbound TCP connection 3204397856 for outside:153.29.17.47/445 (153.29.17.47/445) to inside:x.x.215.62/4600 (x.x.215.62/4600)

<134>:%ASA-session-6-302013: Built outbound TCP connection 3204397857 for outside:93.96.181.119/445 (93.96.181.119/445) to inside:x.x.216.128/4724 (x.x.216.128/4724)

<134>:%ASA-session-6-302013: Built outbound TCP connection 3204397858 for outside:142.117.190.105/445 (142.117.190.105/445) to inside:x.x.211.153/4731 (12.230.211.15session-6-302014: Teardown TCP connection 3204241463 for outside:148.18.251.18/42014: Teardown TCP connection 3204241489 for outside:152.34.30.80/445 to inside:x.x.212.234/3528 duration 0:00:30 bytes 0 SYN Timeout

<134>:%ASA-session-6-302014: Teardown TCP connection 3204241490 fort

<134>:%ASA-session-6-302014: Teardown TCP connection 3204242978 for outside:133.97.126.73/445 to inside:x.x.211.137/3009 duration 0:00:30 bytes 0 SYN Timeout

<134>:%ASA-session-6-302014: Tear12.93/3984)

nection 3204242979 for outside:184.32.145.19/445 to inside:x.x.212 for outside:172.99.172.115/445 (172.99.172.115/445) to inside:x.x.218.192/3260 (x.x.218.192/3260)

<134>:%ASA-session-6-302013: Built outbound TCP connection 3204399383 for outside:94.122.223.124/445 (94.122.223.124/445) to inside:x.x.216.127/4647 (x.x.216.127/4647)

<134>:%ASA-session-6-302013: Built outbound TCP connection 3204399384 for outside:76.58.162.61/445 (76.58.162.61/445) to inside:x.x.212.93/3985 (x.x.212.93/3985)

<134>:%ASA-see:12.230.212.241/1908 duration 0:00:30 bytes 0 SYN Timeout

<134>:%ASA-session-6-62 for outside:63.68.184.29/445 (63.68.184.29/445) to inside:x.x.215.87/1840 outbound UDP connection 3204405586 for outside:168.126.63.1/53 (168.126.63.1/53) to inside:x.x.217.211/4038 (x.x.217.211/4038)

<134>:%ASA-session-6-302013: Built outbound TCP connection 3204405611 for outside:136.102.38.80/445 (136.1012.230.211.180/1197)

<134>:%ASA-session-6-302013: Built outbound TCP connection 2848 duration 0:00:30 bytes 0 SYN Timeout

<134>:%ASA-session-6-302014: Teardown 0:30 bytes 0 SYN Timeout

<134>:%ASA-session-6-302014: Teardown TCP connection 32 0 SYN Timeout

<134>:%ASA-session-6-302014: Teardown TCP connection 3204249238 fside:x.x.211.180/1199 (x.x.211.180/1199)

<134>:%ASA-session-6-302013: Builimeout

<134>:%ASA-session-6-302014: Teardown TCP connection 3204249288 for outsi:%ASA-session-6-302014: Teardown TCP connection 3204249314 for outside:42.29.214sion-6-302014: Teardown TCP connection 3204249340 for outside:181.20.123.79/445  to inside:x.x.211.160/1261 (x.x.211.160/1261)

<134>:%ASA-session-6-302013nection 3204405719 for outside:159.113.13.89/445 (159.113.13.89/445) to inside:10.211.20/4022)

<134>:%ASA-session-6-302013: Built outbound TCP connection 3204402014: Teardown TCP connection 3204249371 for outside:80.49.124.112/445 to inside TCP connection 3204249397 for outside:45.93.39.15/445 to inside:x.x.215.48/3n 3204249423/445) to inside:x.x.216.91/2482 (x.x.216.91/2482)

<134>:%ASA-session-6-302013: Built outbound TCP connection 3204408253 for outside:152.54.105.115/445 (152.54.105.115/445) to inside:x.x.211.150/3239 (x.x.211.150/3239)

<134>:%ASA-session-6-302013: Built outbound TCP connection 3204408254 for outside:64.95.156.117/445 (64.95.156.117/445) to inside:x.x.218.225/4888 (x.x.218.225/4888)

<134>:%ASA-session-6-302013: Built outbound TCP connection 3204408255 for outside:59.51.64.61/445 (59.51.64.61/445) to inside:x.x.211.150/3240 (x.x.211.150/3240)

INTFW(config)# ion-6-302013: Built outbound TCP connection 3204408256 for outside:105.52.141.20/445 (1

Thanks,

Mark

Mark,

Do you know the following Addresses ?

152.54.105.115

64.95.156.117

59.51.64.61

What I am seeing so far is just a lot of tcp connections that are not that normal. And most of them end up on SYN timeout. Can you tell me if outbound TCP traffic (445) for file sharing (Not FTP, FTP goes over 21) is normal? We can set some policies on the firewall to limit the amount of oubound embryonic connections.

Let me know.

Mike

Mike

I don't know those addresses, they're from outside, the first 2 IPs are from US & the last one came from China I think. can you help me setting up policies?

Thanks,

Mark

Uhm, Sure why not.

First, if not 445 traffic should be going out, block that traffic outbound. Second, we can go ahead and set the policy for half-open sessions on that specific port.

Here,

Access-list MPF permit tcp any any eq 445

class-map MPF

match access-list MPF

Policy-map global_policy

class MPF

  set connection per-client-embryonic-max 10

If no TCP 445 traffic should be going outbound, do the following

access-list inside deny tcp any any eq 445

access-list inside permit ip any any

access-group inside in interface inside.

Mike

Mike

Mike

I sent the access-list of our ASA on your private message before I execute this. is it safe to do this, will it not affect the production?

It can't be done. I entered the commands, after that the CPU usage drops so fast. I didn't realized that all the distribution and access switches lost their connections. I removed the commands, now our internet connections fluctuates and the CPU usage of this ASA is now 99%. I don't know what to do with this.

check which IP do more traffic

#sh local-host | i host|count|maximum

and after check the IP detailed for example:

#sh local-host 10.10.10.10 all detail connection

do you have an IPS module ?

Class-map: global-class

       IPS: card status Unresponsive, mode inline fail-open, sensor vs0

         packet input 197451550328, packet output 197459152624, drop 3901726, reset-drop 395164

#sh module

check the resource usage with the 

#show resource usage

So, we know there that it is in fact the traffic hitting the inside interface. Now, I saw something really alarming on one of the access lists that is there and I think that is when the problem of internet connection issue came in. Did you use the commads I gave you or did you use ASDM.

A policy needs to be set while you troubleshoot the inside network to mitigate the impact on the ASA.

Let me know when you have time.

Mike

Mike

Hey Mike,

Sorry for the late reply. We' were so busy because of that CPU usage issue. Well, we found out that it was actually a virus who makes our CPU usage very high. After scanning some PCs on our machine in production, CPU usage suddenly drops to less than 20%. It was wierd but, I'll let you know the details, after we figured out how to totally eliminate the worm actually. Thanks for your help & good luck on your CCIE exam.

Mark

Roberto,

Thanks for the post & sorry I wasn't able to reply. It seems we are on the right track now, but if ever there's an issue again, I'll let you know as well.

Thanks again,

Mark

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: