05-09-2012 05:52 PM - edited 03-11-2019 04:04 PM
Hi,
I know this topic was already discussed before, and I already tried their solution but nothing happened. Bear with me if I'll post this again.
Our company’s Cisco ASA 5520 CPU usage drastically increased up to 93% after installing the antivirus our company purchased.
Upon entering the show commands, which I will post the result later, it shows that the “Dispatch Unit is very high.
I tried to clear the conn of each IP address that has very high bytes, but nothing happened.
I’ll post all the result, and please help me solve this issue. I’m not really familiar with Firewall or security.
INTFW(config)# show proc cpu-usage sorted non-zero
PC Thread 5Sec 1Min 5Min Process
081aa324 6bdaf870 81.3% 81.5% 81.4% Dispatch Unit
08bd08d6 6bda9210 5.7% 5.7% 5.7% Logger
INTFW(config)# show proc cpu-usage sorted non-zero
PC Thread 5Sec 1Min 5Min Process
081aa324 6bdaf870 81.3% 81.5% 81.4% Dispatch Unit
08bd08d6 6bda9210 5.7% 5.7% 5.7% Logger
INTFW(config)# show proc cpu-hog
Process: vpnfol_sync/Bulk Sync - Import , PROC_PC_TOTAL: 23, MAXHOG: 5, LASTHOG: 5
LASTHOG At: 11:27:17 PHST Aug 8 2011
PC: 8da1592 (suspend)
Process: vpnfol_sync/Bulk Sync - Import , NUMHOG: 23, MAXHOG: 5, LASTHOG: 5
LASTHOG At: 11:27:17 PHST Aug 8 2011
PC: 8da1592 (suspend)
Traceback: 8da1c7e 8d9ff8f 8062413
Process: ssh_init, PROC_PC_TOTAL: 4, MAXHOG: 4, LASTHOG: 3
LASTHOG At: 07:41:20 PHST Aug 18 2011
PC: 806dcd5 (suspend)
Process: ssh_init, NUMHOG: 4, MAXHOG: 4, LASTHOG: 3
LASTHOG At: 07:41:20 PHST Aug 18 2011
PC: 806dcd5 (suspend)
Traceback: 8b9d3e6 8bab837 8ba024a 8062413
Process: ssh_init, PROC_PC_TOTAL: 90801, MAXHOG: 5, LASTHOG: 2
LASTHOG At: 04:47:28 PHST Apr 5 2012
PC: 8b9ac8c (suspend)
Process: ssh_init, NUMHOG: 90801, MAXHOG: 5, LASTHOG: 2
LASTHOG At: 04:47:28 PHST Apr 5 2012
PC: 8b9ac8c (suspend)
Traceback: 8b9ac8c 8ba77ed 8ba573e 8ba58e8 8ba6971 8ba02b4 8062413
Process: telnet/ci, PROC_PC_TOTAL: 1, MAXHOG: 3, LASTHOG: 3
LASTHOG At: 08:43:18 PHST Apr 16 2012
PC: 8870ba5 (suspend)
Process: telnet/ci, NUMHOG: 1, MAXHOG: 3, LASTHOG: 3
LASTHOG At: 08:43:18 PHST Apr 16 2012
PC: 8870ba5 (suspend)
Traceback: 8870ba5 9298bf1 92789fe 9279191 80ca7e7 80cacbb 80c14b5
80c1c5f 80c2da6 80c3850 8062413
Process: Unicorn Proxy Thread, PROC_PC_TOTAL: 5, MAXHOG: 3, LASTHOG: 2
LASTHOG At: 20:23:09 PHST Apr 27 2012
PC: 8c0e8e5 (suspend)
Process: Unicorn Proxy Thread, NUMHOG: 5, MAXHOG: 3, LASTHOG: 2
LASTHOG At: 20:23:09 PHST Apr 27 2012
PC: 8c0e8e5 (suspend)
Traceback: 8c0e8e5 8c23428 8c24561 8cff99d 8cfdb0c 8cf9f81 8cf9ef5
8cfa9b0 8cec6c9 8cebf7b 8cec22c 8ce5e2f 8d00cfb 8d01d67
Process: Unicorn Proxy Thread, PROC_PC_TOTAL: 12, MAXHOG: 5, LASTHOG: 4
LASTHOG At: 20:23:09 PHST Apr 27 2012
PC: 8c2bb4d (suspend)
Process: Unicorn Proxy Thread, NUMHOG: 12, MAXHOG: 5, LASTHOG: 4
LASTHOG At: 20:23:09 PHST Apr 27 2012
PC: 8c2bb4d (suspend)
Traceback: 8c2bb4d 8c0ef7a 8c11576 8c11625 8c12748 8c140f8 8c0f074
8c23bae 8f2f1f1 8062413
Process: vpnfol_sync/Bulk Sync - Import , PROC_PC_TOTAL: 488, MAXHOG: 100, LASTHOG: 2
LASTHOG At: 02:44:29 PHST May 6 2012
PC: 80635a5 (suspend)
Process: ssh_init, NUMHOG: 461, MAXHOG: 3, LASTHOG: 2
LASTHOG At: 02:44:29 PHST May 6 2012
PC: 80635a5 (suspend)
Traceback: 80635a5 8133d0b 9224474 923d3c8 9239045 9238e95 9226f50
92263d8 92158bf 920530c 922564a 92254c1 9214606 92050bc
Process: telnet/ci, PROC_PC_TOTAL: 1, MAXHOG: 5, LASTHOG: 5
LASTHOG At: 17:46:33 PHST May 9 2012
PC: 8beab4b (suspend)
Process: telnet/ci, NUMHOG: 1, MAXHOG: 5, LASTHOG: 5
LASTHOG At: 17:46:33 PHST May 9 2012
PC: 8beab4b (suspend)
Traceback: 8beb37e 8bf5961 8870405 92861be 80cf185 80c2c3f 80c3850
8062413
Process: snmp, PROC_PC_TOTAL: 65, MAXHOG: 3, LASTHOG: 3
LASTHOG At: 07:51:40 PHST May 10 2012
PC: 8b37300 (suspend)
Process: snmp, NUMHOG: 65, MAXHOG: 3, LASTHOG: 3
LASTHOG At: 07:51:40 PHST May 10 2012
PC: 8b37300 (suspend)
Traceback: 8b37300 8b35d27 8b32e39 8b358c8 8b10b5e 8b0f7bc 8062413
Process: ssh_init, PROC_PC_TOTAL: 43490, MAXHOG: 4, LASTHOG: 2
LASTHOG At: 08:03:59 PHST May 10 2012
PC: 83cf301 (suspend)
Process: ssh_init, NUMHOG: 43490, MAXHOG: 4, LASTHOG: 2
LASTHOG At: 08:03:59 PHST May 10 2012
PC: 83cf301 (suspend)
Traceback: 83cfb25 83c9883 812ea45 89e51b2 89b8dda 8ba0e44 8ba0278
8062413
Process: Dispatch Unit, PROC_PC_TOTAL: 50959, MAXHOG: 46, LASTHOG: 2
LASTHOG At: 08:16:30 PHST May 10 2012
PC: 81aa324 (suspend)
Process: Dispatch Unit, NUMHOG: 50959, MAXHOG: 46, LASTHOG: 2
LASTHOG At: 08:16:30 PHST May 10 2012
PC: 81aa324 (suspend)
Traceback: 81aa324 8062413
Process: Dispatch Unit, PROC_PC_TOTAL: 4912632, MAXHOG: 1010, LASTHOG: 3
LASTHOG At: 08:16:40 PHST May 10 2012
PC: 81aa50f (suspend)
Process: Dispatch Unit, NUMHOG: 4502524, MAXHOG: 1010, LASTHOG: 3
LASTHOG At: 08:16:40 PHST May 10 2012
PC: 81aa50f (suspend)
Traceback: 81aa50f 8062413
Process: snmp, PROC_PC_TOTAL: 85863, MAXHOG: 4, LASTHOG: 3
LASTHOG At: 08:16:40 PHST May 10 2012
PC: 8c09598 (suspend)
Process: snmp, NUMHOG: 85863, MAXHOG: 4, LASTHOG: 3
LASTHOG At: 08:16:40 PHST May 10 2012
PC: 8c09598 (suspend)
Traceback: 8b300cd 8b1086d 8b0f7bc 8062413
Process: snmp, PROC_PC_TOTAL: 43522, MAXHOG: 4, LASTHOG: 3
LASTHOG At: 08:16:40 PHST May 10 2012
PC: 8b3709e (suspend)
Process: snmp, NUMHOG: 43522, MAXHOG: 4, LASTHOG: 3
LASTHOG At: 08:16:40 PHST May 10 2012
PC: 8b3709e (suspend)
Traceback: 8b3709e 8b35dcb 8b32e39 8b358c8 8b10b5e 8b0f7bc 8062413
Process: Dispatch Unit, NUMHOG: 14404267, MAXHOG: 1012, LASTHOG: 3
LASTHOG At: 08:17:07 PHST May 10 2012
PC: 81aa5f9 (suspend)
Traceback: 81aa5f9 8062413
Process: Dispatch Unit, PROC_PC_TOTAL: 20260397, MAXHOG: 1012, LASTHOG: 3
LASTHOG At: 08:17:08 PHST May 10 2012
PC: 81aa5f9 (suspend)
CPU hog threshold (msec): 2.844
Last cleared: None
INTFW(config)# show int | in error
1762 input errors, 0 CRC, 0 frame, 1762 overrun, 0 ignored, 0 abort
0 output errors, 0 collisions, 0 interface resets
38632851 input errors, 0 CRC, 0 frame, 38632851 overrun, 0 ignored, 0 abort
0 output errors, 0 collisions, 0 interface resets
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 output errors, 0 collisions, 7 interface resets
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 output errors, 0 collisions, 0 interface resets
1 input errors, 0 CRC, 0 frame, 1 overrun, 0 ignored, 0 abort
0 output errors, 0 collisions, 0 interface resets
INTFW(config)# show int
Interface GigabitEthernet0/0 "outside", is up, line protocol is up
Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec
Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
MAC address d0d0.fd3f.0ff4, MTU 1500
IP address x.x.x.6, subnet mask 255.255.255.248
30015960429 packets input, 26267024403964 bytes, 0 no buffer
Received 9057 broadcasts, 0 runts, 0 giants
1762 input errors, 0 CRC, 0 frame, 1762 overrun, 0 ignored, 0 abort
0 L2 decode drops
199746407478 packets output, 25119852006560 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 late collisions, 0 deferred
0 input reset drops, 0 output reset drops, 0 tx hangs
input queue (blocks free curr/low): hardware (255/230)
output queue (blocks free curr/low): hardware (255/24)
Traffic Statistics for "outside":
30002303388 packets input, 25691387461881 bytes
199746407478 packets output, 21463867385699 bytes
629259354 packets dropped
1 minute input rate 1754 pkts/sec, 1668152 bytes/sec
1 minute output rate 11769 pkts/sec, 944305 bytes/sec
1 minute drop rate, 20 pkts/sec
5 minute input rate 1646 pkts/sec, 1415643 bytes/sec
5 minute output rate 11907 pkts/sec, 1263071 bytes/sec
5 minute drop rate, 19 pkts/sec
Interface GigabitEthernet0/1 "inside", is up, line protocol is up
Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec
Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
MAC address d0d0.fd3f.0ff5, MTU 1500
IP address x.x.x.9, subnet mask 255.255.255.248
197887766666 packets input, 24998369433168 bytes, 0 no buffer
Received 278288 broadcasts, 0 runts, 0 giants
38632921 input errors, 0 CRC, 0 frame, 38632921 overrun, 0 ignored, 0 abort
0 L2 decode drops
29089991932 packets output, 26007238507372 bytes, 79 underruns
0 output errors, 0 collisions, 0 interface resets
0 late collisions, 0 deferred
0 input reset drops, 0 output reset drops, 0 tx hangs
input queue (blocks free curr/low): hardware (255/230)
output queue (blocks free curr/low): hardware (255/0)
Traffic Statistics for "inside":
197875091433 packets input, 21381545513997 bytes
29089992011 packets output, 25452507365233 bytes
47959890 packets dropped
1 minute input rate 11609 pkts/sec, 926890 bytes/sec
1 minute output rate 1731 pkts/sec, 1703914 bytes/sec
1 minute drop rate, 3 pkts/sec
5 minute input rate 11612 pkts/sec, 988624 bytes/sec
5 minute output rate 1615 pkts/
INTFW(config)# show conn
----partial result of show conn. Some of the results have an higher bytes but I think this will be enough.
158026 in use, 165954 most used
TCP outside x.x.x.138:1522 inside x.x.x.106:3609, idle 0:00:24, bytes 1231922, flags UIO
TCP outside x.x.x.138:1522 inside x.x.x.106:4583, idle 0:00:05, bytes 108207477, flags UIO
INTFW(config)# show traffic
folink:
received (in 1922566.370 secs):
62152861 packets 4669911582 bytes
1 pkts/sec 2000 bytes/sec
transmitted (in 1922566.370 secs):
1215835634 packets 1396053558570 bytes
0 pkts/sec 726002 bytes/sec
1 minute input rate 1 pkts/sec, 117 bytes/sec
1 minute output rate 55 pkts/sec, 65230 bytes/sec
1 minute drop rate, 0 pkts/sec
5 minute input rate 1 pkts/sec, 117 bytes/sec
5 minute output rate 51 pkts/sec, 59983 bytes/sec
5 minute drop rate, 0 pkts/sec
outside:
received (in 1922872.370 secs):
30003574779 packets 25692551618468 bytes
15000 pkts/sec 13361000 bytes/sec
transmitted (in 1922872.370 secs):
199756000629 packets 21464645138678 bytes
103001 pkts/sec 11162000 bytes/sec
1 minute input rate 1496 pkts/sec, 1370318 bytes/sec
1 minute output rate 11724 pkts/sec, 1001443 bytes/sec
1 minute drop rate, 23 pkts/sec
5 minute input rate 1518 pkts/sec, 1369006 bytes/sec
5 minute output rate 11644 pkts/sec, 992991 bytes/sec
5 minute drop rate, 25 pkts/sec
inside:
received (in 1922876.630 secs):
197884596127 packets 21382322027279 bytes
102001 pkts/sec 11119000 bytes/sec
transmitted (in 1922876.630 secs):
29091209527 packets 25453660568576 bytes
15001 pkts/sec 13237000 bytes/sec
1 minute input rate 11607 pkts/sec, 996877 bytes/sec
1 minute output rate 1476 pkts/sec, 1352799 bytes/sec
1 minute drop rate, 14 pkts/sec
5 minute input rate 11487 pkts/sec, 986769 bytes/sec
5 minute output rate 1453 pkts/sec, 1345452 bytes/sec
5 minute drop rate, 5 pkts/sec
Thanks,
Mark
05-10-2012 08:56 PM
what is the command?
when I entered the show logging, this result shows:
INTFW(config)# show logging
Syslog logging: enabled
Facility: 16
Timestamp logging: disabled
Standby logging: disabled
Debug-trace logging: disabled
Console logging: disabled
Monitor logging: disabled
Buffer logging: disabled
Trap logging: disabled
History logging: disabled
Device ID: disabled
Mail logging: disabled
ASDM logging: level informational, 2153425037 messages logged
I think this is not the one you are asking.
05-10-2012 08:58 PM
I see you have asdm, you can go to monitoring--->Logging and grab the logs from there, or do, logging buffered 6 and then show log.
Mike
05-10-2012 09:07 PM
oh yes we are, and I don't even know how to use it yet.
here is the result, I just did it on CLI instead.
INTFW(config)# show log
Syslog logging: enabled
Facility: 16
Timestamp logging: disabled
Standby logging: disabled
Debug-trace logging: disabled
Console logging: disabled
Monitor logging: disabled
Buffer logging: level informational, 189920 messages logged
Trap logging: disabled
History logging: disabled
Device ID: disabled
Mail logging: disabled
ASDM logging: level informational, 2163529896 messages logged
12.84/3306)
<134>:%ASA-session-6-302013: Built outboection 3204397854 for outside:156.99.135.115/445 (156.99.135.115/445) to inside:x.x.211.122/4070 (x.x.211.122/4070)
<134>:%ASA-session-6-302013: Built outbound TCP connection 3204397855 for outside:48.29.51.119/445 (48.29.51.119/445) to inside:x.x.212.168/4095 (x.x.212.168/4095)
<134>:%ASA-session-6-302013: Built outbound TCP connection 3204397856 for outside:153.29.17.47/445 (153.29.17.47/445) to inside:x.x.215.62/4600 (x.x.215.62/4600)
<134>:%ASA-session-6-302013: Built outbound TCP connection 3204397857 for outside:93.96.181.119/445 (93.96.181.119/445) to inside:x.x.216.128/4724 (x.x.216.128/4724)
<134>:%ASA-session-6-302013: Built outbound TCP connection 3204397858 for outside:142.117.190.105/445 (142.117.190.105/445) to inside:x.x.211.153/4731 (12.230.211.15session-6-302014: Teardown TCP connection 3204241463 for outside:148.18.251.18/42014: Teardown TCP connection 3204241489 for outside:152.34.30.80/445 to inside:x.x.212.234/3528 duration 0:00:30 bytes 0 SYN Timeout
<134>:%ASA-session-6-302014: Teardown TCP connection 3204241490 fort
<134>:%ASA-session-6-302014: Teardown TCP connection 3204242978 for outside:133.97.126.73/445 to inside:x.x.211.137/3009 duration 0:00:30 bytes 0 SYN Timeout
<134>:%ASA-session-6-302014: Tear12.93/3984)
nection 3204242979 for outside:184.32.145.19/445 to inside:x.x.212 for outside:172.99.172.115/445 (172.99.172.115/445) to inside:x.x.218.192/3260 (x.x.218.192/3260)
<134>:%ASA-session-6-302013: Built outbound TCP connection 3204399383 for outside:94.122.223.124/445 (94.122.223.124/445) to inside:x.x.216.127/4647 (x.x.216.127/4647)
<134>:%ASA-session-6-302013: Built outbound TCP connection 3204399384 for outside:76.58.162.61/445 (76.58.162.61/445) to inside:x.x.212.93/3985 (x.x.212.93/3985)
<134>:%ASA-see:12.230.212.241/1908 duration 0:00:30 bytes 0 SYN Timeout
<134>:%ASA-session-6-62 for outside:63.68.184.29/445 (63.68.184.29/445) to inside:x.x.215.87/1840 outbound UDP connection 3204405586 for outside:168.126.63.1/53 (168.126.63.1/53) to inside:x.x.217.211/4038 (x.x.217.211/4038)
<134>:%ASA-session-6-302013: Built outbound TCP connection 3204405611 for outside:136.102.38.80/445 (136.1012.230.211.180/1197)
<134>:%ASA-session-6-302013: Built outbound TCP connection 2848 duration 0:00:30 bytes 0 SYN Timeout
<134>:%ASA-session-6-302014: Teardown 0:30 bytes 0 SYN Timeout
<134>:%ASA-session-6-302014: Teardown TCP connection 32 0 SYN Timeout
<134>:%ASA-session-6-302014: Teardown TCP connection 3204249238 fside:x.x.211.180/1199 (x.x.211.180/1199)
<134>:%ASA-session-6-302013: Builimeout
<134>:%ASA-session-6-302014: Teardown TCP connection 3204249288 for outsi:%ASA-session-6-302014: Teardown TCP connection 3204249314 for outside:42.29.214sion-6-302014: Teardown TCP connection 3204249340 for outside:181.20.123.79/445 to inside:x.x.211.160/1261 (x.x.211.160/1261)
<134>:%ASA-session-6-302013nection 3204405719 for outside:159.113.13.89/445 (159.113.13.89/445) to inside:10.211.20/4022)
<134>:%ASA-session-6-302013: Built outbound TCP connection 3204402014: Teardown TCP connection 3204249371 for outside:80.49.124.112/445 to inside TCP connection 3204249397 for outside:45.93.39.15/445 to inside:x.x.215.48/3n 3204249423/445) to inside:x.x.216.91/2482 (x.x.216.91/2482)
<134>:%ASA-session-6-302013: Built outbound TCP connection 3204408253 for outside:152.54.105.115/445 (152.54.105.115/445) to inside:x.x.211.150/3239 (x.x.211.150/3239)
<134>:%ASA-session-6-302013: Built outbound TCP connection 3204408254 for outside:64.95.156.117/445 (64.95.156.117/445) to inside:x.x.218.225/4888 (x.x.218.225/4888)
<134>:%ASA-session-6-302013: Built outbound TCP connection 3204408255 for outside:59.51.64.61/445 (59.51.64.61/445) to inside:x.x.211.150/3240 (x.x.211.150/3240)
INTFW(config)# ion-6-302013: Built outbound TCP connection 3204408256 for outside:105.52.141.20/445 (1
Thanks,
Mark
05-10-2012 09:15 PM
Mark,
Do you know the following Addresses ?
152.54.105.115
64.95.156.117
59.51.64.61
What I am seeing so far is just a lot of tcp connections that are not that normal. And most of them end up on SYN timeout. Can you tell me if outbound TCP traffic (445) for file sharing (Not FTP, FTP goes over 21) is normal? We can set some policies on the firewall to limit the amount of oubound embryonic connections.
Let me know.
Mike
05-10-2012 10:05 PM
I don't know those addresses, they're from outside, the first 2 IPs are from US & the last one came from China I think. can you help me setting up policies?
Thanks,
Mark
05-10-2012 10:17 PM
Uhm, Sure why not.
First, if not 445 traffic should be going out, block that traffic outbound. Second, we can go ahead and set the policy for half-open sessions on that specific port.
Here,
Access-list MPF permit tcp any any eq 445
class-map MPF
match access-list MPF
Policy-map global_policy
class MPF
set connection per-client-embryonic-max 10
If no TCP 445 traffic should be going outbound, do the following
access-list inside deny tcp any any eq 445
access-list inside permit ip any any
access-group inside in interface inside.
Mike
Mike
05-10-2012 11:15 PM
I sent the access-list of our ASA on your private message before I execute this. is it safe to do this, will it not affect the production?
05-11-2012 04:31 AM
It can't be done. I entered the commands, after that the CPU usage drops so fast. I didn't realized that all the distribution and access switches lost their connections. I removed the commands, now our internet connections fluctuates and the CPU usage of this ASA is now 99%. I don't know what to do with this.
05-11-2012 04:53 AM
check which IP do more traffic
#sh local-host | i host|count|maximum
and after check the IP detailed for example:
#sh local-host 10.10.10.10 all detail connection
05-11-2012 05:07 AM
do you have an IPS module ?
Class-map: global-class
IPS: card status Unresponsive, mode inline fail-open, sensor vs0
packet input 197451550328, packet output 197459152624, drop 3901726, reset-drop 395164
#sh module
05-11-2012 05:18 AM
check the resource usage with the
#show resource usage
05-11-2012 09:18 PM
So, we know there that it is in fact the traffic hitting the inside interface. Now, I saw something really alarming on one of the access lists that is there and I think that is when the problem of internet connection issue came in. Did you use the commads I gave you or did you use ASDM.
A policy needs to be set while you troubleshoot the inside network to mitigate the impact on the ASA.
Let me know when you have time.
Mike
05-15-2012 05:21 PM
Hey Mike,
Sorry for the late reply. We' were so busy because of that CPU usage issue. Well, we found out that it was actually a virus who makes our CPU usage very high. After scanning some PCs on our machine in production, CPU usage suddenly drops to less than 20%. It was wierd but, I'll let you know the details, after we figured out how to totally eliminate the worm actually. Thanks for your help & good luck on your CCIE exam.
Mark
05-15-2012 05:32 PM
Roberto,
Thanks for the post & sorry I wasn't able to reply. It seems we are on the right track now, but if ever there's an issue again, I'll let you know as well.
Thanks again,
Mark
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide