there is intermittent packet loss between cisco router and ASA 5

raghu.jrs · ‎05-12-2013

Hi

We have Cisco router 2800 router which is directly connected to ASA 5510, till now there was no issue every thing was working fine, but from past 2 day's we are facing a problem, when we try to ping to any outside public IP their is a intermittent packet loss & same issue to the remote office through IPSec tunnel,

We are able to reach our ISP router from outside whithout any issue & there is no packet Loss, if we try to reach the ASA their is a intermittent packet loss,

so What could be the issue, Please help me to resolve this

raghu.jrs · ‎05-12-2013

act# sh traffic

outside:

received (in 1344456.090 secs):

38790308 packets 27004452865 bytes

0 pkts/sec 20002 bytes/sec

transmitted (in 1344456.090 secs):

192453628 packets 181680591094 bytes

2 pkts/sec 135002 bytes/sec

1 minute input rate 3343 pkts/sec, 1904704 bytes/sec

1 minute output rate 15347 pkts/sec, 14467560 bytes/sec

1 minute drop rate, 37 pkts/sec

5 minute input rate 3021 pkts/sec, 1997507 bytes/sec

5 minute output rate 5005 pkts/sec, 3437293 bytes/sec

5 minute drop rate, 33 pkts/sec

James Leinweber · ‎05-13-2013

Maybe you have CPU overload problems? What do:

show process cpu-hog

show process cpu-usage non-zero sorted

show interface

suggest to you? In particuarly, is total CPU usage over 60%? Are there lots of CPU-HOG incidents? Do the interface counters show non-zero counts for either "underrun", "overrun", or both?

-- Jim Leinweber, WI State Lab of Hygiene

raghu.jrs · ‎05-14-2013

Hi

Please find the Outputs & this issue is repeating even the CPu load is just 29%

act# show process cpu-hog

Process: CTM message handler, NUMHOG: 1, MAXHOG: 5, LASTHOG: 5

LASTHOG At: 12:19:00 gmt May 12 2013

PC: 8129c7c (suspend)

Call stack: 8063bb3

Process: Unicorn Admin Handler, PROC_PC_TOTAL: 1, MAXHOG: 6, LASTHOG: 6

LASTHOG At: 12:19:25 gmt May 12 2013

PC: 83b9a57 (suspend)

Process: Unicorn Admin Handler, NUMHOG: 1, MAXHOG: 6, LASTHOG: 6

LASTHOG At: 12:19:25 gmt May 12 2013

PC: 83b9a57 (suspend)

Call stack: 84894be 83b9534 83af5da 8066e4a 88cd497 8066e4a 843e60e

8435cb9 8436093 84363b4 843ce7d 8063bb3

Process: fover_FSM_thread, NUMHOG: 1, MAXHOG: 3, LASTHOG: 3

LASTHOG At: 22:58:14 gmt May 13 2013

PC: 8064d65 (suspend)

Call stack: 8064d65 8847f05 88671db 8842611 846e1df 83d4a27 83f48c3

83f4d03 83f73bd 84063eb 840159e 8402cc8 8063bb3

Process: tmatch compile thread, PROC_PC_TOTAL: 1, MAXHOG: 3, LASTHOG: 3

LASTHOG At: 22:58:14 gmt May 13 2013

PC: 8190d92 (suspend)

Process: tmatch compile thread, NUMHOG: 1, MAXHOG: 3, LASTHOG: 3

LASTHOG At: 22:58:14 gmt May 13 2013

PC: 8190d92 (suspend)

Call stack: 8190d92 818d738 818ded9 81994f9 819323b 8063bb3

Process: NIC status poll, PROC_PC_TOTAL: 3, MAXHOG: 16, LASTHOG: 12

LASTHOG At: 23:00:06 gmt May 13 2013

PC: 891be60 (suspend)

Process: NIC status poll, NUMHOG: 3, MAXHOG: 16, LASTHOG: 12

LASTHOG At: 23:00:06 gmt May 13 2013

PC: 891be60 (suspend)

Call stack: 891be60 8063bb3

Process: ssh, NUMHOG: 2, MAXHOG: 4, LASTHOG: 4

LASTHOG At: 23:02:40 gmt May 13 2013

PC: 8064d65 (suspend)

Call stack: 8064d65 83e17ce 83e281d 83fba63 83fbd17 80ca032 80cb6cf

80cc7ba 80cd59a 8063bb3

Process: IKE Daemon, NUMHOG: 1, MAXHOG: 3, LASTHOG: 3

LASTHOG At: 02:51:36 gmt May 14 2013

PC: 8064d65 (suspend)

Call stack: 8064d65 84bd489 8063bb3

Process: fover_FSM_thread, PROC_PC_TOTAL: 5, MAXHOG: 32, LASTHOG: 32

LASTHOG At: 03:30:00 gmt May 14 2013

PC: 8064d65 (suspend)

Process: ssh, NUMHOG: 1, MAXHOG: 32, LASTHOG: 32

LASTHOG At: 03:30:00 gmt May 14 2013

PC: 8064d65 (suspend)

Call stack: 8064d65 924bd88 924b6c1 9246a85 8a0875e 8a08e90 89f4410

8a01365 8c014d5 8c016c9 8c81f52 8066d81 8c7cb5a 8ffbf45

Process: Unicorn Admin Handler, PROC_PC_TOTAL: 14, MAXHOG: 24, LASTHOG: 7

LASTHOG At: 03:30:37 gmt May 14 2013

PC: 8c3acab (suspend)

Process: Unicorn Admin Handler, NUMHOG: 12, MAXHOG: 24, LASTHOG: 7

LASTHOG At: 03:30:37 gmt May 14 2013

PC: 8c3acab (suspend)

Call stack: 8c3b0b2 843e82a 8435cb9 8436093 84363b4 843ce7d 8063bb3

Process: snmp, PROC_PC_TOTAL: 5, MAXHOG: 27, LASTHOG: 10

LASTHOG At: 06:28:13 gmt May 14 2013

PC: 8b88690 (suspend)

Process: snmp, NUMHOG: 5, MAXHOG: 27, LASTHOG: 10

LASTHOG At: 06:28:13 gmt May 14 2013

PC: 8b88690 (suspend)

Call stack: 8b88690 8b86fd7 8b84029 8b86ac8 8b6192e 8b6039c 8063bb3

Process: ssh_init, PROC_PC_TOTAL: 1109, MAXHOG: 3, LASTHOG: 3

LASTHOG At: 07:21:01 gmt May 14 2013

PC: 83e1ca3 (suspend)

Process: ssh_init, NUMHOG: 1109, MAXHOG: 3, LASTHOG: 3

LASTHOG At: 07:21:01 gmt May 14 2013

PC: 83e1ca3 (suspend)

Call stack: 83e281d 83db433 8130c3d 8a28272 89f92ba 8bf19d4 8bf0e08

8063bb3

Process: ldap_client_thread, PROC_PC_TOTAL: 290, MAXHOG: 21, LASTHOG: 13

LASTHOG At: 18:43:42 gmt May 14 2013

PC: 8c8236d (suspend)

Process: ldap_client_thread, NUMHOG: 290, MAXHOG: 21, LASTHOG: 13

LASTHOG At: 18:43:42 gmt May 14 2013

PC: 8c8236d (suspend)

Call stack: 8c8236d 8c64b5a 8c67616 8c676c5 935e023 935b272 93607a8

8c64c4d 85d2a04 8063bb3

Process: snmp, PROC_PC_TOTAL: 2739, MAXHOG: 11, LASTHOG: 9

LASTHOG At: 18:44:21 gmt May 14 2013

PC: 8b8842e (suspend)

Process: snmp, NUMHOG: 2739, MAXHOG: 11, LASTHOG: 9

LASTHOG At: 18:44:21 gmt May 14 2013

PC: 8b8842e (suspend)

Call stack: 8b8842e 8b8707b 8b84029 8b86ac8 8b6192e 8b6039c 8063bb3

Process: snmp, PROC_PC_TOTAL: 2739, MAXHOG: 19, LASTHOG: 9

LASTHOG At: 18:44:21 gmt May 14 2013

PC: 8c5f308 (suspend)

Process: snmp, NUMHOG: 2739, MAXHOG: 19, LASTHOG: 9

LASTHOG At: 18:44:21 gmt May 14 2013

PC: 8c5f308 (suspend)

Call stack: 8b81303 8b6163d 8b6039c 8063bb3

Process: Dispatch Unit, PROC_PC_TOTAL: 8635, MAXHOG: 51, LASTHOG: 3

LASTHOG At: 18:44:57 gmt May 14 2013

PC: 81aa794 (suspend)

Process: Dispatch Unit, NUMHOG: 8632, MAXHOG: 51, LASTHOG: 3

LASTHOG At: 18:44:57 gmt May 14 2013

PC: 81aa794 (suspend)

Call stack: 81aa794 8063bb3

Process: Dispatch Unit, PROC_PC_TOTAL: 68126, MAXHOG: 52, LASTHOG: 5

LASTHOG At: 18:44:59 gmt May 14 2013

PC: 81aa97f (suspend)

Process: Dispatch Unit, NUMHOG: 5967, MAXHOG: 52, LASTHOG: 5

LASTHOG At: 18:44:59 gmt May 14 2013

PC: 81aa97f (suspend)

Call stack: 81aa97f 8063bb3

Process: Dispatch Unit, PROC_PC_TOTAL: 10615, MAXHOG: 52, LASTHOG: 3

LASTHOG At: 18:45:05 gmt May 14 2013

PC: 81aaa69 (suspend)

Process: Dispatch Unit, NUMHOG: 10397, MAXHOG: 52, LASTHOG: 3

LASTHOG At: 18:45:05 gmt May 14 2013

PC: 81aaa69 (suspend)

Call stack: 81aaa69 8063bb3

CPU hog threshold (msec): 3.47

act# show process cpu-usage non-zero sorted

PC Thread 5Sec 1Min 5Min Process

081aa794 a79afa10 27.8% 26.8% 26.6% Dispatch Unit

08c5f308 a799e088 1.9% 0.9% 0.9% snmp

08c24266 a79a93b0 1.1% 0.8% 0.8% Logger

085336c6 a79a05f0 0.2% 0.1% 0.0% IP Thread

08af3b09 a799de90 0.2% 0.2% 0.2% IP SLA Mon Event Processor

08e39ed2 a799ccd8 0.1% 0.1% 0.1% vpnfol_thread_timer

08c601d6 a79a0008 0.1% 0.0% 0.0% udp_thread

08129c7c a79a58a0 0.1% 0.1% 0.1% CTM message handler

08beb5dc a798d0d8 0.0% 0.1% 0.1% ssh

act# show interface

Interface Ethernet0/0 "outside", is up, line protocol is up

Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec

Full-Duplex(Full-duplex), 100 Mbps(100 Mbps)

Input flow control is unsupported, output flow control is off

144947900 packets input, 120260332110 bytes, 0 no buffer

Received 425 broadcasts, 0 runts, 0 giants

90918 input errors, 0 CRC, 0 frame, 90918 overrun, 0 ignored, 0 abort

0 pause input, 0 resume input

0 L2 decode drops

156525316 packets output, 89663180626 bytes, 239300423 underruns

0 pause output, 0 resume output

1 output errors, 2401 collisions, 24 interface resets

6673 late collisions, 17208 deferred

2 input reset drops, 4067 output reset drops, 19 tx hangs

input queue (blocks free curr/low): hardware (255/230)

output queue (blocks free curr/low): hardware (255/0)

Traffic Statistics for "outside":

144894894 packets input, 117600192252 bytes

395836861 packets output, 334657315331 bytes

1599000 packets dropped

1 minute input rate 3520 pkts/sec, 2645833 bytes/sec

1 minute output rate 3025 pkts/sec, 1882533 bytes/sec

1 minute drop rate, 32 pkts/sec

5 minute input rate 2826 pkts/sec, 2168042 bytes/sec

5 minute output rate 10573 pkts/sec, 9609682 bytes/sec

5 minute drop rate, 32 pkts/sec

Interface Ethernet0/1 "inside", is up, line protocol is up

Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec

Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)

Input flow control is unsupported, output flow control is off

486253732 packets input, 400450860201 bytes, 0 no buffer

Received 551611 broadcasts, 0 runts, 0 giants

6882166 input errors, 0 CRC, 0 frame, 6882166 overrun, 0 ignored, 0 abort

0 pause input, 0 resume input

0 L2 decode drops

200741431 packets output, 158213546472 bytes, 0 underruns

0 pause output, 0 resume output

0 output errors, 0 collisions, 2 interface resets

0 late collisions, 0 deferred

14 input reset drops, 0 output reset drops, 0 tx hangs

input queue (blocks free curr/low): hardware (255/230)

output queue (blocks free curr/low): hardware (254/164)

Traffic Statistics for "inside":

484013604 packets input, 388467517994 bytes

200741431 packets output, 154508904349 bytes

7114270 packets dropped

1 minute input rate 5091 pkts/sec, 2156171 bytes/sec

1 minute output rate 5978 pkts/sec, 5500344 bytes/sec

1 minute drop rate, 8 pkts/sec

5 minute input rate 12083 pkts/sec, 10267848 bytes/sec

5 minute output rate 4541 pkts/sec, 3528041 bytes/sec

5 minute drop rate, 10 pkts/sec

Interface Ethernet0/2 "backup", is up, line protocol is up

Hardware is i82546GB rev03, BW 100 Mbps, DLY 100 usec

Auto-Duplex(Full-duplex), Auto-Speed(100 Mbps)

Input flow control is unsupported, output flow control is off

46951587 packets input, 37230631055 bytes, 0 no buffer

Received 318 broadcasts, 0 runts, 0 giants

194 input errors, 0 CRC, 0 frame, 194 overrun, 0 ignored, 0 abort

0 pause input, 0 resume input

0 L2 decode drops

45733131 packets output, 18354000341 bytes, 31696229 underruns

0 pause output, 0 resume output

0 output errors, 0 collisions, 17 interface resets

0 late collisions, 0 deferred

0 input reset drops, 3809 output reset drops, 15 tx hangs

input queue (blocks free curr/low): hardware (255/230)

output queue (blocks free curr/low): hardware (255/0)

Traffic Statistics for "backup":

46919589 packets input, 36381482830 bytes

77433388 packets output, 50460617743 bytes

128580 packets dropped

1 minute input rate 2474 pkts/sec, 2890355 bytes/sec

1 minute output rate 2048 pkts/sec, 321160 bytes/sec

1 minute drop rate, 0 pkts/sec

5 minute input rate 1729 pkts/sec, 1383704 bytes/sec

5 minute output rate 1487 pkts/sec, 680549 bytes/sec

5 minute drop rate, 0 pkts/sec

Interface Ethernet0/3 "ASA5510", is up, line protocol is up

Hardware is i82546GB rev03, BW 100 Mbps, DLY 100 usec

Auto-Duplex(Full-duplex), Auto-Speed(100 Mbps)

Input flow control is unsupported, output flow control is off

Description: LAN/STATE Failover Interface

MAC address d0d0.fd04.0347, MTU 1500

IP address 172.16.1.2, subnet mask 255.255.255.0

15721136 packets input, 18446423252 bytes, 0 no buffer

Received 186 broadcasts, 0 runts, 0 giants

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

0 pause input, 0 resume input

0 L2 decode drops

13252556 packets output, 15559453960 bytes, 0 underruns

0 pause output, 0 resume output

0 output errors, 0 collisions, 1 interface resets

0 late collisions, 0 deferred

3 input reset drops, 0 output reset drops, 0 tx hangs

input queue (blocks free curr/low): hardware (255/230)

output queue (blocks free curr/low): hardware (255/114)

Traffic Statistics for "ASA5510":

15721030 packets input, 17867537424 bytes

13252425 packets output, 15320446732 bytes

0 packets dropped

1 minute input rate 1 pkts/sec, 122 bytes/sec

1 minute output rate 257 pkts/sec, 302767 bytes/sec

1 minute drop rate, 0 pkts/sec

5 minute input rate 1 pkts/sec, 125 bytes/sec

5 minute output rate 245 pkts/sec, 288629 bytes/sec

5 minute drop rate, 0 pkts/sec

Interface Management0/0 "management", is down, line protocol is down

Hardware is i82557, BW 100 Mbps, DLY 100 usec

Auto-Duplex, Auto-Speed

Input flow control is unsupported, output flow control is unsupported

MAC address 0022.5597.2292, MTU 1500

IP address 192.168.1.1, subnet mask 255.255.255.0

0 packets input, 0 bytes, 0 no buffer

Received 0 broadcasts, 0 runts, 0 giants

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

0 pause input, 0 resume input

0 L2 decode drops

0 packets output, 0 bytes, 0 underruns

0 pause output, 0 resume output

0 output errors, 0 collisions, 0 interface resets

0 babbles, 0 late collisions, 0 deferred

0 lost carrier, 0 no carrier

0 input reset drops, 0 output reset drops

input queue (curr/max packets): hardware (0/0) software (0/0)

output queue (curr/max packets): hardware (0/0) software (0/0)

Traffic Statistics for "management":

0 packets input, 0 bytes

0 packets output, 0 bytes

0 packets dropped

1 minute input rate 0 pkts/sec, 0 bytes/sec

1 minute output rate 0 pkts/sec, 0 bytes/sec

1 minute drop rate, 0 pkts/sec

5 minute input rate 0 pkts/sec, 0 bytes/sec

5 minute output rate 0 pkts/sec, 0 bytes/sec

5 minute drop rate, 0 pkts/sec

Management-only interface. Blocked 0 through-the-device packets

James Leinweber · ‎05-14-2013

Your 100 Mbit/s outside interface appears to be severely congested. You have about 0.6% input overruns, where an inbound packet is being dropped because the firewall can't process it fast enough. However, that pales in comparison to your output queue, where the number of underruns (output packets dropped) exceeds the number of packets delivered by 50%. You are failing to send 2/3 of your packets; that may be an interface speed versus traffic issue rather than a CPU issue. Just because the average sending rate is below the interface speed doesn't mean that bursty client traffic can't saturate the link.

Alternatively, is there any kind of speed/duplex mismatch between the router and the firewall? I've seen that produce similar symptoms.

-- Jim Leinweber, WI State Lab of Hygiene

raghu.jrs · ‎05-14-2013

HI

Both the interface between the firewall & the router are Full-Duplex & speed is 100, this setup was working fine from past 4 Years, earlier some time the outside interface traffice was reaching 90MB but there was no issues on that time, now we have only around 50 to 60 MB of interface traffice but no Idead why we are facing this issue,

this issue is repeating even the traffice is around 20 MB, i think there could be some other problem,

We hace changed the cables, between the router & firewall, Failover the ASA & failover the ISP Link, but still the issue is not resolved,

there is intermittent packet loss between cisco router and ASA 5510