All,

William Cooper · ‎11-07-2015

We have a site to site VPN tunnel between an ASA 5505 and an ASA5512.

Experiencing dropped pings between workstations and db server. VPN tunnel is staying up

but dropping packets making the connection unuseable. Traffic from DMZ to Web outside works fine.

Workstation > ASA 5505 inside> 5505 Outside> Fios ISP>Comcast ISP>ASA 5512>5512Inside>

DBSERVER

ping 10.1.1.1 -t

Pinging 10.1.1.1 with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Reply from 10.1.1.1: bytes=32 time=31ms TTL=252
Reply from 10.1.1.1: bytes=32 time=31ms TTL=252
Request timed out.
Reply from 10.1.1.1: bytes=32 time=29ms TTL=252
Reply from 10.1.1.1: bytes=32 time=29ms TTL=252
Request timed out.
Reply from 10.1.1.1: bytes=32 time=29ms TTL=252
Reply from 10.1.1.1: bytes=32 time=30ms TTL=252
Request timed out.
Request timed out.
Reply from 10.1.1.1: bytes=32 time=29ms TTL=252
Request timed out.
Reply from 10.1.1.1: bytes=32 time=31ms TTL=252
IP's edited

Ping statistics for 10.1.1.1:
Packets: Sent = 31, Received = 12, Lost = 19 (61% loss
Approximate round trip times in milli-seconds:
Minimum = 29ms, Maximum = 32ms, Average = 30ms

We opened a ticket with Cisco and they performed the following capture and point the issue at the ISP. Ping from outside interface firewall to firewall do not lose any pings. Fios informs they do not support VPN ? looking for clarification for my own sanity. I still believe it is an issue with the VPN. All cabling swapped for good measure. here is the ASA capture from both sides:

++ Site-to-Site assistance.

++ Packets getting dropped.

++ Tunnel stays up.

++ Applied captures and found that on Site A VPN side packets are going out and we are not seeing any reply back.

SiteAVPN# sh cap cap

22 packets captured

   1: 18:19:08.697626 10.1.1.116 > 10.1.106.2: icmp: echo request
   2: 18:19:13.705072 10.1.1.116 > 10.1.106.2: icmp: echo request
   3: 18:19:13.730888 10.1.106.2 > 10.1.1.116: icmp: echo reply
   4: 18:19:14.707986 10.1.1.116 > 10.1.106.2: icmp: echo request
   5: 18:19:14.734474 10.1.106.2 > 10.1.1.116: icmp: echo reply
   6: 18:19:15.709512 10.1.1.116 > 10.1.106.2: icmp: echo request
   7: 18:19:20.698404 10.1.1.116 > 10.1.106.2: icmp: echo request
   8: 18:19:20.725014 10.1.106.2 > 10.1.1.116: icmp: echo reply
   9: 18:19:21.700571 10.1.1.116 > 10.1.106.2: icmp: echo request
10: 18:19:26.698907 10.1.1.116 > 10.1.106.2: icmp: echo request
11: 18:19:26.724114 10.1.106.2 > 10.1.1.116: icmp: echo reply
12: 18:19:27.700082 10.1.1.116 > 10.1.106.2: icmp: echo request
13: 18:19:32.699030 10.1.1.116 > 10.1.106.2: icmp: echo request
14: 18:19:37.697549 10.1.1.116 > 10.1.106.2: icmp: echo request
15: 18:19:37.723122 10.1.106.2 > 10.1.1.116: icmp: echo reply
16: 18:19:38.699289 10.1.1.116 > 10.1.106.2: icmp: echo request
17: 18:19:38.724785 10.1.106.2 > 10.1.1.116: icmp: echo reply
18: 18:19:39.700250 10.1.1.116 > 10.1.106.2: icmp: echo request

++ On remote end i.e. SiteB ASA packets are reaching and proper reply is going out.
--------------------------------------------------------------------------------------------------------------------------------
Verizon line.
SiteB ASA# sh cap cap

58 packets captured

   1: 18:19:08.650891 802.1Q vlan#1 P0 10.1.1.116 > 10.1.106.2: icmp: echo request
   2: 18:19:08.651760 802.1Q vlan#1 P0 10.1.106.2 > 10.1.1.116: icmp: echo reply
   3: 18:19:13.657818 802.1Q vlan#1 P0 10.1.1.116 > 10.1.106.2: icmp: echo request
   4: 18:19:13.658703 802.1Q vlan#1 P0 10.1.106.2 > 10.1.1.116: icmp: echo reply
   5: 18:19:14.660823 802.1Q vlan#1 P0 10.1.1.116 > 10.1.106.2: icmp: echo request
   6: 18:19:14.661708 802.1Q vlan#1 P0 10.1.106.2 > 10.1.1.116: icmp: echo reply
   7: 18:19:15.661403 802.1Q vlan#1 P0 10.1.1.116 > 10.1.106.2: icmp: echo request
   8: 18:19:15.662319 802.1Q vlan#1 P0 10.1.106.2 > 10.1.1.116: icmp: echo reply
   9: 18:19:20.650891 802.1Q vlan#1 P0 10.1.1.116 > 10.1.106.2: icmp: echo request
10: 18:19:20.651775 802.1Q vlan#1 P0 10.1.106.2 > 10.1.1.116: icmp: echo reply
11: 18:19:21.651592 802.1Q vlan#1 P0 10.1.1.116 > 10.1.106.2: icmp: echo request
12: 18:19:21.652538 802.1Q vlan#1 P0 10.1.106.2 > 10.1.1.116: icmp: echo reply
13: 18:19:26.649258 802.1Q vlan#1 P0 10.1.1.116 > 10.1.106.2: icmp: echo request
14: 18:19:26.649914 802.1Q vlan#1 P0 10.1.106.2 > 10.1.1.116: icmp: echo reply
15: 18:19:27.649960 802.1Q vlan#1 P0 10.1.1.116 > 10.1.106.2: icmp: echo request
16: 18:19:27.650829 802.1Q vlan#1 P0 10.1.106.2 > 10.1.1.116: icmp: echo reply
17: 18:19:32.648236 802.1Q vlan#1 P0 10.1.1.116 > 10.1.106.2: icmp: echo request
18: 18:19:32.648922 802.1Q vlan#1 P0 10.1.106.2 > 10.1.1.116: icmp: echo reply
19: 18:19:37.646542 802.1Q vlan#1 P0 10.1.1.116 > 10.1.106.2: icmp: echo request
20: 18:19:37.647122 802.1Q vlan#1 P0 10.1.106.2 > 10.1.1.116: icmp: echo reply

++ It specifies that return packet is getting drop in the middle and as it only have ISP in the middle so the cause resides at ISP end.

++ Captured esp traffic as well :

Site A VPN# sh cap capout

445 packets captured

   1: 18:44:05.690058 XX.XXX.XXX.45 > XX.xxx.xxxx.67: ip-proto-50, length 100
   2: 18:44:07.462103 XX.XXX.XXX.45 > XX.XXX.XX.67: ip-proto-50, length 100
   3: 18:44:09.270936 XX.XXX.XXX.45 > XX.XXX.XX.67: ip-proto-50, length 196
   4: 18:44:09.297958 XX.XXX.XXX.45 > XX.XXX.XX.67: ip-proto-50, length 196
   5: 18:44:09.323194 XX.XXX.XX.67 > XX.XXX.XXX.45: ip-proto-50, length 196
   6: 18:44:10.690088 XX.XXX.XXX.45 > XX.XXX.XX.67: ip-proto-50, length 100
   7: 18:44:11.874878 XX.XXX.XXX.45 > XX.XXX.XX.67: ip-proto-50, length 196
   8: 18:44:12.462241 XX.XXX.XXX.45 > XX.XXX.XX.67: ip-proto-50, length 100
   9: 18:44:12.486882 XX.XXX.XX.67 > XX.XXX.XXX.45: ip-proto-50, length 100
10: 18:44:13.463232 XX.XXX.XXX.45 > XX.XXX.XX.67: ip-proto-50, length 100
11: 18:44:13.489110 XX.XXX.XX.67 > XX.XXX.XXX.45: ip-proto-50, length 100
12: 18:44:14.464285 XX.XXX.XXX.45 > XX.XXX.XX.67: ip-proto-50, length 100
13: 18:44:14.489919 XX.XXX.XX.67 > XX.XXX.XXX.45: ip-proto-50, length 100
14: 18:44:14.870774 XX.XXX.XXX.45 > XX.XXX.XX.67: ip-proto-50, length 196
15: 18:44:15.465338 XX.XXX.XXX.45 > XX.XXX.XX.67: ip-proto-50, length 100
16: 18:44:15.491963 XX.XXX.XX.67 > XX.XXX.XXX.45: ip-proto-50, length 100

++
# sh cap capout

420 packets captured

   1: 18:45:57.844347 802.1Q vlan#2 P0 XX.XX.XXX.45 > XX.XXX.XX.67: ip-proto-50, length 100
   2: 18:45:57.845644 802.1Q vlan#2 P0 XX.XXX.XX.67 > XXX.XXX.XxX.45: ip-proto-50, length 100
   3: 18:46:00.569993 802.1Q vlan#2 P0 XX.XXX.XXX.45 > XX.XXX.XX.67: ip-proto-50, length 100
   4: 18:46:00.571290 802.1Q vlan#2 P0 XX.XXX.XX.67 > XX.XXX.XXX.45: ip-proto-50, length 100
   5: 18:46:01.572358 802.1Q vlan#2 P0 XX.XXX.XXX.45 > XX.XXX.XX.67: ip

No errors on the ethernet interfaces or when I show IPsec Sa.

Karsten Iwen · ‎11-08-2015

How are you connected to the internet? It looks a little bit like a problem I was facing some time ago.

The DSL-Modem (a ZyXel in "bridged" mode) was dropping IPSec-packets. After replacing the DSL-modem, the problem was fixed.

Deepakumar1 · ‎11-12-2015

Hi Will,

Try to ping from the ASA to remote end public ip. By this way the ASA will going to send the traffic without vpn, which means no esp involved. This will help you to narrow down the issue is with ISP or with your vpn setup.

If there are drops then the issue is between two ASA any device or ISP and if not then we can check the vpn seutp fruther.

Regards,

Deepak Kumar

swj · ‎11-12-2015

Hi William,

I see the update, good amount off troubleshooting done on this. As I understood Ping drops only via tunnel not on the physical outside interface.

From the above note I understood that issue only with ESP traffic.

From the Inside captures on both firewalls it looks Site-B ASA is responding correctly for what the packets it receives. From the above captures it was not clear is all the packets which Site A send was received on Site B.

To understand the situation better. Please help me with the below action plan.

"Ping 10.1.1.1 -l 900 -n 10" (Pinging with size and count). At the same time take the capture on the internal and external interfaces on both firewalls.

By using size we can differentiate the packet on the ESP and on inside also. Please attach the internal captures and show the output of ESP captures.

Thanks

Swj

William Cooper · ‎11-19-2015

All,

Resolved by ISP. Despite continued attempts to get the specifics of the blocked

traffic, the only information I could get was that they reset a card in the path.

Thanks so much for the replies.

dereksiefarth · ‎12-11-2015

William,

Out of curiosity, which ISP reset the card, Verizon or Comcast?

Dropping ping, esp packets across VPN