ttl-exceeded- packet drop in Site to Site VPN tunnel

[loc.nguyen]

Hi,

We have a VPN site to site tunnel between USA and Asia.

The firewall in Asia can ping the Firewall in USA well, no packet loss.

However, a Asia Server  10.89.100.5 ping a  USA server  (10.0.99.73 has packets loss. 

Packet capture ASP drops show something as below. Does it relate to the issue? Please advise.

colo-fw1/pri/act# sh cap asp | i ttl
1: 12:18:02.529300 10.89.100.5 > 10.0.99.73 icmp: echo request Drop-reason: (ttl-exceeded) ttl exceeded, Drop-location: frame 0x000000aab4d031a8 flow (NA)/NA
2: 12:18:07.518619 10.89.100.5 > 10.0.99.73 icmp: echo request Drop-reason: (ttl-exceeded) ttl exceeded, Drop-location: frame 0x000000aab4d031a8 flow (NA)/NA

Thanks

Loc

[MHM Cisco World]

it easy case you have loop in routing 
do packet tracer for traffic and I will help you to find issue 
MHM

[loc.nguyen]

Thanks for quick respond. Here it is:

[root@mdta-vip1 ~]# ping 10.89.100.5
PING 10.89.100.5 (10.89.100.5) 56(84) bytes of data.
64 bytes from 10.89.100.5: icmp_seq=1 ttl=126 time=248 ms
64 bytes from 10.89.100.5: icmp_seq=2 ttl=126 time=248 ms
64 bytes from 10.89.100.5: icmp_seq=3 ttl=126 time=248 ms
64 bytes from 10.89.100.5: icmp_seq=4 ttl=126 time=249 ms
64 bytes from 10.89.100.5: icmp_seq=5 ttl=126 time=251 ms
64 bytes from 10.89.100.5: icmp_seq=6 ttl=126 time=248 ms
64 bytes from 10.89.100.5: icmp_seq=7 ttl=126 time=248 ms
64 bytes from 10.89.100.5: icmp_seq=8 ttl=126 time=248 ms
64 bytes from 10.89.100.5: icmp_seq=9 ttl=126 time=248 ms
64 bytes from 10.89.100.5: icmp_seq=10 ttl=126 time=247 ms
64 bytes from 10.89.100.5: icmp_seq=11 ttl=126 time=248 ms
64 bytes from 10.89.100.5: icmp_seq=13 ttl=126 time=247 ms
64 bytes from 10.89.100.5: icmp_seq=14 ttl=126 time=267 ms
64 bytes from 10.89.100.5: icmp_seq=15 ttl=126 time=248 ms
64 bytes from 10.89.100.5: icmp_seq=16 ttl=126 time=248 ms
64 bytes from 10.89.100.5: icmp_seq=17 ttl=126 time=248 ms
64 bytes from 10.89.100.5: icmp_seq=18 ttl=126 time=248 ms
64 bytes from 10.89.100.5: icmp_seq=19 ttl=126 time=248 ms
64 bytes from 10.89.100.5: icmp_seq=20 ttl=126 time=248 ms
64 bytes from 10.89.100.5: icmp_seq=21 ttl=126 time=248 ms
64 bytes from 10.89.100.5: icmp_seq=22 ttl=126 time=248 ms
^C
--- 10.89.100.5 ping statistics ---
22 packets transmitted, 21 received, 4% packet loss, time 21021ms
rtt min/avg/max/mdev = 247.818/249.255/267.203/4.125 ms
[root@mdta-vip1 ~]#
[root@mdta-vip1 ~]#
[root@mdta-vip1 ~]# traceroute 10.89.100.5
traceroute to 10.89.100.5 (10.89.100.5), 30 hops max, 60 byte packets
1  gateway (10.0.99.250)  0.905 ms * *
2  10.89.100.5 (10.89.100.5)  249.933 ms  250.169 ms  249.527 ms
3  * * *
4  10.89.100.5 (10.89.100.5)  249.127 ms * *
[root@mdta-vip1 ~]#

[MHM Cisco World]

what FW you use ?

MHM

[loc.nguyen]

My end is Cisco ASA. I don't know the other end yet. 

[MHM Cisco World]

ciscoasa# show route 10.0.99.73 longer-prefixes <<- this must point to OUT (interface config with IKEv1 or IKEv2 IPsec S2S)

if it point to IN interface or any nameif interface other than IPsec then ASA have two route 
one is 10.0.0.0/8 or 10.0.0.0/16 and other is 0.0.0.0

the ASA is prefer the 10.0.0.0/8 or 10.0.0.0/16 and hence the packet is looping 
what you need is only tune your route in ASA that it 

MHM

[loc.nguyen]

- 10.0.99.73 is the server at my end. 

- I did not see my firewall has routes to 10.89.100.x ;

- Which side do you think has network loop? 

colo-fw1/pri/act# show route 10.0.99.73

Routing entry for 10.0.99.0 255.255.255.0
Known via "connected", distance 0, metric 0 (connected, via interface)
Routing Descriptor Blocks:
* directly connected, via inside
Route metric is 0, traffic share count is 1

colo-fw1/pri/act# show route 10.89.100.5

% Subnet not in table

colo-fw1/pri/act# show route 10.89.100.0

% Subnet not in table

colo-fw1/pri/act# show route

Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2, V - VPN
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, + - replicated route
SI - Static InterVRF
Gateway of last resort is 216.x.x.33 to network 0.0.0.0

S* 0.0.0.0 0.0.0.0 [1/0] via 216.x.x.33, outside
C 10.0.99.0 255.255.255.0 is directly connected, inside
L 10.0.99.250 255.255.255.255 is directly connected, inside
V 10.10.15.20 255.255.255.255 connected by VPN (advertised), outside
V 10.10.15.35 255.255.255.255 connected by VPN (advertised), outside
V 10.10.115.20 255.255.255.255 connected by VPN (advertised), outside
V 10.10.115.35 255.255.255.255 connected by VPN (advertised), outside
V 10.62.9.48 255.255.255.240 connected by VPN (advertised), outside
V 10.62.105.48 255.255.255.240 connected by VPN (advertised), outside
V 10.88.70.0 255.255.255.0 connected by VPN (advertised), outside
V 10.102.1.0 255.255.255.0 connected by VPN (advertised), outside
V 10.103.1.25 255.255.255.255 connected by VPN (advertised), outside
V 10.103.1.35 255.255.255.255 connected by VPN (advertised), outside
V 10.254.252.0 255.255.255.0 connected by VPN (advertised), outside
C 10.254.254.0 255.255.255.0 is directly connected, netlab
L 10.254.254.1 255.255.255.255 is directly connected, netlab
V 172.16.0.0 255.255.252.0 connected by VPN (advertised), outside
V 172.17.0.0 255.255.252.0 connected by VPN (advertised), outside
V 172.17.15.0 255.255.255.0 connected by VPN (advertised), outside
C 172.31.255.0 255.255.255.252 is directly connected, lanfo
L 172.31.255.1 255.255.255.255 is directly connected, lanfo
C 172.31.255.4 255.255.255.252 is directly connected, statefo
L 172.31.255.5 255.255.255.255 is directly connected, statefo
C 216.x.x.32 255.255.255.240 is directly connected, outside
L 216.x.x.36 255.255.255.255 is directly connected, outside

colo-fw1/pri/act#

[MHM Cisco World · Accepted solution]

the FW that server 10.0.99.73 connect to 
I make simple draw about issue take look 
check point I mention 

[loc.nguyen]

The network team at ASA1 worked with its ISP and change routes for it to a better path. Problem is solved. Thanks for your help.

[MHM Cisco World]

You are so so welcome 

MHM

[loc.nguyen]

Default gateway of server is the firewall. I don't see there is a room for network loop at this side.  

- Packet capture inside interface on firewall :
colo-fw1/pri/act# show cap in | i 10.0.99.73
1: 12:13:40.132576 10.89.100.5 > 10.0.99.73 icmp: echo request
2: 12:13:40.132714 10.0.99.73 > 10.89.100.5 icmp: echo reply
3: 12:13:40.441505 10.89.100.5 > 10.0.99.73 icmp: echo request
4: 12:13:40.441658 10.0.99.73 > 10.89.100.5 icmp: echo reply
7: 12:13:41.143272 10.89.100.5 > 10.0.99.73 icmp: echo request
8: 12:13:41.143455 10.0.99.73 > 10.89.100.5 icmp: echo reply

- Server 10.0.99.73 routing table:

[locngu@mdta-vip1 ~]$ route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.0.99.250 0.0.0.0 UG 100 0 0 ens192
0.0.0.0 10.0.0.250 0.0.0.0 UG 101 0 0 ens224
10.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 ens224
10.0.0.0 0.0.0.0 255.255.255.0 U 100 0 0 ens224
10.0.99.0 0.0.0.0 255.255.255.0 U 0 0 0 ens192
10.0.99.0 0.0.0.0 255.255.255.0 U 100 0 0 ens192
192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0
[locngu@mdta-vip1 ~]$

.250 is firewall IP
colo-fw1/pri/act# show ip
System IP Addresses:
Interface Name IP address Subnet mask Method
Port-channel1 inside 10.0.99.250 255.255.255.0 CONFIG

[MHM Cisco World]

This capture in ASA2 (in my topolgy) inside is good there is no looping.

Below in your original post is from asa2 ( in my topolgy)

colo-fw1/pri/act# sh cap asp | i ttl
1: 12:18:02.529300 10.89.100.5 > 10.0.99.73 icmp: echo request Drop-reason: (ttl-exceeded) ttl exceeded, Drop-location: frame 0x000000aab4d031a8 flow (NA)/NA

MHM

[loc.nguyen]

Sorry I just read this. So you think loop may stay at ASA1 network, don't you?

[MHM Cisco World]

colo-fw1/pri/act# sh cap asp | i ttl 
1: 12:18:02.529300 10.89.100.5 > 10.0.99.73 icmp: echo request Drop-reason: (ttl-exceeded) ttl exceeded, Drop-location: frame 0x000000aab4d031a8 flow (NA)/NA

this Colo-FW1 is ASA2 in my topolgy which connect to server 10.0.99.73 ?
if Yes and then you 

capture traffic in ASA2 Inside interface and you see request/reply ?

MHM

[loc.nguyen]

Yes, I see replied from inside interface:

colo-fw1/pri/act# show cap in | i 10.0.99.73
1: 12:13:40.132576 10.89.100.5 > 10.0.99.73 icmp: echo request
2: 12:13:40.132714 10.0.99.73 > 10.89.100.5 icmp: echo reply
3: 12:13:40.441505 10.89.100.5 > 10.0.99.73 icmp: echo request
4: 12:13:40.441658 10.0.99.73 > 10.89.100.5 icmp: echo reply
7: 12:13:41.143272 10.89.100.5 > 10.0.99.73 icmp: echo request