topic Re: ttl-exceeded- packet drop in Site to Site VPN tunnel in Network Security

ttl-exceeded- packet drop in Site to Site VPN tunnel

loc.nguyen — Tue, 02 Jul 2024 17:30:24 GMT

Hi,

We have a VPN site to site tunnel between USA and Asia.

The firewall in Asia can ping the Firewall in USA well, no packet loss.

However, a Asia Server 10.89.100.5 ping a USA server (10.0.99.73 has packets loss.

Packet capture ASP drops show something as below. Does it relate to the issue? Please advise.

colo-fw1/pri/act# sh cap asp | i ttl
1: 12:18:02.529300 10.89.100.5 > 10.0.99.73 icmp: echo request Drop-reason: (ttl-exceeded) ttl exceeded, Drop-location: frame 0x000000aab4d031a8 flow (NA)/NA
2: 12:18:07.518619 10.89.100.5 > 10.0.99.73 icmp: echo request Drop-reason: (ttl-exceeded) ttl exceeded, Drop-location: frame 0x000000aab4d031a8 flow (NA)/NA

Thanks

Loc

Re: ttl-exceeded- packet drop in Site to Site VPN tunnel

MHM Cisco World — Tue, 02 Jul 2024 17:33:15 GMT

it easy case you have loop in routing
do packet tracer for traffic and I will help you to find issue
MHM

Re: ttl-exceeded- packet drop in Site to Site VPN tunnel

loc.nguyen — Tue, 02 Jul 2024 17:43:51 GMT

Thanks for quick respond. Here it is:

[root@mdta-vip1 ~]# ping 10.89.100.5
PING 10.89.100.5 (10.89.100.5) 56(84) bytes of data.
64 bytes from 10.89.100.5: icmp_seq=1 ttl=126 time=248 ms
64 bytes from 10.89.100.5: icmp_seq=2 ttl=126 time=248 ms
64 bytes from 10.89.100.5: icmp_seq=3 ttl=126 time=248 ms
64 bytes from 10.89.100.5: icmp_seq=4 ttl=126 time=249 ms
64 bytes from 10.89.100.5: icmp_seq=5 ttl=126 time=251 ms
64 bytes from 10.89.100.5: icmp_seq=6 ttl=126 time=248 ms
64 bytes from 10.89.100.5: icmp_seq=7 ttl=126 time=248 ms
64 bytes from 10.89.100.5: icmp_seq=8 ttl=126 time=248 ms
64 bytes from 10.89.100.5: icmp_seq=9 ttl=126 time=248 ms
64 bytes from 10.89.100.5: icmp_seq=10 ttl=126 time=247 ms
64 bytes from 10.89.100.5: icmp_seq=11 ttl=126 time=248 ms
64 bytes from 10.89.100.5: icmp_seq=13 ttl=126 time=247 ms
64 bytes from 10.89.100.5: icmp_seq=14 ttl=126 time=267 ms
64 bytes from 10.89.100.5: icmp_seq=15 ttl=126 time=248 ms
64 bytes from 10.89.100.5: icmp_seq=16 ttl=126 time=248 ms
64 bytes from 10.89.100.5: icmp_seq=17 ttl=126 time=248 ms
64 bytes from 10.89.100.5: icmp_seq=18 ttl=126 time=248 ms
64 bytes from 10.89.100.5: icmp_seq=19 ttl=126 time=248 ms
64 bytes from 10.89.100.5: icmp_seq=20 ttl=126 time=248 ms
64 bytes from 10.89.100.5: icmp_seq=21 ttl=126 time=248 ms
64 bytes from 10.89.100.5: icmp_seq=22 ttl=126 time=248 ms
^C
--- 10.89.100.5 ping statistics ---
22 packets transmitted, 21 received, 4% packet loss, time 21021ms
rtt min/avg/max/mdev = 247.818/249.255/267.203/4.125 ms
[root@mdta-vip1 ~]#
[root@mdta-vip1 ~]#
[root@mdta-vip1 ~]# traceroute 10.89.100.5
traceroute to 10.89.100.5 (10.89.100.5), 30 hops max, 60 byte packets
1 gateway (10.0.99.250) 0.905 ms * *
2 10.89.100.5 (10.89.100.5) 249.933 ms 250.169 ms 249.527 ms
3 * * *
4 10.89.100.5 (10.89.100.5) 249.127 ms * *
[root@mdta-vip1 ~]#

Re: ttl-exceeded- packet drop in Site to Site VPN tunnel

MHM Cisco World — Tue, 02 Jul 2024 17:47:29 GMT

what FW you use ?

MHM

Re: ttl-exceeded- packet drop in Site to Site VPN tunnel

loc.nguyen — Tue, 02 Jul 2024 17:50:20 GMT

My end is Cisco ASA. I don't know the other end yet.

Re: ttl-exceeded- packet drop in Site to Site VPN tunnel

MHM Cisco World — Tue, 02 Jul 2024 18:04:31 GMT

ciscoasa# show route 10.0.99.73 longer-prefixes <<- this must point to OUT (interface config with IKEv1 or IKEv2 IPsec S2S)

if it point to IN interface or any nameif interface other than IPsec then ASA have two route
one is 10.0.0.0/8 or 10.0.0.0/16 and other is 0.0.0.0

the ASA is prefer the 10.0.0.0/8 or 10.0.0.0/16 and hence the packet is looping
what you need is only tune your route in ASA that it

MHM

Re: ttl-exceeded- packet drop in Site to Site VPN tunnel

loc.nguyen — Tue, 02 Jul 2024 18:23:10 GMT

- 10.0.99.73 is the server at my end.

- I did not see my firewall has routes to 10.89.100.x ;

- Which side do you think has network loop?

colo-fw1/pri/act# show route 10.0.99.73

Routing entry for 10.0.99.0 255.255.255.0
Known via "connected", distance 0, metric 0 (connected, via interface)
Routing Descriptor Blocks:
* directly connected, via inside
Route metric is 0, traffic share count is 1

colo-fw1/pri/act# show route 10.89.100.5

% Subnet not in table

colo-fw1/pri/act# show route 10.89.100.0

% Subnet not in table

colo-fw1/pri/act# show route

Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2, V - VPN
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, + - replicated route
SI - Static InterVRF
Gateway of last resort is 216.x.x.33 to network 0.0.0.0

S* 0.0.0.0 0.0.0.0 [1/0] via 216.x.x.33, outside
C 10.0.99.0 255.255.255.0 is directly connected, inside
L 10.0.99.250 255.255.255.255 is directly connected, inside
V 10.10.15.20 255.255.255.255 connected by VPN (advertised), outside
V 10.10.15.35 255.255.255.255 connected by VPN (advertised), outside
V 10.10.115.20 255.255.255.255 connected by VPN (advertised), outside
V 10.10.115.35 255.255.255.255 connected by VPN (advertised), outside
V 10.62.9.48 255.255.255.240 connected by VPN (advertised), outside
V 10.62.105.48 255.255.255.240 connected by VPN (advertised), outside
V 10.88.70.0 255.255.255.0 connected by VPN (advertised), outside
V 10.102.1.0 255.255.255.0 connected by VPN (advertised), outside
V 10.103.1.25 255.255.255.255 connected by VPN (advertised), outside
V 10.103.1.35 255.255.255.255 connected by VPN (advertised), outside
V 10.254.252.0 255.255.255.0 connected by VPN (advertised), outside
C 10.254.254.0 255.255.255.0 is directly connected, netlab
L 10.254.254.1 255.255.255.255 is directly connected, netlab
V 172.16.0.0 255.255.252.0 connected by VPN (advertised), outside
V 172.17.0.0 255.255.252.0 connected by VPN (advertised), outside
V 172.17.15.0 255.255.255.0 connected by VPN (advertised), outside
C 172.31.255.0 255.255.255.252 is directly connected, lanfo
L 172.31.255.1 255.255.255.255 is directly connected, lanfo
C 172.31.255.4 255.255.255.252 is directly connected, statefo
L 172.31.255.5 255.255.255.255 is directly connected, statefo
C 216.x.x.32 255.255.255.240 is directly connected, outside
L 216.x.x.36 255.255.255.255 is directly connected, outside

colo-fw1/pri/act#

Re: ttl-exceeded- packet drop in Site to Site VPN tunnel

MHM Cisco World — Tue, 02 Jul 2024 18:45:52 GMT

the FW that server 10.0.99.73 connect to
I make simple draw about issue take look
check point I mention

Re: ttl-exceeded- packet drop in Site to Site VPN tunnel

loc.nguyen — Tue, 02 Jul 2024 20:03:22 GMT

Default gateway of server is the firewall. I don't see there is a room for network loop at this side.

- Packet capture inside interface on firewall :
colo-fw1/pri/act# show cap in | i 10.0.99.73
1: 12:13:40.132576 10.89.100.5 > 10.0.99.73 icmp: echo request
2: 12:13:40.132714 10.0.99.73 > 10.89.100.5 icmp: echo reply
3: 12:13:40.441505 10.89.100.5 > 10.0.99.73 icmp: echo request
4: 12:13:40.441658 10.0.99.73 > 10.89.100.5 icmp: echo reply
7: 12:13:41.143272 10.89.100.5 > 10.0.99.73 icmp: echo request
8: 12:13:41.143455 10.0.99.73 > 10.89.100.5 icmp: echo reply

- Server 10.0.99.73 routing table:

[locngu@mdta-vip1 ~]$ route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.0.99.250 0.0.0.0 UG 100 0 0 ens192
0.0.0.0 10.0.0.250 0.0.0.0 UG 101 0 0 ens224
10.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 ens224
10.0.0.0 0.0.0.0 255.255.255.0 U 100 0 0 ens224
10.0.99.0 0.0.0.0 255.255.255.0 U 0 0 0 ens192
10.0.99.0 0.0.0.0 255.255.255.0 U 100 0 0 ens192
192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0
[locngu@mdta-vip1 ~]$

.250 is firewall IP
colo-fw1/pri/act# show ip
System IP Addresses:
Interface Name IP address Subnet mask Method
Port-channel1 inside 10.0.99.250 255.255.255.0 CONFIG

Re: ttl-exceeded- packet drop in Site to Site VPN tunnel

loc.nguyen — Tue, 02 Jul 2024 20:05:40 GMT

Do you think the issue stays here?

[locngu@mdta-vip1 ~]$ route -n | grep 100
0.0.0.0 10.0.99.250 0.0.0.0 UG 100 0 0 ens192
10.0.0.0 0.0.0.0 255.255.255.0 U 100 0 0 ens224

Re: ttl-exceeded- packet drop in Site to Site VPN tunnel

MHM Cisco World — Tue, 02 Jul 2024 20:09:50 GMT

This routing for Which device in my topolgy ?

MHM

Re: ttl-exceeded- packet drop in Site to Site VPN tunnel

loc.nguyen — Tue, 02 Jul 2024 20:16:47 GMT

- Server 10.0.99.73 routing table=server 2:

L3SW: we don't have L3 switch.

Re: ttl-exceeded- packet drop in Site to Site VPN tunnel

MHM Cisco World — Tue, 02 Jul 2024 20:23:46 GMT

This capture in ASA2 (in my topolgy) inside is good there is no looping.

Below in your original post is from asa2 ( in my topolgy)

colo-fw1/pri/act# sh cap asp | i ttl
1: 12:18:02.529300 10.89.100.5 > 10.0.99.73 icmp: echo request Drop-reason: (ttl-exceeded) ttl exceeded, Drop-location: frame 0x000000aab4d031a8 flow (NA)/NA

MHM

Re: ttl-exceeded- packet drop in Site to Site VPN tunnel

loc.nguyen — Tue, 02 Jul 2024 20:47:21 GMT

Sorry I just read this. So you think loop may stay at ASA1 network, don't you?

Re: ttl-exceeded- packet drop in Site to Site VPN tunnel

MHM Cisco World — Tue, 02 Jul 2024 21:06:40 GMT

colo-fw1/pri/act# sh cap asp | i ttl
1: 12:18:02.529300 10.89.100.5 > 10.0.99.73 icmp: echo request Drop-reason: (ttl-exceeded) ttl exceeded, Drop-location: frame 0x000000aab4d031a8 flow (NA)/NA

this Colo-FW1 is ASA2 in my topolgy which connect to server 10.0.99.73 ?
if Yes and then you

capture traffic in ASA2 Inside interface and you see request/reply ?

MHM

Re: ttl-exceeded- packet drop in Site to Site VPN tunnel

loc.nguyen — Tue, 02 Jul 2024 21:55:55 GMT

Yes, I see replied from inside interface:

colo-fw1/pri/act# show cap in | i 10.0.99.73
1: 12:13:40.132576 10.89.100.5 > 10.0.99.73 icmp: echo request
2: 12:13:40.132714 10.0.99.73 > 10.89.100.5 icmp: echo reply
3: 12:13:40.441505 10.89.100.5 > 10.0.99.73 icmp: echo request
4: 12:13:40.441658 10.0.99.73 > 10.89.100.5 icmp: echo reply
7: 12:13:41.143272 10.89.100.5 > 10.0.99.73 icmp: echo request

Re: ttl-exceeded- packet drop in Site to Site VPN tunnel

MHM Cisco World — Tue, 02 Jul 2024 21:59:45 GMT

then it temporally loop, if you ping from site to site over vpn and there is no drop then every thing is OK.
MHM

Re: ttl-exceeded- packet drop in Site to Site VPN tunnel

loc.nguyen — Wed, 03 Jul 2024 00:29:00 GMT

Nope, the drop in ASP-DROP is still there.

51: 19:25:17.180944 10.89.100.5 > 10.0.99.73 icmp: echo request Drop-reason: (ttl-exceeded) ttl exceeded, Drop-location: frame 0x000000aab4d031a8 flow (NA)/NA
71: 19:25:22.168951 10.89.100.5 > 10.0.99.73 icmp: echo request Drop-reason: (ttl-exceeded) ttl exceeded, Drop-location: frame 0x000000aab4d031a8 flow (NA)/NA
colo-fw1/pri/act#

Ping still replies normally in the inside interface.

colo-fw1/pri/act# show cap in | i 10.0.99.73
3: 19:27:54.472708 10.89.100.5 > 10.0.99.73 icmp: echo request
4: 19:27:54.472875 10.0.99.73 > 10.89.100.5 icmp: echo reply
5: 19:27:54.540545 10.0.99.73.4001 > 10.89.100.131.50082: P 3096680285:3096680333(48) ack 1114464398 win 238
6: 19:27:54.541414 10.0.99.73.4001 > 10.89.100.148.64767: P 2833687425:2833687473(48) ack 1772863204 win 238
7: 19:27:54.830080 10.89.100.148.64767 > 10.0.99.73.4001: . ack 2833687473 win 1020
8: 19:27:54.830447 10.89.100.131.50082 > 10.0.99.73.4001: . ack 3096680333 win 1020
11: 19:27:55.480352 10.89.100.5 > 10.0.99.73 icmp: echo request

Users report the connection is slow.

Re: ttl-exceeded- packet drop in Site to Site VPN tunnel

loc.nguyen — Wed, 03 Jul 2024 17:04:41 GMT

The network team at ASA1 worked with its ISP and change routes for it to a better path. Problem is solved. Thanks for your help.

Re: ttl-exceeded- packet drop in Site to Site VPN tunnel

MHM Cisco World — Fri, 05 Jul 2024 11:17:34 GMT

You are so so welcome

MHM