cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
6603
Views
5
Helpful
9
Replies

Determining root cause of ASA-4-113019 logged events

greg.kujawa
Level 1
Level 1

This is an odd one. We have five sites all interconnected via site VPN's. Each endpoint is a Cisco ASA 5505. All seems to be well, and the site VPN's have been in place for years. Lately one particular site had been reporting issues per our alarm monitoring provider. Which has since been addressed, but I started looking in detail at the syslogs coming from the ASA's. 

 

Each site connects to the other four sites, and I see only one leg between two sites that's being persistently reset. Each time, there is a Reason: Lost Service event on one side, usually in conjunction with a Reason: IKE Delete event on the other side. But during these time windows, no other sites lose VPN connectivity to the endpoint. Only the one connection between these two particular sites. So it's not like the Internet connectivity is totally broken across the board for the endpoint.

 

All site VPN's are configured with identical parameters and all have functioned well for years now. So I suspect that these resets are occurring quickly and then the tunnel is coming back up again. These two sites have the same ISP for dedicated fiber. The other sites don't have this particular ISP. I'm wondering if it's perhaps some internal routing issue on the ISP's end or something.

 

I'll paste a snippet from the syslogs below. The two endpoints are 69.135.82.90 and 74.143.200.234. If anyone has any suggestions on how to best determine which side is the likely culprit I'd appreciate it. That way I have some details to provide the ISP. 

 

Thanks!

 

Timestamp	Syslog Facility	IP Address	Message
8/11/2019 2:13	Local4.Warning	10.0.1.2	"Aug 11 2019 02:13:53: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 5h:00m:29s, Bytes xmt: 14220274, Bytes rcv: 5409038, Reason: Lost Service"
8/11/2019 2:14	Local4.Warning	10.0.4.1	"%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 5h:00m:39s, Bytes xmt: 5409038, Bytes rcv: 14220274, Reason: IKE Delete"
8/11/2019 2:38	Local4.Warning	10.0.4.1	"%ASA-4-113019: Group = 98.164.34.90, Username = 98.164.34.90, IP = 98.164.34.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 10h:45m:34s, Bytes xmt: 19478, Bytes rcv: 19946, Reason: Idle Timeout"
8/11/2019 4:47	Local4.Warning	10.0.1.2	"Aug 11 2019 04:47:03: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 2h:32m:58s, Bytes xmt: 3303582, Bytes rcv: 2036497, Reason: Lost Service"
8/11/2019 4:47	Local4.Warning	10.0.4.1	"%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 2h:33m:12s, Bytes xmt: 2036737, Bytes rcv: 3303582, Reason: IKE Delete"
8/11/2019 6:28	Local4.Warning	10.0.1.2	"Aug 11 2019 06:28:53: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:41m:37s, Bytes xmt: 1918382, Bytes rcv: 1055216, Reason: Lost Service"
8/11/2019 6:28	Local4.Warning	10.0.4.1	"%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:41m:41s, Bytes xmt: 1055216, Bytes rcv: 1918382, Reason: IKE Delete"
8/11/2019 7:36	Local4.Warning	10.0.1.2	"Aug 11 2019 07:36:43: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:07m:46s, Bytes xmt: 1437875, Bytes rcv: 779629, Reason: Lost Service"
8/11/2019 7:36	Local4.Warning	10.0.4.1	"%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:07m:57s, Bytes xmt: 779869, Bytes rcv: 1437875, Reason: IKE Delete"
8/11/2019 8:45	Local4.Warning	10.0.1.2	"Aug 11 2019 08:45:13: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:08m:24s, Bytes xmt: 1363576, Bytes rcv: 1033718, Reason: Lost Service"
8/11/2019 8:45	Local4.Warning	10.0.4.1	"%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:08m:31s, Bytes xmt: 1035758, Bytes rcv: 1363576, Reason: IKE Delete"
8/11/2019 9:21	Local4.Warning	10.0.1.2	"Aug 11 2019 09:21:03: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:35m:37s, Bytes xmt: 468155, Bytes rcv: 857477, Reason: Lost Service"
8/11/2019 9:21	Local4.Warning	10.0.4.1	"%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:36m:02s, Bytes xmt: 857909, Bytes rcv: 468155, Reason: Lost Service"
8/11/2019 15:35	Local4.Warning	10.0.1.2	"Aug 11 2019 15:35:44: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 6h:14m:13s, Bytes xmt: 4876299, Bytes rcv: 7836608, Reason: Lost Service"
8/11/2019 15:36	Local4.Warning	10.0.4.1	"%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 6h:14m:37s, Bytes xmt: 7838648, Bytes rcv: 4876299, Reason: Lost Service"
8/11/2019 17:21	Local4.Warning	10.0.1.2	"Aug 11 2019 17:21:04: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:44m:54s, Bytes xmt: 1480996, Bytes rcv: 2349008, Reason: Lost Service"
8/11/2019 17:21	Local4.Warning	10.0.4.1	"%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:45m:04s, Bytes xmt: 2349312, Bytes rcv: 1480996, Reason: IKE Delete"
8/11/2019 17:51	Local4.Warning	10.0.1.2	"Aug 11 2019 17:51:04: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:29m:51s, Bytes xmt: 368275, Bytes rcv: 666629, Reason: Lost Service"
8/11/2019 17:51	Local4.Warning	10.0.4.1	"%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:30m:15s, Bytes xmt: 668669, Bytes rcv: 368275, Reason: Lost Service"
8/11/2019 20:45	Local4.Warning	10.0.4.1	"%ASA-4-113019: Group = 98.164.34.90, Username = 98.164.34.90, IP = 98.164.34.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 18h:04m:38s, Bytes xmt: 41190, Bytes rcv: 39557, Reason: Idle Timeout"
8/12/2019 1:24	Local4.Warning	10.0.1.2	"Aug 12 2019 01:24:24: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 7h:32m:53s, Bytes xmt: 9267973, Bytes rcv: 9437701, Reason: Lost Service"
8/12/2019 1:24	Local4.Warning	10.0.4.1	"%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 7h:33m:17s, Bytes xmt: 9437701, Bytes rcv: 9267889, Reason: Lost Service"
8/12/2019 7:37	Local4.Warning	10.0.1.2	"Aug 12 2019 07:37:25: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 6h:12m:14s, Bytes xmt: 4091805, Bytes rcv: 4053067, Reason: Lost Service"
8/12/2019 7:37	Local4.Warning	10.0.4.1	"%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 6h:12m:28s, Bytes xmt: 4053163, Bytes rcv: 4091805, Reason: IKE Delete"
8/12/2019 9:22	Local4.Warning	10.0.1.2	"Aug 12 2019 09:22:25: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:44m:47s, Bytes xmt: 1749285, Bytes rcv: 2127999, Reason: Lost Service"
8/12/2019 9:22	Local4.Warning	10.0.4.1	"%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:45m:11s, Bytes xmt: 2128659, Bytes rcv: 1749285, Reason: Lost Service"
8/12/2019 9:25	Local4.Warning	10.0.1.2	"Aug 12 2019 09:25:25: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:02m:18s, Bytes xmt: 63443, Bytes rcv: 126862, Reason: Lost Service"
8/12/2019 9:25	Local4.Warning	10.0.4.1	"%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:02m:28s, Bytes xmt: 126914, Bytes rcv: 63443, Reason: IKE Delete"
8/12/2019 9:53	Local4.Warning	10.0.1.2	"Aug 12 2019 09:53:55: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:28m:25s, Bytes xmt: 320455, Bytes rcv: 283718, Reason: Lost Service"
8/12/2019 9:54	Local4.Warning	10.0.4.1	"%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:28m:41s, Bytes xmt: 283814, Bytes rcv: 320455, Reason: IKE Delete"
8/12/2019 18:36	Local4.Warning	10.0.4.1	"%ASA-4-113019: Group = 98.164.34.90, Username = 98.164.34.90, IP = 98.164.34.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 21h:50m:26s, Bytes xmt: 66764, Bytes rcv: 62270, Reason: Idle Timeout"
8/12/2019 19:45	Local4.Warning	10.0.4.1	"%ASA-4-113019: Group = 98.164.34.90, Username = 98.164.34.90, IP = 98.164.34.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:06m:47s, Bytes xmt: 8206, Bytes rcv: 6835, Reason: Idle Timeout"
8/12/2019 20:23	Local4.Warning	10.0.4.1	"%ASA-4-113019: Group = 98.164.34.90, Username = 98.164.34.90, IP = 98.164.34.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:30m:47s, Bytes xmt: 70, Bytes rcv: 139, Reason: Idle Timeout"
8/13/2019 1:56	Local4.Warning	10.0.1.2	"Aug 13 2019 01:56:06: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 16h:01m:49s, Bytes xmt: 13091460, Bytes rcv: 28314851, Reason: Lost Service"
8/13/2019 1:56	Local4.Warning	10.0.4.1	"%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 16h:02m:12s, Bytes xmt: 28340411, Bytes rcv: 13084312, Reason: Lost Service"
8/13/2019 3:15	Local4.Warning	10.0.1.2	"Aug 13 2019 03:15:26: %ASA-4-113019: Group = 12.109.127.74, Username = 12.109.127.74, IP = 12.109.127.74, Session disconnected. Session Type: LAN-to-LAN, Duration: 68d 16h:51m:14s, Bytes xmt: 680595762, Bytes rcv: 234860223, Reason: Administrator Reset"
8/13/2019 3:15	Local4.Warning	10.0.1.2	"Aug 13 2019 03:15:26: %ASA-4-113019: Group = 98.164.34.90, Username = 98.164.34.90, IP = 98.164.34.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 12d 23h:36m:53s, Bytes xmt: 3146059, Bytes rcv: 3911683, Reason: Administrator Reset"
8/13/2019 3:15	Local4.Warning	10.0.1.2	"Aug 13 2019 03:15:26: %ASA-4-113019: Group = 104.191.45.105, Username = 104.191.45.105, IP = 104.191.45.105, Session disconnected. Session Type: LAN-to-LAN, Duration: 68d 16h:51m:08s, Bytes xmt: 1396575847, Bytes rcv: 2049197431, Reason: Administrator Reset"
8/13/2019 3:15	Local4.Warning	10.0.1.2	"Aug 13 2019 03:15:26: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:18m:41s, Bytes xmt: 1190611, Bytes rcv: 1667905, Reason: Administrator Reset"
8/13/2019 3:15	Local4.Warning	10.0.4.1	"%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:18m:54s, Bytes xmt: 1667905, Bytes rcv: 1190611, Reason: Lost Service"
8/13/2019 4:01	Local4.Warning	10.0.1.2	"Aug 13 2019 04:01:45: %ASA-4-113019: Group = 12.109.127.74, Username = 12.109.127.74, IP = 12.109.127.74, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:00m:23s, Bytes xmt: 83420, Bytes rcv: 79890, Reason: Unknown"
8/13/2019 4:04	Local4.Warning	10.0.4.1	"%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:03m:16s, Bytes xmt: 39363, Bytes rcv: 24800, Reason: Lost Service"
8/13/2019 10:10	Local4.Warning	10.0.1.2	"Aug 13 2019 10:10:32: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 6h:03m:43s, Bytes xmt: 4505102, Bytes rcv: 9050343, Reason: Lost Service"
8/13/2019 10:10	Local4.Warning	10.0.4.1	"%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 6h:05m:54s, Bytes xmt: 9080348, Bytes rcv: 4505102, Reason: IKE Delete"
8/14/2019 3:14	Local4.Warning	10.0.1.2	"Aug 14 2019 03:14:03: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 17h:03m:21s, Bytes xmt: 20787634, Bytes rcv: 16690388, Reason: Lost Service"
8/14/2019 3:14	Local4.Warning	10.0.4.1	"%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 17h:03m:37s, Bytes xmt: 16691808, Bytes rcv: 20750230, Reason: Lost Service"
8/14/2019 3:24	Local4.Warning	10.0.1.2	"Aug 14 2019 03:24:53: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:10m:32s, Bytes xmt: 2241497, Bytes rcv: 675746, Reason: Lost Service"
8/14/2019 3:25	Local4.Warning	10.0.4.1	"%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:10m:44s, Bytes xmt: 676038, Bytes rcv: 2218777, Reason: IKE Delete"
8/14/2019 13:59	Local4.Warning	10.0.1.2	"Aug 14 2019 13:59:03: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 10h:33m:58s, Bytes xmt: 10642692, Bytes rcv: 9273951, Reason: Lost Service"
8/14/2019 13:59	Local4.Warning	10.0.4.1	"%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 10h:34m:07s, Bytes xmt: 9274821, Bytes rcv: 10639564, Reason: IKE Delete"
8/14/2019 20:40	Local4.Warning	10.0.1.2	"Aug 14 2019 20:40:44: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 6h:41m:31s, Bytes xmt: 8710031, Bytes rcv: 5701109, Reason: Lost Service"
8/14/2019 20:41	Local4.Warning	10.0.4.1	"%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 6h:41m:47s, Bytes xmt: 5701109, Bytes rcv: 8707096, Reason: IKE Delete"
8/14/2019 22:44	Local4.Warning	10.0.4.1	"%ASA-4-113019: Group = 98.164.34.90, Username = 98.164.34.90, IP = 98.164.34.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 2d 2h:19m:44s, Bytes xmt: 700929, Bytes rcv: 741521, Reason: Idle Timeout"
8/15/2019 6:39	Local4.Warning	10.0.1.2	"Aug 15 2019 06:39:24: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 9h:58m:38s, Bytes xmt: 11430601, Bytes rcv: 16676528, Reason: Lost Service"
8/15/2019 6:39	Local4.Warning	10.0.4.1	"%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 9h:58m:37s, Bytes xmt: 16676920, Bytes rcv: 11430601, Reason: IKE Delete"
8/15/2019 6:52	Local4.Warning	10.0.1.2	"Aug 15 2019 06:52:14: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:12m:38s, Bytes xmt: 524768, Bytes rcv: 669097, Reason: Lost Service"
8/15/2019 6:52	Local4.Warning	10.0.4.1	"%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:12m:55s, Bytes xmt: 671137, Bytes rcv: 524768, Reason: IKE Delete"
8/15/2019 7:33	Local4.Warning	10.0.1.2	"Aug 15 2019 07:33:54: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:41m:36s, Bytes xmt: 1403019, Bytes rcv: 1268784, Reason: Lost Service"
8/15/2019 7:34	Local4.Warning	10.0.4.1	"%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:41m:39s, Bytes xmt: 1268936, Bytes rcv: 1403019, Reason: IKE Delete"
8/15/2019 8:01	Local4.Warning	10.0.1.2	"Aug 15 2019 08:01:24: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:27m:27s, Bytes xmt: 1024007, Bytes rcv: 919594, Reason: Lost Service"
8/15/2019 8:01	Local4.Warning	10.0.4.1	"%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:27m:29s, Bytes xmt: 919986, Bytes rcv: 1024007, Reason: IKE Delete"
8/15/2019 8:05	Local4.Warning	10.0.1.2	"Aug 15 2019 08:05:44: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:04m:17s, Bytes xmt: 167093, Bytes rcv: 154160, Reason: Lost Service"
8/15/2019 8:05	Local4.Warning	10.0.4.1	"%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:04m:16s, Bytes xmt: 154688, Bytes rcv: 167053, Reason: IKE Delete"

 

1 Accepted Solution

Accepted Solutions

I observe that the messages mostly are in pairs reporting the same time stamp and nearly same duration. Here are some examples marked to emphasize the comparison

8/11/2019 7:36	Local4.Warning	10.0.1.2	"Aug 11 2019 07:36:43: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:07m:46s, Bytes xmt: 1437875, Bytes rcv: 779629, Reason: Lost Service"
8/11/2019 7:36	Local4.Warning	10.0.4.1	"%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:07m:57s, Bytes xmt: 779869, Bytes rcv: 1437875, Reason: IKE Delete"
8/11/2019 8:45	Local4.Warning	10.0.1.2	"Aug 11 2019 08:45:13: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:08m:24s, Bytes xmt: 1363576, Bytes rcv: 1033718, Reason: Lost Service"
8/11/2019 8:45	Local4.Warning	10.0.4.1	"%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:08m:31s, Bytes xmt: 1035758, Bytes rcv: 1363576, Reason: IKE Delete"
8/11/2019 9:21	Local4.Warning	10.0.1.2	"Aug 11 2019 09:21:03: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:35m:37s, Bytes xmt: 468155, Bytes rcv: 857477, Reason: Lost Service"
8/11/2019 9:21	Local4.Warning	10.0.4.1	"%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:36m:02s, Bytes xmt: 857909, Bytes rcv: 468155, Reason: Lost Service"

 

Sometimes both peers report Lost Service and sometimes one reports Lost Service and the other reports IKE Delete. Do I understand correctly that both peers are served by the same ISP and that no other sites are served by this ISP? It suggests to me that there is something going on in that ISP network that is impacting their connectivity.

 

HTH

 

Rick

HTH

Rick

View solution in original post

9 Replies 9

greg.kujawa
Level 1
Level 1

On average the site VPN tunnel between the two sites only stays up for 1-2 hours before being abruptly disconnected. So I'm going fishing by observing the debug output of these commands on each ASA.

 

debug crypto isakmp 127
debug crypto ipsec 127
debug crypto ikev1 127

 

I observe that the messages mostly are in pairs reporting the same time stamp and nearly same duration. Here are some examples marked to emphasize the comparison

8/11/2019 7:36	Local4.Warning	10.0.1.2	"Aug 11 2019 07:36:43: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:07m:46s, Bytes xmt: 1437875, Bytes rcv: 779629, Reason: Lost Service"
8/11/2019 7:36	Local4.Warning	10.0.4.1	"%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:07m:57s, Bytes xmt: 779869, Bytes rcv: 1437875, Reason: IKE Delete"
8/11/2019 8:45	Local4.Warning	10.0.1.2	"Aug 11 2019 08:45:13: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:08m:24s, Bytes xmt: 1363576, Bytes rcv: 1033718, Reason: Lost Service"
8/11/2019 8:45	Local4.Warning	10.0.4.1	"%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:08m:31s, Bytes xmt: 1035758, Bytes rcv: 1363576, Reason: IKE Delete"
8/11/2019 9:21	Local4.Warning	10.0.1.2	"Aug 11 2019 09:21:03: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:35m:37s, Bytes xmt: 468155, Bytes rcv: 857477, Reason: Lost Service"
8/11/2019 9:21	Local4.Warning	10.0.4.1	"%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:36m:02s, Bytes xmt: 857909, Bytes rcv: 468155, Reason: Lost Service"

 

Sometimes both peers report Lost Service and sometimes one reports Lost Service and the other reports IKE Delete. Do I understand correctly that both peers are served by the same ISP and that no other sites are served by this ISP? It suggests to me that there is something going on in that ISP network that is impacting their connectivity.

 

HTH

 

Rick

HTH

Rick

That is correct. These two sites lose connectivity to each other every couple of hours on average. But at those times they don't drop site VPN tunnels from the other sites. Just with each other. And these two sites have Spectrum Enterprise dedicated fiber Internet. So I can tell the circuits are fine, but there is perhaps something going on in terms of internal routing on the provider's part. The other three sites have providers other than Spectrum.

 

I am running the debug traces on both ASA's now and once the tunnel drops I should have more specifics that I can provide. And depending on those I might open a ticket with Spectrum.

 

Thanks for the extra set of eyes!

You are quite welcome. Frequently extra pair of eyes helps find things and I hope it will be the case here. I will be especially interested in the output of the isakmp debug to see if the isakmp dead peer detection (sometimes called isakmp keep alive) is involved, which might be the case if there is some internal routing issue.

 

HTH

 

Rick

HTH

Rick

I have keepalives disabled on these site VPN's on either end of each connection. I was able to capture a debug of a dropped tunnel. Didn't see anything that would indicate that a Cisco ASA configured event would've caused it (e.g. - DPD being triggered). I'll paste the details below. This was after invoking debug crypto isakmp 254, debug crypto ipsec 254, and debug crypto ikev1 254.

 

Figure I'll open a ticket with Spectrum to look into this. The other site VPN connections that go to other sites with other ISP's stay up. Ugh!

 

Debug on 10.0.4.5 (74.143.200.234)
----------------------------------
IPSEC: Destroy current outbound SPI: 0x4EC39FC1
IPSEC: Deleted outbound encrypt rule, SPI 0x4EC39FC1
    Rule ID: 0xccf57868
IPSEC: Deleted outbound permit rule, SPI 0x4EC39FC1
    Rule ID: 0xcbc4a8e8
IPSEC: Deleted outbound VPN context, SPI 0x4EC39FC1
    VPN handle: 0x0267e89c
IPSEC: Destroy current inbound SPI: 0x2E02B4EB
IPSEC: Deleted inbound decrypt rule, SPI 0x2E02B4EB
    Rule ID: 0xcccdd5a0
IPSEC: Deleted inbound permit rule, SPI 0x2E02B4EB
    Rule ID: 0xcccdd648
IPSEC: Deleted inbound tunnel flow rule, SPI 0x2E02B4EB
    Rule ID: 0xc8314a70
IPSEC: Deleted inbound VPN context, SPI 0x2E02B4EB
    VPN handle: 0x0268199c
IPSEC: Removed SA from last received DB, SPI: 0x2E02B4EB, user: 69.135.82.90, peer: 69.135.82.90, SessionID: 0x00285000
IPSEC(crypto_map_check)-3: Looking for crypto map matching 5-tuple: Prot=6, saddr=10.0.4.5, sport=29295, daddr=10.0.1.5, dport=34560
IPSEC(crypto_map_check)-5: Checking crypto map outside_map 1: skipping because 5-tuple does not match ACL outside_cryptomap_1.
IPSEC(crypto_map_check)-5: Checking crypto map outside_map 2: skipping because 5-tuple does not match ACL outside_cryptomap_2.
IPSEC(crypto_map_check)-3: Checking crypto map outside_map 3: matched.
IPSEC(crypto_map_check)-3: Looking for crypto map matching 5-tuple: Prot=6, saddr=10.0.4.5, sport=29295, daddr=10.0.1.5, dport=34560
IPSEC(crypto_map_check)-5: Checking crypto map outside_map 1: skipping because 5-tuple does not match ACL outside_cryptomap_1.
IPSEC(crypto_map_check)-5: Checking crypto map outside_map 2: skipping because 5-tuple does not match ACL outside_cryptomap_2.
IPSEC(crypto_map_check)-3: Checking crypto map outside_map 3: matched.
IPSEC: Creating IPsec SA
IPSEC: Getting the inbound SPI
IPSEC: New embryonic SA created @ 0xcd0fc160, 
    SCB: 0xCCCDBF20, 
    Direction: inbound
    SPI      : 0x9EEC77C1
    Session ID: 0x00286000
    VPIF num  : 0x00000003
    Tunnel type: l2l
    Protocol   : esp
    Lifetime   : 240 seconds
Aug 16 08:42:20 [IKEv1]IKE Receiver: Packet received on 74.143.200.234:500 from 69.135.82.90:500
Aug 16 08:42:20 [IKEv1]IKE Receiver: Packet received on 74.143.200.234:500 from 69.135.82.90:500
IPSEC: Creating IPsec SA
IPSEC: Adding the outbound SA, SPI: 0x0320CCB2
IPSEC: New embryonic SA created @ 0xcbc4a3e0, 
    SCB: 0xC8316FD0, 
    Direction: outbound
    SPI      : 0x0320CCB2
    Session ID: 0x00286000
    VPIF num  : 0x00000003
    Tunnel type: l2l
    Protocol   : esp
    Lifetime   : 240 seconds
IPSEC: Completed host OBSA update, SPI 0x0320CCB2
IPSEC: Creating outbound VPN context, SPI 0x0320CCB2
    Flags: 0x00000005
    SA   : 0xcbc4a3e0
    SPI  : 0x0320CCB2
    MTU  : 1500 bytes
    VCID : 0x00000000
    Peer : 0x00000000
    SCB  : 0xC22CC10B
    Channel: 0xc82fca00
IPSEC: Completed outbound VPN context, SPI 0x0320CCB2
    VPN handle: 0x0268b054
IPSEC: New outbound encrypt rule, SPI 0x0320CCB2
    Src addr: 10.0.4.0
    Src mask: 255.255.255.0
    Dst addr: 10.0.1.0
    Dst mask: 255.255.255.0
    Src ports
      Upper: 0
      Lower: 0
      Op   : ignore
    Dst ports
      Upper: 0
      Lower: 0
      Op   : ignore
    Protocol: 0
    Use protocol: false
    SPI: 0x00000000
    Use SPI: false
IPSEC: Completed outbound encrypt rule, SPI 0x0320CCB2
    Rule ID: 0xccc8c440
IPSEC: New outbound permit rule, SPI 0x0320CCB2
    Src addr: 74.143.200.234
    Src mask: 255.255.255.255
    Dst addr: 69.135.82.90
    Dst mask: 255.255.255.255
    Src ports
      Upper: 0
      Lower: 0
      Op   : ignore
    Dst ports
      Upper: 0
      Lower: 0
      Op   : ignore
    Protocol: 50
    Use protocol: true
    SPI: 0x0320CCB2
    Use SPI: true
IPSEC: Completed outbound permit rule, SPI 0x0320CCB2
    Rule ID: 0xccd38338
IPSEC: Creating IPsec SA
IPSEC: Updating the inbound SA, SPI: 0x9EEC77C1
IPSEC: New embryonic SA created @ 0xcd0fc160, 
    SCB: 0xCCCDBF20, 
    Direction: inbound
    SPI      : 0x9EEC77C1
    Session ID: 0x00286000
    VPIF num  : 0x00000003
    Tunnel type: l2l
    Protocol   : esp
    Lifetime   : 240 seconds
IPSEC: Completed host IBSA update, SPI 0x9EEC77C1
IPSEC: Creating inbound VPN context, SPI 0x9EEC77C1
    Flags: 0x00000006
    SA   : 0xcd0fc160
    SPI  : 0x9EEC77C1
    MTU  : 0 bytes
    VCID : 0x00000000
    Peer : 0x0268B054
    SCB  : 0xC1F2A111
    Channel: 0xc82fca00
IPSEC: Completed inbound VPN context, SPI 0x9EEC77C1
    VPN handle: 0x026932f4
IPSEC: Updating outbound VPN context 0x0268B054, SPI 0x0320CCB2
    Flags: 0x00000005
    SA   : 0xcbc4a3e0
    SPI  : 0x0320CCB2
    MTU  : 1500 bytes
    VCID : 0x00000000
    Peer : 0x026932F4
    SCB  : 0xC22CC10B
    Channel: 0xc82fca00
IPSEC: Completed outbound VPN context, SPI 0x0320CCB2
    VPN handle: 0x0268b054
IPSEC: Completed outbound inner rule, SPI 0x0320CCB2
    Rule ID: 0xccc8c440
IPSEC: Completed outbound outer SPD rule, SPI 0x0320CCB2
    Rule ID: 0xccd38338
IPSEC: New inbound tunnel flow rule, SPI 0x9EEC77C1
    Src addr: 10.0.1.0
    Src mask: 255.255.255.0
    Dst addr: 10.0.4.0
    Dst mask: 255.255.255.0
    Src ports
      Upper: 0
      Lower: 0
      Op   : ignore
    Dst ports
      Upper: 0
      Lower: 0
      Op   : ignore
    Protocol: 0
    Use protocol: false
    SPI: 0x00000000
    Use SPI: false
IPSEC: Completed inbound tunnel flow rule, SPI 0x9EEC77C1
    Rule ID: 0xccb59970
IPSEC: New inbound decrypt rule, SPI 0x9EEC77C1
    Src addr: 69.135.82.90
    Src mask: 255.255.255.255
    Dst addr: 74.143.200.234
    Dst mask: 255.255.255.255
    Src ports
      Upper: 0
      Lower: 0
      Op   : ignore
    Dst ports
      Upper: 0
      Lower: 0
      Op   : ignore
    Protocol: 50
    Use protocol: true
    SPI: 0x9EEC77C1
    Use SPI: true
IPSEC: Completed inbound decrypt rule, SPI 0x9EEC77C1
    Rule ID: 0xc8314a70
IPSEC: New inbound permit rule, SPI 0x9EEC77C1
    Src addr: 69.135.82.90
    Src mask: 255.255.255.255
    Dst addr: 74.143.200.234
    Dst mask: 255.255.255.255
    Src ports
      Upper: 0
      Lower: 0
      Op   : ignore
    Dst ports
      Upper: 0
      Lower: 0
      Op   : ignore
    Protocol: 50
    Use protocol: true
    SPI: 0x9EEC77C1
    Use SPI: true
IPSEC: Completed inbound permit rule, SPI 0x9EEC77C1
    Rule ID: 0xccb5c878
IPSEC: Added SA to last received DB, SPI: 0x9EEC77C1, user: 69.135.82.90, peer: 69.135.82.90, SessionID: 0x00286000

Debug on 10.0.1.2 (69.135.82.90)
--------------------------------
IPSEC: Destroy current outbound SPI: 0x2E02B4EB
IPSEC: Deleted outbound encrypt rule, SPI 0x2E02B4EB
    Rule ID: 0xccf37d78
IPSEC: Deleted outbound permit rule, SPI 0x2E02B4EB
    Rule ID: 0xcce6c788
IPSEC: Deleted outbound VPN context, SPI 0x2E02B4EB
    VPN handle: 0x0018bdf4
IPSEC: Destroy current inbound SPI: 0x4EC39FC1
IPSEC: Deleted inbound decrypt rule, SPI 0x4EC39FC1
    Rule ID: 0xc8fe5e98
IPSEC: Deleted inbound permit rule, SPI 0x4EC39FC1
    Rule ID: 0xc8fe7ef8
IPSEC: Deleted inbound tunnel flow rule, SPI 0x4EC39FC1
    Rule ID: 0xccd7e610
IPSEC: Deleted inbound VPN context, SPI 0x4EC39FC1
    VPN handle: 0x00194014
IPSEC: Removed SA from last received DB, SPI: 0x4EC39FC1, user: 74.143.200.234, peer: 74.143.200.234, SessionID: 0x0001A000
IPSEC: Creating IPsec SA
IPSEC: Getting the inbound SPI
IPSEC: New embryonic SA created @ 0xccbe92e0, 
    SCB: 0xCCC23C48, 
    Direction: inbound
    SPI      : 0x0320CCB2
    Session ID: 0x00020000
    VPIF num  : 0x00000003
    Tunnel type: l2l
    Protocol   : esp
    Lifetime   : 240 seconds
IPSEC: Creating IPsec SA
IPSEC: Adding the outbound SA, SPI: 0x9EEC77C1
IPSEC: New embryonic SA created @ 0xcce851f0, 
    SCB: 0xCCBF97C0, 
    Direction: outbound
    SPI      : 0x9EEC77C1
    Session ID: 0x00020000
    VPIF num  : 0x00000003
    Tunnel type: l2l
    Protocol   : esp
    Lifetime   : 240 seconds
IPSEC: Completed host OBSA update, SPI 0x9EEC77C1
IPSEC: Creating outbound VPN context, SPI 0x9EEC77C1
    Flags: 0x00000005
    SA   : 0xcce851f0
    SPI  : 0x9EEC77C1
    MTU  : 1500 bytes
    VCID : 0x00000000
    Peer : 0x00000000
    SCB  : 0x6597CE75
    Channel: 0xc82fca80
IPSEC: Completed outbound VPN context, SPI 0x9EEC77C1
    VPN handle: 0x001bc09c
IPSEC: New outbound encrypt rule, SPI 0x9EEC77C1
    Src addr: 10.0.1.0
    Src mask: 255.255.255.0
    Dst addr: 10.0.4.0
    Dst mask: 255.255.255.0
    Src ports
      Upper: 0
      Lower: 0
      Op   : ignore
    Dst ports
      Upper: 0
      Lower: 0
      Op   : ignore
    Protocol: 0
    Use protocol: false
    SPI: 0x00000000
    Use SPI: false
IPSEC: Completed outbound encrypt rule, SPI 0x9EEC77C1
    Rule ID: 0xccd5df38
IPSEC: New outbound permit rule, SPI 0x9EEC77C1
    Src addr: 69.135.82.90
    Src mask: 255.255.255.255
    Dst addr: 74.143.200.234
    Dst mask: 255.255.255.255
    Src ports
      Upper: 0
      Lower: 0
      Op   : ignore
    Dst ports
      Upper: 0
      Lower: 0
      Op   : ignore
    Protocol: 50
    Use protocol: true
    SPI: 0x9EEC77C1
    Use SPI: true
IPSEC: Completed outbound permit rule, SPI 0x9EEC77C1
    Rule ID: 0xccddc338
IPSEC: Creating IPsec SA
IPSEC: Updating the inbound SA, SPI: 0x0320CCB2
IPSEC: New embryonic SA created @ 0xccbe92e0, 
    SCB: 0xCCC23C48, 
    Direction: inbound
    SPI      : 0x0320CCB2
    Session ID: 0x00020000
    VPIF num  : 0x00000003
    Tunnel type: l2l
    Protocol   : esp
    Lifetime   : 240 seconds
IPSEC: Completed host IBSA update, SPI 0x0320CCB2
IPSEC: Creating inbound VPN context, SPI 0x0320CCB2
    Flags: 0x00000006
    SA   : 0xccbe92e0
    SPI  : 0x0320CCB2
    MTU  : 0 bytes
    VCID : 0x00000000
    Peer : 0x001BC09C
    SCB  : 0x659481AD
    Channel: 0xc82fca80
IPSEC: Completed inbound VPN context, SPI 0x0320CCB2
    VPN handle: 0x001c119c
IPSEC: Updating outbound VPN context 0x001BC09C, SPI 0x9EEC77C1
    Flags: 0x00000005
    SA   : 0xcce851f0
    SPI  : 0x9EEC77C1
    MTU  : 1500 bytes
    VCID : 0x00000000
    Peer : 0x001C119C
    SCB  : 0x6597CE75
    Channel: 0xc82fca80
IPSEC: Completed outbound VPN context, SPI 0x9EEC77C1
    VPN handle: 0x001bc09c
IPSEC: Completed outbound inner rule, SPI 0x9EEC77C1
    Rule ID: 0xccd5df38
IPSEC: Completed outbound outer SPD rule, SPI 0x9EEC77C1
    Rule ID: 0xccddc338
IPSEC: New inbound tunnel flow rule, SPI 0x0320CCB2
    Src addr: 10.0.4.0
    Src mask: 255.255.255.0
    Dst addr: 10.0.1.0
    Dst mask: 255.255.255.0
    Src ports
      Upper: 0
      Lower: 0
      Op   : ignore
    Dst ports
      Upper: 0
      Lower: 0
      Op   : ignore
    Protocol: 0
    Use protocol: false
    SPI: 0x00000000
    Use SPI: false
IPSEC: Completed inbound tunnel flow rule, SPI 0x0320CCB2
    Rule ID: 0xccd5e6d0
IPSEC: New inbound decrypt rule, SPI 0x0320CCB2
    Src addr: 74.143.200.234
    Src mask: 255.255.255.255
    Dst addr: 69.135.82.90
    Dst mask: 255.255.255.255
    Src ports
      Upper: 0
      Lower: 0
      Op   : ignore
    Dst ports
      Upper: 0
      Lower: 0
      Op   : ignore
    Protocol: 50
    Use protocol: true
    SPI: 0x0320CCB2
    Use SPI: true
IPSEC: Completed inbound decrypt rule, SPI 0x0320CCB2
    Rule ID: 0xccbf87c8
IPSEC: New inbound permit rule, SPI 0x0320CCB2
    Src addr: 74.143.200.234
    Src mask: 255.255.255.255
    Dst addr: 69.135.82.90
    Dst mask: 255.255.255.255
    Src ports
      Upper: 0
      Lower: 0
      Op   : ignore
    Dst ports
      Upper: 0
      Lower: 0
      Op   : ignore
    Protocol: 50
    Use protocol: true
    SPI: 0x0320CCB2
    Use SPI: true
IPSEC: Completed inbound permit rule, SPI 0x0320CCB2
    Rule ID: 0xcd061e48
IPSEC: Added SA to last received DB, SPI: 0x0320CCB2, user: 74.143.200.234, peer: 74.143.200.234, SessionID: 0x00020000

Syslogs
-------
2019-08-16 08:41:56	Local4.Warning	10.0.1.2	Aug 16 2019 08:41:56: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:10m:30s, Bytes xmt: 1331064, Bytes rcv: 2019519, Reason: Lost Service
2019-08-16 08:42:19	Local4.Warning	10.0.4.1	%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:10m:53s, Bytes xmt: 2021599, Bytes rcv: 1331064, Reason: Lost Service

 

So after opening a ticket with Spectrum I have provided them with lots of these detailed examples. Here's one interesting tidbit. Not sure if it's indicative or not. I ran a traceroute between the endpoints while the tunnel was up, waited until it dropped, and then ran another traceroute immediately thereafter. The routes between these endpoints appear to differ before and after the tunnel dropped.

 

8/16/2019 @ 13:20
-----------------
traceroute 74.143.200.233 source outside

Type escape sequence to abort.
Tracing the route to 74.143.200.233

 1  rrcs-69-135-82-89.central.biz.rr.com (69.135.82.89) 0 msec 0 msec 10 msec
 2  69.23.11.1 0 msec 10 msec
    69.23.11.3 10 msec
 3  agg45.clmkohpe02r.midwest.rr.com (65.189.106.108) 10 msec
    ae1.clmloh0602h.midwest.rr.com (65.189.106.10) 0 msec
    agg45.clmkohpe02r.midwest.rr.com (65.189.106.108) 0 msec
 4  ae1.wevlohoh02h.midwest.rr.com (65.29.17.67) 10 msec
    65.29.17.65 10 msec
    ae1.wevlohoh02h.midwest.rr.com (65.29.17.67) 10 msec
 5  rrcs-74-143-200-233.central.biz.rr.com (74.143.200.233) 0 msec 0 msec 10 msec

---------------------------------------------------------------------------------

traceroute 69.135.82.89 source outside

Type escape sequence to abort.
Tracing the route to 69.135.82.89

 1  rrcs-74-143-200-233.central.biz.rr.com (74.143.200.233) 0 msec 0 msec 0 msec
 2  ae15.wevlohoh02h.midwest.rr.com (69.23.11.7) 0 msec
    ae15.wevlohoh01h.midwest.rr.com (69.23.11.5) 0 msec
    ae15.wevlohoh02h.midwest.rr.com (69.23.11.7) 0 msec
 3  be88.clmcohib01r.midwest.rr.com (65.29.17.66) 10 msec
    be88.clmkohpe02r.midwest.rr.com (65.29.17.64) 20 msec
    be88.clmcohib01r.midwest.rr.com (65.29.17.66) 10 msec
 4  65.189.106.11 10 msec
    ae1.clmloh0601h.midwest.rr.com (65.189.106.109) 90 msec
    65.189.106.11 0 msec
 5  rrcs-69-135-82-89.central.biz.rr.com (69.135.82.89) 0 msec 10 msec 10 msec

===================================================================================

2019-08-16 13:22:06	Local4.Warning	10.0.1.2	Aug 16 2019 13:22:06: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:10m:37s, Bytes xmt: 1939674, Bytes rcv: 4640005, Reason: Lost Service
2019-08-16 13:22:16	Local4.Warning	10.0.4.1	%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:10m:33s, Bytes xmt: 4640005, Bytes rcv: 1939674, Reason: IKE Delete

===================================================================================
8/16/2019 @ 13:23
-----------------
traceroute 74.143.200.233 source outside

Type escape sequence to abort.
Tracing the route to 74.143.200.233

 1  rrcs-69-135-82-89.central.biz.rr.com (69.135.82.89) 0 msec 0 msec 0 msec
 2  69.23.11.1 10 msec 0 msec 10 msec
 3  ae1.clmloh0602h.midwest.rr.com (65.189.106.10) 10 msec 10 msec 10 msec
 4  ae1.wevlohoh02h.midwest.rr.com (65.29.17.67) 10 msec 0 msec 10 msec
 5  rrcs-74-143-200-233.central.biz.rr.com (74.143.200.233) 10 msec 0 msec 10 msec

------------------------------------------------------------------------------------

traceroute 69.135.82.89 source outside

Type escape sequence to abort.
Tracing the route to 69.135.82.89

 1  rrcs-74-143-200-233.central.biz.rr.com (74.143.200.233) 0 msec 0 msec 0 msec
 2  ae15.wevlohoh01h.midwest.rr.com (69.23.11.5) 0 msec 0 msec
    ae15.wevlohoh02h.midwest.rr.com (69.23.11.7) 10 msec
 3  be88.clmkohpe02r.midwest.rr.com (65.29.17.64) 0 msec 10 msec 10 msec
 4  ae1.clmloh0601h.midwest.rr.com (65.189.106.109) 10 msec 10 msec
    65.189.106.11 10 msec
 5  rrcs-69-135-82-89.central.biz.rr.com (69.135.82.89) 0 msec 20 msec 0 msec

 



Just a quick post-mortem on this. It was indeed a provider-side issue. Apparently Spectrum's internetworking had some incorrect references. As of this morning we should be in the clear. Per the engineer's notes on the ticket. Glad to know I wasn't crazy...in this instance at least... :)

 

Our engineers have cleared the LDP session to upstream router and the label mismatch has been resolved as of 5:25 am EDT.

Thanks for the update. Glad to know that it is confirmed as an issue in the ISP network. I am pleased that I was able to confirm your belief that it was an issue with the ISP. This discussion illustrates good steps in documenting and investigating a network issue. Thank you for marking this question as solved. This will help other participants in the community to identify discussions which have helpful information.

 

HTH

 

Rick

HTH

Rick

Actually the provider still hasn't fixed the issue. It happens every 2-3 hours usually. Just between their two endpoints. The other site VPN interconnects are fine. On one end I am running Ping Plotter using another usable public IP. Logging pings back to the other provider endpoint. I do see a few instances of dead hops, that usually coincide when the site VPN interconnect quickly flaps.

 

More often the dead hops I see logged instances where latency is greater than 250 ms. This leads to be a question. I know you can configure the ASA's DPD so that keepalives are issued every X seconds and retry after X seconds if failed. But is there a hard-coded timeout in this mechanism? For example, let's say I have latency of 361 ms. Would the DPD keepalive mechanism give up after a smaller timeout value has been hit and assume the peer is dead?

Review Cisco Networking for a $25 gift card