08-15-2019 06:15 AM - edited 08-15-2019 06:24 AM
This is an odd one. We have five sites all interconnected via site VPN's. Each endpoint is a Cisco ASA 5505. All seems to be well, and the site VPN's have been in place for years. Lately one particular site had been reporting issues per our alarm monitoring provider. Which has since been addressed, but I started looking in detail at the syslogs coming from the ASA's.
Each site connects to the other four sites, and I see only one leg between two sites that's being persistently reset. Each time, there is a Reason: Lost Service event on one side, usually in conjunction with a Reason: IKE Delete event on the other side. But during these time windows, no other sites lose VPN connectivity to the endpoint. Only the one connection between these two particular sites. So it's not like the Internet connectivity is totally broken across the board for the endpoint.
All site VPN's are configured with identical parameters and all have functioned well for years now. So I suspect that these resets are occurring quickly and then the tunnel is coming back up again. These two sites have the same ISP for dedicated fiber. The other sites don't have this particular ISP. I'm wondering if it's perhaps some internal routing issue on the ISP's end or something.
I'll paste a snippet from the syslogs below. The two endpoints are 69.135.82.90 and 74.143.200.234. If anyone has any suggestions on how to best determine which side is the likely culprit I'd appreciate it. That way I have some details to provide the ISP.
Thanks!
Timestamp Syslog Facility IP Address Message 8/11/2019 2:13 Local4.Warning 10.0.1.2 "Aug 11 2019 02:13:53: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 5h:00m:29s, Bytes xmt: 14220274, Bytes rcv: 5409038, Reason: Lost Service" 8/11/2019 2:14 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 5h:00m:39s, Bytes xmt: 5409038, Bytes rcv: 14220274, Reason: IKE Delete" 8/11/2019 2:38 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 98.164.34.90, Username = 98.164.34.90, IP = 98.164.34.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 10h:45m:34s, Bytes xmt: 19478, Bytes rcv: 19946, Reason: Idle Timeout" 8/11/2019 4:47 Local4.Warning 10.0.1.2 "Aug 11 2019 04:47:03: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 2h:32m:58s, Bytes xmt: 3303582, Bytes rcv: 2036497, Reason: Lost Service" 8/11/2019 4:47 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 2h:33m:12s, Bytes xmt: 2036737, Bytes rcv: 3303582, Reason: IKE Delete" 8/11/2019 6:28 Local4.Warning 10.0.1.2 "Aug 11 2019 06:28:53: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:41m:37s, Bytes xmt: 1918382, Bytes rcv: 1055216, Reason: Lost Service" 8/11/2019 6:28 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:41m:41s, Bytes xmt: 1055216, Bytes rcv: 1918382, Reason: IKE Delete" 8/11/2019 7:36 Local4.Warning 10.0.1.2 "Aug 11 2019 07:36:43: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:07m:46s, Bytes xmt: 1437875, Bytes rcv: 779629, Reason: Lost Service" 8/11/2019 7:36 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:07m:57s, Bytes xmt: 779869, Bytes rcv: 1437875, Reason: IKE Delete" 8/11/2019 8:45 Local4.Warning 10.0.1.2 "Aug 11 2019 08:45:13: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:08m:24s, Bytes xmt: 1363576, Bytes rcv: 1033718, Reason: Lost Service" 8/11/2019 8:45 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:08m:31s, Bytes xmt: 1035758, Bytes rcv: 1363576, Reason: IKE Delete" 8/11/2019 9:21 Local4.Warning 10.0.1.2 "Aug 11 2019 09:21:03: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:35m:37s, Bytes xmt: 468155, Bytes rcv: 857477, Reason: Lost Service" 8/11/2019 9:21 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:36m:02s, Bytes xmt: 857909, Bytes rcv: 468155, Reason: Lost Service" 8/11/2019 15:35 Local4.Warning 10.0.1.2 "Aug 11 2019 15:35:44: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 6h:14m:13s, Bytes xmt: 4876299, Bytes rcv: 7836608, Reason: Lost Service" 8/11/2019 15:36 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 6h:14m:37s, Bytes xmt: 7838648, Bytes rcv: 4876299, Reason: Lost Service" 8/11/2019 17:21 Local4.Warning 10.0.1.2 "Aug 11 2019 17:21:04: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:44m:54s, Bytes xmt: 1480996, Bytes rcv: 2349008, Reason: Lost Service" 8/11/2019 17:21 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:45m:04s, Bytes xmt: 2349312, Bytes rcv: 1480996, Reason: IKE Delete" 8/11/2019 17:51 Local4.Warning 10.0.1.2 "Aug 11 2019 17:51:04: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:29m:51s, Bytes xmt: 368275, Bytes rcv: 666629, Reason: Lost Service" 8/11/2019 17:51 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:30m:15s, Bytes xmt: 668669, Bytes rcv: 368275, Reason: Lost Service" 8/11/2019 20:45 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 98.164.34.90, Username = 98.164.34.90, IP = 98.164.34.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 18h:04m:38s, Bytes xmt: 41190, Bytes rcv: 39557, Reason: Idle Timeout" 8/12/2019 1:24 Local4.Warning 10.0.1.2 "Aug 12 2019 01:24:24: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 7h:32m:53s, Bytes xmt: 9267973, Bytes rcv: 9437701, Reason: Lost Service" 8/12/2019 1:24 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 7h:33m:17s, Bytes xmt: 9437701, Bytes rcv: 9267889, Reason: Lost Service" 8/12/2019 7:37 Local4.Warning 10.0.1.2 "Aug 12 2019 07:37:25: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 6h:12m:14s, Bytes xmt: 4091805, Bytes rcv: 4053067, Reason: Lost Service" 8/12/2019 7:37 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 6h:12m:28s, Bytes xmt: 4053163, Bytes rcv: 4091805, Reason: IKE Delete" 8/12/2019 9:22 Local4.Warning 10.0.1.2 "Aug 12 2019 09:22:25: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:44m:47s, Bytes xmt: 1749285, Bytes rcv: 2127999, Reason: Lost Service" 8/12/2019 9:22 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:45m:11s, Bytes xmt: 2128659, Bytes rcv: 1749285, Reason: Lost Service" 8/12/2019 9:25 Local4.Warning 10.0.1.2 "Aug 12 2019 09:25:25: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:02m:18s, Bytes xmt: 63443, Bytes rcv: 126862, Reason: Lost Service" 8/12/2019 9:25 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:02m:28s, Bytes xmt: 126914, Bytes rcv: 63443, Reason: IKE Delete" 8/12/2019 9:53 Local4.Warning 10.0.1.2 "Aug 12 2019 09:53:55: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:28m:25s, Bytes xmt: 320455, Bytes rcv: 283718, Reason: Lost Service" 8/12/2019 9:54 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:28m:41s, Bytes xmt: 283814, Bytes rcv: 320455, Reason: IKE Delete" 8/12/2019 18:36 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 98.164.34.90, Username = 98.164.34.90, IP = 98.164.34.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 21h:50m:26s, Bytes xmt: 66764, Bytes rcv: 62270, Reason: Idle Timeout" 8/12/2019 19:45 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 98.164.34.90, Username = 98.164.34.90, IP = 98.164.34.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:06m:47s, Bytes xmt: 8206, Bytes rcv: 6835, Reason: Idle Timeout" 8/12/2019 20:23 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 98.164.34.90, Username = 98.164.34.90, IP = 98.164.34.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:30m:47s, Bytes xmt: 70, Bytes rcv: 139, Reason: Idle Timeout" 8/13/2019 1:56 Local4.Warning 10.0.1.2 "Aug 13 2019 01:56:06: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 16h:01m:49s, Bytes xmt: 13091460, Bytes rcv: 28314851, Reason: Lost Service" 8/13/2019 1:56 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 16h:02m:12s, Bytes xmt: 28340411, Bytes rcv: 13084312, Reason: Lost Service" 8/13/2019 3:15 Local4.Warning 10.0.1.2 "Aug 13 2019 03:15:26: %ASA-4-113019: Group = 12.109.127.74, Username = 12.109.127.74, IP = 12.109.127.74, Session disconnected. Session Type: LAN-to-LAN, Duration: 68d 16h:51m:14s, Bytes xmt: 680595762, Bytes rcv: 234860223, Reason: Administrator Reset" 8/13/2019 3:15 Local4.Warning 10.0.1.2 "Aug 13 2019 03:15:26: %ASA-4-113019: Group = 98.164.34.90, Username = 98.164.34.90, IP = 98.164.34.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 12d 23h:36m:53s, Bytes xmt: 3146059, Bytes rcv: 3911683, Reason: Administrator Reset" 8/13/2019 3:15 Local4.Warning 10.0.1.2 "Aug 13 2019 03:15:26: %ASA-4-113019: Group = 104.191.45.105, Username = 104.191.45.105, IP = 104.191.45.105, Session disconnected. Session Type: LAN-to-LAN, Duration: 68d 16h:51m:08s, Bytes xmt: 1396575847, Bytes rcv: 2049197431, Reason: Administrator Reset" 8/13/2019 3:15 Local4.Warning 10.0.1.2 "Aug 13 2019 03:15:26: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:18m:41s, Bytes xmt: 1190611, Bytes rcv: 1667905, Reason: Administrator Reset" 8/13/2019 3:15 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:18m:54s, Bytes xmt: 1667905, Bytes rcv: 1190611, Reason: Lost Service" 8/13/2019 4:01 Local4.Warning 10.0.1.2 "Aug 13 2019 04:01:45: %ASA-4-113019: Group = 12.109.127.74, Username = 12.109.127.74, IP = 12.109.127.74, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:00m:23s, Bytes xmt: 83420, Bytes rcv: 79890, Reason: Unknown" 8/13/2019 4:04 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:03m:16s, Bytes xmt: 39363, Bytes rcv: 24800, Reason: Lost Service" 8/13/2019 10:10 Local4.Warning 10.0.1.2 "Aug 13 2019 10:10:32: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 6h:03m:43s, Bytes xmt: 4505102, Bytes rcv: 9050343, Reason: Lost Service" 8/13/2019 10:10 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 6h:05m:54s, Bytes xmt: 9080348, Bytes rcv: 4505102, Reason: IKE Delete" 8/14/2019 3:14 Local4.Warning 10.0.1.2 "Aug 14 2019 03:14:03: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 17h:03m:21s, Bytes xmt: 20787634, Bytes rcv: 16690388, Reason: Lost Service" 8/14/2019 3:14 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 17h:03m:37s, Bytes xmt: 16691808, Bytes rcv: 20750230, Reason: Lost Service" 8/14/2019 3:24 Local4.Warning 10.0.1.2 "Aug 14 2019 03:24:53: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:10m:32s, Bytes xmt: 2241497, Bytes rcv: 675746, Reason: Lost Service" 8/14/2019 3:25 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:10m:44s, Bytes xmt: 676038, Bytes rcv: 2218777, Reason: IKE Delete" 8/14/2019 13:59 Local4.Warning 10.0.1.2 "Aug 14 2019 13:59:03: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 10h:33m:58s, Bytes xmt: 10642692, Bytes rcv: 9273951, Reason: Lost Service" 8/14/2019 13:59 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 10h:34m:07s, Bytes xmt: 9274821, Bytes rcv: 10639564, Reason: IKE Delete" 8/14/2019 20:40 Local4.Warning 10.0.1.2 "Aug 14 2019 20:40:44: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 6h:41m:31s, Bytes xmt: 8710031, Bytes rcv: 5701109, Reason: Lost Service" 8/14/2019 20:41 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 6h:41m:47s, Bytes xmt: 5701109, Bytes rcv: 8707096, Reason: IKE Delete" 8/14/2019 22:44 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 98.164.34.90, Username = 98.164.34.90, IP = 98.164.34.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 2d 2h:19m:44s, Bytes xmt: 700929, Bytes rcv: 741521, Reason: Idle Timeout" 8/15/2019 6:39 Local4.Warning 10.0.1.2 "Aug 15 2019 06:39:24: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 9h:58m:38s, Bytes xmt: 11430601, Bytes rcv: 16676528, Reason: Lost Service" 8/15/2019 6:39 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 9h:58m:37s, Bytes xmt: 16676920, Bytes rcv: 11430601, Reason: IKE Delete" 8/15/2019 6:52 Local4.Warning 10.0.1.2 "Aug 15 2019 06:52:14: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:12m:38s, Bytes xmt: 524768, Bytes rcv: 669097, Reason: Lost Service" 8/15/2019 6:52 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:12m:55s, Bytes xmt: 671137, Bytes rcv: 524768, Reason: IKE Delete" 8/15/2019 7:33 Local4.Warning 10.0.1.2 "Aug 15 2019 07:33:54: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:41m:36s, Bytes xmt: 1403019, Bytes rcv: 1268784, Reason: Lost Service" 8/15/2019 7:34 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:41m:39s, Bytes xmt: 1268936, Bytes rcv: 1403019, Reason: IKE Delete" 8/15/2019 8:01 Local4.Warning 10.0.1.2 "Aug 15 2019 08:01:24: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:27m:27s, Bytes xmt: 1024007, Bytes rcv: 919594, Reason: Lost Service" 8/15/2019 8:01 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:27m:29s, Bytes xmt: 919986, Bytes rcv: 1024007, Reason: IKE Delete" 8/15/2019 8:05 Local4.Warning 10.0.1.2 "Aug 15 2019 08:05:44: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:04m:17s, Bytes xmt: 167093, Bytes rcv: 154160, Reason: Lost Service" 8/15/2019 8:05 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:04m:16s, Bytes xmt: 154688, Bytes rcv: 167053, Reason: IKE Delete"
Solved! Go to Solution.
08-15-2019 11:36 AM
I observe that the messages mostly are in pairs reporting the same time stamp and nearly same duration. Here are some examples marked to emphasize the comparison
8/11/2019 7:36 Local4.Warning 10.0.1.2 "Aug 11 2019 07:36:43: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:07m:46s, Bytes xmt: 1437875, Bytes rcv: 779629, Reason: Lost Service" 8/11/2019 7:36 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:07m:57s, Bytes xmt: 779869, Bytes rcv: 1437875, Reason: IKE Delete" 8/11/2019 8:45 Local4.Warning 10.0.1.2 "Aug 11 2019 08:45:13: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:08m:24s, Bytes xmt: 1363576, Bytes rcv: 1033718, Reason: Lost Service" 8/11/2019 8:45 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:08m:31s, Bytes xmt: 1035758, Bytes rcv: 1363576, Reason: IKE Delete" 8/11/2019 9:21 Local4.Warning 10.0.1.2 "Aug 11 2019 09:21:03: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:35m:37s, Bytes xmt: 468155, Bytes rcv: 857477, Reason: Lost Service" 8/11/2019 9:21 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:36m:02s, Bytes xmt: 857909, Bytes rcv: 468155, Reason: Lost Service"
Sometimes both peers report Lost Service and sometimes one reports Lost Service and the other reports IKE Delete. Do I understand correctly that both peers are served by the same ISP and that no other sites are served by this ISP? It suggests to me that there is something going on in that ISP network that is impacting their connectivity.
HTH
Rick
08-15-2019 10:10 AM
On average the site VPN tunnel between the two sites only stays up for 1-2 hours before being abruptly disconnected. So I'm going fishing by observing the debug output of these commands on each ASA.
debug crypto isakmp 127
debug crypto ipsec 127
debug crypto ikev1 127
08-15-2019 11:36 AM
I observe that the messages mostly are in pairs reporting the same time stamp and nearly same duration. Here are some examples marked to emphasize the comparison
8/11/2019 7:36 Local4.Warning 10.0.1.2 "Aug 11 2019 07:36:43: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:07m:46s, Bytes xmt: 1437875, Bytes rcv: 779629, Reason: Lost Service" 8/11/2019 7:36 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:07m:57s, Bytes xmt: 779869, Bytes rcv: 1437875, Reason: IKE Delete" 8/11/2019 8:45 Local4.Warning 10.0.1.2 "Aug 11 2019 08:45:13: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:08m:24s, Bytes xmt: 1363576, Bytes rcv: 1033718, Reason: Lost Service" 8/11/2019 8:45 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:08m:31s, Bytes xmt: 1035758, Bytes rcv: 1363576, Reason: IKE Delete" 8/11/2019 9:21 Local4.Warning 10.0.1.2 "Aug 11 2019 09:21:03: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:35m:37s, Bytes xmt: 468155, Bytes rcv: 857477, Reason: Lost Service" 8/11/2019 9:21 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:36m:02s, Bytes xmt: 857909, Bytes rcv: 468155, Reason: Lost Service"
Sometimes both peers report Lost Service and sometimes one reports Lost Service and the other reports IKE Delete. Do I understand correctly that both peers are served by the same ISP and that no other sites are served by this ISP? It suggests to me that there is something going on in that ISP network that is impacting their connectivity.
HTH
Rick
08-15-2019 11:43 AM - edited 08-15-2019 11:46 AM
That is correct. These two sites lose connectivity to each other every couple of hours on average. But at those times they don't drop site VPN tunnels from the other sites. Just with each other. And these two sites have Spectrum Enterprise dedicated fiber Internet. So I can tell the circuits are fine, but there is perhaps something going on in terms of internal routing on the provider's part. The other three sites have providers other than Spectrum.
I am running the debug traces on both ASA's now and once the tunnel drops I should have more specifics that I can provide. And depending on those I might open a ticket with Spectrum.
Thanks for the extra set of eyes!
08-15-2019 01:00 PM
You are quite welcome. Frequently extra pair of eyes helps find things and I hope it will be the case here. I will be especially interested in the output of the isakmp debug to see if the isakmp dead peer detection (sometimes called isakmp keep alive) is involved, which might be the case if there is some internal routing issue.
HTH
Rick
08-16-2019 06:04 AM
I have keepalives disabled on these site VPN's on either end of each connection. I was able to capture a debug of a dropped tunnel. Didn't see anything that would indicate that a Cisco ASA configured event would've caused it (e.g. - DPD being triggered). I'll paste the details below. This was after invoking debug crypto isakmp 254, debug crypto ipsec 254, and debug crypto ikev1 254.
Figure I'll open a ticket with Spectrum to look into this. The other site VPN connections that go to other sites with other ISP's stay up. Ugh!
Debug on 10.0.4.5 (74.143.200.234) ---------------------------------- IPSEC: Destroy current outbound SPI: 0x4EC39FC1 IPSEC: Deleted outbound encrypt rule, SPI 0x4EC39FC1 Rule ID: 0xccf57868 IPSEC: Deleted outbound permit rule, SPI 0x4EC39FC1 Rule ID: 0xcbc4a8e8 IPSEC: Deleted outbound VPN context, SPI 0x4EC39FC1 VPN handle: 0x0267e89c IPSEC: Destroy current inbound SPI: 0x2E02B4EB IPSEC: Deleted inbound decrypt rule, SPI 0x2E02B4EB Rule ID: 0xcccdd5a0 IPSEC: Deleted inbound permit rule, SPI 0x2E02B4EB Rule ID: 0xcccdd648 IPSEC: Deleted inbound tunnel flow rule, SPI 0x2E02B4EB Rule ID: 0xc8314a70 IPSEC: Deleted inbound VPN context, SPI 0x2E02B4EB VPN handle: 0x0268199c IPSEC: Removed SA from last received DB, SPI: 0x2E02B4EB, user: 69.135.82.90, peer: 69.135.82.90, SessionID: 0x00285000 IPSEC(crypto_map_check)-3: Looking for crypto map matching 5-tuple: Prot=6, saddr=10.0.4.5, sport=29295, daddr=10.0.1.5, dport=34560 IPSEC(crypto_map_check)-5: Checking crypto map outside_map 1: skipping because 5-tuple does not match ACL outside_cryptomap_1. IPSEC(crypto_map_check)-5: Checking crypto map outside_map 2: skipping because 5-tuple does not match ACL outside_cryptomap_2. IPSEC(crypto_map_check)-3: Checking crypto map outside_map 3: matched. IPSEC(crypto_map_check)-3: Looking for crypto map matching 5-tuple: Prot=6, saddr=10.0.4.5, sport=29295, daddr=10.0.1.5, dport=34560 IPSEC(crypto_map_check)-5: Checking crypto map outside_map 1: skipping because 5-tuple does not match ACL outside_cryptomap_1. IPSEC(crypto_map_check)-5: Checking crypto map outside_map 2: skipping because 5-tuple does not match ACL outside_cryptomap_2. IPSEC(crypto_map_check)-3: Checking crypto map outside_map 3: matched. IPSEC: Creating IPsec SA IPSEC: Getting the inbound SPI IPSEC: New embryonic SA created @ 0xcd0fc160, SCB: 0xCCCDBF20, Direction: inbound SPI : 0x9EEC77C1 Session ID: 0x00286000 VPIF num : 0x00000003 Tunnel type: l2l Protocol : esp Lifetime : 240 seconds Aug 16 08:42:20 [IKEv1]IKE Receiver: Packet received on 74.143.200.234:500 from 69.135.82.90:500 Aug 16 08:42:20 [IKEv1]IKE Receiver: Packet received on 74.143.200.234:500 from 69.135.82.90:500 IPSEC: Creating IPsec SA IPSEC: Adding the outbound SA, SPI: 0x0320CCB2 IPSEC: New embryonic SA created @ 0xcbc4a3e0, SCB: 0xC8316FD0, Direction: outbound SPI : 0x0320CCB2 Session ID: 0x00286000 VPIF num : 0x00000003 Tunnel type: l2l Protocol : esp Lifetime : 240 seconds IPSEC: Completed host OBSA update, SPI 0x0320CCB2 IPSEC: Creating outbound VPN context, SPI 0x0320CCB2 Flags: 0x00000005 SA : 0xcbc4a3e0 SPI : 0x0320CCB2 MTU : 1500 bytes VCID : 0x00000000 Peer : 0x00000000 SCB : 0xC22CC10B Channel: 0xc82fca00 IPSEC: Completed outbound VPN context, SPI 0x0320CCB2 VPN handle: 0x0268b054 IPSEC: New outbound encrypt rule, SPI 0x0320CCB2 Src addr: 10.0.4.0 Src mask: 255.255.255.0 Dst addr: 10.0.1.0 Dst mask: 255.255.255.0 Src ports Upper: 0 Lower: 0 Op : ignore Dst ports Upper: 0 Lower: 0 Op : ignore Protocol: 0 Use protocol: false SPI: 0x00000000 Use SPI: false IPSEC: Completed outbound encrypt rule, SPI 0x0320CCB2 Rule ID: 0xccc8c440 IPSEC: New outbound permit rule, SPI 0x0320CCB2 Src addr: 74.143.200.234 Src mask: 255.255.255.255 Dst addr: 69.135.82.90 Dst mask: 255.255.255.255 Src ports Upper: 0 Lower: 0 Op : ignore Dst ports Upper: 0 Lower: 0 Op : ignore Protocol: 50 Use protocol: true SPI: 0x0320CCB2 Use SPI: true IPSEC: Completed outbound permit rule, SPI 0x0320CCB2 Rule ID: 0xccd38338 IPSEC: Creating IPsec SA IPSEC: Updating the inbound SA, SPI: 0x9EEC77C1 IPSEC: New embryonic SA created @ 0xcd0fc160, SCB: 0xCCCDBF20, Direction: inbound SPI : 0x9EEC77C1 Session ID: 0x00286000 VPIF num : 0x00000003 Tunnel type: l2l Protocol : esp Lifetime : 240 seconds IPSEC: Completed host IBSA update, SPI 0x9EEC77C1 IPSEC: Creating inbound VPN context, SPI 0x9EEC77C1 Flags: 0x00000006 SA : 0xcd0fc160 SPI : 0x9EEC77C1 MTU : 0 bytes VCID : 0x00000000 Peer : 0x0268B054 SCB : 0xC1F2A111 Channel: 0xc82fca00 IPSEC: Completed inbound VPN context, SPI 0x9EEC77C1 VPN handle: 0x026932f4 IPSEC: Updating outbound VPN context 0x0268B054, SPI 0x0320CCB2 Flags: 0x00000005 SA : 0xcbc4a3e0 SPI : 0x0320CCB2 MTU : 1500 bytes VCID : 0x00000000 Peer : 0x026932F4 SCB : 0xC22CC10B Channel: 0xc82fca00 IPSEC: Completed outbound VPN context, SPI 0x0320CCB2 VPN handle: 0x0268b054 IPSEC: Completed outbound inner rule, SPI 0x0320CCB2 Rule ID: 0xccc8c440 IPSEC: Completed outbound outer SPD rule, SPI 0x0320CCB2 Rule ID: 0xccd38338 IPSEC: New inbound tunnel flow rule, SPI 0x9EEC77C1 Src addr: 10.0.1.0 Src mask: 255.255.255.0 Dst addr: 10.0.4.0 Dst mask: 255.255.255.0 Src ports Upper: 0 Lower: 0 Op : ignore Dst ports Upper: 0 Lower: 0 Op : ignore Protocol: 0 Use protocol: false SPI: 0x00000000 Use SPI: false IPSEC: Completed inbound tunnel flow rule, SPI 0x9EEC77C1 Rule ID: 0xccb59970 IPSEC: New inbound decrypt rule, SPI 0x9EEC77C1 Src addr: 69.135.82.90 Src mask: 255.255.255.255 Dst addr: 74.143.200.234 Dst mask: 255.255.255.255 Src ports Upper: 0 Lower: 0 Op : ignore Dst ports Upper: 0 Lower: 0 Op : ignore Protocol: 50 Use protocol: true SPI: 0x9EEC77C1 Use SPI: true IPSEC: Completed inbound decrypt rule, SPI 0x9EEC77C1 Rule ID: 0xc8314a70 IPSEC: New inbound permit rule, SPI 0x9EEC77C1 Src addr: 69.135.82.90 Src mask: 255.255.255.255 Dst addr: 74.143.200.234 Dst mask: 255.255.255.255 Src ports Upper: 0 Lower: 0 Op : ignore Dst ports Upper: 0 Lower: 0 Op : ignore Protocol: 50 Use protocol: true SPI: 0x9EEC77C1 Use SPI: true IPSEC: Completed inbound permit rule, SPI 0x9EEC77C1 Rule ID: 0xccb5c878 IPSEC: Added SA to last received DB, SPI: 0x9EEC77C1, user: 69.135.82.90, peer: 69.135.82.90, SessionID: 0x00286000 Debug on 10.0.1.2 (69.135.82.90) -------------------------------- IPSEC: Destroy current outbound SPI: 0x2E02B4EB IPSEC: Deleted outbound encrypt rule, SPI 0x2E02B4EB Rule ID: 0xccf37d78 IPSEC: Deleted outbound permit rule, SPI 0x2E02B4EB Rule ID: 0xcce6c788 IPSEC: Deleted outbound VPN context, SPI 0x2E02B4EB VPN handle: 0x0018bdf4 IPSEC: Destroy current inbound SPI: 0x4EC39FC1 IPSEC: Deleted inbound decrypt rule, SPI 0x4EC39FC1 Rule ID: 0xc8fe5e98 IPSEC: Deleted inbound permit rule, SPI 0x4EC39FC1 Rule ID: 0xc8fe7ef8 IPSEC: Deleted inbound tunnel flow rule, SPI 0x4EC39FC1 Rule ID: 0xccd7e610 IPSEC: Deleted inbound VPN context, SPI 0x4EC39FC1 VPN handle: 0x00194014 IPSEC: Removed SA from last received DB, SPI: 0x4EC39FC1, user: 74.143.200.234, peer: 74.143.200.234, SessionID: 0x0001A000 IPSEC: Creating IPsec SA IPSEC: Getting the inbound SPI IPSEC: New embryonic SA created @ 0xccbe92e0, SCB: 0xCCC23C48, Direction: inbound SPI : 0x0320CCB2 Session ID: 0x00020000 VPIF num : 0x00000003 Tunnel type: l2l Protocol : esp Lifetime : 240 seconds IPSEC: Creating IPsec SA IPSEC: Adding the outbound SA, SPI: 0x9EEC77C1 IPSEC: New embryonic SA created @ 0xcce851f0, SCB: 0xCCBF97C0, Direction: outbound SPI : 0x9EEC77C1 Session ID: 0x00020000 VPIF num : 0x00000003 Tunnel type: l2l Protocol : esp Lifetime : 240 seconds IPSEC: Completed host OBSA update, SPI 0x9EEC77C1 IPSEC: Creating outbound VPN context, SPI 0x9EEC77C1 Flags: 0x00000005 SA : 0xcce851f0 SPI : 0x9EEC77C1 MTU : 1500 bytes VCID : 0x00000000 Peer : 0x00000000 SCB : 0x6597CE75 Channel: 0xc82fca80 IPSEC: Completed outbound VPN context, SPI 0x9EEC77C1 VPN handle: 0x001bc09c IPSEC: New outbound encrypt rule, SPI 0x9EEC77C1 Src addr: 10.0.1.0 Src mask: 255.255.255.0 Dst addr: 10.0.4.0 Dst mask: 255.255.255.0 Src ports Upper: 0 Lower: 0 Op : ignore Dst ports Upper: 0 Lower: 0 Op : ignore Protocol: 0 Use protocol: false SPI: 0x00000000 Use SPI: false IPSEC: Completed outbound encrypt rule, SPI 0x9EEC77C1 Rule ID: 0xccd5df38 IPSEC: New outbound permit rule, SPI 0x9EEC77C1 Src addr: 69.135.82.90 Src mask: 255.255.255.255 Dst addr: 74.143.200.234 Dst mask: 255.255.255.255 Src ports Upper: 0 Lower: 0 Op : ignore Dst ports Upper: 0 Lower: 0 Op : ignore Protocol: 50 Use protocol: true SPI: 0x9EEC77C1 Use SPI: true IPSEC: Completed outbound permit rule, SPI 0x9EEC77C1 Rule ID: 0xccddc338 IPSEC: Creating IPsec SA IPSEC: Updating the inbound SA, SPI: 0x0320CCB2 IPSEC: New embryonic SA created @ 0xccbe92e0, SCB: 0xCCC23C48, Direction: inbound SPI : 0x0320CCB2 Session ID: 0x00020000 VPIF num : 0x00000003 Tunnel type: l2l Protocol : esp Lifetime : 240 seconds IPSEC: Completed host IBSA update, SPI 0x0320CCB2 IPSEC: Creating inbound VPN context, SPI 0x0320CCB2 Flags: 0x00000006 SA : 0xccbe92e0 SPI : 0x0320CCB2 MTU : 0 bytes VCID : 0x00000000 Peer : 0x001BC09C SCB : 0x659481AD Channel: 0xc82fca80 IPSEC: Completed inbound VPN context, SPI 0x0320CCB2 VPN handle: 0x001c119c IPSEC: Updating outbound VPN context 0x001BC09C, SPI 0x9EEC77C1 Flags: 0x00000005 SA : 0xcce851f0 SPI : 0x9EEC77C1 MTU : 1500 bytes VCID : 0x00000000 Peer : 0x001C119C SCB : 0x6597CE75 Channel: 0xc82fca80 IPSEC: Completed outbound VPN context, SPI 0x9EEC77C1 VPN handle: 0x001bc09c IPSEC: Completed outbound inner rule, SPI 0x9EEC77C1 Rule ID: 0xccd5df38 IPSEC: Completed outbound outer SPD rule, SPI 0x9EEC77C1 Rule ID: 0xccddc338 IPSEC: New inbound tunnel flow rule, SPI 0x0320CCB2 Src addr: 10.0.4.0 Src mask: 255.255.255.0 Dst addr: 10.0.1.0 Dst mask: 255.255.255.0 Src ports Upper: 0 Lower: 0 Op : ignore Dst ports Upper: 0 Lower: 0 Op : ignore Protocol: 0 Use protocol: false SPI: 0x00000000 Use SPI: false IPSEC: Completed inbound tunnel flow rule, SPI 0x0320CCB2 Rule ID: 0xccd5e6d0 IPSEC: New inbound decrypt rule, SPI 0x0320CCB2 Src addr: 74.143.200.234 Src mask: 255.255.255.255 Dst addr: 69.135.82.90 Dst mask: 255.255.255.255 Src ports Upper: 0 Lower: 0 Op : ignore Dst ports Upper: 0 Lower: 0 Op : ignore Protocol: 50 Use protocol: true SPI: 0x0320CCB2 Use SPI: true IPSEC: Completed inbound decrypt rule, SPI 0x0320CCB2 Rule ID: 0xccbf87c8 IPSEC: New inbound permit rule, SPI 0x0320CCB2 Src addr: 74.143.200.234 Src mask: 255.255.255.255 Dst addr: 69.135.82.90 Dst mask: 255.255.255.255 Src ports Upper: 0 Lower: 0 Op : ignore Dst ports Upper: 0 Lower: 0 Op : ignore Protocol: 50 Use protocol: true SPI: 0x0320CCB2 Use SPI: true IPSEC: Completed inbound permit rule, SPI 0x0320CCB2 Rule ID: 0xcd061e48 IPSEC: Added SA to last received DB, SPI: 0x0320CCB2, user: 74.143.200.234, peer: 74.143.200.234, SessionID: 0x00020000 Syslogs ------- 2019-08-16 08:41:56 Local4.Warning 10.0.1.2 Aug 16 2019 08:41:56: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:10m:30s, Bytes xmt: 1331064, Bytes rcv: 2019519, Reason: Lost Service 2019-08-16 08:42:19 Local4.Warning 10.0.4.1 %ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:10m:53s, Bytes xmt: 2021599, Bytes rcv: 1331064, Reason: Lost Service
08-16-2019 10:45 AM
So after opening a ticket with Spectrum I have provided them with lots of these detailed examples. Here's one interesting tidbit. Not sure if it's indicative or not. I ran a traceroute between the endpoints while the tunnel was up, waited until it dropped, and then ran another traceroute immediately thereafter. The routes between these endpoints appear to differ before and after the tunnel dropped.
8/16/2019 @ 13:20 ----------------- traceroute 74.143.200.233 source outside Type escape sequence to abort. Tracing the route to 74.143.200.233 1 rrcs-69-135-82-89.central.biz.rr.com (69.135.82.89) 0 msec 0 msec 10 msec 2 69.23.11.1 0 msec 10 msec 69.23.11.3 10 msec 3 agg45.clmkohpe02r.midwest.rr.com (65.189.106.108) 10 msec ae1.clmloh0602h.midwest.rr.com (65.189.106.10) 0 msec agg45.clmkohpe02r.midwest.rr.com (65.189.106.108) 0 msec 4 ae1.wevlohoh02h.midwest.rr.com (65.29.17.67) 10 msec 65.29.17.65 10 msec ae1.wevlohoh02h.midwest.rr.com (65.29.17.67) 10 msec 5 rrcs-74-143-200-233.central.biz.rr.com (74.143.200.233) 0 msec 0 msec 10 msec --------------------------------------------------------------------------------- traceroute 69.135.82.89 source outside Type escape sequence to abort. Tracing the route to 69.135.82.89 1 rrcs-74-143-200-233.central.biz.rr.com (74.143.200.233) 0 msec 0 msec 0 msec 2 ae15.wevlohoh02h.midwest.rr.com (69.23.11.7) 0 msec ae15.wevlohoh01h.midwest.rr.com (69.23.11.5) 0 msec ae15.wevlohoh02h.midwest.rr.com (69.23.11.7) 0 msec 3 be88.clmcohib01r.midwest.rr.com (65.29.17.66) 10 msec be88.clmkohpe02r.midwest.rr.com (65.29.17.64) 20 msec be88.clmcohib01r.midwest.rr.com (65.29.17.66) 10 msec 4 65.189.106.11 10 msec ae1.clmloh0601h.midwest.rr.com (65.189.106.109) 90 msec 65.189.106.11 0 msec 5 rrcs-69-135-82-89.central.biz.rr.com (69.135.82.89) 0 msec 10 msec 10 msec =================================================================================== 2019-08-16 13:22:06 Local4.Warning 10.0.1.2 Aug 16 2019 13:22:06: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:10m:37s, Bytes xmt: 1939674, Bytes rcv: 4640005, Reason: Lost Service 2019-08-16 13:22:16 Local4.Warning 10.0.4.1 %ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:10m:33s, Bytes xmt: 4640005, Bytes rcv: 1939674, Reason: IKE Delete =================================================================================== 8/16/2019 @ 13:23 ----------------- traceroute 74.143.200.233 source outside Type escape sequence to abort. Tracing the route to 74.143.200.233 1 rrcs-69-135-82-89.central.biz.rr.com (69.135.82.89) 0 msec 0 msec 0 msec 2 69.23.11.1 10 msec 0 msec 10 msec 3 ae1.clmloh0602h.midwest.rr.com (65.189.106.10) 10 msec 10 msec 10 msec 4 ae1.wevlohoh02h.midwest.rr.com (65.29.17.67) 10 msec 0 msec 10 msec 5 rrcs-74-143-200-233.central.biz.rr.com (74.143.200.233) 10 msec 0 msec 10 msec ------------------------------------------------------------------------------------ traceroute 69.135.82.89 source outside Type escape sequence to abort. Tracing the route to 69.135.82.89 1 rrcs-74-143-200-233.central.biz.rr.com (74.143.200.233) 0 msec 0 msec 0 msec 2 ae15.wevlohoh01h.midwest.rr.com (69.23.11.5) 0 msec 0 msec ae15.wevlohoh02h.midwest.rr.com (69.23.11.7) 10 msec 3 be88.clmkohpe02r.midwest.rr.com (65.29.17.64) 0 msec 10 msec 10 msec 4 ae1.clmloh0601h.midwest.rr.com (65.189.106.109) 10 msec 10 msec 65.189.106.11 10 msec 5 rrcs-69-135-82-89.central.biz.rr.com (69.135.82.89) 0 msec 20 msec 0 msec
08-17-2019 07:10 AM - edited 08-17-2019 07:11 AM
Just a quick post-mortem on this. It was indeed a provider-side issue. Apparently Spectrum's internetworking had some incorrect references. As of this morning we should be in the clear. Per the engineer's notes on the ticket. Glad to know I wasn't crazy...in this instance at least... :)
Our engineers have cleared the LDP session to upstream router and the label mismatch has been resolved as of 5:25 am EDT.
08-17-2019 08:29 AM
Thanks for the update. Glad to know that it is confirmed as an issue in the ISP network. I am pleased that I was able to confirm your belief that it was an issue with the ISP. This discussion illustrates good steps in documenting and investigating a network issue. Thank you for marking this question as solved. This will help other participants in the community to identify discussions which have helpful information.
HTH
Rick
10-25-2019 08:32 AM - edited 10-25-2019 11:14 AM
Actually the provider still hasn't fixed the issue. It happens every 2-3 hours usually. Just between their two endpoints. The other site VPN interconnects are fine. On one end I am running Ping Plotter using another usable public IP. Logging pings back to the other provider endpoint. I do see a few instances of dead hops, that usually coincide when the site VPN interconnect quickly flaps.
More often the dead hops I see logged instances where latency is greater than 250 ms. This leads to be a question. I know you can configure the ASA's DPD so that keepalives are issued every X seconds and retry after X seconds if failed. But is there a hard-coded timeout in this mechanism? For example, let's say I have latency of 361 ms. Would the DPD keepalive mechanism give up after a smaller timeout value has been hit and assume the peer is dead?
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide