Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
- Cisco Community
- Technology and Support
- Networking
- Routing
- Determining root cause of ASA-4-113019 logged events
Options
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
6599
Views
5
Helpful
9
Replies
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-15-2019 06:15 AM - edited 08-15-2019 06:24 AM
This is an odd one. We have five sites all interconnected via site VPN's. Each endpoint is a Cisco ASA 5505. All seems to be well, and the site VPN's have been in place for years. Lately one particular site had been reporting issues per our alarm monitoring provider. Which has since been addressed, but I started looking in detail at the syslogs coming from the ASA's.
Each site connects to the other four sites, and I see only one leg between two sites that's being persistently reset. Each time, there is a Reason: Lost Service event on one side, usually in conjunction with a Reason: IKE Delete event on the other side. But during these time windows, no other sites lose VPN connectivity to the endpoint. Only the one connection between these two particular sites. So it's not like the Internet connectivity is totally broken across the board for the endpoint.
All site VPN's are configured with identical parameters and all have functioned well for years now. So I suspect that these resets are occurring quickly and then the tunnel is coming back up again. These two sites have the same ISP for dedicated fiber. The other sites don't have this particular ISP. I'm wondering if it's perhaps some internal routing issue on the ISP's end or something.
I'll paste a snippet from the syslogs below. The two endpoints are 69.135.82.90 and 74.143.200.234. If anyone has any suggestions on how to best determine which side is the likely culprit I'd appreciate it. That way I have some details to provide the ISP.
Thanks!
Timestamp Syslog Facility IP Address Message 8/11/2019 2:13 Local4.Warning 10.0.1.2 "Aug 11 2019 02:13:53: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 5h:00m:29s, Bytes xmt: 14220274, Bytes rcv: 5409038, Reason: Lost Service" 8/11/2019 2:14 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 5h:00m:39s, Bytes xmt: 5409038, Bytes rcv: 14220274, Reason: IKE Delete" 8/11/2019 2:38 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 98.164.34.90, Username = 98.164.34.90, IP = 98.164.34.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 10h:45m:34s, Bytes xmt: 19478, Bytes rcv: 19946, Reason: Idle Timeout" 8/11/2019 4:47 Local4.Warning 10.0.1.2 "Aug 11 2019 04:47:03: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 2h:32m:58s, Bytes xmt: 3303582, Bytes rcv: 2036497, Reason: Lost Service" 8/11/2019 4:47 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 2h:33m:12s, Bytes xmt: 2036737, Bytes rcv: 3303582, Reason: IKE Delete" 8/11/2019 6:28 Local4.Warning 10.0.1.2 "Aug 11 2019 06:28:53: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:41m:37s, Bytes xmt: 1918382, Bytes rcv: 1055216, Reason: Lost Service" 8/11/2019 6:28 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:41m:41s, Bytes xmt: 1055216, Bytes rcv: 1918382, Reason: IKE Delete" 8/11/2019 7:36 Local4.Warning 10.0.1.2 "Aug 11 2019 07:36:43: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:07m:46s, Bytes xmt: 1437875, Bytes rcv: 779629, Reason: Lost Service" 8/11/2019 7:36 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:07m:57s, Bytes xmt: 779869, Bytes rcv: 1437875, Reason: IKE Delete" 8/11/2019 8:45 Local4.Warning 10.0.1.2 "Aug 11 2019 08:45:13: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:08m:24s, Bytes xmt: 1363576, Bytes rcv: 1033718, Reason: Lost Service" 8/11/2019 8:45 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:08m:31s, Bytes xmt: 1035758, Bytes rcv: 1363576, Reason: IKE Delete" 8/11/2019 9:21 Local4.Warning 10.0.1.2 "Aug 11 2019 09:21:03: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:35m:37s, Bytes xmt: 468155, Bytes rcv: 857477, Reason: Lost Service" 8/11/2019 9:21 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:36m:02s, Bytes xmt: 857909, Bytes rcv: 468155, Reason: Lost Service" 8/11/2019 15:35 Local4.Warning 10.0.1.2 "Aug 11 2019 15:35:44: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 6h:14m:13s, Bytes xmt: 4876299, Bytes rcv: 7836608, Reason: Lost Service" 8/11/2019 15:36 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 6h:14m:37s, Bytes xmt: 7838648, Bytes rcv: 4876299, Reason: Lost Service" 8/11/2019 17:21 Local4.Warning 10.0.1.2 "Aug 11 2019 17:21:04: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:44m:54s, Bytes xmt: 1480996, Bytes rcv: 2349008, Reason: Lost Service" 8/11/2019 17:21 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:45m:04s, Bytes xmt: 2349312, Bytes rcv: 1480996, Reason: IKE Delete" 8/11/2019 17:51 Local4.Warning 10.0.1.2 "Aug 11 2019 17:51:04: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:29m:51s, Bytes xmt: 368275, Bytes rcv: 666629, Reason: Lost Service" 8/11/2019 17:51 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:30m:15s, Bytes xmt: 668669, Bytes rcv: 368275, Reason: Lost Service" 8/11/2019 20:45 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 98.164.34.90, Username = 98.164.34.90, IP = 98.164.34.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 18h:04m:38s, Bytes xmt: 41190, Bytes rcv: 39557, Reason: Idle Timeout" 8/12/2019 1:24 Local4.Warning 10.0.1.2 "Aug 12 2019 01:24:24: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 7h:32m:53s, Bytes xmt: 9267973, Bytes rcv: 9437701, Reason: Lost Service" 8/12/2019 1:24 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 7h:33m:17s, Bytes xmt: 9437701, Bytes rcv: 9267889, Reason: Lost Service" 8/12/2019 7:37 Local4.Warning 10.0.1.2 "Aug 12 2019 07:37:25: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 6h:12m:14s, Bytes xmt: 4091805, Bytes rcv: 4053067, Reason: Lost Service" 8/12/2019 7:37 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 6h:12m:28s, Bytes xmt: 4053163, Bytes rcv: 4091805, Reason: IKE Delete" 8/12/2019 9:22 Local4.Warning 10.0.1.2 "Aug 12 2019 09:22:25: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:44m:47s, Bytes xmt: 1749285, Bytes rcv: 2127999, Reason: Lost Service" 8/12/2019 9:22 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:45m:11s, Bytes xmt: 2128659, Bytes rcv: 1749285, Reason: Lost Service" 8/12/2019 9:25 Local4.Warning 10.0.1.2 "Aug 12 2019 09:25:25: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:02m:18s, Bytes xmt: 63443, Bytes rcv: 126862, Reason: Lost Service" 8/12/2019 9:25 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:02m:28s, Bytes xmt: 126914, Bytes rcv: 63443, Reason: IKE Delete" 8/12/2019 9:53 Local4.Warning 10.0.1.2 "Aug 12 2019 09:53:55: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:28m:25s, Bytes xmt: 320455, Bytes rcv: 283718, Reason: Lost Service" 8/12/2019 9:54 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:28m:41s, Bytes xmt: 283814, Bytes rcv: 320455, Reason: IKE Delete" 8/12/2019 18:36 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 98.164.34.90, Username = 98.164.34.90, IP = 98.164.34.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 21h:50m:26s, Bytes xmt: 66764, Bytes rcv: 62270, Reason: Idle Timeout" 8/12/2019 19:45 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 98.164.34.90, Username = 98.164.34.90, IP = 98.164.34.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:06m:47s, Bytes xmt: 8206, Bytes rcv: 6835, Reason: Idle Timeout" 8/12/2019 20:23 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 98.164.34.90, Username = 98.164.34.90, IP = 98.164.34.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:30m:47s, Bytes xmt: 70, Bytes rcv: 139, Reason: Idle Timeout" 8/13/2019 1:56 Local4.Warning 10.0.1.2 "Aug 13 2019 01:56:06: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 16h:01m:49s, Bytes xmt: 13091460, Bytes rcv: 28314851, Reason: Lost Service" 8/13/2019 1:56 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 16h:02m:12s, Bytes xmt: 28340411, Bytes rcv: 13084312, Reason: Lost Service" 8/13/2019 3:15 Local4.Warning 10.0.1.2 "Aug 13 2019 03:15:26: %ASA-4-113019: Group = 12.109.127.74, Username = 12.109.127.74, IP = 12.109.127.74, Session disconnected. Session Type: LAN-to-LAN, Duration: 68d 16h:51m:14s, Bytes xmt: 680595762, Bytes rcv: 234860223, Reason: Administrator Reset" 8/13/2019 3:15 Local4.Warning 10.0.1.2 "Aug 13 2019 03:15:26: %ASA-4-113019: Group = 98.164.34.90, Username = 98.164.34.90, IP = 98.164.34.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 12d 23h:36m:53s, Bytes xmt: 3146059, Bytes rcv: 3911683, Reason: Administrator Reset" 8/13/2019 3:15 Local4.Warning 10.0.1.2 "Aug 13 2019 03:15:26: %ASA-4-113019: Group = 104.191.45.105, Username = 104.191.45.105, IP = 104.191.45.105, Session disconnected. Session Type: LAN-to-LAN, Duration: 68d 16h:51m:08s, Bytes xmt: 1396575847, Bytes rcv: 2049197431, Reason: Administrator Reset" 8/13/2019 3:15 Local4.Warning 10.0.1.2 "Aug 13 2019 03:15:26: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:18m:41s, Bytes xmt: 1190611, Bytes rcv: 1667905, Reason: Administrator Reset" 8/13/2019 3:15 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:18m:54s, Bytes xmt: 1667905, Bytes rcv: 1190611, Reason: Lost Service" 8/13/2019 4:01 Local4.Warning 10.0.1.2 "Aug 13 2019 04:01:45: %ASA-4-113019: Group = 12.109.127.74, Username = 12.109.127.74, IP = 12.109.127.74, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:00m:23s, Bytes xmt: 83420, Bytes rcv: 79890, Reason: Unknown" 8/13/2019 4:04 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:03m:16s, Bytes xmt: 39363, Bytes rcv: 24800, Reason: Lost Service" 8/13/2019 10:10 Local4.Warning 10.0.1.2 "Aug 13 2019 10:10:32: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 6h:03m:43s, Bytes xmt: 4505102, Bytes rcv: 9050343, Reason: Lost Service" 8/13/2019 10:10 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 6h:05m:54s, Bytes xmt: 9080348, Bytes rcv: 4505102, Reason: IKE Delete" 8/14/2019 3:14 Local4.Warning 10.0.1.2 "Aug 14 2019 03:14:03: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 17h:03m:21s, Bytes xmt: 20787634, Bytes rcv: 16690388, Reason: Lost Service" 8/14/2019 3:14 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 17h:03m:37s, Bytes xmt: 16691808, Bytes rcv: 20750230, Reason: Lost Service" 8/14/2019 3:24 Local4.Warning 10.0.1.2 "Aug 14 2019 03:24:53: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:10m:32s, Bytes xmt: 2241497, Bytes rcv: 675746, Reason: Lost Service" 8/14/2019 3:25 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:10m:44s, Bytes xmt: 676038, Bytes rcv: 2218777, Reason: IKE Delete" 8/14/2019 13:59 Local4.Warning 10.0.1.2 "Aug 14 2019 13:59:03: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 10h:33m:58s, Bytes xmt: 10642692, Bytes rcv: 9273951, Reason: Lost Service" 8/14/2019 13:59 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 10h:34m:07s, Bytes xmt: 9274821, Bytes rcv: 10639564, Reason: IKE Delete" 8/14/2019 20:40 Local4.Warning 10.0.1.2 "Aug 14 2019 20:40:44: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 6h:41m:31s, Bytes xmt: 8710031, Bytes rcv: 5701109, Reason: Lost Service" 8/14/2019 20:41 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 6h:41m:47s, Bytes xmt: 5701109, Bytes rcv: 8707096, Reason: IKE Delete" 8/14/2019 22:44 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 98.164.34.90, Username = 98.164.34.90, IP = 98.164.34.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 2d 2h:19m:44s, Bytes xmt: 700929, Bytes rcv: 741521, Reason: Idle Timeout" 8/15/2019 6:39 Local4.Warning 10.0.1.2 "Aug 15 2019 06:39:24: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 9h:58m:38s, Bytes xmt: 11430601, Bytes rcv: 16676528, Reason: Lost Service" 8/15/2019 6:39 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 9h:58m:37s, Bytes xmt: 16676920, Bytes rcv: 11430601, Reason: IKE Delete" 8/15/2019 6:52 Local4.Warning 10.0.1.2 "Aug 15 2019 06:52:14: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:12m:38s, Bytes xmt: 524768, Bytes rcv: 669097, Reason: Lost Service" 8/15/2019 6:52 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:12m:55s, Bytes xmt: 671137, Bytes rcv: 524768, Reason: IKE Delete" 8/15/2019 7:33 Local4.Warning 10.0.1.2 "Aug 15 2019 07:33:54: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:41m:36s, Bytes xmt: 1403019, Bytes rcv: 1268784, Reason: Lost Service" 8/15/2019 7:34 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:41m:39s, Bytes xmt: 1268936, Bytes rcv: 1403019, Reason: IKE Delete" 8/15/2019 8:01 Local4.Warning 10.0.1.2 "Aug 15 2019 08:01:24: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:27m:27s, Bytes xmt: 1024007, Bytes rcv: 919594, Reason: Lost Service" 8/15/2019 8:01 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:27m:29s, Bytes xmt: 919986, Bytes rcv: 1024007, Reason: IKE Delete" 8/15/2019 8:05 Local4.Warning 10.0.1.2 "Aug 15 2019 08:05:44: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:04m:17s, Bytes xmt: 167093, Bytes rcv: 154160, Reason: Lost Service" 8/15/2019 8:05 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:04m:16s, Bytes xmt: 154688, Bytes rcv: 167053, Reason: IKE Delete"
Solved! Go to Solution.
Labels:
- Labels:
-
WAN
1 Accepted Solution
Accepted Solutions
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-15-2019 11:36 AM
I observe that the messages mostly are in pairs reporting the same time stamp and nearly same duration. Here are some examples marked to emphasize the comparison
8/11/2019 7:36 Local4.Warning 10.0.1.2 "Aug 11 2019 07:36:43: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:07m:46s, Bytes xmt: 1437875, Bytes rcv: 779629, Reason: Lost Service" 8/11/2019 7:36 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:07m:57s, Bytes xmt: 779869, Bytes rcv: 1437875, Reason: IKE Delete" 8/11/2019 8:45 Local4.Warning 10.0.1.2 "Aug 11 2019 08:45:13: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:08m:24s, Bytes xmt: 1363576, Bytes rcv: 1033718, Reason: Lost Service" 8/11/2019 8:45 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:08m:31s, Bytes xmt: 1035758, Bytes rcv: 1363576, Reason: IKE Delete" 8/11/2019 9:21 Local4.Warning 10.0.1.2 "Aug 11 2019 09:21:03: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:35m:37s, Bytes xmt: 468155, Bytes rcv: 857477, Reason: Lost Service" 8/11/2019 9:21 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:36m:02s, Bytes xmt: 857909, Bytes rcv: 468155, Reason: Lost Service"
Sometimes both peers report Lost Service and sometimes one reports Lost Service and the other reports IKE Delete. Do I understand correctly that both peers are served by the same ISP and that no other sites are served by this ISP? It suggests to me that there is something going on in that ISP network that is impacting their connectivity.
HTH
Rick
HTH
Rick
Rick
9 Replies 9
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-15-2019 10:10 AM
On average the site VPN tunnel between the two sites only stays up for 1-2 hours before being abruptly disconnected. So I'm going fishing by observing the debug output of these commands on each ASA.
debug crypto isakmp 127
debug crypto ipsec 127
debug crypto ikev1 127
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-15-2019 11:36 AM
I observe that the messages mostly are in pairs reporting the same time stamp and nearly same duration. Here are some examples marked to emphasize the comparison
8/11/2019 7:36 Local4.Warning 10.0.1.2 "Aug 11 2019 07:36:43: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:07m:46s, Bytes xmt: 1437875, Bytes rcv: 779629, Reason: Lost Service" 8/11/2019 7:36 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:07m:57s, Bytes xmt: 779869, Bytes rcv: 1437875, Reason: IKE Delete" 8/11/2019 8:45 Local4.Warning 10.0.1.2 "Aug 11 2019 08:45:13: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:08m:24s, Bytes xmt: 1363576, Bytes rcv: 1033718, Reason: Lost Service" 8/11/2019 8:45 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:08m:31s, Bytes xmt: 1035758, Bytes rcv: 1363576, Reason: IKE Delete" 8/11/2019 9:21 Local4.Warning 10.0.1.2 "Aug 11 2019 09:21:03: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:35m:37s, Bytes xmt: 468155, Bytes rcv: 857477, Reason: Lost Service" 8/11/2019 9:21 Local4.Warning 10.0.4.1 "%ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:36m:02s, Bytes xmt: 857909, Bytes rcv: 468155, Reason: Lost Service"
Sometimes both peers report Lost Service and sometimes one reports Lost Service and the other reports IKE Delete. Do I understand correctly that both peers are served by the same ISP and that no other sites are served by this ISP? It suggests to me that there is something going on in that ISP network that is impacting their connectivity.
HTH
Rick
HTH
Rick
Rick
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-15-2019 11:43 AM - edited 08-15-2019 11:46 AM
That is correct. These two sites lose connectivity to each other every couple of hours on average. But at those times they don't drop site VPN tunnels from the other sites. Just with each other. And these two sites have Spectrum Enterprise dedicated fiber Internet. So I can tell the circuits are fine, but there is perhaps something going on in terms of internal routing on the provider's part. The other three sites have providers other than Spectrum.
I am running the debug traces on both ASA's now and once the tunnel drops I should have more specifics that I can provide. And depending on those I might open a ticket with Spectrum.
Thanks for the extra set of eyes!
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-15-2019 01:00 PM
You are quite welcome. Frequently extra pair of eyes helps find things and I hope it will be the case here. I will be especially interested in the output of the isakmp debug to see if the isakmp dead peer detection (sometimes called isakmp keep alive) is involved, which might be the case if there is some internal routing issue.
HTH
Rick
HTH
Rick
Rick
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-16-2019 06:04 AM
I have keepalives disabled on these site VPN's on either end of each connection. I was able to capture a debug of a dropped tunnel. Didn't see anything that would indicate that a Cisco ASA configured event would've caused it (e.g. - DPD being triggered). I'll paste the details below. This was after invoking debug crypto isakmp 254, debug crypto ipsec 254, and debug crypto ikev1 254.
Figure I'll open a ticket with Spectrum to look into this. The other site VPN connections that go to other sites with other ISP's stay up. Ugh!
Debug on 10.0.4.5 (74.143.200.234)
----------------------------------
IPSEC: Destroy current outbound SPI: 0x4EC39FC1
IPSEC: Deleted outbound encrypt rule, SPI 0x4EC39FC1
Rule ID: 0xccf57868
IPSEC: Deleted outbound permit rule, SPI 0x4EC39FC1
Rule ID: 0xcbc4a8e8
IPSEC: Deleted outbound VPN context, SPI 0x4EC39FC1
VPN handle: 0x0267e89c
IPSEC: Destroy current inbound SPI: 0x2E02B4EB
IPSEC: Deleted inbound decrypt rule, SPI 0x2E02B4EB
Rule ID: 0xcccdd5a0
IPSEC: Deleted inbound permit rule, SPI 0x2E02B4EB
Rule ID: 0xcccdd648
IPSEC: Deleted inbound tunnel flow rule, SPI 0x2E02B4EB
Rule ID: 0xc8314a70
IPSEC: Deleted inbound VPN context, SPI 0x2E02B4EB
VPN handle: 0x0268199c
IPSEC: Removed SA from last received DB, SPI: 0x2E02B4EB, user: 69.135.82.90, peer: 69.135.82.90, SessionID: 0x00285000
IPSEC(crypto_map_check)-3: Looking for crypto map matching 5-tuple: Prot=6, saddr=10.0.4.5, sport=29295, daddr=10.0.1.5, dport=34560
IPSEC(crypto_map_check)-5: Checking crypto map outside_map 1: skipping because 5-tuple does not match ACL outside_cryptomap_1.
IPSEC(crypto_map_check)-5: Checking crypto map outside_map 2: skipping because 5-tuple does not match ACL outside_cryptomap_2.
IPSEC(crypto_map_check)-3: Checking crypto map outside_map 3: matched.
IPSEC(crypto_map_check)-3: Looking for crypto map matching 5-tuple: Prot=6, saddr=10.0.4.5, sport=29295, daddr=10.0.1.5, dport=34560
IPSEC(crypto_map_check)-5: Checking crypto map outside_map 1: skipping because 5-tuple does not match ACL outside_cryptomap_1.
IPSEC(crypto_map_check)-5: Checking crypto map outside_map 2: skipping because 5-tuple does not match ACL outside_cryptomap_2.
IPSEC(crypto_map_check)-3: Checking crypto map outside_map 3: matched.
IPSEC: Creating IPsec SA
IPSEC: Getting the inbound SPI
IPSEC: New embryonic SA created @ 0xcd0fc160,
SCB: 0xCCCDBF20,
Direction: inbound
SPI : 0x9EEC77C1
Session ID: 0x00286000
VPIF num : 0x00000003
Tunnel type: l2l
Protocol : esp
Lifetime : 240 seconds
Aug 16 08:42:20 [IKEv1]IKE Receiver: Packet received on 74.143.200.234:500 from 69.135.82.90:500
Aug 16 08:42:20 [IKEv1]IKE Receiver: Packet received on 74.143.200.234:500 from 69.135.82.90:500
IPSEC: Creating IPsec SA
IPSEC: Adding the outbound SA, SPI: 0x0320CCB2
IPSEC: New embryonic SA created @ 0xcbc4a3e0,
SCB: 0xC8316FD0,
Direction: outbound
SPI : 0x0320CCB2
Session ID: 0x00286000
VPIF num : 0x00000003
Tunnel type: l2l
Protocol : esp
Lifetime : 240 seconds
IPSEC: Completed host OBSA update, SPI 0x0320CCB2
IPSEC: Creating outbound VPN context, SPI 0x0320CCB2
Flags: 0x00000005
SA : 0xcbc4a3e0
SPI : 0x0320CCB2
MTU : 1500 bytes
VCID : 0x00000000
Peer : 0x00000000
SCB : 0xC22CC10B
Channel: 0xc82fca00
IPSEC: Completed outbound VPN context, SPI 0x0320CCB2
VPN handle: 0x0268b054
IPSEC: New outbound encrypt rule, SPI 0x0320CCB2
Src addr: 10.0.4.0
Src mask: 255.255.255.0
Dst addr: 10.0.1.0
Dst mask: 255.255.255.0
Src ports
Upper: 0
Lower: 0
Op : ignore
Dst ports
Upper: 0
Lower: 0
Op : ignore
Protocol: 0
Use protocol: false
SPI: 0x00000000
Use SPI: false
IPSEC: Completed outbound encrypt rule, SPI 0x0320CCB2
Rule ID: 0xccc8c440
IPSEC: New outbound permit rule, SPI 0x0320CCB2
Src addr: 74.143.200.234
Src mask: 255.255.255.255
Dst addr: 69.135.82.90
Dst mask: 255.255.255.255
Src ports
Upper: 0
Lower: 0
Op : ignore
Dst ports
Upper: 0
Lower: 0
Op : ignore
Protocol: 50
Use protocol: true
SPI: 0x0320CCB2
Use SPI: true
IPSEC: Completed outbound permit rule, SPI 0x0320CCB2
Rule ID: 0xccd38338
IPSEC: Creating IPsec SA
IPSEC: Updating the inbound SA, SPI: 0x9EEC77C1
IPSEC: New embryonic SA created @ 0xcd0fc160,
SCB: 0xCCCDBF20,
Direction: inbound
SPI : 0x9EEC77C1
Session ID: 0x00286000
VPIF num : 0x00000003
Tunnel type: l2l
Protocol : esp
Lifetime : 240 seconds
IPSEC: Completed host IBSA update, SPI 0x9EEC77C1
IPSEC: Creating inbound VPN context, SPI 0x9EEC77C1
Flags: 0x00000006
SA : 0xcd0fc160
SPI : 0x9EEC77C1
MTU : 0 bytes
VCID : 0x00000000
Peer : 0x0268B054
SCB : 0xC1F2A111
Channel: 0xc82fca00
IPSEC: Completed inbound VPN context, SPI 0x9EEC77C1
VPN handle: 0x026932f4
IPSEC: Updating outbound VPN context 0x0268B054, SPI 0x0320CCB2
Flags: 0x00000005
SA : 0xcbc4a3e0
SPI : 0x0320CCB2
MTU : 1500 bytes
VCID : 0x00000000
Peer : 0x026932F4
SCB : 0xC22CC10B
Channel: 0xc82fca00
IPSEC: Completed outbound VPN context, SPI 0x0320CCB2
VPN handle: 0x0268b054
IPSEC: Completed outbound inner rule, SPI 0x0320CCB2
Rule ID: 0xccc8c440
IPSEC: Completed outbound outer SPD rule, SPI 0x0320CCB2
Rule ID: 0xccd38338
IPSEC: New inbound tunnel flow rule, SPI 0x9EEC77C1
Src addr: 10.0.1.0
Src mask: 255.255.255.0
Dst addr: 10.0.4.0
Dst mask: 255.255.255.0
Src ports
Upper: 0
Lower: 0
Op : ignore
Dst ports
Upper: 0
Lower: 0
Op : ignore
Protocol: 0
Use protocol: false
SPI: 0x00000000
Use SPI: false
IPSEC: Completed inbound tunnel flow rule, SPI 0x9EEC77C1
Rule ID: 0xccb59970
IPSEC: New inbound decrypt rule, SPI 0x9EEC77C1
Src addr: 69.135.82.90
Src mask: 255.255.255.255
Dst addr: 74.143.200.234
Dst mask: 255.255.255.255
Src ports
Upper: 0
Lower: 0
Op : ignore
Dst ports
Upper: 0
Lower: 0
Op : ignore
Protocol: 50
Use protocol: true
SPI: 0x9EEC77C1
Use SPI: true
IPSEC: Completed inbound decrypt rule, SPI 0x9EEC77C1
Rule ID: 0xc8314a70
IPSEC: New inbound permit rule, SPI 0x9EEC77C1
Src addr: 69.135.82.90
Src mask: 255.255.255.255
Dst addr: 74.143.200.234
Dst mask: 255.255.255.255
Src ports
Upper: 0
Lower: 0
Op : ignore
Dst ports
Upper: 0
Lower: 0
Op : ignore
Protocol: 50
Use protocol: true
SPI: 0x9EEC77C1
Use SPI: true
IPSEC: Completed inbound permit rule, SPI 0x9EEC77C1
Rule ID: 0xccb5c878
IPSEC: Added SA to last received DB, SPI: 0x9EEC77C1, user: 69.135.82.90, peer: 69.135.82.90, SessionID: 0x00286000
Debug on 10.0.1.2 (69.135.82.90)
--------------------------------
IPSEC: Destroy current outbound SPI: 0x2E02B4EB
IPSEC: Deleted outbound encrypt rule, SPI 0x2E02B4EB
Rule ID: 0xccf37d78
IPSEC: Deleted outbound permit rule, SPI 0x2E02B4EB
Rule ID: 0xcce6c788
IPSEC: Deleted outbound VPN context, SPI 0x2E02B4EB
VPN handle: 0x0018bdf4
IPSEC: Destroy current inbound SPI: 0x4EC39FC1
IPSEC: Deleted inbound decrypt rule, SPI 0x4EC39FC1
Rule ID: 0xc8fe5e98
IPSEC: Deleted inbound permit rule, SPI 0x4EC39FC1
Rule ID: 0xc8fe7ef8
IPSEC: Deleted inbound tunnel flow rule, SPI 0x4EC39FC1
Rule ID: 0xccd7e610
IPSEC: Deleted inbound VPN context, SPI 0x4EC39FC1
VPN handle: 0x00194014
IPSEC: Removed SA from last received DB, SPI: 0x4EC39FC1, user: 74.143.200.234, peer: 74.143.200.234, SessionID: 0x0001A000
IPSEC: Creating IPsec SA
IPSEC: Getting the inbound SPI
IPSEC: New embryonic SA created @ 0xccbe92e0,
SCB: 0xCCC23C48,
Direction: inbound
SPI : 0x0320CCB2
Session ID: 0x00020000
VPIF num : 0x00000003
Tunnel type: l2l
Protocol : esp
Lifetime : 240 seconds
IPSEC: Creating IPsec SA
IPSEC: Adding the outbound SA, SPI: 0x9EEC77C1
IPSEC: New embryonic SA created @ 0xcce851f0,
SCB: 0xCCBF97C0,
Direction: outbound
SPI : 0x9EEC77C1
Session ID: 0x00020000
VPIF num : 0x00000003
Tunnel type: l2l
Protocol : esp
Lifetime : 240 seconds
IPSEC: Completed host OBSA update, SPI 0x9EEC77C1
IPSEC: Creating outbound VPN context, SPI 0x9EEC77C1
Flags: 0x00000005
SA : 0xcce851f0
SPI : 0x9EEC77C1
MTU : 1500 bytes
VCID : 0x00000000
Peer : 0x00000000
SCB : 0x6597CE75
Channel: 0xc82fca80
IPSEC: Completed outbound VPN context, SPI 0x9EEC77C1
VPN handle: 0x001bc09c
IPSEC: New outbound encrypt rule, SPI 0x9EEC77C1
Src addr: 10.0.1.0
Src mask: 255.255.255.0
Dst addr: 10.0.4.0
Dst mask: 255.255.255.0
Src ports
Upper: 0
Lower: 0
Op : ignore
Dst ports
Upper: 0
Lower: 0
Op : ignore
Protocol: 0
Use protocol: false
SPI: 0x00000000
Use SPI: false
IPSEC: Completed outbound encrypt rule, SPI 0x9EEC77C1
Rule ID: 0xccd5df38
IPSEC: New outbound permit rule, SPI 0x9EEC77C1
Src addr: 69.135.82.90
Src mask: 255.255.255.255
Dst addr: 74.143.200.234
Dst mask: 255.255.255.255
Src ports
Upper: 0
Lower: 0
Op : ignore
Dst ports
Upper: 0
Lower: 0
Op : ignore
Protocol: 50
Use protocol: true
SPI: 0x9EEC77C1
Use SPI: true
IPSEC: Completed outbound permit rule, SPI 0x9EEC77C1
Rule ID: 0xccddc338
IPSEC: Creating IPsec SA
IPSEC: Updating the inbound SA, SPI: 0x0320CCB2
IPSEC: New embryonic SA created @ 0xccbe92e0,
SCB: 0xCCC23C48,
Direction: inbound
SPI : 0x0320CCB2
Session ID: 0x00020000
VPIF num : 0x00000003
Tunnel type: l2l
Protocol : esp
Lifetime : 240 seconds
IPSEC: Completed host IBSA update, SPI 0x0320CCB2
IPSEC: Creating inbound VPN context, SPI 0x0320CCB2
Flags: 0x00000006
SA : 0xccbe92e0
SPI : 0x0320CCB2
MTU : 0 bytes
VCID : 0x00000000
Peer : 0x001BC09C
SCB : 0x659481AD
Channel: 0xc82fca80
IPSEC: Completed inbound VPN context, SPI 0x0320CCB2
VPN handle: 0x001c119c
IPSEC: Updating outbound VPN context 0x001BC09C, SPI 0x9EEC77C1
Flags: 0x00000005
SA : 0xcce851f0
SPI : 0x9EEC77C1
MTU : 1500 bytes
VCID : 0x00000000
Peer : 0x001C119C
SCB : 0x6597CE75
Channel: 0xc82fca80
IPSEC: Completed outbound VPN context, SPI 0x9EEC77C1
VPN handle: 0x001bc09c
IPSEC: Completed outbound inner rule, SPI 0x9EEC77C1
Rule ID: 0xccd5df38
IPSEC: Completed outbound outer SPD rule, SPI 0x9EEC77C1
Rule ID: 0xccddc338
IPSEC: New inbound tunnel flow rule, SPI 0x0320CCB2
Src addr: 10.0.4.0
Src mask: 255.255.255.0
Dst addr: 10.0.1.0
Dst mask: 255.255.255.0
Src ports
Upper: 0
Lower: 0
Op : ignore
Dst ports
Upper: 0
Lower: 0
Op : ignore
Protocol: 0
Use protocol: false
SPI: 0x00000000
Use SPI: false
IPSEC: Completed inbound tunnel flow rule, SPI 0x0320CCB2
Rule ID: 0xccd5e6d0
IPSEC: New inbound decrypt rule, SPI 0x0320CCB2
Src addr: 74.143.200.234
Src mask: 255.255.255.255
Dst addr: 69.135.82.90
Dst mask: 255.255.255.255
Src ports
Upper: 0
Lower: 0
Op : ignore
Dst ports
Upper: 0
Lower: 0
Op : ignore
Protocol: 50
Use protocol: true
SPI: 0x0320CCB2
Use SPI: true
IPSEC: Completed inbound decrypt rule, SPI 0x0320CCB2
Rule ID: 0xccbf87c8
IPSEC: New inbound permit rule, SPI 0x0320CCB2
Src addr: 74.143.200.234
Src mask: 255.255.255.255
Dst addr: 69.135.82.90
Dst mask: 255.255.255.255
Src ports
Upper: 0
Lower: 0
Op : ignore
Dst ports
Upper: 0
Lower: 0
Op : ignore
Protocol: 50
Use protocol: true
SPI: 0x0320CCB2
Use SPI: true
IPSEC: Completed inbound permit rule, SPI 0x0320CCB2
Rule ID: 0xcd061e48
IPSEC: Added SA to last received DB, SPI: 0x0320CCB2, user: 74.143.200.234, peer: 74.143.200.234, SessionID: 0x00020000
Syslogs
-------
2019-08-16 08:41:56 Local4.Warning 10.0.1.2 Aug 16 2019 08:41:56: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:10m:30s, Bytes xmt: 1331064, Bytes rcv: 2019519, Reason: Lost Service
2019-08-16 08:42:19 Local4.Warning 10.0.4.1 %ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:10m:53s, Bytes xmt: 2021599, Bytes rcv: 1331064, Reason: Lost Service
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-16-2019 10:45 AM
So after opening a ticket with Spectrum I have provided them with lots of these detailed examples. Here's one interesting tidbit. Not sure if it's indicative or not. I ran a traceroute between the endpoints while the tunnel was up, waited until it dropped, and then ran another traceroute immediately thereafter. The routes between these endpoints appear to differ before and after the tunnel dropped.
8/16/2019 @ 13:20
-----------------
traceroute 74.143.200.233 source outside
Type escape sequence to abort.
Tracing the route to 74.143.200.233
1 rrcs-69-135-82-89.central.biz.rr.com (69.135.82.89) 0 msec 0 msec 10 msec
2 69.23.11.1 0 msec 10 msec
69.23.11.3 10 msec
3 agg45.clmkohpe02r.midwest.rr.com (65.189.106.108) 10 msec
ae1.clmloh0602h.midwest.rr.com (65.189.106.10) 0 msec
agg45.clmkohpe02r.midwest.rr.com (65.189.106.108) 0 msec
4 ae1.wevlohoh02h.midwest.rr.com (65.29.17.67) 10 msec
65.29.17.65 10 msec
ae1.wevlohoh02h.midwest.rr.com (65.29.17.67) 10 msec
5 rrcs-74-143-200-233.central.biz.rr.com (74.143.200.233) 0 msec 0 msec 10 msec
---------------------------------------------------------------------------------
traceroute 69.135.82.89 source outside
Type escape sequence to abort.
Tracing the route to 69.135.82.89
1 rrcs-74-143-200-233.central.biz.rr.com (74.143.200.233) 0 msec 0 msec 0 msec
2 ae15.wevlohoh02h.midwest.rr.com (69.23.11.7) 0 msec
ae15.wevlohoh01h.midwest.rr.com (69.23.11.5) 0 msec
ae15.wevlohoh02h.midwest.rr.com (69.23.11.7) 0 msec
3 be88.clmcohib01r.midwest.rr.com (65.29.17.66) 10 msec
be88.clmkohpe02r.midwest.rr.com (65.29.17.64) 20 msec
be88.clmcohib01r.midwest.rr.com (65.29.17.66) 10 msec
4 65.189.106.11 10 msec
ae1.clmloh0601h.midwest.rr.com (65.189.106.109) 90 msec
65.189.106.11 0 msec
5 rrcs-69-135-82-89.central.biz.rr.com (69.135.82.89) 0 msec 10 msec 10 msec
===================================================================================
2019-08-16 13:22:06 Local4.Warning 10.0.1.2 Aug 16 2019 13:22:06: %ASA-4-113019: Group = 74.143.200.234, Username = 74.143.200.234, IP = 74.143.200.234, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:10m:37s, Bytes xmt: 1939674, Bytes rcv: 4640005, Reason: Lost Service
2019-08-16 13:22:16 Local4.Warning 10.0.4.1 %ASA-4-113019: Group = 69.135.82.90, Username = 69.135.82.90, IP = 69.135.82.90, Session disconnected. Session Type: LAN-to-LAN, Duration: 1h:10m:33s, Bytes xmt: 4640005, Bytes rcv: 1939674, Reason: IKE Delete
===================================================================================
8/16/2019 @ 13:23
-----------------
traceroute 74.143.200.233 source outside
Type escape sequence to abort.
Tracing the route to 74.143.200.233
1 rrcs-69-135-82-89.central.biz.rr.com (69.135.82.89) 0 msec 0 msec 0 msec
2 69.23.11.1 10 msec 0 msec 10 msec
3 ae1.clmloh0602h.midwest.rr.com (65.189.106.10) 10 msec 10 msec 10 msec
4 ae1.wevlohoh02h.midwest.rr.com (65.29.17.67) 10 msec 0 msec 10 msec
5 rrcs-74-143-200-233.central.biz.rr.com (74.143.200.233) 10 msec 0 msec 10 msec
------------------------------------------------------------------------------------
traceroute 69.135.82.89 source outside
Type escape sequence to abort.
Tracing the route to 69.135.82.89
1 rrcs-74-143-200-233.central.biz.rr.com (74.143.200.233) 0 msec 0 msec 0 msec
2 ae15.wevlohoh01h.midwest.rr.com (69.23.11.5) 0 msec 0 msec
ae15.wevlohoh02h.midwest.rr.com (69.23.11.7) 10 msec
3 be88.clmkohpe02r.midwest.rr.com (65.29.17.64) 0 msec 10 msec 10 msec
4 ae1.clmloh0601h.midwest.rr.com (65.189.106.109) 10 msec 10 msec
65.189.106.11 10 msec
5 rrcs-69-135-82-89.central.biz.rr.com (69.135.82.89) 0 msec 20 msec 0 msec
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-17-2019 07:10 AM - edited 08-17-2019 07:11 AM
Just a quick post-mortem on this. It was indeed a provider-side issue. Apparently Spectrum's internetworking had some incorrect references. As of this morning we should be in the clear. Per the engineer's notes on the ticket. Glad to know I wasn't crazy...in this instance at least... :)
Our engineers have cleared the LDP session to upstream router and the label mismatch has been resolved as of 5:25 am EDT.
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-17-2019 08:29 AM
Thanks for the update. Glad to know that it is confirmed as an issue in the ISP network. I am pleased that I was able to confirm your belief that it was an issue with the ISP. This discussion illustrates good steps in documenting and investigating a network issue. Thank you for marking this question as solved. This will help other participants in the community to identify discussions which have helpful information.
HTH
Rick
HTH
Rick
Rick
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-25-2019 08:32 AM - edited 10-25-2019 11:14 AM
Actually the provider still hasn't fixed the issue. It happens every 2-3 hours usually. Just between their two endpoints. The other site VPN interconnects are fine. On one end I am running Ping Plotter using another usable public IP. Logging pings back to the other provider endpoint. I do see a few instances of dead hops, that usually coincide when the site VPN interconnect quickly flaps.
More often the dead hops I see logged instances where latency is greater than 250 ms. This leads to be a question. I know you can configure the ASA's DPD so that keepalives are issued every X seconds and retry after X seconds if failed. But is there a hard-coded timeout in this mechanism? For example, let's say I have latency of 361 ms. Would the DPD keepalive mechanism give up after a smaller timeout value has been hit and assume the peer is dead?
Learn, share, save
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide
Customers Also Viewed These Support Documents
