5545X ASA needs to be rebooted approximately every 8 days since company moved to mobile work platform

[jerryburrisiii]

Hey Friends,

I am at a complete loss. I have no idea where to go from here.

Since my company has moved to a remote work platform as most others have, we are experiencing extremely high latency on our outside interface every 8 days or so. We have approximately 300 employees that work from home via AnyConnect full tunnel VPN. This incident happened again this morning. I am wondering if anyone can look at this and be able to tell me what I need to change. I pulled this information after the reboot, not during the outage. Please, tell me if you need more information. Any help is greatly appreciated.

ASA-5545x/pri/act# sho ver

Cisco Adaptive Security Appliance Software Version 9.8(3)18
Firepower Extensible Operating System Version 2.2(2.111)
Device Manager Version 7.10(1)

Compiled on Wed 12-Dec-18 17:03 PST by builders
System image file is "disk0:/asa983-18-smp-k8.bin"
Config file at boot was "startup-config"

EDF-ASA-5545x up 6 hours 59 mins
failover cluster up 3 years 311 days

Hardware: ASA5545, 12288 MB RAM, CPU Lynnfield 2660 MHz, 1 CPU (8 cores)
ASA: 6455 MB RAM, 1 CPU (1 core)
Internal ATA Compact Flash, 8192MB
BIOS Flash MX25L6445E @ 0xffbb0000, 8192KB

Encryption hardware device : Cisco ASA Crypto on-board accelerator (revision 0x1)
Boot microcode : CNPx-MC-BOOT-2.00
SSL/IKE microcode : CNPx-MC-SSL-SB-PLUS-0005
IPSec microcode : CNPx-MC-IPSEC-MAIN-0026
Number of accelerators: 1
Baseboard Management Controller (revision 0x1) Firmware Version: 2.4

0: Int: Internal-Data0/0 : address is 003a.7dcf.b4a0, irq 11
1: Ext: GigabitEthernet0/0 : address is 003a.7dcf.b4a5, irq 5
2: Ext: GigabitEthernet0/1 : address is 003a.7dcf.b4a1, irq 5
3: Ext: GigabitEthernet0/2 : address is 003a.7dcf.b4a6, irq 10
4: Ext: GigabitEthernet0/3 : address is 003a.7dcf.b4a2, irq 10
5: Ext: GigabitEthernet0/4 : address is 003a.7dcf.b4a7, irq 5
6: Ext: GigabitEthernet0/5 : address is 003a.7dcf.b4a3, irq 5
7: Ext: GigabitEthernet0/6 : address is 003a.7dcf.b4a8, irq 10
8: Ext: GigabitEthernet0/7 : address is 003a.7dcf.b4a4, irq 10
9: Int: Internal-Data0/1 : address is 0000.0001.0002, irq 0
10: Int: Internal-Control0/0 : address is 0000.0001.0001, irq 0
11: Int: Internal-Data0/2 : address is 0000.0001.0003, irq 0
12: Ext: Management0/0 : address is 003a.7dcf.b4a0, irq 0
13: Int: Internal-Data0/3 : address is 0000.0100.0001, irq 0

Licensed features for this platform:
Maximum Physical Interfaces : Unlimited perpetual
Maximum VLANs : 300 perpetual
Inside Hosts : Unlimited perpetual
Failover : Active/Active perpetual
Encryption-DES : Enabled perpetual
Encryption-3DES-AES : Enabled perpetual
Security Contexts : 2 perpetual
Carrier : Disabled perpetual
AnyConnect Premium Peers : 2500 perpetual
AnyConnect Essentials : Disabled perpetual
Other VPN Peers : 2500 perpetual
Total VPN Peers : 2500 perpetual
AnyConnect for Mobile : Enabled perpetual
AnyConnect for Cisco VPN Phone : Enabled perpetual
Advanced Endpoint Assessment : Enabled perpetual
Shared License : Disabled perpetual
Total TLS Proxy Sessions : 2 perpetual
Botnet Traffic Filter : Disabled perpetual
IPS Module : Disabled perpetual
Cluster : Enabled perpetual
Cluster Members : 2 perpetual

This platform has an ASA5545 VPN Premium license.

Failover cluster licensed features for this platform:
Maximum Physical Interfaces : Unlimited perpetual
Maximum VLANs : 300 perpetual
Inside Hosts : Unlimited perpetual
Failover : Active/Active perpetual
Encryption-DES : Enabled perpetual
Encryption-3DES-AES : Enabled perpetual
Security Contexts : 4 perpetual
Carrier : Disabled perpetual
AnyConnect Premium Peers : 2500 perpetual
AnyConnect Essentials : Disabled perpetual
Other VPN Peers : 2500 perpetual
Total VPN Peers : 2500 perpetual
AnyConnect for Mobile : Enabled perpetual
AnyConnect for Cisco VPN Phone : Enabled perpetual
Advanced Endpoint Assessment : Enabled perpetual
Shared License : Disabled perpetual
Total TLS Proxy Sessions : 4 perpetual
Botnet Traffic Filter : Disabled perpetual
IPS Module : Disabled perpetual
Cluster : Enabled perpetual

This platform has an ASA5545 VPN Premium license.

The Running Activation Key feature: 5000 AnyConnect Premium sessions exceed the limit on the platform, reduced to 2500 AnyConnect Premium sessions.

Serial Number: FCH20117RKH
Running Permanent Activation Key: 0x910fc87f 0xb81ec108 0x1df20dbc 0xc0a82c1c 0x0136f292
Configuration register is 0x1

Image type : Release
Key version : A

ASA-5545x/pri/act# sho vpn-sessiondb
---------------------------------------------------------------------------
VPN Session Summary
---------------------------------------------------------------------------
Active : Cumulative : Peak Concur : Inactive
----------------------------------------------
AnyConnect Client : 249 : 386 : 249 : 0
SSL/TLS/DTLS : 249 : 386 : 249 : 0
Site-to-Site VPN : 15 : 172 : 15
IKEv2 IPsec : 6 : 159 : 6
IKEv1 IPsec : 9 : 13 : 9
---------------------------------------------------------------------------
Total Active and Inactive : 264 Total Cumulative : 558
Device Total VPN Capacity : 2500
Device Load : 11%
---------------------------------------------------------------------------

---------------------------------------------------------------------------
Tunnels Summary
---------------------------------------------------------------------------
Active : Cumulative : Peak Concurrent
----------------------------------------------
IKEv1 : 9 : 13 : 9
IKEv2 : 6 : 159 : 6
IPsec : 51 : 5871 : 59
AnyConnect-Parent : 249 : 386 : 249
SSL-Tunnel : 244 : 695 : 245
DTLS-Tunnel : 240 : 611 : 240
---------------------------------------------------------------------------
Totals : 799 : 7735
---------------------------------------------------------------------------

ASA-5545x/pri/act# sho vpn-sessiondb anyconnect

Session Type: AnyConnect

Username : XXXXXXX Index : 45
Assigned IP : 10.15.21.114 Public IP : XX.108.47.216
Protocol : AnyConnect-Parent SSL-Tunnel DTLS-Tunnel
License : AnyConnect Premium
Encryption : AnyConnect-Parent: (1)none SSL-Tunnel: (1)AES-GCM-256 DTLS-Tunnel: (1)AES128
Hashing : AnyConnect-Parent: (1)none SSL-Tunnel: (1)SHA384 DTLS-Tunnel: (1)SHA1
Bytes Tx : 42097479 Bytes Rx : 8435178
Group Policy : Always_On_VPN Tunnel Group : DefaultWEBVPNGroup
Login Time : 13:59:49 EDT Wed Apr 15 2020
Duration : 0h:02m:39s
Inactivity : 0h:00m:00s
VLAN Mapping : N/A VLAN : none
Audt Sess ID : 0a07000a0002d0005e974b95
Security Grp : none

ASA-5545x/pri/act# sho vpn-sessiondb detail anyconnect

Session Type: AnyConnect Detailed

Username : XXXXXXXXXX Index : 45
Assigned IP : 10.15.21.114 Public IP : XX.108.47.216
Protocol : AnyConnect-Parent SSL-Tunnel DTLS-Tunnel
License : AnyConnect Premium
Encryption : AnyConnect-Parent: (1)none SSL-Tunnel: (1)AES-GCM-256 DTLS-Tunnel: (1)AES128
Hashing : AnyConnect-Parent: (1)none SSL-Tunnel: (1)SHA384 DTLS-Tunnel: (1)SHA1
Bytes Tx : 45409592 Bytes Rx : 11496912
Pkts Tx : 48230 Pkts Rx : 39284
Pkts Tx Drop : 316 Pkts Rx Drop : 0
Group Policy : Always_On_VPN Tunnel Group : DefaultWEBVPNGroup
Login Time : 13:59:49 EDT Wed Apr 15 2020
Duration : 0h:08m:19s
Inactivity : 0h:00m:00s
VLAN Mapping : N/A VLAN : none
Audt Sess ID : 0a07000a0002d0005e974b95
Security Grp : none

AnyConnect-Parent Tunnels: 1
SSL-Tunnel Tunnels: 1
DTLS-Tunnel Tunnels: 1

AnyConnect-Parent:
Tunnel ID : 45.1
Public IP : XX.108.47.216
Encryption : none Hashing : none
TCP Src Port : 18555 TCP Dst Port : 443
Auth Mode : Certificate
Idle Time Out: 60 Minutes Idle TO Left : 51 Minutes
Client OS : win
Client OS Ver: 10.0.18363
Client Type : AnyConnect
Client Ver : Cisco AnyConnect VPN Agent for Windows 4.8.02042
Bytes Tx : 8052 Bytes Rx : 0
Pkts Tx : 6 Pkts Rx : 0
Pkts Tx Drop : 0 Pkts Rx Drop : 0

SSL-Tunnel:
Tunnel ID : 45.2
Assigned IP : 10.15.21.114 Public IP : XX.108.47.216
Encryption : AES-GCM-256 Hashing : SHA384
Ciphersuite : ECDHE-RSA-AES256-GCM-SHA384
Encapsulation: TLSv1.2 TCP Src Port : 18559
TCP Dst Port : 443 Auth Mode : Certificate
Idle Time Out: 60 Minutes Idle TO Left : 51 Minutes
Client OS : Windows
Client Type : SSL VPN Client
Client Ver : Cisco AnyConnect VPN Agent for Windows 4.8.02042
Bytes Tx : 8052 Bytes Rx : 1478
Pkts Tx : 6 Pkts Rx : 18
Pkts Tx Drop : 0 Pkts Rx Drop : 0

DTLS-Tunnel:
Tunnel ID : 45.3
Assigned IP : 10.15.21.114 Public IP : XX.108.47.216
Encryption : AES128 Hashing : SHA1
Ciphersuite : AES128-SHA
Encapsulation: DTLSv1.0 UDP Src Port : 65511
UDP Dst Port : 443 Auth Mode : Certificate
Idle Time Out: 60 Minutes Idle TO Left : 60 Minutes
Client OS : Windows
Client Type : DTLS VPN Client
Client Ver : Cisco AnyConnect VPN Agent for Windows 4.8.02042
Bytes Tx : 45397115 Bytes Rx : 11508006
Pkts Tx : 48246 Pkts Rx : 39302
Pkts Tx Drop : 316 Pkts Rx Drop : 0

ASA-5545x/pri/act# sho processes cpu-usage non-zero
Hardware: ASA5545
Cisco Adaptive Security Appliance Software Version 9.8(3)18
ASLR enabled, text region 7fdd565b4000-7fdd5a8cef5c
PC Thread 5Sec 1Min 5Min Process
0x00007fdd58c9b565 0x00002aaac6cafb80 0.2% 0.0% 0.0% vpnfol_thread_timer
0x00007fdd58a5eadb 0x00002aaac6cb1140 0.0% 0.3% 0.5% snmp
0x00007fdd589d8fcc 0x00002aaac6ccefc0 0.3% 0.4% 0.4% Logger
0x00007fdd573be9f9 0x00002aaac6cbe720 0.2% 0.2% 0.2% CP Processing
0x00007fdd57877382 0x00002aaac6cb9760 0.6% 0.5% 0.5% ARP Thread
- - 53.8% 55.4% 53.3% DATAPATH-0-2356

ASA-5545x/pri/act# sho memory
Free memory: 5276001353 bytes (78%)
Used memory: 1492617184 bytes (22%)
------------- ------------------
Total memory: 6768618537 bytes (100%)