Anyconnect - Internal server error

[richard.priest]

Hi,

can anyone advise on how best to resolve this issue? I've setup Anyconnect and it's working fine for a number of users, however recently any new clients connecting are receiving the error "VPN Server Internal error".

The users can authenticate to the webpage fine and download the client, but once installed and post authentication the error appears are they're unable to connect. Apologies, this is incorrect, the client returns the VPN server error as soon as you hit connect. Currently it's setup to authenticate via SAML, but changing to local or TACACs, I get the exact same error when testing.

We only have a very small number of users, (6) and there are 8 licenses on the device (5508-x in HA failover and 4 licenses per unit, so 8 total available).

However even after testing witn no-one else connected the same error occurs so I don't think it's a license issue?

I've tried a number of different anyconnect versions, from 

4.7.04056

4.8.03036, and 

4.9.00086

All have the same error.

[richard.priest]

Update,

if I take all of the files from ProgramData and appdata and copy to a new machine before installing the AnyConnect client I can then connect.

So it's a client config issue somewhere?

[Josue Brenes]

Hi Richard,

Can you enable the “debug webvpn AnyConnect 255”, try to connect and share the result of that output?

Also, is the connection using an xml profile? If yes, can you compare the profile on a working and non-working machine and look for any possible difference between them?

Lastly, a DART from the failed connection would be useful.

Rate if it helps.

Regards,

Josue Brenes

TAC - VPN Engineer.

[richard.priest]

Hi,

When I enable the anyconnect debug, I get absolutely zero output on the terminal and on syslogs. (I have terminal monitor turned on).

All I see on syslogs is the initial connection and the it being terminated with seemingly no reason

I did start playing about with XML profiles, but thought it may be the cause for some reason so removed them all and any references with the config.

What's a DART and how do I get one?

[richard.priest]

Found the DART! bundle attached

Cheers

[Josue Brenes]

Richard,

In this particular case, the DART is not showing any relevant information about the failed connection besides the error:

The following error message was received from the secure gateway:
VPN Server internal error.

Would you be able to share the ASA's config over here?

At least the output of the commands: 

sh run webvpn

sh run ssl

sh version

sh run tunnel-group <name_of_tunnel_group>

sh run group-policy <name_of_group_policy>

Rate if it helps.

Regards,

Josue Brenes

TAC - VPN Engineer.

[richard.priest]

Thanks Josue, really appreicate the assistance,

outputs are:

Show Webvpn

webvpn
enable OUTSIDE
hsts
enable
max-age 31536000
include-sub-domains
no preload
anyconnect image disk0:/anyconnect-win-4.8.03036-webdeploy-k9.pkg 2
anyconnect enable
saml idp https://sts.windows.net/4fb68470-0753-42ef-98d9-b1d034772ace/
url sign-in https://login.microsoftonline.com/4fb68470-0753-42ef-98d9-b1d034772ace/saml2
url sign-out https://login.microsoftonline.com/common/wsfederationwa=wsignout1.0
base-url https://xxxxxxxx.xxxxxx.xx
trustpoint idp AzureAD-AC-SAML
trustpoint sp ASDM_TrustPoint0
no signature
no force re-authentication
tunnel-group-list enable
cache
disable
error-recovery disable

Sho run SSL

ssl trust-point ASDM_TrustPoint0 OUTSIDE
ssl trust-point ASDM_Launcher_Access_TrustPoint_0 NOC-LAN-V950
ssl trust-point ASDM_Launcher_Access_TrustPoint_0 NOC-LAN-V950 vpnlb-ip

Sho ver

Cisco Adaptive Security Appliance Software Version 9.8(4)22
Firepower Extensible Operating System Version 2.2(2.124)
Device Manager Version 7.13(1)

Compiled on Fri 29-May-20 00:37 PDT by builders
System image file is "disk0:/asa984-22-lfbff-k8.SPA"
Config file at boot was "startup-config"

AIM-NOC-FW01 up 1 day 19 hours
failover cluster up 160 days 22 hours

Hardware: ASA5508, 8192 MB RAM, CPU Atom C2000 series 2000 MHz, 1 CPU (8 cores)
Internal ATA Compact Flash, 8000MB
BIOS Flash M25P64 @ 0xfed01000, 16384KB

Encryption hardware device : Cisco ASA Crypto on-board accelerator (revision 0x1)
Number of accelerators: 1

1: Ext: GigabitEthernet1/1 : address is ecbd.1d0b.788b, irq 255
2: Ext: GigabitEthernet1/2 : address is ecbd.1d0b.788c, irq 255
3: Ext: GigabitEthernet1/3 : address is ecbd.1d0b.788d, irq 255
4: Ext: GigabitEthernet1/4 : address is ecbd.1d0b.788e, irq 255
5: Ext: GigabitEthernet1/5 : address is ecbd.1d0b.788f, irq 255
6: Ext: GigabitEthernet1/6 : address is ecbd.1d0b.7890, irq 255
7: Ext: GigabitEthernet1/7 : address is ecbd.1d0b.7891, irq 255
8: Ext: GigabitEthernet1/8 : address is ecbd.1d0b.7892, irq 255
9: Int: Internal-Data1/1 : address is ecbd.1d0b.788a, irq 255
10: Int: Internal-Data1/2 : address is 0000.0001.0002, irq 0
11: Int: Internal-Control1/1 : address is 0000.0001.0001, irq 0
12: Int: Internal-Data1/3 : address is 0000.0001.0003, irq 0
13: Ext: Management1/1 : address is ecbd.1d0b.788a, irq 0
14: Int: Internal-Data1/4 : address is 0000.0100.0001, irq 0

Licensed features for this platform:
Maximum Physical Interfaces : Unlimited perpetual
Maximum VLANs : 50 perpetual
Inside Hosts : Unlimited perpetual
Failover : Active/Active perpetual
Encryption-DES : Enabled perpetual
Encryption-3DES-AES : Enabled perpetual
Security Contexts : 2 perpetual
Carrier : Disabled perpetual
AnyConnect Premium Peers : 4 perpetual
AnyConnect Essentials : Disabled perpetual
Other VPN Peers : 100 perpetual
Total VPN Peers : 100 perpetual
AnyConnect for Mobile : Disabled perpetual
AnyConnect for Cisco VPN Phone : Disabled perpetual
Advanced Endpoint Assessment : Disabled perpetual
Shared License : Disabled perpetual
Total TLS Proxy Sessions : 320 perpetual
Botnet Traffic Filter : Disabled perpetual
Cluster : Disabled perpetual
VPN Load Balancing : Enabled perpetual

Failover cluster licensed features for this platform:
Maximum Physical Interfaces : Unlimited perpetual
Maximum VLANs : 50 perpetual
Inside Hosts : Unlimited perpetual
Failover : Active/Active perpetual
Encryption-DES : Enabled perpetual
Encryption-3DES-AES : Enabled perpetual
Security Contexts : 4 perpetual
Carrier : Disabled perpetual
AnyConnect Premium Peers : 8 perpetual
AnyConnect Essentials : Disabled perpetual
Other VPN Peers : 100 perpetual
Total VPN Peers : 100 perpetual
AnyConnect for Mobile : Disabled perpetual
AnyConnect for Cisco VPN Phone : Disabled perpetual
Advanced Endpoint Assessment : Disabled perpetual
Shared License : Disabled perpetual
Total TLS Proxy Sessions : 320 perpetual
Botnet Traffic Filter : Disabled perpetual
Cluster : Disabled perpetual
VPN Load Balancing : Enabled perpetual

The Running Activation Key feature: 640 TLS Proxy sessions exceed the limit on the platform, reduced to 320 TLS Proxy sessions.

Serial Number: JAD193301UQ
Running Permanent Activation Key: 0x3d1af26d 0xf8c33571 0xccb2b11c 0xa428208c 0xc12b1ab5
Configuration register is 0x1
Image type : Release
Key Version : A
Configuration last modified by admin at 20:27:07.498 GMT/BDT Sat Jul 4 2020

sho run tunnel-group

tunnel-group NOC-SSL-VPN-GroupPolicy type remote-access
tunnel-group NOC-SSL-VPN-GroupPolicy general-attributes
authentication-server-group TACACS+
default-group-policy NOC-SSL-VPN-GroupPolicy
tunnel-group NOC-SSL-VPN-GroupPolicy webvpn-attributes
authentication saml
group-alias NOC-SSL-VPN enable
saml identity-provider https://sts.windows.net/4fb68470-0753-42ef-98d9-b1d034772ace/

Sho run group-policy

group-policy NOC-SSL-VPN-GroupPolicy internal
group-policy NOC-SSL-VPN-GroupPolicy attributes
vpn-tunnel-protocol ssl-client
split-tunnel-policy tunnelspecified
ipv6-split-tunnel-policy tunnelspecified
split-tunnel-network-list value NOC-Network-Split-Tunnel
default-domain value aimes.net
address-pools value NOC-SSL-VPN-POOL
ipv6-address-pools value NOC-SSL-VPN-POOL-IPV6
webvpn
anyconnect keep-installer none

The sho run SSL was interesting, wh are there two trustpoints listed and what's that 'vpnlb-ip' command do?

[Josue Brenes · Accepted solution]

Hi Richard, 

Based on the show tech you shared over private message, your ASA was missing the SAML identity-provider, which was causing the error when connecting.

Rate if it helps.

Regards,

Josue Brenes

TAC - VPN Engineer.

[richard.priest]

Yeah many many thanks to Josue, he spotted that I had multiple tunnel groups, one was fully utilising SAML for authentication, but another tunnel group was only partially configured, this was causing the Anyconnect to throw this error.

I didn't realise that you could add to the URL i.e. /test which was really helpful to learn!