Slow Memory Leak and/or Denial of Service with AnyConnect on 3845

[Hans Johnson]

Hi Everyone,

I'm having an issue with the WebVPN/AnyConnect setup on our edge 3845. Over the course of a week or so, the 3845 eats all its RAM, with it all going to the SSLVPN_PROCESS. As a side note, I can see the tcp connection count going up continuously over that time, as though connections are never being closed. The 3845 is equipped with an AIM-VPN/SSL-3. Other than the apparent memory leak, it works well.

Any thoughts would be greatly appreciated, it's getting annoying having to reload the router every several days.

The relevant parts of my configuration are as follows:

webvpn gateway Our-VPN
 hostname vpn.ourvpn.org
 ip address 172.16.0.1 port 443
 ssl encryption 3des-sha1 aes-sha1
 ssl trustpoint holdenvillage.org
 inservice
!
webvpn install svc flash:/webvpn/anyconnect-win-4.0.00061-k9.pkg sequence 1
!
webvpn install svc flash:/webvpn/anyconnect-macosx-i386-3.1.05187-k9.pkg sequence 2
!
webvpn context Holden-VPN
 ssl authenticate verify all
 !
 !
 policy group PG-SSL
   functions svc-enabled
   filter tunnel VPN-ACL
   svc address-pool "VPN-POOL" netmask 255.255.255.0
   svc keep-client-installed
   svc rekey method new-tunnel
   svc split dns "holdenvillage.org"
   svc split include 10.2.0.0 255.255.0.0
   svc split include 10.1.0.0 255.255.0.0
   svc split include 172.17.0.0 255.255.0.0
   svc split include 192.168.2.0 255.255.255.0
   svc dns-server primary 172.17.15.2
   svc dns-server secondary 10.2.0.3
 virtual-template 2
 default-group-policy PG-SSL
 aaa authentication list VPN-LOGIN
 gateway Our-VPN
 inservice

[Karsten Iwen]

When looking at your installed AnyConnect versions, I would assume that your IOS could also be quite ancient. All kind of memory leaks are typically bugs in IOS which are often resolved in newer releases. Before investigating that issue, I would first try to update the IOS on the router.

[Hans Johnson]

Router is running 15.1M9, the second to last IOS available for the unit. Previously, I was running it on a 3825 running 15.1M10 (the last version available) with the same result.

If it is an honest to god memory leak and/or denial of service, then I need to figure something else out, but if there is a configuration workaround to make things run, I'm good.

For better or worse, this router belongs to a non-profit organization that doesn't have the budget to go out and buy a more modern ASA or similar, so we generally stick one or two hardware generations behind.

[Karsten Iwen]

Ok, then that's the wrong direction ...

The shown config is really quite basic and I wouldn't see anything problematic there.

You probably don't have a SmartNet? Then perhaps there is a workaround to get out of the situation. What do you do when you see that the memory is completely used? Have you tried what happens when you take the VPN-gateway out of service and reenable it again?

[Hans Johnson]

Yeah, no SmartNet. Just for gits and shiggles I updated the Anyconnect client to a recent 4.3 build, and same thing.

When the memory is used up, the router basically goes OOM and starts having issues, so I reload it via a console server we have (SSH process usually dies, but I can get in over the console port).

There is no change when I disable the webvpn gateway and context. In that situation, I still see the SSLVPN_PROCESS occupying significant RAM. When I re-enable it, the same just keeps going up from there.

The one thing I have noticed is that if I do a "show tcp brief numeric" I see all sorts of Established connections to port 443. I actually changed IP addresses of the VPN interface yesterday, and I'm still seeing established connections for an IP that is no longer connected with the router. I do have "service tcp-keepalives-in" running as well.

[Karsten Iwen]

Do you have a restrictive incoming ACL on the outside interface? This ACL should only allow the traffic that is really needed (like access to the VPN-gateway).