Buffer leak from SSLVPN

LukaszTJB · ‎09-10-2014

Hello,

I’m running a Cisco 3845 with a AIM-VPN/SSL-3 Module. My WAN Interface (a DSL Connection) is configured with NAT overload. This is also the interface where the sslvpn connection is terminating. If a user connects from the internet to the SSLVPN in full tunnel mode and uses the internet a buffer leak occurs in the middle pool and after certain time the router crashes. The IOS which is running on the device is c3845-adventerprisek9-mz.151-4.M7.bin. I also tried out the latest Version c3845-adventerprisek9-mz.151-4.M8.bin but it doesn't make any differences.

sh buffers
Buffer elements:
674 in free list (500 max allowed)
3064635 hits, 0 misses, 617 created

Public buffer pools:
Small buffers, 104 bytes (total 183, permanent 150, peak 183 @ 01:03:44):
     176 in free list (50 min, 300 max allowed)
     959716 hits, 194 misses, 13 trims, 46 created
     0 failures (0 no memory)
Middle buffers, 600 bytes (total 823, permanent 400, peak 823 @ 00:45:03):
     458 in free list (400 min, 800 max allowed)
     91468 hits, 225 misses, 69 trims, 492 created
     0 failures (0 no memory)
Big buffers, 1536 bytes (total 741, permanent 500, peak 773 @ 01:03:46):
     685 in free list (500 min, 1000 max allowed)
     1333349 hits, 519 misses, 75 trims, 316 created
     0 failures (0 no memory)
VeryBig buffers, 4520 bytes (total 45, permanent 45, peak 48 @ 04:16:10):
     44 in free list (35 min, 65 max allowed)
     183629 hits, 1 misses, 3 trims, 3 created
     0 failures (0 no memory)
Large buffers, 5024 bytes (total 35, permanent 35, peak 36 @ 04:16:10):
     35 in free list (25 min, 65 max allowed)
     0 hits, 0 misses, 1 trims, 1 created
     0 failures (0 no memory)
Huge buffers, 18024 bytes (total 4, permanent 4, peak 5 @ 04:16:10):
     4 in free list (2 min, 8 max allowed)
     0 hits, 0 misses, 1 trims, 1 created
     0 failures (0 no memory)

The 'sh buffer' output above is quite fresh after a reload of the device. When users generate a lot of traffic to the internet from a Full tunnel Connection the used buffers in the middle pool rises until no IO memory is left on the device. I can check this with 'show memory statistic history' :

      555555555555555555555555555555555555555555555555555556666666
      999999999999999999999999999999999999999999999999999990000000
100
   90
   80
   70
   60 ############################################################
   50 ############################################################
   40 ############################################################
   30 ############################################################
   20 ############################################################
   10 ############################################################
     0....5....1....1....2....2....3....3....4....4....5....5....6
               0    5    0    5    0    5    0    5    0    5    0
               Free memory per minute (last 60 minutes)
              * = maximum # = average

About 50mins ago 1% of the IO memory become wasted due to a leak. This was the time where I tested it with a Anyconnect client from my mobile device and opened one website.

Does anyone have an Idea how to fix this? Appreciated any kind of help.

kind regards

Lukasz

LukaszTJB · ‎09-10-2014

It looks like even if someone tries to access resources from the LAN through a full tunnel a leak happens as well:

sh buffers
Buffer elements:
674 in free list (500 max allowed)
3359687 hits, 0 misses, 617 created

Public buffer pools:
Small buffers, 104 bytes (total 150, permanent 150, peak 183 @ 02:55:10):
     143 in free list (50 min, 300 max allowed)
     1051942 hits, 194 misses, 46 trims, 46 created
     0 failures (0 no memory)
Middle buffers, 600 bytes (total 21962, permanent 400, peak 21962 @ 00:04:45):
     407 in free list (400 min, 800 max allowed)
     141634 hits, 15220 misses, 123 trims, 21685 created
     0 failures (0 no memory)
Big buffers, 1536 bytes (total 582, permanent 500, peak 773 @ 02:55:13):
     521 in free list (500 min, 1000 max allowed)
     1377048 hits, 543 misses, 260 trims, 342 created
     0 failures (0 no memory)
VeryBig buffers, 4520 bytes (total 45, permanent 45, peak 48 @ 06:07:36):
     44 in free list (35 min, 65 max allowed)
     184351 hits, 1 misses, 3 trims, 3 created
     0 failures (0 no memory)
Large buffers, 5024 bytes (total 35, permanent 35, peak 36 @ 06:07:36):
     35 in free list (25 min, 65 max allowed)
     0 hits, 0 misses, 1 trims, 1 created
     0 failures (0 no memory)
Huge buffers, 18024 bytes (total 4, permanent 4, peak 5 @ 06:07:36):
     4 in free list (2 min, 8 max allowed)
     0 hits, 0 misses, 1 trims, 1 created
     0 failures (0 no memory)

IO Memory:

      1111111111111223334455566666666666666666666666666666655555555
      999999999999937159371590000000000000000000000000000009999999
100
   90
   80
   70
   60                                            *######################################
   50                      *########################################
   40                                  *###########################################
   30                              ##############################################
   20 ############################################################
   10 ############################################################
     0....5....1....1....2....2....3....3....4....4....5....5....6
               0    5    0    5    0    5    0    5    0    5    0
               Free memory per minute (last 60 minutes)
              * = maximum # = average

This happens after I downloaded a ~60MB file from a CIFS share through a full tunnel.

LukaszTJB · ‎09-10-2014

I continued using VPN and downloaded more files from the CIFS server until all IO Memory was exhausted. The last messages from the router where theses:

186>1 2014-09-10T18:02:07.182019+02:00 hostname 190 - - - Sep 10 16:02:06.166: %SYS-2-MALLOCFAIL: Memory allocation of 780 bytes failed from 0x606625D0, alignment 32
<186>1 2014-09-10T18:02:07.182019+02:00 hostname 191 - - - Pool: I/O Free: 16544 Cause: Memory fragmentation
<186>1 2014-09-10T18:02:07.182074+02:00 hostname 192 - - - Alternate Pool: None Free: 0 Cause: No Alternate pool
<186>1 2014-09-10T18:02:07.182074+02:00 hostname 193 - - - -Process= "Pool Manager", ipl= 0, pid= 7
<186>1 2014-09-10T18:02:07.182089+02:00 hostname 194 - - - -Traceback= 63F40FD8z 6065CD98z 63F5DEECz 63F5E268z 63F0D888z 63F0D86Cz
<186>1 2014-09-10T18:02:37.187832+02:00 hostname 195 - - - Sep 10 16:02:36.183: %SYS-2-MALLOCFAIL: Memory allocation of 780 bytes failed from 0x606625D0, alignment 32
<186>1 2014-09-10T18:02:37.187832+02:00 hostname 196 - - - Pool: I/O Free: 16544 Cause: Memory fragmentation
<186>1 2014-09-10T18:02:37.187924+02:00 hostname 197 - - - Alternate Pool: None Free: 0 Cause: No Alternate pool
<186>1 2014-09-10T18:02:37.187924+02:00 hostname 198 - - - -Process= "Pool Manager", ipl= 0, pid= 7
<186>1 2014-09-10T18:02:37.188421+02:00 hostname 199 - - - -Traceback= 63F40FD8z 6065CD98z 63F5DEECz 63F5E178z 63F0D888z 63F0D86Cz
<185>1 2014-09-10T18:02:43.119863+02:00 hostname 200 - - - Sep 10 16:02:43.111: %VPN_HW-1-PACKET_ERROR: slot: 0 Packet Encryption/Decryption error, UNKNOWN, handle=0x100DC

After that the ssh session got terminated and Internet access was barley possible. I had to restart the device.

LukaszTJB · ‎09-11-2014

Ok, it looks like sslvpn code is broken in IOS 15.1(4)M - see CSCug17485. However, the memory leak only occurs if clients connect from the internet to the NAT-Enabled WAN interface and generated bulk traffic. If I connect from the inside of the network to the webvpn Server (same Interface) and generate traffic everything is fine.

IOS 15.0 hasn't this issue but there I can't get windows7 clients to authenticate probably using the anyconnect client. First I got the error message "anyconnect cannot confirm it is connected to your secure gateway". After some research I tried to import the Certificate by hand and the next error message i got is "anyconnect connection attempt has failed due to network or pc issue". I deleted all tempfiles associated with anyconnect, checked firewall and antivirus, reinstalled the client and tried different version but no success. Anyway a connection from a android mobile device with the latest anyconnect client works fine.

Anybody an idea how to get this up & running on IOS 15.0-1.M10

kind regards

Lukasz