Strange ASA 5515 memory usage

[Michael Muenz]

Hey guys,

enclosed a picture of Cacti monitoring one of my ASA's (p1). The memory usage is growing rapidly, not sure if since enabling OSPF or since ugprade to ASA 9.5.2.2, but the funny part is, after hitting the limit of 4GB, now picture p2 shows me 8GB. 

Is this normal behavior?

Thanks!

[Aditya Ganjoo · Accepted solution]

Hi, Could you share the output of show memory and show version of the affected ASA ?

Just check if your conditions are matching this bug:

http://cdets.cisco.com/apps/dumpcr?&content=summary&format=html&identifier=CSCux15273

Regards,

Aditya

[Michael Muenz]

Thanks for you fast answer, here the output:

Customer-ASA1-DOM# sh ver

Cisco Adaptive Security Appliance Software Version 9.5(2)2
Device Manager Version 7.5(2)153

Compiled on Tue 22-Dec-15 10:06 PST by builders
System image file is "disk0:/asa952-2-smp-k8.bin"
Config file at boot was "startup-config"

Customer-ASA1-DOM up 13 days 14 hours
failover cluster up 126 days 0 hours

Hardware: ASA5515, 8192 MB RAM, CPU Clarkdale 3058 MHz, 1 CPU (4 cores)
ASA: 3598 MB RAM, 1 CPU (1 core)
Internal ATA Compact Flash, 8192MB
BIOS Flash MX25L6445E @ 0xffbb0000, 8192KB

Encryption hardware device : Cisco ASA Crypto on-board accelerator (revision 0x1)
Boot microcode : CNPx-MC-BOOT-2.00
SSL/IKE microcode : CNPx-MC-SSL-SB-PLUS-0005
IPSec microcode : CNPx-MC-IPSEC-MAIN-0026
Number of accelerators: 1
Baseboard Management Controller (revision 0x1) Firmware Version: 2.4

0: Int: Internal-Data0/0 : address is 188b.9d72.3433, irq 11
1: Ext: GigabitEthernet0/0 : address is 188b.9d72.3437, irq 10
2: Ext: GigabitEthernet0/1 : address is 188b.9d72.3434, irq 10
3: Ext: GigabitEthernet0/2 : address is 188b.9d72.3438, irq 5
4: Ext: GigabitEthernet0/3 : address is 188b.9d72.3435, irq 5
5: Ext: GigabitEthernet0/4 : address is 188b.9d72.3439, irq 10
6: Ext: GigabitEthernet0/5 : address is 188b.9d72.3436, irq 10
7: Int: Internal-Data0/1 : address is 0000.0001.0002, irq 0
8: Int: Internal-Control0/0 : address is 0000.0001.0001, irq 0
9: Int: Internal-Data0/2 : address is 0000.0001.0003, irq 0
10: Ext: Management0/0 : address is 188b.9d72.3433, irq 0

Licensed features for this platform:
Maximum Physical Interfaces : Unlimited perpetual
Maximum VLANs : 100 perpetual
Inside Hosts : Unlimited perpetual
Failover : Active/Active perpetual
Encryption-DES : Enabled perpetual
Encryption-3DES-AES : Enabled perpetual
Security Contexts : 2 perpetual
Carrier : Disabled perpetual
AnyConnect Premium Peers : 2 perpetual
AnyConnect Essentials : Disabled perpetual
Other VPN Peers : 250 perpetual
Total VPN Peers : 250 perpetual
AnyConnect for Mobile : Disabled perpetual
AnyConnect for Cisco VPN Phone : Disabled perpetual
Advanced Endpoint Assessment : Disabled perpetual
Shared License : Disabled perpetual
Total UC Proxy Sessions : 2 perpetual
Botnet Traffic Filter : Disabled perpetual
IPS Module : Disabled perpetual
Cluster : Enabled perpetual
Cluster Members : 2 perpetual

This platform has an ASA 5515 Security Plus license.

Failover cluster licensed features for this platform:
Maximum Physical Interfaces : Unlimited perpetual
Maximum VLANs : 100 perpetual
Inside Hosts : Unlimited perpetual
Failover : Active/Active perpetual
Encryption-DES : Enabled perpetual
Encryption-3DES-AES : Enabled perpetual
Security Contexts : 4 perpetual
Carrier : Disabled perpetual
AnyConnect Premium Peers : 4 perpetual
AnyConnect Essentials : Disabled perpetual
Other VPN Peers : 250 perpetual
Total VPN Peers : 250 perpetual
AnyConnect for Mobile : Disabled perpetual
AnyConnect for Cisco VPN Phone : Disabled perpetual
Advanced Endpoint Assessment : Disabled perpetual
Shared License : Disabled perpetual
Total UC Proxy Sessions : 4 perpetual
Botnet Traffic Filter : Disabled perpetual
IPS Module : Disabled perpetual
Cluster : Enabled perpetual

This platform has an ASA 5515 Security Plus license.

Serial Number: X
Running Permanent Activation Key: X
Configuration register is 0x1

Image type : Release
Key version : A

Configuration last modified by enable_15 at 11:59:52.028 CEST Wed Feb 24 2016

Customer-ASA1-DOM# sh mem
Free memory: 175129907 bytes ( 5%)
Used memory: 3819765072 bytes (95%)
------------- ------------------
Total memory: 3772506931 bytes (100%)

[Aditya Ganjoo]

Hi,

Could you also send the output of show mem detail ?

Aditya

[Michael Muenz]

Sure:

Customer-ASA1-DOM# sh mem detail
Free memory heap: 222393168 bytes ( 6%)
Free memory system: 18446744073660725331 bytes (488978400%)
Used memory:
Allocated memory in use: 3217353440 bytes (85%)
Reserved memory (DMA): 150994944 bytes ( 4%)
Memory overhead: 0 bytes ( 0%)
----------------------------- ------------------
Total memory: 3772506931 bytes (100%)

Least free memory: 404155155 bytes (11%)
Most used memory: 3368351776 bytes (89%)
MEMPOOL_HEAPCACHE_0 POOL STATS:

Non-mmapped bytes allocated = 452984832
Number of free chunks = 995
Number of mmapped regions = 0
Mmapped bytes allocated = 0
Max memory footprint = 452984832
Keepcost = 222090096
Max contiguous free mem = 222090096
Allocated memory in use = 230591664
Free memory = 222393168

----- fragmented memory statistics -----

fragment size count total
(bytes) (bytes)
---------------- ---------- --------------
32 295 9440
48 249 11952
64 244 15616
80 14 1120
96 1 96**
96 14 1344
112 8 896
128 17 2176
144 4 576
160 7 1120
176 1 176
192 4 768
208 8 1664
224 1 224
240 5 1200
256 33 9792
384 33 14528
512 14 7664
1024 8 11520
1536 6 9920
2048 10 24048
3072 6 22048
4096 3 13904
6144 5 34816
8192 2 16640
16384 2 44432
32768 1 45392
222090096 1 222090096*

* - top most releasable chunk.
** - contiguous memory on top of heap.

----- allocated memory statistics -----

fragment size count total
(bytes) (bytes)
---------------- ---------- --------------
80 1697 135760
96 13262 1273152
112 3775 422800
128 1943 248704
144 3695 532080
160 489 78240
176 359 63184
192 438 84096
208 439 91312
224 330 73920
240 243 58320
256 3016 772096
384 1144 439296
512 1212 620544
768 554 425472
1024 1045 1070080
1536 258 396288
2048 287 587776
3072 75 230400
4096 392 1605632
6144 55 337920
8192 188 1540096
12288 120 1474560
16384 588 9633792
24576 27 663552
32768 62 2031616
49152 28 1376256
65536 229 15007744
98304 18 1769472
131072 20 2621440
196608 19 3735552
262144 13 3407872
393216 16 6291456
524288 3 1572864
786432 9 7077888
1048576 12 12582912
1572864 5 7864320
2097152 3 6291456
3145728 5 15728640
4194304 4 16777216
8388608 1 8388608
12582912 3 37748736

MEMPOOL_DMA POOL STATS:

Non-mmapped bytes allocated = 150994944
Number of free chunks = 90
Number of mmapped regions = 0
Mmapped bytes allocated = 0
Max memory footprint = 150994944
Keepcost = 23893280
Max contiguous free mem = 23893280
Allocated memory in use = 127006848
Free memory = 23988096

----- fragmented memory statistics -----

fragment size count total
(bytes) (bytes)
---------------- ---------- --------------
32 1 32
48 6 288
64 1 64
96 1 96**
112 1 112
256 26 7472
384 32 15344
512 3 1872
1024 13 16096
4096 1 5632
6144 1 6304
8192 2 19536
16384 1 21744
23893280 1 23893280*

* - top most releasable chunk.
** - contiguous memory on top of heap.

----- allocated memory statistics -----

fragment size count total
(bytes) (bytes)
---------------- ---------- --------------
160 1 160
240 7 1680
256 2 512
512 44 22528
768 1 768
1024 165 168960
2048 5 10240
8192 1 8192
12288 14 172032
16384 3 49152
32768 38 1245184
49152 5 245760
65536 1 65536
98304 4 393216
131072 9 1179648
196608 2 393216
262144 4 1048576
393216 5 1966080
524288 2 1048576
786432 3 2359296
1048576 11 11534336
1572864 6 9437184
3145728 3 9437184
6291456 2 12582912
8388608 1 8388608
12582912 2 25165824

MEMPOOL_GLOBAL_SHARED POOL STATS:

Non-mmapped bytes allocated = 135168
Number of free chunks = 4
Number of mmapped regions = 0
Mmapped bytes allocated = 0
Max memory footprint = 0
Keepcost = 109840
Max contiguous free mem = 109840
Allocated memory in use = 4064
Free memory = 131104

----- fragmented memory statistics -----

fragment size count total
(bytes) (bytes)
---------------- ---------- --------------
432 1 432
12288 1 14896

----- allocated memory statistics -----

fragment size count total
(bytes) (bytes)
---------------- ---------- --------------
96 1 96
112 1 112
160 1 160
208 3 624

Summary for all pools:

Non-mmapped bytes allocated = 604114944
Number of free chunks = 1089
Number of mmapped regions = 0
Mmapped bytes allocated = 0
Max memory footprint = 603979776
Keepcost = 246093216
Allocated memory in use = 357602576
Free memory = 246512368

[Michael Muenz]

Oh ... syslog throws out:

2

Feb 25 2016

06:58:03

System Memory usage reached 101%

[Gian Paolo Caparrelli]

Got the same problem:

2

Mar 21 2016

14:25:18

System Memory usage reached 181%

# sh mem        
Free memory:      18446744070756716172 bytes (488978801%)
Used memory:        6817501424 bytes (-488978701%)
-------------     ------------------
Total memory:       3772503244 bytes (100%)

# sh ver
Cisco Adaptive Security Appliance Software Version 9.5(2)
Device Manager Version 7.5(2)153

[Michael Muenz]

This is fixed with the newly released 9.6.1 (no release notes available yet)

[Aditya Ganjoo]

Hi,

Can we please mark the correct answer as this is a cosmetic bug CSCux15273 on the ASA ?

Regards,

Aditya

Please rate helpful posts.

[Michael Muenz]

I marked it as correct because als TAC said it's this bug, but it's not just cosmetic. My ASA stops working when reaching around 200% usage. 

[Gian Paolo Caparrelli]

Updated to 9.6.1 but now I have problems with sfr

1

Mar 21 2016

16:52:14

Module sfr experienced a data channel communication failure, data channel is DOWN.

[myanuary]

My ASA is 9.6.1 but also have similar symtomps, allocated memory 281%

After checking, 9.6.1 also has similar bug : https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuz23576

Its fixed on 9.6.2

[Aditya Ganjoo]

Hi,

This issue seems cosmetic.

Customer-ASA1-DOM# sh mem detail
Free memory heap: 222393168 bytes ( 6%)
Free memory system: 18446744073660725331 bytes (488978400%)

You are hitting the following bug:

https://tools.cisco.com/bugsearch/bug/CSCux15273/?reffering_site=dumpcr

Regards,

Aditya

Please rate helpful posts.

[Michael Muenz]

Hm ... thanks!

Now I checked my central syslog and I found many ASAs with 9.5.2.2 trowing out this memory error:

Today 14:02:46 LOCAL4 CRIT 172.9.4.31 %ASA-2-321006: Syslog Message Details System Memory usage reached 331%
Today 14:01:54 LOCAL7 CRIT 172.9.4.17 %ASA-2-321006: Syslog Message Details System Memory usage reached 133%
Today 14:01:18 LOCAL4 CRIT 172.9.4.21 %ASA-2-321006: Syslog Message Details System Memory usage reached 101%

Also the system with 331% doesn't respond to me SSH connection :( 

Since only on system is running OSPF this must be something related to 9.5.2.

[Tim Glen]

Hello,

The bug states, "When memory is used on the ASA and then freed, the ASA may not account for the freed memory, and although it is released back to the system, outputs such as show memory do not indicate this."

My NMS is also showing what appears to be a memory leak, see attached png. 

Does this bug also incorrectly report memory usage to ASA's SNMP Engine?

I am also on 9.5(2).

Cisco Adaptive Security Appliance Software Version 9.5(2)
 

Thank you