01-22-2024 07:42 AM
Hello,
I upgraded an standalone FRP 2130 ASA appliance mode from version 9.15(1)15 to 9.18(3). Used for remote vpn, everything is working correctly,..But I noticed after 4 days that the memory usage is higher (previously 20%, now +75%). and I see in our monitoring tool which is based on snmp the status is: UNKNOWN, Even though I can retrieve the information successfully :
$ snmpwalk -v 2c -c comm ip sysdesc
SNMPv2-MIB::sysDescr.0 = STRING: Cisco Adaptive Security Appliance Version 9.18(3)
Any ideas, please?
# sh memory
Free memory: 3305546696 bytes (23%)
Used memory: 11062033464 bytes (77%)
------------- ------------------
Total memory: 14367580160 bytes (100%)
Thank you
Solved! Go to Solution.
01-23-2024 06:31 AM
Only this chunk type stands out, SNMP-related (SNMP logging):
Chunk type: 563, name: "NsConf log"
pool_type: 1 (MEMPOOL_GLOBAL_SHARED) flags: 00000401 ( DYNAMIC NO_LIMIT)
addr: 0x000000555d1d8f50, data start: 0x000000555d1dd010, end: 0x000000555d225010
next: 0x000000555d47c600, next_sibling: 0x000000fcfc7b9c10, prev_sibling: 0x0000000000000000
real_max: 2048, real_size: 144, first free index: 0
total_inuse: 70198585, total_free: 711, inuse_hwm: 70198587
alignment: 0, base alignment: 0
chunk size: 311507, chunk data size: 0, num chunks: 34277, total size: 2087590847
Per-chunk statistics: siblings created 35497, siblings trimmed 1221
So, it appears the issue is caused by SNMP. The leak is very fast, because you probably poll the device at very high rate. Do you?
There are two bugs which match: CSCwe65245 and internal CSCwf70825. Both were fixed in 9.18.4, but I would still upgrade to 9.18.4.8. Of course, you need to work with TAC to confirm my findings. And also check disk0:, if you have SNMP log files of big size: "dir /r".
HTH
01-22-2024 08:08 AM
01-22-2024 08:30 AM
Hello @MHM Cisco World,
Thanks for your feedback
I don't think it's related to this bug !
9.18(3) not mentioned, fixed in : https://bst.cisco.com/bugsearch/bug/CSCvx54562
+ I use ASA code not FTD.
Thank you.
01-22-2024 08:42 AM
This high memory usage unusual
And I think 90% it relate to bug
One more bug about high memory usage
https://bst.cisco.com/bugsearch/bug/CSCwi20848?rfs=qvlogin
MHM
01-22-2024 11:17 AM
If this is a leak, you will see something unusual in the "show memory top" output: memory allocation for blocks of the same size will stand out.
HTH
01-22-2024 11:59 AM
Hello @tvotna
Thanks for your feedback
Attached "show memory top" output, can you please check?
Thank you
01-22-2024 12:24 PM
Google is best friend here
Write in search your fpr ver and asa and high memory usage and check all bug you get.
There is nothing here you can do.
Believe me.
Goodluck friend
MHM
01-23-2024 03:08 AM
@MHM Cisco World, picking up random bugs from Google search is not the best strategy when people ask for help. The fact that you don't know how to troubleshoot memory leak issues doesn't mean that others cannot do this.
@cisco.13, this indeed looks like a bug. From the output it's evident that almost all heap memory is allocated by the same function 0xaaaf81994c in 262144 byte blocks. It's obvious that the memory was allocated by chunk manager process, hence we need "show chunkstat" output. As the "show chunkstat" output is huge, write it to flash: "show chunkstat | redir disk0:/chunkstat.txt".
Also, please provide "show vpn-sessiondb" and "show counters".
There are few memory leak bugs which were fixed in 9.18.3, but it's impossible to identify an exact one, until "show chunkstat" is decoded. Most likely the issue is caused by VPN subsystem or SNMP. If memory utilization becomes critical, open TAC case and provide above outputs, or I can try to decode them myself if you don't have TAC support. In general, you'll probably need to upgrade to the latest 9.18 interim, which is 9.18.4.8, if I'm not mistaken.
01-23-2024 03:53 AM
Hello @tvotna
Thank you very much for your feedback and help.
Yes, I opened a TAC case, they are also looking.
Currently
Free memory: 1660771944 bytes (12%)
Used memory: 12706808216 bytes (88%)
------------- ------------------
Total memory: 14367580160 bytes (100%)
Here are the files
Thank you.
01-23-2024 06:31 AM
Only this chunk type stands out, SNMP-related (SNMP logging):
Chunk type: 563, name: "NsConf log"
pool_type: 1 (MEMPOOL_GLOBAL_SHARED) flags: 00000401 ( DYNAMIC NO_LIMIT)
addr: 0x000000555d1d8f50, data start: 0x000000555d1dd010, end: 0x000000555d225010
next: 0x000000555d47c600, next_sibling: 0x000000fcfc7b9c10, prev_sibling: 0x0000000000000000
real_max: 2048, real_size: 144, first free index: 0
total_inuse: 70198585, total_free: 711, inuse_hwm: 70198587
alignment: 0, base alignment: 0
chunk size: 311507, chunk data size: 0, num chunks: 34277, total size: 2087590847
Per-chunk statistics: siblings created 35497, siblings trimmed 1221
So, it appears the issue is caused by SNMP. The leak is very fast, because you probably poll the device at very high rate. Do you?
There are two bugs which match: CSCwe65245 and internal CSCwf70825. Both were fixed in 9.18.4, but I would still upgrade to 9.18.4.8. Of course, you need to work with TAC to confirm my findings. And also check disk0:, if you have SNMP log files of big size: "dir /r".
HTH
01-23-2024 08:00 AM
Hello @tvotna,
Thank you very much for your analysis, I will try v9.18.4.8
Possible to share a document how do you diagnose memory problems please (if possible)?
Thank you.
01-23-2024 08:09 AM
Perfect Q
I am also interested what number indicates that SNMP is consume memory.????
MHM
01-24-2024 12:41 AM
@cisco.13, unfortunately, I don't have one. The commands "show memory", "show memory detail", "show memory top" and sometimes "show chunkstat" are the most helpful. If you upgrade, you'll find "show chunkstat top" helpful too. Then look for something unusual, e.g. abnormally high number of memory allocations by the same pc (program counter) or for the same chunk type. Collect stats few times and check if numbers increase. Then apply common sense. Disable suspected subsystems, e.g. SNMP, and check again.
If the leak is fast, it can only be caused by some function which is invoked frequently. You mentioned VPN and SNMP in the very beginning, but the number of VPN sessions is very small, so most likely we can rule out VPN. SNMP, on the other hand, is known to cause many issues as of 9.14, including memory leaks. It's unlikely that regular firewall functions, e.g. connections or NAT can cause leaks like this, simply because corresponding code is old and pretty stable. Of course, there is always a possibility that you configured something very specific on the box, which is rarely used and hence rarely tested by dev team. So it's worth to analyze config from this angle too.
There are few other methods, but they require decoding of memory addresses, which can only be done by TAC, as we don't have symbol files. Also, exact bug is always difficult to spot, because bug descriptions created by TAC are always poorly written.
@MHM Cisco World, "NsConf" is abbreviation for Netsnmp Configuration.
01-24-2024 02:41 AM
Friend let me answer you
as I mention it is bug you need to check and it smart step you open TAC.
now why suspect about chunk NsConf Log <<- not need to be SNMP issue
the memory is divide into room and there is specific room for each process
the total_inuse room for this process is so high and there is so little free left
total_inuse: 70198585, total_free: 711, inuse_hwm: 70198587
you mention you upgrade to 9.18(3) so you can check 9.18(4) and later ver. the bug solved (check each bug one by one)
each bug give you some detail you can use it to see if it relate to your issue or not.
sorry I can not help you so much here
thanks a lot
have a nice day
MHM
01-24-2024 04:42 AM
Hello @tvotna @MHM Cisco World, Thanks for your feedback
I imagine that TACs have tools to analyze memory leak problems, otherwise, it's complicated and takes a lot of time!
I tested the v9.18.4.8 (+9.18.4.x & 9.18.3.x), it has another bug : loss of 'smart license' after reboot, another similar case has been reported to TAC. only on FPR-2130, not on ASAv.
Thank you
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide