ASA 5585 SSP20 内存偏高问题

hexingke24 · ‎03-31-2014

本帖最后由 hexingke24 于 2014-3-31 19:26 编辑

这台ASA 5585 SSP20主要做NAT，最近发现内存使用率很高(12GB内存)，大概三个月前才只有34%，现在高峰期可以达到82%，一般都维持在72%左右。一个月前重启过一次设备，就陆续出现一些不能上网的问题，之前没关注内存的问题，最近发现内存使用率很高。
三个月前ASA的一些信息：
------------------ show memory ------------------
F ree memory: 8481390208 bytes (66%)
Used memory: 4403511680 bytes (34%)
------------- ------------------
Total memory: 12884901888 bytes (100%)
------------------ show conn count ------------------
184103 in use, 847809 most used
------------------ show xlate count ------------------
241275 in use, 653771 most used
现在ASA的一些信息：
------------------ show memory ------------------
F ree memory: 3603259008bytes (28%)
Used memory: 9281642880 bytes (72%)
------------- ------------------
Total memory: 12884901888 bytes (100%)

------------------ show conn count ------------------
197568 in use, 550139 most used

------------------ show xlate count ------------------
259627 in use, 619869 most used

对比可以看到，一些连接数和nat转换数没有多大变化，就是内存变化很大。
下面的 show memorydetail 有人可以看懂吗？
非常感谢。
ASA5585# show memory detail
F ree memory: 3589808144 bytes (28%)
Used memory:
Allocatedmemory in use: 7227715568 bytes (56%)
Reserved memory: 2067378176 bytes (16%)
----------------------------- ------------------
Total memory: 12884901888 bytes (100%)

Least f ree memory: 3436660384 bytes(27%)
Most used memory: 9448241504 bytes (73%)

MEMPOOL_DMA POOL STATS:
Non-mmapped bytes allocated = 669515776
Number of f ree chunks = 258
Number of mmapped regions = 0
Mmapped bytes allocated = 0
Max memory footprint = 669515776
Keepcost = 437875744
Max contiguous f ree mem = 437875744
Allocated memory in use = 231530416
F ree memory = 437985360

----- fragmented memory statistics -----
fragment size count total
(bytes) (bytes)
---------------- ---------- --------------
48 3 144
64 2 128
96 1 96**
96 1 96
256 122 39232
384 64 30720
512 64 38912
437875744 1 437875744*

* - top most releasable chunk.
** - contiguous memory on top of heap.

----- allocated memory statistics -----
fragment size count total
(bytes) (bytes)
---------------- ---------- --------------
80 1 80
96 1 96
144 1 144
224 2 448
240 3 720
256 4 1024
384 2 768
512 8 4096
1024 258 264192
2048 2 4096
8192 6 49152
12288 76 933888
16384 2 32768
32768 5 163840
65536 2 131072
98304 2 196608
131072 3 393216
196608 3 589824
262144 8 2097152
393216 4 1572864
524288 2 1048576
786432 6 4718592
1048576 7 7340032
1572864 1 1572864
2097152 1 2097152
3145728 7 22020096
6291456 4 25165824
8388608 4 33554432
12582912 3 37748736

MEMPOOL_GLOBAL_SHARED POOL STATS:
Non-mmapped bytes allocated = 10817523712
Number of f ree chunks = 28534
Number of mmapped regions = 0
Mmapped bytes allocated = 0
Max memory footprint = 10817523712
Keepcost = 2880411392
Max contiguous f ree mem = 2880411392
Allocated memory in use = 7227712352
F ree memory = 3589811360

----- fragmented memory statistics -----
fragment size count total
(bytes) (bytes)
---------------- ---------- --------------
32 1473 47136
48 1098 52704
64 1252 80128
80 246 19680
96 1 96**
96 11 1056
112 442 49504
128 584 74752
144 266 38304
160 561 89760
192 262 50304
208 355 73840
224 329 73696
240 47 11280
256 755 248768
384 981 424736
512 933 591136
768 953 847440
1024 1757 2003584
1536 560 976272
2048 855 2018160
3072 3925 14512816
4096 4404 22667856
6144 846 5434048
8192 1 10240
32768 680 30959216
49152 358 18345376
65536 223 19792496
98304 1995 244124608
131072 2379 332193936
12582912 1 13587008
2880411392 1 2880411392*

* - top most releasable chunk.
** - contiguous memory on top of heap.

----- allocated memory statistics -----
fragment size count total
(bytes) (bytes)
---------------- ---------- --------------
80 1329 106320
96 11639 1117344
112 4739 530768
128 11658 1492224
144 5784 832896
160 1207 193120
176 1973 347248
192 550 105600
208 326 67808
224 19965 4472160
240 1100 264000
256 3003 768768
384 14361 5514624
512 20789 10643968
768 266 204288
1024 12665 12968960
1536 1119 1718784
2048 166 339968
3072 92 282624
4096 1678 6873088
6144 33968 208699392
8192 423 3465216
12288 226 2777088
16384 63 1032192
24576 21 516096
32768 8820 289013760
49152 1527 75055104
65536 258 16908288
98304 65 6389760
131072 35226 4617142272
196608 24 4718592
262144 58 15204352
393216 18 7077888
524288 214 112197632
786432 25 19660800
1048576 74 77594624
1572864 1 1572864
2097152 211 442499072
3145728 3 9437184
4194304 1 4194304
8388608 2 16777216
12582912 9 113246208

Summary for all pools:
Non-mmapped bytes allocated = 11487039488
Number of f ree chunks = 28792
Number of mmapped regions = 0
Mmapped bytes allocated = 0
Max memory footprint = 11487039488
Keepcost = 3318287136
Allocated memory in use = 7459242768
F ree memory = 4027796720

yangliu2 · ‎03-31-2014

你ASA是什么版本 ?
请收集再收集一次新的 show memory detail
和 show chuckstat

hexingke24 · ‎03-31-2014

yangliu2 发表于 2014-3-31 23:51
你ASA是什么版本 ?
请收集再收集一次新的 show memory detail
和 show chuckstat

感谢您的回答。
这台ASA的版本是8.4(5)7，Cisco提供的一本内部版本，正常使用一年了。
刚看了一下show memory detail，发现内存使用更高了，达到80%。
ASA5585# sh mem det
F ree memory: 2536878896 bytes (20%)
Used memory:
Allocated memory in use: 8280644816 bytes (64%)
Reserved memory: 2067378176 bytes (16%)
----------------------------- ------------------
Total memory: 12884901888 bytes (100%)

最新的 show memory detail 和 show chunkstat 信息请看附件文件。

yangliu2 · ‎04-01-2014

请收集：
sh mem binsize 6144
sh mem binsize 32768
sh mem binsize 131072
sh mem binsize 524288
sh mem binsize 2097152
请把输出内容直接贴出来吧，最好别以附件形式上传。

yangliu2 · ‎04-01-2014

像是大量的Threat-detection 统计信息占用了你的 memory
你可以先关闭此功能：
ciscoasa(config)# clear configure threat-detection

hexingke24 · ‎04-01-2014

yangliu2 发表于 2014-4-1 15:15
请收集：
sh mem binsize 6144
sh mem binsize 32768

ASA5585(config)# sh mem binsize 6144
MEMPOOL_DMA pool bin stats:
MEMPOOL_GLOBAL_SHARED pool bin stats:
pc = 0x19bae88, size = 6769664 , count = 1024
pc = 0x19bae88, size = 6769664 , count = 1024
pc = 0x19bae88, size = 6769664 , count = 1024
pc = 0x19bae88, size = 6769664 , count = 1024
pc = 0x19bae88, size = 6769664 , count = 1024
pc = 0x19bae88, size = 6769664 , count = 1024
pc = 0x19bae88, size = 6769664 , count = 1024
pc = 0x19bae88, size = 6769664 , count = 1024
pc = 0x19bae88, size = 6769664 , count = 1024
pc = 0x19bae88, size = 6769664 , count = 1024
pc = 0x19bae88, size = 6769664 , count = 1024
pc = 0x19bae88, size = 6769664 , count = 1024
pc = 0x19bae88, size = 6769664 , count = 1024
pc = 0x19bae88, size = 6769664 , count = 1024
pc = 0x19bae88, size = 6769664 , count = 1024
pc = 0x19bae88, size = 6769664 , count = 1024
pc = 0x19bae88, size = 6769664 , count = 1024
pc = 0x19bae88, size = 6769664 , count = 1024
pc = 0x19bae88, size = 6769664 , count = 1024
pc = 0x19bae88, size = 6769664 , count = 1024
pc = 0x19bae88, size = 6769664 , count = 1024
pc = 0x19bae88, size = 6769664 , count = 1024
pc = 0x19bae88, size = 6769664 , count = 1024
pc = 0x19bae88, size = 6769664 , count = 1024
pc = 0x19bae88, size = 6769664 , count = 1024
pc = 0x19bae88, size = 6769664 , count = 1024
pc = 0x19bae88, size = 6769664 , count = 1024
pc = 0x19bae88, size = 6769664 , count = 1024
pc = 0x19bae88, size = 6769664 , count = 1024
pc = 0x19bae88, size = 6769664 , count = 1024
pc = 0x19bae88, size = 6769664 , count = 1024
pc = 0x19bae88, size = 6763053 , count = 1023
pc = 0x19bb109, size = 6611 , count = 1
pc = 0x19bae88, size = 6769664 , count = 1024
pc = 0x19bae88, size = 6769664 , count = 1024
pc = 0x19bae88, size = 6769664 , count = 1024
pc = 0x19bae88, size = 6769664 , count = 1024
pc = 0x19bae88, size = 6769664 , count = 1024
pc = 0x19bae88, size = 6769664 , count = 1024
pc = 0x19bae88, size = 6769664 , count = 1024
pc = 0x19bae88, size = 1229646 , count = 186
pc = 0x19bb109, size = 74833 , count = 11
pc = 0xdc311f, size = 6352 , count = 1
pc = 0x19bae0f, size = 117774 , count = 18
pc = 0x1ed244b, size = 6296 , count = 1
pc = 0x1254c12, size = 7128 , count = 1
pc = 0x1254a5f, size = 7136 , count = 1
pc = 0xcbdc71, size = 8000 , count = 1
pc = 0xe1fce8, size = 6352 , count = 1
pc = 0xe0a409, size = 6352 , count = 1
pc = 0x150778a, size = 7976 , count = 1
pc = 0x1509e3f, size = 7976 , count = 1
pc = 0x14fe570, size = 8000 , count = 1
pc = 0xa2afe6, size = 7200 , count = 1
pc = 0xb3c806, size = 8008 , count = 1
Number of entries were > 1024, aggregation happened on each 1024 blocks
ASA5585(config)# sh mem binsize 32768
MEMPOOL_DMA pool bin stats:
pc = 0x51dbdd, size = 152444 , count = 4
pc = 0x1a43cdd, size = 43008 , count = 1
MEMPOOL_GLOBAL_SHARED pool bin stats:
pc = 0x19bae0f, size = 45701489 , count = 1023
pc = 0x19bae88, size = 32979 , count = 1
pc = 0x19bae0f, size = 45746432 , count = 1024
pc = 0x19bae0f, size = 45880689 , count = 1023
pc = 0x424427, size = 32768 , count = 1
pc = 0x19bae0f, size = 43314432 , count = 1024
pc = 0x19bae0f, size = 44485632 , count = 1024
pc = 0x19bae0f, size = 42258146 , count = 1022
pc = 0x19bb109, size = 72422 , count = 2
pc = 0x19bae0f, size = 39992832 , count = 1024
pc = 0x19bae0f, size = 24485317 , count = 619
pc = 0x424427, size = 4096000 , count = 125
pc = 0x19bb109, size = 239922 , count = 6
pc = 0x138d0bb, size = 192832 , count = 5
pc = 0xb3e702, size = 524288 , count = 16
pc = 0x14fdf68, size = 34760 , count = 1
pc = 0x19bae88, size = 173087 , count = 5
pc = 0x1ed244b, size = 129528 , count = 3
pc = 0x19cce33, size = 32768 , count = 1
pc = 0x5bccf5, size = 262208 , count = 8
pc = 0x10e0d00, size = 33032 , count = 1
Number of entries were > 1024, aggregation happened on each 1024 blocks
ASA5585(config)# sh mem binsize 131072
MEMPOOL_DMA pool bin stats:
pc = 0x40e594, size = 150479 , count = 1
pc = 0x40d269, size = 294686 , count = 2
MEMPOOL_GLOBAL_SHARED pool bin stats:
pc = 0x19bae0f, size = 146066432, count = 1024
pc = 0x19bae0f, size = 146808117, count = 1019
pc = 0x5c369e, size = 491904 , count = 3
pc = 0x57e379, size = 262320 , count = 2
pc = 0x19bae0f, size = 152952832, count = 1024
pc = 0x19bae0f, size = 155244032, count = 1024
pc = 0x19bae0f, size = 156101632, count = 1024
pc = 0x19bae0f, size = 154527232, count = 1024
pc = 0x19bae0f, size = 154322432, count = 1024
pc = 0x19bae0f, size = 153900032, count = 1024
pc = 0x19bae0f, size = 155743232, count = 1024
pc = 0x19bae0f, size = 155116032, count = 1024
pc = 0x19bae0f, size = 155551232, count = 1024
pc = 0x19bae0f, size = 156626432, count = 1024
pc = 0x19bae0f, size = 157164032, count = 1024
pc = 0x19bae0f, size = 155461632, count = 1024
pc = 0x19bae0f, size = 156805632, count = 1024
pc = 0x19bae0f, size = 155551232, count = 1024
pc = 0x19bae0f, size = 156613632, count = 1024
pc = 0x19bae0f, size = 157189632, count = 1024
pc = 0x19bae0f, size = 157356032, count = 1024
pc = 0x19bae0f, size = 157394432, count = 1024
pc = 0x19bae0f, size = 157343232, count = 1024
pc = 0x19bae0f, size = 157343232, count = 1024
pc = 0x19bae0f, size = 157330432, count = 1024
pc = 0x19bae0f, size = 157368832, count = 1024
pc = 0x19bae0f, size = 157394432, count = 1024
pc = 0x19bae0f, size = 157381632, count = 1024
pc = 0x19bae0f, size = 157368832, count = 1024
pc = 0x19bae0f, size = 157394432, count = 1024
pc = 0x19bae0f, size = 157432832, count = 1024
pc = 0x19bae0f, size = 157407232, count = 1024
pc = 0x19bae0f, size = 157381632, count = 1024
pc = 0x19bae0f, size = 157381632, count = 1024
pc = 0x19bae0f, size = 157407232, count = 1024
pc = 0x19bae0f, size = 157420032, count = 1024
pc = 0x19bae0f, size = 157432832, count = 1024
pc = 0x19bae0f, size = 157317632, count = 1024
pc = 0x19bae0f, size = 157394432, count = 1024
pc = 0x19bae0f, size = 157420032, count = 1024
pc = 0x19bae0f, size = 157381632, count = 1024
pc = 0x19bae0f, size = 157279232, count = 1024
pc = 0x19bae0f, size = 157356032, count = 1024
pc = 0x19bae0f, size = 106582728, count = 696
pc = 0x138d0bb, size = 178912 , count = 1
pc = 0xa27d86, size = 524288 , count = 4
pc = 0xd3bfab, size = 192040 , count = 1
pc = 0x137c517, size = 3278240 , count = 20
pc = 0x19bb109, size = 323622 , count = 2
pc = 0x153765c, size = 172200 , count = 1
pc = 0x1a7e59b, size = 139272 , count = 1
pc = 0x1ed244b, size = 150288 , count = 1
Number of entries were > 1024, aggregation happened on each 1024 blocks
ASA5585(config)# sh mem binsize 524288
MEMPOOL_DMA pool bin stats:
pc = 0x40e594, size = 544079 , count = 1
pc = 0x40d269, size = 588943 , count = 1
MEMPOOL_GLOBAL_SHARED pool bin stats:
pc = 0x19bae0f, size = 142122507, count = 197
pc = 0x795a0a, size = 524288 , count = 1
pc = 0x1338210, size = 557056 , count = 1
pc = 0x1ed244b, size = 589824 , count = 1
ASA5585(config)# sh mem binsize 2097152
MEMPOOL_DMA pool bin stats:
pc = 0x40e594, size = 3080271 , count = 1
MEMPOOL_GLOBAL_SHARED pool bin stats:
pc = 0x19bae0f, size = 23084807 , count = 9
pc = 0xa2af2b, size = 2621440 , count = 1
pc = 0x19bb109, size = 2138323 , count = 1

hexingke24 · ‎04-01-2014

yangliu2 发表于 2014-4-1 15:22
像是大量的Threat-detection 统计信息占用了你的 memory
你可以先关闭此功能：
ciscoasa(config)# clear ...

非常感谢您的回答。
昨晚按照您说的关闭Threat-detection后，内存从原来的82%降到了76%，同时我把pat-xlate 的时间降到1分钟。
我已经把您需要我搜集的binzise贴出来了，烦请您帮忙看一下。
再次感谢。

yangliu2 · ‎04-02-2014

你好，请在收集一次 show chunkstat

hexingke24 · ‎04-03-2014

yangliu2 发表于 2014-4-3 14:53
你好，请在收集一次 show chunkstat

感谢您的回答。
由于show chunkstat的信息太多，超过了最大的字数要求，所以我还是以附件的形式发出来。
如果有什么要求，请告知。

yangliu2 · ‎04-03-2014

你确定已经关闭了 threat-detection 吗？
请将 show run threat-detection 发给我。

hexingke24 · ‎04-08-2014

yangliu2 发表于 2014-4-4 10:45
你确定已经关闭了 threat-detection 吗？
请将 show run threat-detection 发给我。

确定全部threat-detection已经关闭了，都是no threat-detection命令。

yangliu2 · ‎04-09-2014

这个问题肯定是 threat-detection 引起的，目前看来就算关闭了 threat-detection feature 也没能使它彻底释放 memory。如果你需要，通过 “ clear local all" 将所有的connection 清除。注意此命令会将所有ASA上的 session清楚，你最好申请维护窗口做。

hexingke24 · ‎04-10-2014

yangliu2 发表于 2014-4-9 16:33
这个问题肯定是 threat-detection 引起的，目前看来就算关闭了 threat-detection feature 也没能使它彻底 ...

感谢您的回答。
今天早上进行了clear local all操作，但是内存没有改变。请问还有其他解决方法吗？重启？

songl · ‎04-18-2014

为啥用中间版本，最好用正式版本 8.4.7 可以考虑。

yangliu2 · ‎04-20-2014

建议你关闭所有TD 后重启一下ASA ，问题应该可以解决

陈建0 · ‎05-31-2014

好帖要顶,楼主的头像还是不错滴