Solved: What do these snmpwalk results indicate?

vonnpiolo · ‎05-24-2024

Hello,

We're currently doing fine tuning work in our monitoring systems for Cisco ASA and Firepower and we would like details monitoring of the free and used memory.

We identified these two OID's to do the job:

1.3.6.1.4.1.9.9.221.1.1.1.1.7: Used Memory
1.3.6.1.4.1.9.9.221.1.1.1.1.8: Free Memory

Here are the results of the snmpwalk:

[admin@device~]$ snmpwalk -v2c -c xxxxxxxxx [IP ADDRESS] 1.3.6.1.4.1.9.9.221.1.1.1.1.7
SNMPv2-SMI::enterprises.9.9.221.1.1.1.1.7.2.1 = Gauge32: 1422924688
SNMPv2-SMI::enterprises.9.9.221.1.1.1.1.7.2.3 = Gauge32: 209056
SNMPv2-SMI::enterprises.9.9.221.1.1.1.1.7.2.4 = Gauge32: 571447456
SNMPv2-SMI::enterprises.9.9.221.1.1.1.1.7.2.5 = Gauge32: 261709328
SNMPv2-SMI::enterprises.9.9.221.1.1.1.1.7.2.6 = Gauge32: 752

[admin@device~]$ snmpwalk -v2c -c xxxxxxxxx [IP ADDRESS] 1.3.6.1.4.1.9.9.221.1.1.1.1.8
SNMPv2-SMI::enterprises.9.9.221.1.1.1.1.8.2.1 = Gauge32: 1064529848
SNMPv2-SMI::enterprises.9.9.221.1.1.1.1.8.2.3 = Gauge32: 8179552
SNMPv2-SMI::enterprises.9.9.221.1.1.1.1.8.2.4 = Gauge32: 810575712
SNMPv2-SMI::enterprises.9.9.221.1.1.1.1.8.2.5 = Gauge32: 201761264
SNMPv2-SMI::enterprises.9.9.221.1.1.1.1.8.2.6 = Gauge32: 134416

The question is: what are the meaning of each of those Gauges?

Here is the result when we run "show memory"

> show memory
Free memory: 12733046336 bytes (89%)
Used memory: 1634509248 bytes (11%)
------------- ------------------
Total memory: 14367555584 bytes (100%)

Here are the results when we run "show memory details":

> show memory detail

Heap Memory:
Free Memory:
Heapcache Pool: 179741136 bytes ( 1% )
Global Shared Pool: 132272 bytes ( 0% )
Message Layer Pool: 6110624 bytes ( 0% )
System: 12553076736 bytes ( 87% )
Used Memory:
Heapcache Pool: 1195990576 bytes ( 8% )
Global Shared Pool: 2896 bytes ( 0% )
Reserved for messaging: 2277984 bytes ( 0% )
MMAP usage: 63082496 bytes ( 0% )
System Overhead: 367140864 bytes ( 3% )
------------------------------------- ----------------
Total Memory: 14367555584 bytes ( 100% )

Warning: The information reported here is computationally expensive to
determine, and may result in CPU hogs and performance impact.

-----------------------------------------------------------------------

Additional DMA Memory
----------------------------------------------
Total Memory 1073741824 bytes ( 100% )
Used Memory 795477840 bytes ( 74% )
Free Memory 278263984 bytes ( 26% )
----------------------------------------------

MEMPOOL_MSGLYR POOL STATS:

Non-mmapped bytes allocated = 8388608
Number of free chunks = 60
Number of mmapped regions = 0
Mmapped bytes allocated = 0
Max memory footprint = 8388608
Keepcost = 5942544
Max contiguous free mem = 5942544
Allocated memory in use = 2277984
Free memory = 6110624

----- fragmented memory statistics -----

fragment size count total
(bytes) (bytes)
---------------- ---------- --------------
32 18 576
48 15 720
64 12 768
80 1 80
96 1 96**
112 1 112
128 1 128
144 1 144
160 2 320
208 1 208
240 1 240
256 1 304
1536 1 2016
6144 1 7440
8192 1 9408
65536 1 75168
5942544 1 5942544*

* - top most releasable chunk.
** - contiguous memory on top of heap.

----- allocated memory statistics -----

fragment size count total
(bytes) (bytes)
---------------- ---------- --------------
80 108 8640
96 136 13056
112 172 19264
128 105 13440
144 106 15264
160 53 8480
176 160 28160
192 7 1344
224 8 1792
240 14 3360
256 159 40704
384 9 3456
512 39 19968
1024 147 150528
1536 37 56832
4096 1 4096
8192 22 180224
16384 98 1605632

MEMPOOL_HEAPCACHE_0 POOL STATS:

Non-mmapped bytes allocated = 1375731712
Number of free chunks = 32438
Number of mmapped regions = 0
Mmapped bytes allocated = 0
Max memory footprint = 1375731712
Keepcost = 178332416
Max contiguous free mem = 178332416
Allocated memory in use = 1195990576
Free memory = 179741136

----- fragmented memory statistics -----

fragment size count total
(bytes) (bytes)
---------------- ---------- --------------
32 14913 477216
48 11903 571344
64 5620 359680
96 1 96**
178332416 1 178332416*

* - top most releasable chunk.
** - contiguous memory on top of heap.

----- allocated memory statistics -----

fragment size count total
(bytes) (bytes)
---------------- ---------- --------------
80 141511 11320880
96 20737 1990752
112 8314 931168
128 6302 806656
144 6420 924480
160 685 109600
176 1008 177408
192 235 45120
208 260 54080
224 1250 280000
240 497 119280
256 2698 690688
384 542 208128
512 598 306176
768 706 542208
1024 1086 1112064
1536 1182 1815552
2048 626 1282048
3072 97 297984
4096 7950 32563200
6144 299 1837056
8192 262 2146304
12288 477 5861376
16384 676 11075584
24576 111 2727936
32768 90 2949120
49152 98 4816896
65536 140 9175040
98304 169 16613376
131072 218 28573696
196608 74 14548992
262144 71 18612224
393216 61 23986176
524288 49 25690112
786432 43 33816576
1048576 28 29360128
1572864 12 18874368
2097152 27 56623104
3145728 14 44040192
4194304 33 138412032
6291456 8 50331648
8388608 3 25165824
12582912 23 289406976

MEMPOOL_DMA POOL STATS:

Non-mmapped bytes allocated = 1073741824
Number of free chunks = 1
Number of mmapped regions = 0
Mmapped bytes allocated = 0
Max memory footprint = 1073741824
Keepcost = 278263888
Max contiguous free mem = 278263888
Allocated memory in use = 795477840
Free memory = 278263984

----- fragmented memory statistics -----

fragment size count total
(bytes) (bytes)
---------------- ---------- --------------
96 1 96**
278263888 1 278263888*

* - top most releasable chunk.
** - contiguous memory on top of heap.

----- allocated memory statistics -----

fragment size count total
(bytes) (bytes)
---------------- ---------- --------------
131072 2 262144
262144 2 524288
393216 4 1572864
524288 1 524288
786432 4 3145728
1048576 7 7340032
1572864 1 1572864
3145728 2 6291456
4194304 2 8388608
12582912 3 37748736

MEMPOOL_GLOBAL_SHARED POOL STATS:

Non-mmapped bytes allocated = 135168
Number of free chunks = 1
Number of mmapped regions = 0
Mmapped bytes allocated = 0
Max memory footprint = 0
Keepcost = 59584
Max contiguous free mem = 0
Allocated memory in use = 2896
Free memory = 132272

----- fragmented memory statistics -----

fragment size count total
(bytes) (bytes)
---------------- ---------- --------------
112 649 72688

----- allocated memory statistics -----

fragment size count total
(bytes) (bytes)
---------------- ---------- --------------
168 1 168
184 3 552

Summary for all pools:

Non-mmapped bytes allocated = 2457997312
Number of free chunks = 32500
Number of mmapped regions = 0
Mmapped bytes allocated = 0
Max memory footprint = 2457862144
Keepcost = 462598432
Allocated memory in use = 1993749296
Free memory

Thank you very much for your help.

tvotna · ‎05-27-2024

First of all, use 1.3.6.1.4.1.9.9.221.1.1.1.1.20 and 1.3.6.1.4.1.9.9.221.1.1.1.1.18 instead of OIDs you mentioned. These counters are 64-bit. Each OID is indexed by a memory pool id. Walk 1.3.6.1.4.1.9.9.221.1.1.1.1.3 for pool names. The per-pool values are not very helpful for ordinary people, so it doesn't make much sense to monitor them. The only OID which is helpful is a "free memory system" - cempMemPoolHCFree. Its value should match "Free memory" line from the "show memory" output. The exact OID which needs to be polled depends on the platform and mode (single/multiple), e.g. for ASA running in multiple context mode:

ASA 5585-X: .1.3.6.1.4.1.9.9.221.1.1.1.1.20.5.1
Firepower 4100/9300: .1.3.6.1.4.1.9.9.221.1.1.1.1.20.1.1
ASA 5500-X: .1.3.6.1.4.1.9.9.221.1.1.1.1.20.2.1

The exact OID can be determined by comparing snmpwalk results with "show memory" output.

Finally, if you still want to poll per-pool OIDs, it's recommended to configure "no snmp-server enable oid mempool" to prevent CPU-intensive computations of values for MEMPOOL_GLOBAL_SHARED pool (and MEMPOOL_HEAPCACHE in newer versions). Otherwise, SNMP polling can sometimes cause CPU hogs, which in turn can lead to unexpected failovers and other such things. This command is configured by default as of CSCwd38583.

HTH

View solution in original post

tvotna · ‎05-27-2024

First of all, use 1.3.6.1.4.1.9.9.221.1.1.1.1.20 and 1.3.6.1.4.1.9.9.221.1.1.1.1.18 instead of OIDs you mentioned. These counters are 64-bit. Each OID is indexed by a memory pool id. Walk 1.3.6.1.4.1.9.9.221.1.1.1.1.3 for pool names. The per-pool values are not very helpful for ordinary people, so it doesn't make much sense to monitor them. The only OID which is helpful is a "free memory system" - cempMemPoolHCFree. Its value should match "Free memory" line from the "show memory" output. The exact OID which needs to be polled depends on the platform and mode (single/multiple), e.g. for ASA running in multiple context mode:

ASA 5585-X: .1.3.6.1.4.1.9.9.221.1.1.1.1.20.5.1
Firepower 4100/9300: .1.3.6.1.4.1.9.9.221.1.1.1.1.20.1.1
ASA 5500-X: .1.3.6.1.4.1.9.9.221.1.1.1.1.20.2.1

The exact OID can be determined by comparing snmpwalk results with "show memory" output.

Finally, if you still want to poll per-pool OIDs, it's recommended to configure "no snmp-server enable oid mempool" to prevent CPU-intensive computations of values for MEMPOOL_GLOBAL_SHARED pool (and MEMPOOL_HEAPCACHE in newer versions). Otherwise, SNMP polling can sometimes cause CPU hogs, which in turn can lead to unexpected failovers and other such things. This command is configured by default as of CSCwd38583.

HTH

vonnpiolo · ‎05-29-2024

Hi tvotna,

Thank you!

Best regards,

Vonn