cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
2373
Views
0
Helpful
10
Replies

ISR4351 - %PLATFORM-4-ELEMENT_WARNING: SIP1:smand: RP/0: Used Memory

Jagvella
Level 1
Level 1

Hi guys,

I am seeing this error keep popping up in the router ISR4351 logs. Not sure what is causing this.

%PLATFORM-4-ELEMENT_WARNING: SIP1: smand: RP/0: Used Memory value 93% exceeds warning level 88%

Router ver: 16.6.7

Similar router elsewhere with same IOS version, not having this at all. Some output:

================================================================

Router#show platform hardware qfp active datapath utilization
CPP 0: Subdev 0 5 secs 1 min 5 min 60 min
Input: Priority (pps) 0 0 0 0
(bps) 0 0 0 0
Non-Priority (pps) 116 135 145 151
(bps) 247736 322920 422648 385848
Total (pps) 116 135 145 151
(bps) 247736 322920 422648 385848
Output: Priority (pps) 3 3 2 2
(bps) 6472 6208 4640 4720
Non-Priority (pps) 225 139 134 145
(bps) 1196160 493288 473368 444720
Total (pps) 228 142 136 147
(bps) 1202632 499496 478008 449440
Processing: Load (pct) 1 1 1 1

Router#show platform resources
**State Acronym: H - Healthy, W - Warning, C - Critical
Resource Usage Max Warning Critical State
----------------------------------------------------------------------------------------------------
RP0 (ok, active) W
Control Processor 12.50% 100% 90% 95% H
DRAM 3527MB(93%) 3781MB 90% 95% W
ESP0(ok, active) H
QFP H
DRAM 192115KB(73%) 262144KB 80% 90% H
IRAM 426KB(20%) 2048KB 80% 90% H
CPU Utilization 1.00% 100% 90% 95% H

Router#show platform hardware qfp active infrastructure exmem statistics
QFP exmem statistics

Type: Name: DRAM, QFP: 0
Total: 268435456
InUse: 196725760
Free: 71709696
Lowest free water mark: 55018496
Type: Name: IRAM, QFP: 0
Total: 2097152
InUse: 436224
Free: 1660928
Lowest free water mark: 1660928
Type: Name: SRAM, QFP: 0
Total: 0
InUse: 0
Free: 0
Lowest free water mark: 0

 

Router#sh proc cpu his

333333333332222222222333333333322222222222222222222444442222
100
90
80
70
60
50
40
30
20
10
0....5....1....1....2....2....3....3....4....4....5....5....6
0 5 0 5 0 5 0 5 0 5 0
CPU% per second (last 60 seconds)


444544444443343344433395333343943443434453443377777777887777
100
90
80
70
60
50
40
30
20
10 * ** * * ############
0....5....1....1....2....2....3....3....4....4....5....5....6
0 5 0 5 0 5 0 5 0 5 0
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%

222 1 2 2
838666679777765565555566666666669877365676355556551667867865655465555556
100
90
80
70
60
50
40
30 * *
20 *** * *
10 *************************************************************** ******
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7..
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%

10 Replies 10

Ramblin Tech
Spotlight
Spotlight

Your control-plane route processor (RP0) has a max allocation of 3781MB of DRAM and will throw a warning alarm when 90% has been allocated (critical alarm at 95%). Output shows current allocation of 3527MB or 93% of max, hence the alarm.

Most efficient way to get to root cause of memory consumption issue is a TAC case, if you have a support contract. Otherwise, you can poke around with various "show memory" or "show proc memory" commands to try to see which IOS process is consuming the DRAM. Whether or not you can do something about the memory consumption with a config change is another question.

As 16.6.7 is nearly 4 years old, it is possible that there is a memory leak that has since been identified and addressed in a later release. An IOS upgrade might be in your future and may be a good idea regardless of this particular memory issue, if you have an unacceptable exposure to a number of risks that this release might be vulnerable to:

https://www.cisco.com/c/en/us/support/routers/4000-series-integrated-services-routers-isr/products-security-advisories-list.html

 

Disclaimer: I am long in CSCO

Leo Laohoo
Hall of Fame
Hall of Fame

Data Plane memory utilization is very high.  

Post the complete output to the command "sh platform software status con brief".

Wait, I think I know what this is.  

Look at the logs of the routers, is the logs getting spammed with CALL-HOME & SMAR LIC messages?  If it is, then this could be FN - 72323 - Cisco IOS XE Software: QuoVadis Root CA 2 Decommission Might Affect Smart Licensing, Smart Call Home, and Other Functionality.

Jagvella
Level 1
Level 1

 

No there are no logs related to call-home msgs.

Here is the sh platform output:

Router#sh platform software status con brief
Load Average
Slot Status 1-Min 5-Min 15-Min
RP0 Healthy 1.53 1.49 1.56

Memory (kB)
Slot Status Total Used (Pct) Free (Pct) Committed (Pct)
RP0 Warning 3871952 3614220 (93%) 257732 ( 7%) 3454796 (89%)

CPU Utilization
Slot CPU User System Nice Idle IRQ SIRQ IOwait
RP0 0 0.89 0.69 0.00 98.40 0.00 0.00 0.00
1 10.00 0.90 0.00 89.10 0.00 0.00 0.00
2 2.40 1.20 0.00 96.40 0.00 0.00 0.00
3 2.39 1.29 0.00 96.30 0.00 0.00 0.00
4 1.90 1.60 0.00 96.50 0.00 0.00 0.00
5 1.70 1.70 0.00 96.60 0.00 0.00 0.00
6 58.50 3.90 0.00 37.60 0.00 0.00 0.00
7 57.24 24.87 0.00 17.88 0.00 0.00 0.00


@Jagvella wrote:

RP0 Warning 3871952 3614220 (93%)


Control-plane memory utilization is at critical level. 

Post the complete output to the following commands: 

sh crypto pki trustpool | i cn
sh processes memory platform sorted location rp active  

 

 

Router#sh crypto pki trustpool | i cn
cn=QuoVadis Root CA 2
cn=QuoVadis Root CA 2
cn=VeriSign Class 3 Public Primary Certification Authority - G5
cn=VeriSign Class 3 Public Primary Certification Authority - G5
cn=Cisco RXC-R2
cn=Cisco RXC-R2
cn=Cisco Root CA 2048
cn=ACT2 SUDI CA
cn=Cisco Licensing Root CA
cn=Cisco Licensing Root CA
cn=Licensing Root - DEV
cn=Licensing Root - DEV
cn=Cisco Root CA M2
cn=Cisco Manufacturing CA SHA2
cn=Cisco Root CA M2
cn=Cisco Root CA M2
cn=Cisco Root CA 2048
cn=Cisco Manufacturing CA
cn=Cisco Root CA 2048
cn=Cisco Root CA 2048
cn=Cisco Root CA M1
cn=Cisco Root CA M1

Router#sh processes memory platform sorted location rp active
System memory: 3871952K total, 3610252K used, 261700K free,
Lowest: 261700K
Pid Text Data Stack Dynamic RSS Total Name
--------------------------------------------------------------------------------
4834 251552 1021252 136 124 1021252 3431352 linux_iosd-imag
14442 11895 251924 136 36924 251924 2262808 fman_fp_image
11348 89 242268 132 36776 242268 1791484 cpp_cp_svr
30542 7095 112140 136 2264 112140 1282316 fman_rp
11035 18227 681084 136 400 681084 1173604 qfp-ucode-utah
7336 68 148840 132 17212 148840 1147972 cpp_sp_svr
5732 151 121524 132 2988 121524 1121236 cpp_driver
6596 363 104908 136 3604 104908 1110892 cpp_ha_top_leve
26416 230 63204 136 3588 63204 1037272 bsm
27850 202 81132 136 7124 81132 955732 cli_agent
4311 231 42924 132 3772 42924 849232 cman_fp
32355 1770 151736 136 113092 151736 841052 confd
22596 472 54020 136 4236 54020 768560 iomd
22984 472 54400 136 4188 54400 768512 iomd
4602 151 79412 136 268 79412 634708 smd
28337 97 99552 136 3556 99552 620736 dbm
11814 98 76268 136 144 76268 572796 ndbmand
3329 460 60108 136 29416 60108 495128 smand
27765 1080 49660 136 4488 49660 409612 cmand
13183 624 28476 136 1560 28476 347432 cmcc
14564 624 27360 136 1416 27360 347288 cmcc
12449 624 28220 136 1416 28220 347288 cmcc
8888 498 10160 132 132 10160 270960 libvirtd
735 36 19140 136 1424 19140 265192 psd
30070 326 20552 136 2312 20552 251900 hman
31467 113 17984 136 1664 17984 247724 lman
14430 326 16260 136 1144 16260 240224 hman
14114 326 16068 136 1144 16068 240224 hman
13583 326 16072 136 1144 16072 240224 hman
10020 326 16280 132 1144 16280 240220 hman
21416 74 16496 136 1092 16496 226176 dmiauthd
5254 137 19784 136 4224 19784 211540 syncfd
7684 110 17468 136 2628 17468 209260 nesd
16019 82 15944 136 1280 15944 207748 vtyserverutild
465 39 15076 136 2292 15076 200212 plogd
18496 39 14368 136 1668 14368 199588 plogd
18247 39 14356 136 1668 14356 199588 plogd
17117 39 14748 136 1668 14748 199588 plogd
9327 39 14372 132 1668 14372 199584 plogd
22655 52 10676 136 596 10676 195996 ngiolite
8430 26 10824 136 1464 10824 176620 tams_proc
26414 236 12048 136 168 12048 172944 ncsshd
7582 72 10360 136 872 10360 171824 tamd_proc
6705 16 9236 136 688 9236 165192 tam_svcs_esg_cf
24119 47 6424 136 260 6424 135032 ncsshd_bp
32278 9 3256 136 260 3256 64360 rotee
26297 9 3340 136 260 3340 63976 rotee
25931 9 3468 136 260 3468 63976 rotee
23650 9 3332 136 260 3332 63976 rotee
17907 9 3288 136 260 3288 63976 rotee
10828 9 3376 136 260 3376 63976 rotee
25097 9 3364 132 260 3364 63972 rotee
24965 9 3452 132 260 3452 63972 rotee
9058 9 3340 132 260 3340 63972 rotee
8525 9 3324 132 260 3324 63972 rotee
7259 9 3276 132 260 3276 63972 rotee
1370 9 3276 132 260 3276 63972 rotee
32463 9 3332 136 260 3332 63856 rotee
31985 9 3324 136 260 3324 63856 rotee
31765 9 3240 136 260 3240 63856 rotee
31402 9 3432 136 260 3432 63856 rotee
30954 9 3332 136 260 3332 63856 rotee
30242 9 3328 136 260 3328 63856 rotee
29556 9 3324 136 260 3324 63856 rotee
29321 9 3328 136 260 3328 63856 rotee
28461 9 3320 136 260 3320 63856 rotee
27951 9 3340 136 260 3340 63856 rotee
27755 9 3456 136 260 3456 63856 rotee
27336 9 3264 136 260 3264 63856 rotee
26739 9 3268 136 260 3268 63856 rotee
25983 9 3332 136 260 3332 63856 rotee
25545 9 3232 136 260 3232 63856 rotee
22138 9 3268 136 260 3268 63856 rotee
20936 9 3312 136 260 3312 63856 rotee
18400 9 3292 136 260 3292 63856 rotee
17303 9 3328 136 260 3328 63856 rotee
17183 9 3252 136 260 3252 63856 rotee
17068 9 3336 136 260 3336 63856 rotee
15578 9 3332 136 260 3332 63856 rotee
13128 9 3340 136 260 3340 63856 rotee
12029 9 3240 136 260 3240 63856 rotee
10634 9 3240 136 260 3240 63856 rotee
9912 9 3440 136 260 3440 63856 rotee
9179 9 3336 136 260 3336 63856 rotee
7055 9 3332 136 260 3332 63856 rotee
6612 9 3324 136 260 3324 63856 rotee
5632 9 3476 136 260 3476 63856 rotee
4939 9 3264 136 260 3264 63856 rotee
4655 9 3400 136 260 3400 63856 rotee
3462 9 3448 136 260 3448 63856 rotee
2839 9 3344 136 260 3344 63856 rotee
2438 9 3320 136 260 3320 63856 rotee
1722 9 3344 136 260 3344 63856 rotee
903 9 3432 136 260 3432 63856 rotee
24660 9 3472 132 260 3472 63852 rotee
22417 9 3240 132 260 3240 63852 rotee
21566 9 3332 132 260 3332 63852 rotee
17669 9 3244 132 260 3244 63852 rotee
17285 9 3324 132 260 3324 63852 rotee
16224 9 3312 132 260 3312 63852 rotee
13550 9 3264 132 260 3264 63852 rotee
12885 9 3440 132 260 3440 63852 rotee
12701 9 3320 132 260 3320 63852 rotee
11991 9 3344 132 260 3344 63852 rotee
11809 9 3320 132 260 3320 63852 rotee
10870 9 3088 132 260 3088 63852 rotee
10673 9 3284 132 260 3284 63852 rotee
10604 9 3408 132 260 3408 63852 rotee
9959 9 3264 132 260 3264 63852 rotee
9916 9 3096 132 260 3096 63852 rotee
9199 9 3176 132 260 3176 63852 rotee
8163 9 3276 132 260 3276 63852 rotee
7503 9 3340 132 260 3340 63852 rotee
6723 9 3232 132 260 3232 63852 rotee
5896 9 3480 132 260 3480 63852 rotee
5281 9 3236 132 260 3236 63852 rotee
4645 9 3372 132 260 3372 63852 rotee
3989 9 3340 132 260 3340 63852 rotee
3268 9 3436 132 260 3436 63852 rotee
8858 572 3480 132 132 3480 45748 virtlogd
1 1377 5804 132 1396 5804 39592 systemd
208 398 3424 132 264 3424 37252 systemd-udevd
196 302 4528 132 132 4528 30208 systemd-journal
8849 46 2292 132 132 2292 29760 rpcbind
11305 2631 12176 132 1020 12176 25688 dhcpd
32592 880 7284 136 4540 7284 22588 periodic.sh
24097 880 6372 132 3596 6372 21640 rollback_timer.
24978 880 5916 132 3088 5916 21132 pvp.sh
24851 880 5652 132 2868 5652 20912 psvp.sh
8860 880 5564 132 2684 5564 20728 droputil.sh
8016 880 5348 132 2604 5348 20648 pvp.sh
7028 880 5396 132 2604 5396 20648 pvp.sh
5961 880 5504 132 2604 5504 20648 pvp.sh
882 880 5508 132 2604 5508 20648 pvp.sh
8834 880 5500 132 2592 5500 20636 reflector.sh
13175 880 4828 136 2032 4828 20080 btrace_rotate.s
11877 880 4828 136 2032 4828 20080 btrace_rotate.s
11078 880 4752 136 2032 4752 20080 btrace_rotate.s
3850 880 4732 136 2032 4732 20080 btrace_rotate.s
26848 880 4896 136 2028 4896 20076 btrace_rotate.s
31097 880 4836 136 1988 4836 20036 pman.sh
2328 880 4836 136 1988 4836 20036 pman.sh
29293 880 4836 132 1984 4836 20028 pman.sh
26885 880 4780 132 1984 4780 20028 pman.sh
26213 880 4680 132 1984 4680 20028 pman.sh
25325 880 4700 132 1984 4700 20028 pman.sh
25536 880 4820 136 1976 4820 20024 pman.sh
16829 880 4820 136 1976 4820 20024 pman.sh
4365 880 4748 136 1976 4748 20024 pman.sh
32025 880 4828 136 1972 4828 20020 pman.sh
31492 880 4684 136 1972 4684 20020 pman.sh
27500 880 4788 136 1972 4788 20020 pman.sh
25695 880 4784 136 1972 4784 20020 pman.sh
20691 880 4776 136 1972 4776 20020 pman.sh
10513 880 4824 136 1972 4824 20020 pman.sh
6741 880 4720 136 1972 4720 20020 pman.sh
1212 880 4728 136 1972 4728 20020 pman.sh
29891 880 4828 136 1968 4828 20016 pman.sh
25849 880 4684 132 1972 4684 20016 pman.sh
15295 880 4772 136 1968 4772 20016 pman.sh
4753 880 4812 132 1972 4812 20016 pman.sh
31670 880 4708 132 1968 4708 20012 pman.sh
30707 880 4824 132 1968 4824 20012 pman.sh
23405 880 4716 136 1964 4716 20012 pman.sh
5521 880 4720 132 1968 4720 20012 pman.sh
3990 880 4832 132 1968 4832 20012 pman.sh
1739 880 4780 132 1968 4780 20012 pman.sh
27960 880 4680 132 1964 4680 20008 pman.sh
8137 880 4636 132 1840 4636 19884 pman.sh
1892 880 4560 132 1840 4560 19884 pman.sh
17478 880 4604 136 1732 4604 19780 opdatamgrd.sh
10224 880 4532 136 1716 4532 19764 pman.sh
9240 880 4664 136 1716 4664 19764 pman.sh
8277 880 4624 136 1716 4624 19764 pman.sh
21798 880 4640 132 1716 4640 19760 pman.sh
21500 880 4528 132 1716 4528 19760 pman.sh
20978 880 4580 132 1716 4580 19760 pman.sh
16721 880 4648 132 1716 4648 19760 pman.sh
16310 880 4548 132 1716 4548 19760 pman.sh
15406 880 4548 132 1716 4548 19760 pman.sh
12341 880 4544 132 1716 4544 19760 pman.sh
11539 880 4548 132 1716 4548 19760 pman.sh
11282 880 4616 132 1716 4616 19760 pman.sh
10412 880 4580 132 1716 4580 19760 pman.sh
10391 880 4656 132 1716 4656 19760 pman.sh
9200 880 4548 132 1716 4548 19760 pman.sh
7111 880 4548 132 1716 4548 19760 pman.sh
6378 880 4680 132 1716 4680 19760 pman.sh
5659 880 4628 132 1716 4628 19760 pman.sh
5004 880 4688 132 1716 4688 19760 pman.sh
4344 880 4684 132 1716 4684 19760 pman.sh
3624 880 4664 132 1716 4664 19760 pman.sh
2982 880 4600 132 1716 4600 19760 pman.sh
2349 880 4636 132 1716 4636 19760 pman.sh
10307 880 4340 132 1556 4340 19600 iptbl.sh
31885 880 4424 136 1536 4424 19584 confd-startup.s
28939 880 4324 136 1440 4324 19488 flash_check.sh
8843 880 3520 132 684 3520 18728 oom.sh
18899 880 3492 136 648 3492 18696 cdb_update_q.sh
5187 880 2904 136 260 2904 18308 sort_files_by_i
24098 880 2956 132 132 2956 18176 auxinit.sh
8857 880 2804 132 132 2804 18176 libvirtd.sh
12466 37 1908 136 132 1908 18116 inet_gethost
8937 95 1856 132 132 1856 17892 rpc.mountd
24101 151 1968 132 132 1968 15200 xinetd
24099 151 1980 132 132 1980 15200 xinetd
12675 151 1996 132 132 1996 15200 xinetd
12670 151 2016 132 132 2016 15200 xinetd
8881 151 1992 132 132 1992 15200 xinetd
12458 37 1600 136 132 1600 11784 inet_gethost
8854 48 1684 132 132 1684 9784 sntp
8835 27 5252 132 132 5252 8368 klogd
4496 24 1536 136 132 1536 6716 inotifywait
4490 24 1476 136 132 1476 6716 inotifywait
4439 24 1508 136 132 1508 6716 inotifywait
3190 24 1528 136 132 1528 6716 inotifywait
1732 24 1460 136 132 1460 6716 inotifywait
25224 24 1596 132 132 1596 6712 inotifywait
24973 24 1500 132 132 1500 6712 inotifywait
24674 24 1548 132 132 1548 6712 inotifywait
10643 24 1576 132 132 1576 6712 inotifywait
10164 24 1608 132 132 1608 6712 inotifywait
9997 24 1704 132 132 1704 6712 inotifywait
9188 24 1552 132 132 1552 6712 inotifywait
8642 24 1552 132 132 1552 6712 inotifywait
7480 24 1612 132 132 1612 6712 inotifywait
1524 24 1592 132 132 1592 6712 inotifywait
5189 34 1160 136 132 1160 4560 sleep
5153 34 1312 136 132 1312 4560 sleep
5055 34 1240 136 132 1240 4560 sleep
4216 34 1312 136 132 1312 4560 sleep
3489 34 1308 136 132 1308 4560 sleep
9222 34 1136 132 132 1136 4552 sleep


@Jagvella wrote:

Router#sh crypto pki trustpool | i cn
cn=QuoVadis Root CA 2
cn=QuoVadis Root CA 2


You are hitting multiple memory leaks caused by multiple bugs:

First two lines is already "QuoVadis".  This confirms FN - 72323 - Cisco IOS XE Software: QuoVadis Root CA 2 Decommission Might Affect Smart Licensing, Smart Call Home, and Other Functionality.


@Jagvella wrote:

4834 251552 1021252 136 124 1021252 3431352 linux_iosd-imag


Next one is this memory leak of the "linux_iosd-imag" which is caused by telemetry (SNMP &/or DNAC).  I suspect this is either CSCwd07516 or the very popular CSCvh89372.


@Jagvella wrote:

%PLATFORM-4-ELEMENT_WARNING: SIP1: smand: RP/0: Used Memory value 93% exceeds warning level 88%


Lastly, this could be another popular bug CSCvd45973.

Looking over Leo's references, QuoVadis appears to apply to 16.6.7, but none of the others note they do.  (BTW, you can fix the QuoVadis issue w/o doing a IOS update.)

Reviewing the last Everest release (16.6.9), there was a resolved memory leak.  (16.6.9 also last release that doesn't require smart licensing.)

Joseph W. Doherty
Hall of Fame
Hall of Fame

I may have missed anyone else suggesting this, but if this is a production router, I would suggest you consider a reload ASAP!  (Free memory depletion leads to "bad things".)

With such a reload, if you have a maintenance contract, you might consider moving to a newer release too.

Often moving to a newer release would be a TAC recommendation, although their release recommendation might differ from what you might select.  (Generally, moving to a newer release on the same software train [bug fixes] is fairly safe.)

Did reload but it just comes back up to same high percentage immediately.

First message, after reload, same or lower usage percentage compared to last message before reload?

Is usage percentage growing, and if so, by how much over what time periods?

Without an external syslog server, you might be unable to answer, but did messages also start immediately after prior boot?  Whether they did or not, again history of memory usage over time?

Review Cisco Networking for a $25 gift card