Hi Vinit,

purevbatya · ‎01-25-2016

Hi, I've ASR1001 router, and it keeps showing this error shown below.

*Jan 25 09:11:10.279: %PLATFORM-3-ELEMENT_CRITICAL: SIP0: smand: RP/0: Committed Memory value 111% exceeds critical level 100%

so i've searched from google and found almost nothing.

Is there any1 know how to solve it?

Here is some info that might need.

#show platform software status control-processor

RP0: online, statistics updated 7 seconds ago

Load Average: healthy

1-Min: 0.00, status: healthy, under 8.00

5-Min: 0.01, status: healthy, under 8.00

15-Min: 0.00, status: healthy, under 10.00

Memory (kb): critical

Total: 3975012

Used: 3935964 (99%)

Free: 39048 (1%)

Committed: 4392508 (111%), status: critical, over 100%

Per-core Statistics

CPU0: CPU Utilization (percentage of time spent)

User: 3.20, System: 6.80, Nice: 0.00, Idle: 89.98

IRQ: 0.00, SIRQ: 0.00, IOwait: 0.00

CPU1: CPU Utilization (percentage of time spent)

User: 1.50, System: 0.90, Nice: 0.00, Idle: 97.59

IRQ: 0.00, SIRQ: 0.00, IOwait: 0.00

#show memory statistics

Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)

Processor 7F06FC497010 1129267976 392478128 736789848 463192776 488455092

lsmpi_io 7F06FBC941A8 6295128 6294304 824 824 412

#show version

Cisco IOS XE Software, Version 03.12.00.S - Standard Support Release

Cisco IOS Software, ASR1000 Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 15.4(2)S, RELEASE SOFTWARE (fc2)

Technical Support: http://www.cisco.com/techsupport

Compiled Wed 26-Mar-14 21:28 by mcpre

purevbatya · ‎01-25-2016

any1?

darylturner · ‎07-27-2017

Was this problem ever resolved? We're having a very similar issue.

Vinit Jain · ‎01-25-2016

Hello,

Could you please share the output of:

- show platform software status control-processor brief

- Show platform software process list r0 sort memory

- Show platform software process list 0 sort memory

were there any changes made before you started seeing this problem?

Regards

Vinit

Thanks
--Vinit

purevbatya · ‎01-25-2016

Hi Vinit,

There were no big changes or i don't remember.

All i remember is the committed memory was keep increasing.

Pls check attached file. Output was too long to post so i posting as attachment.

PS: we enabled WCCP a while ago, but now it's disabled.

purevbatya · ‎01-25-2016

Also it might help:

ROUTER#monitor platform software process r0
top - 04:52:21 up 325 days, 6:42, 0 users, load average: 0.00, 0.00, 0.00
Tasks: 264 total, 2 running, 262 sleeping, 0 stopped, 0 zombie
Cpu(s): 1.7%us, 3.3%sy, 0.0%ni, 95.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 3975012k total, 3932140k used, 42872k free, 22180k buffers
Swap: 0k total, 0k used, 0k free, 803044k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
29546 root 20 0 2622m 1.4g 16m S 0 36.0 471:50.79 cpp_cp_svr
24361 root 20 0 2535m 740m 58m S 2 19.1 8303:29 linux_iosd-imag
30404 root 20 0 1604m 232m 22m S 7 6.0 32808:10 fman_fp_image
21706 root 20 0 983m 212m 205m S 0 5.5 57:21.42 fman_rp
25736 root 20 0 122m 115m 4644 S 0 3.0 181:49.40 smand
30565 root 20 0 709m 53m 4572 S 0 1.4 40:44.33 cpp_sp_svr
29481 root 20 0 821m 38m 2288 S 0 1.0 155:02.45 cpp_driver
5879 root 20 0 195m 29m 4704 R 0 0.8 2192:53 mcpcc-lc-ms
30018 root 20 0 603m 28m 2312 S 0 0.7 170:42.14 cpp_ha_top_leve
29976 root 20 0 753m 27m 2312 S 0 0.7 39:01.13 cpp_stats_svr
29301 root 20 0 580m 25m 1632 S 0 0.7 19:37.43 cpp_cdm_svr
22160 root 20 0 25048 12m 11m S 0 0.3 72:58.47 imand
20688 root 20 0 35116 11m 6540 S 0 0.3 256:40.08 cmand
28814 root 20 0 388m 7836 2436 S 0 0.2 125:08.16 cman_fp
22470 root 20 0 15456 6140 3692 S 3 0.2 8276:01 hman
20782 root 20 0 20372 5232 1696 S 0 0.1 163:28.35 emd
31773 root 20 0 28492 4860 2428 S 0 0.1 103:19.29 cmcc
32592 root 20 0 13332 4652 3004 S 0 0.1 132:33.73 hman
32711 root 20 0 16368 4528 2808 S 0 0.1 85:57.55 imccd
32305 root 20 0 19716 4448 1632 S 0 0.1 148:23.65 emd
24435 root 20 0 5720 4444 1044 S 0 0.1 39:49.68 periodic.sh
30140 root 20 0 19728 4440 1616 S 0 0.1 149:07.65 emd
24634 root 20 0 12880 4024 1668 S 0 0.1 37:34.90 plogd
24175 root 20 0 14392 3848 1840 S 0 0.1 41:50.31 lman
2818 root 20 0 12404 3552 1580 S 0 0.1 35:20.73 plogd
24812 root 20 0 15496 3508 1924 S 0 0.1 36:13.45 psd
ROUTER#

Vinit Jain · ‎01-25-2016

Thanks, Interestingly, i dont see the output for the SIP0 card or F0 card in the show plat soft status control brief command. Are they up and running fine?

Could you please also share show platform and show log command in a file.

Regards

Vinit

Thanks
--Vinit

Vinit Jain · ‎01-25-2016

Clearly the RP has run out of memory and thats the reason you are seeing this error. the most memory is being consumed by cpp_cp_svr, linux_iosd-imag and fman_fp_image and other sub-processes.

Could you please share the below outputs: Please provide 2-3 iterations of the below output with a time gap of 15-20 minutes.

show platform software memory forwarding-manager FP active brief
show platform software memory qfp-control-process qfp active brief

I would also like to know what all features are you running on the router:

IPSec

PFR

NBAR

QoS

Netflow

Please let me know which of the above features are being used?

It clearly seems to be a memory leak condition and i have a suspected defect - CSCut21885, but once i have output and information, it can be confirmed.

You may take a look at the defect as well at the Bug toolkit.

Regards

Vinit

Thanks
--Vinit

purevbatya · ‎01-26-2016

Hi Vinit,

thank you for your reply.

IPSec - no,

PFR - no,

NBAR - no,

QoS - yes,

Netflow - yes

and atm i'm collecting output of..

show platform software memory forwarding-manager FP active brief
show platform software memory qfp-control-process qfp active brief

will send you soon.

purevbatya · ‎01-26-2016

Here comes the output file.

purevbatya · ‎01-27-2016

any update?

bobbydazzler · ‎11-27-2017

Hello,

Have you found out what was causing the pb at the end?

I have a similar pb (CPU and memory maxing out) few days after enabling WCCP (with GRE redirect/return).

I have disabled it but still maxing out, I will try a reload over night.

Looking forward to your reply.

Best regards

RP/0: Committed Memory value % exceeds critical level