What is Routekernel Process for the memory process

[R T]

I saw much DEAD memory is holding by Routekernel Process on my C4500. And recently received a lot of memory error. Do you have any idea for this issue? Should I reboot the router as it has been running over 1.5 year.

<LOG>

Dec 22 15:06:45.382 GMT: %SYS-2-MALLOCFAIL: Memory allocation of 4112 bytes failed from 0x103A5C54, alignment 0
Pool: Processor Free: 21235140 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "Spanning Tree", ipl= 0, pid= 117
-Traceback= 10DDF50C 10DDFC58 118BECD4 118C11F4 118C1458 11B2ACA0 103A5C58 104EF3E8 1071D8A0 1071EFD8 1176FA70 1176FD94 1176FF08 11771770 117776EC 1174BCC8

#sh memory dead totals
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 13F371B4 470584904 449324764 21260140 19027304 705948
Dead Proc Summary for: Processor

PC Total Name
0x1268B1CC 4391 Routekernel Process
0x11FA216C 3360 Routekernel Process
0x124E22A4 3176 Routekernel Process
0x104335B8 678 Filesys Init Process
0x11FB491C 660 Routekernel Process
0x11FB4968 660 Routekernel Process
0x12532AD0 492 Routekernel Process
0x12532AEC 492 Routekernel Process
0x11DF365C 461 Routekernel Process
0x124FFBD8 461 Routekernel Process
0x1042E550 447 Filesys Init Process
0x11FA7EB4 421 Routekernel Process
0x11FB49C8 420 Routekernel Process
0x11DEAB58 397 Routekernel Process
0x124E227C 397 Routekernel Process
0x124E7060 397 Routekernel Process
0x11DF35F0 397 Routekernel Process
0x1125D974 328 Virtual Exec
0x1026E510 328 Virtual Exec
...

[acampbell]

Hi RT,

I think you should be looking at a spanning tree issue.
The route kernel "dead memory" is probably a consequence of a spanning tree loop
or something similar.

This wee link explain in a few sentences IOS, IOS/XE and kernels:-
https://books.google.co.uk/books?id=cevlBQAAQBAJ&pg=PA4&lpg=PA4&dq=what+is+the+cisco+route+kernel&source=bl&ots=H7pvbRGFtL&sig=-MzVKkQ5IqCbU8Jp0mhIhQSU4O8&hl=en&sa=X&ved=0ahUKEwjb2ajN0PHJAhWCgBoKHUWQBxAQ6AEIOzAF#v=onepage&q=what%20is%20the%20cisco%20route%20kernel&f=false

Start an investigation in to your spanning tree situation.
Where is the route bridge per vlan ?
Has any new device been added that is claiming to be the route bridge ?
etc etc.

Regards
Alex

[R T]

Hi Alex,

Thanks. But I think it is not related to spanning tree. Because sometimes we got an error for SSH Process, sometimes it's Virtual Exec. The issue is not happening for a specific process.

Best regards,

RT

<LOG>

Dec 18 18:12:52.940 GMT: %SYS-2-MALLOCFAIL: Memory allocation of 4112 bytes failed from 0x103A5C54, alignment 0
Pool: Processor Free: 20786904 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "SpanTree Helper", ipl= 0, pid= 220
-Traceback= 10DDF50C 10DDFC58 118BECD4 118C11F4 118C1458 11B2ACA0 103A5C58 104EF3E8 1071D8A0 1071EFD8 1176FA70 1176FC58 1176FF78 117795A4 11766FB0 11754E28
Dec 21 23:40:14.674 GMT: %SYS-2-MALLOCFAIL: Memory allocation of 123640 bytes failed from 0x117CF880, alignment 0
Pool: Processor Free: 21366276 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "SSH Process", ipl= 0, pid= 70
-Traceback= 10DDF50C 10DDFC58 118BECD4 118C11F4 118C1458 11B2ACA0 117CF884 117CC678 117D55CC 1023E99C 1023EA20 10D63188 10DE55E0 10DE62B8 10D90F8C 10D912C8
Dec 22 03:46:28.551 GMT: %SYS-2-MALLOCFAIL: Memory allocation of 395648 bytes failed from 0x117D085C, alignment 0
Pool: Processor Free: 21620420 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "Virtual Exec", ipl= 0, pid= 269
-Traceback= 10DDF50C 10DDFC58 118BECD4 118C11F4 118C1458 11B2ACA0 117D0860 117CF830 117CC678 117D55CC 1023E99C 1023EA20 10D63188 10DE55E0 10DE62B8 10D90F8C
Dec 22 15:06:45.382 GMT: %SYS-2-MALLOCFAIL: Memory allocation of 4112 bytes failed from 0x103A5C54, alignment 0
Pool: Processor Free: 21235140 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "Spanning Tree", ipl= 0, pid= 117
-Traceback= 10DDF50C 10DDFC58 118BECD4 118C11F4 118C1458 11B2ACA0 103A5C58 104EF3E8 1071D8A0 1071EFD8 1176FA70 1176FD94 1176FF08 11771770 117776EC 1174BCC8

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages wha2tsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Discovering the cause of your dead memory, would likely require working with TAC.  It might be due to some bug in your IOS, or just the way you use your 4500.  (I.e. Cisco IOSs don't garbage collect, so certain "normal" usage can run you out of memory, especially if you don't have much free memory to begin with.)

Rebooting your 4500 will "clean the slate", and may allow you to run for some time without memory issues.  (BTW, I've seen some Cisco devices in continuous operation, w/o any issues, for over 15 years.  So, constant scheduled rebooting, isn't a general requirement, but for certain issues, it can be a work around.  If you do need to schedule reoccurring reboots, some devices now support a warm reload feature, which reboots much faster as the IOS isn't reloaded, just working/variable memory is reset.)