%SYS-2-MALLOCFAIL: + %TCP-2-INVALIDTCB + %SYS-3-MGDTIMER + Process= "OER Border Router",

[Sigeesh Sivadasan]

Hi ,

Cisco 2851  is running with IOS : c2800nm-advsecurityk9-mz.151-4.M3.bin. The router crashed several time  leaving below errors

%SYS-2-MALLOCFAIL: , %SYS-3-MGDTIMER: and %TCP-2-INVALIDTCB.

=================================================================================================

Sep 20 22:35:55 GMT: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x4280C88C, alignment 4

Pool: Processor  Free: 104032  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "OER Border Router", ipl= 0, pid= 306

-Traceback= 427E2450z 427FD054z 430E010Cz 430E0308z 43127578z 42F75C94z 42F68DB0z 42F7F4A4z 42F6D594z 42F94B7Cz 42F950ACz 42F95270z 42F6E2A0z 42F6E3F8z 42F6A02Cz 427B1E98z

Sep 20 22:35:55 GMT: %TCP-2-INVALIDTCB: Invalid TCB pointer: 0x486B5BBC -Process= "OER Border Router", ipl= 0, pid= 306

-Traceback= 41E2891Cz 41E30C88z 42F94664z 42F6DA68z 42F68D04z 42F68DBCz 42F7F4A4z 42F6D594z 42F94B7Cz 42F950ACz 42F95270z 42F6E2A0z 42F6E3F8z 42F6A02Cz 427B1E98z 427B1E7Cz

*Sep 20 22:35:55 GMT: %TCP-2-INVALIDTCB: Invalid TCB pointer: 0x486B5BBC -Process= "OER Border Router", ipl= 0, pid= 306

-Traceback= 41E2891Cz 41E30C88z 42F94664z 42F6D0D8z 42F6D680z 42F94B7Cz 42F950ACz 42F95270z 42F6E2A0z 42F6E3F8z 42F6A02Cz 427B1E98z 427B1E7Cz

*Sep 20 22:35:55 GMT: %SYS-3-MGDTIMER: Next timer has bad reverse linkage, timer = 488419E8. -Process= "OER Master Controller", ipl= 4, pid= 305

------------------ show process memory ------------------

Processor Pool Total:  153687072 Used:  111309836 Free:   42377236

       I/O Pool Total:   14680064 Used:    7177344 Free:    7502720

305   0   33140496    4207820   28942940     239918     239915 OER Master Contr

306   0  160723104   44539956   30977404     139097     139095 OER Border Route

130   0      24880       7240      21732     454108     454107 IP Input   

[skarthic · Accepted solution]

Hi Sigeesh,

It looks like your router is leaking memory probably due to some OER related process. Please review a couple of other crashfiles and check if all of them have " %SYS-2-MALLOCFAIL" error messages.

If yes, then continously monitor the output of 'show process memory sorted' output and see what process is leaking the memory. Btw, can you explain what is your OER design? Just want to make sure you are not storing a lot of info resulting in router going out of memory

Thanks.

PS: For memory/crash troubleshooting, its always better to check with TAC.

[Sigeesh Sivadasan]

yes, the  MALLOCFAIL was found in all the crash file, we upgrade the IOS... now it’s looks fine.. thanks for ur inputs.