OSPF down on N7K

[Amirkia Kalvandi]

Ive got an issue last week on N7K ios 6.2.12and suddenly OSPF down cause low memory condition 

./routing-sw/libs/timers/active_timer.c[927] for size 8196
2016 Jul 15 22:08:41 -B %OSPF-3-ATIMERS_ERROR: malloc failed in
heap_create
2016 Jul 15 22:08:41 -B %OSPF-3-ERROR: -Traceback: librsw.so+0xe1b85
librsw.so+0xd902e libospf.so+0x19f078 libmtrack.so+0x5437
libmtrack.so+0x9850
libutils.so+0x12c03 libospf.so+0x23a020 libutils.so+0xd093
libutils.so+0xedb7 libospf.so+0x232a32 libospf.so+0x24256b libospf.*
2016 Jul 15 22:08:41 -B %OSPF-1-LOWMEM: ospf-100 [4803] Low memory
condition: Shutting down the OSPF process. Please restart OSPF manually
2016 Jul 15 22:08:42 -B ospf[4803]: Warning! Malloc Failure in
../routing-sw/libs/timers/active_timer.c[927] for size 8196
2016 Jul 15 22:08:41 -B %OSPF-3-ATIMERS_ERROR: malloc failed in
heap_create
2016 Jul 15 22:08:41 -B %OSPF-3-ERROR: -Traceback: librsw.so+0xe1b85
librsw.so+0xd902e librsw.so+0xd8cd5 librsw.so+0xd9001 libospf.so+0x19f078
lib
mtrack.so+0x5437 libsdwrap.so+0x9796 librsw.so+0xe19e2 librsw.so+0xd902e
libospf.so+0x2425bd libospf.so+0xeedbb 0x8054817 0x807*
2016 Jul 15 22:08:41 -B %OSPF-4-SYSLOG_SL_MSG_WARNING: OSPF-1-LOWMEM:
message repeated 1 times in last 17418055 sec
2016 Jul 15 22:08:42 -B ospf[4803]: Warning! Malloc Failure in
../routing-sw/libs/timers/active_timer.c[927] for size 8196
2016 Jul 15 22:08:41 -B %OSPF-3-ATIMERS_ERROR: malloc failed in
heap_create
2016 Jul 15 22:08:41 -B %OSPF-3-ERROR: -Traceback: librsw.so+0xe1b85
librsw.so+0xd902e libospf.so+0x2425bd libospf.so+0xeedbb 0x8054817
0x8070704
0x807bb0a 0x807f740 libospf.so+0x973a2 librsw.so+0xaa81f
libpthread.so.0+0x6140 libc.so.6+0xcedee
2016 Jul 15 22:08:41 -B %OSPF-3-TLV_INTF: ospf-100 [4803] Failed to
convert to TLV for x.x.x.x (0x1)x.x.x.x (0x80006c80) (0x3333) (4)
2016 Jul 15 22:08:42 -B ospf[4803]: Warning! Malloc Failure in
../routing-sw/libs/timers/active_timer.c[927] for size 8196
2016 Jul 15 22:08:41 -B %OSPF-3-ATIMERS_ERROR: malloc failed in

[Georg Pauwen]

Hello,

this might or might not have anything to do with the errors in your log, but are you running OSPF v2 on all routers ? If not, mmake sure your routers support RFC 1583:

N7K#configure terminal
N7K(config)# feature ospf
N7K(config)# router ospf 1
N7K(config-router)# rfc1583compatibility
N7K(config-router)# vrf C
N7K(config-router-vrf)# rfc1583compatibility

[Richard Burts]

The original post seems to suggest that the N7K ran OK for a while and then developed this issue. Can you verify that this is the case? Or did it do this immediately after booting up?

The symptoms show that the N7K has a low memory issue. You might check to make sure that there is not some issue with installed memory. I think it more likely that this is the result of a memory leak (especially if the N7K has run for a while before encountering this issue). The immediate way to resolve the issue is to reboot the N7K. The longer term solution is to change the image that you run. If this N7K is covered by a maintenance contract it might be worth opening a case with Cisco TAC to verify that this is a memory leak issue.

HTH

Rick

[Amirkia Kalvandi]

Hi Rick,

its working for 2 years and OSPF gone suddenly and we have a plan to upgrade the FW but we are not sure after upgrading it will be fixed also our smart net have been finished a year ago .

thank you for your kind help and support

amirkia

[Richard Burts]

amirkia

Thanks for the additional information. if the N7K has been running two years and then develops the problem it seems quite likely that it is a memory leak. I would say that it is very likely that if you upgrade the software version that the problem will be fixed. Of course there is some possibility that you might upgrade from a version that has the memory leak to a different version that also has a memory leak but that possibility is not likely. You would need current smart net to get the image for the new version.

In the short term if you reboot the N7K I believe that things will be back to normal and you would have some time to arrange for smart net and for an upgrade to a new version of code.

HTH

Rick 

[Amirkia Kalvandi]

Dear Rick,

We have got the same error on both chassis and I don't think it's from ram 

thx

amirkia

[Amirkia Kalvandi]

Hi,

We are running OSPF v2 

tnx

amirkia