%FIB-3-NOMEM - IOS bug or ARP cache problem

[johnlloyd_13]

hi all,

today, one of our 1841 had a slow connectivity and packet drop issue. my telnet session was very slow and found syslogs below.

there was no high CPU but i've observed a high memory usage. i've also observed ARP table output was very long (from 1.0.0.0/24 up to i don't know what).

i've found a similar issue found on link below but i'm not sure whether it's related and can also affect an 1841 router. please advise if my IOS hit a bug or was it just an ARP cache/memory or CEF/FIB problem? thanks in advance!

https://supportforums.cisco.com/thread/204558

#sh ver

Cisco IOS Software, 1841 Software (C1841-IPBASE-M), Version 12.4(15)T10, RELEASE SOFTWARE (fc3)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2009 by Cisco Systems, Inc.

Compiled Mon 14-Sep-09 12:41 by prod_rel_team

*Nov 11 20:56:39.758 SGT: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes

failed from 0x6028FAE0, alignment 16

*Nov 11 20:56:39.762 SGT: %FIB-3-NOMEM: Malloc Failure, disabling CEF, 

-Traceback= 0x60AF1A14 0x60CAA824 0x60CAFC78 0x60CB6414 0x6088E43C 0x60CDFED0

0x60E96140 0x616BBBC8 0x616BBBAC

#sh proc cpu sort

CPU utilization for five seconds: 1%/0%; one minute: 2%; five minutes: 1%

#sh mem stat his

------------------ History of Processor Mempool ------------------

<SNIP>  10:46:14 AM Monday Nov 12 2012 SGT

    999999999999999999999999999999999999999999999999999999999999

    333333333333333333333333333333333333333333333333333333333333

100                                                            

90 ************************************************************

80 ************************************************************

70 ************************************************************

60 ************************************************************

50 ************************************************************

40 ************************************************************

30 ************************************************************

20 ************************************************************

10 ************************************************************

   0....5....1....1....2....2....3....3....4....4....5....5....6

             0    5    0    5    0    5    0    5    0    5    0

               Free memory per second (last 60 seconds)

[skarthic]

Hi John,

Your FIB-NO-MEM message is due to the low memory condition. We will have to concentrate on why the router is is having low memory.

Please collect the following outputs ASAP

show process memory sorted

show memory alloc totals

Thanks,

Karthic

[skarthic]

If you have a huge ARP cache, it might have to do with static routes pointing out of the interface rather than using the next hop ip.

Can you also post

show run | in ip route

Thanks,

Karthic

[johnlloyd_13]

hi karthic,

i will post the show command output if the issue happens again. i've observed the connection has been stable (telnet also became faster) after our client reluctantly rebooted the router, although the ARP table is growing bigger again.

yes, you're right on the static route issue as well. the network engineer who setup the router used an FE port as exit interface instead of the ISP's next hop IP. this one has been resolved earlier.

so this ain't due to an IOS bug (i hope not)?

[skarthic · Accepted solution]

Yes, It should be due to the static route. Just check the output of 'show memory stat' once in 3 days and see if the memory usage is stable.

If nothing seems alarming after a week or so, your issue is resolved.

Thanks,

Karthic

[Raju Sekharan]

Hi

Please paste the following output

Show proc mem sort

Do you have any static default route pointing to any broadcast interfface?

attach your configs too

Thank you

Raju