2812 with 16-ESW CEF low memory issue.

[JohnTylerPearce]

According to my boss every 3 to 4 months he has to restart our 2821 with a 16-esw module installed because of a low memory issue dealing with

CEF. Here is the exact error message.

%% Low on memory; try again later

Jun  8 11:18:51.777: %FIB-2-FIBDOWN: CEF has been disabled due to a low memory condition. It can be re-enabled by configuring "ip cef [distributed]"
Jun  8 11:19:51.823: %FIB-2-FIBDOWN: CEF has been disabled due to a low memory condition. It can be re-enabled by configuring "ip cef [distributed]"
%% Low on memory; try again later

%% Low on memory; try again later

%% Low on memory; try again later

Jun  8 11:20:51.868: %FIB-2-FIBDOWN: CEF has been disabled due to a low memory condition. It can be re-enabled by configuring "ip cef [distributed]"
Jun  8 11:21:51.914: %FIB-2-FIBDOWN: CEF has been disabled due to a low memory condition. It can be re-enabled by configuring "ip cef [distributed

From what I can tell, I believe we have 256 megs DRAM on this router. I haven't done to much research yet, because I've been super busy. We're not running

BGP or anything. We are running EIGRP but it's honestly only one peer with less than 30 routes. I'm going to do some more search tomorrow, but was just

wondering if anyone had any ideas. Obviously it appears as if the CEF tables FIB/AT tare getting to large and hitting a watermark causing it to be disabled.

[nkarpysh]

Hi John,

You are right it can be related to growing number of routes and you can check those with

sho ip route summ

However the fact the reload clears it for months can also mean a general memory leak in particular process. It is also recomended to monitor the memory with

show proc mem sorted

show proc mem

and watch if there are any process holding a lot and continiously increasing (possibly slowly) and never releasing memory.

Nik

[JohnTylerPearce]

That's a pretty good idea. I'm going to research that command and include that in my research tomorrow. I was thinking it could.... possibly be a bug but I'm not 100% yet. Thanks for the idea Nikolav!

[JohnTylerPearce]

Update

------------

So I was looking at the Flash and found the following.

-#- --length-- -----date/time------ path
1     50933220 Oct 5 2007 15:32:06 +00:00 c2800nm-advipservicesk9-mz.124-15.T.bin
2          806 Oct 5 2007 15:33:30 +00:00 Cisco.Pub.Key.txt
3      7540505 Oct 5 2007 15:34:20 +00:00 IOS-S290-CLI.pkg
4      1697952 Oct 5 2007 15:35:08 +00:00 securedesktop-ios-3.1.1.45-k9.pkg
5      6036480 Oct 5 2007 15:39:50 +00:00 SDM.TAR
6         1038 Oct 5 2007 15:41:00 +00:00 home.shtml
7         1826 Oct 5 2007 15:41:36 +00:00 sdmconfig-28xx.cfg
8       113152 Oct 5 2007 15:42:02 +00:00 HOME.TAR
9       416354 Oct 5 2007 15:42:58 +00:00 sslclient-win-1.1.3.173.pkg
10      861696 Oct 5 2007 15:43:44 +00:00 ES.TAR
11     1164288 Oct 5 2007 15:44:20 +00:00 COMMON.TAR
12       22952 Nov 9 2007 15:18:38 +00:00 config_11.09.07
13    51054864 Feb 29 2008 06:14:14 +00:00 c2800nm-advipservicesk9-mz.124-15.T3.bin
14       20569 Apr 8 2008 09:39:42 +00:00 config.new
15       20569 Apr 8 2008 09:39:46 +00:00 config.sav
16       20569 Apr 8 2008 09:39:50 +00:00 config.bak
17       23246 May 16 2008 20:06:40 +00:00 copy
18    67760352 Jul 20 2011 20:54:56 +00:00 c2800nm-adventerprisek9-mz.151-4.M1.bin
19    11523641 Jul 21 2011 15:12:52 +00:00 c3750-ipservices-mz.122-50.SE5.bin

57245696 bytes available (199258112 bytes used)

From this math

57245696 bytes available (199258112 bytes used) = -142012416

Is this even technically possible? Will it start using DRAM?

When I do a 'show version' I get the following.

Cisco 2821 (revision 1.0) with 505856K/18432K bytes of memory.

Processor board ID FTX1129A10L

3 Gigabit Ethernet interfaces

1 Serial interface

1 terminal line

2 Virtual Private Network (VPN) Modules

DRAM configuration is 64 bits wide with parity enabled.

239K bytes of non-volatile configuration memory.

250880K bytes of ATA CompactFlash (Read/Write)

Totals                    :     766240/778720     ( 98%) [240]

Cisco 2821 (revision 1.0) with 505856K/18432K bytes of memory.
Processor board ID FTX1129A10L
3 Gigabit Ethernet interfaces
1 Serial interface
1 terminal line
2 Virtual Private Network (VPN) Modules
DRAM configuration is 64 bits wide with parity enabled.
239K bytes of non-volatile configuration memory.
250880K bytes of ATA CompactFlash (Read/Write)

Now from what I understand the two CEF data structures FIB and AT are stored each in DRAM, is this correct?

Also, according to the configuration I believe we have 512 megs DRAM?

This is what I get after I run the 'show cef memory'

Totals                    :     766240/778720     ( 98%) [240]

[nkarpysh]

Hi John,

Your flash  is

250880K bytes of ATA CompactFlash (Read/Write)  - 250 M,

So math is

57245696 bytes available +199258112 bytes used = 250 MB

That is ok. And even if that is allused - it want use DRAM it just fail to save anything and possibly start sending error messgaes to console.

As for a CEF afaik it stores routes in TCAM - though DRAM is responsible to maintain TCAM, so if TCAM/DRAM exausted then there will be a problem. Here we see CEF reaching it's limitation. Possibly worth considering if some routes can be summarized to remove host routes from CEF table. That can help I guess.

Nik