Solved: IP CEF - ARP cache issue. Help!!

Nicholas Richardson · ‎11-01-2011

We have a Cisco 881 router, which is crashing. We have seen that the ARP cache fills up so much it causes things to crash, our phones go down.. We dont know why this however IP CEF seems to be doing it, when we disable it goes away however disabling IP CEF causes our L2TP tunnel to become inoperable also. So why does IP CEF cause thousands of AR entries and how can we limit that!? Below is the error, sample of the ARP cache and our config. You will notice we also have a /31 given to us on WAN interface, this was given to us by our service provider. Any help gratefully appreciated, this is really strange I cant find other examples on internet.

The error:

Nov 1 04:21:57.474: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x81F083F4, alignment 16

Pool: Processor Free: 55176 Cause: Not enough free memory

Alternate Pool: I/O Free: 2352 Cause: Not enough free memory

-Process= "IP ARP Adjacency", ipl= 4, pid= 93, -Traceback= 0x81ECDDBCz 0x81EEFEACz 0x81F083F8z 0x80EF09C0z 0x80EF2A98z 0x80EF30A8z 0x80EEBAD0z 0x80EB20E0z 0x80E77ABCz 0x80EB5D58z 0x80E6D590z 0x80E6D728z 0x80EFA8DCz 0x80EFC4A8z 0x80EFC7FCz 0x80F4B6F4z

Nov 1 04:21:57.474: %COMMON_FIB-3-NOMEM: Memory allocation failure for validating prefix in IPv4 CEF [0x80E5C570] (fatal) (1 subsequent failure).

Nov 1 04:21:57.474: %COMMON_FIB-4-DISABLING: IPv4 CEF is being disabled due to a fatal error.

----

ARP Cache grows and grows! (this is only a small part of it)

Internet 92.244.102.35 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 93.106.58.124 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 93.152.2.184 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 93.186.25.33 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 94.30.30.45 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 94.65.135.105 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 94.229.169.157 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 95.26.58.189 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 95.71.47.220 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 95.145.233.189 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 95.157.171.219 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 95.158.166.29 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 95.158.240.98 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 95.175.133.195 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 95.177.36.236 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 95.177.36.251 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 95.177.83.81 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 95.177.112.74 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 95.177.112.89 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 95.177.115.217 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 95.177.120.2 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 95.177.120.8 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 95.177.124.190 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 95.182.12.183 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 96.47.67.105 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 96.55.244.191 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 98.190.6.100 0 0007.7dd9.f802 ARPA FastEthernet4

Protocol Address Age (min) Hardware Addr Type Interface

Internet 98.234.32.235 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 98.252.226.26 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 99.38.10.203 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 99.109.224.219 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 107.20.249.243 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 108.20.186.227 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 108.86.142.112 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 109.91.172.101 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 109.124.49.200 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 109.204.16.245 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 109.204.16.252 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 109.204.18.221 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 109.204.19.17 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 109.204.19.45 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 109.204.19.201 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 109.204.19.211 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 109.204.85.47 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 109.205.186.36 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 110.77.234.108 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 110.168.77.87 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 110.247.149.121 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 112.205.49.228 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 112.210.175.127 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 113.193.168.17 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 114.42.221.29 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 115.128.5.228 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 115.184.122.151 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 115.232.185.22 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 117.203.146.240 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 117.254.242.187 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 120.60.38.252 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 121.45.140.38 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 121.131.138.101 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 122.102.190.84 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 122.108.13.150 0 0007.7dd9.f802 ARPA FastEthernet4

Protocol Address Age (min) Hardware Addr Type Interface

Internet 122.172.28.101 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 122.175.137.98 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 122.176.254.29 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 122.177.29.83 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 123.21.184.150 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 123.238.66.114 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 124.43.23.194 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 124.82.35.40 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 124.168.77.84 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 124.208.84.162 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 124.244.155.215 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 126.107.20.125 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 128.211.201.213 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 129.6.15.28 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 129.242.185.231 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 130.37.34.52 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 130.83.203.72 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 130.159.196.118 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 141.2.212.16 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 142.68.86.23 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 149.3.130.112 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 149.13.32.15 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 151.25.14.212 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 151.29.245.230 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 169.254.105.207 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 174.7.191.135 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 174.36.215.20 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 174.37.211.68 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 174.101.224.116 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 177.16.236.144 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 178.16.128.74 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 178.35.158.192 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 178.36.99.58 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 178.74.253.234 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 178.125.39.47 0 0007.7dd9.f802 ARPA FastEthernet4

Protocol Address Age (min) Hardware Addr Type Interface

Internet 178.125.101.227 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 178.165.47.48 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 178.176.19.60 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 178.187.145.12 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 178.239.102.36 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 178.239.102.37 - 0007.7d99.043c ARPA FastEthernet4

Internet 178.252.106.192 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 182.93.214.99 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 183.107.161.196 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 183.182.90.82 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 183.185.112.81 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 184.17.83.33 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 186.205.70.102 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 188.2.249.152 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 188.25.156.11 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 188.27.215.132 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 188.52.196.71 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 188.54.38.56 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 188.65.176.153 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 188.66.72.107 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 188.187.35.39 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 188.249.145.161 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 189.81.57.140 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 190.191.98.97 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 190.245.158.24 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 192.168.0.1 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 192.168.10.5 - 0007.7d99.0438 ARPA Vlan10

Internet 192.168.100.10 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 192.168.237.1 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 192.168.254.1 - 0007.7d99.0438 ARPA Vlan2

Internet 192.168.254.2 28 64d9.893b.61c1 ARPA Vlan2

Internet 192.168.254.3 28 d0c2.829e.af41 ARPA Vlan2

Protocol Address Age (min) Hardware Addr Type Interface

Internet 192.168.254.111 0 0016.76df.8e42 ARPA Vlan2

Internet 192.168.254.240 0 0015.17c0.b3b4 ARPA Vlan2

Internet 192.175.48.1 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 194.165.188.82 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 195.74.128.6 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 195.74.130.12 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 196.41.255.44 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 199.7.51.72 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 199.47.216.144 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 199.47.216.146 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 199.47.216.147 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 199.47.216.148 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 199.47.216.149 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 199.47.217.144 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 199.47.217.146 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 199.47.217.149 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 199.47.217.172 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 199.47.217.173 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 201.1.196.233 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 202.83.41.183 0 0007.7dd9.f802 ARPA FastEthernet4

Internet 204.9.163.158 0 0007.7dd9.f802 ARPA FastEthernet4

----

Our config

#show run

Building configuration...

Current configuration : 6521 bytes

!

! Last configuration change at 08:51:40 GMT Tue Nov 1 2011 by admin

!

version 15.0

no service pad

service timestamps debug datetime msec

service timestamps log datetime msec

no service password-encryption

!

boot-start-marker

boot-end-marker

!

logging buffered 51200 warnings

!

no aaa new-model

!

monitor session 1 source interface Fa3

monitor session 1 destination interface Fa1

memory-size iomem 10

clock timezone GMT 0

clock summer-time BST date Mar 27 2011 1:00 Oct 30 2011 2:00

!

ip source-route

!

ip dhcp excluded-address 192.168.254.1 192.168.254.99

!

ip cef

no ip domain lookup

ip domain name yourdomain.com

ip name-server 8.8.8.8

no ipv6 cef

l2tp-class l2tpclass1

!

vpdn enable

!

license udi pid CISCO881-K9 sn FCZ152692N2

!

vtp mode client

!

pseudowire-class communicator

encapsulation l2tpv2

protocol l2tpv2 l2tpclass1

ip local interface FastEthernet4

!

interface FastEthernet0

switchport access vlan 4

!

interface FastEthernet1

description Phones

switchport access vlan 3

!

interface FastEthernet2

switchport access vlan 4

!

interface FastEthernet3

description Trunk to switch

switchport mode trunk

!

interface FastEthernet4

description WAN

ip address x.x.x.x 255.255.255.254

ip nat outside

ip virtual-reassembly

duplex auto

speed auto

no cdp enable

!

interface Virtual-PPP1

ip unnumbered Vlan3

ppp chap hostname x.x.x.x

ppp chap password 0 xxxxxxx

ppp direction callout

no cdp enable

pseudowire x.x.x.x 10 pw-class x

!

interface Vlan1

description $ETH-SW-LAUNCH$$INTF-INFO-HWIC 4ESW$

no ip address

ip tcp adjust-mss 1452

shutdown

!

interface Vlan2

ip address 192.168.254.1 255.255.255.0

ip nat inside

ip virtual-reassembly

!

interface Vlan3

ip address 10.60.97.129 255.255.255.192

ip helper-address 10.50.32.32

ip tcp adjust-mss 1452

!

interface Vlan4

ip address x.x.x.x 255.255.255.248

!

ip forward-protocol nd

no ip http server

ip http access-class 23

ip http authentication local

no ip http secure-server

ip http timeout-policy idle 60 life 86400 requests 10000

!

ip dns server

ip nat inside source list 15 interface FastEthernet4 overload

ip route 0.0.0.0 0.0.0.0 FastEthernet4

ip route 10.0.0.0 255.0.0.0 Virtual-PPP1

ip route x.x.x.x 255.255.255.248 Vlan4

!

logging 192.168.254.111

access-list 15 permit any

access-list 23 permit x.x.x.x

Many thanks!

Nicholas

Joseph W. Doherty · ‎11-01-2011

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

ip route 0.0.0.0 0.0.0.0 FastEthernet4

change to:

ip route 0.0.0.0 0.0.0.0 x.x.x.x !next hop IP

View solution in original post

Joseph W. Doherty · ‎11-01-2011

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

ip route 0.0.0.0 0.0.0.0 FastEthernet4

change to:

ip route 0.0.0.0 0.0.0.0 x.x.x.x !next hop IP

Nicholas Richardson · ‎11-01-2011

couldnt see the wood for the trees!

many thanks

Nicholas

softimpera · ‎09-06-2013

Hello,

If I have 2 web servers in the network with NAT.

Who is in my case next hop ip ? The ip from WAN (GigabitEthernet0/1) ?

----------------

ip dhcp pool pool1

import all

network 192.168.1.0 255.255.255.0

dns-server x.x.x.1 x.x.x.2

default-router 192.168.1.1

!

interface GigabitEthernet0/0

description $ETH-LAN$$FW_INSIDE$

ip address 192.168.1.1 255.255.255.0

no ip redirects

no ip unreachables

no ip proxy-arp

ip nat inside

ip virtual-reassembly in

zone-member security in-zone

duplex auto

speed auto

no cdp enable

no mop enabled

!

interface GigabitEthernet0/1

description $ETH-WAN$$FW_OUTSIDE$

ip address 88.x.x.157 255.255.255.128

no ip redirects

no ip unreachables

no ip proxy-arp

ip verify unicast reverse-path

ip nat outside

ip ips sdm_ips_rule in

ip virtual-reassembly in

zone-member security out-zone

duplex auto

speed auto

no cdp enable

no mop enabled

!

ip forward-protocol nd

!

ip http server

ip http access-class 2

ip http authentication local

ip http secure-server

!

ip nat inside source list 1 interface GigabitEthernet0/1 overload

ip nat inside source static 192.168.1.88 34.x.x.178

ip nat inside source static 192.168.1.89 34.x.x.179

ip route 0.0.0.0 0.0.0.0 GigabitEthernet0/1

-----------------

How can I do this ?

Joseph W. Doherty · ‎09-06-2013

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

It would be the IP address of the device on the "otherside" of G0/1. Sh ARP or a traceroute should show it.

softimpera · ‎09-06-2013

Hello,

Thank''s for your answer

So GigabitEthernet0/1 is the Wan and has an ip address 88.x.x.157 255.255.255.128 directly connected to internet provider.

So my commads should be :

no ip route 0.0.0.0 0.0.0.0 GigabitEthernet0/1

ip route 0.0.0.0 0.0.0.0 88.x.x.157

Will work this with nat ?

Joseph W. Doherty · ‎09-06-2013

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Yes to the first and believe also yes to the second.