11-01-2011 02:11 AM - edited 03-04-2019 02:07 PM
We have a Cisco 881 router, which is crashing. We have seen that the ARP cache fills up so much it causes things to crash, our phones go down.. We dont know why this however IP CEF seems to be doing it, when we disable it goes away however disabling IP CEF causes our L2TP tunnel to become inoperable also. So why does IP CEF cause thousands of AR entries and how can we limit that!? Below is the error, sample of the ARP cache and our config. You will notice we also have a /31 given to us on WAN interface, this was given to us by our service provider. Any help gratefully appreciated, this is really strange I cant find other examples on internet.
The error:
Nov 1 04:21:57.474: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x81F083F4, alignment 16
Pool: Processor Free: 55176 Cause: Not enough free memory
Alternate Pool: I/O Free: 2352 Cause: Not enough free memory
-Process= "IP ARP Adjacency", ipl= 4, pid= 93, -Traceback= 0x81ECDDBCz 0x81EEFEACz 0x81F083F8z 0x80EF09C0z 0x80EF2A98z 0x80EF30A8z 0x80EEBAD0z 0x80EB20E0z 0x80E77ABCz 0x80EB5D58z 0x80E6D590z 0x80E6D728z 0x80EFA8DCz 0x80EFC4A8z 0x80EFC7FCz 0x80F4B6F4z
Nov 1 04:21:57.474: %COMMON_FIB-3-NOMEM: Memory allocation failure for validating prefix in IPv4 CEF [0x80E5C570] (fatal) (1 subsequent failure).
Nov 1 04:21:57.474: %COMMON_FIB-4-DISABLING: IPv4 CEF is being disabled due to a fatal error.
----
ARP Cache grows and grows! (this is only a small part of it)
Internet 92.244.102.35 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 93.106.58.124 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 93.152.2.184 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 93.186.25.33 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 94.30.30.45 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 94.65.135.105 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 94.229.169.157 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 95.26.58.189 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 95.71.47.220 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 95.145.233.189 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 95.157.171.219 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 95.158.166.29 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 95.158.240.98 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 95.175.133.195 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 95.177.36.236 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 95.177.36.251 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 95.177.83.81 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 95.177.112.74 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 95.177.112.89 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 95.177.115.217 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 95.177.120.2 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 95.177.120.8 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 95.177.124.190 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 95.182.12.183 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 96.47.67.105 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 96.55.244.191 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 98.190.6.100 0 0007.7dd9.f802 ARPA FastEthernet4
Protocol Address Age (min) Hardware Addr Type Interface
Internet 98.234.32.235 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 98.252.226.26 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 99.38.10.203 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 99.109.224.219 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 107.20.249.243 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 108.20.186.227 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 108.86.142.112 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 109.91.172.101 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 109.124.49.200 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 109.204.16.245 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 109.204.16.252 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 109.204.18.221 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 109.204.19.17 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 109.204.19.45 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 109.204.19.201 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 109.204.19.211 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 109.204.85.47 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 109.205.186.36 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 110.77.234.108 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 110.168.77.87 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 110.247.149.121 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 112.205.49.228 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 112.210.175.127 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 113.193.168.17 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 114.42.221.29 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 115.128.5.228 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 115.184.122.151 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 115.232.185.22 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 117.203.146.240 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 117.254.242.187 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 120.60.38.252 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 121.45.140.38 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 121.131.138.101 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 122.102.190.84 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 122.108.13.150 0 0007.7dd9.f802 ARPA FastEthernet4
Protocol Address Age (min) Hardware Addr Type Interface
Internet 122.172.28.101 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 122.175.137.98 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 122.176.254.29 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 122.177.29.83 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 123.21.184.150 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 123.238.66.114 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 124.43.23.194 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 124.82.35.40 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 124.168.77.84 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 124.208.84.162 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 124.244.155.215 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 126.107.20.125 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 128.211.201.213 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 129.6.15.28 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 129.242.185.231 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 130.37.34.52 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 130.83.203.72 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 130.159.196.118 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 141.2.212.16 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 142.68.86.23 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 149.3.130.112 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 149.13.32.15 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 151.25.14.212 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 151.29.245.230 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 169.254.105.207 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 174.7.191.135 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 174.36.215.20 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 174.37.211.68 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 174.101.224.116 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 177.16.236.144 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 178.16.128.74 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 178.35.158.192 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 178.36.99.58 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 178.74.253.234 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 178.125.39.47 0 0007.7dd9.f802 ARPA FastEthernet4
Protocol Address Age (min) Hardware Addr Type Interface
Internet 178.125.101.227 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 178.165.47.48 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 178.176.19.60 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 178.187.145.12 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 178.239.102.36 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 178.239.102.37 - 0007.7d99.043c ARPA FastEthernet4
Internet 178.252.106.192 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 182.93.214.99 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 183.107.161.196 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 183.182.90.82 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 183.185.112.81 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 184.17.83.33 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 186.205.70.102 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 188.2.249.152 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 188.25.156.11 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 188.27.215.132 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 188.52.196.71 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 188.54.38.56 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 188.65.176.153 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 188.66.72.107 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 188.187.35.39 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 188.249.145.161 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 189.81.57.140 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 190.191.98.97 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 190.245.158.24 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 192.168.0.1 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 192.168.10.5 - 0007.7d99.0438 ARPA Vlan10
Internet 192.168.100.10 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 192.168.237.1 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 192.168.254.1 - 0007.7d99.0438 ARPA Vlan2
Internet 192.168.254.2 28 64d9.893b.61c1 ARPA Vlan2
Internet 192.168.254.3 28 d0c2.829e.af41 ARPA Vlan2
Protocol Address Age (min) Hardware Addr Type Interface
Internet 192.168.254.111 0 0016.76df.8e42 ARPA Vlan2
Internet 192.168.254.240 0 0015.17c0.b3b4 ARPA Vlan2
Internet 192.175.48.1 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 194.165.188.82 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 195.74.128.6 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 195.74.130.12 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 196.41.255.44 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 199.7.51.72 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 199.47.216.144 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 199.47.216.146 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 199.47.216.147 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 199.47.216.148 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 199.47.216.149 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 199.47.217.144 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 199.47.217.146 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 199.47.217.149 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 199.47.217.172 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 199.47.217.173 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 201.1.196.233 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 202.83.41.183 0 0007.7dd9.f802 ARPA FastEthernet4
Internet 204.9.163.158 0 0007.7dd9.f802 ARPA FastEthernet4
----
Our config
#show run
Building configuration...
Current configuration : 6521 bytes
!
! Last configuration change at 08:51:40 GMT Tue Nov 1 2011 by admin
!
version 15.0
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
!
boot-start-marker
boot-end-marker
!
logging buffered 51200 warnings
!
no aaa new-model
!
monitor session 1 source interface Fa3
monitor session 1 destination interface Fa1
memory-size iomem 10
clock timezone GMT 0
clock summer-time BST date Mar 27 2011 1:00 Oct 30 2011 2:00
!
ip source-route
!
!
ip dhcp excluded-address 192.168.254.1 192.168.254.99
!
!
ip cef
no ip domain lookup
ip domain name yourdomain.com
ip name-server 8.8.8.8
no ipv6 cef
l2tp-class l2tpclass1
!
!
!
vpdn enable
!
license udi pid CISCO881-K9 sn FCZ152692N2
!
!
vtp mode client
!
!
pseudowire-class communicator
encapsulation l2tpv2
protocol l2tpv2 l2tpclass1
ip local interface FastEthernet4
!
!
interface FastEthernet0
switchport access vlan 4
!
interface FastEthernet1
description Phones
switchport access vlan 3
!
interface FastEthernet2
switchport access vlan 4
!
interface FastEthernet3
description Trunk to switch
switchport mode trunk
!
interface FastEthernet4
description WAN
ip address x.x.x.x 255.255.255.254
ip nat outside
ip virtual-reassembly
duplex auto
speed auto
no cdp enable
!
interface Virtual-PPP1
ip unnumbered Vlan3
ppp chap hostname x.x.x.x
ppp chap password 0 xxxxxxx
ppp direction callout
no cdp enable
pseudowire x.x.x.x 10 pw-class x
!
interface Vlan1
description $ETH-SW-LAUNCH$$INTF-INFO-HWIC 4ESW$
no ip address
ip tcp adjust-mss 1452
shutdown
!
interface Vlan2
ip address 192.168.254.1 255.255.255.0
ip nat inside
ip virtual-reassembly
!
interface Vlan3
ip address 10.60.97.129 255.255.255.192
ip helper-address 10.50.32.32
ip tcp adjust-mss 1452
!
interface Vlan4
ip address x.x.x.x 255.255.255.248
!
!
ip forward-protocol nd
no ip http server
ip http access-class 23
ip http authentication local
no ip http secure-server
ip http timeout-policy idle 60 life 86400 requests 10000
!
ip dns server
ip nat inside source list 15 interface FastEthernet4 overload
ip route 0.0.0.0 0.0.0.0 FastEthernet4
ip route 10.0.0.0 255.0.0.0 Virtual-PPP1
ip route x.x.x.x 255.255.255.248 Vlan4
!
logging 192.168.254.111
access-list 15 permit any
access-list 23 permit x.x.x.x
Many thanks!
Nicholas
Solved! Go to Solution.
11-01-2011 02:47 AM
Disclaimer
The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.
Liability Disclaimer
In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.
Posting
ip route 0.0.0.0 0.0.0.0 FastEthernet4
change to:
ip route 0.0.0.0 0.0.0.0 x.x.x.x !next hop IP
11-01-2011 02:47 AM
Disclaimer
The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.
Liability Disclaimer
In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.
Posting
ip route 0.0.0.0 0.0.0.0 FastEthernet4
change to:
ip route 0.0.0.0 0.0.0.0 x.x.x.x !next hop IP
11-01-2011 03:05 AM
couldnt see the wood for the trees!
many thanks
Nicholas
09-06-2013 09:58 AM
Hello,
If I have 2 web servers in the network with NAT.
Who is in my case next hop ip ? The ip from WAN (GigabitEthernet0/1) ?
----------------
ip dhcp pool pool1
import all
network 192.168.1.0 255.255.255.0
dns-server x.x.x.1 x.x.x.2
default-router 192.168.1.1
!
interface GigabitEthernet0/0
description $ETH-LAN$$FW_INSIDE$
ip address 192.168.1.1 255.255.255.0
no ip redirects
no ip unreachables
no ip proxy-arp
ip nat inside
ip virtual-reassembly in
zone-member security in-zone
duplex auto
speed auto
no cdp enable
no mop enabled
!
interface GigabitEthernet0/1
description $ETH-WAN$$FW_OUTSIDE$
ip address 88.x.x.157 255.255.255.128
no ip redirects
no ip unreachables
no ip proxy-arp
ip verify unicast reverse-path
ip nat outside
ip ips sdm_ips_rule in
ip virtual-reassembly in
zone-member security out-zone
duplex auto
speed auto
no cdp enable
no mop enabled
!
ip forward-protocol nd
!
ip http server
ip http access-class 2
ip http authentication local
ip http secure-server
!
ip nat inside source list 1 interface GigabitEthernet0/1 overload
ip nat inside source static 192.168.1.88 34.x.x.178
ip nat inside source static 192.168.1.89 34.x.x.179
ip route 0.0.0.0 0.0.0.0 GigabitEthernet0/1
-----------------
How can I do this ?
09-06-2013 10:23 AM
Disclaimer
The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.
Liability Disclaimer
In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.
Posting
It would be the IP address of the device on the "otherside" of G0/1. Sh ARP or a traceroute should show it.
09-06-2013 12:07 PM
Hello,
Thank''s for your answer
So GigabitEthernet0/1 is the Wan and has an ip address 88.x.x.157 255.255.255.128 directly connected to internet provider.
So my commads should be :
no ip route 0.0.0.0 0.0.0.0 GigabitEthernet0/1
ip route 0.0.0.0 0.0.0.0 88.x.x.157
Will work this with nat ?
09-06-2013 06:00 PM
Disclaimer
The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.
Liability Disclaimer
In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.
Posting
Yes to the first and believe also yes to the second.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide