Internet stops working on C1111-8P

[pro100bear]

Hi,

For some reasons internet partially stops working on my C1111-8P after 3-4 days. Some big websites, like google or fb are reachable, but most of the internet is not. Ping to this router form the internet is not working. Config is very simple:

interface GigabitEthernet0/0/1
 description ISP
 ip address dhcp
 ip nat outside
 ip access-group WAN in
 negotiation auto

ISP is Verizon fios.

After a reboot everything is working fine. Memory and CPU are at a very low usage all the time.

What should I look at first?

Thank you.

[Kasun Bandara]

Hi,

1. check your access list names 'WAN' for any unwanted blocking rules

2. check DNS resolving for working and not working web sites to identify any DNS issues

3. check with ISP for any outages or misconfiguration, because if you can access few sites then other sites also should be able to access

4. if you are using any Firewall between, check for misconfigurations

[pro100bear]

Thank you. The thing is this router works perfectly most of the time. When it happened first time I thought it is a problem with ISP. Wanted for 6 hours and then rebooted the router. But yesterday it happened again. I would say there is something wrong with the router.

[balaji.bandi]

You need to post full config.

what site was not working ?

[pro100bear]

Thank you. Attached.

Amazon, Pinterest, Netflix did not work. Ping to 8.8.8.8 did not work too. But at the same time Google and most of the google services were reachable.

[Georg Pauwen]

Hello,

I have made some changes to the config, monitor and check if that makes a difference:

version 17.6
service timestamps debug datetime msec
service timestamps log datetime msec
platform qfp utilization monitor load 80
platform punt-keepalive disable-kernel-core
platform hardware throughput crypto 50000
!
hostname BR-01
!
boot-start-marker
boot-end-marker
!
logging console emergencies
enable secret 9 $9$QBrFjH7YtFoEF.$/jD/ANRAOoZbuwn9xpSR8bu2xjMH1D16DFfWljlpDMs
enable password 07173020
!
no aaa new-model
clock timezone UTC -5 0
!
ip dhcp excluded-address 10.0.1.0 10.0.1.4
!
ip dhcp pool POOL
network 10.0.1.0 255.255.255.0
default-router 10.0.1.1
dns-server 8.8.8.8 4.2.2.2
!
login on-success log
!
subscriber templating
multilink bundle-name authenticated
!
crypto pki trustpoint TP-self-signed-15656755
enrollment selfsigned
subject-name cn=IOS-Self-Signed-Certificate-15656755
revocation-check none
rsakeypair TP-self-signed-15656755
!
crypto pki trustpoint SLA-TrustPoint
enrollment pkcs12
revocation-check crl
!
crypto pki certificate chain TP-self-signed-15656755
crypto pki certificate chain SLA-TrustPoint
!
crypto pki certificate pool
cabundle nvram:ios_core.p7b
!
license udi pid C1111-8P sn
memory free low-watermark processor 70210
!
diagnostic bootup level minimal
!
spanning-tree extend system-id
!
username cisco privilege 15 secret 9
!
redundancy
mode none
!
vlan internal allocation policy ascending
no cdp run
!
interface GigabitEthernet0/0/0
no ip address
shutdown
negotiation auto
!
interface GigabitEthernet0/0/1
description ISP
ip address dhcp
ip nat outside
negotiation auto
!
interface GigabitEthernet0/1/0
!
interface GigabitEthernet0/1/1
!
interface GigabitEthernet0/1/2
!
interface GigabitEthernet0/1/3
!
interface GigabitEthernet0/1/4
!
interface GigabitEthernet0/1/5
!
interface GigabitEthernet0/1/6
!
interface GigabitEthernet0/1/7
!
interface Vlan1
ip address 10.0.1.1 255.255.255.0
ip nat inside
!
ip http server
ip http authentication local
ip http secure-server
ip forward-protocol nd
!
ip nat inside source list 1 interface GigabitEthernet0/0/1 overload
ip route 0.0.0.0 0.0.0.0 GigabitEthernet0/0/1 dhcp
!
access-list 1 permit 10.0.1.0 0.0.0.255
!
control-plane
!
line con 0
transport input none
stopbits 1
line vty 0 4
password 07173020
login
length 0
transport input ssh
line vty 5 14
password 07173020
login
transport input ssh
!
call-home
! If contact email address in call-home is configured as sch-smart-licensing@cisco.com
! the email address configured in Cisco Smart License Portal will be used as contact email address to send SCH notifications.
contact-email-addr sch-smart-licensing@cisco.com
profile "CiscoTAC-1"
active
destination transport-method http
ntp server time.google.com prefer source GigabitEthernet0/0/1
!
end

[pro100bear]

Thank you. I will try this config, but honestly I doubt it would change anything. The thing is everything is working perfectly for a few days.

I thought maybe ISP give a new IP, but the router does not update it...

[pro100bear]

Hi,

It stopped again and here is what I found in the logs:

Nov 24 03:52:59.011: %ARP-4-ARPLEARNCROSS: 15360 Learned ARP entries are installed in the ARP table and reached the max limit

I then run "clear arp-cache" and it helped with no the entire device rebooting.

Though I am not sure if it is all related ...

[paul driver]

Hello
you are maxing out in your arp queries and the probable reason for that is you have a default route pointing to a physical interface as such it tells the rtr to arp for everything off tha wan interface for it directly connected which it isn’t

no ip route 0.0.0.0 0.0.0.0 gig0/0/1
ip route 0.0.0.0 0.0.0.0 gig0/0/1 dhcp

Edited-
also change you dhcp pool to point you your own rtr and enable dns

ip dhcp pool POOL
no dns-server 8.8.8.8 4.2.2.2
dns server 10.0.1.1

ip dns server

[pro100bear]

Thank you for your help. I added dhcp to ip route. Will see in a few days.

But the main question is why it is happening. From what I found on the internet, this ARP cached should clean up itself from time to time... It is not the slowest device.

[paul driver]

Hello
That error code suggests the rtr is reaching its maximum arp entry limitation, As such that amended default route should help also you could also try tweaking the arp timeouts on the wan interface and NAT tcp translations timer, just keep eye of the rtr cpu/memory process if you do.

conf t
ip nat translation tcp-timeout 3600
int gig0/0/1
arp timeout 7200

[Georg Pauwen]

Hello,

it could be a bug, or some sort of external hack attempt. With just the defaults configured on your C1111, I have never seen anyone reporting this error before.

While doing more research, configure the simple EEM script below to automate the arp cache clearing, that way, at least you don't have to do it manually.

event manager applet CLEAR_ARP
event syslog pattern "%ARP-4-ARPLEARNCROSS"
action 1.0 cli command "enable"
action 2.0 cli command "clear arp-cache"

[pro100bear]

---

@Georg Pauwen wrote:

Hello,

it could be a bug, or some sort of external hack attempt. With just the defaults configured on your C1111, I have never seen anyone reporting this error before.

---

Maybe... I will try to install MD version. I am currently on ED, maybe there is a bug or something.

[pro100bear]

I think I found the bug. Can someone please confirm if I am right?

So I run show arp and exported the data. I did the same in 20 minutes and compared the lists. These two lists are on the screenshot. The first one is on the left and the second one is on the right.

Checking the line Internet 1.215.116.138 245 54e0.3275.1fc8 on the first list. The age is 245 minutes. The default cache timeout is 240 minutes + a random jitter that is no more than 30 minutes.

So in a few minutes, it should be cleared and removed from the list. Right? Ok, updated the list every few minutes and found this:

Internet 1.215.116.138 2 54e0.3275.1fc8

Now it says 2 instead of 245. So instead of deleting this line, the system just reset the times for this IP and MAC combination. Am I right? Is this the bug that causes the problem with the ARP and the internet?