Buy or Renew
Buy or Renew
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
cancel
Start a conversation
1423
Views
5
Helpful
22
Replies

network connectivity issue with arp table having thousands of entries

Go to solution
Elito Haylett
Beginner
Beginner

‎05-23-2023 12:49 PM - last edited on ‎06-08-2023 12:52 AM by Community Manager

I have an issue happening in my network that at first we thought was a wireless problem because we were receiving a lot of mac flapping due to roaming clients which we thought affected the network but upon further investigation it is happening to both wired and wireless clients. I'm connected to the network with via ethernet and when i tried to browse certain sites it timed out and other sites were responsive but slow loading. Some devices at time can't connect to the network because they don't get an IP address. To get the network to be responsive again I have to issue a

clear arp

command. I tried to debug arp on the router to see if any messages would be displayed but there were none. I checked the interfaces the AP's and clients are connected to for errors and there were none. The CPU utilization on the Cisco ISR4431 router didn't even exceed User and System 10%. After I cleared the arp cache, a few seconds later every site that I couldn't browse to before started loading without any problems.

In the wireless forum this was posted by one of the Wireless VIP as possible things to look at so I'm in desperate need of help because it does impact my network.

> "it affects both wired and wireless clients"
Then it's not a wireless problem - it's a switching or routing problem.
> "I was able to check the arp cache prior to and after and there were entries in the table"
And what were those entries?

The fact that it can affect some destinations and not others is downright weird because ARP cache should only be relevant to local devices, nothing beyond the next hop. Some ideas on possible problems - pure guesswork at this point because we don't have any real detail to work with:
- Person in the middle type attack - some device is redirecting traffic via another node on the network - hair-pinning the traffic - by hijacking the ARP entries, potentially for the router

(default gateway)

IP
- Proxy ARP enabled by mistake with a bad routing design resulting in your ARP cache trying to create an entry for every device on the internet - that would explain why clearing the ARP cache temporarily helps
- If your DHCP or devices have wrong default gateway configured then resulting in ICMP redirects to the correct gateway then your devices could start filling up with /32 routes to every IP on the internet.

Here's a truncated list of devices in the arp table there thousands of these entries that it shows devices outside of my network and beyond my next hop gateway.

ECH-ISR4431-138#show arp
Protocol Address Age (min) Hardware Addr Type Interface
Internet 1.34.163.232 1 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 2.57.121.229 28 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 2.180.35.216 67 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.0.126 168 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.1.2 218 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.1.162 131 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.2.123 37 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.2.176 191 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.2.202 163 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.2.216 94 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.3.139 71 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Protocol Address Age (min) Hardware Addr Type Interface
Internet 3.5.3.161 0 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.3.185 43 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.3.211 28 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.3.216 254 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.6.101 79 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.6.112 101 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.6.130 241 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.6.143 136 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.6.160 121 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.7.133 88 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.7.170 181 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Protocol Address Age (min) Hardware Addr Type Interface
Internet 3.5.7.203 36 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.8.19 25 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.8.106 254 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.8.160 133 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.9.11 166 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.9.134 140 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.9.171 254 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.10.138 192 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.10.150 97 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.10.151 144 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.10.180 118 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Protocol Address Age (min) Hardware Addr Type Interface
Internet 3.5.10.193 155 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.11.119 220 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.11.134 13 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.11.146 141 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.11.149 156 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.11.194 254 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.11.199 4 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.11.201 13 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.11.226 5 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.16.12 145 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.16.103 187 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Protocol Address Age (min) Hardware Addr Type Interface
Internet 3.5.16.172 82 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.17.165 139 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.17.221 208 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.19.141 44 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.20.19 12 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.20.205 108 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.20.215 126 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.21.122 109 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.21.148 27 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.21.183 169 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.21.204 45 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Protocol Address Age (min) Hardware Addr Type Interface
Internet 3.5.25.20 63 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.25.42 2 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.25.47 30 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.25.92 87 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.25.105 198 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.25.110 13 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.25.114 120 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.25.116 2 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.25.139 58 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.25.187 14 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.25.204 147 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Protocol Address Age (min) Hardware Addr Type Interface
Internet 3.5.25.205 254 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.25.229 224 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.25.231 200 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.25.242 153 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.27.104 170 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.27.119 38 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.27.135 91 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.27.141 190 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.27.142 129 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.27.156 126 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.27.163 100 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Protocol Address Age (min) Hardware Addr Type Interface
Internet 3.5.27.182 111 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.27.196 233 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.28.18 1 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.28.23 173 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.28.101 53 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.28.132 213 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.28.139 2 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.28.154 196 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.28.157 138 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.28.162 205 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.28.164 130 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2



This only started happening since upgrading from an ISR2921 to the ISR4335 and then the ISR4435. All I did was copied the config over from the previous 2921 to the upgraded devices....


sanitized version of the config

ECH-ISR4431-138#show runn
Building configuration...

Current configuration : 21034 bytes
!
! Last configuration change at 12:05:29 est Tue May 23 2023 by ehaylett
! NVRAM config last updated at 19:07:54 est Mon May 15 2023 by ehaylett
!
version 17.9
service timestamps debug datetime msec
service timestamps log datetime localtime
service call-home
platform qfp utilization monitor load 80
no platform punt-keepalive disable-kernel-core
platform hardware throughput level 1000000
!
hostname ECH-ISR4431-138
!
boot-start-marker
boot system bootflash:isr4400-universalk9.17.09.02a.SPA.bin
boot-end-marker
!
!
vrf definition Mgmt-intf
!
address-family ipv4
exit-address-family
!
address-family ipv6
exit-address-family
!
logging buffered informational
logging console informational
aaa new-model
!
!
aaa authentication login default local
aaa authorization console
aaa authorization exec default local
!
!
aaa session-id common
clock timezone est -4 0
clock calendar-valid
!
!
!
!
ip nbar http-services
!
!
!
!
!
ip name-server 68.237.161.12 71.250.0.12
ip ddns update method dyndns
HTTP
add http://xxxxxxxxx@members.dyndns.org/nic/updatesystem=dyndns&hostname=<h>&myip=<a>
interval maximum 0 0 5 0
!
ip dhcp excluded-address 172.168.100.0 172.168.100.24
ip dhcp excluded-address 172.168.120.0 172.168.120.1
!
ip dhcp pool ECH-NET-100
network 172.168.100.0 255.255.255.0
default-router 172.168.100.1
dns-server 68.237.161.12 71.250.0.12
lease 3
!
ip dhcp pool ECH-VOICE-NET-120
network 172.168.120.0 255.255.255.0
lease infinite
!
!
!
login on-success log
!
!
!
!
!
!
!
subscriber templating
!
!
!
!
!
multilink bundle-name authenticated
!
!
!
!
!
!
!
crypto pki trustpoint SLA-TrustPoint
enrollment pkcs12
revocation-check crl
!
crypto pki trustpoint TP-self-signed-3693526534
enrollment selfsigned
subject-name cn=IOS-Self-Signed-Certificate-3693526534
revocation-check none
rsakeypair TP-self-signed-3693526534
!
!
crypto pki certificate chain SLA-TrustPoint
certificate ca 01
30820321 30820209 A0030201 02020101 300D0609 2A864886 F70D0101 0B050030
32310E30 0C060355 040A1305 43697363 6F312030 1E060355 04031317 43697363
6F204C69 63656E73 696E6720 526F6F74 20434130 1E170D31 33303533 30313934
3834375A 170D3338 30353330 31393438 34375A30 32310E30 0C060355 040A1305
43697363 6F312030 1E060355 04031317 43697363 6F204C69 63656E73 696E6720
526F6F74 20434130 82012230 0D06092A 864886F7 0D010101 05000382 010F0030
82010A02 82010100 A6BCBD96 131E05F7 145EA72C 2CD686E6 17222EA1 F1EFF64D
CBB4C798 212AA147 C655D8D7 9471380D 8711441E 1AAF071A 9CAE6388 8A38E520
1C394D78 462EF239 C659F715 B98C0A59 5BBB5CBD 0CFEBEA3 700A8BF7 D8F256EE
4AA4E80D DB6FD1C9 60B1FD18 FFC69C96 6FA68957 A2617DE7 104FDC5F EA2956AC
7390A3EB 2B5436AD C847A2C5 DAB553EB 69A9A535 58E9F3E3 C0BD23CF 58BD7188
68E69491 20F320E7 948E71D7 AE3BCC84 F10684C7 4BC8E00F 539BA42B 42C68BB7
C7479096 B4CB2D62 EA2F505D C7B062A4 6811D95B E8250FC4 5D5D5FB8 8F27D191
C55F0D76 61F9A4CD 3D992327 A8BB03BD 4E6D7069 7CBADF8B DF5F4368 95135E44
DFC7C6CF 04DD7FD1 02030100 01A34230 40300E06 03551D0F 0101FF04 04030201
06300F06 03551D13 0101FF04 05300301 01FF301D 0603551D 0E041604 1449DC85
4B3D31E5 1B3E6A17 606AF333 3D3B4C73 E8300D06 092A8648 86F70D01 010B0500
03820101 00507F24 D3932A66 86025D9F E838AE5C 6D4DF6B0 49631C78 240DA905
604EDC