network connectivity issue with arp table having thousands of entries

[Elito Haylett]

I have an issue happening in my network that at first we thought was a wireless problem because we were receiving a lot of mac flapping due to roaming clients which we thought affected the network but upon further investigation it is happening to both wired and wireless clients. I'm connected to the network with via ethernet and when i tried to browse certain sites it timed out and other sites were responsive but slow loading. Some devices at time can't connect to the network because they don't get an IP address. To get the network to be responsive again I have to issue a

clear arp

command. I tried to debug arp on the router to see if any messages would be displayed but there were none. I checked the interfaces the AP's and clients are connected to for errors and there were none. The CPU utilization on the Cisco ISR4431 router didn't even exceed User and System 10%. After I cleared the arp cache, a few seconds later every site that I couldn't browse to before started loading without any problems.

In the wireless forum this was posted by one of the Wireless VIP as possible things to look at so I'm in desperate need of help because it does impact my network.

> "it affects both wired and wireless clients"
Then it's not a wireless problem - it's a switching or routing problem.
> "I was able to check the arp cache prior to and after and there were entries in the table"
And what were those entries?

The fact that it can affect some destinations and not others is downright weird because ARP cache should only be relevant to local devices, nothing beyond the next hop. Some ideas on possible problems - pure guesswork at this point because we don't have any real detail to work with:
- Person in the middle type attack - some device is redirecting traffic via another node on the network - hair-pinning the traffic - by hijacking the ARP entries, potentially for the router

(default gateway)

IP
- Proxy ARP enabled by mistake with a bad routing design resulting in your ARP cache trying to create an entry for every device on the internet - that would explain why clearing the ARP cache temporarily helps
- If your DHCP or devices have wrong default gateway configured then resulting in ICMP redirects to the correct gateway then your devices could start filling up with /32 routes to every IP on the internet.

Here's a truncated list of devices in the arp table there thousands of these entries that it shows devices outside of my network and beyond my next hop gateway.

ECH-ISR4431-138#show arp
Protocol Address Age (min) Hardware Addr Type Interface
Internet 1.34.163.232 1 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 2.57.121.229 28 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 2.180.35.216 67 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.0.126 168 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.1.2 218 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.1.162 131 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.2.123 37 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.2.176 191 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.2.202 163 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.2.216 94 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.3.139 71 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Protocol Address Age (min) Hardware Addr Type Interface
Internet 3.5.3.161 0 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.3.185 43 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.3.211 28 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.3.216 254 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.6.101 79 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.6.112 101 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.6.130 241 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.6.143 136 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.6.160 121 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.7.133 88 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.7.170 181 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Protocol Address Age (min) Hardware Addr Type Interface
Internet 3.5.7.203 36 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.8.19 25 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.8.106 254 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.8.160 133 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.9.11 166 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.9.134 140 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.9.171 254 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.10.138 192 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.10.150 97 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.10.151 144 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.10.180 118 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Protocol Address Age (min) Hardware Addr Type Interface
Internet 3.5.10.193 155 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.11.119 220 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.11.134 13 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.11.146 141 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.11.149 156 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.11.194 254 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.11.199 4 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.11.201 13 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.11.226 5 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.16.12 145 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.16.103 187 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Protocol Address Age (min) Hardware Addr Type Interface
Internet 3.5.16.172 82 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.17.165 139 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.17.221 208 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.19.141 44 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.20.19 12 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.20.205 108 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.20.215 126 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.21.122 109 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.21.148 27 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.21.183 169 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.21.204 45 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Protocol Address Age (min) Hardware Addr Type Interface
Internet 3.5.25.20 63 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.25.42 2 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.25.47 30 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.25.92 87 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.25.105 198 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.25.110 13 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.25.114 120 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.25.116 2 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.25.139 58 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.25.187 14 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.25.204 147 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Protocol Address Age (min) Hardware Addr Type Interface
Internet 3.5.25.205 254 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.25.229 224 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.25.231 200 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.25.242 153 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.27.104 170 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.27.119 38 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.27.135 91 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.27.141 190 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.27.142 129 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.27.156 126 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.27.163 100 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Protocol Address Age (min) Hardware Addr Type Interface
Internet 3.5.27.182 111 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.27.196 233 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.28.18 1 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.28.23 173 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.28.101 53 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.28.132 213 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.28.139 2 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.28.154 196 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.28.157 138 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.28.162 205 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.28.164 130 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2

This only started happening since upgrading from an ISR2921 to the ISR4335 and then the ISR4435. All I did was copied the config over from the previous 2921 to the upgraded devices....


sanitized version of the config

ECH-ISR4431-138#show runn
Building configuration...

Current configuration : 21034 bytes
!
! Last configuration change at 12:05:29 est Tue May 23 2023 by ehaylett
! NVRAM config last updated at 19:07:54 est Mon May 15 2023 by ehaylett
!
version 17.9
service timestamps debug datetime msec
service timestamps log datetime localtime
service call-home
platform qfp utilization monitor load 80
no platform punt-keepalive disable-kernel-core
platform hardware throughput level 1000000
!
hostname ECH-ISR4431-138
!
boot-start-marker
boot system bootflash:isr4400-universalk9.17.09.02a.SPA.bin
boot-end-marker
!
!
vrf definition Mgmt-intf
!
address-family ipv4
exit-address-family
!
address-family ipv6
exit-address-family
!
logging buffered informational
logging console informational
aaa new-model
!
!
aaa authentication login default local
aaa authorization console
aaa authorization exec default local
!
!
aaa session-id common
clock timezone est -4 0
clock calendar-valid
!
!
!
!
ip nbar http-services
!
!
!
!
!
ip name-server 68.237.161.12 71.250.0.12
ip ddns update method dyndns
HTTP
add http://xxxxxxxxx@members.dyndns.org/nic/updatesystem=dyndns&hostname=<h>&myip=<a>
interval maximum 0 0 5 0
!
ip dhcp excluded-address 172.168.100.0 172.168.100.24
ip dhcp excluded-address 172.168.120.0 172.168.120.1
!
ip dhcp pool ECH-NET-100
network 172.168.100.0 255.255.255.0
default-router 172.168.100.1
dns-server 68.237.161.12 71.250.0.12
lease 3
!
ip dhcp pool ECH-VOICE-NET-120
network 172.168.120.0 255.255.255.0
lease infinite
!
!
!
login on-success log
!
!
!
!
!
!
!
subscriber templating
!
!
!
!
!
multilink bundle-name authenticated
!
!
!
!
!
!
!
crypto pki trustpoint SLA-TrustPoint
enrollment pkcs12
revocation-check crl
!
crypto pki trustpoint TP-self-signed-3693526534
enrollment selfsigned
subject-name cn=IOS-Self-Signed-Certificate-3693526534
revocation-check none
rsakeypair TP-self-signed-3693526534
!
!
crypto pki certificate chain SLA-TrustPoint
certificate ca 01
30820321 30820209 A0030201 02020101 300D0609 2A864886 F70D0101 0B050030
32310E30 0C060355 040A1305 43697363 6F312030 1E060355 04031317 43697363
6F204C69 63656E73 696E6720 526F6F74 20434130 1E170D31 33303533 30313934
3834375A 170D3338 30353330 31393438 34375A30 32310E30 0C060355 040A1305
43697363 6F312030 1E060355 04031317 43697363 6F204C69 63656E73 696E6720
526F6F74 20434130 82012230 0D06092A 864886F7 0D010101 05000382 010F0030
82010A02 82010100 A6BCBD96 131E05F7 145EA72C 2CD686E6 17222EA1 F1EFF64D
CBB4C798 212AA147 C655D8D7 9471380D 8711441E 1AAF071A 9CAE6388 8A38E520
1C394D78 462EF239 C659F715 B98C0A59 5BBB5CBD 0CFEBEA3 700A8BF7 D8F256EE
4AA4E80D DB6FD1C9 60B1FD18 FFC69C96 6FA68957 A2617DE7 104FDC5F EA2956AC
7390A3EB 2B5436AD C847A2C5 DAB553EB 69A9A535 58E9F3E3 C0BD23CF 58BD7188
68E69491 20F320E7 948E71D7 AE3BCC84 F10684C7 4BC8E00F 539BA42B 42C68BB7
C7479096 B4CB2D62 EA2F505D C7B062A4 6811D95B E8250FC4 5D5D5FB8 8F27D191
C55F0D76 61F9A4CD 3D992327 A8BB03BD 4E6D7069 7CBADF8B DF5F4368 95135E44
DFC7C6CF 04DD7FD1 02030100 01A34230 40300E06 03551D0F 0101FF04 04030201
06300F06 03551D13 0101FF04 05300301 01FF301D 0603551D 0E041604 1449DC85
4B3D31E5 1B3E6A17 606AF333 3D3B4C73 E8300D06 092A8648 86F70D01 010B0500
03820101 00507F24 D3932A66 86025D9F E838AE5C 6D4DF6B0 49631C78 240DA905
604EDCDE FF4FED2B 77FC460E CD636FDB DD44681E 3A5673AB 9093D3B1 6C9E3D8B
D98987BF E40CBD9E 1AECA0C2 2189BB5C 8FA85686 CD98B646 5575B146 8DFC66A8
467A3DF4 4D565700 6ADF0F0D CF835015 3C04FF7C 21E878AC 11BA9CD2 55A9232C
7CA7B7E6 C1AF74F6 152E99B7 B1FCF9BB E973DE7F 5BDDEB86 C71E3B49 1765308B
5FB0DA06 B92AFE7F 494E8A9E 07B85737 F3A58BE1 1A48A229 C37C1E69 39F08678
80DDCD16 D6BACECA EEBC7CF9 8428787B 35202CDC 60E4616A B623CDBD 230E3AFB
418616A9 4093E049 4D10AB75 27E86F73 932E35B5 8862FDAE 0275156F 719BB2F0
D697DF7F 28
quit
crypto pki certificate chain TP-self-signed-3693526534
certificate self-signed 01
30820330 30820218 A0030201 02020101 300D0609 2A864886 F70D0101 05050030
31312F30 2D060355 04031326 494F532D 53656C66 2D536967 6E65642D 43657274
69666963 6174652D 33363933 35323635 3334301E 170D3233 30343039 31393530
34395A17 0D333330 34303831 39353034 395A3031 312F302D 06035504 03132649
4F532D53 656C662D 5369676E 65642D43 65727469 66696361 74652D33 36393335
32363533 34308201 22300D06 092A8648 86F70D01 01010500 0382010F 00308201
0A028201 0100A795 4597C3EC 40CF915E 48B27C42 BA61B5A8 B24A3E7E EFED37D8
6E47F36D 150E6532 C92C4F17 C4B628A3 3A218AD7 F458A71B 5964717F EAEA40BA
B11A7065 F62C1350 42262381 564873A2 5278A22F 5C1A6B46 70483C1D 97297847
F45454B7 D19AA687 4F760A37 F45CE895 38C02B4C A6305A7B C1C39166 6F3931C6
AE0BD754 5185EF16 CAD723C1 B1BEA4B9 1C6261F6 F571B9BD D8235C94 7ABAB454
92DC1CF0 806A1AE3 FFC08834 E8A9BC1F 59258BCF B026043F 03A0614A 76CB2A2A
329445BA E84FD4B9 DCA3ABAA 2A9F7FC0 D888CECE 5356F272 ACACD9AB BE3E020D
2D1DC9C7 CA9FAE01 9EDEDA0C CBC51BDF AE547421 9261B6E4 5B4038A8 D7DA7D7C
7E3F6242 B8350203 010001A3 53305130 0F060355 1D130101 FF040530 030101FF
301F0603 551D2304 18301680 148CB853 78168CFC AA6B7AB1 440168B7 C93A9C27
8D301D06 03551D0E 04160414 8CB85378 168CFCAA 6B7AB144 0168B7C9 3A9C278D
300D0609 2A864886 F70D0101 05050003 82010100 18C5D784 B651E4CB 0B50D86D
A13CB8FE 8F59E68D D92DD5E4 D2BF3DC0 BBAE1174 A7D4CD07 EE5D2E00 681DBD06
7E71B837 BAC3DB79 A9B6B391 73527FD8 993E3F3C 881DF055 03AF6320 B110FF9B
EC27BA3F 72061567 4A39C655 F2CE6AD0 CEAC86FB 04FC93C9 25CE11F2 E89D67A3
9B8CC7E9 8A186EA7 5C214C5B AA2DBA15 B749F18C D532BCF3 65F8887F B8289398
0BE343D5 F875E765 F555E5E5 88ED15C8 D7A195EB 2D186779 90450C10 C0BC51CD
52D18FD0 390BCA06 5111D00B DF2A81FD A77004C8 959D63BF 2DCE467B ED5509E4
39FAF74D 0198F99D 011DCBB3 CD9D19AA A146E090 5A27BDBE FB5A6B14 9E060C1E
E4396CDE 610B9A60 487EEC4D 8D854685 CD091677
quit
!
!
!
!
!
!
!
!
!
!
voice-card 0/4
no watchdog
!
license feature hseck9
license udi pid ISR4431/K9 sn xxxxxxxxx
license boot level uck9
license boot level securityk9
memory free low-watermark processor 62760
!
diagnostic bootup level minimal
!
spanning-tree extend system-id
!
!
enable secret 9 $14$z3Ao$N5UHHEg.eKsgXE$Lg1fr1pDklFcA00lLhYq1TmsdRyd765ki14ofSUpiMs
!
username xxxxxxx privilege 15 password 7 xxxxxxxxxxxx
username webui privilege 15 password 7 xxxxxxxxxxxxxx
!
redundancy
mode none
!
!
!
crypto ikev2 keyring ECH-ISR4431-138
peer ECH-ISR4331-138
address 162.84.130.90
pre-shared-key local xxxxx
pre-shared-key remote xxxxx
!
!
!
crypto ikev2 profile ECH-ISR4431-138_Profile
match identity remote address 162.84.130.90 255.255.255.255
authentication remote pre-share
authentication local pre-share
keyring local xxxxxxxxx
!
!
!
vlan internal allocation policy ascending
!
!
class-map match-all WEBUI-MULTIMEDIA_CONFERENCING-DSCP
match dscp af41
class-map match-all WEBUI-BROADCAST_VIDEO-NBAR
match protocol attribute traffic-class broadcast-video
match protocol attribute business-relevance business-relevant
class-map match-all WEBUI-VOICE-NBAR
match protocol attribute traffic-class voip-telephony
match protocol attribute business-relevance business-relevant
class-map match-all WEBUI-BULK_DATA-NBAR
match protocol attribute traffic-class bulk-data
match protocol attribute business-relevance business-relevant
class-map match-all WEBUI-SIGNALING-NBAR
match protocol attribute traffic-class signaling
match protocol attribute business-relevance business-relevant
class-map match-all WEBUI-NETWORK_CONTROL-DSCP
match dscp cs6
class-map match-all WEBUI-SCAVENGER-NBAR
match protocol attribute business-relevance business-irrelevant
class-map match-all WEBUI-SCAVENGER-DSCP
match dscp cs1
class-map match-all WEBUI-NETWORK_CONTROL-NBAR
match protocol attribute traffic-class network-control
match protocol attribute business-relevance business-relevant
class-map match-all WEBUI-SIGNALING-DSCP
match dscp cs3
class-map match-all WEBUI-BULK_DATA-DSCP
match dscp af11
class-map match-all WEBUI-BROADCAST_VIDEO-DSCP
match dscp cs5
class-map match-all WEBUI-MULTIMEDIA_CONFERENCING-NBAR
match protocol attribute traffic-class multimedia-conferencing
match protocol attribute business-relevance business-relevant
class-map match-all WEBUI-VOICE-DSCP
match dscp ef
class-map match-all WEBUI-NETWORK_MANAGEMENT-NBAR
match protocol attribute traffic-class ops-admin-mgmt
match protocol attribute business-relevance business-relevant
class-map match-all WEBUI-MULTIMEDIA_STREAMING-DSCP
match dscp af31
class-map match-all WEBUI-REALTIME_INTERACTIVE-NBAR
match protocol attribute traffic-class real-time-interactive
match protocol attribute business-relevance business-relevant
class-map match-all WEBUI-TRANSACTIONAL_DATA-DSCP
match dscp af21
class-map match-all WEBUI-REALTIME_INTERACTIVE-DSCP
match dscp cs4
class-map match-all WEBUI-TRANSACTIONAL_DATA-NBAR
match protocol attribute traffic-class transactional-data
match protocol attribute business-relevance business-relevant
class-map match-all WEBUI-NETWORK_MANAGEMENT-DSCP
match dscp cs2
class-map match-all WEBUI-MULTIMEDIA_STREAMING-NBAR
match protocol attribute traffic-class multimedia-streaming
match protocol attribute business-relevance business-relevant
class-map type inspect match-any Web_app
match protocol tcp
match protocol udp
match protocol ftp
match protocol icmp
class-map type inspect match-all Web
match class-map Web_app
match access-group name Web_acl
!
policy-map WEBUI-MARKING-IN
class WEBUI-VOICE-NBAR
set dscp ef
class WEBUI-BROADCAST_VIDEO-NBAR
set dscp cs5
class WEBUI-REALTIME_INTERACTIVE-NBAR
set dscp cs4
class WEBUI-MULTIMEDIA_CONFERENCING-NBAR
set dscp af41
class WEBUI-MULTIMEDIA_STREAMING-NBAR
set dscp af31
class WEBUI-SIGNALING-NBAR
set dscp cs3
class WEBUI-NETWORK_CONTROL-NBAR
set dscp cs6
class WEBUI-NETWORK_MANAGEMENT-NBAR
set dscp cs2
class WEBUI-TRANSACTIONAL_DATA-NBAR
set dscp af21
class WEBUI-BULK_DATA-NBAR
set dscp af11
class WEBUI-SCAVENGER-NBAR
set dscp cs1
class class-default
set dscp default
policy-map WEBUI-QUEUING-OUT
class WEBUI-VOICE-DSCP
priority percent 10
class WEBUI-BROADCAST_VIDEO-DSCP
priority percent 10
class WEBUI-REALTIME_INTERACTIVE-DSCP
priority percent 13
class WEBUI-NETWORK_CONTROL-DSCP
bandwidth percent 2
class WEBUI-SIGNALING-DSCP
bandwidth percent 2
class WEBUI-NETWORK_MANAGEMENT-DSCP
bandwidth percent 3
class WEBUI-MULTIMEDIA_CONFERENCING-DSCP
bandwidth percent 10
fair-queue
random-detect dscp-based
class WEBUI-MULTIMEDIA_STREAMING-DSCP
bandwidth percent 10
fair-queue
random-detect dscp-based
class WEBUI-TRANSACTIONAL_DATA-DSCP
bandwidth percent 10
fair-queue
random-detect dscp-based
class WEBUI-BULK_DATA-DSCP
bandwidth percent 4
fair-queue
random-detect dscp-based
class WEBUI-SCAVENGER-DSCP
bandwidth percent 1
class class-default
bandwidth percent 25
fair-queue
random-detect dscp-based
policy-map type inspect INSIDE-OUTSIDE-POLICY
class type inspect Web
inspect
class class-default
drop log
!
zone security INSIDE
description Zone for inside interfaces
zone security OUTSIDE
description Zone for outside interfaces
zone security default
zone-pair security INSIDE-OUTSIDE source INSIDE destination OUTSIDE
service-policy type inspect INSIDE-OUTSIDE-POLICY
!
!
!
!
!
!
!
!
crypto isakmp policy 10
encryption aes
hash sha256
authentication pre-share
group 5
crypto isakmp key xxxxxxxxxxx address 108.58.36.170
crypto isakmp invalid-spi-recovery
!
!
crypto ipsec transform-set ECH esp-aes esp-sha256-hmac
mode tunnel
crypto ipsec transform-set ECH-ISR4431-138 esp-aes 256 esp-sha512-hmac
mode tunnel
crypto ipsec df-bit clear
!
!
!
crypto map CMAP 1 ipsec-isakmp
set peer 108.58.36.170
set transform-set ECH
set pfs group5
match address VPN-TRAFFIC
!
crypto map ECH-ISR4431-138 1 ipsec-isakmp
set peer 162.84.130.90
set transform-set ECH-ISR4431-138
set ikev2-profile ECH-ISR4431-138_Profile
match address VPN-TRAFFIC
!
!
!
!
!
!
!
!
interface GigabitEthernet0/0/0
no ip address
shutdown
media-type rj45
negotiation auto
service-policy output WEBUI-QUEUING-OUT
!
interface GigabitEthernet0/0/1
no ip address
shutdown
media-type rj45
negotiation auto
service-policy output WEBUI-QUEUING-OUT
!
interface GigabitEthernet0/0/2
description WAN Outside
ip address dhcp
ip nbar protocol-discovery
ip nat outside
media-type rj45
negotiation auto
crypto map CMAP
service-policy input WEBUI-MARKING-IN
service-policy output WEBUI-QUEUING-OUT
!
interface GigabitEthernet0/0/3
no ip address
shutdown
media-type rj45
negotiation auto
service-policy output WEBUI-QUEUING-OUT
!
interface GigabitEthernet0/1/0
description ECH-CAT3560C-138
switchport trunk native vlan 100
switchport mode trunk
!
interface GigabitEthernet0/1/1
description ECH-CAP1852I-138A
switchport trunk native vlan 100
switchport mode trunk
!
interface GigabitEthernet0/1/2
description ECH-CAP1852I-138B
switchport trunk native vlan 100
switchport mode trunk
!
interface GigabitEthernet0/1/3
description ECH-CAP1815I-138C
switchport trunk native vlan 100
switchport mode trunk
!
interface GigabitEthernet0/1/4
description ECH-CAP1815I-138D
switchport trunk native vlan 100
switchport mode trunk
!
interface GigabitEthernet0/1/5
description ECH-CAP1815I-138xx
switchport access vlan 100
switchport trunk native vlan 100
switchport mode access
!
interface GigabitEthernet0/1/6
description Lutron Wireless Gateway
switchport access vlan 100
switchport mode access
!
interface GigabitEthernet0/1/7
description Verizon FIOS Set Top Box Gateway
switchport access vlan 140
switchport mode access
!
interface GigabitEthernet0/2/0
switchport access vlan 100
switchport trunk native vlan 100
spanning-tree portfast
!
interface GigabitEthernet0/2/1
switchport access vlan 100
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet0/2/2
switchport access vlan 100
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet0/2/3
switchport access vlan 100
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet0/2/4
switchport access vlan 100
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet0/2/5
switchport access vlan 100
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet0/2/6
switchport access vlan 100
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet0/2/7
switchport access vlan 100
switchport mode access
!
interface GigabitEthernet0/3/0
switchport access vlan 100
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet0/3/1
shutdown
!
interface GigabitEthernet0/3/2
shutdown
!
interface GigabitEthernet0/3/3
shutdown
!
interface GigabitEthernet0/3/4
shutdown
!
interface GigabitEthernet0/3/5
shutdown
!
interface GigabitEthernet0/3/6
shutdown
!
interface GigabitEthernet0/3/7
shutdown
!
interface Service-Engine0/4/0
!
interface GigabitEthernet0
vrf forwarding Mgmt-intf
no ip address
shutdown
negotiation auto
!
interface Vlan1
no ip address
shutdown
!
interface Vlan100
description ECH-NET-100 (Main Network)
ip address 172.168.100.1 255.255.255.0
ip nat inside
!
interface Vlan110
description ECH-NET-110 (Family Network)
ip dhcp relay source-interface Vlan110
ip address 172.168.110.1 255.255.255.0
ip nat inside
!
interface Vlan120
description ECH-VOICE-NET (Voice Network)
ip dhcp relay source-interface Vlan120
ip address 172.168.120.1 255.255.255.0
ip nat inside
!
interface Vlan138
description ECH-GUEST-NET (Guest Network)
ip dhcp relay source-interface Vlan138
ip address 172.168.138.1 255.255.255.0
ip nat inside
!
interface Vlan140
description ECH-NET-140 (Devices Network)
ip dhcp relay source-interface Vlan140
ip address 172.168.140.1 255.255.255.0
ip nat inside
!
interface Vlan150
description ECH-VPN-NET (AnyConnect VPN Network)
ip dhcp relay source-interface Vlan150
ip address 172.168.150.1 255.255.255.0
ip nat inside
!
ip http server
ip http authentication local
ip http secure-server
ip forward-protocol nd
ip tftp source-interface Vlan100
ip nat inside source route-map NAT_RMAP_1 interface GigabitEthernet0/0/2 overload
ip nat inside source route-map track-primary-if interface GigabitEthernet0/0/2 overload
ip route 0.0.0.0 0.0.0.0 GigabitEthernet0/0/2
!
!
ip access-list extended ECH_GUEST
10 remark Deny Guest VLAN138 access to other VLANs
10 deny ip any 172.168.100.0 0.0.0.255
20 deny ip any 172.168.101.0 0.0.0.255
30 deny ip any 172.168.102.0 0.0.0.255
40 deny ip any 172.168.103.0 0.0.0.255
50 deny ip any 172.168.110.0 0.0.0.255
60 deny ip any 172.168.120.0 0.0.0.255
70 deny ip any 172.168.140.0 0.0.0.255
80 deny ip any 172.168.150.0 0.0.0.255
90 permit ip any any
ip access-list extended NAT-ACL
10 deny ip any 172.168.140.0 0.0.0.255
ip access-list extended VPN-TRAFFIC
10 remark Site to Site VPN
10 permit ip 172.168.100.0 0.0.0.255 172.168.101.0 0.0.0.255
20 permit ip 172.168.100.0 0.0.0.255 172.168.102.0 0.0.0.255
30 permit ip 172.168.100.0 0.0.0.255 172.168.103.0 0.0.0.255
ip access-list extended Web_acl
10 permit ip any any
!
logging host 172.168.100.4
ip access-list extended 100
10 remark NAT_ACL
10 remark IPSec_Rule
10 deny ip 172.168.100.0 0.0.0.255 172.168.101.0 0.0.0.255
20 deny ip 172.168.100.0 0.0.0.255 172.168.102.0 0.0.0.255
30 deny ip any host 172.168.100.161
40 deny ip any host 172.168.100.162
50 deny ip any host 172.168.100.163
60 deny ip any host 172.168.100.164
70 deny ip any host 172.168.100.165
80 deny ip any host 172.168.100.166
90 deny ip any host 172.168.100.167
100 deny ip any host 172.168.100.168
110 deny ip any host 172.168.100.169
120 deny ip any host 172.168.100.170
130 permit ip 10.1.10.0 0.0.0.3 any
140 permit ip 172.168.100.0 0.0.0.255 any
150 permit ip 172.168.110.0 0.0.0.255 any
160 permit ip 172.168.120.0 0.0.0.255 any
170 permit ip 172.168.138.0 0.0.0.255 any
180 permit ip 172.168.140.0 0.0.0.255 any
190 permit ip 172.168.150.0 0.0.0.255 any
ip access-list extended 109
10 remark IPSec Rule
10 deny ip 172.168.100.0 0.0.0.255 172.168.110.0 0.0.0.255
20 deny ip 172.168.100.0 0.0.0.255 172.168.103.0 0.0.0.255
30 deny ip 172.168.100.0 0.0.0.255 172.168.102.0 0.0.0.255
40 deny ip 172.168.100.0 0.0.0.255 172.168.101.0 0.0.0.255
50 deny ip any 172.168.150.0 0.0.0.255
60 remark NAT Rule
60 permit ip 172.168.100.0 0.0.0.255 any
70 permit ip 172.168.110.0 0.0.0.255 any
80 permit ip 172.168.120.0 0.0.0.255 any
90 permit ip 172.168.138.0 0.0.0.255 any
100 permit ip 172.168.140.0 0.0.0.255 any
110 permit ip 172.168.150.0 0.0.0.255 any
!
route-map track-primary-if permit 1
match ip address 197
set interface GigabitEthernet0/0/2
!
route-map NAT_RMAP_1 permit 1
match ip address 100
match interface GigabitEthernet0/0/2
!
!
!
!
!
control-plane
!
!
mgcp behavior rsip-range tgcp-only
mgcp behavior comedia-role none
mgcp behavior comedia-check-media-src disable
mgcp behavior comedia-sdp-force disable
!
mgcp profile default
!
!
!
!
!
banner login ^CUnauthorized Access To This Device is Prohibited And Will Be Punishable By The Full Extent Of The Law^C
!
line con 0
stopbits 1
line aux 0
line vty 0 4
length 0
transport input ssh
line vty 5 14
transport input ssh
!
call-home
! If contact email address in call-home is configured as sch-smart-licensing@cisco.com
! the email address configured in Cisco Smart License Portal will be used as contact email address to send SCH notifications.
contact-email-addr sch-smart-licensing@cisco.com
profile "CiscoTAC-1"
active
destination transport-method http
ntp server ip time-a-wwv.nist.gov prefer source GigabitEthernet0/0/2
ntp server ip time-d-g.nist.gov source GigabitEthernet0/0/2
ntp server ip time-a-g.nist.gov source GigabitEthernet0/0/2
ntp server ip time-b-g.nist.gov source GigabitEthernet0/0/2
ntp server ip time-c-g.nist.gov source GigabitEthernet0/0/2
!
!
!
!
!
event manager applet noshut_port
event timer cron cron-entry "0 7 * * *"
action 010 cli command "enable"
action 020 cli command "config t"
action 030 cli command "interface Vlan110"
action 040 cli command "no shut"
action 050 cli command "end"
action 060 syslog msg "interface Vlan110 has been restored"
event manager applet ClearArp-0
event timer cron cron-entry "0 7 * * *"
action 010 cli command "clear arp"
action 020 syslog msg "Clear Arp Command Issued"
event manager applet ClearArp-1
event timer cron cron-entry "0 18 * * *"
action 010 cli command "clear arp"
action 020 syslog msg "Clear Arp Command Issued"
event manager applet shutdown_port
event timer cron cron-entry "30 2 * * *"
action 010 cli command "enable"
action 020 cli command "config t"
action 030 cli command "interface Vlan110"
action 040 cli command "shut"
action 050 cli command "end"
action 060 syslog msg "interface Vlan110 has been shutdown"

!
end

[Joseph W. Doherty]

Yup, that's how it should be done.

[Elito Haylett]

i removed the

route-map track-primary-if permit 1

[MHM Cisco World]

It can issue if you use route toward interface 

Sultion' ask ISP if you can use dhcp cleint and they can push

defualt route via dhcp op

Check NAT translate as I mention above.

[Elito Haylett]

Thank you for the clarity. I'll reach out to my ISP as well as check the NAt configuration again....

thanks

[paul driver · Accepted solution]

Hello
Your issue suggests its possibly due to your manual default static route, at present the rtr is ARP'ing for every ip address residing off your wan interface, you need to negate this from occurring.

no ip route 0.0.0.0 0.0.0.0 GigabitEthernet0/0/2
ip route 0.0.0.0 0.0.0.0 GigabitEthernet0/0/2 dhcp

[Elito Haylett · Accepted solution]

Hello Paul,

I did modify the config to reflect the second line....

thanks...

ECH

[MHM Cisco World]

and big thanks to @Flavio Miranda  he get the issue here. 
MHM

[Elito Haylett]

Exactly.. he did... and i did thank him for pointing me in the right direction.....