Mac Flapping in a Cisco Mobility Express Network

[Elito Haylett]

Hello everyone I've read several post on theis forum regarding MAC Flapping in a Wireless Network but I haven't seen anything regarding Mobility Express and a single router with ethernet switch modules. When the issue occurs it causes some clients to not be able to communicate on the network or access the Internet. The only way I'm able to restore operation is to issue the "clear arp" command so I know this is definitely affecting the Arp Table.

The topology is a single location with 4 floors and an AP on each of the floors. The celing to the floor is about 13' so the AP's are approximately that far apart give or take a 1 feet or 2. There are no performance issues or dropped packets on the router which will indicate a loop.

I know it's defintiely a roaming issue caused by the clients cell phones as they move between floors as indicated in all the other posts. From what I read in the other posts I believe most of them was just messages they were seeing and no network issues were encountered. My issue is definitely causing me some headeaches because of the complaints. 

I'm including part of my configration which includes the switchport settings, the cdp neighbors as well as the error from the log so you can get an idea of what's going on.

May 21 11:00:00.362: %HA_EM-6-LOG: noshut_port: interface Vlan110 has been restored
May 21 07:00:02: %LINK-3-UPDOWN: Interface Vlan110, changed state to up
May 21 07:00:03: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan110, changed state to up
May 21 07:37:41: %SMART_LIC-3-COMM_FAILED: Communications failure with the Cisco Smart License Utility (CSLU) : Unable to resolve server hostname/domain name
May 21 07:58:44: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection from 113.196.82.10
May 21 08:03:11: %SW_MATM-4-MACFLAP_NOTIF: Host 925d.400d.0123 in vlan 110 is flapping between port Gi0/1/1 and port Gi0/1/2
May 21 08:37:50: %SMART_LIC-3-COMM_FAILED: Communications failure with the Cisco Smart License Utility (CSLU) : Unable to resolve server hostname/domain name
May 21 09:04:03: %SW_MATM-4-MACFLAP_NOTIF: Host b252.664d.1a87 in vlan 110 is flapping between port Gi0/1/1 and port Gi0/1/2
May 21 09:05:16: %SW_MATM-4-MACFLAP_NOTIF: Host a288.c712.3f89 in vlan 100 is flapping between port Gi0/1/4 and port Gi0/1/3
May 21 09:07:35: %SW_MATM-4-MACFLAP_NOTIF: Host a288.c712.3f89 in vlan 100 is flapping between port Gi0/1/4 and port Gi0/1/3
May 21 09:08:14: %SW_MATM-4-MACFLAP_NOTIF: Host b252.664d.1a87 in vlan 110 is flapping between port Gi0/1/1 and port Gi0/1/2
May 21 09:09:22: %SW_MATM-4-MACFLAP_NOTIF: Host a288.c712.3f89 in vlan 100 is flapping between port Gi0/1/4 and port Gi0/1/3
May 21 09:11:16: %SW_MATM-4-MACFLAP_NOTIF: Host b252.664d.1a87 in vlan 110 is flapping between port Gi0/1/1 and port Gi0/1/2
May 21 09:14:19: %SW_MATM-4-MACFLAP_NOTIF: Host a288.c712.3f89 in vlan 100 is flapping between port Gi0/1/4 and port Gi0/1/3
May 21 09:15:28: %SW_MATM-4-MACFLAP_NOTIF: Host a288.c712.3f89 in vlan 100 is flapping between port Gi0/1/4 and port Gi0/1/3
May 21 09:17:08: %SW_MATM-4-MACFLAP_NOTIF: Host a288.c712.3f89 in vlan 100 is flapping between port Gi0/1/4 and port Gi0/1/3
May 21 09:18:27: %SW_MATM-4-MACFLAP_NOTIF: Host a288.c712.3f89 in vlan 100 is flapping between port Gi0/1/4 and port Gi0/1/3
May 21 09:19:43: %SW_MATM-4-MACFLAP_NOTIF: Host a288.c712.3f89 in vlan 100 is flapping between port Gi0/1/4 and port Gi0/1/3
May 21 09:21:06: %SW_MATM-4-MACFLAP_NOTIF: Host a288.c712.3f89 in vlan 100 is flapping between port Gi0/1/4 and port Gi0/1/3
May 21 09:22:09: %SW_MATM-4-MACFLAP_NOTIF: Host a288.c712.3f89 in vlan 100 is flapping between port Gi0/1/4 and port Gi0/1/3
May 21 09:37:59: %SMART_LIC-3-COMM_FAILED: Communications failure with the Cisco Smart License Utility (CSLU) : Unable to resolve server hostname/domain name
May 21 09:42:12: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection from 113.196.82.10
May 21 10:32:44: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection from 113.196.82.10
May 21 10:38:08: %SMART_LIC-3-COMM_FAILED: Communications failure with the Cisco Smart License Utility (CSLU) : Unable to resolve server hostname/domain name
May 21 10:38:44: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection from 113.196.82.10
May 21 10:56:32: %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: ehaylett] [Source: 172.168.100.53] [localport: 22] at 10:56:32 est Sun May 21 2023
May 21 11:01:40: %SYS-6-LOGOUT: User ehaylett has exited tty session 867(172.168.100.53)
May 21 11:04:32: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection from 113.196.82.10
May 21 11:05:11: %IOSXE-3-PLATFORM: R0/0: cpp_cp: QFP:0.0 Thread:005 TS:00003326382879648999 %IPSEC-3-REPLAY_ERROR: IPSec SA receives anti-replay error, DP Handle 9, src_addr 108.58.36.170, dest_addr 98.113.183.177, SPI 0xfa1d2ef0
May 21 11:07:30: %SW_MATM-4-MACFLAP_NOTIF: Host dadb.a644.d834 in vlan 110 is flapping between port Gi0/1/2 and port Gi0/1/1
May 21 11:12:01: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection from 113.196.82.10
May 21 11:16:53: %SW_MATM-4-MACFLAP_NOTIF: Host dadb.a644.d834 in vlan 110 is flapping between port Gi0/1/1 and port Gi0/1/2
May 21 11:18:45: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection from 113.196.82.10
May 21 11:20:11: %IOSXE-3-PLATFORM: R0/0: cpp_cp: QFP:0.0 Thread:001 TS:00003327282878227079 %IPSEC-3-REPLAY_ERROR: IPSec SA receives anti-replay error, DP Handle 7, src_addr 108.58.36.170, dest_addr 98.113.183.177, SPI 0xcead5763
May 21 11:25:59: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection from 113.196.82.10
May 21 11:31:20: %SW_MATM-4-MACFLAP_NOTIF: Host a288.c712.3f89 in vlan 100 is flapping between port Gi0/1/2 and port Gi0/1/1
May 21 11:33:35: %SW_MATM-4-MACFLAP_NOTIF: Host 925d.400d.0123 in vlan 110 is flapping between port Gi0/1/1 and port Gi0/1/2
May 21 11:35:11: %IOSXE-3-PLATFORM: R0/0: cpp_cp: QFP:0.0 Thread:002 TS:00003328183926907659 %IPSEC-3-REPLAY_ERROR: IPSec SA receives anti-replay error, DP Handle 7, src_addr 108.58.36.170, dest_addr 98.113.183.177, SPI 0xcead5763
May 21 11:35:42: %SW_MATM-4-MACFLAP_NOTIF: Host dadb.a644.d834 in vlan 110 is flapping between port Gi0/1/1 and port Gi0/1/2
May 21 11:38:17: %SMART_LIC-3-COMM_FAILED: Communications failure with the Cisco Smart License Utility (CSLU) : Unable to resolve server hostname/domain name
May 21 11:47:04: %SW_MATM-4-MACFLAP_NOTIF: Host dadb.a644.d834 in vlan 110 is flapping between port Gi0/1/1 and port Gi0/1/2
May 21 11:48:18: %SW_MATM-4-MACFLAP_NOTIF: Host dadb.a644.d834 in vlan 110 is flapping between port Gi0/1/1 and port Gi0/1/2
May 21 11:50:11: %IOSXE-3-PLATFORM: R0/0: cpp_cp: QFP:0.0 Thread:004 TS:00003329083924260342 %IPSEC-3-REPLAY_ERROR: IPSec SA receives anti-replay error, DP Handle 7, src_addr 108.58.36.170, dest_addr 98.113.183.177, SPI 0xcead5763
May 21 12:01:08: %SW_MATM-4-MACFLAP_NOTIF: Host dadb.a644.d834 in vlan 110 is flapping between port Gi0/1/1 and port Gi0/1/2
May 21 12:01:36: %SW_MATM-4-MACFLAP_NOTIF: Host a288.c712.3f89 in vlan 100 is flapping between port Gi0/1/2 and port Gi0/1/1
May 21 12:05:11: %IOSXE-3-PLATFORM: R0/0: cpp_cp: QFP:0.0 Thread:001 TS:00003329983902061698 %IPSEC-3-REPLAY_ERROR: IPSec SA receives anti-replay error, DP Handle 9, src_addr 108.58.36.170, dest_addr 98.113.183.177, SPI 0x36207491
May 21 12:07:37: %SW_MATM-4-MACFLAP_NOTIF: Host deea.1a6c.ae1a in vlan 100 is flapping between port Gi0/1/1 and port Gi0/1/2
May 21 12:09:07: %SW_MATM-4-MACFLAP_NOTIF: Host a288.c712.3f89 in vlan 100 is flapping between port Gi0/1/2 and port Gi0/1/1
May 21 12:09:52: %SW_MATM-4-MACFLAP_NOTIF: Host a288.c712.3f89 in vlan 100 is flapping between port Gi0/1/1 and port Gi0/1/2
May 21 12:09:58: %SW_MATM-4-MACFLAP_NOTIF: Host a288.c712.3f89 in vlan 100 is flapping between port Gi0/1/1 and port Gi0/1/2
May 21 12:10:27: %SW_MATM-4-MACFLAP_NOTIF: Host a288.c712.3f89 in vlan 100 is flapping between port Gi0/1/1 and port Gi0/1/2

interface GigabitEthernet0/1/0
description ECH-CAT3560C-138
switchport trunk native vlan 100
switchport mode trunk
!
interface GigabitEthernet0/1/1
description ECH-CAP1852I-138A
switchport trunk native vlan 100
switchport mode trunk
!
interface GigabitEthernet0/1/2
description ECH-CAP1852I-138B
switchport trunk native vlan 100
switchport mode trunk
!
interface GigabitEthernet0/1/3
description ECH-CAP1815I-138C
switchport trunk native vlan 100
switchport mode trunk
!
interface GigabitEthernet0/1/4
description ECH-CAP1815I-138D
switchport trunk native vlan 100
switchport mode trunk
!
interface GigabitEthernet0/1/5
description ECH-CAP1815I-138xx
switchport access vlan 100
switchport trunk native vlan 100
switchport mode access

ECH-ISR4431-138#sho cdp neigh
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone,
D - Remote, C - CVTA, M - Two-port Mac Relay

Device ID Local Intrfce Holdtme Capability Platform Port ID
ECH-CAT3560C-138 Gig 0/1/0 152 S I WS-C3560C Gig 0/1
ECH-CAP1852I-138-A Gig 0/1/1 117 R T AIR-AP185 Gig 0
ECH-CAP1852I-138-B Gig 0/1/2 166 R T AIR-AP185 Gig 0
ECH-CAP1815I-138-C Gig 0/1/3 159 R T AIR-AP181 Gig 0
ECH-CAP1815I-138-D Gig 0/1/4 166 R T AIR-AP181 Gig 0

Total cdp entries displayed : 5

Please Help... Thanks

[marce1000]

 - Basically the MAC Flapping is normal when clients are roaming , for the mobility express controller consider https://www.cisco.com/c/en/us/support/docs/wireless/wireless-lan-controller-software/200046-tac-recommended-aireos.html  and or upgrade accordingly , if applicable , 

 M.

[Elito Haylett]

thanks for the reply and the link. I am running the latest release 8.10.185.0 on all the AP's. I'll read through the document to see if this issue is addressed in it.

[Rich R · Accepted solution]

I don't think we have enough info to answer.  MAC flapping is only at layer 2 on the switch (as the device moves between switch ports) but if the MAC still stays in the same VLAN that should never affect the ARP cache.  If devices are moving between SSIDs (into different VLANs) then your ARP cache timer should not be longer than your DHCP lease time otherwise that might cause issues if a new device gets an IP address for which you still have an old ARP cache entry pointing to an old device MAC which has already left.
Where do you clear the ARP cache to clear the problem?
What does the ARP cache look like before and after clearing for that device?

[Elito Haylett]

Hello Richard,

Thanks for your reply. Two different devices, moving within their respected VLANS associated to the SSID of their VLANS. The log messages shows them moving between floors within a few minutes of each other. I didn't mess with any of the default timers for ARP on the router. For DHCP I have the lease period of 3 days. The device is a Cisco Router ISR-4431 with Gig Ethernet switch module for layer 2 connectivity. That's where I'm issuing the "clear arp" command. Looking at the arp table it has over a thousand or more entries in the table. I never thought to capture it when the problem takes place so I don't know what it looks like.

ECH 

[Elito Haylett]

The issue happened again about an hour ago and I was able to check the arp cache prior to and after and there were entries in the table. Everytime it happens it affects both wired and wireless clients. I'm connected to the network with via ethernet and when i tried to browse certain sites it timed out and other sites were responsive but slow loading. I tried to debug arp on the router to see if any messages would be displayed but there were none. I checked the interfaces the AP's are connected to for errors and there were none. The CPU utilization on the router didn't even exceed User and System 10%. After I cleared the arp cache, a few seconds later every site that I couldn't browse to before started loading without any problems. I eliminated one of the AP's from one of the floors to see if there will be less hopping from one to another but the problem still exist. I'll try to eliminate another AP on Friday to have them spaced further apart to see if this will solve the problem but then sacrificing coverage which may lead to other complaints. Some how I have to do the process of elimination to try to pinpoint the issue and work from there. 

[Rich R]

> "it affects both wired and wireless clients"
Then it's not a wireless problem - it's a switching or routing problem.
> "I was able to check the arp cache prior to and after and there were entries in the table"
And what were those entries?

The fact that it can affect some destinations and not others is downright weird because ARP cache should only be relevant to local devices, nothing beyond the next hop.  Some ideas on possible problems - pure guesswork at this point because we don't have any real detail to work with:
- Person in the middle type attack - some device is redirecting traffic via another node on the network - hair-pinning the traffic - by hijacking the ARP entries, potentially for the router (default gateway) IP
- Proxy ARP enabled by mistake with a bad routing design resulting in your ARP cache trying to create an entry for every device on the internet - that would explain why clearing the ARP cache temporarily helps
- If your DHCP or devices have wrong default gateway configured then resulting in ICMP redirects to the correct gateway then your devices could start filling up with /32 routes to every IP on the internet.

[Elito Haylett]

here's a truncated list of devices in the arp table:

ECH-ISR4431-138#show arp
Protocol Address Age (min) Hardware Addr Type Interface
Internet 1.34.163.232 1 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 2.57.121.229 28 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 2.180.35.216 67 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.0.126 168 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.1.2 218 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.1.162 131 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.2.123 37 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.2.176 191 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.2.202 163 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.2.216 94 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.3.139 71 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Protocol Address Age (min) Hardware Addr Type Interface
Internet 3.5.3.161 0 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.3.185 43 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.3.211 28 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.3.216 254 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.6.101 79 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.6.112 101 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.6.130 241 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.6.143 136 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.6.160 121 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.7.133 88 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.7.170 181 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Protocol Address Age (min) Hardware Addr Type Interface
Internet 3.5.7.203 36 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.8.19 25 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.8.106 254 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.8.160 133 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.9.11 166 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.9.134 140 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.9.171 254 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.10.138 192 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.10.150 97 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.10.151 144 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.10.180 118 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Protocol Address Age (min) Hardware Addr Type Interface
Internet 3.5.10.193 155 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.11.119 220 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.11.134 13 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.11.146 141 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.11.149 156 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.11.194 254 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.11.199 4 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.11.201 13 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.11.226 5 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.16.12 145 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.16.103 187 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Protocol Address Age (min) Hardware Addr Type Interface
Internet 3.5.16.172 82 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.17.165 139 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.17.221 208 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.19.141 44 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.20.19 12 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.20.205 108 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.20.215 126 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.21.122 109 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.21.148 27 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.21.183 169 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.21.204 45 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Protocol Address Age (min) Hardware Addr Type Interface
Internet 3.5.25.20 63 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.25.42 2 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.25.47 30 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.25.92 87 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.25.105 198 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.25.110 13 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.25.114 120 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.25.116 2 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.25.139 58 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.25.187 14 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.25.204 147 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Protocol Address Age (min) Hardware Addr Type Interface
Internet 3.5.25.205 254 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.25.229 224 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.25.231 200 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.25.242 153 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.27.104 170 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.27.119 38 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.27.135 91 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.27.141 190 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.27.142 129 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.27.156 126 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.27.163 100 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Protocol Address Age (min) Hardware Addr Type Interface
Internet 3.5.27.182 111 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.27.196 233 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.28.18 1 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.28.23 173 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.28.101 53 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.28.132 213 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.28.139 2 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.28.154 196 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.28.157 138 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.28.162 205 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2
Internet 3.5.28.164 130 f01c.2d8c.f6ce ARPA GigabitEthernet0/0/2

thousands of these entries so what you're saying is correct that it shows devices beyond my next hop gateway. Thanks for pointing me in the right direction and possibly isolating it to a routing issue. I have to check my routing config because we never experience this in the past only since upgrading to the ISR4335 and then the ISR4435. All I did was copied the config over from the previous 2921 to the upgraded devices....

I'll troubleshoot further and move this to the routing community for further assistance..... Thanks for your help. 

[Rich R]

Without knowing details of your network and devices I can't really say whether that ARP cache looks good or not but what matters is any differences before/after clearing it.  So capture the full ARP cache before clearing and then a few minutes after and then compare them.  Obviously some entries will disappear if the devices are no longer active but if you see a large number disappear then look at what those are and why they were there.  If you see entries that have changed then look closer at them and why they changed - might give some clue to what's wrong.  You might also want to look at the mac-address-table on the VLAN before and after to see whether that looks correct.

[Elito Haylett]

Hi Rich,

providing an update after several days of stability in the network. As mentioned in my prior post, thank you for pointing me in the right direction. It turned out the problem wasn't a layer 2 issue but a layer 3 and more specifically with proxy-arp. The default route was configured as "ip route 0.0.0.0 0.0.0.0 int g0/0/0" instead of a more specific address. After I changed the command to "ip route to 0.0.0.0 0.0.0.0 dhcp" all the external macs disappeared and only the interfaces and clients in the network were in the arp table. There are no more issues with network outages or browsing so again Thank you very much.....

Regards,

ECH

[Rich R · Accepted solution]

Common mistake - glad you worked it out in the end.
As a general rule you should never give an interface instead of a next hop IP for a route on a LAN.  It's fine on a point to point link like a dialer interface (and maybe even a point to point ethernet) but on a regular LAN is a recipe for disaster as you discovered.  Sometimes you may want to qualify a next hop IP with an interface as well, which says that route can only be reached via that IP through that interface (not a recursive lookup via another interface), and that's ok because you're making it more not less restrictive.