The problem is client users keep randomly getting disconnected and at this time they are unable to ping their gateway IP which exist on the router.
At the very moment when the ARP table on the router is checked the relevant ARP entry for that user who is unable to ping its gateway does not exist.
If we clear the ARP cache on the router the problem resolves and the clients are able to reach the gateway IP . But moments later the issue re-appears and stays put until the ARP cache is cleared again .
The setup involves EoGRE where the router is acting as a DHCP relay agent and has multiple EoGRE tunnel interfaces configured . The IP addresses of the tunnel interfaces are used as gateway IP for the clients across various mobile users who get access to the network via access points across multiple locations .
The tunnels is established between the access points and the router sitting at the Data center.
The issue happens for all mobile users connected via different tunnel interfaces and the symptoms remain the same .
we have also checked that during the issue when a new user in the network requests for an IP address , even if it gets the IP it is unable to ping its gateway IP ( tunnel IP for that connection) as soon as the ARP cache is cleared the the same mobile client is able to ping the gateway and access internet .
we have taken a packet capture on the router and have seen that when an mobile user that is unable to ping its gateway sends an ARP broadcast for its gateway IP , the router sometimes replies to the ARP requests and sometimes rejects the request and the same behavior was seen when ARP debug was enabled .
here is a truncated debug ARP log that was captured during the issue .
IP ARP: rcvd req src 10.9.142.21 00e0.2015.0fbc, dst 10.9.142.23 GigabitEthernet0/0/0
IP ARP: rcvd req src 0.0.0.0 741b.b26c.21bb, dst 10.122.37.253 Tunnel20
IP ARP req rejected src 0.0.0.0, dst 10.122.37.253 interface Tunnel20 is arp internal
IP ARP: rcvd req src 10.122.16.215 f48b.322b.4189, dst 10.122.16.1 Tunnel15
IP ARP req rejected src 10.122.16.215, dst 10.122.16.1 interface Tunnel15 is arp internal
IP ARP: sent rep src 10.122.16.1 0000.0000.0015, dst 10.122.16.215 f48b.322b.4189 Tunnel15
IP ARP: rcvd req src 10.122.37.201 c49a.0269.215d, dst 10.122.36.1 Tunnel20
IP ARP req rejected src 10.122.37.201, dst 10.122.36.1 interface Tunnel20 is arp internal
IP ARP: sent rep src 10.122.36.1 0000.0000.0020,dst 10.122.37.201 c49a.0269.215d Tunnel20
IP ARP: sent req src 10.122.36.1 0000.0000.0020,dst 10.122.39.138 0008.2290.cefb Tunnel20
IP ARP: sent req src 10.122.36.1 0000.0000.0020,dst 10.122.37.106 acc3.3a1e.fc4d Tunnel20
IP ARP: sent req src 10.122.0.1 0000.0000.0011,dst 10.122.0.203 7cc7.0963.6c33 Tunnel11
IP ARP: sent req src 10.122.0.1 0000.0000.0011,dst 10.122.2.114 0694.ed1f.000b Tunnel11
IP ARP: sent req src 10.122.0.1 0000.0000.0011,dst 10.122.2.90 0694.f8ff.000b Tunnel11
IP ARP: rcvd req src 10.122.16.254 1c7b.21ad.97fd, dst 10.122.16.1 Tunnel15
IP ARP req rejected src 10.122.16.254, dst 10.122.16.1 interface Tunnel15 is arp internal
IP ARP: sent rep src 10.122.16.1 0000.0000.0015,dst 10.122.16.254 1c7b.21ad.97fd Tunnel15
IP ARP: rcvd req src 10.122.36.125 0c41.3e8c.41ee, dst 10.122.36.125 Tunnel20
IP ARP req rejected src 10.122.36.125, dst 10.122.36.125 interface Tunnel20 is arp internal
IP ARP: rcvd rep src 10.122.2.114 0694.ed1f.000b, dst 10.122.0.1 Tunnel11
IP ARP rep rejected src 10.122.2.114, dst 10.122.0.1 interface Tunnel11 is arp internal
IP ARP: rcvd req src 10.122.22.33 f06b.cace.9375, dst 10.122.20.1 Tunnel16
IP ARP req rejected src 10.122.22.33, dst 10.122.20.1 interface Tunnel16 is arp internal
IP ARP: sent rep src 10.122.20.1 0000.0000.0016,dst 10.122.22.33 f06b.cace.9375 Tunnel16
IP ARP: rcvd req src 10.122.33.32 a470.d64e.6d0d, dst 10.122.32.1 Tunnel19
IP ARP req rejected src 10.122.33.32, dst 10.122.32.1 interface Tunnel19 is arp internal
IP ARP: sent rep src 10.122.32.1 0000.0000.0019,dst 10.122.33.32 a470.d64e.6d0d Tunnel19
Data center router is an ASR 1000 on release 15.4(2).
Has anyone else come across this issue recently or in the past ?
