3650 proxy arp

[Josh Sprang]

Not sure if this belongs in switching for wireless.  I have a weird issue on a 3650 switch.  

-The network is all flat 192.168.1.0/24.  

-Switch interface VLAN is the default gateway of all PCs and wireless at @192.168.1.1.  

-Switch has default route to ASA @192.168.1.3

-3650 has a WLC loaded.  

Everything works except for iPhones running 9.1.3.  All other wired and wireless devices work, even iPhones running version 8.   iPhones join wireless fine and get correct default gateway of 192.168.1.1.  Switch can ping phone, but the ASA cannot.  Pinging the ASA from the phone responds and an ARP entry for the phone will appear on the ASA and the phone will be able to reach the internet.  Flushing ARP entry from the ASA will cause the phone to not be able to reach the internet.  The ASA will not have an ARP entry for the phone during this time.  Pinging the ASA causes an ARP entry and phone is able to reach the internet.   Enabling proxy ARP on the switch interface VLAN causes the issue to be fixed.  Packet caps on the ASA show traffic from the phone reaching the internet with no response.  I am stumped.  TAC wants a collaboration between the switching team and wireless team tomorrow but I would like to give the customer some answers.  Thanks

[Philip D'Ath]

What software version are you using on the 3650?

[Josh Sprang]

IOS-XE 03.06.04.E FC2     

I should also point out this customer has a private line to a provider that is routed and can be used as a backup internet through their equipment in the event of primary ISP failure.  When we fall over and use the backup internet over the hosted service everything works without proxy-arp configuration...   As I mentioned before all clients (wired and wireless)  are on the same 192.168.1.0/24 at the main site...  

clients------>3650----->ASA---->isp

         |

         |   <----(routed /30 with eigrp routing enabled for backup internet)

hosted service------>hosted service ASA----->hosted service ISP

[Philip D'Ath]

I think your software version is good.

The bit above sounds important.  How does the 3650 know which circuit to use?

Is the link between the 3650 and the ASA a point to point subnet, or part of a larger subnet with other machines?

[Josh Sprang]

Sorry for the late reply.  Local ASA advertises a default route which is preferred over remote ASA default route to the 3650 via EIGRP.  Note that everything except iphones running 9.1.3 work fine and everything uses the flat VLAN.  Local ASA/3650/wired and wireless clients are all in the same /24.  Local ASA plugs directly into 3650. 

Another /24 comes over the private leg in the hosted environment which allows the users to access their hosted servers.  

Thanks

[pwwiddicombe]

Playing with proxy-arp and (unknowingly) icmp-redirects can lead to wild and wonderful problems, particularly where the clients are all on the same network as multiple routers.

It's possible that, since the default gateway is the 1.1 address, that when it is directed to an alternate gateway on the same network, that the device will be sent an icmp-redirect (i.e. from the default gateway, to use the alternate gateway for THAT destination).  Unfortunately, many devices will then only go BACK to the intended default gateway when they receive another icmp-redirect, which never comes.

I have seen this with VMWare servers, particularly, where the WAN link goes down; and the WAN router then sends a redirect to it's default gateway, which is the Internet firewall (as the intended destination is unavailable temporarily).  From then on, the VMWare server obediently keeps sending traffic to all redirected hosts, to the firewall; who is quite happy to continue ignoring the traffic it wasn't supposed to carry in the first place.  Only recourse was to reboot the vmware server - with the real fix of putting clients/servers in their own network space; OR to disable icmp redirects (which had an impact on the WAN router handling all firewalled internet traffic).

[Josh Sprang]

Thanks for the response.  "no ip redirects" is and has been configured.  I am fully aware of problems this and proxy arp can cause, and how the components work, but thanks for the explanation.    This  why I want to figure out why the ASA will not receive ARP from this particular phone unless we enable proxy arp.    Any thoughts as to why this particular phone and software revision is causing this particular issue?    It is important to note that all other wireless/wired clients on the same subnet work fine with proxy arp disabled..