Problem with ARP when moving ip to new failover pair

[manoman]

We are replacing an old Cisco ASA 5525x failover pair with a new FPR1150 failover pair.
Everything is setup and running side by side.
We cannot replace them in one go, so we have to move vlan by vlan.
The configuration is basically vlans with servers and corresponding NAT rules:

object network servername
 host #internal_ip#
object network servername
 nat (vlan112,outside) static #external_ip# dns

What we do:
First we remove all configuration for the vlans and IP addresses to be moved.
After that we insert new configuration for the same IP addresses in the new failover pair.

Problem:
I takes 3 hours before this works, due to the ARP timeout in the provider’s equipment.
When they clear the cache manually, it works immediately.

I thought that new ip/mac from the new failover pair would be sent as gratuitous arp and just work
I read that Cisco ASA does not send GARP for NATed Ips. Is this what (not) happens here?

What can I do to fix this? Or is there a work around to send this update so the ARP cache is updated?

Thanks,

[MHM Cisco World]

after you add IP shut/no shut the interface this will force ASA to send G-ARP

MHM

[manoman]

I will try that! Outside interface on new failover pair? Or the new inside interface? (Maybe a stupid question)

[MHM Cisco World]

any interface that you change it IP you need to shut/no shut otherwise the ASA not send G-ARP

MHM

[manoman]

I tried this on both the internal interface and outside. Did not help, unfortunately.

What else can I try?

[tvotna]

So far as I remember, "shut / no shut" doesn't send GARP, nor it is sent for a global IP address in a NAT rule. This issue exists for ages, but Cisco is reluctant to fix it. Indeed, this is not an AI or some other fancy feature which can be sold to customers, so why fix it?

CSCuy57115 ASA: GARP should be generated for interface SHUT/NO SHUT events
CSCsy85614 ASA should send gratuitous arp for configured global IP addresses

Use the following procedure to send GARP manually.

ASA/CONTEXT/pri/act(config)# sh run nat
nat (inside,outside) source static obj-1.2.3.4 obj-10.41.169.90

ASA/CONTEXT/pri/act(config)# debug menu ipaddrutl 5 10.41.169.90
Gratuitous ARP sent for 10.41.169.90

ASA/CONTEXT/pri/act(config)# sh cap cap1
1 packet captured
   1: 17:05:33.451804       802.1Q vlan#1999 P0 arp who-has 10.41.169.90 (a2:ae:65:0:0:1c) tell 10.41.169.90
1 packet shown

[manoman]

I get this result (I use my outside IP of course):

cisco# debug menu ipaddrutl 5 10.41.169.90
Gratuitous ARP not sent for 10.41.169.90

This was on the old ASA, since I have been moving a test IP back and forth.

[tvotna]

You can send GARP for any IP address taken from the connected subnet. If 10.41.169.90 doesn't match connected subnet on this firewall, the GARP is not sent.

[manoman]

So if the outside of the new firewall is 10.41.168.22 and the IP I am moving is 10.41.169.90, no GARP will be sent?
But it will be sent of the IP I am moving is 10.41.168.90?
We have IP:s on multiple subnets on the outside. Some will match and some won't.
(I will have to test this)

Is there any workaround or am I toast?

[manoman]

Tested this now and the GARP worked when the firewall and IP to be moved is on the same subnet.

cisco# debug menu ipaddrutl 5 10.41.169.90
Gratuitous ARP sent for 10.41.169.90

I could reach the IP directly after this.

So... how do I move all IP:s not on the same subnet? Is there a workaround or trick to make the new firewall send the GARP on the correct subnet?

[MHM Cisco World]

these three way I test.

using Ping to broadcast not work, it update the MAC in SW but not update the ARP in host.

[manoman]

Would it be possible to change the IP address of the firewall multiple times to one on each subnet containing IPs to be moved?
And then move the IPs on that subnet? VPN tunnels will get into trouble in a process like this, but is there something else that will cause problems? Will the already moved IPs be affected?

What about if I ask our provider to decrease the timeout from 3-4 hours to 5 minutes on our network during shorter periods.
Can this cause problems?

Please tell me that an easy solution exists.

[MHM Cisco World]

sorry can you more elaborate 
thanks 
MHM

[manoman]

Yes, of course. Basically I was thinking something like below steps, just to make the GARP work. I will still not be able to solve everything, but maybe some more.

Step 1- Set external IP of new firewall to IP on subnet 1.
Step 2 - Move IP addresses on subnet 1 (GARP will work)
Step 3 - Set external IP of new firewall to IP on subnet 2.
Step 4 - Move IP addresses on subnet 2 (GARP will work)

The question is - will this give me other problems I do not think of?

Second question is if it is possible to ask our provider to decrease the ARP timeout in the upstream equipment from 3-4 hours to 5 minutes on our network during shorter periods of migration or if this will cause other problems.

[tvotna]

All you need to do is to send GARP for your IP addresses from the connected subnet. This includes ASA interface IP address and NAT public IP addresses (if any) which belong to the connected subnet. For all other NAT IP addresses (which do not belong to the connected subnet) your provider should change routes to route traffic to these IPs trough the new firewall IP address. That's it.