Problem with ARP when moving ip to new failover pair

[manoman]

We are replacing an old Cisco ASA 5525x failover pair with a new FPR1150 failover pair.
Everything is setup and running side by side.
We cannot replace them in one go, so we have to move vlan by vlan.
The configuration is basically vlans with servers and corresponding NAT rules:

object network servername
 host #internal_ip#
object network servername
 nat (vlan112,outside) static #external_ip# dns

What we do:
First we remove all configuration for the vlans and IP addresses to be moved.
After that we insert new configuration for the same IP addresses in the new failover pair.

Problem:
I takes 3 hours before this works, due to the ARP timeout in the provider’s equipment.
When they clear the cache manually, it works immediately.

I thought that new ip/mac from the new failover pair would be sent as gratuitous arp and just work
I read that Cisco ASA does not send GARP for NATed Ips. Is this what (not) happens here?

What can I do to fix this? Or is there a work around to send this update so the ARP cache is updated?

Thanks,

[manoman]

Yes, I know that one solution is to ask the provider to empty the cache which will make the traffic find its way to the new friewall. They probably won't be happy to do this whenever I need to move some IPs. I cannot move all at the same time, unfortunately.

[tvotna]

I didn't say that ISP needs to clear ARP. Do all of your #external_ip#s currently belong to the same subnet or few different subnets? If they all belong to the same subnet, is it the same subnet as the subnet of the outside interface?

If all of the external IPs are taken from the outside interface subnet, you need to migrate them all at once (create new NAT rules, change outside interface IP address and send GARP from the firewall). You cannot migrate rules step by step.

If external IPs are taken from some other subnet, not the one which is assigned to the outside interface, it is IP routing on the ISP side that is responsible for traffic delivery, and not ARP. In this case, in order to deliver packets to the new outside interface IP address, ISP needs to change routes to reach those external IPs. And ARP is only needed here to reach firewall's outside interface IP.

object network servername
 host #internal_ip#
object network servername
 nat (vlan112,outside) static #external_ip#

I hope this makes sense.

[manoman]

I can migrate rules one by one, or almost. I just need to take one internal vlan at a time.
Works fine except for the ARP timeout problem.

[MHM Cisco World]

there are two subnet use by same interface and same IP ?

MHM

[manoman]

Yes. For example:
10.10.10.0/24
10.10.11.0/24

Firewall has IP on 10.10.11.0/24

Firewall handles IPs on both subnets

When I move IP on 10.10.11.0/24, manual GARP works
When I move IP on 10.10.10.0/24, manual GARP does not work

I now understand why. I need to find a way to move these IPs some other way.
After the ARP cache timeout the IPs work, but it is too long. Are there any risks for the provider to lower this during shorter migration periods?

[MHM Cisco World]

If you can ask them why short the time instead make them clear arp for interface connect to your ASA

MHM

[manoman]

I will be doing this during the night and then it would be easier to make them decrease this value the day before. I cannot ask them to stay up to do this. The move will be done in multiple steps too, during multiple nights. If I could make them clear arp whenever I wanted, I would do it like that.

[MHM Cisco World]

there is two issue, hope we can solve it in one step 
1- ARP for NAT (all NAT use no-proxy so I dont think this issue for MAC-IP table)
2- ARP for new FW interface using the IP of Old ASA, this I think alot about it we need to make SW-Router-Hosts update there MAC-IP, shut/no shut not send G-ARP as we test, so can you try 
ping <subnet broadcast> <<- subnet broadcast for example the interface  have 10.0.0.0/24 the broadcast will be 10.0.0.255,
this make all device in same subnet KNOW the new MAC of new FW interface.
try this hope it work for you

MHM

[tvotna]

@MHM Cisco World, "all NAT use no-proxy"? This is funny. How would static NAT work if its global IP is borrowed from the outside interface subnet?