It was a beautiful Saturday morning a few weeks ago. As I was enjoying a cup of coffee on the porch, I quickly realized it was time to get ready to go to work. It was an extremely hectic day with many high severity cases that rang in. One of them was very interesting that I decided to discuss it with everyone today.
In this case the problem was that the Iron-port server that was off the Email interface was not reachable from the outside.
Partial config on the ASA:
Static (Email,Outside) 14.36.10.138 10.10.10.10 net 255.255.255.255
Route Outside 0.0.0.0 0.0.0.0 14.36.10.142
- ASA's default route was pointing to R1
- Packets destined to 14.36.10.138 did not arrive on the ASA's outside interface. Verified that with the capture command:
- cap capout int outside match ip any host 14.36.10.138
- sh cap capout det
- I was able to ssh to the ASA's outside address. These packets arrived – obviously!
- All other statics configured worked as expected except for this IP address 14.36.10.138.
- Customer does not have access to R1.
- R1 belonged to the ISP and they put in a call to the ISP and no response from them.
- There is no way they will allow us to reboot R1.
So, how do we resolve this issue? My suggestion to change the translated IP address to another unused address quickly went out the window for obvious name resolution issues. ARP timeout is usually set for 14400 which is 4 hours. No one has the patience to wait it out.
Proxy arp: Well, the ASA proxy arps for all its global addresses, meaning, we will answer who we are if someone asked us. When an ARP request is made for the global IP address of the Iron-port server, 14.36.10.138, the security appliance responds with its own MAC address.
If the router were to ask the ASA, it would reply as below:
28: 04:09:44.939007 arp who-has 14.36.10.138 tell 14.36.10.142
29: 04:09:44.939053 arp reply 14.36.10.138 is-at 0:16:c7:9f:73:ef
Gratuitous arp: In this case we needed the ASA to send a gratuitous arp meaning we wanted the ASA to announce to everyone that it owns this IP address 14.36.10.138. An ARP announcement is not intended to solicit a reply; instead it updates any cached entries in the ARP tables of other hosts that receive the packet.
1: 03:38:35.873840 0016.c79f.73ef ffff.ffff.ffff 0x0806 42: arp who-has 14.36.10.138 tell 14.36.10.138
We don't do this for the global addresses that we own but only for the interface addresses. Once we get this enhancement defect resolved CSCsy85614 we would be able to remove the static line and add it back again and force the ASA to grat arp and R1 will automatically update its arp cache to the correct MAC address; assuming it wasn't programmed with a static arp for this IP address.
Now, with no visibility to R1 how could we (Cisco) solve this problem? How do we make R1 learn the ASA's outside interface MAC address for this IP address 14.36.10.138?
Fortunately, there were no tunnels terminated on this ASA. With their permission, I changed the ASA's outside interface IP address to 14.36.10.138 and pinged R1. Once I got a reply I knew R1 has learned the ASA's outside interface MAC for the IP address 14.36.10.138 and immediately changed the interface IP on the ASA outside interface back to what it was 14.36.10.132. Iron-port server was reachable from the internet. Packets arrived ! Problem solved!
Whew ! everyone was finally happy.
6 Comments
