Have to clear arp to get DMZ host to respond

[the-lebowski]

I have an issue with a DMZ host (vm connected to un-routed DMZ vlan) and its reach ability via the internet.  

Like so:

internet router (HSRP) > DMZ switch > Transparent FW > un-routed VLAN on core (HSRP) > trunk to ESXi hosts

One vm on the 'public' vlan is configured with a public IP address (No NAT) started having issues.  It will respond to traffic for a couple minutes then it will stop.  If I clear the arp cache on the internet router (running HSRP) it comes back to life but after a minute or so it stops responding again.   HSRP hasn't changed in a long time (3+ yrs) and no other network changes that I am aware of.  

On the vm itself I do not see an arp entry for the edge router hsrp when its broken.  I do see it after I clear the arp on the edge router that is until it stops working again.   What is strange is thatI can reach the host via public IP from the LAN when the arp issue is occurring, but it always breaks for packets to and from the internet.  

Not working:

[host@host ~]$ arp

Address                  HWtype  HWaddress           Flags Mask            Iface

ns2                         ether   00:30:48:9f:f9:61   C                     eth0

fw1                         ether   e8:b7:48:3c:f3:40   C                     eth0

r2                            ether   e8:b7:48:3c:f3:40   C                     eth0

Working:

[host@host~]$ arp

Address                  HWtype  HWaddress           Flags Mask            Iface

ns2                         ether   00:30:48:9f:f9:61   C                     eth0

r3-hsrp                   ether   30:e4:db:55:a6:81   C                     eth0

fw1                         ether  e8:b7:48:3c:f3:40   C                     eth0

r2                            ether   00:00:0c:07:ac:65   C                     eth0

I have other hosts almost identical to this one with no issue but this vm in particular just refuses to remain reachable.   There is no duplicate IP, do duplicate MAC and I cannot seem to figure out what the problem is.  I tried changing the IP, NAT'ing, vmotioning to another host (same host as other dmz hosts reside on) with no joy.  

[Jon Marshall]

What device is r2 ?

Some comments on your output -

1) when it is not working r2 has the same mac address as the firewall whereas when it is working r2 has a different mac address

2) when it is working r2 has the HSRP VIP mac address and not the r3-hrsp entry ie. HSRP VIP mac addresses use 00:00:0c:07:ac:xx

Something is obviously not right and it is to do with the arp cache on the server as far as I can see.

Can you compare the cache to a server that is working.

If there is a difference then obviously check the IP settings on the server but then look at firewall as well.

Somehow that server is getting completely the wrong mac addresses or that is what it looks like.

It's late where I am so i'll check this thread tomorrow if nobody else has answered in the meantime.

Jon

[the-lebowski]

Its fixed:  Itwas a static NAT statement doing a proxy arp on the routed FW.  I edited all NAT statements on the routed FW to specific interfaces (inside,outside) and updated a recently added NAT statement to include no-proxy-arp route lookup which solved the issue.

r2 is the other HSRP internet router.  

1. That is correct and fw1 is actually the routed firewall.  So traffic leaving the LAN will traverse that fw to get to the server in question.  I do see fw1 MAC address for almost every public ip in our space on r3's arp table.  So maybe that is what is causing the issue? fw1 responding on its behalf and thus not being able to route the traffic?  

2. Correct.

I agree MAC addressing is messed up but I don't know how.  

What is strange is that when its still working the APR entries change on the client to drop r3 and get populated with fw1 mac addresses.   

fw1 e8:b7:48:3c:f3:40   
r2  00:00:0c:07:ac:65   
r2  00:00:0c:07:ac:65  

fw1 e8:b7:48:3c:f3:40   
r2  e8:b7:48:3c:f3:40   
r2  e8:b7:48:3c:f3:40