General Network Failure

[davealessi]

I have a database server and a web server communicating under the ASA 5500 router.  The servers are connected to a hub, and then to the router.

I am intermittantly experiencing "General Networ Failure" type exceptions on the web server when he tries to access the database.  I have worked extensively with Microsoft on this issue, and their analysis is that the cisco router is causing the problem.

Server Side Trace

1531 10:00:05 AM 10/18/2010 146.3886710  192.168.1.10 192.168.1.50 ARP ARP:Request, 192.168.1.10 asks for 192.168.1.50
1532 10:00:05 AM 10/18/2010 146.3886710  192.168.1.50 192.168.1.10 ARP ARP:Response, 192.168.1.50 at 00-21-9B-8D-5F-84
1533 10:00:05 AM 10/18/2010 146.3886710 sqlservr.exe 192.168.1.10 192.168.1.50 TDS TDS:Response, Version = 7.300000(No version information available, using the default version), SPID = 60, PacketID = 1, Flags=...AP..., SrcPort=1433, DstPort=62929, PayloadLen=31, Seq=126375893 - 126375924, Ack=369473217, Win=64312 {TDS:5, TCP:4, IPv4:3}
1534 10:00:05 AM 10/18/2010 146.3896480  192.168.1.50 192.168.1.10 ARP ARP:Response, 192.168.1.50 at C4-7D-4F-80-F8-AF

Client side trace

3136 9:58:43 AM 10/18/2010 60.1155471  192.168.1.10 192.168.1.50 ARP ARP:Request, 192.168.1.10 asks for 192.168.1.50
3137 9:58:43 AM 10/18/2010 60.1155669  192.168.1.50 192.168.1.10 ARP ARP:Response, 192.168.1.50 at 00-21-9B-8D-5F-84

According to microsoft, the trace item #1531 is the database requesting the mac address of the web server.  The web server on 192.168.1.50 responds correctly in frame #1532 with its mac address.  Then, in frame #1534, another ARP response is received with the routers mac address.  MS says this is the issue.

Could the router possibly be responding to a ARP request for address 192.168.1.50?  The routers address is 192.168.1.1.

My ASA version is 7.2(4).  My ASDM version is 5.2(4).

[Jon Marshall]

Dave

What device owns this mac-address from 1534 - C4-7D-4F-80-F8-AF

Is is the router or the ASA. For both devices check for proxy-arp on the interfaces. If that is the problem it's not as straighforward as simply disabling it wihout knowing about your setup. NAT statements rely on proxy-arp and firewalls like the ASA often have proxy-arp enabled on the outside interface because without it non of the static NATs would work.

So we would need more info about the setup.

Edit - apologies, you have already said it is the router mac-address so check for proxy-arp on that interface.

Jon

[Kimberly Adams]

Dave,

Are you sure you are using a hub and not a switch?  If you were using a hub, the ASA wouldn't be in the mix due to broadcast on the hubs.  The two systems on a hub would broadcast to each other and would talk directly and not use the ASA.

If you would like to setup a capture on the ASA to track the traffic, let me know and I will help you setup a capture on the ASA.  The more information on the Network side of your network the better for us to help you find the issue.  Makes and models of the equipment is also helpful.

Thanks,

Kimberly

[davealessi]

My mistake...It is a switch. I think the model is shown below.

16-Port 10/100M Rack Mount Switch

How can I determine if the ASA has proxy arp enabled? Where can I see this in the GUI?

General info:

5 servers. All Dell. 4 running w2003. 1 running w2008. 2 Are database servers (SQL 2005). 2 are web servers (1 w2008, 1 w2003). The w2008 server has the problem. 1 development server.

All boxes connected with two adapters (A and B) to the switch. Switch connected to ASA. ASA connected to internet.

[Jon Marshall]

Dave

Just to confirm, is the mac-address being returned in 1534 the mac-address of the router or ASA. Whichever it is you need to check for proxy-arp on that.

Jon

[davealessi]

There is only one device...the ASA.

The mac address in 1534 is the address of the ASA.

Can you tell me how to check the proxy arp status on the asa. I can’t find it in the gui.

[Jon Marshall]

Dave

From config mode on ASA -

ASA# sh running-config sysopt

to disable proxy-arp on an interface -

ASA(config)# sysopt noproxyarp

but i would stress again that if it is enabled you should not just disable without understanding what it does ie. it could, depending on the interface and your NAT statements, stop traffic flowing.

Jon

[davealessi]

I disabled the proxy-apr on the inside interface, and tested to be sure that all the communications were working through the asa. This appears to have resolved my network issue. I am not sure if I can leave it disabled, as it may interfere with other routing that I do in the future.

Can you explain why I the router is responding to this arp request in the first place. The servers are communicating with one another on their own subnet, connected via a switch. I guess they do the arp request periodically. MS says the arp cache is refreshed every 10 minutes. Why would the asa ever get in the middle of this communication?

I need a permanent solution, and don’t know what to do.

[Jon Marshall]

davealessi wrote:
I disabled the proxy-apr on the inside interface, and tested to be sure that all the communications were working through the asa.  This appears to have resolved my network issue.  I am not sure if I can leave it disabled, as it may interfere with other routing that I do in the future.
Can you explain why I the router is responding to this arp request in the first place.  The servers are communicating with one another on their own subnet, connected via a switch.  I guess they do the arp request periodically.  MS says the arp cache is refreshed every 10 minutes.  Why would the asa ever get in the middle of this communication?
I need a permanent solution, and don’t know what to do.

Dave

If this is not interfering with any traffic flows then it is a permanent solution.

Basically an ASA needs proxy-arp to respond with a mac-address for IP addresses that are not actually assigned to it. So a typical setup on an ASA -

static (inside,outside) 177.10.10.10 192.168.5.10 netmask 255.255.255.255

means that on the inside interface of the ASA there is a device with an IP of 192.168.5.10. You want to present this to the outside (typically where the internet is) as 177.10.10.10.  So when a device (usually the upstream L3 router) arps for 177.10.10.10 the ASA has to respond to this arp and send back it's outside interface mac-address so the packet is sent to it. It can only do this if proxy-arp is enabled. If it was disabled then it would not respond to the arp request.

Now this is a typical thing to do on an ASA ie. present an inside private IP as an outside public IP so you will find that the vast majority of ASA firewalls have proxy-arp enabled on their outside interfaces.

However it is not a typical thing to want to do the opposite ie.

static (outside,inside) 192.168.5.10 177.10.10.10 netmask 255.255.255.255

which means present the outside address of 177.10.10.10 as 192.168.5.1 to the inside. This is a far less common setup. So usually there is no need to have proxy-arp enabled on the inside interface of the ASA. But it does depend on what NATs you are doing on your ASA. If no traffic is affected then i suspect you will be okay. Note also that you may never need the 2nd type of static statement on your firewall as it is a far less common need.

Jon

[davealessi]

Thanks for your reply.

I don't think I have a need for "case 2", where we present a public address to the local network. This is a web server that just responds to http requests. Also sends email, and one server sends ftp every night. All those functions appear to be working with proxy arp disabled.

I did not intentionally enable proxy arp when I configured this box. It must be the default configuration? If this option conflicts with normal server to server operation on a network, I don’t understand why it is on.

I have invested more hours that I can count on this network problem, and suffered customer interruptions as well. I'm guessing more than 40 hours working with Microsoft. I feel really burned here.

If Microsoft is correct that the arp table is cleared every 10 minutes, then this (arp request) is happening throughout the day. I don’t understand why sometimes it causes a problem, and other times not.

I may have a solution, but don't feel I understand why.

[gatlin007]

As Jon has noted something in your configuration, most likely associated with NAT was causing the ASA to proxy-arp for inside (trusted) addresses.  I don't advocate posting your firewall config here as it's a security issue.  If you hire a security engineer to evaluate your config the reason will surface as to why proxy-arp was occurring.  From my experience it's generally based on a 'over-reaching' NAT configuration.

Chris

[thamesreach]

Genius! That fixed the issue for me. It wasn't particularly service affecting since the internal host was due to be retired, but the ASA log was filling up with all these entries relating to just one host. Turns out it was a NAT rule causing it. As soon as I deleted it, all entries stopped appearing. It was as strange one though since when pinging the host, the first ping would fail (with the ASA answering on the ARP request with its MAC address) then all subsequent pings would work.

Anyway, that's a few months of on/off troubleshooting I don't have to do anymore.