First connection to servers fail, then subsequent connections work, but then 15 Mins later same thing

[bascheew]

Ok Cisco community, I'm running out of ideas on this problem and I could use some direction. I have a scenario where hosts will stop communicating with other hosts on the LAN. For example with a web server, the first time a webpage is rquested the page cannot be displayed. The second and all subsequent requests work fine until it sits idle for about 10-15 minutes and then it all happens again. This is happening on a 3750x stack with a 2921 router for remote office routing and an ASA 5515x for edge security -- it's a very simple environment.

• This isn't just happening from one host to another, it appears that it's happening from all hosts to all hosts.
  
• This happens on both the physical and VMware virtual network.
  
• Connections that are external coming through NAT do not seem to have this problem.
  
• The problem happens across other VLANs as well as within the same broadcast domain. So that eliminates a routing issue.
  
• Before I connect to a server I checked the local ARP cache of a client and I see the entry for the destination host, then I checked the switch MAC address table and the entry was there. But upon making the first connection, the same issue persisted. This would lead me to believe that the issue is not ARP related since it was in the local cache when this happened.
  
• There are only about 250 MAC addresses in the table.
  
• I have tried IPs instead of hostnames with the same result. So that rules out DNS.
  
• I have confirmed the 3750x stack is the spanning tree bridge for all VLANs and I don't see any spanning tree events throughout the day.
  
• The switches CPU averages about 30% which from what I've seen is normal on a 3750x stack.
  

I'm grasping at straws at this point, so I'm open to any suggestions.

Thank you!

[paul driver]

Hello

Have you scanned your network for possible viruses pertaining to a potential DOS attacks?

Is this occurring between all hosts to server or host to host also?

Res
Paul

Sent from Cisco Technical Support iPad App

[bascheew]

This is happening from host to host as well.

If we keep a continuous ping going from a host to the server, then that client doesn't have the problem to that server as long as the ping is running.

We will check for a virus at the application layer, but it seems that a DoS attack or other storm would also be manifest on the switch as well.  I don't see any ports that are averaging over 5Mbits.

[bascheew]

I started some packet captures on the switch to see what was going on under the hood.  I found that the ASA was responding to ARP requests for internal servers.  After digging around as to why this would happen I found that "proxy ARP" was probably the feature that would cause this.  Sure enough "proxy ARP" was enabled on the inside interface.  Once this was disabled the ASA stopped responding to ARP requests for hosts on the inside interface.

So far the problem seems to be resolved, I'll report back on any new findings.