Cisco ASA Dropping Packets to Specific VLAN

[AMD_GAMER]

Have a bizarre problem I cannot put my finger on. Below is the following configuration:

Core Switch: Cisco Nexus 7K running 6.1.1 - Address 10.0.0.1

ASA Model: Cisco ASA 5510 running 8.4.5 - Address 10.0.0.18 (directly connected to Nexus)

Both devices are on VLAN10 which has a scope of 10.0.0.0/22. All of our servers are on VLAN10 and have their default gateway to the Nexus (10.0.0.1). All of the IP routes exist on the Nexus. We have about 20 different locations (each with their own subnet) that terminate to the Nexus either via fiber or hosted WAN. These connections are all Layer3 routed.

The problem I am having is that I am getting about 7-12% packet loss when traffic comes from 10.0.0.0/22 and is destined for a remote Site-to-Site VPN that is behind the ASA. The Nexus has routes to all the remote sites. The packet loss can be seen from pings, SMB traffic, HTTP traffic, and SSL. Also, I am only able to get about 100KB/sec through the VPN tunnel. However, from another location outside the datacenter (Ex: 10.1.0.0/23), I can transfer to the remote site with 20 mbit throughput and 0 dropped packets. Therefore, the issue is that only VLAN10 traffic is slow to the remote sites. Would having the ASA's inside interface on the same network be causing the issue? I can't believe it is a VPN config problem as I can get 20 mbit through the tunnel from other IP scopes on the network. Any ideas....I am stumped.

[Marius Gunnerud]

if you issue the command show crypto ipsec sa do you see approx. the same number of encrypted as decrypted packets? 

is there a large amount of send and/or receive errors?

What MTU do you have configured for the ASA interfaces?  Try setting the MTU to 1360

Run a packet tracer with the source IP of a local server and source interface the server subnet interface with destination outside interface.  Might be that the traffic is hitting the wrong NAT statement.

--

Please remember to select a correct answer and rate helpful posts

[AMD_GAMER]

Nope. I don't see any errors.

 #pkts encaps: 8631, #pkts encrypt: 8631, #pkts digest: 8631
      #pkts decaps: 8249, #pkts decrypt: 8249, #pkts verify: 8249
      #pkts compressed: 0, #pkts decompressed: 0
      #pkts not compressed: 8631, #pkts comp failed: 0, #pkts decomp failed: 0
      #pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0
      #PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0
      #send errors: 0, #recv errors: 0

The MTU was set to 1500, so I changed to 1360 but it made no difference. Also did a packet tracer and all comes back good.

Would this have to do with proxy ARP by chance since the ASA is on the same subnet as the servers? The servers default gateway is the Nexus, but it seems that servers could talk directly to the ASA via Layer2. I wonder if the ARP entries are getting screwed up somehow. If I ping 3 different servers and the Nexus from a remote site, the Nexus never drops a packet, but all 3 servers will drop packets but at different times.

[AMD_GAMER]

Turned off ProxyARP on ASA's without success. I did some packet captures on remote sites and I am seeing TCP retransmissions and duplicate acknowledgements. I'm opening a Cisco TAC case as I've run out of ideas.

[Marius Gunnerud]

Let us know what TAC says.

[AMD_GAMER]

Cisco TAC was able to get to the bottom of it in 5 minutes. Here is the explanation:

---------------------------------------------------------------------------------------------

We were able to identify a scenario in vlan 10 where the Nexus 7000 was forced to software switch the traffic from devices that live on vlan 10.  This is because of the hair pinning that was occurring which drives an IP redirect message to be generated.  Please find the resource below that explains this scenario in greater detail:

http://www.cisco.com/c/en/us/support/docs/ip/routing-information-protocol-rip/13714-43.html

Hope this may help someone else in the future.

[Marius Gunnerud]

So the issue was on the nexus switch, and the solution was to disable ICMP redirect?

Glad you found a solution