L2L tunnel up, not passing traffic...all of a sudden

[WStoffel1]

I've had a tunnel in place on a 5505 to a remote network i don't control...so my troubleshooting there is limited.  But the tunnel has been in place for over a year without issue.  Suddenly it doesn't appear to be passing traffic.  But it is in at least one direction.  

Remote network:192.168.191.0/24

Local ASA side: 10.220.78.0/24

I had a constant ping started from 192.168.191.10 > 10.220.78.23

Which is a Windows server pinging a Windows workstation.

When i debug icmp on the ASA i get:

ICMP echo request from outside:192.168.191.10 to inside:10.220.78.23 ID=1 seq=2866 len=32
ICMP echo reply from inside:10.220.78.23 to outside:192.168.191.10 ID=1 seq=2866 len=32

Which confirms to me that the remote network is in fact traversing the tunnel and hitting the 10.220.78.23 device, which is in fact responding, and the reply is being sent out the ASA.

The tunnel negotiates and comes up any time I reset it, by all accounts it looks correct.

The problem is not limited to ICMP as I'm unable to net use or map drives, nor can 192.168.191.10 print to the printer at 10.220.78.20.

But once i saw the icmp trace output I pretty much figured it has to be on the remote end...so....

My question, can I absolutely infer from this that the issue resides on the remote end?

[David_Che]

Hi,

It seems that packet(DF-bit set) size > path mtu, then caused packet drop.

As ICMP message is 100 bytes by default,  it always works good if only end to end connectivity is ok.

You may try check the path mtu size when encounter the issue.

ping 'server ip address '  -f -l xxxx  to get the maximum size 

workaround:

option1: clear df-bit on both sides

option2:' ip tcp adjust-mss ' for TCP traffic

option3: lower client MTU

Regards,

David

[WStoffel1]

It's a windows server sending the icmp request, 32 bytes by default.  So that length is actually correct.

However if I set the DF bit, i get a somewhat odd response, 1410 is my MTU...?

[WStoffel1]

Some additional info.  Aside from the ping they have running from the remote network, which is shown in the above icmp trace, if i run packet tracer from the local network to the remote, tunnel's up/traffic is allowed.  Not a big surprise since the tunnel does negotiate and stay up.

I captured packets from the ASA and I can see the local 10.220.78.23 device sending the reply to 192.168.191.10.  Matching up with the icmp trace.

I had them run a packet capture on their firewall and confirmed, the ICMP requests from 192.168.191.10 are being encapsulated and sent on the tunnel.  Again confirmed in my mind since i see the requests on the ASA.  But they don't ever see the response.

There's no tcp adjust mss command on the ASA but there's this in the config:

ASA# sh run all sys
no sysopt connection timewait
sysopt connection tcpmss 1380
sysopt connection tcpmss minimum 0
sysopt connection permit-vpn
sysopt connection reclassify-vpn
no sysopt connection preserve-vpn-flows
no sysopt nodnsalias inbound
no sysopt nodnsalias outbound
no sysopt radius ignore-secret
no sysopt noproxyarp inside
no sysopt noproxyarp outside

Any other ideas?

[David_Che]

As you know, host is enabled PMTU discovery by default, which means packet's df-bit set to 1, in other words, if this packet size > intermediate device outgoing interface MTU, this packet will be dropped, and send icmp too big message to the source,  however this message sometimes will not reach to the source due to blocked or disable. This will cause end-2-end application failure.

You may try to optimize 'sysopt connection tcpmss' to lower vaule like '1300' or less until resolve the issue.

[WStoffel1]

sysopt connection tcpmss 1000

still same result.  i've opened a ticket with the carrier, still pending on that.  but I can see icmp requests from the remote end, pass my asa, the device responds, i see the icmp reply and then it appears to drop when exiting the asa.

i'll also start running packet captures from both ends, and start looking at sequence numbers and go from there..thanks for the ideas though.