Need troubleshooting assistance on a 5505

[kylebrogers]

I am working with a small off that has a 5505 acting as a basic firewall.  Behind it are off-the-shelf unmanaged switches. 

Two users have to work with an outside vendor and are having issues.  They have a Sonicwall remote VPN client on each of their desktops and use this to connect to the vendor.  They then RDP into VMWare-based Windows 7 desktops at the vendor's site to do their work.  Randomly throughout the day (6-10 times per day) while they are actively working the RDP session will disconnect.  It will auto-reconnect after a few seconds.  The VPN log on the clients never show any issues.  I believe this is an RDP problem because while the RDP session is disconnecting, their VPN client is not (it is set to NOT auto-reconnect if it gets disconnected so that I will know for sure if it gets disconnected). 

I don't see anything in the ASA's logs about denying connections involving their PCs and the remote VPN peer IP. 

Does anyone have any suggestions on how I can try and troubleshoot this in the ASA?  Is there any way for me to log JUST info about connections between their inside IPs and the VPN peer IP?  I know that setting up a syslog server is an option but I'd like to see if there is a way to capture only the info I want first.  

[Jouni Forss]

Hi,

Naturally the first thing that makes the problem harder to pinpoint on the ASA is the fact that the users are using a VPN Client software towards the remote site.

This again means that you wont see the actual RDP (TCP/3389) traffic destined for the remote sites hosts.

So the points where you could see the actual TCP connections and possible capture the actual traffic would be

• Run Wireshark capture on the actual user computers
• Run some capture or monitor logs on the remote Sonicwall if possible
• Run Wireshark capture on the remote site computers

I guess you can always capture the traffic between he host and the remote Sonicwall public IP address on the ASA (The encrypted traffic). Then you can monitor when the connection disconnects and upload the capture file from the ASA to a computer and open it with wireshark and see if there is somekind of pause in the traffic from the remote site.

A simple capture could forexample be

access-list VPN-CAPTURE permit ip host host

access-list VPN-CAPTURE permit ip host host

capture VPN-CAPTURE type raw-data packet-length 1522 access-list VPN-CAPTURE interface inside buffer 33500000 circular-buffer

Then you could use the following command to see if the is any traffic hitting the capture

show capture

You could use the following command to view the contents of the capture (even though it wouldnt tell much and would be hard to read)

show capture VPN-CAPTURE

Then you could copy the capture contents to a local PC to be opened by Wireshark

copy /pcap capture:VPN-CAPTURE tftp:///VPN-CAPTURE.pcap

With the Wireshark software the capture file would be the easiest to go through and compare the timestamps to the time when the user expirienced the disconnect.

But I would still have to say that the ASA in this case is probably the worst place to monitor what happens to the connections (unless we can see a clear disruption in the connections on ASA in general). It should be done on the actual hosts or on the Sonicwall.

Naturally you can try something like Ping Plotter and monitor some remote Internet host with ICMP and see if there is any packet loss in the Internet connections from your site  to the Internet while this happens. If there is none it might point to the fact that the remote end is expiriencing the problems and not your site.

Naturally if there is something else related to the ASA that is happening then a separate Syslog server would perhaps tell if there is something else happening on the ASA.

Hope this helps

- Jouni

[kylebrogers]

Thank you for the advice.  Since the vendor is less than responsive (their stance is "well, no one else is reporting issues"), I can only troubleshoot from my side.  I will try Wireshark.  I am not onsite at all during the day so it's going to be a 15-20 hour capture file. I will let you know what I find. 

[kylebrogers]

I set up Wireshark to monitor the Sonicwall VPN client's network interface (instead of her regular network interface).  That way I could see the communications between her PC and the Windows 7 system she's RDPing into on the other side of the tunnel.  I also kept a continual ping between her PC and the Windows 7 PC on the other side of the tunnel.  I wanted to see if I would drop ping packets at that same time that she was getting disconnected.  I noticed a couple of things.  Around the time that she gets disconnected, she usually gets several "Destination Unreachable (Fragmentation Needed)" responses.  These are for traffic that is IN the tunnel since I'm watching the VPN interface and not her regular network adapter (where the VPN is terminated).  I have attached a screenshot of the wireshark info for one of these.

Here is some relevant info:

I should mention that the vendor uses a very large subnet mask (I believe 255.255.224.0) so the user's VPN adapter IP and the remote system's IP are in the same subnet. 

User's VPN adapter IP:  192.168.0.111

User's VPN adapter MAC:  ends in 26:16:50 (you see this as Redcreek_26:16:50)

Remote Windows 7 (VMWare) IP:  192.168.2.43 (no idea about the MAC)

The packet's source comes from Cisco_54:01:60 which I assume must be a device on the other side of the tunnel.  While I know it couldn't be something from our network because this packet is inside the VPN tunnel, I check the ASA (our only Cisco device) and none of the interfaces have that MAC address. 

It looks to me like some Cisco device on the remote side of the tunnel is having MTU issues.  Am I crazy?

I have attached a copy of the packet

[malshbou]

The client has used tunnel MTU larger than (1414) while DF bit is set.

Not sure how sonicwall VPN is transported, but if it is TCP-based (SSL or TLS), then you may check the following:

- check if path MTU discovery (PMTUD) is active at the client (IP packets sizes before and after the ICMP message)

- work around through reducing the MSS value on ASA, though this is at the transport-level, but it could be reflected to the tunnel MTU to be lowered.

- get the MTU value of the VPN adapter:

CMD> netsh interface ipv4 show interface

- if you can observe that the same packet sizes that triggered the ICMP Unreachable message can be sent normally during the "non-failure" times, then it is likely that the failure appears with some change in routing (dynamic) at the peer side where a hop with lower MTU is traveresed.

Hope this helps.

------------------

Mashal Alshboul

[kylebrogers]

I'd thought about adjusting the ASA MSS but assumed that it wouldn't work since the fragmented VPN packets will get dropped off and re-assembled into a packet with its own DNF bit set. 

I will definitely check the MTU on the VPN adapter.  I wasn't aware that I could do that.  Thanks for the tip.  I will let you know what I find out. 

What's wierd is that the "framgentation needed" is coming from something behind their VPN server, meaning the packet it can't handle doesn't have any of the VPN overhead adding size to it.  It's just the standard data packet.  It's a bizarre situation. 

[malshbou]

By adjusting the ASA MSS, it can intercept the TCP negotiation between the VPN client and server so that they exhange smaller TCP segements, hence smaller MTU . The TCP/IP stack normally can derive MTU values based on the MSS exhanged, so we hope by this workaround that the source originally sends smaller packets.

The command i provided works for Windows 7 exactly as is. For Win XP, you will need to enter submodes for almost each part of the command.

------------------
Mashal Alshboul

[kylebrogers]

That makes sense.  I was only thinking of packets in an existing stream and not about how it can effect the initial negotiation.  Thanks   I'll crank that down as well tonight and let you know what happens.