ASA VPN Problems...Routing the cause?

[caplinktech]

I have deployed dozens or IPSEC site to site and RA VPNs, along with dozens of SSL VPN configurations using ASA5505s and ASA55X0 devices.

I am having an issue with getting a tunnel to establish at a new clients location and the only thing that strikes me as a potential cause of the problem is the way the public interface IP is routed to the ASA possibly combined with a static route on the ASA.

Essentially when tracerouting from outside their network, we are seeing RFC1918 IP space in the traceroute which appears that it may be on the edge of their network....

1  206.x.x.129 (206.x.x.129)  0.827 ms  1.375 ms  1.463 ms
2  206.x.x.69 (206.x.x.69)  0.753 ms  0.627 ms  0.956 ms
3  hr1-bo3-te-1-0-1.waltham3bo3.savvis.net (204.70.202.45)  0.835 ms  1.115 ms  0.960 ms
4  cr2-te-0-4-0-0.boston.savvis.net (204.70.202.13)  4.418 ms  3.606 ms  3.661 ms
5  er1-tengig-2-2.newyorknyd.savvis.net (204.70.198.173)  8.895 ms  7.746 ms  8.246 ms
6  cr2-te-0-15-3-0.newyork.savvis.net (204.70.196.69)  25.140 ms  8.118 ms  7.697 ms
7  er1-te-2-0-0.newyork.savvis.net (204.70.197.9)  8.115 ms  7.612 ms  7.587 ms
8  208.174.224.134 (208.174.224.134)  7.391 ms  7.520 ms  7.525 ms
9  vlan52.ebr2.newyork2.level3.net (4.69.138.254)  7.904 ms  7.789 ms  8.101 ms
10  ae-6-6.ebr2.newyork1.level3.net (4.69.141.21)  7.741 ms  8.147 ms  7.483 ms
11  ae-3-3.ebr2.dallas1.level3.net (4.69.137.121)  43.946 ms  44.136 ms  44.111 ms
12  4.69.151.165 (4.69.151.165)  49.844 ms  49.527 ms  48.384 ms
13  ae-42-90.car2.dallas1.level3.net (4.69.145.196)  49.813 ms  48.804 ms  49.394 ms
14  comtel-tele.car2.dallas1.level3.net (4.71.170.66)  48.512 ms  48.544 ms  49.294 ms
15  10.10.250.37 (10.10.250.37)  45.176 ms  44.965 ms  44.799 ms
16  10.10.250.58 (10.10.250.58)  52.270 ms  49.178 ms  48.918 ms
17  64.x.x.236 (64.x.x.236)  49.153 ms  49.265 ms  49.139 ms

Additionally on the ASAs where we are looking to terminate the VPN, there is a static route that points 10.0.0.0/8 to another router on there internal network.  I am wondering if this is having an effect on the ability to temrinate the VPN on this device.  I have never seen public routes before that show private IP space in the traceroutes.  I realize that frequently packets pass internally across a provider's network using MPLS, but usually these internal IP addresses are hidden from the public view via a traceroute since they are switched with MPLS.

The symptoms that I am seeing make it look like there is packet loss along the connection to the ASA.  For example, if I access the public ASA IP via a web browser to establish a SSL VPN tunnel, the first time I go to access the URL, the certificate error will pop up, it wil allow me to continue to the login screen, and then after that, it stops functioning.  Additional attempts to even go back to the base URL (where the cert error would pop up) fail with a page not found error and it remains that way for about 30 minutes before we can get to the same point again.

It seems as though some initial packets make it through the edge routers before they are potentially dropped as being spoofed packets because of the RFC1918 space?  Or could it be related to the static route on the ASA?

Any insight as to what may cause this would be appreciated.  Additionally, any information on the traceroute and routing in general with RFC1918 address being visible from a public traceroute would be appreciated.

[barry]

Hi

I think it is unlikely that the RFC 1918 addresses in use within the ISP cloud will cause you any issues. This is pretty normal; ISPs tend to use the 10.x.x.x address space within their backbones to conserve the now run out IP V4 publically routable address space.

A traceroute can return any IP address configured on a layer 3 device (router or layer 3 switch). Generally speaking this will be the IP address of the interface the traceroute request was received over, however this isn't always the case. The returned address does not need to be globally routable.

The key however is that your devices will never need to talk *directly* to these addresses. Your devices only ever need to talk directly to the destination IP address you are attempting to contact - your devices should not even be aware of the hops in the path. IP itself is connectionless so it is quote valid for different packets, even within a single stream, to take different routes between source and destination and this should not cause massive issues.

The only addresses that are of concern are the real source and destination IP addresses; unless you are trying to use IP source routing which I think is unlikely.

In terms of troubleshooting your issue further, I would attempt to install a packet sniffer such as wireshark on your PC and capture the packet exchanges. If there is packet loss, it should show up in the TCP trace. You could also install a similar PC running wireshark to mirror the outside card of your ASA. With the two traces you should absolutely be able to confirm if there is packet loss. If there is packet loss,I think it is unlikely that it will be related to the use of RFC 1918 addresses within the ISP cloud.

Hope this helps.

Barry

[caplinktech]

Additional debugging and weird behavior...

If I SSH into the ASA using the external IP from off site, I can connect to the ASA without a problem.  While connected to the ASA, if I fire up the Anyconnect client and attempt to bring up a SSL VPN, the Anyconnect client indicates Host unreachable, but i remain connected and functioning within my SSH session.

If I attempt to establish a SSL VPN to the ASA, I get the host unreachable error, and if I then attempt to establish an SSH session, I immediately get a Connection Closed by Host error and am unable to connect for approximately 5 minutes.

I'm beginning to think there is some type of issue with security certificates as they are configured on the ASA.  Is there a way to wipe out all certificates and rsa keys from the ASA in 1 command?

[barry]

Hi

Daft question from me time. It's not possible your ISP is doing some kind of IPS/IDS in front of your ASA is it? The disconnected sessions and timeout periods sound like you are being shunned to me...

Barry

[caplinktech]

I do not believe so and I also do not know why they would detect a https session as an intrusion.

It is these weird timeouts and stuff that had/have me thinking it may have been some type of routing problem where my client's ISP filters traffic at their border because it sees RFC1918 as the source of the traffic and thinks it is a spoof.  If these 10.10.250.X addresses were not showing up in the traceroute at their network edge, I would dismiss it for the same reasons you stated.  My concern is that they are using this space at their edge because the traceroute doesn't show what I would normally assume/hope would be another hop from XO/Level3 to a publically routable IP on their network edge.

Additionally, when I enable debugging level logging and then attempt to access the public IP of the firewall, not a thing regarding any traffic from the source IP I attempt to connect from shows up in the debugging log.  So it seems like it never receives the request to access the page.  I would expect that since it is an SSL connection, something would be emitted related to security being negotiated/secured.  Instead, the source IP never shows up, and I just get a page cannot be displayed.

Since there are timeout periods as you referred to them as, I am wondering if some type of route dampening is occurring?

This has me puzzled.