Sorry, not quite understand where the issue is.
You were saying that if the VPN connection is through an ASA firewall, it works just fine, however, if it's not through the ASA but other internet connection, this is when it fails. Then, not quite sure how we can help in this forum since connection through ASA works just fine.
I would assume that it is more to do with routing issue if it works fine through ASA but not through other connection.