01-24-2014 09:29 PM
Hi all.
I have a couple ASA 5505 in use at two separate locations. They are configured in Spoke/Hub; the two are linked with site to site vpn, and remote access vpn to main site will also allow traffic to traverse to the second site.
Everything was working great. Then main site needed to move to a new location, which also resulted in new ISP and new public IPs.
Both still get internet access, the remote access vpn still works fine. The site to site VPN is acting a little strangely. It was working just fine for a week. Yesterday, the site to site VPN stopped functioning properly. I cannot ping or navigate to the remote site anymore. If i use packet-tracer (packet-tracer input inside icmp 192.168.1.50 1 1 192.168.2.50) it will drop encrypted traffic due to ACL. However, when I try the same command immediately after (or any other ip to any other ip) the vpn will encrypt and allow flow-creation. Alas, I still cannot ping remote hosts.
I have verified and double checked that there is no firewall/anti-virus interference on the remote host that I am attempting to connect to. I have also checked and double checked the following:
-Remote site internet connectivity is fine.
-Remote site IP addresses have not changed.
-Strange behavior is present from both firewalls when trying to use packet-tracer
-Unable to ping from either direction in the site to site VPN. Remote access VPN can still ping the main site local hosts.
-Crypto map settings appear correct.
I would gladly post configs, but at this very moment I do not have access to them. Any help/suggestions would be appreciated.
01-25-2014 09:54 AM
Hi,
The "packet-tracer" command you used doesnt really correspond to the typical ICMP that a host would send.
You would use
packet-tracer input inside icmp 192.168.1.50 8 0 192.168.2.50
Also, if the L2L VPN connection is down and you attempt to use "packet-tracer" the result is always a VPN Phase DROP. This is because the first "packet-tracer" initiates the VPN negotiation and usually when you issue the same command again the VPN negotiation has already finished.
Can you confirm that the traffic is sent succesfully encrypted/encapsulated through the L2L VPN when you attempt the connections or send ICMP Echos?
You can naturally use the command
show crypto ipsec sa
or
show crypto ipsec sa peer
And first confirm that traffic is getting to the VPN connection.
Next you should probably confirm whats seen on the remote site. Does it decrypt/decapsulate traffic?
Naturally as one site moved I would confirm that no configurations were removed by accident.
You could go as far as capturing traffic on the ASAs and/or the hosts to confirm where the flow of traffic stops.
On some occasions this kind of troubleshooting might simply end in a situation where everything is configured correctly and there is no clear reason for the problem. In this case it might be some bug.
Sometimes the solution might simply have been to reconfigure the VPN connection from scratch. I have seen this work for some here in CSC also.
But I would suggest first following where the traffic flow stops with captures, logs and output of the different counters on the ASAs.
- Jouni
01-25-2014 10:12 AM
I used the version you recommended from both ends of the L2L tunnel. The result was the same; both allow traffic all the way through flow-creation.
I then verified security associations via show crypto ipsec sa. Just to be thorough, i used both methods you stated from both ends of the L2L tunnel. Both show the SA.
I'm still just getting my feet wet in Cisco, where can I see if the decryption is actually happening?
As far as configurations removed by accident, I keep a change log of every change command I issue to the ASA. Wouldn't the configuration removal have resulted in a downtime immediately rather than a week after the fact?
Thanks so much for your assistance.
01-25-2014 10:34 AM
Hi,
True, the configuration changes should have shown earlier. Was too much caught up listing things I didn't really think that one through
If I were to presume that the L2L VPN doesnt work at all at the moment and if there is very few attempts to pass traffic through it you could simply look at the counters of the command
show crypto ipsec sa
or
show crypto ipsec sa peer
You should see counters for SAs / network pairs on the L2L VPN connection and how much traffic has been encrypted/encapsulated (sent) and how much has been decrypted/decapsulated (received). This would be the fast way to determine if you are seeing the traffic on both ASA units.
Naturally you could use ASA logs to confirm the connection is built on the originating ASA and see if the same connection can be seen built on the remote ASA. If that is the case then you could naturally start confirming that the remote end sees some return traffic for this connection that has been built on the ASA (ASA builds a connection as long as it allows the first packet of the connection through itself)
The ASAs could be configured to capture the traffic you want and you could then very clearly confirm where the traffic stops.
Just to give you an example of a capture configuration on the ASA
access-list VPN-CAPTURE permit ip
access-list VPN-CAPTURE permit ip
capture VPN-CAPTURE type raw-data access-list VPN-CAPTURE interface inside buffer 5000000 circular buffer
You can naturally change the capture and ACL names and modify the ACL to include only certain local/remote host or even go as far as defining ports (though then have to make sure that the ACL really matches both directions of traffic). Also your local interface might no be named the default "inside".
You can then attempt connections and issue the following command to determine if anything has been captured
show capture
If traffic was captured you could then use the following command to show the output on the CLI
show capture VPN-CAPTURE
I would suggest though that you would attempt sending the capture with TFTP to some host so you can open it up with Wireshark for example
copy /pcap capture:VPN-CAPTURE tftp://x.x.x.x/VPN-CAPTURE.pcap
You can then remove the capture and its contents from the ASA with the command
no capture VPN-CAPTURE
The ACL you will have to remove separately.
But as I said, checking the logs while making connection attempts and checking the VPN counters might be easier things to start with to get a picture where the traffic stops and the capture could then be used at either device to get a clealer look at the situation.
As you mentioned already, using "packet-tracer" command that matches the L2L VPN configuration already provides a output that indicates that the VPN configuration is matched and the VPN negotiation goes through.
As for possible configuration changes that might have happend on the ASA I will mention this even though it doesnt match your problem as it started well after the ISP change.
In certain situation where people use the ASDM and its Wizard to configure a VPN they might change the setting that enables the VPN traffic to Bypass the interface ACLs. This configuration change might essentially cause that either end ASA might start blocking the incoming traffic from VPN. I mean the traffic that is arriving to one of the ASA firewalls and being decrypted/decapsulated.
The command on the CLI is
sysopt connection permit-vpn
This is the default setting on an ASA and doesnt show up in the basic CLI configuration
no sysopt connection permit-vpn
This setting shows up in the CLI configuration and also means that the ACL of the interface that builds the VPN connection now controls traffic coming even from VPN connection.
Hope this helps
- Jouni
01-25-2014 03:25 PM
So interestingly, while the packet tracer works.... no packets are going out when I ping. The captures are 0 bytes after multiple ping attempts, and encryption/decryption all read 0 on the security associations. However when I run packet-tracer again with the icmp 8 0 packet, it goes through and generates 62 bytes.
01-25-2014 04:52 PM
More and more I suspect the firewall is not the problem, especially since no configurations were changed. (Only myself and my partner have access to the configs, and he was busy with another location when the outage occured). Could anything change on the ISP end that would affect this? The main site (the one that changed locations) also has a remote-access VPN configured. That VPN is completely functional and fine.
01-27-2014 12:24 PM
I have just noticed now that packet-tracer fails on VPN (subtype: encrypt) the first time I run it. However, if I run the packet-tracer again the vpn encryption does not fail.
Thoughts?
01-28-2014 12:05 AM
Hi,
Mentioned about that earlier with the "packet-tracer" command. Its normal behaviour if the VPN connection is down during the first time you issue the command
packet-tracer input inside icmp 192.168.1.50 8 0 192.168.2.50
Also, if the L2L VPN connection is down and you attempt to use "packet-tracer" the result is always a VPN Phase DROP. This is because the first "packet-tracer" initiates the VPN negotiation and usually when you issue the same command again the VPN negotiation has already finished.
- Jouni
01-29-2014 11:38 AM
I'm sorry, was a little sleep deprived and must have forgotten you already said something about it. So what must be happening then is that the firewalls are indeed connecting via VPN, and the problem lies elsewhere?
01-30-2014 04:09 PM
Got back in town and was able to get running configs. Two things here confuse me. Maybe I have a configuration wrong and my brain is just not letting me see the error.
1) I set up a site-to-site VPN from a different firewall (5505 but different software version) and it exhibited the exact same behavior. I could use Packet-Tracer, but not actually ping anything or initiate network traffic across it.
2) I configured a remote access VPN for FW2 and it works completely fine. I am able to ping the host through the remote-access VPN where I still cannot via site-to-site VPN.
FW1 public ip information has been replaced with 1.1.1.1
FW2 public ip information has been replaced with 2.2.2.2
Thanks for the help.
FW1
____________________________________________
ASA Version 8.2(5)
!
hostname fw1
enable password /r2oDuwuK7ckZKJB encrypted
passwd lcALn1AgoxqCj3uZ encrypted
names
!
interface Ethernet0/0
switchport access vlan 2
!
interface Ethernet0/1
!
interface Ethernet0/2
!
interface Ethernet0/3
!
interface Ethernet0/4
!
interface Ethernet0/5
!
interface Ethernet0/6
!
interface Ethernet0/7
!
interface Vlan1
nameif inside
security-level 100
ip address 192.168.1.1 255.255.255.0
!
interface Vlan2
nameif outside
security-level 0
ip address 1.1.1.1 255.255.255.128
!
ftp mode passive
dns domain-lookup outside
dns server-group defaultdns
name-server 1.1.1.1
name-server 1.1.1.1
same-security-traffic permit intra-interface
object-group network Exchange
network-object host 192.168.1.102
access-list outside_access_in extended permit tcp any interface outside eq 3389
access-list outside_access_in extended permit tcp any interface outside eq ldap
access-list outside_access_in extended permit tcp any interface outside eq 379
access-list outside_access_in extended permit tcp any interface outside eq 390
access-list outside_access_in extended permit tcp any interface outside eq 3268
access-list outside_access_in extended permit tcp any interface outside eq ldaps
access-list outside_access_in extended permit tcp any interface outside eq 3269
access-list outside_access_in extended permit tcp any interface outside eq imap4
access-list outside_access_in extended permit tcp any interface outside eq 993
access-list outside_access_in extended permit tcp any interface outside eq pop3
access-list outside_access_in extended permit tcp any interface outside eq 563
access-list outside_access_in extended permit tcp any interface outside eq www
access-list outside_access_in extended permit tcp any interface outside eq https
access-list outside_access_in extended permit tcp any interface outside eq smtp
access-list outside_access_in extended permit tcp any interface outside eq 465
access-list outside_access_in extended permit tcp any interface outside eq 691
access-list outside_access_in extended permit tcp any interface outside eq 102
access-list outside_access_in extended permit tcp any interface outside eq 135
access-list outside_access_in extended permit tcp any interface outside eq 522
access-list outside_access_in extended permit tcp any interface outside eq domain
access-list outside_access_in extended permit tcp any interface outside eq 717
access-list outside_access_in extended permit tcp any interface outside eq 2525
access-list outside_access_in extended permit tcp any interface outside eq 587
access-list nonat remark ACL for Nat Bypass
access-list nonat extended permit ip 192.168.1.0 255.255.255.0 10.11.12.0 255.255.255.0
access-list nonat extended permit ip 192.168.1.0 255.255.255.0 192.168.2.0 255.255.255.0
access-list vpn-SplitTunnel remark ACL for VPN Split Tunnel
access-list vpn_SplitTunnel standard permit 192.168.1.0 255.255.255.0
access-list vpn_SplitTunnel standard permit 192.168.2.0 255.255.255.0
access-list nsfw2_LAN extended permit ip 192.168.1.0 255.255.255.0 192.168.2.0 255.255.255.0
access-list nsfw2_LAN extended permit ip 10.11.12.0 255.255.255.0 192.168.2.0 255.255.255.0
pager lines 24
mtu inside 1500
mtu outside 1500
ip local pool vpnpool 10.11.12.1-10.11.12.255
icmp unreachable rate-limit 1 burst-size 1
no asdm history enable
arp timeout 14400
global (outside) 10 interface
nat (inside) 0 access-list nonat
nat (inside) 10 192.168.1.0 255.255.255.0
static (inside,outside) tcp interface 3389 192.168.1.100 3389 netmask 255.255.255.255
static (inside,outside) tcp interface ldap 192.168.1.102 ldap netmask 255.255.255.255
static (inside,outside) tcp interface 379 192.168.1.102 379 netmask 255.255.255.255
static (inside,outside) tcp interface 390 192.168.1.102 390 netmask 255.255.255.255
static (inside,outside) tcp interface 3268 192.168.1.102 3268 netmask 255.255.255.255
static (inside,outside) tcp interface ldaps 192.168.1.102 ldaps netmask 255.255.255.255
static (inside,outside) tcp interface 3269 192.168.1.102 3269 netmask 255.255.255.255
static (inside,outside) tcp interface imap4 192.168.1.102 imap4 netmask 255.255.255.255
static (inside,outside) tcp interface 993 192.168.1.102 993 netmask 255.255.255.255
static (inside,outside) tcp interface pop3 192.168.1.102 pop3 netmask 255.255.255.255
static (inside,outside) tcp interface 563 192.168.1.102 563 netmask 255.255.255.255
static (inside,outside) tcp interface www 192.168.1.102 www netmask 255.255.255.255
static (inside,outside) tcp interface https 192.168.1.102 https netmask 255.255.255.255
static (inside,outside) tcp interface smtp 192.168.1.102 smtp netmask 255.255.255.255
static (inside,outside) tcp interface 465 192.168.1.102 465 netmask 255.255.255.255
static (inside,outside) tcp interface 691 192.168.1.102 691 netmask 255.255.255.255
static (inside,outside) tcp interface 102 192.168.1.102 102 netmask 255.255.255.255
static (inside,outside) tcp interface 135 192.168.1.102 135 netmask 255.255.255.255
static (inside,outside) tcp interface 522 192.168.1.102 522 netmask 255.255.255.255
static (inside,outside) tcp interface domain 192.168.1.102 domain netmask 255.255.255.255
static (inside,outside) tcp interface 717 192.168.1.102 717 netmask 255.255.255.255
static (inside,outside) tcp interface 2525 192.168.1.102 2525 netmask 255.255.255.255
static (inside,outside) tcp interface 587 192.168.1.102 587 netmask 255.255.255.255
access-group outside_access_in in interface outside
route outside 0.0.0.0 0.0.0.0 1.1.1.2 1
timeout xlate 3:00:00
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02
timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00
timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00
timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute
timeout tcp-proxy-reassembly 0:01:00
timeout floating-conn 0:00:00
dynamic-access-policy-record DfltAccessPolicy
aaa authentication http console LOCAL
aaa authentication ssh console LOCAL
http server enable
http 192.168.1.0 255.255.255.0 inside
no snmp-server location
no snmp-server contact
snmp-server enable traps snmp authentication linkup linkdown coldstart
crypto ipsec transform-set strong-des esp-3des esp-md5-hmac
crypto ipsec transform-set nsfw2 esp-aes-256 esp-sha-hmac
crypto ipsec security-association lifetime seconds 28800
crypto ipsec security-association lifetime kilobytes 4608000
crypto dynamic-map dynmap 30 set transform-set strong-des
crypto map ns 1 match address nsfw2_LAN
crypto map ns 1 set peer 2.2.2.2
crypto map ns 1 set transform-set nsfw2
crypto map ns 65535 ipsec-isakmp dynamic dynmap
crypto map ns interface outside
crypto isakmp identity address
crypto isakmp enable outside
crypto isakmp policy 1
authentication pre-share
encryption aes-256
hash sha
group 2
lifetime 86400
crypto isakmp policy 11
authentication pre-share
encryption 3des
hash md5
group 2
lifetime 86400
console timeout 0
management-access inside
dhcpd lease 3000
dhcpd option 3 ip 192.168.1.1
!
dhcpd address 192.168.1.120-192.168.1.250 inside
dhcpd enable inside
!
threat-detection basic-threat
threat-detection statistics access-list
no threat-detection statistics tcp-intercept
webvpn
group-policy ns internal
group-policy ns attributes
vpn-idle-timeout 120
vpn-tunnel-protocol IPSec
split-tunnel-policy tunnelspecified
split-tunnel-network-list value vpn_SplitTunnel
tunnel-group ns-VPN type remote-access
tunnel-group ns-VPN general-attributes
address-pool vpnpool
default-group-policy ns
tunnel-group ns-VPN ipsec-attributes
pre-shared-key *****
tunnel-group 2.2.2.2 type ipsec-l2l
tunnel-group 2.2.2.2 ipsec-attributes
pre-shared-key *****
!
class-map inspection_default
match default-inspection-traffic
!
!
policy-map type inspect dns preset_dns_map
parameters
message-length maximum client auto
message-length maximum 512
policy-map global_policy
class inspection_default
inspect dns preset_dns_map
inspect ftp
inspect h323 h225
inspect h323 ras
inspect ip-options
inspect netbios
inspect rsh
inspect rtsp
inspect skinny
inspect sqlnet
inspect sunrpc
inspect tftp
inspect sip
inspect xdmcp
inspect icmp
inspect icmp error
FW2
_______________________________
ASA Version 8.2(5)
!
hostname fw2
enable password rDlrx/ijZiwp44Mi encrypted
passwd Ft0mv6GdiaYo9Cge encrypted
names
!
interface Ethernet0/0
switchport access vlan 2
!
interface Ethernet0/1
!
interface Ethernet0/2
!
interface Ethernet0/3
!
interface Ethernet0/4
!
interface Ethernet0/5
!
interface Ethernet0/6
!
interface Ethernet0/7
!
interface Vlan1
nameif inside
security-level 100
ip address 192.168.2.1 255.255.255.0
!
interface Vlan2
nameif outside
security-level 0
ip address 2.2.2.2 255.255.255.248
!
ftp mode passive
access-list nsfw1_LAN extended permit ip 192.168.2.0 255.255.255.0 192.168.1.0 255.255.255.0
access-list nsfw1_LAN extended permit ip 192.168.2.0 255.255.255.0 10.11.12.0 255.255.255.0
access-list nonat remark ACL for NAT Bypass
access-list nonat extended permit ip 192.168.2.0 255.255.255.0 192.168.1.0 255.255.255.0
access-list nonat extended permit ip 192.168.2.0 255.255.255.0 10.11.12.0 255.255.255.0
access-list nonat extended permit ip 192.168.2.0 255.255.255.0 10.10.10.0 255.255.255.0
access-list vpn_SplitTunnel remark ACL for VPN Split Tunnel
access-list vpn_SplitTunnel standard permit 192.168.2.0 255.255.255.0
pager lines 24
mtu inside 1500
mtu outside 1500
ip local pool ravpnpool 10.10.10.1-10.10.10.10
icmp unreachable rate-limit 1 burst-size 1
no asdm history enable
arp timeout 14400
global (outside) 10 interface
nat (inside) 0 access-list nonat
nat (inside) 10 192.168.2.0 255.255.255.0
route outside 0.0.0.0 0.0.0.0 2.2.2.3 1
timeout xlate 3:00:00
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02
timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00
timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00
timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute
timeout tcp-proxy-reassembly 0:01:00
timeout floating-conn 0:00:00
dynamic-access-policy-record DfltAccessPolicy
aaa authentication ssh console LOCAL
no snmp-server location
no snmp-server contact
snmp-server enable traps snmp authentication linkup linkdown coldstart
crypto ipsec transform-set nsfw2 esp-aes-256 esp-sha-hmac
crypto ipsec transform-set ra-nsfw2 esp-3des esp-md5-hmac
crypto ipsec security-association lifetime seconds 28800
crypto ipsec security-association lifetime kilobytes 4608000
crypto dynamic-map dynmap 30 set transform-set ra-nsfw2
crypto map ns2 1 match address nsfw1_LAN
crypto map ns2 1 set peer 1.1.1.1
crypto map ns2 1 set transform-set nsfw2
crypto map ns2 65535 ipsec-isakmp dynamic dynmap
crypto map ns2 interface outside
crypto isakmp identity address
crypto isakmp enable outside
crypto isakmp policy 1
authentication pre-share
encryption aes-256
hash sha
group 2
lifetime 86400
crypto isakmp policy 12
authentication pre-share
encryption 3des
hash md5
group 2
lifetime 86400
console timeout 0
management-access inside
!
dhcpd address 192.168.2.60-192.168.2.200 inside
dhcpd enable inside
!
threat-detection basic-threat
threat-detection statistics access-list
no threat-detection statistics tcp-intercept
group-policy ns2 internal
group-policy ns2 attributes
vpn-idle-timeout 120
vpn-tunnel-protocol IPSec
split-tunnel-policy tunnelspecified
split-tunnel-network-list value vpn_SplitTunnel
tunnel-group 1.1.1.1 type ipsec-l2l
tunnel-group 1.1.1.1 ipsec-attributes
pre-shared-key *****
tunnel-group ph-VPN type remote-access
tunnel-group ph-VPN general-attributes
address-pool ravpnpool
default-group-policy ns2
tunnel-group ph-VPN ipsec-attributes
pre-shared-key *****
!
class-map inspection_default
match default-inspection-traffic
!
!
policy-map type inspect dns preset_dns_map
parameters
message-length maximum client auto
message-length maximum 512
policy-map global_policy
class inspection_default
inspect dns preset_dns_map
inspect ftp
inspect h323 h225
inspect h323 ras
inspect ip-options
inspect netbios
inspect rsh
inspect rtsp
inspect skinny
inspect esmtp
inspect sqlnet
inspect sunrpc
inspect tftp
inspect sip
inspect xdmcp
inspect icmp
inspect icmp error
01-31-2014 10:17 AM
Bump, looking for help still.
02-03-2014 09:30 AM
Bump again please. We have a workaround in place but I would like to have this tunnel functioning as it should.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide