Phase 2 VPN issue

[MavsX4142]

Hi Guys- I know there are a ton of threads on phase 2 issues, and i've been reading all of them, but am still having issues. Phase 1 of the vpn completes. Phase 2 is my issue. Specifically the branch site asa is not doing encaps but has a few decaps, and then HQ asa is getting encaps but no decaps, in addition hit counts are not incrementing. I've been staring at this for a week so maybe i am missing something obvious? I've removed a lot of the config that does't apply here as well as changed the public IPs into X.X.X.X.

From what i am reading when there is an issue with the encaps and decaps there are 1 of a 4 things wrong..

1 ) Verify the other end has a route outside for the interesting traffic.
2 ) Check that both VPN ACL’s are not mismatched.
3 ) Double check NAT’s to make sure the traffic is not NAT’ing correctly.
4 ) Is what you are trying to ping even responding back? Often what you’re sending traffic to is not able to accept or is not responding to this traffic. I prefer to put a packet capture on the remote end firewall to see if the traffic is coming back into that firewall.

Thank you so much for any input that can lead me to figuring this out.

brach config

ASA Version 7.2(4)
!

names
!
interface Vlan1
 no nameif
 no security-level
 no ip address
!
interface Vlan100
 nameif Outside
 security-level 0
 ip address X.X.X.X 255.255.255.0
!
interface Vlan330
 nameif WiFi-Management
 security-level 100
 ip address 10.3.30.1 255.255.255.0
!
interface Vlan331
 nameif WiFi-Access
 security-level 50
 ip address 10.3.31.1 255.255.255.0
!
interface Vlan333
 nameif Business-Center
 security-level 75
 ip address 10.3.33.1 255.255.255.0
!
interface Vlan999
 shutdown
 nameif Unused-Ports
 security-level 0
 no ip address
!
interface Ethernet0/0
 switchport access vlan 100
 speed 10
 duplex full
!
interface Ethernet0/1
 switchport trunk allowed vlan 330-333
 switchport mode trunk
!
interface Ethernet0/2
 switchport access vlan 330
!
interface Ethernet0/3
 switchport access vlan 331
 shutdown
!
interface Ethernet0/4
 shutdown
!
interface Ethernet0/5
 switchport access vlan 333
!
interface Ethernet0/6
 switchport access vlan 333
!
interface Ethernet0/7
 switchport access vlan 333
!
ftp mode passive
dns server-group DefaultDNS
 domain-name XXXXXX.com
access-list VPN-LEG-PUB-MGMT extended permit ip 10.3.33.0 255.255.255.0 10.3.1.0 255.255.255.0
access-list NONAT extended permit ip 10.3.33.0 255.255.255.0 10.3.1.0 255.255.255.0
pager lines 24
mtu Outside 1500
mtu WiFi-Management 1500
mtu WiFi-Access 1500
mtu Business-Center 1500
mtu Unused-Ports 1500
no failover
icmp unreachable rate-limit 1 burst-size 1
asdm image disk0:/asdm-524.bin
no asdm history enable
arp timeout 14400
global (Outside) 1 interface
nat (WiFi-Access) 1 10.3.31.0 255.255.255.0
nat (Business-Center) 0 access-list NONAT
nat (Business-Center) 1 10.3.33.0 255.255.255.0
route Outside 0.0.0.0 0.0.0.0 ISP Gateway 1
timeout xlate 3:00:00
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02
timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00
timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00
timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute
no snmp-server location
no snmp-server contact
snmp-server enable traps snmp authentication linkup linkdown coldstart
crypto ipsec transform-set 3des esp-3des esp-sha-hmac
crypto ipsec transform-set aes-256 esp-3des esp-sha-hmac
crypto map vpnmap 10 match address VPN-LEG-PUB-MGMT
crypto map vpnmap 10 set pfs
crypto map vpnmap 10 set peer HQ public IP address
crypto map vpnmap 10 set transform-set 3des
crypto map vpnmap interface Outside
crypto isakmp identity address
crypto isakmp enable Outside
crypto isakmp policy 9
 authentication pre-share
 encryption aes-256
 hash sha
 group 2
 lifetime 3600
crypto isakmp policy 10
 authentication pre-share
 encryption 3des
 hash sha
 group 2
 lifetime 86400
crypto isakmp nat-traversal  20
telnet timeout 5

ssh timeout 5
console timeout 0
dhcpd dns 199.45.32.38 151.197.0.38
dhcpd lease 10800
!
dhcpd address 10.3.33.100-10.3.33.130 Business-Center
dhcpd enable Business-Center
!

tunnel-group HQ public IP address type ipsec-l2l
tunnel-group HQ public IP address ipsec-attributes
 pre-shared-key *
!
class-map inspection_default
 match default-inspection-traffic
!
!
policy-map type inspect dns preset_dns_map
 parameters
  message-length maximum 512
policy-map global_policy
 class inspection_default
  inspect dns preset_dns_map
  inspect ftp
  inspect h323 h225
  inspect h323 ras
  inspect netbios
  inspect rsh
  inspect rtsp
  inspect skinny
  inspect esmtp
  inspect sqlnet
  inspect sunrpc
  inspect tftp
  inspect sip
  inspect xdmcp
!
service-policy global_policy global
prompt hostname context
Cryptochecksum:ab0eab021b7d40edea91f18a503d224b
: end
[OK]
public#

HQ config

:

ASA Version 8.2(5)
!

interface Ethernet0/0
 nameif outside
 security-level 0
 ip address X.X.X.X 255.255.255.240
!
interface Ethernet0/1
 speed 100
 duplex full
 no nameif
 no security-level
 no ip address
!
interface Ethernet0/1.14
 vlan 14
 nameif inside
 security-level 100
 ip address 10.1.4.1 255.255.255.248
!
interface Ethernet0/1.31
 vlan 31
 nameif leg-pub-mgmt
 security-level 50
 ip address 10.3.1.1 255.255.255.0
!
interface Ethernet0/1.41
 shutdown
 vlan 41
 nameif internet-guest
 security-level 25
 no ip address
!
interface Ethernet0/1.42
 shutdown
 vlan 42
 nameif internet-employee
 security-level 50
 no ip address
!
interface Ethernet0/1.43
 vlan 43
 nameif cap
 security-level 50
 ip address 10.4.3.1 255.255.255.0
!
interface Ethernet0/2
 shutdown
 no nameif
 no security-level
 no ip address
!
interface Ethernet0/3
 shutdown
 no nameif
 no security-level
 no ip address
!
interface Management0/0
 nameif management
 security-level 100
 ip address 192.168.1.1 255.255.255.0
 management-only
!
boot system disk0:/asa825-k8.bin
ftp mode passive
clock timezone EST -5
clock summer-time EDT recurring
dns server-group DefaultDNS
 domain-name XXXXXX.com

access-list NONAT-LEG-PUB-MGMT extended permit ip 10.3.1.0 255.255.255.0 10.3.33.0 255.255.255.0

access-list VPN-OAK-BUS extended permit ip 10.3.1.0 255.255.255.0 10.3.33.0 255.255.255.0
pager lines 24
logging timestamp

no asdm history enable
arp timeout 14400
global (outside) 1 interface

nat (leg-pub-mgmt) 0 access-list NONAT-LEG-PUB-MGMT
nat (leg-pub-mgmt) 1 10.3.1.0 255.255.255.0

access-group INBOUND in interface outside
route outside 0.0.0.0 0.0.0.0 X.X.X.X 1

crypto ipsec transform-set 3des esp-3des esp-sha-hmac
crypto ipsec transform-set aes-256 esp-3des esp-sha-hmac
crypto ipsec security-association lifetime seconds 28800
crypto ipsec security-association lifetime kilobytes 4608000

crypto map vpnmap 333 match address VPN-OAK-BUS
crypto map vpnmap 333 set pfs
crypto map vpnmap 333 set peer BRANCH SITE IP ADDRESS
crypto map vpnmap 333 set transform-set 3des

crypto map vpnmap interface outside
crypto ca trustpoint WebVPN

crypto isakmp identity hostname
crypto isakmp enable outside
crypto isakmp policy 9
 authentication pre-share
 encryption aes-256
 hash sha
 group 2
 lifetime 3600
crypto isakmp policy 10
 authentication pre-share
 encryption 3des
 hash sha
 group 2
 lifetime 86400
telnet timeout 5
ssh 10.1.8.0 255.255.255.0 inside
ssh timeout 60
console timeout 0

dhcpd dns 208.67.222.222 208.67.220.220 interface internet-guest
dhcpd lease 43200 interface internet-guest
!
dhcpd dns 208.67.222.222 208.67.220.220 interface internet-employee
dhcpd lease 43200 interface internet-employee
!
dhcpd address 192.168.1.2-192.168.1.254 management
dhcpd enable management
!
priority-queue outside
  tx-ring-limit 256
threat-detection basic-threat
threat-detection statistics access-list
no threat-detection statistics tcp-intercept
ntp server 10.1.1.7
ssl trust-point WebVPN outside
webvpn
 enable outside
 svc image disk0:/anyconnect-win-3.1.01065-k9.pkg 1

tunnel-group BRANCH SITE IP ADDRESS type ipsec-l2l
tunnel-group BRANCH SITE IP ADDRESS ipsec-attributes
 pre-shared-key *****
!
class-map VPN-Priority
 match access-list VPN-Priority
!
!
policy-map VPN-QOS
 class VPN-Priority
  priority
!
service-policy VPN-QOS interface outside
prompt hostname context
no call-home reporting anonymous
call-home
 profile CiscoTAC-1
  no active
  destination address http https://tools.cisco.com/its/service/oddce/services/DDCEService
  destination address email callhome@cisco.com
  destination transport-method http
  subscribe-to-alert-group diagnostic
  subscribe-to-alert-group environment
  subscribe-to-alert-group inventory periodic monthly
  subscribe-to-alert-group configuration periodic monthly
  subscribe-to-alert-group telemetry periodic daily
Cryptochecksum:c04f670ab6e503e49ccbdf0f54e83081
: end
[OK]
asa5510#

[Jouni Forss · Accepted solution]

Hi,

Well I can't really find anything wrong with the configurations at the moment.

Since the actual L2L VPN connections is up it would usually point to a problem with either the actual hosts behind the ASA or perhaps a NAT configurations.

Your NAT0 configurations however seem fine and really the only configuration should be able to override and cause problems for it is a "static" configuration that forward the traffic the wrong interface. Since you say that traffic is getting encapsulated from the HQ then it would seem that the problem isn't there atleast and it seems that you have posted the whole Branch configuration that doesnt contain any "static" configurations.

If you want to test traffic from either site to the other sites ASA you could consider doing this. Issue the command "show run management-access" on both units. If the unit doesnt show any "management-access" configurations lines as output of the "show" command then you should be able to issue a "management-access <interface nameif>" for the interface that holds an IP address in the network that is using the L2L VPN connection. Enabling this command on the ASA for an interface should enable you to send ICMP to it through the L2L VPN. It should also enable you to connect to that interface for management purposes through the VPN (provided you have allowed management connections from the appropriate source networks). This command can only be enabled for one interface on the ASA at a given time.

You could now send ICMP from both sites to the other sites ASAs LAN interface and see if that traffic goes through. If the ICMP goes through it would point to a problem with the actual hosts behind the ASAs. Though in your case it seems that the problem is at the Branch site if its not sending any traffic to the L2L VPN.

There have been some bugs on the ASA where the ASA stops encrypting traffic and sending it to the L2L VPN but your software levels dont seem to match the ones where I have seen this problem.

You can naturally also confirm at branch site with the "packet-tracer" command that its test goes through when simulating packets coming from its LAN to the HQ sites LAN through the VPN

packet-tracer input Business-Center tcp 10.3.33.100 12345 10.3.1.100 80

- Jouni

[MavsX4142]

Hi Jouni-

Alright, I had a different reply, but am editing it because i couldn't wait to reply back. So I suspected that since these are public machines, that icmp echo reply is disabled..amongst other things. 

I can't ping machines at their private static ips. i can't ping network at 10.3.33.1. I enable management-access (interface) on each ASA, and suddenly i am getting replies back from 10.3.33.1. Why is it not replying back without this command? I still dont get any echo replies with that command, so that points to a firewall issue on the clients. Then the thought occurred to me. Maybe echos are disabled but RDP is working. So i disabled the management-access command and opened up remote desktop entered in the ip address of the client(since i knew what it was, i was there last week) and bam. I am able to RDP into the machine. So apparently there was nothing wrong with my vpn, I just wasn't opening up the right kind of traffic! Know i know what might be wrong with my other site too..just need to fire up RDP rather than the command prompt!

I just did a show crypto ipsec sa on both and now the encaps and decaps are much higher.

Any idea about why it is not responding to the ping of 10.3.33.1 though?

[Jouni Forss]

Hi,

Cisco firewalls have a default behaviour that they wont let you ICMP a remote interface. What I mean by this is that if the user is behind "inside" interface and the user sends ICMP to the the address of "outside" interface then this will be blocked by the ASA.

In your case your case the ICMP to the internal interface of the Branch ASA is coming through the external interface of that ASA (through the L2L VPN) and therefore the same rule applies. The traffic is blocked.

The "management-access" command is a command specifically meant for a situation where you have a VPN connection and you want to access that remote internal interface of an ASA through that VPN connection. So when that command is applied for a specific interface then you should be able to ICMP through a VPN connection to it and also use management connections.

- Jouni