Access to Remote site through VPN

[ematelyan]

Hello,

I'm trying to troubleshoot an issue with VPN, and I'm hoping someone could give me a hand.

We have 3 offices, each with an ASA 5505 as the router/firewall, connected to the cable modem

(NC office) <----IPSEC----->(PA office) <----IPSEC-----> (CT office)

Internally we have a full mesh VPN, so all offices can talk to each other directly.

I have people at home, using remote access VPN into the PA office, and I need them to be able to connect to the other two offices from there.

I was able to get it to work to the CT office, but I can't get it to work for the NC office.  (What I mean is, users can remote access VPN into the PA office, and access resources in the PA and CT offices, but they can't get to the NC office).

Could someone take a look at these 2 configs, and let me know if I'm missing something?  I'm newer to this, so some of these configs don't have the best naming conventions, but I'm getting there

PA OFFICE

Result of the command: "show run"

: Saved
:
ASA Version 8.2(5)
!
hostname WayneASA

names
!
interface Ethernet0/0
switchport access vlan 2
!
interface Ethernet0/1
!
interface Ethernet0/2
!
interface Ethernet0/3
!
interface Ethernet0/4
!
interface Ethernet0/5
!
interface Ethernet0/6
!
interface Ethernet0/7
!
interface Vlan1
nameif inside
security-level 100
ip address 192.168.1.1 255.255.255.0
!
interface Vlan2
nameif outside
security-level 0
ip address 70.91.18.205 255.255.255.252
!
ftp mode passive
clock timezone EST -5
clock summer-time EDT recurring
dns domain-lookup inside
dns domain-lookup outside
dns server-group DefaultDNS
name-server 75.75.75.75
name-server 75.75.76.76
domain-name 3gtms.com
same-security-traffic permit intra-interface
object-group protocol TCPUDP
protocol-object udp
protocol-object tcp
access-list inside_access_in extended permit ip any any
access-list IPSec_Access extended permit ip 192.168.1.0 255.255.255.0 192.168.2.0 255.255.255.0
access-list IPSec_Access extended permit ip 192.168.10.0 255.255.255.224 192.168.2.0 255.255.255.0
access-list IPSec_Access extended permit ip 192.168.10.0 255.255.255.224 192.168.5.0 255.255.255.0
access-list inside_nat0 extended permit ip 192.168.1.0 255.255.255.0 192.168.10.0 255.255.255.224
access-list inside_nat0 extended permit ip 192.168.1.0 255.255.255.0 192.168.2.0 255.255.255.0
access-list inside_nat0 extended permit ip 192.168.1.0 255.255.255.0 192.168.5.0 255.255.255.0
access-list TunnelSplit1 standard permit 192.168.10.0 255.255.255.224
access-list TunnelSplit1 standard permit 192.168.1.0 255.255.255.0
access-list outside_1_cryptomap extended permit ip 192.168.1.0 255.255.255.0 192.168.2.0 255.255.255.0
access-list outside_2_cryptomap extended permit ip 192.168.1.0 255.255.255.0 192.168.5.0 255.255.255.0
access-list outside_cryptomap extended permit ip 192.168.1.0 255.255.255.0 192.168.5.0 255.255.255.0
access-list RemoteTunnel_splitTunnelAcl standard permit 192.168.1.0 255.255.255.0
access-list RemoteTunnel_splitTunnelAcl_1 standard permit 192.168.1.0 255.255.255.0
access-list RemoteTunnel_splitTunnelAcl_1 standard permit 192.168.2.0 255.255.255.0
access-list RemoteTunnel_splitTunnelAcl_1 standard permit 192.168.5.0 255.255.255.0
access-list out_access_in extended permit udp any host 70.91.18.205 eq sip
access-list out_access_in extended permit tcp any host 70.91.18.205 eq 5000
access-list out_access_in extended permit udp any host 70.91.18.205 range 9000 9049
access-list out_access_in extended permit tcp any host 70.91.18.205 eq sip
access-list out_access_in extended permit object-group TCPUDP any host 70.91.18.205 eq 5090
access-list out_access_in extended permit udp any host 70.91.18.205 eq 5000
access-list outside-nat0 remark NAT0 for VPNPool to Remote Sites
access-list outside-nat0 extended permit ip 192.168.10.0 255.255.255.224 192.168.2.0 255.255.255.0
access-list outside-nat0 extended permit ip 192.168.10.0 255.255.255.224 192.168.5.0 255.255.255.0
pager lines 24
logging asdm informational
mtu inside 1500
mtu outside 1500
ip local pool VPNPool 192.168.10.1-192.168.10.30 mask 255.255.255.224
no failover
icmp unreachable rate-limit 1 burst-size 1
no asdm history enable
arp timeout 14400
global (outside) 1 interface
nat (inside) 0 access-list inside_nat0
nat (inside) 1 0.0.0.0 0.0.0.0
nat (outside) 0 access-list outside-nat0
access-group inside_access_in in interface inside
access-group out_access_in in interface outside
route outside 0.0.0.0 0.0.0.0 70.91.18.206 1
timeout xlate 3:00:00
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02
timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00
timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00
timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute
timeout tcp-proxy-reassembly 0:01:00
timeout floating-conn 0:00:00
dynamic-access-policy-record DfltAccessPolicy
aaa authentication ssh console LOCAL
http server enable
http 0.0.0.0 0.0.0.0 inside
no snmp-server location
no snmp-server contact
snmp-server enable traps snmp authentication linkup linkdown coldstart
crypto ipsec transform-set ESP-AES-256-MD5 esp-aes-256 esp-md5-hmac
crypto ipsec transform-set ESP-DES-SHA esp-des esp-sha-hmac
crypto ipsec transform-set ESP-DES-MD5 esp-des esp-md5-hmac
crypto ipsec transform-set ESP-AES-192-MD5 esp-aes-192 esp-md5-hmac
crypto ipsec transform-set ESP-3DES-MD5 esp-3des esp-md5-hmac
crypto ipsec transform-set ESP-AES-256-SHA esp-aes-256 esp-sha-hmac
crypto ipsec transform-set ESP-AES-128-SHA esp-aes esp-sha-hmac
crypto ipsec transform-set ESP-AES-192-SHA esp-aes-192 esp-sha-hmac
crypto ipsec transform-set ESP-AES-128-MD5 esp-aes esp-md5-hmac
crypto ipsec transform-set ESP-3DES-SHA esp-3des esp-sha-hmac
crypto ipsec transform-set VPNTransformSet esp-3des esp-md5-hmac
crypto ipsec security-association lifetime seconds 28800
crypto ipsec security-association lifetime kilobytes 4608000
crypto dynamic-map SYSTEM_DEFAULT_CRYPTO_MAP 65535 set pfs group1
crypto dynamic-map SYSTEM_DEFAULT_CRYPTO_MAP 65535 set transform-set ESP-AES-128-SHA ESP-AES-128-MD5 ESP-AES-192-SHA ESP-AES-192-MD5 ESP-AES-256-SHA ESP-AES-256-MD5 ESP-3DES-SHA ESP-3DES-MD5 ESP-DES-SHA ESP-DES-MD5
crypto map IPSec_map 1 match address IPSec_Access
crypto map IPSec_map 1 set peer 50.199.234.229
crypto map IPSec_map 1 set transform-set VPNTransformSet
crypto map IPSec_map 2 match address outside_2_cryptomap
crypto map IPSec_map 2 set pfs group1
crypto map IPSec_map 2 set peer 98.101.139.210
crypto map IPSec_map 2 set transform-set VPNTransformSet
crypto map IPSec_map 65535 ipsec-isakmp dynamic SYSTEM_DEFAULT_CRYPTO_MAP
crypto map IPSec_map interface outside
crypto map outside_map 1 match address outside_1_cryptomap
crypto map outside_map 1 set peer 50.199.234.229
crypto isakmp enable outside
crypto isakmp policy 1
authentication pre-share
encryption 3des
hash sha
group 2
lifetime 43200
telnet 192.168.1.0 255.255.255.0 inside
telnet timeout 5
ssh 0.0.0.0 0.0.0.0 inside
ssh timeout 60
console timeout 0
management-access inside
dhcpd auto_config outside
!
dhcpd address 192.168.1.100-192.168.1.199 inside
dhcpd dns 75.75.75.75 75.75.76.76 interface inside
dhcpd enable inside
!

threat-detection basic-threat
threat-detection statistics access-list
no threat-detection statistics tcp-intercept
webvpn
group-policy RemoteTunnel internal
group-policy RemoteTunnel attributes
dns-server value 75.75.75.75 75.75.76.76
vpn-tunnel-protocol IPSec
split-tunnel-policy tunnelspecified
split-tunnel-network-list value RemoteTunnel_splitTunnelAcl_1
username dfavier password vUA99P1dT3fvnDZy encrypted
username dfavier attributes
service-type remote-access
username rduske password vu0Zdx0n3oZWFSaX encrypted
username rduske attributes
service-type remote-access
username eric password 0vcSd5J/TLsFy7nU encrypted privilege 15
username lestofts password URsSXKLozQMSeCBk encrypted
username lestofts attributes
service-type remote-access
username jpwiggins password 3WyoRxmI6LZjGHZE encrypted
username jpwiggins attributes
service-type remote-access
username tomleonard password cQXk0RJCBtxyzZ4K encrypted
username tomleonard attributes
service-type remote-access
username algobel password 4AjIefFXCbu7.T9v encrypted
username algobel attributes
service-type remote-access
tunnel-group RemoteTunnel type remote-access
tunnel-group RemoteTunnel general-attributes
address-pool VPNPool
default-group-policy RemoteTunnel
tunnel-group RemoteTunnel ipsec-attributes
pre-shared-key *****
tunnel-group 50.199.234.229 type ipsec-l2l
tunnel-group 50.199.234.229 ipsec-attributes
pre-shared-key *****
tunnel-group 98.101.139.210 type ipsec-l2l
tunnel-group 98.101.139.210 ipsec-attributes
pre-shared-key *****
!
class-map inspection_default
match default-inspection-traffic
!
!
policy-map type inspect dns preset_dns_map
parameters
  message-length maximum client auto
  message-length maximum 512
policy-map global_policy
class inspection_default
  inspect dns preset_dns_map
  inspect ftp
  inspect h323 h225
  inspect h323 ras
  inspect rsh
  inspect rtsp
  inspect esmtp
  inspect sqlnet
  inspect skinny 
  inspect sunrpc
  inspect xdmcp
  inspect sip 
  inspect netbios
  inspect tftp
  inspect ip-options
  inspect icmp
  inspect pptp
!
service-policy global_policy global
prompt hostname context
no call-home reporting anonymous
Cryptochecksum:6d1ffe8d570d467e1ea6fd60e9457ba1
: end

CT OFFICE

Result of the command: "show run"

: Saved
:
ASA Version 8.2(5)
!
hostname RaleighASA
enable password Ml95GJgphVRqpdJ7 encrypted
passwd 2KFQnbNIdI.2KYOU encrypted
names
!
interface Ethernet0/0
switchport access vlan 2
!
interface Ethernet0/1
!
interface Ethernet0/2
!
interface Ethernet0/3
!
interface Ethernet0/4
!
interface Ethernet0/5
!
interface Ethernet0/6
!
interface Ethernet0/7
!
interface Vlan1
nameif inside
security-level 100
ip address 192.168.5.1 255.255.255.0
!
interface Vlan2
nameif outside
security-level 0
ip address 98.101.139.210 255.0.0.0
!
ftp mode passive
clock timezone EST -5
clock summer-time EDT recurring
dns domain-lookup inside
dns server-group DefaultDNS
name-server 24.25.5.60
name-server 24.25.5.61
same-security-traffic permit intra-interface
object-group protocol TCPUDP
protocol-object udp
protocol-object tcp
access-list Wayne_Access extended permit ip 192.168.5.0 255.255.255.0 192.168.1.0 255.255.255.0
access-list Wayne_Access extended permit ip 192.168.5.0 255.255.255.0 192.168.10.0 255.255.255.0
access-list Shelton_Access extended permit ip 192.168.5.0 255.255.255.0 192.168.2.0 255.255.255.0
access-list out_access_in extended permit tcp any host 98.101.139.210 eq www
access-list out_access_in extended permit tcp any host 98.101.139.210 eq ftp
access-list out_access_in extended permit udp any host 98.101.139.210 eq tftp
access-list out_access_in extended permit udp any host 98.101.139.210 eq sip
access-list out_access_in extended permit tcp any host 98.101.139.210 eq 5090
access-list out_access_in extended permit tcp any host 98.101.139.210 eq 2001
access-list out_access_in extended permit tcp any host 98.101.139.210 eq 5080
access-list out_access_in extended permit tcp any host 98.101.139.210 eq ssh
access-list out_access_in extended permit tcp any host 98.101.139.210 eq 81
access-list out_access_in extended permit tcp any host 98.101.139.210 eq 56774
access-list out_access_in extended permit tcp any host 98.101.139.210 eq 5000
access-list out_access_in extended permit tcp any host 98.101.139.210 eq 902
access-list out_access_in extended permit tcp any host 98.101.139.210 eq netbios-ssn
access-list out_access_in extended permit tcp any host 98.101.139.210 eq 445
access-list out_access_in extended permit tcp any host 98.101.139.210 eq https
access-list out_access_in extended permit object-group TCPUDP any host 98.101.139.210 eq 3389
access-list out_access_in extended permit object-group TCPUDP any host 98.101.139.210 range 5480 5487
access-list out_access_in extended permit udp any host 98.101.139.210 range 9000 9050
access-list inside_nat0 extended permit ip 192.168.5.0 255.255.255.0 192.168.1.0 255.255.255.0
access-list inside_nat0 extended permit ip 192.168.5.0 255.255.255.0 192.168.2.0 255.255.255.0
access-list inside_nat0 extended permit ip 192.168.5.0 255.255.255.0 192.168.10.0 255.255.255.0
pager lines 24
logging asdm informational
mtu inside 1500
mtu outside 1500
icmp unreachable rate-limit 1 burst-size 1
no asdm history enable
arp timeout 14400
global (outside) 1 interface
nat (inside) 0 access-list inside_nat0
nat (inside) 1 0.0.0.0 0.0.0.0

access-group out_access_in in interface outside
route outside 0.0.0.0 0.0.0.0 98.101.139.209 1
timeout xlate 3:00:00
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02
timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00
timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00
timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute
timeout tcp-proxy-reassembly 0:01:00
timeout floating-conn 0:00:00
dynamic-access-policy-record DfltAccessPolicy
aaa authentication ssh console LOCAL
http server enable
http 0.0.0.0 0.0.0.0 inside
no snmp-server location
no snmp-server contact
snmp-server enable traps snmp authentication linkup linkdown coldstart
crypto ipsec transform-set WayneTransform esp-3des esp-md5-hmac
crypto ipsec transform-set SheltonTransform esp-3des esp-md5-hmac
crypto ipsec transform-set ESP-AES-128-SHA esp-aes esp-sha-hmac
crypto ipsec transform-set ESP-AES-128-MD5 esp-aes esp-md5-hmac
crypto ipsec transform-set ESP-AES-192-SHA esp-aes-192 esp-sha-hmac
crypto ipsec transform-set ESP-AES-192-MD5 esp-aes-192 esp-md5-hmac
crypto ipsec transform-set ESP-AES-256-SHA esp-aes-256 esp-sha-hmac
crypto ipsec transform-set ESP-AES-256-MD5 esp-aes-256 esp-md5-hmac
crypto ipsec transform-set ESP-3DES-SHA esp-3des esp-sha-hmac
crypto ipsec transform-set ESP-3DES-MD5 esp-3des esp-md5-hmac
crypto ipsec transform-set ESP-DES-SHA esp-des esp-sha-hmac
crypto ipsec transform-set ESP-DES-MD5 esp-des esp-md5-hmac
crypto ipsec security-association lifetime seconds 28800
crypto ipsec security-association lifetime kilobytes 4608000
crypto map IPSec_map 1 match address Wayne_Access
crypto map IPSec_map 1 set pfs group1
crypto map IPSec_map 1 set peer 70.91.18.205
crypto map IPSec_map 1 set transform-set WayneTransform
crypto map IPSec_map 2 match address Shelton_Access
crypto map IPSec_map 2 set pfs group1
crypto map IPSec_map 2 set peer 50.199.234.229
crypto map IPSec_map 2 set transform-set SheltonTransform
crypto map IPSec_map interface outside
crypto isakmp enable outside
crypto isakmp policy 1
authentication pre-share
encryption 3des
hash sha
group 2
lifetime 43200
telnet timeout 5
ssh 0.0.0.0 0.0.0.0 inside
ssh timeout 5
console timeout 0
management-access inside
dhcpd auto_config outside
!
dhcpd address 192.168.5.100-192.168.5.199 inside
dhcpd dns 24.25.5.60 24.25.5.61 interface inside
dhcpd enable inside
!

threat-detection basic-threat
threat-detection statistics access-list
no threat-detection statistics tcp-intercept
webvpn
username eric password 0vcSd5J/TLsFy7nU encrypted privilege 15
tunnel-group 50.199.234.229 type ipsec-l2l
tunnel-group 50.199.234.229 ipsec-attributes
pre-shared-key *****
tunnel-group 70.91.18.205 type ipsec-l2l
tunnel-group 70.91.18.205 ipsec-attributes
pre-shared-key *****
!
class-map inspection_default
match default-inspection-traffic
!
!
policy-map type inspect dns preset_dns_map
parameters
  message-length maximum client auto
  message-length maximum 512
policy-map global_policy
class inspection_default
  inspect dns preset_dns_map
  inspect ftp
  inspect h323 h225
  inspect h323 ras
  inspect rsh
  inspect rtsp
  inspect esmtp
  inspect sqlnet
  inspect skinny 
  inspect sunrpc
  inspect xdmcp
  inspect sip 
  inspect netbios
  inspect tftp
  inspect ip-options
  inspect icmp
!
service-policy global_policy global
prompt hostname context
no call-home reporting anonymous
Cryptochecksum:3d770ba9647ffdc22b3637e1e5b9a955
: end

[Jouni Forss]

Hi,

I vaguely remember that we at some point already were looking to getting this working here on the forums?

To me it seems you might need the following configurations

PA OFFICE

access-list outside_2_cryptomap extended permit ip 192.168.10.0 255.255.255.0 192.168.5.0 255.255.255.0

- Jouni

[ematelyan]

Hey Jouni,

Yes, you helped me with getting PA connected to CT.

I just swapped out the router in NC with an ASA 5505 last night, so now it's a little easier to manage.

I added the command to the PA ASA.  I tried to ping NC ASA, 192.168.5.1, but it's not making it.

I also tried to RDP to one of the servers in NC, same deal.

[Jouni Forss]

Hi,

Can you generate some traffic from VPN Client to the other site and then take the outputs

PA OFFICE

show crypto ipsec sa peer 98.101.139.210

CT OFFICE

show crypto ipsec sa peer 70.91.18.205

- Jouni

[ematelyan]

The connection between PA and CT is working at the moment, just to point out.  But there's the outputs from them.

PA OFFICE

Result of the command: "show crypto ipsec sa peer 98.101.139.210"

peer address: 98.101.139.210
    Crypto map tag: IPSec_map, seq num: 2, local addr: 70.91.18.205

      access-list outside_2_cryptomap extended permit ip 192.168.1.0 255.255.255.0 192.168.5.0 255.255.255.0
      local ident (addr/mask/prot/port): (192.168.1.0/255.255.255.0/0/0)
      remote ident (addr/mask/prot/port): (192.168.5.0/255.255.255.0/0/0)
      current_peer: 98.101.139.210

      #pkts encaps: 383752, #pkts encrypt: 383752, #pkts digest: 383752
      #pkts decaps: 423629, #pkts decrypt: 423629, #pkts verify: 423629
      #pkts compressed: 0, #pkts decompressed: 0
      #pkts not compressed: 383752, #pkts comp failed: 0, #pkts decomp failed: 0
      #pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0
      #PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0
      #send errors: 0, #recv errors: 0

      local crypto endpt.: 70.91.18.205, remote crypto endpt.: 98.101.139.210

      path mtu 1500, ipsec overhead 58, media mtu 1500
      current outbound spi: 3D24A93C
      current inbound spi : BF762A3D

    inbound esp sas:
      spi: 0xBF762A3D (3212192317)
         transform: esp-3des esp-md5-hmac no compression
         in use settings ={L2L, Tunnel, PFS Group 1, }
         slot: 0, conn_id: 18243584, crypto-map: IPSec_map
         sa timing: remaining key lifetime (kB/sec): (3874827/22189)
         IV size: 8 bytes
         replay detection support: Y
         Anti replay bitmap:
          0xFFFFFFFF 0xFFFFFFFF
    outbound esp sas:
      spi: 0x3D24A93C (1025812796)
         transform: esp-3des esp-md5-hmac no compression
         in use settings ={L2L, Tunnel, PFS Group 1, }
         slot: 0, conn_id: 18243584, crypto-map: IPSec_map
         sa timing: remaining key lifetime (kB/sec): (3871278/22189)
         IV size: 8 bytes
         replay detection support: Y
         Anti replay bitmap:
          0x00000000 0x00000001

CT OFFICE

Result of the command: "show crypto ipsec sa peer 70.91.18.205"

There are no ipsec sas for peer 70.91.18.205

[Jouni Forss]

Hi,

The other end not showing any output doesnt make sense.

If there is a L2L VPN between them and its in use there should be output.

The purpose of those commands were to show output at PA office for the command "show crypto ipsec sa peer " and show output at CT for the command "show crypto ipsec sa peer "

This is to confirm if there is any mention of the VPN Pool network and the CT LAN network in the output.

As there is no output it would seem that the PA ASA has not forwarded any traffic from the VPN client to the L2L VPN connection between the sites. Therefore no connections cant get from the VPN Client user connected to PA to the site CT.

From what I looked you were only missing the mentioned ACL on the other site which would tell the ASA to forward traffic from the VPN Pool to the CT LAN through the L2L VPN.

The Split Tunnel and NAT rules also seemed to be correct so I am not sure what the problem really is.

I guess you could next connect with VPN client to the PA site and take the output at PA site with "show crypto ipsec sa peer "

This should list if there is anything coming from the client to the PA ASA even.

You can also check the VPN Client statistics through the client software while its active. The most interesting one would ofcourse be if the route section shows the CT network in the list.

Again it seems that everything should be in place already

• You have the PA Split Tunnel ACL that lists the network 192.168.5.0/24
• You have the PA "outside" interface NAT0 configuration that tells the ASA that it shouldnt NAT traffic between the VPN Pool and the CT site
• We added the cryptomap ACL that tells the ASA should forward traffic from VPN Pool to the CT site through the L2L VPN
• The CT site has the corresponding crypto map ACL configured that allows the traffic from the PA VPN Pool to the CT LAN
• There is a NAT0 configuration on the "inside" interface of CT ASA that tells the ASA not to NAT the traffic between these mentioned networks.

Everything should be there.

The only thing  that hits my eye that the VPN Pool is configured with mask 255.255.255.224 and its referenced in the configuration both with 255.255.255.0 and 255.255.255.224. I guess it might not hurt to change those network masks in the configurations if nothing else works. It might actually be wise just to keep the configuration concistent.

I guess you could also compare the configuration of PA and the other site for which the VPN Client can access the remote site network. There shouldnt really be anything else different there other than addresses used.

- Jouni

[ematelyan]

Well here's the results of the "show crypto ipsec sa peer 100.45.29.45" while the client is connected to the PA ASA through VPN.

And just to clarify, the access to the CT (192.168.2.0) site through Remote VPN is working fine.  It's the access to the NC (192.168.5.0) site that's not working.  Just making sure, as you mentioned "CT", and I kind of got confused.

Lastly, I modified the subnet masks for 192.168.10.0 to match (255.255.255.224).  Tried again, but still can't ping.

Would a packet capture give some more info as to why it's not going through?  If so, I could use some guidance on the best way of doing that...

Result of the command: "show crypto ipsec sa peer 100.45.29.45"

peer address: 100.45.29.45
    Crypto map tag: SYSTEM_DEFAULT_CRYPTO_MAP, seq num: 65535, local addr: 70.91.18.205

      local ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)
      remote ident (addr/mask/prot/port): (192.168.10.2/255.255.255.255/0/0)
      current_peer: 100.45.29.45, username: eric
      dynamic allocated peer ip: 192.168.10.2

      #pkts encaps: 1537, #pkts encrypt: 1537, #pkts digest: 1537
      #pkts decaps: 932, #pkts decrypt: 932, #pkts verify: 932
      #pkts compressed: 0, #pkts decompressed: 0
      #pkts not compressed: 1537, #pkts comp failed: 0, #pkts decomp failed: 0
      #pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0
      #PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0
      #send errors: 0, #recv errors: 0

      local crypto endpt.: 70.91.18.205, remote crypto endpt.: 100.45.29.45

      path mtu 1500, ipsec overhead 74, media mtu 1500
      current outbound spi: 4749A63F
      current inbound spi : BE19E8E5

    inbound esp sas:
      spi: 0xBE19E8E5 (3189369061)
         transform: esp-aes esp-sha-hmac no compression
         in use settings ={RA, Tunnel, }
         slot: 0, conn_id: 18358272, crypto-map: SYSTEM_DEFAULT_CRYPTO_MAP
         sa timing: remaining key lifetime (sec): 28701
         IV size: 16 bytes
         replay detection support: Y
         Anti replay bitmap:
          0xFFFFFFFF 0xFFFFFFFF
    outbound esp sas:
      spi: 0x4749A63F (1196009023)
         transform: esp-aes esp-sha-hmac no compression
         in use settings ={RA, Tunnel, }
         slot: 0, conn_id: 18358272, crypto-map: SYSTEM_DEFAULT_CRYPTO_MAP
         sa timing: remaining key lifetime (sec): 28701
         IV size: 16 bytes
         replay detection support: Y
         Anti replay bitmap:
          0x00000000 0x00000001

[Jouni Forss]

Hi,

Your original post states the network 192.168.5.0/24 being CT Office?

And didnt remember that the ASA would show the "0.0.0.0" with "show crypto ipsec sa" even though the client is using Split Tunnel

One idea was to compare the configuration between the VPN client and the 192.168.2.0/24 site to the VPN Client to the 192.168.5.0/24 since they should be configured in the same way. And one of them is working and the other is not.

If we really are not getting any output when checking the connection between the main site and the remote site you CANT reach with VPN Client then it would seem that the traffic is either not even going to the VPN Client connection or something on the first ASA is stopping the traffic.

We cant use packet capture to to help with this as all the traffic is encrypted. Or I am not sure if the ASA can get anything between the 2 VPN connections (VPN CLient and the L2L VPN to the site) Naturally you could capture traffic on the 192.168.5.0/24 site but at the moment it seems that the traffic isnt even reaching the site so I dont know if that really helps at all.

- Jouni

[ematelyan]

Hey Jouni,

Sorry about the typo.  Just to clarify,

PA Office (main office that I'm VPNing into) - 192.168.1.0

CT Office (the one that is working) - 192.168.2.0

NC Office (the one that is not working) - 192.168.5.0

I've been going through both configs, taking out any commands that are exactly the same, or were not relevant to the issue at hand. Just to make it easier to compare the two.

I'm going to start comparing the two now, but I'm going to post both of them, just in case I forget cause it's getting late.  Maybe you can catch something that I'm missing.

NC Office - Not working

interface Vlan1
nameif inside
security-level 100
ip address 192.168.5.1 255.255.255.0
!
interface Vlan2
nameif outside
security-level 0
ip address 98.101.139.210 255.0.0.0
dns domain-lookup inside
dns domain-lookup outside
dns server-group DefaultDNS
name-server 24.25.5.60
name-server 24.25.5.61
same-security-traffic permit intra-interface
object-group protocol TCPUDP
protocol-object udp
protocol-object tcp
access-list Wayne_Access extended permit ip 192.168.5.0 255.255.255.0 192.168.1.0 255.255.255.0
access-list Wayne_Access extended permit ip 192.168.5.0 255.255.255.0 192.168.10.0 255.255.255.224
access-list Shelton_Access extended permit ip 192.168.5.0 255.255.255.0 192.168.2.0 255.255.255.0
access-list out_access_in extended permit tcp any host 98.101.139.210 eq www
access-list out_access_in extended permit tcp any host 98.101.139.210 eq ftp
access-list out_access_in extended permit udp any host 98.101.139.210 eq tftp
access-list out_access_in extended permit udp any host 98.101.139.210 eq sip
access-list out_access_in extended permit tcp any host 98.101.139.210 eq 5090
access-list out_access_in extended permit tcp any host 98.101.139.210 eq 2001
access-list out_access_in extended permit tcp any host 98.101.139.210 eq 5080
access-list out_access_in extended permit tcp any host 98.101.139.210 eq ssh
access-list out_access_in extended permit tcp any host 98.101.139.210 eq 81
access-list out_access_in extended permit tcp any host 98.101.139.210 eq 56774
access-list out_access_in extended permit tcp any host 98.101.139.210 eq 5000
access-list out_access_in extended permit tcp any host 98.101.139.210 eq 902
access-list out_access_in extended permit tcp any host 98.101.139.210 eq netbios-ssn
access-list out_access_in extended permit tcp any host 98.101.139.210 eq 445
access-list out_access_in extended permit tcp any host 98.101.139.210 eq https
access-list out_access_in extended permit object-group TCPUDP any host 98.101.139.210 eq 3389
access-list out_access_in extended permit object-group TCPUDP any host 98.101.139.210 range 5480 5487
access-list out_access_in extended permit udp any host 98.101.139.210 range 9000 9050
access-list out_access_in extended permit icmp any any
access-list inside_nat0 extended permit ip 192.168.5.0 255.255.255.0 192.168.1.0 255.255.255.0
access-list inside_nat0 extended permit ip 192.168.5.0 255.255.255.0 192.168.2.0 255.255.255.0
access-list inside_nat0 extended permit ip 192.168.5.0 255.255.255.0 192.168.10.0 255.255.255.224
global (outside) 1 interface
nat (inside) 0 access-list inside_nat0
nat (inside) 1 0.0.0.0 0.0.0.0
static (inside,outside) tcp interface www 192.168.5.52 8080 netmask 255.255.255.255
static (inside,outside) tcp interface ftp 192.168.5.10 ftp netmask 255.255.255.255
static (inside,outside) udp interface tftp 192.168.5.10 tftp netmask 255.255.255.255
static (inside,outside) udp interface sip 192.168.5.11 sip netmask 255.255.255.255
static (inside,outside) tcp interface 5090 192.168.5.11 5090 netmask 255.255.255.255
static (inside,outside) tcp interface 2001 192.168.5.10 2001 netmask 255.255.255.255
static (inside,outside) tcp interface 5080 192.168.5.11 5080 netmask 255.255.255.255
static (inside,outside) tcp interface ssh 192.168.5.200 ssh netmask 255.255.255.255
static (inside,outside) tcp interface 81 192.168.5.20 www netmask 255.255.255.255
static (inside,outside) tcp interface 56774 192.168.5.10 1823 netmask 255.255.255.255
static (inside,outside) tcp interface 5000 192.168.5.11 5000 netmask 255.255.255.255
static (inside,outside) udp interface 3389 192.168.5.10 3389 netmask 255.255.255.255
static (inside,outside) tcp interface 3389 192.168.5.10 3389 netmask 255.255.255.255
access-group out_access_in in interface outside
route outside 0.0.0.0 0.0.0.0 98.101.139.209 1
dynamic-access-policy-record DfltAccessPolicy
aaa authentication ssh console LOCAL
http server enable
http 0.0.0.0 0.0.0.0 inside
snmp-server enable traps snmp authentication linkup linkdown coldstart
crypto ipsec transform-set WayneTransform esp-3des esp-md5-hmac
crypto ipsec transform-set SheltonTransform esp-3des esp-md5-hmac
crypto ipsec transform-set ESP-3DES-MD5 esp-3des esp-md5-hmac
crypto ipsec security-association lifetime seconds 28800
crypto ipsec security-association lifetime kilobytes 4608000
crypto map IPSec_map 1 match address Wayne_Access
crypto map IPSec_map 1 set pfs group1
crypto map IPSec_map 1 set peer 70.91.18.205
crypto map IPSec_map 1 set transform-set WayneTransform
crypto map IPSec_map 2 match address Shelton_Access
crypto map IPSec_map 2 set pfs group1
crypto map IPSec_map 2 set peer 50.199.234.229
crypto map IPSec_map 2 set transform-set SheltonTransform
crypto map IPSec_map interface outside
crypto isakmp enable outside
crypto isakmp policy 1
authentication pre-share
encryption 3des
hash sha
group 2
lifetime 43200
telnet timeout 5
ssh 0.0.0.0 0.0.0.0 inside
ssh timeout 5
console timeout 0
management-access inside
dhcpd auto_config outside
!
dhcpd address 192.168.5.100-192.168.5.199 inside
dhcpd dns 24.25.5.60 24.25.5.61 interface inside
dhcpd enable inside

username eric password 0vcSd5J/TLsFy7nU encrypted privilege 15
tunnel-group 50.199.234.229 type ipsec-l2l
tunnel-group 50.199.234.229 ipsec-attributes
pre-shared-key *****
tunnel-group 70.91.18.205 type ipsec-l2l
tunnel-group 70.91.18.205 ipsec-attributes
pre-shared-key *****

CT Office - Working

interface Vlan1
nameif inside
security-level 100
ip address 192.168.2.1 255.255.255.0
!
interface Vlan2
nameif outside
security-level 0
ip address Shelton-Firewall 255.255.255.252
dns domain-lookup inside
dns domain-lookup outside
dns server-group DefaultDNS
name-server 75.75.75.75
name-server 75.75.76.76
object-group protocol TCPUDP
protocol-object udp
protocol-object tcp
access-list inside_access_in extended permit ip any any
access-list inside_nat0 extended permit ip 192.168.2.0 255.255.255.0 192.168.11.0 255.255.255.224
access-list inside_nat0 extended permit ip 192.168.2.0 255.255.255.0 Wayne-Network 255.255.255.0
access-list inside_nat0 extended permit ip 192.168.2.0 255.255.255.0 192.168.5.0 255.255.255.0
access-list inside_nat0 extended permit ip 192.168.2.0 255.255.255.0 192.168.10.0 255.255.255.224
access-list TunnelSplit1 standard permit 192.168.11.0 255.255.255.224
access-list TunnelSplit1 standard permit 192.168.2.0 255.255.255.0
access-list IPSec_Access extended permit ip 192.168.2.0 255.255.255.0 Wayne-Network 255.255.255.0
access-list IPSec_Access extended permit ip 192.168.2.0 255.255.255.0 192.168.10.0 255.255.255.224
access-list RemoteTunnel_splitTunnelAcl standard permit 192.168.2.0 255.255.255.0
access-list outside_access_in extended permit object-group TCPUDP any host Shelton-Firewall eq 5065
access-list outside_access_in extended permit udp any host Shelton-Firewall eq 14000
access-list outside_access_in extended permit udp any host Shelton-Firewall eq 14002
access-list outside_access_in extended permit udp any host Shelton-Firewall eq 14003
access-list outside_access_in extended permit udp any host Shelton-Firewall eq 14004
access-list outside_access_in extended permit udp any host Shelton-Firewall eq 14005
access-list outside_access_in extended permit udp any host Shelton-Firewall eq 14006
access-list outside_access_in extended permit udp any host Shelton-Firewall eq 14007
access-list outside_access_in extended permit udp any host Shelton-Firewall eq 14008
access-list outside_access_in extended permit udp any host Shelton-Firewall eq 14009
access-list outside_access_in extended permit udp any host Shelton-Firewall eq 14001
access-list outside_access_in extended permit tcp any host Shelton-Firewall eq 5000
access-list outside_access_in extended permit udp any host Shelton-Firewall eq 5000
access-list Raleigh_IPSec extended permit ip 192.168.2.0 255.255.255.0 192.168.5.0 255.255.255.0
ip local pool VPNPool 192.168.11.1-192.168.11.30 mask 255.255.255.224
global (outside) 1 interface
nat (inside) 0 access-list inside_nat0
nat (inside) 1 0.0.0.0 0.0.0.0
static (inside,outside) tcp interface 5065 192.168.2.4 5065 netmask 255.255.255.255
static (inside,outside) udp interface 14000 192.168.2.4 14000 netmask 255.255.255.255
static (inside,outside) udp interface 14001 192.168.2.4 14001 netmask 255.255.255.255
static (inside,outside) udp interface 14002 192.168.2.4 14002 netmask 255.255.255.255
static (inside,outside) udp interface 14003 192.168.2.4 14003 netmask 255.255.255.255
static (inside,outside) udp interface 14004 192.168.2.4 14004 netmask 255.255.255.255
static (inside,outside) udp interface 14005 192.168.2.4 14005 netmask 255.255.255.255
static (inside,outside) udp interface 14006 192.168.2.4 14006 netmask 255.255.255.255
static (inside,outside) udp interface 14007 192.168.2.4 14007 netmask 255.255.255.255
static (inside,outside) udp interface 14008 192.168.2.4 14008 netmask 255.255.255.255
static (inside,outside) udp interface 14009 192.168.2.4 14009 netmask 255.255.255.255
static (inside,outside) tcp interface 5000 192.168.2.4 5000 netmask 255.255.255.255
static (inside,outside) udp interface 5000 192.168.2.4 5000 netmask 255.255.255.255
access-group inside_access_in in interface inside
access-group outside_access_in in interface outside
route outside 0.0.0.0 0.0.0.0 50.199.234.230 1
dynamic-access-policy-record DfltAccessPolicy
aaa authentication ssh console LOCAL
http server enable
http 0.0.0.0 0.0.0.0 inside
snmp-server enable traps snmp authentication linkup linkdown coldstart
crypto ipsec transform-set VPNTransformSet esp-3des esp-md5-hmac
crypto ipsec transform-set ESP-3DES-MD5 esp-3des esp-md5-hmac
crypto ipsec security-association lifetime seconds 28800
crypto ipsec security-association lifetime kilobytes 4608000
crypto dynamic-map DynamicMap 1 set pfs group1
crypto dynamic-map DynamicMap 1 set transform-set VPNTransformSet
crypto dynamic-map DynamicMap 1 set reverse-route
crypto dynamic-map SYSTEM_DEFAULT_CRYPTO_MAP 65535 set pfs group1
crypto dynamic-map SYSTEM_DEFAULT_CRYPTO_MAP 65535 set transform-set ESP-AES-128-SHA ESP-AES-128-MD5 ESP-AES-192-SHA ESP-AES-192-MD5 ESP-AES-256-SHA ESP-AES-256-MD5 ESP-3DES-MD5 ESP-DES-SHA ESP-DES-MD5
crypto map VPNMap 1 ipsec-isakmp dynamic DynamicMap
crypto map IPSec_map 1 match address IPSec_Access
crypto map IPSec_map 1 set peer 70.91.18.205
crypto map IPSec_map 1 set transform-set VPNTransformSet
crypto map IPSec_map 2 match address Raleigh_IPSec
crypto map IPSec_map 2 set pfs group1
crypto map IPSec_map 2 set peer 98.101.139.210
crypto map IPSec_map 2 set transform-set ESP-3DES-MD5
crypto map IPSec_map 65535 ipsec-isakmp dynamic SYSTEM_DEFAULT_CRYPTO_MAP
crypto map IPSec_map interface outside
crypto isakmp enable outside
crypto isakmp policy 1
authentication pre-share
encryption 3des
hash sha
group 2
lifetime 43200
telnet timeout 5
ssh 0.0.0.0 0.0.0.0 inside
ssh timeout 60
console timeout 0
management-access inside
dhcpd auto_config outside
!
dhcpd address 192.168.2.100-192.168.2.199 inside
dhcpd dns 75.75.75.75 75.75.76.76 interface inside
dhcpd enable inside

group-policy RemoteTunnel internal
group-policy RemoteTunnel attributes
dns-server value 75.75.75.75 75.75.76.76
vpn-tunnel-protocol IPSec
split-tunnel-policy tunnelspecified
split-tunnel-network-list value RemoteTunnel_splitTunnelAcl
username eric password 0vcSd5J/TLsFy7nU encrypted
tunnel-group 50.199.234.229 type ipsec-l2l
tunnel-group 70.91.18.205 type ipsec-l2l
tunnel-group 70.91.18.205 ipsec-attributes
pre-shared-key *****
tunnel-group RemoteTunnel type remote-access
tunnel-group RemoteTunnel general-attributes
address-pool VPNPool
default-group-policy RemoteTunnel
tunnel-group RemoteTunnel ipsec-attributes
pre-shared-key *****
tunnel-group 98.101.139.210 type ipsec-l2l
tunnel-group 98.101.139.210 ipsec-attributes
pre-shared-key *****

[Jouni Forss]

Hi,

I guess we could try to confirm that the PA and NC can form the L2L VPN connection with regards to the connection between the local NC network and the VPN Pool at PA site.

You could try this "packet-tracer" command TWICE on the NC ASA and copy/paste the output here

packet-tracer input inside tcp 192.168.5.100 12345 192.168.10.100 80

This should tell us if the 2 ASAs can form the required L2L VPN connectivity to make it possible for the VPN Clients even form connections.

After the "packet-tracer" output also take this output on the NC ASA

show crypto ipsec sa peer 70.91.18.205

Naturally if the "packet-tracer" ends with a DROP result there probably isnt anything interesting in this output. But as I said, take the "packet-tracer" output twice as the first one might fails usually with a DROP result.

- Jouni

[ematelyan]

Here's the first one,

Result of the command: "packet-tracer input inside tcp 192.168.5.100 12345 192.168.10.100 80"

Phase: 1
Type: CAPTURE
Subtype:
Result: ALLOW
Config:
Additional Information:
MAC Access list

Phase: 2
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
Additional Information:
MAC Access list

Phase: 3
Type: ROUTE-LOOKUP
Subtype: input
Result: ALLOW
Config:
Additional Information:
in   0.0.0.0         0.0.0.0         outside

Phase: 4
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group inside_access_in in interface inside
access-list inside_access_in extended permit ip any any
Additional Information:

Phase: 5
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 6
Type: NAT
Subtype:
Result: ALLOW
Config:
nat (inside) 1 0.0.0.0 0.0.0.0
  match ip inside any outside any
    dynamic translation to pool 1 (98.101.139.210 [Interface PAT])
    translate_hits = 92342, untranslate_hits = 4034
Additional Information:
Dynamic translate 192.168.5.100/12345 to 98.101.139.210/15854 using netmask 255.255.255.255

Phase: 7
Type: NAT
Subtype: host-limits
Result: ALLOW
Config:
nat (inside) 1 0.0.0.0 0.0.0.0
  match ip inside any inside any
    dynamic translation to pool 1 (No matching global)
    translate_hits = 0, untranslate_hits = 0
Additional Information:

Phase: 8
Type: HOST-LIMIT
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 9
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 10
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 217608, packet dispatched to next module

Result:
input-interface: inside
input-status: up
input-line-status: up
output-interface: outside
output-status: up
output-line-status: up
Action: allow

Here's the second one,

Result of the command: "packet-tracer input inside tcp 192.168.5.100 12345 192.168.10.100 80"

Phase: 1
Type: CAPTURE
Subtype:
Result: ALLOW
Config:
Additional Information:
MAC Access list

Phase: 2
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
Additional Information:
MAC Access list

Phase: 3
Type: ROUTE-LOOKUP
Subtype: input
Result: ALLOW
Config:
Additional Information:
in   0.0.0.0         0.0.0.0         outside

Phase: 4
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group inside_access_in in interface inside
access-list inside_access_in extended permit ip any any
Additional Information:

Phase: 5
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 6
Type: NAT
Subtype:
Result: ALLOW
Config:
nat (inside) 1 0.0.0.0 0.0.0.0
  match ip inside any outside any
    dynamic translation to pool 1 (98.101.139.210 [Interface PAT])
    translate_hits = 92351, untranslate_hits = 4034
Additional Information:
Dynamic translate 192.168.5.100/12345 to 98.101.139.210/43802 using netmask 255.255.255.255

Phase: 7
Type: NAT
Subtype: host-limits
Result: ALLOW
Config:
nat (inside) 1 0.0.0.0 0.0.0.0
  match ip inside any inside any
    dynamic translation to pool 1 (No matching global)
    translate_hits = 0, untranslate_hits = 0
Additional Information:

Phase: 8
Type: HOST-LIMIT
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 9
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 10
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 217631, packet dispatched to next module

Result:
input-interface: inside
input-status: up
input-line-status: up
output-interface: outside
output-status: up
output-line-status: up
Action: allow

And here's the last one

Result of the command: "show crypto ipsec sa peer 70.91.18.205"

peer address: 70.91.18.205
    Crypto map tag: IPSec_map, seq num: 1, local addr: 98.101.139.210

      access-list Wayne_Access extended permit ip 192.168.5.0 255.255.255.0 192.168.1.0 255.255.255.0
      local ident (addr/mask/prot/port): (192.168.5.0/255.255.255.0/0/0)
      remote ident (addr/mask/prot/port): (192.168.1.0/255.255.255.0/0/0)
      current_peer: 70.91.18.205

      #pkts encaps: 5862, #pkts encrypt: 5862, #pkts digest: 5862
      #pkts decaps: 2469, #pkts decrypt: 2469, #pkts verify: 2469
      #pkts compressed: 0, #pkts decompressed: 0
      #pkts not compressed: 5862, #pkts comp failed: 0, #pkts decomp failed: 0
      #pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0
      #PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0
      #send errors: 0, #recv errors: 0

      local crypto endpt.: 98.101.139.210, remote crypto endpt.: 70.91.18.205

      path mtu 1500, ipsec overhead 58, media mtu 1500
      current outbound spi: FE583E8C
      current inbound spi : 61927304

    inbound esp sas:
      spi: 0x61927304 (1636987652)
         transform: esp-3des esp-md5-hmac no compression
         in use settings ={L2L, Tunnel, PFS Group 1, }
         slot: 0, conn_id: 16384, crypto-map: IPSec_map
         sa timing: remaining key lifetime (kB/sec): (4372671/12441)
         IV size: 8 bytes
         replay detection support: Y
         Anti replay bitmap:
          0xFFFFFFFF 0xFFFFFFFF
    outbound esp sas:
      spi: 0xFE583E8C (4267196044)
         transform: esp-3des esp-md5-hmac no compression
         in use settings ={L2L, Tunnel, PFS Group 1, }
         slot: 0, conn_id: 16384, crypto-map: IPSec_map
         sa timing: remaining key lifetime (kB/sec): (4372223/12441)
         IV size: 8 bytes
         replay detection support: Y
         Anti replay bitmap:
          0x00000000 0x00000001

[Jouni Forss]

Hi,

I guess you still havent changed the 192.168.10.0 255.255.255.224 to 192.168.10.0 255.255.255.0 on all the configurations?

As we can see from the above the traffic wouldnt match the L2L VPN rules and would just be pushed out the local Internet connection

I guess with the current configurations using the original mask you would have to try something like this

packet-tracer input inside tcp 192.168.5.100 12345 192.168.10.10 80

- Jouni

[ematelyan]

Hey Jouni,

I was originally trying to change them all to 192.168.10.0 255.255.255.224, but I missed one. 

All of the commands with 192.168.10.0 have a subnet mask of 255.255.255.224 now

I ran this command twice on the NC ASA,

packet-tracer input inside tcp 192.168.5.100 12345 192.168.10.10 80

Result of the command: "packet-tracer input inside tcp 192.168.5.100 12345 192.168.10.10 80"

Phase: 1
Type: CAPTURE
Subtype:
Result: ALLOW
Config:
Additional Information:
MAC Access list

Phase: 2
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
Additional Information:
MAC Access list

Phase: 3
Type: ROUTE-LOOKUP
Subtype: input
Result: ALLOW
Config:
Additional Information:
in   0.0.0.0         0.0.0.0         outside

Phase: 4
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group inside_access_in in interface inside
access-list inside_access_in extended permit ip any any
Additional Information:

Phase: 5
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 6
Type: NAT-EXEMPT
Subtype:
Result: ALLOW
Config:
  match ip inside 192.168.5.0 255.255.255.0 outside 192.168.10.0 255.255.255.224
    NAT exempt
    translate_hits = 1, untranslate_hits = 0
Additional Information:

Phase: 7
Type: NAT
Subtype:
Result: ALLOW
Config:
nat (inside) 1 0.0.0.0 0.0.0.0
  match ip inside any outside any
    dynamic translation to pool 1 (98.101.139.210 [Interface PAT])
    translate_hits = 95161, untranslate_hits = 4069
Additional Information:

Phase: 8
Type: NAT
Subtype: host-limits
Result: ALLOW
Config:
nat (inside) 1 0.0.0.0 0.0.0.0
  match ip inside any inside any
    dynamic translation to pool 1 (No matching global)
    translate_hits = 0, untranslate_hits = 0
Additional Information:

Phase: 9
Type: HOST-LIMIT
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 10
Type: VPN
Subtype: encrypt
Result: DROP
Config:
Additional Information:

Result:
input-interface: inside
input-status: up
input-line-status: up
output-interface: outside
output-status: up
output-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule

Result of the command: "packet-tracer input inside tcp 192.168.5.100 12345 192.168.10.10 80"

Phase: 1
Type: CAPTURE
Subtype:
Result: ALLOW
Config:
Additional Information:
MAC Access list

Phase: 2
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
Additional Information:
MAC Access list

Phase: 3
Type: ROUTE-LOOKUP
Subtype: input
Result: ALLOW
Config:
Additional Information:
in   0.0.0.0         0.0.0.0         outside

Phase: 4
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group inside_access_in in interface inside
access-list inside_access_in extended permit ip any any
Additional Information:

Phase: 5
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 6
Type: NAT-EXEMPT
Subtype:
Result: ALLOW
Config:
  match ip inside 192.168.5.0 255.255.255.0 outside 192.168.10.0 255.255.255.224
    NAT exempt
    translate_hits = 3, untranslate_hits = 0
Additional Information:

Phase: 7
Type: NAT
Subtype:
Result: ALLOW
Config:
nat (inside) 1 0.0.0.0 0.0.0.0
  match ip inside any outside any
    dynamic translation to pool 1 (98.101.139.210 [Interface PAT])
    translate_hits = 95439, untranslate_hits = 4090
Additional Information:

Phase: 8
Type: NAT
Subtype: host-limits
Result: ALLOW
Config:
nat (inside) 1 0.0.0.0 0.0.0.0
  match ip inside any inside any
    dynamic translation to pool 1 (No matching global)
    translate_hits = 0, untranslate_hits = 0
Additional Information:

Phase: 9
Type: HOST-LIMIT
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 10
Type: VPN
Subtype: encrypt
Result: ALLOW
Config:
Additional Information:

Phase: 11
Type: VPN
Subtype: ipsec-tunnel-flow
Result: ALLOW
Config:
Additional Information:

Phase: 12
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 13
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 222391, packet dispatched to next module

Result:
input-interface: inside
input-status: up
input-line-status: up
output-interface: outside
output-status: up
output-line-status: up
Action: allow

[Jouni Forss · Accepted solution]

Hi,

I might have found the problem.

To be honest I am a bit tired and concentration is hard Especially when jumping between several device configurations. So second pair of eyes might be in order.

At the moment it seems to me that this configuration is the problem at the PA SITE

access-list IPSec_Access extended permit ip 192.168.10.0 255.255.255.224 192.168.5.0 255.255.255.0

This is an ACL that defines the local and remote networks for a L2L VPN connection.

Now when we look to which L2L VPN connection this belong we see the following

crypto map IPSec_map 1 match address IPSec_Access

crypto map IPSec_map 1 set peer 50.199.234.229

crypto map IPSec_map 1 set transform-set VPNTransformSet

Now we see that the peer IP address is 50.199.234.229. What site is this? Its the IP address of the CT Site that is working right?

Now what the ACL line I mentioned earlier basically tells is that when the network 192.168.10.0 255.255.255.224 wants to connect to network 192.168.5.0/24 it should be forwarded to the CT Site. And naturally this should not be the case as we want that traffic to go to the NC Site

Also worth noting is that on the PA SITE the above connection is configured with the priority "1" so it gets matched first against a connection. If the L2L VPN configurations were in other order then the VPN Client connection might be actually working. But this is just something I wanted to point out. The actual resolution to the problem is naturally removing the configuration that is causing the actual problem in which ASA is trying to forward the traffic to a totally wrong place.

So can you next remove this ACL line from the PA ASA

no access-list IPSec_Access extended permit ip 192.168.10.0 255.255.255.224 192.168.5.0 255.255.255.0

Then test the VPN Client to NC SITE connection again.

Hope this is finally the solution

- Jouni

[ematelyan]

That did it!!!  Through the remote vpn, I can ping 192.168.5.1, and RDP to 192.168.5.10

You don't know how happy that makes me!

I owe you one!  I wish I could send you a digital beer or something :)