Re: LAN-to-LAN IPsec VPN tunnel traffic not being routed

mattkl3com · ‎02-25-2011

I am trying to set up a LAN-to-LAN VPN tunnel between two sites. One site has a 5505, and the other site has a 5510. It looks like the tunnel is being established fine (both ISAKMP and IPSEC SAs look OK), but traffic doesn't appear to be routing across the internet between the devices.

Configuration for 5505 (reduced):

ASA Version 8.2(1)

!

interface Vlan1

nameif inside

security-level 100

ip address 192.168.103.26 255.255.255.0

!

interface Vlan2

mac-address 0040.1018.fab7

nameif outside

security-level 0

ip address <asa_5505_ext_IP> 255.255.255.240

!

interface Vlan5

no forward interface Vlan1

nameif dmz

security-level 50

no ip address

!

same-security-traffic permit intra-interface

access-list outside_access_out extended permit ip any any
access-list inside_access_in extended permit ip any any
access-list inside_access_in extended permit ip any interface outside
access-list inside_access_out extended permit ip any any

access-list no-nat extended permit ip any 192.168.103.240 255.255.255.240

access-list no-nat extended permit ip 192.168.103.0 255.255.255.0 192.168.110.0 255.255.255.0

access-list vpn_list extended permit ip 192.168.103.0 255.255.255.0 192.168.110.0 255.255.255.0

access-group inside_access_in in interface inside
access-group inside_access_out out interface inside
access-group outside_access_in in interface outside
access-group outside_access_out out interface outside

ip local pool vpnpool 192.168.103.240-192.168.103.254 mask 255.255.255.0

icmp unreachable rate-limit 1 burst-size 1

icmp permit any inside

icmp permit any outside

global (inside) 1 interface

global (outside) 1 interface

nat (inside) 0 access-list no-nat

nat (inside) 1 0.0.0.0 0.0.0.0 dns

route outside 0.0.0.0 0.0.0.0 <internet_gateway_IP1> 1

crypto ipsec transform-set ESP-3DES-SHA esp-3des esp-sha-hmac

crypto ipsec transform-set ESP-3DES-SHA mode transport

crypto ipsec security-association lifetime seconds 28800

crypto ipsec security-association lifetime kilobytes 4608000

crypto map l2lmap 10 match address vpn_list

crypto map l2lmap 10 set peer <asa_5510_ext_IP>

crypto map l2lmap 10 set transform-set ESP-3DES-SHA

crypto map l2lmap interface outside

crypto isakmp identity address

crypto isakmp enable inside

crypto isakmp enable outside

crypto isakmp policy 5

authentication pre-share

encryption 3des

hash sha

group 2

lifetime 86400

tunnel-group <asa_5510_ext_IP> type ipsec-l2l

tunnel-group <asa_5510_ext_IP> ipsec-attributes

pre-shared-key *

class-map inspection_default

match default-inspection-traffic

policy-map global_policy

class inspection_default

inspect pptp

inspect http

inspect icmp

inspect ftp

Configuration for 5510 (reduced):

ASA Version 8.2(2)

!

interface Ethernet0/0

nameif outside

security-level 0

ip address <asa_5510_ext_IP> 255.255.255.0

!

interface Ethernet0/1

nameif inside

security-level 100

ip address 192.168.110.150 255.255.255.0

!

interface Ethernet0/2

no nameif

no security-level

no ip address

!

interface Ethernet0/3

shutdown

no nameif

no security-level

no ip address

!

interface Management0/0

nameif management

security-level 100

ip address 192.168.1.1 255.255.255.0

management-only

access-list inside_access_in extended permit ip any any

access-list outside_access_out extended permit ip any any

access-list no-nat extended permit ip 192.168.110.0 255.255.255.0 192.168.110.0 255.255.255.0

access-list no-nat extended permit ip 192.168.110.0 255.255.255.0 192.168.103.0 255.255.255.0

access-list vpn_list extended permit ip 192.168.110.0 255.255.255.0 192.168.103.0 255.255.255.0

access-group outside_access_out out interface outside
access-group inside_access_in in interface inside

ip local pool vpnpool 192.168.110.50-192.168.110.99 mask 255.255.255.0

icmp unreachable rate-limit 1 burst-size 1

icmp permit any outside

icmp permit any inside

global (outside) 1 interface

global (inside) 1 interface

nat (inside) 0 access-list no-nat

nat (inside) 1 192.168.110.0 255.255.255.0 dns

route outside 0.0.0.0 0.0.0.0 <internet_gateway_IP2> 1

crypto ipsec transform-set ESP_3DES_SHA esp-3des esp-sha-hmac

crypto ipsec transform-set ESP_3DES_SHA mode transport

crypto ipsec security-association lifetime seconds 28800

crypto ipsec security-association lifetime kilobytes 4608000

crypto map l2lmap 20 match address vpn_list

crypto map l2lmap 20 set peer <asa_5505_ext_IP>

crypto map l2lmap 20 set transform-set ESP_3DES_SHA

crypto map l2lmap interface outside

crypto isakmp identity address

crypto isakmp enable outside

crypto isakmp policy 10

authentication pre-share

encryption 3des

hash sha

group 2

lifetime 86400

telnet timeout 5

tunnel-group <asa_5505_ext_IP> type ipsec-l2l

tunnel-group <asa_5505_ext_IP> ipsec-attributes

pre-shared-key *****

class-map inspection_default

match default-inspection-traffic

policy-map type inspect dns migrated_dns_map_1

parameters

message-length maximum client auto

message-length maximum 512

policy-map global_policy

class inspection_default

inspect dns migrated_dns_map_1

inspect ftp

inspect h323 h225

inspect h323 ras

inspect rsh

inspect rtsp

inspect esmtp

inspect sqlnet

inspect skinny

inspect sunrpc

inspect xdmcp

inspect sip

inspect netbios

inspect tftp

inspect ip-options

inspect icmp

ISAKMP/IKE SAs:

Active SA: 1

Rekey SA: 0 (A tunnel will report 1 Active and 1 Rekey SA during rekey)

Total IKE SA: 1

1 IKE Peer: <asa_5505_ext_IP>

Type : L2L Role : initiator

Rekey : no State : MM_ACTIVE

Encrypt : 3des Hash : SHA

Auth : preshared Lifetime: 86400

Lifetime Remaining: 85964

Active SA: 1

Rekey SA: 0 (A tunnel will report 1 Active and 1 Rekey SA during rekey)

Total IKE SA: 1

1 IKE Peer: <asa_5510_ext_IP>

Type : L2L Role : responder

Rekey : no State : MM_ACTIVE

Encrypt : 3des Hash : SHA

Auth : preshared Lifetime: 86400

Lifetime Remaining: 85436

IPSec SAs:

interface: outside

Crypto map tag: l2lmap, seq num: 20, local addr: <asa_5510_ext_IP>

access-list vpn_list extended permit ip 192.168.110.0 255.255.255.0 192.168.103.0 255.255.255.0

local ident (addr/mask/prot/port): (192.168.110.0/255.255.255.0/0/0)

remote ident (addr/mask/prot/port): (192.168.103.0/255.255.255.0/0/0)

current_peer: <asa_5505_ext_IP>

#pkts encaps: 27, #pkts encrypt: 27, #pkts digest: 27

#pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0

#pkts compressed: 0, #pkts decompressed: 0

#pkts not compressed: 27, #pkts comp failed: 0, #pkts decomp failed: 0

#post-frag successes: 0, #post-frag failures: 0, #fragments created: 0

#PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0

#send errors: 0, #recv errors: 0

local crypto endpt.: <asa_5510_ext_IP>, remote crypto endpt.: <asa_5505_ext_IP>

path mtu 1500, ipsec overhead 58, media mtu 1500

current outbound spi: 5BE7008C

current inbound spi : E5EA0BB1

inbound esp sas:

spi: 0xE5EA0BB1 (3857320881)

transform: esp-3des esp-sha-hmac no compression

in use settings ={L2L, Transport, }

slot: 0, conn_id: 237568, crypto-map: l2lmap

sa timing: remaining key lifetime (kB/sec): (4374000/28312)

IV size: 8 bytes

replay detection support: Y

Anti replay bitmap:

0x00000000 0x00000001

outbound esp sas:

spi: 0x5BE7008C (1541865612)

transform: esp-3des esp-sha-hmac no compression

in use settings ={L2L, Transport, }

slot: 0, conn_id: 237568, crypto-map: l2lmap

sa timing: remaining key lifetime (kB/sec): (4373998/28312)

IV size: 8 bytes

replay detection support: Y

Anti replay bitmap:

0x00000000 0x00000001

Crypto map tag: l2lmap, seq num: 10, local addr: <asa_5505_ext_IP>

access-list palm_ud_vpn permit ip 192.168.103.0 255.255.255.0 192.168.110.0 255.255.255.0

local ident (addr/mask/prot/port): (192.168.103.0/255.255.255.0/0/0)

remote ident (addr/mask/prot/port): (192.168.110.0/255.255.255.0/0/0)

current_peer: <asa_5510_ext_IP>

#pkts encaps: 417, #pkts encrypt: 417, #pkts digest: 417

#pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0

#pkts compressed: 0, #pkts decompressed: 0

#pkts not compressed: 417, #pkts comp failed: 0, #pkts decomp failed: 0

#post-frag successes: 0, #post-frag failures: 0, #fragments created: 0

#PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0

#send errors: 0, #recv errors: 0

local crypto endpt.: <asa_5505_ext_IP>, remote crypto endpt.: <asa_5510_ext_IP>

path mtu 1500, ipsec overhead 58, media mtu 1500

current outbound spi: E5EA0BB1

inbound esp sas:

spi: 0x5BE7008C (1541865612)

transform: esp-3des esp-sha-hmac no compression

in use settings ={L2L, Transport, }

slot: 0, conn_id: 311296, crypto-map: l2lmap

sa timing: remaining key lifetime (kB/sec): (3915000/28238)

IV size: 8 bytes

replay detection support: Y

Anti replay bitmap:

0x00000000 0x00000001

outbound esp sas:

spi: 0xE5EA0BB1 (3857320881)

transform: esp-3des esp-sha-hmac no compression

in use settings ={L2L, Transport, }

slot: 0, conn_id: 311296, crypto-map: l2lmap

sa timing: remaining key lifetime (kB/sec): (3915000/28238)

IV size: 8 bytes

replay detection support: Y

Anti replay bitmap:

0x00000000 0x00000001

I tried to debug this using:

capture test interface outside match ipsec any any

Capture from 5510:

1: 11:27:12.273499 192.168.110.120 > 192.168.103.205: ip-proto-50, length 100

2: 11:27:13.273469 192.168.110.120 > 192.168.103.205: ip-proto-50, length 100

3: 11:27:14.273469 192.168.110.120 > 192.168.103.205: ip-proto-50, length 100

4: 11:27:15.273453 192.168.110.120 > 192.168.103.205: ip-proto-50, length 100

Capture from 5505:

1: 10:51:58.726738 802.1Q vlan#2 P0 192.168.103.205 > 192.168.110.120: ip-proto-50, length 100

2: 10:51:59.726570 802.1Q vlan#2 P0 192.168.103.205 > 192.168.110.120: ip-proto-50, length 100

3: 10:52:00.726402 802.1Q vlan#2 P0 192.168.103.205 > 192.168.110.120: ip-proto-50, length 100

4: 10:52:01.726250 802.1Q vlan#2 P0 192.168.103.205 > 192.168.110.120: ip-proto-50, length 100

What am I missing?

Collin Clark · ‎02-25-2011

Try adding a route to the other ASA-

On the 5505-

route 192.168.110.0 255.255.255.0 [public ip of 5510]

On the 5510-

route 192.168.103.0 255.255.255.0 [public ip of 5505]

mattkl3com · ‎02-25-2011

The route command actually requires an interface to be specified as well.

I think I had tried this already, but I tried it again, using the outside interface, and still no luck.

show crypto ipsec sa

still shows

#pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0

after running my ping commands.

Collin Clark · ‎02-25-2011

Can you try packet tracer from one either the 5505 or 5510 and post the results?

In case you're not familiar with Packet Tracer, here's a quick training video.

http://www.cisco.com/web/learning/le31/le29/configuring_asa_pix_security_appliances.html

Federico Coto Fajardo · ‎02-25-2011

Since the tunnel is between two ASAs you can add this command on both sides:

management-access inside

Then try to PING between inside IPs on both sides:

ping inside x.x.x.x --> internal IP of the peer ASA

This is the easiest way I know to test if traffic is passing through the tunnel.

Also check that both ASAs are encrypting the traffic:

sh cry ips sa

The above will work assuming the inside IPs on both sides are part of the interesting traffic.

Hope it helps.

Federico.

mattkl3com · ‎02-25-2011

It's not clear to me why I would need to use the management-access inside command at all. From the command reference:

To allow management access to an interface other than the one from which you entered the adaptive
security appliance when using VPN, use the management-access command in global configuration
mode.

I'm am already logged in to the ASA via SSH for management purposes.

I already did the ping inside x.x.x.x before, but I ran it again, with no luck.

From 5505:

Sending 5, 100-byte ICMP Echos to 192.168.110.150, timeout is 2 seconds:
?????
Success rate is 0 percent (0/5)

From 5510:


Sending 5, 100-byte ICMP Echos to 192.168.103.26, timeout is 2 seconds:
?????
Success rate is 0 percent (0/5)

Same result from show crypto ipsec sa. See above.

mattkl3com · ‎02-25-2011

Here are the results from the packet tracer:

From the 5510:

firewall(config)# packet-tracer input inside icmp 192.168.110.120 8 0 192.168.103.$
Phase: 1
Type: FLOW-LOOKUP
Subtype:
Result: ALLOW
Config:
Additional Information:
Found no matching flow, creating a new flow
Phase: 2
Type: ROUTE-LOOKUP
Subtype: input
Result: ALLOW
Config:
Additional Information:
in   192.168.103.0   255.255.255.0   outside
Phase: 3
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group inside_access_in in interface inside
access-list inside_access_in extended permit ip any any
Additional Information:
Phase: 4
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Phase: 5
Type: INSPECT
Subtype: np-inspect
Result: ALLOW
Config:
class-map inspection_default
 match default-inspection-traffic
policy-map global_policy
 class inspection_default
  inspect icmp
service-policy global_policy global
Additional Information:
Phase: 6
Type: INSPECT
Subtype: np-inspect
Result: ALLOW
Config:
Additional Information:
Phase: 7
Type: NAT-EXEMPT
Subtype:
Result: ALLOW
Config:
  match ip inside 192.168.110.0 255.255.255.0 outside 192.168.103.0 255.255.255.0
    NAT exempt
    translate_hits = 1568, untranslate_hits = 0
Additional Information:
Phase: 8
Type: NAT
Subtype: host-limits
Result: ALLOW
Config:
static (inside,outside) tcp  https 192.168.110.120 https netmask 255.255.255.255
  match tcp inside host 192.168.110.120 eq 443 outside any
    static translation to /443
    translate_hits = 0, untranslate_hits = 12
Additional Information:
Phase: 9
Type: NAT
Subtype:
Result: ALLOW
Config:
nat (inside) 1 192.168.110.0 255.255.255.0 dns
  match ip inside 192.168.110.0 255.255.255.0 outside any
    dynamic translation to pool 1 ( [Interface PAT])
    translate_hits = 207560, untranslate_hits = 18
Additional Information:
Phase: 10
Type: VPN
Subtype: encrypt
Result: ALLOW
Config:
Additional Information:
Phase: 11
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group outside_access_out out interface outside
access-list outside_access_out extended permit ip any any
Additional Information:
Phase: 12
Type: VPN
Subtype: ipsec-tunnel-flow
Result: ALLOW
Config:
Additional Information:
Phase: 13
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Phase: 14
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 332724, packet dispatched to next module
Result:
input-interface: inside
input-status: up
input-line-status: up
output-interface: outside
output-status: up
output-line-status: up
Action: allow

From the 5505:

ciscoasa(config)# packet-tracer input inside icmp 192.168.103.205 8 0 192.168.$

Phase: 1
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
Additional Information:
MAC Access list

Phase: 2
Type: FLOW-LOOKUP
Subtype:
Result: ALLOW
Config:
Additional Information:
Found no matching flow, creating a new flow

Phase: 3
Type: ROUTE-LOOKUP
Subtype: input
Result: ALLOW
Config:
Additional Information:
in   192.168.110.0   255.255.255.0   outside

Phase: 4
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group inside_access_in in interface inside
access-list inside_access_in extended permit ip any any
Additional Information:

Phase: 5
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 6
Type: INSPECT
Subtype: np-inspect
Result: ALLOW
Config:
class-map inspection_default
 match default-inspection-traffic
policy-map global_policy
 class inspection_default
  inspect icmp
service-policy global_policy global
Additional Information:

Phase: 7
Type: INSPECT
Subtype: np-inspect
Result: ALLOW
Config:
Additional Information:

Phase: 8
Type: VPN
Subtype: ipsec-tunnel-flow
Result: ALLOW
Config:
Additional Information:

Phase: 9
Type: NAT-EXEMPT
Subtype:
Result: ALLOW
Config:
  match ip inside 192.168.103.0 255.255.255.0 outside 192.168.110.0 255.255.255.0
    NAT exempt
    translate_hits = 170, untranslate_hits = 0
Additional Information:

Phase: 10
Type: NAT
Subtype: host-limits
Result: ALLOW
Config:
static (inside,outside) tcp  ssh 192.168.103.205 ssh netmask 255.255.255.255
  match tcp inside host 192.168.103.205 eq 22 outside any
    static translation to /22
    translate_hits = 278, untranslate_hits = 49588
Additional Information:

Phase: 11
Type: NAT
Subtype:
Result: ALLOW
Config:
nat (inside) 1 0.0.0.0 0.0.0.0 dns
  match ip inside any outside any
    dynamic translation to pool 1 ( [Interface PAT])
    translate_hits = 19685912, untranslate_hits = 1807976
Additional Information:

Phase: 12
Type: VPN
Subtype: encrypt
Result: ALLOW
Config:
Additional Information:

Phase: 13
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group outside_access_out out interface outside
access-list outside_access_out extended permit ip any any
Additional Information:

Phase: 14
Type: VPN
Subtype: ipsec-tunnel-flow
Result: ALLOW
Config:
Additional Information:

Phase: 15
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 16
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 21584066, packet dispatched to next module

Result:
input-interface: inside
input-status: up
input-line-status: up
output-interface: outside
output-status: up
output-line-status: up
Action: allow

Note that I am connected to the 5510 via SSH from behind the 5505.

I didn't bother to post this originally because I posted the capture of the IPsec from the outside interface already, which I believe showed that the ping was making it from the inside to the outside. Is there something else you're looking for here?

Thanks for the link to the videos. I didn't watch any of them yet. They look to be kind of old, and the one on the packet tracer looks like it was done for ASA version 7.2.

Federico Coto Fajardo · ‎02-25-2011

management-access inside is to allow you to be able to access the ASA through the tunnel (pass traffic through the tunnel).

Being able to connect to the outside IP of the remote ASA does not prove traffic is passing through the tunnel...

being able to connect to the inside IP of the remote ASA proves the traffic is passing through.

Both ASAs seem to be able to encrypt (send) traffic but no packets received.
This means no traffic is passing through the tunnel (arriving at the other side).

The test I wanted with the management-access inside is to PING the inside IP of the peer ASA and check if that ASA decrypts packets.

If you do :
sh run all sysopt
You see
sysopt connection permit-vpn
correct?

If still does not work, is there any change that ESP is being blocked on the path?

Federico.

mattkl3com · ‎02-25-2011

Yes, I have sysopt connection permit-vpn. Here's the full output for the 5505. Settings are the same for the 5510.

no sysopt connection timewait
sysopt connection tcpmss 1380
sysopt connection tcpmss minimum 0
sysopt connection permit-vpn
sysopt connection reclassify-vpn
no sysopt connection preserve-vpn-flows
no sysopt nodnsalias inbound
no sysopt nodnsalias outbound
no sysopt radius ignore-secret
no sysopt noproxyarp inside
no sysopt noproxyarp outside
no sysopt noproxyarp dmz

Good idea on the ESP being possibly blocked. I was assuming that it would be open, but that does fit with what I'm seeing. I'm following up with some of the network administrators on that.

Collin Clark · ‎02-25-2011

I'm going to be offline for a while. Here's a great T/S link on VPN's. If you have time you might want to read over it and see if you're missing anything.

http://www.cisco.com/en/US/products/ps6120/products_tech_note09186a00807e0aca.shtml

mattkl3com · ‎02-28-2011

Here's what I got back from the network administrator:

I can assure you that these protocols are open as well.

We have 4 Site-to-Site VPN tunnels and 53 Remote Access IPSEC/SSL VPN tunnels terminating on an ASA 5520 that is in the same rule set as your ASA 5510. We also have 2 VPN devices in the same rule set that are also running correctly.

So, I'm still stuck for now :-)

Is it correct that I should be seeing the 192.168.x.x IP addresses in the ipsec capture on the outside interface?

The route statements that Collin suggested made some sense, though I wasn't sure if they were necessary. I figured the remote peer setup stuff probably took care of that.

Is there an easy way to verify the actual IP packets coming from the outside interface have the IP addresses of the firewall outside interface for the source and the remote peer as the destination?

Collin Clark · ‎02-28-2011

I setup a L2L a few weeks ago and I was pulling my hair out. I was seeing exactly what you are. I ended up opening a TAC case and the route statements fixed my problem. Are you able to open a TAC case?

Collin Clark · ‎02-25-2011

Your NAT statements do not match on each end. As a test can you;

ASA5505

Remove access-list no-nat extended permit ip any 192.168.103.240 255.255.255.240

ASA5510

Remove access-list no-nat extended permit ip 192.168.110.0 255.255.255.0 192.168.110.0 255.255.255.0

mattkl3com · ‎02-28-2011

I fail to see what relevance these access lists have on my issue. The ICMP traffic I'm using to test does not match either of these. These don't show up in the packet-tracer output either.

(I removed them for a sec anyway, and no change)

mattkl3com · ‎02-28-2011

Figured this out. I had an extra line in the crypto config that needed to be removed from both devices:

crypto ipsec transform-set ESP-3DES-SHA mode transport

I also removed the route statements to see if it would still work, and it is still working after:

clear route

clear crypto ipsec sa

Review of the "Most Common L2L and Remote Access IPsec VPN Troubleshooting Solutions" page sent by Collin prompted me to find this.

Thanks!