Problems with VPN-SITE-TO-SITE interesting traffic.

[Rafael Jimenez]

There is an existing network, The HQ and two BR. Each BR is connected to the HQ via site-to-site vpn and are working fine.

I had to add a third BR. Currently the vpn tunnel is up, but I’m unable to get traffic pass through the tunnel.

I need your help guys, I´m out of ideas.

[Jon Marshall]

We need a lot more information.

Perhaps post relevant configurations ?

Jon

[Rafael Jimenez]

attached the relevant config for HQ and new BR. 

public ips changed.

[Rafael Jimenez]

the ACL in HQ related with the new BR site.

HQ Access-list

access-list outside2_access_in extended permit ip object-group Remote-networks object-group internal-networks
access-list outside2_access_in extended permit ip object-group Remote-networks object-group Remote-networks
access-list outside2_access_in extended permit ip object-group VPNPOOLS object-group DM_INLINE_NETWORK_2
access-group outside_access_in in interface outside
access-group outside2_access_in in interface outside2

ASA Version 9.1(6)4  in both appliances, 5510 and 5505.

[Jon Marshall]

You read my mind :)

By default IPSEC traffic should be allowed through without an acl although it depends on what you set on the ASA.

Can you answer other points in my last post.

Jon

[Rafael Jimenez]

Thanks Jon, I appreciate your help with this issue.

[Jon Marshall]

Can you run a packet tracer from the BR site to an IP at the HQ site and post output.

Jon

[Rafael Jimenez]

unfortunately I dont have access to the BR right now.
I´m posting an packet tracer that I did some days before.
may be help for the moment, I always receive a drop in the encryp phase.

fw1# packet-tracer input outside tcp 10.1.2.254 5060 10.15.15.1 5060 de$

Phase: 1
Type: ROUTE-LOOKUP
Subtype: input
Result: ALLOW
Config:
Additional Information:
in 10.15.15.1 255.255.255.255 inside

Phase: 2
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
nat (inside,outside) source static HH-BAQ HH-BAQ destination static HH-VPN_BAQ HH-VPN_BAQ no-proxy-arp route-lookup
Additional Information:
NAT divert to egress interface inside
Untranslate 10.15.15.1/5060 to 10.15.15.1/5060

Phase: 3
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group OUTSIDE_IN in interface outside
access-list OUTSIDE_IN extended permit ip object-group HH-VPN_BAQ object-group HH-BAQ
object-group network HH-VPN_BAQ
network-object 10.1.0.0 255.255.0.0
network-object 10.2.0.0 255.255.0.0
network-object 10.3.0.0 255.255.0.0
network-object 10.4.0.0 255.255.0.0
network-object 10.5.0.0 255.255.0.0
object-group network HH-BAQ
network-object 10.9.1.0 255.255.255.0
network-object 10.15.15.1 255.255.255.255
network-object 192.168.0.0 255.255.255.0
network-object 10.10.10.0 255.255.255.0
Additional Information:
Forward Flow based lookup yields rule:
in id=0xcb4b05c0, priority=13, domain=permit, deny=false
hits=0, user_data=0xc96190a0, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
src ip/id=10.1.0.0, mask=255.255.0.0, port=0
dst ip/id=10.15.15.1, mask=255.255.255.255, port=0, dscp=0x0
input_ifc=outside, output_ifc=any

Phase: 4
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0xcb4c1238, priority=0, domain=inspect-ip-options, deny=true
hits=355, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0
input_ifc=outside, output_ifc=any

Phase: 5
Type: INSPECT
Subtype: inspect-sip
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0xcbdca550, priority=70, domain=inspect-sip, deny=false
hits=2, user_data=0xcbdc9730, cs_id=0x0, use_real_addr, flags=0x0, protocol=6
src ip/id=0.0.0.0, mask=0.0.0.0, port=0
dst ip/id=0.0.0.0, mask=0.0.0.0, port=5060, dscp=0x0
input_ifc=outside, output_ifc=any

Phase: 6
Type: VPN
Subtype: ipsec-tunnel-flow
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0xcb3e68b0, priority=13, domain=ipsec-tunnel-flow, deny=true
hits=17, user_data=0x0, cs_id=0x0, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0
input_ifc=outside, output_ifc=any

Phase: 7
Type: HOST-LIMIT
Subtype:
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0xcb49d130, priority=0, domain=host-limit, deny=false
hits=332, user_data=0x0, cs_id=0x0, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0
input_ifc=outside, output_ifc=any

Phase: 8
Type: NAT
Subtype: rpf-check
Result: ALLOW
Config:
nat (inside,outside) source static HH-BAQ HH-BAQ destination static HH-VPN_BAQ HH-VPN_BAQ no-proxy-arp route-lookup
Additional Information:
Forward Flow based lookup yields rule:
out id=0xcb4f1280, priority=6, domain=nat-reverse, deny=false
hits=1, user_data=0xcb4dc918, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
src ip/id=10.1.0.0, mask=255.255.0.0, port=0
dst ip/id=10.15.15.1, mask=255.255.255.255, port=0, dscp=0x0
input_ifc=outside, output_ifc=inside

Phase: 9
Type: USER-STATISTICS
Subtype: user-statistics
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
out id=0xcbdd2268, priority=0, domain=user-statistics, deny=false
hits=325, user_data=0xcbdb1d48, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0
input_ifc=any, output_ifc=inside

Phase: 10
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
in id=0xcb4997d8, priority=0, domain=inspect-ip-options, deny=true
hits=442, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0
input_ifc=inside, output_ifc=any

Phase: 11
Type: VPN
Subtype: encrypt
Result: DROP
Config:
Additional Information:
Reverse Flow based lookup yields rule:
out id=0xcbcb1b18, priority=70, domain=encrypt, deny=false
hits=1, user_data=0x0, cs_id=0xcb43a1c8, reverse, flags=0x0, protocol=0
src ip/id=10.15.15.1, mask=255.255.255.255, port=0
dst ip/id=10.1.0.0, mask=255.255.0.0, port=0, dscp=0x0
input_ifc=any, output_ifc=outside

Result:
input-interface: outside
input-status: up
input-line-status: up
output-interface: inside
output-status: up
output-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule

[Jon Marshall · Accepted solution]

Assuming the tunnel is fully up I suggest running debugging on IPSEC when you get access to the BR and see if you get any error messages that would indicate anything.

Other than that I can't see anything wrong with the configuration you posted unless the HQ ASA is doing some other filtering on the VPN traffic.

Jon

[Rafael Jimenez]

I don´t see any filtering on the vpn traffic.

Whats means the phase 11 and why the src ip and dst ip are swapped.

Phase: 11
Type: VPN
Subtype: encrypt
Result: DROP
Config:
Additional Information:
Reverse Flow based lookup yields rule:
out id=0xcbcb1b18, priority=70, domain=encrypt, deny=false
hits=1, user_data=0x0, cs_id=0xcb43a1c8, reverse, flags=0x0, protocol=0
src ip/id=10.15.15.1, mask=255.255.255.255, port=0
dst ip/id=10.1.0.0, mask=255.255.0.0, port=0, dscp=0x0
input_ifc=any, output_ifc=outside

[Rafael Jimenez]

Hello Jon,

everything looks good, but I don´t know why I can´t access the HQ servers (https://10.1.2.10, or ssh 10.1.2.254).

fw1# sh vpn-sessiondb detail l2l

Session Type: LAN-to-LAN Detailed

Connection : HQ.HQ.HQ.HQ
Index : 1 IP Addr : HQ.HQ.HQ.HQ
Protocol : IKEv1 IPsec
Encryption : IKEv1: (1)AES128 IPsec: (1)AES128
Hashing : IKEv1: (1)SHA1 IPsec: (1)SHA1
Bytes Tx : 917461 Bytes Rx : 3707195
Login Time : 07:21:10 PEST Thu Jan 21 2016
Duration : 1h:51m:42s
IKEv1 Tunnels: 1
IPsec Tunnels: 1

IKEv1:
Tunnel ID : 1.1
UDP Src Port : 500 UDP Dst Port : 500
IKE Neg Mode : Main Auth Mode : preSharedKeys
Encryption : AES128 Hashing : SHA1
Rekey Int (T): 86400 Seconds Rekey Left(T): 79698 Seconds
D/H Group : 5
Filter Name :

IPsec:
Tunnel ID : 1.3
Local Addr : 10.9.0.0/255.255.0.0/0/0
Remote Addr : 10.1.0.0/255.255.0.0/0/0
Encryption : AES128 Hashing : SHA1
Encapsulation: Tunnel
Rekey Int (T): 28800 Seconds Rekey Left(T): 22345 Seconds
Rekey Int (D): 4608000 K-Bytes Rekey Left(D): 4603853 K-Bytes
Idle Time Out: 30 Minutes Idle TO Left : 30 Minutes
Bytes Tx : 940236 Bytes Rx : 4246718
Pkts Tx : 8288 Pkts Rx : 10163

NAC:
Reval Int (T): 0 Seconds Reval Left(T): 0 Seconds
SQ Int (T) : 0 Seconds EoU Age(T) : 6747 Seconds
Hold Left (T): 0 Seconds Posture Token:
Redirect URL :

fw1# sh crypto ipsec sa detail
interface: outside
Crypto map tag: outside_map, seq num: 100, local addr: <MY-PUBLIC-IP>

access-list <BR-cryptomap> extended permit ip 10.9.0.0 255.255.0.0 10.1.0.0 255.255.0.0
local ident (addr/mask/prot/port): (10.9.0.0/255.255.0.0/0/0)
remote ident (addr/mask/prot/port): (10.1.0.0/255.255.0.0/0/0)
current_peer: <HQ-PUBLICIP>

#pkts encaps: 11317, #pkts encrypt: 11317, #pkts digest: 11317
#pkts decaps: 13288, #pkts decrypt: 13288, #pkts verify: 13288
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 11317, #pkts comp failed: 0, #pkts decomp failed: 0
#pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0
#PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0
#TFC rcvd: 0, #TFC sent: 0
#Valid ICMP Errors rcvd: 0, #Invalid ICMP Errors rcvd: 0
#pkts no sa (send): 0, #pkts invalid sa (rcv): 0
#pkts encaps failed (send): 0, #pkts decaps failed (rcv): 0
#pkts invalid prot (rcv): 0, #pkts verify failed: 0
#pkts invalid identity (rcv): 0, #pkts invalid len (rcv): 0
#pkts invalid pad (rcv): 0,
#pkts invalid ip version (rcv): 0,
#pkts replay rollover (send): 0, #pkts replay rollover (rcv): 0
#pkts replay failed (rcv): 0
#pkts min mtu frag failed (send): 0, #pkts bad frag offset (rcv): 0
#pkts internal err (send): 0, #pkts internal err (rcv): 0

local crypto endpt.: <MYPUBLICIP>, remote crypto endpt.: <HQ-PUBLICIP>
path mtu 1500, ipsec overhead 74(44), media mtu 1500
PMTU time remaining (sec): 0, DF policy: copy-df
ICMP error validation: disabled, TFC packets: disabled
current outbound spi: C78F1563
current inbound spi : C7DBCE13

inbound esp sas:
spi: 0xC7DBCE13 (3353071123)
transform: esp-aes esp-sha-hmac no compression
in use settings ={L2L, Tunnel, IKEv1, }
slot: 0, conn_id: 4096, crypto-map: outside_map
sa timing: remaining key lifetime (kB/sec): (4367833/20502)
IV size: 16 bytes
replay detection support: Y
Anti replay bitmap:
0xFF7F77FF 0xF9FFFDFF
outbound esp sas:
spi: 0xC78F1563 (3348043107)
transform: esp-aes esp-sha-hmac no compression
in use settings ={L2L, Tunnel, IKEv1, }
slot: 0, conn_id: 4096, crypto-map: outside_map
sa timing: remaining key lifetime (kB/sec): (4372817/20500)
IV size: 16 bytes
replay detection support: Y
Anti replay bitmap:
0x00000000 0x00000001

======

[Jon Marshall]

Apologies for delay in getting back.

It looks good to me and it suggests packets are getting encrypted and decrypted for the tunnel which is confusing based on your packet tracer output.

Are you sure it is not a problem at the HQ end on their ASA or even with the device you are trying to connect to ?

Can you post the same configuration you posted for this BR for a BR that actually works ?

Jon

[Rafael Jimenez]

Thanks John, your help have been enormous because I confirmed the config was ok. I requested some info to the HQ and found the problem is a routing problem. They need setup the routes in all L3 devices that are involved.

[Jon Marshall]

Glad to hear you got it working and thanks for letting me know.

Jon

[Jon Marshall]

Are you trying to send traffic between the new BR and other BRs or just from the BR to HQ ?

What are the source and destination IPs you are testing with ?

I'm assuming if you have any acls applied to other interfaces ie. not the crypto map interface you have allowed the relevant traffic ?

Jon