Two IPSEC tunnels up, but only can ping mgmt interfaces between one

Jesserony · ‎08-26-2022

Hello,

I have an ASA5516 that i want to temporarily run two IPSEC site to site tunnels so we can start setting up a new data center while waiting for our SDWAN equipment. One tunnel to our current data center (CoLo) subnet and DMZ (running an ASA5505), and one to our office in Dover (running an ASA5508X).

Both tunnels initialize and look normal when looking at them in ASDM.

We currently have no equipment behind the new ASA to do ping tests with, so i am using the inside interfaces of the ASAs to test.

For the data center tunnel, i can ping back and forth without issue, even to and from hosts behind the data center ASA.

For the Dover tunnel, when i ping from either ASA to the other, i get no ping response. On either end of the tunnel i see Byte Tx increment when i do the ping tests, but the other side Rx doesnt increment, and vise-versa.

Could someone please take a look at my configs and see if there are any glaring issues?

Thank you in advance for any suggestions!

Jesse

Config for ASA5516 with tunnel to 192.168.170.0 working and 192.168.10.0 not working:

!
interface GigabitEthernet1/2
nameif Tierpoint
security-level 5
ip address !TierpointIP! 255.255.255.248
!
interface GigabitEthernet1/3
nameif inside
security-level 100
ip address 192.168.190.22 255.255.255.0
!
object network Internal_RFC1918-10
subnet 10.0.0.0 255.0.0.0
object network Internal_RFC1918-172.16
subnet 172.16.0.0 255.240.0.0
object network Internal_RFC1918-192.168
subnet 192.168.0.0 255.255.0.0
object network New_CoLo-SW01
host 192.168.190.21
object network Office-CoLo
subnet 192.168.190.0 255.255.255.0
object network OldCoLo-DMZ
subnet 172.20.43.0 255.255.255.0
object network Office-OldCoLo
subnet 192.168.170.0 255.255.255.0
object network Office-dover
subnet 192.168.10.0 255.255.255.0
object network Dover_inside
subnet 192.168.254.56 255.255.255.248
object-group network Internal_RFC1918
network-object object Internal_RFC1918-10
network-object object Internal_RFC1918-192.168
network-object object Internal_RFC1918-172.16
object-group network local-network
network-object object Office-CoLo
object-group network remote-network
network-object object Office-OldCoLo
network-object object OldCoLo-DMZ
object-group network remote-dover
network-object object Office-dover
network-object object Dover_inside
access-list mpls_access_in extended permit ip any any
access-list inside_access_in extended permit ip any any
access-list DMZ_access_in extended permit ip any any
access-list Tierpoint_cryptomap_1 extended permit ip object-group local-network object-group remote-network
access-list Tierpoint_access_in extended permit tcp any object New_CoLo-SW01 eq ssh
access-list Tierpoint_access_in extended permit ip any any
access-list outside_access_in extended permit tcp any object New_CoLo-SW01 eq ssh
access-list outside_access_in extended permit ip any any
access-list Tierpoint_cryptomap_4 extended permit ip object-group local-network object-group remote-dover

nat (any,any) source static Internal_RFC1918 Internal_RFC1918 destination static Internal_RFC1918 Internal_RFC1918 no-proxy-arp
nat (any,outside) source dynamic any interface inactive
nat (any,Tierpoint) source dynamic any interface
nat (inside,Tierpoint) source static local-network local-network destination static remote-dover remote-dover no-proxy-arp route-lookup
nat (inside,Tierpoint) source static local-network local-network destination static remote-network remote-network no-proxy-arp route-lookup
!
object network New_CoLo-SW01
nat (inside,Tierpoint) static interface service tcp ssh 6758

route Tierpoint 0.0.0.0 0.0.0.0 !TierpointGatewayIP! 1
route outside 0.0.0.0 0.0.0.0 !AnotherISPgateway! 2

crypto ipsec ikev1 transform-set ESP-3DES-MD5 esp-3des esp-md5-hmac
crypto ipsec security-association pmtu-aging infinite
crypto map Tierpoint_map1 1 match address Tierpoint_cryptomap_4
crypto map Tierpoint_map1 1 set peer !IP of non-working peer!
crypto map Tierpoint_map1 1 set ikev1 transform-set ESP-3DES-MD5
crypto map Tierpoint_map1 2 match address Tierpoint_cryptomap_1
crypto map Tierpoint_map1 2 set peer !IP of working peer!
crypto map Tierpoint_map1 2 set ikev1 transform-set ESP-3DES-MD5
crypto map Tierpoint_map1 interface Tierpoint
crypto ca trustpool policy
crypto isakmp identity address
crypto ikev1 policy 1
authentication pre-share
encryption 3des
hash md5
group 2
lifetime 86400

management-access inside

group-policy DfltGrpPolicy attributes
vpn-tunnel-protocol ikev1 l2tp-ipsec ssl-clientless
dynamic-access-policy-record DfltAccessPolicy

tunnel-group !IP of working peer! type ipsec-l2l
tunnel-group !IP of working peer! ipsec-attributes
ikev1 pre-shared-key *****
tunnel-group !IP of non-working peer!type ipsec-l2l
tunnel-group !IP of non-working peer!ipsec-attributes
ikev1 pre-shared-key *****
!

------------------------------------------------------

Config of non-working Dover end of tunnel:

!
interface GigabitEthernet1/3
description LINK SW01:G1/0/20
nameif inside
security-level 100
ip address 192.168.254.58 255.255.255.248
!
!
interface GigabitEthernet1/7
nameif Comcast_Test
security-level 2
ip address !ISP IP! 255.255.255.248
!
object network Internal_RFC1918-10
subnet 10.0.0.0 255.0.0.0
object network Internal_RFC1918-172.16
subnet 172.16.0.0 255.24.0.0
object network Internal_RFC1918-192.168
subnet 192.168.0.0 255.255.0.0
object network Dover_Data_Subnet
subnet 192.168.10.0 255.255.255.0
object network New_CoLo_Subnet
subnet 192.168.190.0 255.255.255.0

object-group network remote-network
network-object object CGI-Network
network-object object CGI-DMZ
network-object object DCIS
object-group network Internal_RFC1918
network-object object Internal_RFC1918-10
network-object object Internal_RFC1918-172.16
network-object object Internal_RFC1918-192.168

object-group network VPN_Local
network-object object Dover_Data_Subnet
network-object 192.168.254.56 255.255.255.248
object-group network VPN_Remote
network-object object New_CoLo_Subnet

access-list inside_access_in extended permit ip any any
access-list Comcast_Test_cryptomap extended permit ip object-group VPN_Local object-group VPN_Remote

access-list Comcast_Test_access_in extended permit ip any any

icmp unreachable rate-limit 1 burst-size 1
icmp permit any mpls
icmp permit any echo-reply outside
icmp permit any outside
icmp permit any inside
icmp permit any Comcast_Test

nat (inside,outside) source static Server_RDGWA-Dov Server_RDGWA-Dov-Outside
nat (DMZ,any) source static CGI-Network CGI-Network destination static DMZ-Network DMZ-Network no-proxy-arp route-lookup
nat (inside,any) source static CGI-Network CGI-Network destination static DMZ-Network DMZ-Network no-proxy-arp route-lookup
nat (DMZ,any) source static CGI-DMZ CGI-DMZ destination static DMZ-Network DMZ-Network no-proxy-arp route-lookup
nat (inside,any) source static CGI-DMZ CGI-DMZ destination static DMZ-Network DMZ-Network no-proxy-arp route-lookup
nat (inside,Comcast_Test) source static VPN_Local VPN_Local destination static VPN_Remote VPN_Remote no-proxy-arp route-lookup
!
nat (any,any) after-auto source static Internal_RFC1918 Internal_RFC1918 destination static Internal_RFC1918 Internal_RFC1918 no-proxy-arp
nat (any,any) after-auto source static Internal_RFC1918 Internal_RFC1918 destination static DCIS-Hosts DCIS-Hosts no-proxy-arp
nat (any,outside) after-auto source dynamic any interface

route Comcast_Test 0.0.0.0 0.0.0.0 !ISP gateway IP! 2
route inside 192.168.8.0 255.255.252.0 192.168.254.57 1

crypto ipsec ikev1 transform-set ESP-3DES-MD5 esp-3des esp-md5-hmac
crypto ipsec security-association pmtu-aging infinite
crypto map Comcast_Test_map0 1 match address Comcast_Test_cryptomap
crypto map Comcast_Test_map0 1 set peer !Peer IP!
crypto map Comcast_Test_map0 1 set ikev1 transform-set ESP-3DES-MD5
crypto map Comcast_Test_map0 1 set nat-t-disable
crypto map Comcast_Test_map0 1 set reverse-route
crypto map Comcast_Test_map0 interface Comcast_Test

crypto ikev1 enable Comcast_Test
crypto ikev1 policy 1
authentication pre-share
encryption 3des
hash md5
group 2
lifetime 86400

management-access inside

group-policy DfltGrpPolicy attributes
vpn-tunnel-protocol ikev1 l2tp-ipsec ssl-clientless
dynamic-access-policy-record DfltAccessPolicy

tunnel-group !Peer IP! type ipsec-l2l
tunnel-group !Peer IP! ipsec-attributes
ikev1 pre-shared-key *****
!

Rob Ingram · ‎08-26-2022

@Jesserony As you've configured "mangement-access inside" ping from a device (switch, router) behind the main site's firewall to the dover site ASA's inside interface, rather than from the main site's ASA. If you ping from the ASA the source would be the egress interface, which is not in the crypto ACL?

Jesserony · ‎08-27-2022

Thanks Rob and MHM Cisco World - i simplified the config on both ends, not usign object groups this time, and just using 192.168.10.0/24 to 192.168.190.0/24.

Also, we now have a few actual hosts on the 190.0 subnet, 190.200 and 190.60, which i can ping across the working tunnel to the 192.168.170.0/24 subnet.

But i still cant ping them from Dover. I am not seeing anything in the logs where the pings are being denied. Im seeing on the .190 end (the one with the other working tunnel) that Bytes Tx is increasing as the host behind it gets pinged from Dover, but the bytes Tx stays at 0. While on the Dover end, both Tx and Rx are increasing.

Any other ideas? Here are the relevant config parts.

New data center, with 1 working tunnel:

access-list Tierpoint_cryptomap_1 extended permit ip object-group local-network object-group remote-network

access-list Tierpoint_cryptomap_4 extended permit ip object Office-CoLo object Office-dover

nat (any,any) source static Internal_RFC1918 Internal_RFC1918 destination static Internal_RFC1918 Internal_RFC1918 no-proxy-arp
nat (any,outside) source dynamic any interface inactive
nat (any,Tierpoint) source dynamic any interface
nat (inside,Tierpoint) source static Office-CoLo Office-CoLo destination static Office-dover Office-dover no-proxy-arp route-lookup
nat (inside,Tierpoint) source static local-network local-network destination static remote-network remote-network no-proxy-arp route-lookup
!
object network New_CoLo-SW01
nat (inside,Tierpoint) static interface service tcp ssh 6758

crypto ipsec ikev1 transform-set ESP-3DES-MD5 esp-3des esp-md5-hmac
crypto ipsec security-association pmtu-aging infinite
crypto map Tierpoint_map1 1 match address Tierpoint_cryptomap_4
crypto map Tierpoint_map1 1 set peer xxxx
crypto map Tierpoint_map1 1 set ikev1 transform-set ESP-3DES-MD5
crypto map Tierpoint_map1 2 match address Tierpoint_cryptomap_1
crypto map Tierpoint_map1 2 set peer xxxx
crypto map Tierpoint_map1 2 set ikev1 transform-set ESP-3DES-MD5
crypto map Tierpoint_map1 interface Tierpoint

group-policy DfltGrpPolicy attributes
vpn-tunnel-protocol ikev1 l2tp-ipsec ssl-clientless

tunnel-group xxxx type ipsec-l2l
tunnel-group xxxx ipsec-attributes
ikev1 pre-shared-key *****
tunnel-group xxxx type ipsec-l2l
tunnel-group xxxx ipsec-attributes
ikev1 pre-shared-key *****

--------------------------------------------
Dover, tunnel not working at all

access-list Tierpoint_cryptomap_1 extended permit ip object-group local-network object-group remote-network

access-list Tierpoint_cryptomap_4 extended permit ip object Office-CoLo object Office-dover

nat (any,any) source static Internal_RFC1918 Internal_RFC1918 destination static Internal_RFC1918 Internal_RFC1918 no-proxy-arp
nat (any,outside) source dynamic any interface inactive
nat (any,Tierpoint) source dynamic any interface
nat (inside,Tierpoint) source static Office-CoLo Office-CoLo destination static Office-dover Office-dover no-proxy-arp route-lookup
nat (inside,Tierpoint) source static local-network local-network destination static remote-network remote-network no-proxy-arp route-lookup
!
object network New_CoLo-SW01
nat (inside,Tierpoint) static interface service tcp ssh 6758

crypto ipsec ikev1 transform-set ESP-3DES-MD5 esp-3des esp-md5-hmac
crypto ipsec security-association pmtu-aging infinite
crypto map Tierpoint_map1 1 match address Tierpoint_cryptomap_4
crypto map Tierpoint_map1 1 set peer xxxx
crypto map Tierpoint_map1 1 set ikev1 transform-set ESP-3DES-MD5
crypto map Tierpoint_map1 2 match address Tierpoint_cryptomap_1
crypto map Tierpoint_map1 2 set peer xxxx
crypto map Tierpoint_map1 2 set ikev1 transform-set ESP-3DES-MD5
crypto map Tierpoint_map1 interface Tierpoint

group-policy DfltGrpPolicy attributes
vpn-tunnel-protocol ikev1 l2tp-ipsec ssl-clientless

tunnel-group xxxx type ipsec-l2l
tunnel-group xxxx ipsec-attributes
ikev1 pre-shared-key *****
tunnel-group xxxx type ipsec-l2l
tunnel-group xxxx ipsec-attributes
ikev1 pre-shared-key *****

Rob Ingram · ‎08-27-2022

@Jesserony NAT or routing problem.

Please confirm what IP address/device are you pinging to and from what device are you generating this traffic? - for the non-working tunnel.

Run packet-tracer from the CLI on the ASA from Dover to CoLo and on the ASA from CoLo to Dover to simulate the traffic flow, provide the outputs for review.

Jesserony · ‎08-27-2022

Thanks again Rob, trying to ping from 192.168.10.239 to 192.168.190.200. It is responding to the pings from the .170 subnets tunnel so i dont think it is something with that host.

This is from the Dover side:
packet-tracer input inside tcp 192.168.10.239 80 192.168.190.200 80

Phase: 1
Type: ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
Additional Information:
found next-hop 192.168.190.200 using egress ifc Comcast_Test

Phase: 2
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
nat (inside,Comcast_Test) source static Dover_Data_Subnet Dover_Data_Subnet destination static New_CoLo_Subnet New_CoLo_Subnet no-proxy-arp route-lookup
Additional Information:
NAT divert to egress interface Comcast_Test
Untranslate 192.168.190.200/80 to 192.168.190.200/80

Phase: 3
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group inside_access_in in interface inside
access-list inside_access_in extended permit ip any any
Additional Information:

Phase: 4
Type: NAT
Subtype:
Result: ALLOW
Config:
nat (inside,Comcast_Test) source static Dover_Data_Subnet Dover_Data_Subnet destination static New_CoLo_Subnet New_CoLo_Subnet no-proxy-arp route-lookup
Additional Information:
Static translate 192.168.10.239/80 to 192.168.10.239/80

Phase: 5
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 6
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 7
Type: SFR
Subtype:
Result: ALLOW
Config:
class-map sfr
match access-list sfr_redirect
policy-map global_policy
class sfr
sfr fail-open
service-policy global_policy global
Additional Information:

Phase: 8
Type: FLOW-EXPORT
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 9
Type: VPN
Subtype: encrypt
Result: ALLOW
Config:
Additional Information:

Phase: 10
Type: NAT
Subtype: rpf-check
Result: ALLOW
Config:
nat (inside,Comcast_Test) source static Dover_Data_Subnet Dover_Data_Subnet destination static New_CoLo_Subnet New_CoLo_Subnet no-proxy-arp route-lookup
Additional Information:

Phase: 11
Type: VPN
Subtype: ipsec-tunnel-flow
Result: ALLOW
Config:
Additional Information:

Phase: 12
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 13
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 14
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 10707604, packet dispatched to next module

Result:
input-interface: inside
input-status: up
input-line-status: up
output-interface: Comcast_Test
output-status: up
output-line-status: up
Action: allow

Here is from the new data center:
packet-tracer input inside tcp 192.168.190.200 80 192.168.10.239 80

Phase: 1
Type: ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
Additional Information:
found next-hop 63.254.156.9 using egress ifc Tierpoint

Phase: 2
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
nat (any,any) source static Internal_RFC1918 Internal_RFC1918 destination static Internal_RFC1918 Internal_RFC1918 no-proxy-arp
Additional Information:
NAT divert to egress interface Tierpoint
Untranslate 192.168.10.239/80 to 192.168.10.239/80

Phase: 3
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group inside_access_in in interface inside
access-list inside_access_in extended permit ip any any
Additional Information:

Phase: 4
Type: NAT
Subtype:
Result: ALLOW
Config:
nat (any,any) source static Internal_RFC1918 Internal_RFC1918 destination static Internal_RFC1918 Internal_RFC1918 no-proxy-arp
Additional Information:
Static translate 192.168.190.200/80 to 192.168.190.200/80

Phase: 5
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 6
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 7
Type: VPN
Subtype: encrypt
Result: ALLOW
Config:
Additional Information:

Phase: 8
Type: NAT
Subtype: rpf-check
Result: ALLOW
Config:
nat (any,any) source static Internal_RFC1918 Internal_RFC1918 destination static Internal_RFC1918 Internal_RFC1918 no-proxy-arp
Additional Information:

Phase: 9
Type: VPN
Subtype: ipsec-tunnel-flow
Result: ALLOW
Config:
Additional Information:

Phase: 10
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 11
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 12
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 108471, packet dispatched to next module

Result:
input-interface: inside
input-status: up
input-line-status: up
output-interface: Tierpoint
output-status: up
output-line-status: up
Action: allow

Rob Ingram · ‎08-27-2022

@Jesserony both packet-tracer output confirms this should work.

So when you generate real traffic what are you generating traffic to/from? You previously said "We currently have no equipment behind the new ASA to do ping tests with, so i am using the inside interfaces of the ASAs to test." - and I said egress would come from the ASA's outside interface which would not be part of the encryption domain. When testing generate traffic "through" the ASA not "to" the ASA.

Jesserony · ‎08-27-2022

Hi Rob, we now have some hosts behind the .190 subnet ASA, but we didnt in the original post.

192.168.10.239 is a windows PC and 192.168.190.200 is a Windows server.

Any other ideas on where the problem might lie?

Rob Ingram · ‎08-27-2022

@Jesserony well the packet-tracer output confirmed it should work, is windows firewall turned on these devices blocking communication?

Jesserony · ‎08-28-2022

Thanks Rob. Yes, Windows Firewall was on. I tried disabling it this morning but that didnt help. I ended up routing to the working tunnel and having it carry the Dover traffic. Not ideal but at least we have some access from Dover now, and it also shows that the PC in Dover was able to communicate, just not over THAT tunnel for some reason...

MHM Cisco World · ‎08-28-2022

try ping one more time and share the output of follwoing
show crypto isakmp sa

Jesserony · ‎08-31-2022

Thanks, going to try setting up a new one from the working ASA (.190 subnet) to a test ASA on a test network in Dover, on that same circuit. I'll let you all know our result.

MHM Cisco World · ‎08-31-2022

Waiting your reply.

Jesserony · ‎09-14-2022

Hi MHM, i was able to successfully establish and pass traffic through my test tunnel. While the original tunnel is still up.

So i re-created the production tunnel and am running into the same issue. I ping 192.168.190.200 from Dover and the tunnel establishes, i see bytes rx and tx on the Dover end of the tunnel, but on the other end just Tx but 0 Rx.

As previously requested, here are the results of show crypto isakmp sa:

Dover End:

Result of the command: "show crypto isakmp sa"

IKEv1 SAs:

Active SA: 1
Rekey SA: 0 (A tunnel will report 1 Active and 1 Rekey SA during rekey)
Total IKE SA: 1

1 IKE Peer: xxxxxx
Type : L2L Role : responder
Rekey : no State : MM_ACTIVE

There are no IKEv2 SAs

New Data Center End:

Result of the command: "show crypto isakmp sa"

IKEv1 SAs:

Active SA: 2
Rekey SA: 0 (A tunnel will report 1 Active and 1 Rekey SA during rekey)
Total IKE SA: 2

1 IKE Peer: xxxx
Type : L2L Role : initiator
Rekey : no State : MM_ACTIVE
2 IKE Peer: xxxx
Type : L2L Role : initiator
Rekey : no State : MM_ACTIVE

There are no IKEv2 SAs

Jesserony · ‎09-14-2022

Something else that might be meaningful - when i ping 192.168.190.200 from my Albany subnet, that has no tunnel to that subnet configured at all, it says Reply from 192.168.190.200: Destination net unreachable.

Now if i do the same from a host in Dover, where the tunnel is up but not passing traffic back, it says Reply from 70.43.126.128: Destination net unreachable.

Jesserony · ‎09-15-2022

Did some more digging on that 70.43.126.128, it belongs to Windstream. We are utilizing Windstream SDWAN on the ASA in Dover, but for this VPN we have a separate interface connected directly to a Comcast circuit, completely separate from the SDWAN. On this ASA, the IP in question is being advertised by BGP from the Velos. Also it is advertising 192.168.0.0/16, so i can see why it would get routed this way. That is, unless a more specific route to 192.168.190.0/24 is in place... which should be by the tunnel... and where i can see the traffic bytes increasing on both ends of the tunnel... yet getting Destination net unreachable from 70.43.126.128... i am confused, please help!

B 70.43.126.128 255.255.255.254 [20/0] via 192.168.252.13, 4w6d

B 192.168.0.0 255.255.0.0 [20/1] via 192.168.252.13, 4w6d