ASA5516 Site-to-Site VPN not communicating one way

[ringpowersolutions]

We have site-to-site VPNs between 3 sites using ASA Firewalls.  We can ping devices from the first site (Head Office) to the second site (Remote Office) without issues, and the Remote Office can also ping the Head Office.  The Head Office and the Remote Office can ping the third site (DataCenter) without issue.  We are also able to access data across the VPN in both Offices to the DataCenter.  The DataCenter is unable to ping either the Head Office or Remote Office, or initiate any data traffic to the sites.

Previously, we had only the Head Office and DataCenter and they were able to ping and transfer data without issue.  For business requirements, we needed to add the Remote Office and the issues have started since then and are unable to remove the VPN.  The Remote Office is now live as users are able to work, but automated processes triggered from the DataCenter to the Head Office have stopped working.

We have run packet-tracer input [INT] icmp [INT_IP] 8 1 [EXT_IP] detail on all sites and the response is that the traffic is allowed over the VPN on all sites.

We have run clear isakmp sa on the DataCenter side and that has not resolved the issue.

The command show crypto isakmp sa detail run on the DataCenter shows two active tunnels.

The command show crypto ipsec sa shows the correct local address, correct current peer, correct local and remote idents and the ACL that was configured for the traffic.  It does not show any errors with the packets across the VPN.

For troubleshooting purposes, we temporarily set all ACLs to ip permit any any and icmp permit any any on our firewalls to no success.

We verified the policy-map global_policy settings included icmp and that had resolved issues we had with the Remote Office not being able to ping the DataCenter, but did not resolve the issue we have with the DataCenter sending communications through the VPN to the other sites.

The DataCenter currently has a single Internet connection and we only have a static route pointing all unknown traffic to the next hop IP Address for all external traffic.  This was the setup before the Remote Office was added and was working until recently.

Any idea what could be causing this problem, or any commands to run to resolve the issue?

[ringpowersolutions]

@MHM Cisco World used the command and no changes.  show asp table vpn-context detail is attached and I can see the HEX is pointing to the 10.168.0.0 VPN range.  I am unable to see that Hex value in the show ipsec sa detail command.  I have also included the ACL Filter that is currently applied to both VPNs.

Regarding NAT, I have no ability to add no route-lookup to the NAT rules, only route-lookup.  The NAT rules for the VPNs don't have route-lookup enabled, but does have no-proxy-arp applied.  This is true to both the DC and the other offices, where communication between them both is working without issue.

[MHM Cisco World]

user_data=0x2b7c264 <<-

VPN CTX  = 0x02B7C264

Peer IP  = 10.168.0.0 <<- Peer IP is same as remote LAN ???
Pointer  = 0x4D3789B0
State    = UP
Flags    = ENCR+ESP
SA       = 0x2C40EEF5
SPI      = 0xA7C64889 
Group    = 110
Pkts     = 1384786
Bad Pkts = 0
Bad SPI  = 0
Spoof    = 0
Bad Crypto = 0
Rekey Pkt  = 1
Rekey Call = 1
VPN Filter = acl_FW_VPN_Office <<- this CAN make traffic drop in one direction 

pr-fw-1# sh ipsec sa detail
    Crypto map tag: cm_VPN_Hub, seq num: 12, local addr: #.#.#.#

      access-list cracl_VPN_HeadOff extended permit ip 10.169.0.0 255.255.0.0 10.168.0.0 255.255.224.0 <<- remote LAN
      local ident (addr/mask/prot/port): (10.169.0.0/255.255.0.0/0/0)
      remote ident (addr/mask/prot/port): (10.168.0.0/255.255.224.0/0/0)
      current_peer: *.*.*.*


      #pkts encaps: 1418098, #pkts encrypt: 1418098, #pkts digest: 1418098
      #pkts decaps: 1163267, #pkts decrypt: 1163267, #pkts verify: 1163267
      #pkts compressed: 0, #pkts decompressed: 0
      #pkts not compressed: 1418098, #pkts comp failed: 0, #pkts decomp failed: 0
      #pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0
      #PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 4
      #TFC rcvd: 0, #TFC sent: 0
      #Valid ICMP Errors rcvd: 0, #Invalid ICMP Errors rcvd: 0
      #pkts no sa (send): 0, #pkts invalid sa (rcv): 0
      #pkts encaps failed (send): 0, #pkts decaps failed (rcv): 0
      #pkts invalid prot (rcv): 0, #pkts verify failed: 0
      #pkts invalid identity (rcv): 0, #pkts invalid len (rcv): 0
      #pkts invalid pad (rcv): 0,
      #pkts invalid ip version (rcv): 0,
      #pkts replay rollover (send): 0, #pkts replay rollover (rcv): 0
      #pkts replay failed (rcv): 0
      #pkts min mtu frag failed (send): 0, #pkts bad frag offset (rcv): 0
      #pkts internal err (send): 0, #pkts internal err (rcv): 0

      local crypto endpt.: #.#.#.#/0, remote crypto endpt.: *.*.*.*/0
      path mtu 1500, ipsec overhead 74(44), media mtu 1500
      PMTU time remaining (sec): 0, DF policy: copy-df
      ICMP error validation: disabled, TFC packets: disabled
      current outbound spi: A7C64889
      current inbound spi : 8C90A26C

    inbound esp sas:
      spi: 0x8C90A26C (2358289004)
         SA State: active
         transform: esp-aes-256 esp-sha-hmac no compression
         in use settings ={L2L, Tunnel, IKEv1, }
         slot: 0, conn_id: 238034944, crypto-map: cm_VPN_Hub
         sa timing: remaining key lifetime (kB/sec): (4218543/10330)
         IV size: 16 bytes
         replay detection support: Y
         Anti replay bitmap:
          0xFFFFFFFF 0xFFFFFFFF
    outbound esp sas:
      spi: 0xA7C64889 (2814789769)
         SA State: active
         transform: esp-aes-256 esp-sha-hmac no compression
         in use settings ={L2L, Tunnel, IKEv1, }
         slot: 0, conn_id: 238034944, crypto-map: cm_VPN_Hub
         sa timing: remaining key lifetime (kB/sec): (4140821/10291)
         IV size: 16 bytes
         replay detection support: Y
         Anti replay bitmap:
          0x00000000 0x00000001


    

 check my note 
MHM

[ringpowersolutions]

The Peer IP on the working site also shows the remote subnets when using the show asp table vpn-context detail command.  It matches up with the subnet that the VPN traffic should be encrypting for that connection.

The access-list has a permit ip any any and no deny statements before that permit statement.

[MHM Cisco World]

pr-fw-1# sh ipsec sa detail
interface: inet
    Crypto map tag: cm_VPN_Hub, seq num: 12, local addr: #.#.#.#

      access-list cracl_VPN_HeadOff extended permit ip 10.169.0.0 255.255.0.0 10.168.0.0 255.255.224.0

Crypto map tag: cm_VPN_Hub, seq num: 12, local addr: #.#.#.#

      access-list cracl_VPN_HeadOff extended permit ip 10.169.0.0 255.255.0.0 10.167.1.0 255.255.255.0
      local ident 

This same peer have two line in one acl  cracl_VPN_HeadOff

I can not know since peer ip is remove

MHM

[MHM Cisco World]

Phase: 7
Type: VPN
Subtype: encrypt
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
out id=0x7f0b4d77d5f0, priority=70, domain=encrypt, deny=false
hits=14500, user_data=0x2b7c264, cs_id=0x7f0bafc380d0, reverse, flags=0x0, protocol=0
src ip/id=#.#.#.#, mask=255.255.0.0, port=0, tag=any
dst ip/id=#.#.#.#, mask=255.255.224.0, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=inet

Phase: 8
Type: ACCESS-LIST
Subtype: filter-aaa
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
out id=0x7f0b4d382ee0, priority=13, domain=filter-aaa, deny=false
hits=187, user_data=0x7f0ba35a8700, filter_id=0xb(acl_FW_VPN_RemOff), protocol=0
src ip=#.#.#.#, mask=255.255.255.0, port=0
dst ip=#.#.#.#, mask=255.255.255.0, port=0

the filter is use after decrypt not after encrypt 
so in your Packet-tracer 

packet-tracer input lan2 tcp LOCAL-LAN 12345  REMOTE-LAN https detail <<- can you do this again and share result 

MHM

[ringpowersolutions]

Please find attached packet tracer log.

[MHM Cisco World]

Can check photo I share 

Can you check

Show route remote lan longest 

Are it point to correct egress interface?

MHM

[ringpowersolutions]

I was unable to run the command as posted as "longest" was not recognised by the ASAs.  Below is the closest command I could run.

DC# sh route 10.168.2.0 longer-prefixes

Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2, V - VPN
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, + - replicated route
Gateway of last resort is #.#.#.# to network 0.0.0.0

C 10.169.0.0 255.255.255.0 is directly connected, mgmt
L 10.169.0.1 255.255.255.255 is directly connected, mgmt
C 10.169.1.32 255.255.255.240 is directly connected, if_mgmt
L 10.169.1.33 255.255.255.255 is directly connected, if_mgmt
C 10.169.2.0 255.255.255.0 is directly connected, lan2
L 10.169.2.1 255.255.255.255 is directly connected, lan2
C 10.169.6.0 255.255.255.0 is directly connected, servers
L 10.169.6.1 255.255.255.255 is directly connected, servers
C 10.169.60.0 255.255.255.0 is directly connected, cng
L 10.169.60.1 255.255.255.255 is directly connected, cng
C 10.169.254.0 255.255.255.0 is directly connected, if_drac
L 10.169.254.1 255.255.255.255 is directly connected, if_drac

The DC only has a single IP that terminates our traffic, so anything not local to the DC is pushed out of the Default Gateway.  We don't have a specific route for traffic to the remote offices as all VPN traffic flows out of the Default Gateway.  This is a setup that worked previously.  We can ping the Office public IPs from the DC ASA and vice versa.

[MHM Cisco World]

Can you add route for remote LAN point to WAN interface 

MHM

[ringpowersolutions]

I have done this now and still no change.

[MHM Cisco World]

DC# sh route 10.168.0.0 longer-prefixes <<- this your remote LAN do this command again share here 

Thanks 

MHM

[ringpowersolutions]

DC# show route 10.168.0.0 longer-prefixes

Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2, V - VPN
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, + - replicated route
Gateway of last resort is #.#.#.# to network 0.0.0.0

S 10.168.0.0 255.255.224.0 [1/0] via #.#.#.#, inet
C 10.169.0.0 255.255.255.0 is directly connected, mgmt
L 10.169.0.1 255.255.255.255 is directly connected, mgmt
C 10.169.1.32 255.255.255.240 is directly connected, if_mgmt
L 10.169.1.33 255.255.255.255 is directly connected, if_mgmt
C 10.169.2.0 255.255.255.0 is directly connected, lan2
L 10.169.2.1 255.255.255.255 is directly connected, lan2
C 10.169.6.0 255.255.255.0 is directly connected, servers
L 10.169.6.1 255.255.255.255 is directly connected, servers
C 10.169.60.0 255.255.255.0 is directly connected, cng
L 10.169.60.1 255.255.255.255 is directly connected, cng
C 10.169.254.0 255.255.255.0 is directly connected, if_drac
L 10.169.254.1 255.255.255.255 is directly connected, if_drac

[MHM Cisco World]

asa# clear ipsec sa peer <remote peer ip> <<- can you clear the crypto for this peer only 
MHM

[ringpowersolutions]

I have cleared this for the specific peer with no change.  I then removed the VPN access list completely to ensure that wasn't causing an issue and cleared the ipsec sa for that peer again, with no change.  I ran this command and clear isakmp sa on both sides and have no change with how the VPNs are operating.  I am now noticing more entries when I run show asp table vpn-context detail that there are multiple entries for the Head Office connection.  When I run show ipsec sa, I can see that the encrypt/decrypt are pointing to one entry that no long shows an ACL filter.  When I run a packet-tracer, I can see the user_data matches the information for the current ipsec tunnel to the Head Office.

[ringpowersolutions · Accepted solution]

We have found the issue.  Problem was the Offices had two NAT statements that were conflicting.  Below is the commands used to troubleshoot this.

capture asp type asp-drop all
show capture asp | include 10.169.2.69
   8: 13:24:43.496235       802.1Q vlan#13 P0 10.169.2.69.52411 > 10.168.8.27.161:  udp 78 Drop-reason: <b>(no-adjacency) No valid adjacency</b>

The "(no-adjacency) No valid adjacency" from the capture of asp-drop traffic seems to be a routing issue from what we found online.  We checked the Routing table and everything looked fine.  We checked the NAT entries and found the following statements:

nat (if_mgmt,any) source static OBJ-10.168.0.0-16 OBJ-10.168.0.0-16 destination static OBJ-10.169.0.0-16 OBJ-10.169.0.0-16 no-proxy-arp
nat (if_swr,any) source static OBJ-10.168.0.0-16 OBJ-10.168.0.0-16 destination static OBJ-10.169.0.0-16 OBJ-10.169.0.0-16 no-proxy-arp

We replaced the NAT statement for if_mgmt as shown below.  This then resolved the issue immediately.

object-group network OBJ-10.168.1.32-20
network-object 10.168.1.32 255.255.240.0

no nat (if_mgmt,any) source static OBJ-10.168.0.0-16 OBJ-10.168.0.0-16 destination static OBJ-10.169.0.0-16 OBJ-10.169.0.0-16 no-proxy-arp
nat (if_mgmt,any) source static OBJ-10.168.1.32-20 OBJ-10.168.1.32-20 destination static OBJ-10.169.0.0-16 OBJ-10.169.0.0-16 no-proxy-arp

Thank you @MHM Cisco World and @Rob Ingram for your help with this issue.