Multi SA site-to-site not encrypting packets

[silverdragan]

I have a setup that runs site-to-site IPSEC between the main location and satellite locations. The main location sends multiple subnets (192.168.0.0/24 & 10.1.0.0/16) and the satellite locations send one subnet (usually another rfc1918 address).  The random issue that I am having is that sometimes that firewalls (at either the main or satellite locations) stop encrypting packets for one specific SA only while the second SA works just fine. For example:

MAIN 192.168.0.0/24 <> SATELLITE 172.25.0.0/23 works

MAIN 10.1.0.0/16 <> SATELLITE 172.25.0.0/23 does not work

Here is the tunnel output:

peer address: SATELLITE-SITE-1
    Crypto map tag: att, seq num: 500, local addr: MAIN-SITE

      access-list ACL-VPN-TUNNEL extended permit ip 10.1.0.0 255.255.0.0 172.25.0.0 255.255.254.0
      local ident (addr/mask/prot/port): (10.1.0.0/255.255.0.0/0/0)
      remote ident (addr/mask/prot/port): (172.25.0.0/255.255.254.0/0/0)
      current_peer: SATELLITE-SITE-1


      #pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
      #pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
      #pkts compressed: 0, #pkts decompressed: 0
      #pkts not compressed: 0, #pkts comp failed: 0, #pkts decomp failed: 0
      #pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0
      #PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0
      #TFC rcvd: 0, #TFC sent: 0
      #Valid ICMP Errors rcvd: 0, #Invalid ICMP Errors rcvd: 0
      #send errors: 0, #recv errors: 0

      local crypto endpt.: MAIN-SITE/500, remote crypto endpt.: SATELLITE-SITE-1/500
      path mtu 1500, ipsec overhead 74(44), media mtu 1500
      PMTU time remaining (sec): 0, DF policy: copy-df
      ICMP error validation: disabled, TFC packets: disabled
      current outbound spi: 56B871BE
      current inbound spi : 1DBE2071
              
    inbound esp sas:
      spi: 0x1DBE2071 (498999409)
         transform: esp-aes-192 esp-sha-hmac no compression
         in use settings ={L2L, Tunnel, IKEv2, }
         slot: 0, conn_id: 69685248, crypto-map: att
         sa timing: remaining key lifetime (kB/sec): (4008960/28753)
         IV size: 16 bytes
         replay detection support: Y
         Anti replay bitmap:
          0x00000000 0x00000001
    outbound esp sas:
      spi: 0x56B871BE (1454928318)
         transform: esp-aes-192 esp-sha-hmac no compression
         in use settings ={L2L, Tunnel, IKEv2, }
         slot: 0, conn_id: 69685248, crypto-map: att
         sa timing: remaining key lifetime (kB/sec): (3962880/28753)
         IV size: 16 bytes
         replay detection support: Y
         Anti replay bitmap:
          0x00000000 0x00000001

    Crypto map tag: att, seq num: 500, local addr: MAIN-SITE

      access-list ACL-VPN-TUNNEL extended permit ip 192.168.2.0 255.255.255.0 172.25.0.0 255.255.254.0
      local ident (addr/mask/prot/port): (192.168.2.0/255.255.255.0/0/0)
      remote ident (addr/mask/prot/port): (172.25.0.0/255.255.254.0/0/0)
      current_peer: SATELLITE-SITE-1


      #pkts encaps: 529, #pkts encrypt: 529, #pkts digest: 529
      #pkts decaps: 718, #pkts decrypt: 718, #pkts verify: 718
      #pkts compressed: 0, #pkts decompressed: 0
      #pkts not compressed: 529, #pkts comp failed: 0, #pkts decomp failed: 0
      #pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0
      #PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0
      #TFC rcvd: 0, #TFC sent: 0
      #Valid ICMP Errors rcvd: 0, #Invalid ICMP Errors rcvd: 0
      #send errors: 0, #recv errors: 0

      local crypto endpt.: MAIN-SITE/500, remote crypto endpt.: SATELLITE-SITE-1/500
      path mtu 1500, ipsec overhead 74(44), media mtu 1500
      PMTU time remaining (sec): 0, DF policy: copy-df
      ICMP error validation: disabled, TFC packets: disabled
      current outbound spi: 2AF034A7
      current inbound spi : 5E9259BE

    inbound esp sas:
      spi: 0x5E9259BE (1586649534)
         transform: esp-aes-192 esp-sha-hmac no compression
         in use settings ={L2L, Tunnel, IKEv2, }
         slot: 0, conn_id: 69685248, crypto-map: att
         sa timing: remaining key lifetime (kB/sec): (3962507/28729)
         IV size: 16 bytes
         replay detection support: Y
         Anti replay bitmap:
          0xFFFFFFFF 0xFFFFFFFF
    outbound esp sas:
      spi: 0x2AF034A7 (720385191)
         transform: esp-aes-192 esp-sha-hmac no compression
         in use settings ={L2L, Tunnel, IKEv2, }
         slot: 0, conn_id: 69685248, crypto-map: att
         sa timing: remaining key lifetime (kB/sec): (4193235/28729)
         IV size: 16 bytes
         replay detection support: Y
         Anti replay bitmap:
          0x00000000 0x00000001

Here is a packet-trace output which shows everything checks out:

packet-tracer input inside icmp 10.1.102.100 0 6 172.25.0.144

Phase: 1
Type: ROUTE-LOOKUP
Subtype: input
Result: ALLOW
Config:
Additional Information:
in   172.25.0.0      255.255.254.0   att

Phase: 2
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
nat (any,att) source static MAIN-SITE-to-REMOTE-SITES MAIN-SITE-to-REMOTE-SITES destination static NAT-EXEMPTION-REMOTE-SITES NAT-EXEMPTION-REMOTE-SITES no-proxy-arp route-lookup
Additional Information:
NAT divert to egress interface att
Untranslate 172.25.0.144/0 to 172.25.0.144/0

Phase: 3
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group any-traffic in interface inside
access-list any-traffic extended permit ip any any
Additional Information:

Phase: 4
Type: NAT
Subtype:
Result: ALLOW
Config:
nat (any,att) source static MAIN-SITE-to-REMOTE-SITES MAIN-SITE-to-REMOTE-SITES destination static NAT-EXEMPTION-REMOTE-SITES NAT-EXEMPTION-REMOTE-SITES no-proxy-arp route-lookup
Additional Information:
Static translate 10.1.102.100/0 to 10.1.102.100/0

Phase: 5
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 6
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 7
Type: INSPECT
Subtype: np-inspect
Result: ALLOW
Config:
class-map inspection_default
 match default-inspection-traffic
policy-map global_policy
 class inspection_default
  inspect icmp
service-policy global_policy global
Additional Information:

Phase: 8
Type: INSPECT
Subtype: np-inspect
Result: ALLOW
Config:
Additional Information:

Phase: 9
Type: FLOW-EXPORT
Subtype:      
Result: ALLOW
Config:
Additional Information:

Phase: 10
Type: VPN
Subtype: encrypt
Result: ALLOW
Config:
Additional Information:

Phase: 11
Type: NAT
Subtype: rpf-check
Result: ALLOW
Config:
nat (any,att) source static MAIN-SITE-to-REMOTE-SITES MAIN-SITE-to-REMOTE-SITES destination static NAT-EXEMPTION-REMOTE-SITES NAT-EXEMPTION-REMOTE-SITES no-proxy-arp route-lookup
Additional Information:

Phase: 12
Type: USER-STATISTICS
Subtype: user-statistics
Result: ALLOW
Config:       
Additional Information:

Phase: 13
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 639783064, packet dispatched to next module

Result:
input-interface: inside
input-status: up
input-line-status: up
output-interface: att
output-status: up
output-line-status: up
Action: allow

[Francesco Molino · Accepted solution]

Hi

What's your asa model and version?
Are you sure you don't have any overlapping vpn?
Can you following the 2nd option of this doc and share the complete output into a text file please: https://community.cisco.com/t5/security-documents/asa-how-to-troubleshoot-vpn-l2l-ensure-traffic-is-passing/ta-p/3166082

When this isn't working, what do you do to make it working again?

[silverdragan]

I am running Cisco ASA 5515 / 9.1.7 on the Main site and Sonicwall on the Satellite location, but I've experienced the same problem between Cisco ASA 5515 and Cisco ASA 5505 running either 9.1.7 or 9.1.8.

Output requested:

Looks like the SPI's are not matching. Fixing this issue in the past have involved clearing the tunnels on both ends or shutting down the tunnel on the satellite location (which I can do), but the issue now is that I do not control the other side nor I have the option to take it down for a longer period of time.

Phase: 10
Type: VPN
Subtype: encrypt
Result: ALLOW
Config:
Additional Information:
 Forward Flow based lookup yields rule:
 out id=0x7fff3d329b90, priority=70, domain=encrypt, deny=false
        hits=5208, user_data=0x1793b1fc, cs_id=0x7fff3b0b0f50, reverse, flags=0x0, protocol=0
        src ip/id=10.1.0.0, mask=255.255.0.0, port=0, tag=0
        dst ip/id=172.25.0.0, mask=255.255.254.0, port=0, tag=0, dscp=0x0
        input_ifc=any, output_ifc=att

VPN CTX  = 0x1793B1FC

Peer IP  = 172.25.0.0
Pointer  = 0x3F99D0D0
State    = UP
Flags    = ENCR+ESP
SA       = 0x355DD45B
SPI      = 0x6ED029DF
Group    = 1
Pkts     = 27209
Bad Pkts = 0
Bad SPI  = 0
Spoof    = 0
Bad Crypto = 0
Rekey Pkt  = 1
Rekey Call = 1
VPN Filter = <none>

peer address: REMOTE-SITE-1
    Crypto map tag: att, seq num: 500, local addr: MAIN-SITE

      access-list ACL-VPN-TUNNEL extended permit ip 10.1.0.0 255.255.0.0 172.25.0.0 255.255.254.0
      local ident (addr/mask/prot/port): (10.1.0.0/255.255.0.0/0/0)
      remote ident (addr/mask/prot/port): (172.25.0.0/255.255.254.0/0/0)
      current_peer: REMOTE-SITE-1


      #pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
      #pkts decaps: 17, #pkts decrypt: 17, #pkts verify: 17
      #pkts compressed: 0, #pkts decompressed: 0
      #pkts not compressed: 0, #pkts comp failed: 0, #pkts decomp failed: 0
      #pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0
      #PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0
      #TFC rcvd: 0, #TFC sent: 0
      #Valid ICMP Errors rcvd: 0, #Invalid ICMP Errors rcvd: 0
      #send errors: 0, #recv errors: 0

      local crypto endpt.: MAIN-SITE/500, remote crypto endpt.: REMOTE-SITE-1/500
      path mtu 1500, ipsec overhead 74(44), media mtu 1500
      PMTU time remaining (sec): 0, DF policy: copy-df
      ICMP error validation: disabled, TFC packets: disabled
      current outbound spi: 7A6009ED
      current inbound spi : 9DD7EB01
              
    inbound esp sas:
      spi: 0x9DD7EB01 (2648173313)
         transform: esp-aes-192 esp-sha-hmac no compression
         in use settings ={L2L, Tunnel, IKEv2, }
         slot: 0, conn_id: 69685248, crypto-map: att
         sa timing: remaining key lifetime (kB/sec): (4331519/28482)
         IV size: 16 bytes
         replay detection support: Y
         Anti replay bitmap:
          0x00000000 0x0003FFFF
    outbound esp sas:
      spi: 0x7A6009ED (2053114349)
         transform: esp-aes-192 esp-sha-hmac no compression
         in use settings ={L2L, Tunnel, IKEv2, }
         slot: 0, conn_id: 69685248, crypto-map: att
         sa timing: remaining key lifetime (kB/sec): (4101120/28425)
         IV size: 16 bytes
         replay detection support: Y
         Anti replay bitmap:
          0x00000000 0x00000001

[Francesco Molino]

Ok SPI doesn't match and that's why traffic won't go through.
Do you have any other VPNs in your ASA? Maybe some overlapping?
Can you share output of show cry isak, show cryp ipsec?
You'll need to clear it on both side to make sure traffic will go through.

[silverdragan]

I completely removed the crypto map configuration on my end the following entries remain:

Peer IP     = 172.25.0.0
Pointer     = 0x41C51F90
State     = UP
Flags     = DECR+ESP
SA     = 0x441BE6CF
SPI     = 0x734251A8
Group     = 1
Pkts     = 10
Bad Pkts = 0
Bad SPI     = 0
Spoof     = 0
Bad Crypto = 0
Rekey Pkt  = 1
Rekey Call = 2
VPN Filter = <none>

Peer IP     = 172.25.0.0
Pointer     = 0x3F99D0D0
State     = UP
Flags     = ENCR+ESP
SA     = 0x355DD45B
SPI     = 0x6ED029DF
Group     = 1
Pkts     = 29395
Bad Pkts = 0  
Bad SPI  = 0  
Spoof    = 0  
Bad Crypto = 0
Rekey Pkt  = 1
Rekey Call = 1
VPN Filter = <none>

sh crypto isakmp or sh crypto ipsec don't have any entries for 172.25.0.0/23 nor they are matching any of the SPI's shown above.

[Francesco Molino]

The status is still up.
Can you clear all crypto out of hours and see what happens?
Can you share the output of sh cry isa and ipsec + config crypto (without psk) please?

[silverdragan]

Apparently I am hitting two bugs according to Cisco TAC: CSCvb29688, CSCvd42057 and will need to either reload the device or remove the crypto map configuration.

[Francesco Molino]

Ok thanks for letting me know. 1 has reload asa as workaround. The other has non but some versions don't have this bug. If you have a maintenance window, instead of just reloading the box, upgrade it to be sure it won't happen again.

[silverdragan]

I've upgraded the devices to 9.9.1 and so far (couple of hours) haven't seen any issues.

[Francesco Molino]

👍🏻