Tunnel between ASA5505 and Watchguard only sending traffic one way

[James Dykes]

I have an ASA5505 with a VPN to a customer's Watchguard firewall. Up until today, the VPN tunnel between the two devices was working. Now today, traffic is only getting encrypted from the Watchguard's side.

I can see the traffic from servers on our side hit the inside interface of the ASA from a packet capture, but this side is not encapsulating the packets and sending them along. I'm getting this from the debug logs:

May 16 16:04:37 [IKEv1 DEBUG]: Group = XXX.XXX.171.225, IP = XXX.XXX.171.225, processing hash payload

May 16 16:04:37 [IKEv1 DEBUG]: Group = XXX.XXX.171.225, IP = XXX.XXX.171.225, processing notify payload

May 16 16:04:37 [IKEv1 DEBUG]: Group = XXX.XXX.171.225, IP = XXX.XXX.171.225, Received keep-alive of type DPD R-U-THERE (seq number 0x768b90f)

May 16 16:04:37 [IKEv1 DEBUG]: Group = XXX.XXX.171.225, IP = XXX.XXX.171.225, Sending keep-alive of type DPD R-U-THERE-ACK (seq number 0x768b90f)

May 16 16:04:37 [IKEv1 DEBUG]: Group = XXX.XXX.171.225, IP = XXX.XXX.171.225, constructing blank hash payload

May 16 16:04:37 [IKEv1 DEBUG]: Group = XXX.XXX.171.225, IP = XXX.XXX.171.225, constructing qm hash payload

May 16 16:04:37 [IKEv1]: IP = XXX.XXX.171.225, IKE_DECODE SENDING Message (msgid=9a32f537) with payloads : HDR + HASH (8) + NOTIFY (11) + NONE (0) total length : 84

May 16 16:04:39 [IKEv1]: IP = XXX.XXX.171.225, IKE_DECODE RECEIVED Message (msgid=c4a6844a) with payloads : HDR + HASH (8) + DELETE (12) + NONE (0) total length : 68

May 16 16:04:39 [IKEv1 DEBUG]: Group = XXX.XXX.171.225, IP = XXX.XXX.171.225, processing hash payload

May 16 16:04:39 [IKEv1 DEBUG]: Group = XXX.XXX.171.225, IP = XXX.XXX.171.225, processing delete

May 16 16:04:39 [IKEv1]: Group = XXX.XXX.171.225, IP = XXX.XXX.171.225, Could not find centry for IPSec SA delete with reason message - SPI 0xCFA1A169

May 16 16:04:45 [IKEv1]: IP = XXX.XXX.171.225, IKE_DECODE RECEIVED Message (msgid=9af40d52) with payloads : HDR + HASH (8) + DELETE (12) + NONE (0) total length : 68

May 16 16:04:45 [IKEv1 DEBUG]: Group = XXX.XXX.171.225, IP = XXX.XXX.171.225, processing hash payload

May 16 16:04:45 [IKEv1 DEBUG]: Group = XXX.XXX.171.225, IP = XXX.XXX.171.225, processing delete

May 16 16:04:45 [IKEv1]: Group = XXX.XXX.171.225, IP = XXX.XXX.171.225, Could not find centry for IPSec SA delete with reason message - SPI 0xCFA1A169

May 16 16:04:51 [IKEv1]: IP = XXX.XXX.171.225, IKE_DECODE RECEIVED Message (msgid=a592f52b) with payloads : HDR + HASH (8) + DELETE (12) + NONE (0) total length : 68

May 16 16:04:51 [IKEv1 DEBUG]: Group = XXX.XXX.171.225, IP = XXX.XXX.171.225, processing hash payload

May 16 16:04:51 [IKEv1 DEBUG]: Group = XXX.XXX.171.225, IP = XXX.XXX.171.225, processing delete

May 16 16:04:51 [IKEv1]: Group = XXX.XXX.171.225, IP = XXX.XXX.171.225, Could not find centry for IPSec SA delete with reason message - SPI 0xCFA1A169

May 16 16:04:57 [IKEv1]: IP = XXX.XXX.171.225, IKE_DECODE RECEIVED Message (msgid=f6714889) with payloads : HDR + HASH (8) + DELETE (12) + NONE (0) total length : 68

May 16 16:04:57 [IKEv1 DEBUG]: Group = XXX.XXX.171.225, IP = XXX.XXX.171.225, processing hash payload

May 16 16:04:57 [IKEv1 DEBUG]: Group = XXX.XXX.171.225, IP = XXX.XXX.171.225, processing delete

May 16 16:04:57 [IKEv1]: Group = XXX.XXX.171.225, IP = XXX.XXX.171.225, Could not find centry for IPSec SA delete with reason message - SPI 0xCFA1A169

And the following from my capture:

   1: 15:45:04.400217 802.1Q vlan#1 P0 10.50.10.19 > 192.168.92.110: icmp: echo request

   2: 15:45:04.400461 802.1Q vlan#1 P0 192.168.92.110 > 10.50.10.19: icmp: echo reply

   3: 15:45:08.963222 802.1Q vlan#1 P0 10.50.10.19 > 192.168.92.110: icmp: echo request

   4: 15:45:08.963405 802.1Q vlan#1 P0 192.168.92.110 > 10.50.10.19: icmp: echo reply

   5: 15:45:13.964381 802.1Q vlan#1 P0 10.50.10.19 > 192.168.92.110: icmp: echo request

   6: 15:45:13.964641 802.1Q vlan#1 P0 192.168.92.110 > 10.50.10.19: icmp: echo reply

   7: 15:45:18.969859 802.1Q vlan#1 P0 10.50.10.19 > 192.168.92.110: icmp: echo request

   8: 15:45:18.970103 802.1Q vlan#1 P0 192.168.92.110 > 10.50.10.19: icmp: echo reply

   9: 15:45:29.183843 802.1Q vlan#1 P0 10.50.10.19.137 > 192.168.92.110.137:  udp 50

  10: 15:45:29.184621 802.1Q vlan#1 P0 192.168.92.110.137 > 10.50.10.19.137:  udp 157

  11: 15:45:30.678721 802.1Q vlan#1 P0 10.50.10.19.137 > 192.168.92.110.137:  udp 50

  12: 15:45:30.678981 802.1Q vlan#1 P0 192.168.92.110.137 > 10.50.10.19.137:  udp 157

  13: 15:45:32.178488 802.1Q vlan#1 P0 10.50.10.19.137 > 192.168.92.110.137:  udp 50

  14: 15:45:32.178655 802.1Q vlan#1 P0 192.168.92.110.137 > 10.50.10.19.137:  udp 157

  15: 15:45:39.479681 802.1Q vlan#1 P0 10.50.10.19 > 192.168.92.110: icmp: echo request

  16: 15:45:39.479940 802.1Q vlan#1 P0 192.168.92.110 > 10.50.10.19: icmp: echo reply

  17: 15:45:43.468313 802.1Q vlan#1 P0 10.50.10.19 > 192.168.92.110: icmp: echo request

  18: 15:45:43.468557 802.1Q vlan#1 P0 192.168.92.110 > 10.50.10.19: icmp: echo reply

  19: 15:45:52.726204 802.1Q vlan#1 P0 10.50.10.19 > 192.168.92.109: icmp: echo request

  20: 15:45:52.726433 802.1Q vlan#1 P0 192.168.92.109 > 10.50.10.19: icmp: echo reply

  21: 15:45:57.470358 802.1Q vlan#1 P0 10.50.10.19 > 192.168.92.109: icmp: echo request

  22: 15:45:57.470495 802.1Q vlan#1 P0 192.168.92.109 > 10.50.10.19: icmp: echo reply

  23: 15:46:02.471151 802.1Q vlan#1 P0 10.50.10.19 > 192.168.92.109: icmp: echo request

  24: 15:46:02.471304 802.1Q vlan#1 P0 192.168.92.109 > 10.50.10.19: icmp: echo reply

  25: 15:46:07.472784 802.1Q vlan#1 P0 10.50.10.19 > 192.168.92.109: icmp: echo request

  26: 15:46:07.472967 802.1Q vlan#1 P0 192.168.92.109 > 10.50.10.19: icmp: echo reply

  27: 15:46:24.932294 802.1Q vlan#1 P0 192.168.92.110 > 10.50.10.19: icmp: echo request

  28: 15:46:29.691660 802.1Q vlan#1 P0 192.168.92.110 > 10.50.10.19: icmp: echo request

  29: 15:46:34.683634 802.1Q vlan#1 P0 192.168.92.110 > 10.50.10.19: icmp: echo request

  30: 15:46:39.691401 802.1Q vlan#1 P0 192.168.92.110 > 10.50.10.19: icmp: echo request

  31: 15:47:07.429878 802.1Q vlan#1 P0 192.168.92.110 > 10.50.10.19: icmp: echo request

  32: 15:47:08.692163 802.1Q vlan#1 P0 192.168.92.110 > 10.50.10.19: icmp: echo request

  33: 15:47:10.189672 802.1Q vlan#1 P0 192.168.92.110 > 10.50.10.19: icmp: echo request

  34: 15:47:11.687372 802.1Q vlan#1 P0 192.168.92.110 > 10.50.10.19: icmp: echo request

  35: 15:47:13.185049 802.1Q vlan#1 P0 192.168.92.110 > 10.50.10.19: icmp: echo request

  36: 15:47:14.682597 802.1Q vlan#1 P0 192.168.92.110 > 10.50.10.19: icmp: echo request

  37: 15:47:16.195821 802.1Q vlan#1 P0 192.168.92.110 > 10.50.10.19: icmp: echo request

  38: 15:47:17.693384 802.1Q vlan#1 P0 192.168.92.110 > 10.50.10.19: icmp: echo request

  39: 15:47:19.191259 802.1Q vlan#1 P0 192.168.92.110 > 10.50.10.19: icmp: echo request

  40: 15:47:20.688685 802.1Q vlan#1 P0 192.168.92.110 > 10.50.10.19: icmp: echo request

  41: 15:47:22.186254 802.1Q vlan#1 P0 192.168.92.110 > 10.50.10.19: icmp: echo request

Here's the relevant parts of the config:

object-group network Warmoth_Office

network-object 10.50.10.0 255.255.255.0

access-list outside_20_cryptomap extended permit ip 192.168.92.0 255.255.255.0 object-group Warmoth_Office

access-list inside_nat0_outbound extended permit ip 192.168.92.0 255.255.255.0 object-group Warmoth_Office

nat (inside) 0 access-list inside_nat0_outbound

crypto ipsec transform-set ESP-3DES-SHA esp-3des esp-sha-hmac

crypto ipsec transform-set ESP-AES-128-SHA esp-aes esp-sha-hmac

crypto dynamic-map outside_dyn_map 20 set pfs

crypto dynamic-map outside_dyn_map 20 set transform-set ESP-3DES-SHA

crypto map outside_map 20 match address outside_20_cryptomap

crypto map outside_map 20 set pfs group5

crypto map outside_map 20 set peer XXX.XXX.171.225

crypto map outside_map 20 set transform-set ESP-AES-128-SHA

crypto map outside_map 65535 ipsec-isakmp dynamic outside_dyn_map

crypto map outside_map interface outside

crypto isakmp enable outside

crypto isakmp policy 5

authentication pre-share

encryption aes

hash sha

group 5

lifetime 28800

crypto isakmp policy 10

authentication pre-share

encryption 3des

hash sha

group 2

lifetime 86400

tunnel-group XXX.XXX.171.225 type ipsec-l2l

tunnel-group XXX.XXX.171.225 ipsec-attributes

pre-shared-key *

And a packet-tracer output:

3888-FWL001(config)# packet-tracer input inside icmp 192.168.92.111 0 8 10.50.$

Phase: 1

Type: CAPTURE

Subtype:

Result: ALLOW

Config:

Additional Information:

MAC Access list

Phase: 2

Type: ACCESS-LIST

Subtype:

Result: ALLOW

Config:

Implicit Rule

Additional Information:

MAC Access list

Phase: 3

Type: FLOW-LOOKUP

Subtype:

Result: ALLOW

Config:

Additional Information:

Found no matching flow, creating a new flow

Phase: 4

Type: ROUTE-LOOKUP

Subtype: input

Result: ALLOW

Config:

Additional Information:

in   0.0.0.0         0.0.0.0         outside

Phase: 5

Type: IP-OPTIONS

Subtype:

Result: ALLOW

Config:

Additional Information:

Phase: 6

Type: INSPECT

Subtype: np-inspect

Result: ALLOW

Config:

Additional Information:

Phase: 7

Type: CAPTURE

Subtype:

Result: ALLOW

Config:

Additional Information:

Phase: 8

Type: NAT-EXEMPT

Subtype:

Result: ALLOW

Config:

  match ip inside 192.168.92.0 255.255.255.0 outside 10.50.10.0 255.255.255.0

    NAT exempt

    translate_hits = 1141, untranslate_hits = 25861

Additional Information:

Phase: 9

Type: NAT

Subtype:

Result: ALLOW

Config:

static (inside,outside) 216.211.136.84 192.168.92.111 netmask 255.255.255.255

  match ip inside host 192.168.92.111 outside any

    static translation to 216.211.136.84

    translate_hits = 2702284, untranslate_hits = 276218777

Additional Information:

Phase: 10

Type: NAT

Subtype: host-limits

Result: ALLOW

Config:

static (inside,outside) 216.211.136.84 192.168.92.111 netmask 255.255.255.255

  match ip inside host 192.168.92.111 outside any

    static translation to 216.211.136.84

    translate_hits = 2702284, untranslate_hits = 276218795

Additional Information:

Phase: 11

Type: HOST-LIMIT

Subtype:

Result: ALLOW

Config:

Additional Information:

Phase: 12

Type: VPN

Subtype: encrypt

Result: ALLOW

Config:

Additional Information:

Phase: 13

Type: CAPTURE

Subtype:

Result: ALLOW

Config:

Additional Information:

Phase: 14

Type: FLOW-CREATION

Subtype:

Result: ALLOW

Config:

Additional Information:

New flow created with id 287710247, packet dispatched to next module

Result:

input-interface: inside

input-status: up

input-line-status: up

output-interface: outside

output-status: up

output-line-status: up

Action: allow

I don't understand why the traffic from this side is not encapsulating at all.