ASA to ASA overnight ping stops. Bouncing OUTSIDE-IN ACL restarts it.

[clyde.a.huffman.ctr@mail.mil]

I set my Linux server to ping the remote PC (over an IPSEC L2L tunnel) every 600 seconds overnight.  When I return in the morning the ping is stalled and will not restart.  There is no log entry for denial in the logs.  Bouncing the remote INSIDE-IN ACL restarts the ping.  Note that the ACL is applied to outside control-plane.  Also note that the remote PC is powered down over night....  I can't find any reason that the ping doesn't pick up after the remote PC is powered back on.  Any ideas?  Thanks :-)

LOCAL ASA 5520 (1.1.1.92) # debug icmp trace

ICMP echo request from inside:192.168.168.140 to outside:192.168.180.3 ID=17620 seq=715 len=56
ICMP echo request from inside:192.168.168.140 to outside:192.168.180.3 ID=17620 seq=716 len=56
ICMP echo request from inside:192.168.168.140 to outside:192.168.180.3 ID=17620 seq=717 len=56
ICMP echo request from inside:192.168.168.140 to outside:192.168.180.3 ID=17620 seq=718 len=56
ICMP echo request from inside:192.168.168.140 to outside:192.168.180.3 ID=17620 seq=719 len=56
ICMP echo request from inside:192.168.168.140 to outside:192.168.180.3 ID=17620 seq=720 len=56

---------------------------------> FIX ACTION <----------------------------------------
-------> bounce REMOTE ASA 5505 OUTSIDE-IN
-------> access-list OUTSIDE-IN extended permit ip any any log warnings
-------> access-list OUTSIDE-IN extended permit ip any any log warnings inactive
---------------------------------> PING RESTARTS <----------------------------------------

ICMP echo reply from outside:192.168.180.3 to inside:192.168.168.140 ID=17620 seq=720 len=56
ICMP echo request from inside:192.168.168.140 to outside:192.168.180.3 ID=17620 seq=721 len=56
ICMP echo reply from outside:192.168.180.3 to inside:192.168.168.140 ID=17620 seq=721 len=56
ICMP echo request from inside:192.168.168.140 to outside:192.168.180.3 ID=17620 seq=722 len=56
ICMP echo reply from outside:192.168.180.3 to inside:192.168.168.140 ID=17620 seq=722 len=56
ICMP echo request from inside:192.168.168.140 to outside:192.168.180.3 ID=17620 seq=723 len=56
ICMP echo reply from outside:192.168.180.3 to inside:192.168.168.140 ID=17620 seq=723 len=56
ICMP echo request from inside:192.168.168.140 to outside:192.168.180.3 ID=17620 seq=724 len=56
ICMP echo reply from outside:192.168.180.3 to inside:192.168.168.140 ID=17620 seq=724 len=56

REMOTE ASA 5505 (2.2.2.2)

access-list OUTSIDE-IN extended permit ip any any log warnings inactive
access-list OUTSIDE-IN extended permit tcp 1.1.1.88 255.255.255.252 host 2.2.2.2 eq https log warnings
access-list OUTSIDE-IN extended permit tcp 2.2.2.0 255.255.255.252 host 2.2.2.2 eq https log warnings
access-list OUTSIDE-IN extended permit ip host 3.3.3.230 any log
access-list OUTSIDE-IN extended permit ip 192.168.180.0 255.255.255.0 any log
access-list OUTSIDE-IN extended permit ip 192.168.168.0 255.255.255.0 any log
access-list OUTSIDE-IN extended permit icmp 192.168.168.0 255.255.255.0 any log
access-list OUTSIDE-IN extended permit icmp 192.168.180.0 255.255.255.0 any log
access-list OUTSIDE-IN extended deny ip any any log interval 600

access-group OUTSIDE-IN in interface outside control-plane

my ISP (3.3.3.230)

icmp permit any inside
icmp permit any echo inside
icmp permit any echo-reply inside
icmp permit 2.2.2.0 255.255.255.252 outside
icmp permit host 3.3.3.230 outside
icmp permit 1.1.1.88 255.255.255.248 outside

asa180(config)#
asa180(config)# sh cry is sa

IKEv1 SAs:

Active SA: 1
Rekey SA: 0 (A tunnel will report 1 Active and 1 Rekey SA during rekey)
Total IKE SA: 1

1 IKE Peer: 1.1.1.92
Type : L2L Role : responder
Rekey : no State : MM_ACTIVE

There are no IKEv2 SAs
asa180(config)#

asa180(config)# sh cry ipsec sa
interface: outside
Crypto map tag: CMAP, seq num: 20, local addr: 2.2.2.2

access-list LAN_Traffic_200 extended permit ip 192.168.180.0 255.255.255.0 192.168.168.0 255.255.255.0
local ident (addr/mask/prot/port): (192.168.180.0/255.255.255.0/0/0)
remote ident (addr/mask/prot/port): (192.168.168.0/255.255.255.0/0/0)
current_peer: 1.1.1.92

#pkts encaps: 2562, #pkts encrypt: 2562, #pkts digest: 2562
#pkts decaps: 2200, #pkts decrypt: 2200, #pkts verify: 2200
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 2562, #pkts comp failed: 0, #pkts decomp failed: 0
#pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0
#PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0
#TFC rcvd: 0, #TFC sent: 0
#Valid ICMP Errors rcvd: 0, #Invalid ICMP Errors rcvd: 0
#send errors: 0, #recv errors: 0

local crypto endpt.: 2.2.2.2/0, remote crypto endpt.: 1.1.1.92/0
path mtu 1500, ipsec overhead 74(44), media mtu 1500
PMTU time remaining (sec): 0, DF policy: copy-df
ICMP error validation: disabled, TFC packets: disabled
current outbound spi: EABA0F61
current inbound spi : D7335959

inbound esp sas:
spi: 0xD7335959 (3610466649)
transform: esp-aes-256 esp-sha-hmac no compression
in use settings ={L2L, Tunnel, PFS Group 2, IKEv1, }
slot: 0, conn_id: 1859584, crypto-map: CMAP
sa timing: remaining key lifetime (kB/sec): (4373833/23554)
IV size: 16 bytes
replay detection support: Y
Anti replay bitmap:
0xFFFFFFFF 0xFFFFFFFF
outbound esp sas:
spi: 0xEABA0F61 (3938062177)
transform: esp-aes-256 esp-sha-hmac no compression
in use settings ={L2L, Tunnel, PFS Group 2, IKEv1, }
slot: 0, conn_id: 1859584, crypto-map: CMAP
sa timing: remaining key lifetime (kB/sec): (4373810/23554)
IV size: 16 bytes
replay detection support: Y
Anti replay bitmap:
0x00000000 0x00000001

asa180(config)#
asa180(config)#
asa180(config)#

[clyde.a.huffman.ctr@mail.mil]

I think that it has to do with the total number of pings entered.  I stopped a running ping that was getting replies (in the middle of the day) and it would not restart until I bounced the remote ASA OUTSIDE-IN ACL.... If I stick to a few pings at a time, it keeps working.  Is there a threashold for pings (from a particular source)?

LOOK AT THIS: a threshold on icmp?

asa180(config)# show run all threat-detection
threat-detection rate icmp-drop rate-interval 600 average-rate 100 burst-rate 400
threat-detection rate icmp-drop rate-interval 3600 average-rate 80 burst-rate 320

[clyde.a.huffman.ctr@mail.mil]

It's worse then I thought, it is stopping all traffic on the IPSEC tunnel, not just icmp.  :-(  Why is the 5505 doing this?  I'm suspecting that it is because it is applied to the outside control-plane...

[Georg Pauwen · Accepted solution]

Hello,

do you have "inspect icmp” configured in the global_policy map ? If not, you might want to get rid of the ICMP access list altogether and just use the global policy.

If you do have to use the access list, change it to the one below:

access-list OUTSIDE-IN extended permit icmp 192.168.168.0 255.255.255.0 any echo log

access-list OUTSIDE-IN extended permit icmp 192.168.180.0 255.255.255.0 any echo log

access-list OUTSIDE-IN extended permit icmp 192.168.168.0 255.255.255.0 any time-exceeded log
access-list OUTSIDE-IN extended permit icmp 192.168.180.0 255.255.255.0 any time-exceeded log

access-list OUTSIDE-IN extended permit icmp 192.168.168.0 255.255.255.0 any timestamp-reply log
access-list OUTSIDE-IN extended permit icmp 192.168.180.0 255.255.255.0 any timestamp-reply  log

access-list OUTSIDE-IN extended permit icmp 192.168.168.0 255.255.255.0 any unreachable log
access-list OUTSIDE-IN extended permit icmp 192.168.180.0 255.255.255.0 any unreachable  log

[clyde.a.huffman.ctr@mail.mil]

Hi Georg, thanks for your help, good to hear from you again. I've locked down icmp on OUTSIDE-IN and removed the control-plane. The tunnel stays up but the ACL ip is open (no deny any any at the end):-(.  Is it safe enough that NAT keeps internal IP addresses hidden from direct attacks.

Actually I don't understand how the NAT to the internet was working with the following configuration. How did an internet session get back in - wouldn't "deny ip any any" stop it? If I understand this I should be able to lock down the OUTSIDE-IN.

Office Network - Primary WAN 3.3.3.230 - Backup WAN 1.1.1.92 - LAN 192.168.168.0
Remote Employee - WAN 2.2.2.2 - LAN 192.168.180.0
Tunnel 1.1.1.92 to 2.2.2.2

Remote Site

access-list OUTSIDE-IN extended permit ip host 1.1.1.92 any log
access-list OUTSIDE-IN extended permit ip host 2.2.2.2 any log
access-list OUTSIDE-IN extended permit ip host 3.3.3.230 any log
access-list OUTSIDE-IN extended permit ip 192.168.180.0 255.255.255.0 any log
access-list OUTSIDE-IN extended permit ip 192.168.168.0 255.255.255.0 any log
access-list OUTSIDE-IN extended permit icmp 192.168.180.0 255.255.255.0 any log
access-list OUTSIDE-IN extended permit icmp 192.168.168.0 255.255.255.0 any log
access-list OUTSIDE-IN extended deny ip any any log interval 600

access-group OUTSIDE-IN in interface outside control-plane

nat (inside,outside) source static 192.168.180.0_24 192.168.180.0_24 destination static 192.168.168.0_24 192.168.168.0_24 no-proxy-arp route-lookup
nat (inside,outside) after-auto source dynamic any interface

route outside 0.0.0.0 0.0.0.0 2.2.2.1 5
route outside 192.168.168.0 255.255.255.0 1.1.1.92 1

tunnel-group 1.1.1.92 type ipsec-l2l
tunnel-group 1.1.1.92 ipsec-attributes

I FOUND THIS:

By default all traffic from higher security zone such as “inside” going to lower security zone “outside” is allowed without the need of an ACL. Return traffic is allowed while the traffic was initiated from “inside”. This is only true for stateful TCP traffic. You’ll find yourself not being able to ping from an internal host to the outside world without implementing one of the options below.

[clyde.a.huffman.ctr@mail.mil]

Just got inspect icmp working :-)  Thanks :0)  Funny that this policy is applied to the outside....

access-list ICMP_INSPECT extended permit icmp 192.168.168.055.255.255.0 192.168.180.0 255.255.255.0
class-map ICMP_CLASS
match access-list ICMP_INSPECT
exit
policy-map MY_POLICY
class ICMP_CLASS
inspect icmp
exit
service-policy MY_POLICY interface outside
end

# no icmp permit 192.168.168.0 255.255.255.0 outside

# deb icmp tr
debug icmp trace enabled at level 1
# ICMP echo request from outside:192.168.168.140 to inside:192.168.180.3 ID=32495 seq=1 len=56
ICMP echo reply from inside:192.168.180.3 to outside:192.168.168.140 ID=32495 seq=1 len=56
ICMP echo request from outside:192.168.168.140 to inside:192.168.180.3 ID=32495 seq=2 len=56
ICMP echo reply from inside:192.168.180.3 to outside:192.168.168.140 ID=32495 seq=2 len=56
ICMP echo request from outside:192.168.168.140 to inside:192.168.180.3 ID=32495 seq=3 len=56

[Georg Pauwen]

Hello,

I haven't followed exactly what you have (re)configured, but is the PING still stopping overnight (as this was the original problem) ?

[clyde.a.huffman.ctr@mail.mil]

Hi Georg,

It was very strange.  The following configuration caused the tunnel to stop passing traffic even though the tunnel was up.  To restart traffic I removed the "inactive" from the "permit ip any any" and immediately put it back on.  The tunnel stopped even when I was not pinging - so it was not an icmp issue.  I have since removed the "control-plane" from the ""access-group OUTSIDE-IN in interface outside" and removed the "deny ip any any" at the end of the OUTSIDE-IN ACL.  Now everything works but I'd feel better if I could add a "deny any any" statement :-(

access-list OUTSIDE-IN extended permit ip any any log warnings inactive
access-list OUTSIDE-IN extended permit tcp 1.1.1.88 255.255.255.252 host 2.2.2.2 eq https log warnings
access-list OUTSIDE-IN extended permit tcp 2.2.2.0 255.255.255.252 host 2.2.2.2 eq https log warnings
access-list OUTSIDE-IN extended permit ip host 3.3.3.230 any log
access-list OUTSIDE-IN extended permit ip 192.168.180.0 255.255.255.0 any log
access-list OUTSIDE-IN extended permit ip 192.168.168.0 255.255.255.0 any log
access-list OUTSIDE-IN extended permit icmp 192.168.168.0 255.255.255.0 any log
access-list OUTSIDE-IN extended permit icmp 192.168.180.0 255.255.255.0 any log
access-list OUTSIDE-IN extended deny ip any any log interval 600

access-group OUTSIDE-IN in interface outside control-plane