Solved: ASA 5508-X NAT/PAT Intermittenly unresponsive

TobyB · ‎02-24-2019

Hello,

I recently installed a new 5508-X running 9.9(2) in our environment. All is well save for one server that we have a static NAT for. At a random point in the day, the NAT seems to break, and no traffic is passed to or from said server. xlate shows the NAT as idle, and the server remains accessible over the LAN and l2l VPN, but no true internet access on the server. I have confirmed with our ISP that the static is functioning properly, and have even migrated to a different static to verify this. My most recent troubleshooting step was to move to PAT for the websever/other services, and the behavior is still present.

I see nothing in the logs when the connection drops. Debugging doesn't show any TCP connections being built when I try to access the server from the outside, or when I try to access the internet from the server when i had the NAT in place. I have other servers/services using NAT that have no issues. I have not been able to nail down a consistent time that the behavior starts, but it seems to be when the server itself goes idle after work hours. The behavior started the day that I installed the firewall. Throwing the old firewall back in place resolves the issue (pre 8.3 if that matters.) I am not sure where to look next to troubleshoot this, but everything points to it being my ASA. Any help is greatly appreciated. Config below.

names

name 172.31.250.1 HS-L3 description HS-3560

!

interface GigabitEthernet1/1

nameif outside

security-level 0

ip address 5.5.5.5 255.255.255.0

!

interface GigabitEthernet1/2

nameif inside

security-level 100

ip address 172.31.250.100 255.255.255.0

!

interface GigabitEthernet1/3

shutdown

no nameif

no security-level

no ip address

!

interface GigabitEthernet1/4

shutdown

no nameif

no security-level

no ip address

!

interface GigabitEthernet1/5

shutdown

no nameif

no security-level

no ip address

!

interface GigabitEthernet1/6

shutdown

no nameif

no security-level

no ip address

!

interface GigabitEthernet1/7

shutdown

no nameif

no security-level

no ip address

!

interface GigabitEthernet1/8

shutdown

no nameif

no security-level

no ip address

!

interface Management1/1

management-only

no nameif

no security-level

ip address 192.168.45.45 255.255.255.0

!

banner motd *--UNAUTHORIZED ACCESS IS STRICTLY PROHIBITED BY LAW--*

ftp mode passive

clock timezone EST -5

clock summer-time EST recurring

dns server-group DefaultDNS

domain-name HSk12.org

same-security-traffic permit inter-interface

object network obj_any

subnet 0.0.0.0 0.0.0.0

object network HS-LDAPS

host 172.31.0.12

object network HS-RDP-55550

host 172.31.0.22

object network HS-HVAC-WWW

host 172.31.2.250

object network HS-HVAC-1911

host 172.31.2.250

object network HS-HVAC-3011

host 172.31.2.250

object network HS-APPS3

host 172.31.0.26

object network HS-IPTVCOMMAND

host 172.31.72.10

object network HS-SMOOTHWALL

host 172.31.250.3

object network PROBLEM-SERVER

host 172.31.0.30

object-group network Airwatch

description Airwatch Whitelist Addresses

network-object host 63.128.77.234

network-object host 209.66.96.114

network-object host 63.128.77.238

network-object host 209.208.228.192

network-object host 209.208.228.198

network-object host 216.235.137.253

object-group network Edlio

description Edlio Whitelist for LDAP

network-object 192.40.145.0 255.255.255.0

object-group network OBJ-LOCAL-SITE

network-object 172.31.0.0 255.255.0.0

object-group network OBJ-REMOTE-SITE

network-object 172.31.0.0 255.255.0.0

access-list HS-Filter-In remark *--Inbound Access List--*

access-list HS-Filter-In remark *--Smoothwall ACL Start--*

access-list HS-Filter-In extended permit tcp any gt 1023 host 172.31.250.3 eq www

access-list HS-Filter-In extended permit tcp any gt 1023 host 172.31.250.3 eq 805

access-list HS-Filter-In extended permit tcp any gt 1023 host 172.31.250.3 eq 800

access-list HS-Filter-In extended permit tcp any gt 1023 host 172.31.250.3 eq 442

access-list HS-Filter-In extended permit tcp any gt 1023 host 172.31.250.3 eq 441

access-list HS-Filter-In extended permit tcp any gt 1023 host 172.31.250.3 eq https

access-list HS-Filter-In remark *--Smoothwall ACL End--*

access-list HS-Filter-In remark *--RDP ALLOW--*

access-list HS-Filter-In extended permit tcp any gt 1023 host 172.31.0.22 eq 55550

access-list HS-Filter-In remark *--eSports ACL Start--*

access-list HS-Filter-In remark *--League ACL Entries--*

access-list HS-Filter-In extended permit tcp any eq 5223 any

access-list HS-Filter-In extended permit tcp host 23.13.146.217 host 172.31.250.100

access-list HS-Filter-In extended permit tcp host 23.79.196.176 eq https host 172.31.250.100

access-list HS-Filter-In remark *--eSports ACL End--*

access-list HS-Filter-In remark *--HVAC ACL Start--*

access-list HS-Filter-In extended permit tcp any gt 1023 host 172.31.2.250

access-list HS-Filter-In remark *--HVAC ACL End--*

access-list HS-Filter-In extended permit tcp any gt 1023 host 172.31.0.30 eq https

access-list HS-Filter-In remark *--PROBLEM SERVER ACL Start--*

access-list HS-Filter-In extended permit tcp any gt 1023 host 172.31.0.30 eq 8443

access-list HS-Filter-In extended permit tcp any gt 1023 host 172.31.0.30 eq 7880

access-list HS-Filter-In extended permit tcp any gt 1023 host 172.31.0.30 eq 5071

access-list HS-Filter-In extended permit tcp any gt 1023 host 172.31.0.30 eq ftp

access-list HS-Filter-In extended permit tcp any gt 1023 host 172.31.0.30 eq ftp-data

access-list HS-Filter-In extended permit tcp any gt 1023 host 172.31.0.30 eq www

access-list HS-Filter-In remark *--Remote Access to PROBLEM SERVER--*

access-list HS-Filter-In extended permit tcp any gt 1023 host 172.31.0.30 eq 55550

access-list HS-Filter-In extended permit tcp host 204.14.14.18 gt 1023 host 172.31.0.30 eq sqlnet

access-list HS-Filter-In extended permit tcp host 204.14.12.2 gt 1023 host 172.31.0.30 eq sqlnet

access-list HS-Filter-In remark *--ASM Connection to PROBLEM SERVER--*

access-list HS-Filter-In extended permit tcp host 189.72.146.214 gt 1023 host 172.31.0.30 eq telnet

access-list HS-Filter-In remark *--PROBLEM SERVER ACL End--*

access-list HS-Filter-In remark *--IPTV ACL Start--*

access-list HS-Filter-In extended permit udp any gt 1023 host 172.31.71.101 eq snmp

access-list HS-Filter-In extended permit tcp any gt 1023 host 172.31.72.10 eq www

access-list HS-Filter-In extended permit tcp any host 172.31.72.10

access-list HS-Filter-In remark *--IPTV ACL End--*

access-list HS-Filter-In remark *--LDAPS ACL Start--*

access-list HS-Filter-In extended permit tcp object-group Edlio gt 1023 host 172.31.0.12 eq ldaps

access-list HS-Filter-In extended permit tcp host 198.154.98.170 gt 1023 host 172.31.0.12 eq ldaps

access-list HS-Filter-In remark *--LDAP ACL End--*

access-list HS-Filter-In extended deny udp any any log warnings

access-list HS-Filter-In extended deny tcp any any log warnings

access-list HS-Filter-In extended deny icmp any any log warnings

access-list HS-Filter-In extended deny gre any any log warnings

access-list HS-Filter-In extended deny esp any any log warnings

access-list HS-Filter-In extended deny ip any any log warnings

access-list VPN extended permit ip 172.31.0.0 255.255.0.0 172.31.0.0 255.255.0.0

access-list VPN extended permit ip object-group OBJ-LOCAL-SITE object-group OBJ-REMOTE-SITE

access-list NONAT extended permit ip 172.31.0.0 255.255.0.0 172.31.0.0 255.255.0.0

pager lines 24

logging enable

logging timestamp

logging buffer-size 60000

logging buffered debugging

logging trap debugging

logging asdm warnings

logging host inside 172.31.0.26

mtu outside 1500

mtu inside 1500

no failover

no monitor-interface service-module

icmp unreachable rate-limit 1 burst-size 1

no asdm history enable

arp timeout 14400

no arp permit-nonconnected

arp rate-limit 16384

nat (inside,outside) source static OBJ-LOCAL-SITE OBJ-LOCAL-SITE destination static OBJ-REMOTE-SITE OBJ-REMOTE-SITE no-proxy-arp route-lookup

!

object network obj_any

nat (inside,outside) dynamic interface

object network HS-LDAPS

nat (inside,outside) static 5.5.6.152 service tcp ldaps ldaps

object network HS-RDP-55550

nat (inside,outside) static 5.5.7.116 service tcp 55550 55550

object network HS-HVAC-WWW

nat (inside,outside) static 5.5.6.152 service tcp www www

object network HS-HVAC-1911

nat (inside,outside) static XXX.XXX.6.152 service tcp 1911 1911

object network HS-HVAC-3011

nat (inside,outside) static 5.5.6.152 service tcp 3011 3011

object network HS-APPS3

nat (inside,outside) static 5.5.7.136

object network HS-IPTVCOMMAND

nat (inside,outside) static 5.5.7.115

object network HS-SMOOTHWALL

nat (inside,outside) static 5.5.7.116

object network PROBLEM-SERVER

nat (inside,outside) static 5.5.7.137

access-group HS-Filter-In in interface outside

route outside 0.0.0.0 0.0.0.0 5.5.6.1 1

route inside 172.31.0.0 255.255.248.0 HS-L3 1

route inside 172.31.16.0 255.255.254.0 HS-L3 1

route inside 172.31.28.0 255.255.252.0 HS-L3 1

route inside 172.31.72.0 255.255.255.0 HS-L3 1

timeout xlate 3:00:00

timeout pat-xlate 0:00:30

timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 sctp 0:02:00 icmp 0:00:02

timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00

timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00

timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute

timeout tcp-proxy-reassembly 0:01:00

timeout floating-conn 0:00:00

timeout conn-holddown 0:00:15

timeout igp stale-route 0:01:10

user-identity default-domain LOCAL

aaa authentication serial console LOCAL

aaa authentication ssh console LOCAL

aaa authorization exec authentication-server

aaa authentication login-history

snmp-server host inside 172.31.0.60 community *****

no snmp-server location

no snmp-server contact

snmp-server community *****

no service sw-reset-button

crypto ipsec ikev1 transform-set VPN esp-aes esp-md5-hmac

crypto ipsec security-association pmtu-aging infinite

crypto map VPN 1 match address VPN

crypto map VPN 1 set pfs

crypto map VPN 1 set peer 6.6.6.6

crypto map VPN 1 set ikev1 transform-set VPN

crypto map VPN 1 set security-association lifetime seconds 28800

crypto map VPN 1 set security-association lifetime kilobytes 4608000

crypto map VPN interface outside

crypto ca trustpoint _SmartCallHome_ServerCA

no validation-usage

crl configure

crypto ca trustpool policy

crypto ca certificate chain _SmartCallHome_ServerCA

certificate ca 0509

308205b7 3082039f a0030201 02020205 09300d06 092a8648 86f70d01 01050500

3045310b 30090603 55040613 02424d31 19301706 0355040a 13105175 6f566164

6973204c 696d6974 6564311b 30190603 55040313 1251756f 56616469 7320526f

6f742043 41203230 1e170d30 36313132 34313832 3730305a 170d3331 31313234

31383233 33335a30 45310b30 09060355 04061302 424d3119 30170603 55040a13

1051756f 56616469 73204c69 6d697465 64311b30 19060355 04031312 51756f56

61646973 20526f6f 74204341 20323082 0222300d 06092a86 4886f70d 01010105

00038202 0f003082 020a0282 0201009a 18ca4b94 0d002daf 03298af0 0f81c8ae

4c19851d 089fab29 4485f32f 81ad321e 9046bfa3 86261a1e fe7e1c18 3a5c9c60

172a3a74 8333307d 615411cb edabe0e6 d2a27ef5 6b6f18b7 0a0b2dfd e93eef0a

c6b310e9 dcc24617 f85dfda4 daff9e49 5a9ce633 e62496f7 3fba5b2b 1c7a35c2

d667feab 66508b6d 28602bef d760c3c7 93bc8d36 91f37ff8 db1113c4 9c7776c1

aeb7026a 817aa945 83e205e6 b956c194 378f4871 6322ec17 6507958a 4bdf8fc6

5a0ae5b0 e35f5e6b 11ab0cf9 85eb44e9 f80473f2 e9fe5c98 8cf573af 6bb47ecd

d45c022b 4c39e1b2 95952d42 87d7d5b3 9043b76c 13f1dedd f6c4f889 3fd175f5

92c391d5 8a88d090 ecdc6dde 89c26571 968b0d03 fd9cbf5b 16ac92db eafe797c

adebaff7 16cbdbcd 252be51f fb9a9fe2 51cc3a53 0c48e60e bdc9b476 0652e611

13857263 0304e004 362b2019 02e874a7 1fb6c956 66f07525 dc67c10e 616088b3

3ed1a8fc a3da1db0 d1b12354 df44766d ed41d8c1 b222b653 1cdf351d dca1772a

31e42df5 e5e5dbc8 e0ffe580 d70b63a0 ff33a10f ba2c1515 ea97b3d2 a2b5bef2

8c961e1a 8f1d6ca4 6137b986 7333d797 969e237d 82a44c81 e2a1d1ba 675f9507

a32711ee 16107bbc 454a4cb2 04d2abef d5fd0c51 ce506a08 31f991da 0c8f645c

03c33a8b 203f6e8d 673d3ad6 fe7d5b88 c95efbcc 61dc8b33 77d34432 35096204

921610d8 9e2747fb 3b21e3f8 eb1d5b02 03010001 a381b030 81ad300f 0603551d

130101ff 04053003 0101ff30 0b060355 1d0f0404 03020106 301d0603 551d0e04

1604141a 8462bc48 4c332504 d4eed0f6 03c41946 d1946b30 6e060355 1d230467

30658014 1a8462bc 484c3325 04d4eed0 f603c419 46d1946b a149a447 3045310b

30090603 55040613 02424d31 19301706 0355040a 13105175 6f566164 6973204c

696d6974 6564311b 30190603 55040313 1251756f 56616469 7320526f 6f742043

41203282 02050930 0d06092a 864886f7 0d010105 05000382 0201003e 0a164d9f

065ba8ae 715d2f05 2f67e613 4583c436 f6f3c026 0c0db547 645df8b4 72c946a5

03182755 89787d76 ea963480 1720dce7 83f88dfc 07b8da5f 4d2e67b2 84fdd944

fc775081 e67cb4c9 0d0b7253 f8760707 4147960c fbe08226 93558cfe 221f6065

7c5fe726 b3f73290 9850d437 7155f692 2178f795 79faf82d 26876656 3077a637

78335210 58ae3f61 8ef26ab1 ef187e4a 5963ca8d a256d5a7 2fbc561f cf39c1e2

fb0aa815 2c7d4d7a 63c66c97 443cd26f c34a170a f890d257 a21951a5 2d9741da

074fa950 da908d94 46e13ef0 94fd1000 38f53be8 40e1b46e 561a20cc 6f588ded

2e458fd6 e9933fe7 b12cdf3a d6228cdc 84bb226f d0f8e4c6 39e90488 3cc3baeb

557a6d80 9924f56c 01fbf897 b0945beb fdd26ff1 77680d35 6423acb8 55a103d1

4d4219dc f8755956 a3f9a849 79f8af0e b911a07c b76aed34 d0b62662 381a870c

f8e8fd2e d3907f07 912a1dd6 7e5c8583 99b03808 3fe95ef9 3507e4c9 626e577f

a75095f7 bac89be6 8ea201c5 d666bf79 61f33c1c e1b9825c 5da0c3e9 d848bd19

a2111419 6eb2861b 683e4837 1a88b75d 965e9cc7 ef276208 e291195c d2f121dd

ba174282 97718153 31a99ff6 7d62bf72 e1a3931d cc8a265a 0938d0ce d70d8016

b478a53a 874c8d8a a5d54697 f22c10b9 bc5422c0 01506943 9ef4b2ef 6df8ecda

f1e3b1ef df918f54 2a0b25c1 2619c452 100565d5 8210eac2 31cd2e

quit

crypto ikev1 enable outside

crypto ikev1 policy 5

authentication pre-share

encryption aes

hash md5

group 2

lifetime 86400

telnet timeout 5

ssh stricthostkeycheck

ssh 0.0.0.0 0.0.0.0 outside

ssh 0.0.0.0 0.0.0.0 inside

ssh timeout 5

ssh version 2

ssh key-exchange group dh-group1-sha1

console timeout 0

management-access outside

no threat-detection basic-threat

threat-detection statistics access-list

no threat-detection statistics tcp-intercept

ntp server 172.31.0.12

ssl cipher default low

ssl cipher tlsv1 low

ssl cipher tlsv1.1 low

ssl cipher tlsv1.2 low

ssl cipher dtlsv1 low

dynamic-access-policy-record DfltAccessPolicy

username admin password XXX privilege 15

username admin attributes

service-type admin

tunnel-group 6.6.6.6 type ipsec-l2l

tunnel-group 6.6.6.6 ipsec-attributes

ikev1 pre-shared-key *****

!

class-map inspection_default

match default-inspection-traffic

!

policy-map type inspect dns migrated_dns_map_1

parameters

message-length maximum client auto

message-length maximum 512

no tcp-inspection

policy-map global_policy

class inspection_default

inspect dns migrated_dns_map_1

inspect ftp

inspect h323 h225

inspect h323 ras

inspect ip-options

inspect netbios

inspect rsh

inspect rtsp

inspect skinny

inspect esmtp

inspect sqlnet

inspect sunrpc

inspect tftp

inspect sip

inspect xdmcp

policy-map type inspect dns migrated_dns_map_2

parameters

message-length maximum client auto

message-length maximum 512

no tcp-inspection

!

Sheraz.Salim · ‎02-26-2019

You need to change the order of NAT rule.

object network PROBLEM-SERVER
host 172.31.0.30
!
object network PROBLEM-SERVER-MAP-IP
host 5.5.7.137
!
nat (inside,outside) 2 source static PROBLEM-SERVER PROBLEM-SERVER-MAP-IP

please do not forget to rate.

View solution in original post

venkat_n7 · ‎02-25-2019

i have few questions.

1. can you do packet trace to from server to outside when the NAT functionality fails and see what it says.

2. do yo need to use gt-1023 port on acls?

Please rate comments and support
with regards,
Venkat

TobyB · ‎02-25-2019

1. Packet Trace. Please let me know if I did something incorrectly with my command, I am still learning a lot with ASA troubleshooting:

FW# packet-tracer input inside tcp 172.31.0.30 443 9.9.9.9 https detailed

Phase: 1
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
Additional Information:
Forward Flow based lookup yields rule:
in id=0x2aaad9c49bf0, priority=1, domain=permit, deny=false
hits=2286783, user_data=0x0, cs_id=0x0, l3_type=0x8
src mac=0000.0000.0000, mask=0000.0000.0000
dst mac=0000.0000.0000, mask=0100.0000.0000
input_ifc=inside, output_ifc=any

Phase: 2
Type: ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
Additional Information:
found next-hop 5.5.5.1 using egress ifc outside

Phase: 3
Type: NAT
Subtype:
Result: ALLOW
Config:
object network PROBLEM SERVER
nat (inside,outside) static 5.5.5.5
Additional Information:
Static translate 172.31.0.30/443 to 5.5.5.5/443
Forward Flow based lookup yields rule:
in id=0x2aaad9cbfb20, priority=6, domain=nat, deny=false
hits=53, user_data=0x2aaad9cbeda0, cs_id=0x0, flags=0x0, protocol=0
src ip/id=172.31.0.30, mask=255.255.255.255, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=inside, output_ifc=outside

Phase: 4
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x2aaad8918ef0, priority=0, domain=nat-per-session, deny=false
hits=325850, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=6
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=any

Phase: 5
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x2aaad9c51f10, priority=0, domain=inspect-ip-options, deny=true
hits=191699, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=inside, output_ifc=any

Phase: 6
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
in id=0x2aaad8918ef0, priority=0, domain=nat-per-session, deny=false
hits=325852, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=6
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=any

Phase: 7
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
in id=0x2aaad9bba1c0, priority=0, domain=inspect-ip-options, deny=true
hits=202298, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=outside, output_ifc=any

Phase: 8
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 194493, packet dispatched to next module
Module information for forward flow ...
snp_fp_inspect_ip_options
snp_fp_tcp_normalizer
snp_fp_translate
snp_fp_adjacency
snp_fp_fragment
snp_fp_tracer_drop
snp_ifc_stat

Module information for reverse flow ...
snp_fp_inspect_ip_options
snp_fp_translate
snp_fp_tcp_normalizer
snp_fp_adjacency
snp_fp_fragment
snp_fp_tracer_drop
snp_ifc_stat

Phase: 9
Type: ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
Additional Information:
found next-hop 5.5.5.1 using egress ifc outside

Phase: 10
Type: ADJACENCY-LOOKUP
Subtype: next-hop and adjacency
Result: ALLOW
Config:
Additional Information:
adjacency Active
next-hop mac address 0000.8888.4444 hits 63 reference 105

Result:
input-interface: inside
input-status: up
input-line-status: up
output-interface: outside
output-status: up
output-line-status: up
Action: allow

2. Is there a better/different option, or is that unnecessary? I am going from some notes that I had from a colleague plus the structure of my legacy firewall. It was pre 8.3 though, so I am still getting accustomed to some of the new commands.

Thank you for replying, I appreciate the help with this.

venkat_n7 · ‎02-25-2019

i dont see any drops on packet trace. so far i can say is capture the logs during the failure and see what happend.

and can you add

"logging trap informational"
also can you export the logs from firewall to a remote syslog server so that we can see what happened when the "NAT" failed.

your acl structure is old fashioned like prior to 8.3 as you said. you have to use object-group format, which makes more sense to read and audit the firewall config.

Below is suggested structure for configuration format to follow.

object-group network HS-LDAP-SVR
network-object host 172.31.0.12
object-group network HS-RDP-55550
network-object host 172.31.0.22
object-group network HS-HVAC-WWW
network-object host 172.31.2.250
object-group network PROBLEM-SVR
network-object host 172.31.0.30

object-group network HOST_204.14.14.18
network-object host 204.14.14.18
object-group network HOST_204.14.12.2
network-object host 204.14.12.2

object-group network SQL_HOSTS
group-object HOST_204.14.14.18
group-object HOST_204.14.12.2

object-group network 189-72-146-214.PVOCE702_IP
network-object host 189.72.146.214

object-group service tcpWWW tcp
port-object eq www
object-group service TCP_TELNET tcp
port-object eq telnet

object-group service tcpFTPDATA tcp
port-object eq ftp
port-object eq ftp-data

object-group service tcpSSL tcp
port-object eq 8443
port-object eq 7880
port-object eq 5071

object-group service tcpRMT_ACCESS tcp
port-object eq 55550
object-group service tcpSQL tcp
port-object eq sqlnet

access-list HS-Filter-In remark ***--PROBLEM SERVER ACL Start--***
access-list HS-Filter-In extended permit tcp any object-group PROBLEM-SVR object-group tcpSSL log
access-list HS-Filter-In extended permit tcp any object-group PROBLEM-SVR object-group tcpWWW log
access-list HS-Filter-In extended permit tcp any object-group PROBLEM-SVR object-group tcpFTPDATA log
access-list HS-Filter-In remark *--Remote Access to PROBLEM SERVER--*
access-list HS-Filter-In extended permit tcp any object-group PROBLEM-SVR object-group tcpRMT_ACCESS log
access-list HS-Filter-In extended permit tcp object-group SQL_HOSTS object-group PROBLEM-SVR object-group tcpSQL log

access-list HS-Filter-In remark *--ASM Connection to PROBLEM SERVER--*
access-list HS-Filter-In extended permit tcp object-group 189-72-146-214.PVOCE702_IP object-group PROBLEM-SVR object-group TCP_TELNET log

Please rate comments and support
with regards,
Venkat

TobyB · ‎02-26-2019

I have adjusted my syslog settings so it can take a look at the logs when I lose connection, checking on it every hour.

I am also working to rewrite my ACLs using the method you posted, thank you for the information. I will reply again once I have logs during a failure.

venkat_n7 · ‎02-26-2019

Great

Please rate comments and support
with regards,
Venkat

TobyB · ‎02-26-2019

I see this repeated in the logs before I noticed the failure.

2019-02-26 11:43:35 Local4.Info 172.31.250.100 Feb 26 2019 11:43:35: %ASA-6-302013: Built outbound TCP connection 1155584 for outside:52.37.78.48/443 (52.37.78.48/443) to inside:172.31.0.30/58724 (5.5.5.5/58724)

2019-02-26 11:43:35 Local4.Info 172.31.250.100 Feb 26 2019 11:43:35: %ASA-6-302014: Teardown TCP connection 1155576 for outside:52.37.78.48/443 to inside:172.31.0.30/58723 duration 0:00:00 bytes 1931 TCP FINs from inside

2019-02-26 11:43:35 Local4.Info 172.31.250.100 Feb 26 2019 11:43:35: %ASA-6-106015: Deny TCP (no connection) from 172.31.0.30/58723 to 52.37.78.48/443 flags RST on interface inside

I check my mobile device to see if the service is still working, webpage times out

2019-02-26 11:43:43 Local4.Info 172.31.250.100 Feb 26 2019 11:43:43: %ASA-6-106100: access-list HS-Filter-In permitted tcp outside/34.211.131.161(2197) -> inside/172.31.0.30(443)

2019-02-26 11:43:43 Local4.Info 172.31.250.100 Feb 26 2019 11:43:43: %ASA-6-302013: Built inbound TCP connection 1155765 for outside:34.211.131.161/2197 (34.211.131.161/2197) to inside:172.31.0.30/443 (5.5.5.5/443)

ACL hits showing permitted but service still unavailable, no tcp builds

2019-02-26 11:44:17 Local4.Info 172.31.250.100 Feb 26 2019 11:44:17: %ASA-6-106100: access-list HS-Filter-In permitted tcp outside/34.211.131.161(21578) -> inside/172.31.0.30(443)

2019-02-26 11:44:17 Local4.Info 172.31.250.100 Feb 26 2019 11:44:17: %ASA-6-106100: access-list HS-Filter-In permitted tcp outside/34.211.131.161(65136) -> inside/172.31.0.30(443)

Large chunks of tcp builds from a client accessing service via VPN can be seen (service still works on my LAN and over VPN, just thought this seemed like a rather large chunk of requests?) There are several large groups of these. This user is not on site in the evenings when the service fails though, so I don’t believe this is to blame?

2019-02-26 11:45:51 Local4.Info 172.31.250.100 Feb 26 2019 11:45:51: %ASA-6-302013: Built inbound TCP connection 1159261 for outside:172.31.11.58/52057 (172.31.11.58/52057) to inside:172.31.0.30/443 (172.31.0.30/443)

2019-02-26 11:45:51 Local4.Info 172.31.250.100 Feb 26 2019 11:45:51: %ASA-6-302013: Built inbound TCP connection 1159262 for outside:172.31.11.58/52056 (172.31.11.58/52056) to inside:172.31.0.30/443 (172.31.0.30/443)

2019-02-26 11:45:51 Local4.Info 172.31.250.100 Feb 26 2019 11:45:51: %ASA-6-302013: Built inbound TCP connection 1159263 for outside:172.31.11.58/52058 (172.31.11.58/52058) to inside:172.31.0.30/443 (172.31.0.30/443)

2019-02-26 11:45:51 Local4.Info 172.31.250.100 Feb 26 2019 11:45:51: %ASA-6-302013: Built inbound TCP connection 1159264 for outside:172.31.11.58/52059 (172.31.11.58/52059) to inside:172.31.0.30/443 (172.31.0.30/443)

2019-02-26 11:45:51 Local4.Info 172.31.250.100 Feb 26 2019 11:45:51: %ASA-6-302013: Built inbound TCP connection 1159265 for outside:172.31.11.58/52060 (172.31.11.58/52060) to inside:172.31.0.30/443 (172.31.0.30/443)

2019-02-26 11:45:51 Local4.Info 172.31.250.100 Feb 26 2019 11:45:51: %ASA-6-302013: Built inbound TCP connection 1159266 for outside:172.31.11.58/52061 (172.31.11.58/52061) to inside:172.31.0.30/443 (172.31.0.30/443)

2019-02-26 11:45:51 Local4.Info 172.31.250.100 Feb 26 2019 11:45:51: %ASA-6-302013: Built inbound TCP connection 1159267 for outside:172.31.11.58/52062 (172.31.11.58/52062) to inside:172.31.0.30/443 (172.31.0.30/443)

2019-02-26 11:45:51 Local4.Info 172.31.250.100 Feb 26 2019 11:45:51: %ASA-6-302013: Built inbound TCP connection 1159268 for outside:172.31.11.58/52063 (172.31.11.58/52063) to inside:172.31.0.30/443 (172.31.0.30/443)

2019-02-26 11:45:51 Local4.Info 172.31.250.100 Feb 26 2019 11:45:51: %ASA-6-302013: Built inbound TCP connection 1159270 for outside:172.31.11.58/52066 (172.31.11.58/52066) to inside:172.31.0.30/443 (172.31.0.30/443)

2019-02-26 11:45:51 Local4.Info 172.31.250.100 Feb 26 2019 11:45:51: %ASA-6-302013: Built inbound TCP connection 1159273 for outside:172.31.11.58/52069 (172.31.11.58/52069) to inside:172.31.0.30/443 (172.31.0.30/443)

2019-02-26 11:45:51 Local4.Info 172.31.250.100 Feb 26 2019 11:45:51: %ASA-6-302013: Built inbound TCP connection 1159269 for outside:172.31.11.58/52065 (172.31.11.58/52065) to inside:172.31.0.30/443 (172.31.0.30/443)

I remove the NAT statement for server, RDP to server, access the internet via browser on the affected server, replace NAT statement, open a browser on the server again, and then the connections become available again. Only solution that I have found to restore access temporarily.

2019-02-26 11:43:43 Local4.Info 172.31.250.100 Feb 26 2019 11:43:43: %ASA-6-106100: access-list HS-Filter-In permitted tcp outside/34.211.131.161(2197) -> inside/172.31.0.30(443)

2019-02-26 11:43:43 Local4.Info 172.31.250.100 Feb 26 2019 11:43:43: %ASA-6-302013: Built inbound TCP connection 1155765 for outside:34.211.131.161/2197 (34.211.131.161/2197) to inside:172.31.0.30/443 (5.5.5.5/443)

I don't see anything out of the ordinary here, just the sudden drop of TCP connections being built. Strange that it built one when I checked on my mobile device if the service was working or not. Again, thank you for taking the time to look.

Sheraz.Salim · ‎02-26-2019

You need to change the order of NAT rule.

object network PROBLEM-SERVER
host 172.31.0.30
!
object network PROBLEM-SERVER-MAP-IP
host 5.5.7.137
!
nat (inside,outside) 2 source static PROBLEM-SERVER PROBLEM-SERVER-MAP-IP

please do not forget to rate.

TobyB · ‎02-26-2019

I thought that the way I had things set up that it would be dynamically taken care of. I've made the changes, will monitor to see if behavior continues and report back. Thank you!

TobyB · ‎02-27-2019

Apologies for taking a while to respond to if this has worked or not...several power outages this week have interrupted my ability to monitor performance. It has been acting better, but I don't have a clear grasp on if it is solved or yet.

TobyB · ‎02-28-2019

Still seeing the same behavior. My syslog server did not come back up properly after the power outage last night, so I am working on capturing more logs now.

TobyB · ‎03-04-2019

After having more time to observe and catching logs, the issue persists, but the behavior is now different after adjusting the NAT statement. Before moving my NAT rule, one the connection was severed, I had to remove then re-add the NAT rule to bring access back online. Now, access comes back after a short period of time (roughly 1 hour seems to be the trend) with no intervention from me. Logs are still the same as posted previously, no warnings/errors, there is just a random point where TCP builds stop happening, and a random point where they resume for that one NAT.

Sheraz.Salim · ‎03-04-2019

Is this server is access able on the VPN. as you have site-to-site vpn configured. if so could you also confirm the site-to-site vpn user having this issue too?

what is the average load of traffic on your ASA box?

I have look into your config. Cant see any issue in regards to the NAT rules and other config. none spotted any issue.

please do not forget to rate.

TobyB · ‎03-11-2019

Follow up on this. After initially making the changes, the behavior changed but I still had some issues. A week later it has now gone away completely. I believe that there was more than one issue at play here, but this step is what stabilized things. Thank you for your help.