Cisco ASA 9.1 5540 NAT statement not getting hit

[efreymuth_2]

Hello all,

I am an amateur when it comes to the true science behind some of what I am trying to configure so I love to hear explanations as to why it is not working, as well as get it fixed.  I have a Cisco 5540 running 9.1.  I have an outside, p_wired, dmz, private interfaces setup and working.  Everyone can access the internet like I would expect.  The dmz_webserver can access the outside in order to do updates but I cannot get to the website that I want to host on the dmz_webserver from the public internet.  Below is my current running config.  The immediate packet-tracer command shows a result of allow, so I am truly lost.  Any help is truly appreciated.  I have been reading and studying for almost 2 weeks because I like to try and figure things like this out myself.  

packet-tracer input outside tcp 18.218.108.31 1234 192.168.2.100 80 detailed

The p_wired interface has good internet access and I can carry out all tasks needed.  I can access the dmz interface from the p_wired as I would like because of the security-level settings are working.  The dmz has good internet access to the server and any other device I connect to it.  The private network is not a concern and is working as expected.

ASA Version 9.1(7)23
!
hostname ciscoasa
enable password [removed]
names
dns-guard
!
interface GigabitEthernet0/0
nameif outside
security-level 0
ip address dhcp setroute
!
interface GigabitEthernet0/1
nameif p_wired
security-level 50
ip address 172.16.1.1 255.255.0.0
!
interface GigabitEthernet0/2
nameif dmz
security-level 25
ip address 192.168.2.1 255.255.255.0
!
interface GigabitEthernet0/3
nameif private
security-level 100
ip address 10.0.0.1 255.0.0.0
!
interface Management0/0
shutdown
no nameif
no security-level
no ip address
!
ftp mode passive
dns domain-lookup p_wired
dns domain-lookup dmz
dns server-group DefaultDNS
name-server 8.8.8.8
name-server 4.4.2.2
same-security-traffic permit inter-interface
same-security-traffic permit intra-interface
object network dmz_webserver
host 192.168.2.100
object network outside_acl
object network dmz_acl
object service HTTP-8080
service tcp source eq 8080
object service HTTP-80
service tcp source eq www
object network dmz_subnet
subnet 192.168.2.0 255.255.255.0
access-list outside_acl extended permit tcp any4 object dmz_webserver eq www
access-list outside_acl extended permit tcp any4 object dmz_webserver eq 8080
access-list outside_acl extended permit tcp any object dmz_webserver eq www
access-list outside_acl extended permit tcp any any eq www
access-list outside_acl extended permit tcp any any eq 8080
access-list outside_acl extended permit ip any any
pager lines 24
logging enable
mtu outside 1500
mtu p_wired 1500
mtu dmz 1500
mtu private 1500
no failover
icmp unreachable rate-limit 1 burst-size 1
no asdm history enable
arp timeout 14400
arp permit-nonconnected

nat (p_wired,outside) source dynamic any interface
nat (dmz,outside) source static any dmz_webserver service HTTP-80 HTTP-80
nat (dmz,outside) source static any dmz_webserver service HTTP-8080 HTTP-8080
nat (dmz,outside) source dynamic any interface
!
object network dmz_webserver
nat (dmz,outside) static interface
access-group outside_acl in interface outside
timeout xlate 3:00:00
timeout pat-xlate 0:00:30
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02
timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00
timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00
timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute
timeout tcp-proxy-reassembly 0:01:00
timeout floating-conn 0:00:00
dynamic-access-policy-record DfltAccessPolicy
user-identity default-domain LOCAL
no snmp-server location
no snmp-server contact
crypto ipsec security-association pmtu-aging infinite
crypto ca trustpool policy
telnet timeout 5
ssh stricthostkeycheck
ssh timeout 5
ssh key-exchange group dh-group1-sha1
console timeout 0
dhcpd dns 8.8.8.8 4.4.2.2
!
dhcpd address 172.16.100.1-172.16.100.100 p_wired
dhcpd enable p_wired
!
dhcpd address 192.168.2.100-192.168.2.120 dmz
dhcpd enable dmz
!
dhcpd address 10.10.10.1-10.10.10.100 private
dhcpd enable private
!
threat-detection basic-threat
threat-detection statistics host number-of-rate 2
threat-detection statistics access-list
threat-detection statistics tcp-intercept rate-interval 30 burst-rate 400 average-rate 200
webvpn
anyconnect-essentials
cache
disable
!
!
!
policy-map global_policy
!
prompt hostname context
no call-home reporting anonymous
Cryptochecksum:[removed]
: end

---- Below is the result of NAT translation after I ran the packet-tracer command at the beginning twice.

ciscoasa(config)# show nat
Manual NAT Policies (Section 1)
1 (p_wired) to (outside) source dynamic any interface
translate_hits = 324, untranslate_hits = 5
2 (dmz) to (outside) source static any dmz_webserver service HTTP-80 HTTP-80
translate_hits = 2, untranslate_hits = 2
3 (dmz) to (outside) source static any dmz_webserver service HTTP-8080 HTTP-8080
translate_hits = 0, untranslate_hits = 0
4 (dmz) to (outside) source dynamic any interface
translate_hits = 0, untranslate_hits = 0

Auto NAT Policies (Section 2)
1 (dmz) to (outside) source static dmz_webserver interface
translate_hits = 0, untranslate_hits = 0

If you need anything else to help me out please let me know.  I know the dmz_webserver is working and the ports are listening because I have verified with the netstat command and I can access the website from either a dmz or p_wired connected device.

Thanks,

Eldon

[GRANT3779]

I would remove the following two nats

nat (dmz,outside) source static any
dmz_webserver service HTTP-80 HTTP-80
nat (dmz,outside) source static any dmz_webserver service HTTP-8080 HTTP-8080
Amend the following nat
nat (dmz,outside) source dynamic any interface
To

nat (dmz,outside) after-auto source dynamic any interface

[efreymuth_2]

Thanks for the reply.  I am open to any troubleshooting at this point.  I did as you described and I had tried that setup before, but it was worth another try.  This didn't work, and actually caused the packet-tracer to drop the test packet-tracer and the external attempt to access the website failed (ERR_CONNECTION_TIMED_OUT).  When I run Wireshark capture I can see that the SYN packets always perform a resend and never get an ACK.  Not sure if that helps.

Manual NAT Policies (Section 1)
1 (p_wired) to (outside) source dynamic any interface
translate_hits = 1831, untranslate_hits = 25

Auto NAT Policies (Section 2)
1 (dmz) to (outside) source static dmz_webserver interface
translate_hits = 0, untranslate_hits = 0

Manual NAT Policies (Section 3)
1 (dmz) to (outside) source dynamic any interface
translate_hits = 0, untranslate_hits = 0

ciscoasa(config)# packet-tracer input outside tcp 18.218.108.31 1234 192.168.2$

Phase: 1
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
Additional Information:
Forward Flow based lookup yields rule:
in id=0x79b61700, priority=1, domain=permit, deny=false
hits=568403, user_data=0x0, cs_id=0x0, l3_type=0x8
src mac=0000.0000.0000, mask=0000.0000.0000
dst mac=0000.0000.0000, mask=0100.0000.0000
input_ifc=outside, output_ifc=any

Phase: 2
Type: ROUTE-LOOKUP
Subtype: input
Result: ALLOW
Config:
Additional Information:
in 192.168.2.0 255.255.255.0 dmz

Phase: 3
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group outside_acl in interface outside
access-list outside_acl extended permit tcp any4 object dmz_webserver eq www
Additional Information:
Forward Flow based lookup yields rule:
in id=0x79cf0c70, priority=13, domain=permit, deny=false
hits=4, user_data=0x7610d640, cs_id=0x0, use_real_addr, flags=0x0, protocol=6
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
dst ip/id=192.168.2.100, mask=255.255.255.255, port=80, tag=0, dscp=0x0
input_ifc=outside, output_ifc=any

Phase: 4
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x794d5ed0, priority=0, domain=nat-per-session, deny=false
hits=25264, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=6
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0, dscp=0x0
input_ifc=any, output_ifc=any

Phase: 5
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x79c052e0, priority=0, domain=inspect-ip-options, deny=true
hits=10321, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0, dscp=0x0
input_ifc=outside, output_ifc=any

Phase: 6
Type: NAT
Subtype: rpf-check
Result: DROP
Config:
object network dmz_webserver
nat (dmz,outside) static interface
Additional Information:
Forward Flow based lookup yields rule:
out id=0x79cc3260, priority=6, domain=nat-reverse, deny=false
hits=1, user_data=0x79381300, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
dst ip/id=192.168.2.100, mask=255.255.255.255, port=0, tag=0, dscp=0x0
input_ifc=outside, output_ifc=dmz

[Rob Ingram]

Hi,
You are being dropped for RPF check.

Can you run the packet-tracer again, with the outside interface IP address as the destination rather than the private IP address of the dmz server. Post the output here for review.

[efreymuth_2]

Thank you for all the help so far.  I cannot show my appreciation enough for your time and knowledge sharing.

Now we are onto a path of teaching me something.  This is the first time I have got the packet-tracer to allow the packet through the destination G0/0 IP (outside).  Below are the results you requested.  Not sure if this helps your analysis but my G0/0 (outside) is given an IP from the ISP.  My ISP router is in bridge mode just to pass traffic to my ASA.

packet-tracer input outside tcp 18.218.108.31 80 [removed g0/0 ip] 80

Phase: 1
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
Additional Information:
Forward Flow based lookup yields rule:
in id=0x79b61700, priority=1, domain=permit, deny=false
hits=574261, user_data=0x0, cs_id=0x0, l3_type=0x8
src mac=0000.0000.0000, mask=0000.0000.0000
dst mac=0000.0000.0000, mask=0100.0000.0000
input_ifc=outside, output_ifc=any

Phase: 2
Type: ROUTE-LOOKUP
Subtype: input
Result: ALLOW
Config:
Additional Information:
in 0.0.0.0 0.0.0.0 outside

Phase: 3
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group outside_acl in interface outside
access-list outside_acl extended permit tcp any any eq www
Additional Information:
Forward Flow based lookup yields rule:
in id=0x79cc36f8, priority=13, domain=permit, deny=false
hits=7, user_data=0x7610d4c0, cs_id=0x0, use_real_addr, flags=0x0, protocol=6
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
dst ip/id=0.0.0.0, mask=0.0.0.0, port=80, tag=0, dscp=0x0
input_ifc=outside, output_ifc=any

Phase: 4
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x794d5ed0, priority=0, domain=nat-per-session, deny=false
hits=25952, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=6
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0, dscp=0x0
input_ifc=any, output_ifc=any

Phase: 5
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x79c052e0, priority=0, domain=inspect-ip-options, deny=true
hits=10738, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0, dscp=0x0
input_ifc=outside, output_ifc=any

Phase: 6
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
in id=0x794d5ed0, priority=0, domain=nat-per-session, deny=false
hits=25954, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=6
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0, dscp=0x0
input_ifc=any, output_ifc=any

Phase: 7
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
in id=0x79c052e0, priority=0, domain=inspect-ip-options, deny=true
hits=10740, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0, dscp=0x0
input_ifc=outside, output_ifc=any

Phase: 8
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 10978, packet dispatched to next module
Module information for forward flow ...
snp_fp_tracer_drop
snp_fp_inspect_ip_options
snp_fp_tcp_normalizer
snp_fp_translate
snp_fp_adjacency
snp_fp_fragment
snp_ifc_stat

Module information for reverse flow ...
snp_fp_tracer_drop
snp_fp_inspect_ip_options
snp_fp_translate
snp_fp_tcp_normalizer
snp_fp_adjacency
snp_fp_fragment
snp_ifc_stat

Result:
input-interface: outside
input-status: up
input-line-status: up
output-interface: outside
output-status: up
output-line-status: up
Action: allow

[GRANT3779]

Hi There,

If you add the following static NAT for the webserver and new object group just to keep things clean for testing -

object-group service OBJ_G_WEB_PORTS
description WEB_PORTS
service-object tcp eq 80
service-object tcp eq 443

nat (dmz,outside) source static dmz_webserver interface service OBJ_G_WEB_PORTS OBJ_G_WEB_PORTS

Remove all the current DMZ related NATs first and then add the above.

If there is nothing else that sits behind the DMZ requiring a static NAT then configure PAT for the rest of the DMZ after that will be processed after the static NAT -

object network OBJ_DMZ
subnet 192.168.2.0 255.255.255.0
!
object network OBJ_DMZ
nat (dmz,outside) dynamic interface

Before any testing I would clear any open xlates from the table (if this is a test environment)

[efreymuth_2]

Could you clarify or refine your recommendations?  I am getting an error trying to implement your first NAT statement.

ciscoasa(config-network-object)# object-group service OBJ_G_WEB_PORTS
ciscoasa(config-service-object-group)# service-object tcp eq 80
ciscoasa(config-service-object-group)# service-object tcp eq 443
ciscoasa(config-service-object-group)# exit
ciscoasa(config)# nat (dmz,outside) source static dmz_webserver interface serv$
ERROR: OBJ_G_WEB_PORTS is not a valid service object name
ciscoasa(config)#

[GRANT3779]

Hi, 

Looking at this again, you will be bound by one port at a time for the the NAT using services. Try the following, removing previous DMZ NATs. 

object network OBJ_DMZ_80

host 192.168.2.100

nat (dmz,outside) static interface service tcp 80 80

object network OBJ_DMZ_8080

host 192.168.2.100
nat (dmz,outside) static interface service tcp 8080 8080

I would only test the server dmz nats for now then add dmz subnet pat once all good. Then tidy up test / redundant objects. 

[efreymuth_2]

Those two nat statements executed successfully, however that didn't seem to do anything.  I get an ACL drop on the below packet tracer command, and when I try to navigate to the website from the public internet the NAT is not getting any hits.  

What is next on the list to try?

packet-tracer input outside tcp 18.218.108.31 123 [removed ip] 80

[GRANT3779]

Can I see a current show run nat and also a show nat output, please. 

[GRANT3779]

Also, output of show access-list
Do you see hits on outside ACL?

[efreymuth_2]

I don't see hits to the ACL.  The (5) hits to the Line 1 ACL below is from previous testing.  I haven't altered the ACL for this specific set of testing.  I also can't get NMAP to report any of the ports as open.  So, I am still lost and appreciate any help or recommendations you offer.

ciscoasa(config)# show run nat
nat (p_wired,outside) source dynamic any interface
!
object network OBJ_DMZ_80
nat (dmz,outside) static interface service tcp www www
object network OBJ_DMZ_8080
nat (dmz,outside) static interface service tcp 8080 8080

ciscoasa(config)# show nat
Manual NAT Policies (Section 1)
1 (p_wired) to (outside) source dynamic any interface
translate_hits = 13885, untranslate_hits = 593

Auto NAT Policies (Section 2)
1 (dmz) to (outside) source static OBJ_DMZ_80 interface service tcp www www
translate_hits = 0, untranslate_hits = 0
2 (dmz) to (outside) source static OBJ_DMZ_8080 interface service tcp 8080 8080
translate_hits = 0, untranslate_hits = 0

ciscoasa(config)# show access-list
access-list cached ACL log flows: total 0, denied 0 (deny-flow-max 4096)
alert-interval 300
access-list outside_acl; 6 elements; name hash: 0x6b8df462
access-list outside_acl line 1 extended permit tcp any4 object dmz_webserver eq www (hitcnt=5) 0x7c82cf95
access-list outside_acl line 1 extended permit tcp any4 host 192.168.2.100 eq www (hitcnt=5) 0x7c82cf95
access-list outside_acl line 2 extended permit tcp any4 object dmz_webserver eq 8080 (hitcnt=0) 0x4d1442bd
access-list outside_acl line 2 extended permit tcp any4 host 192.168.2.100 eq 8080 (hitcnt=0) 0x4d1442bd
access-list outside_acl line 3 extended permit tcp any object dmz_webserver eq www (hitcnt=0) 0xc62102d5
access-list outside_acl line 3 extended permit tcp any host 192.168.2.100 eq www (hitcnt=0) 0xc62102d5
access-list outside_acl line 4 extended permit tcp any any eq www (hitcnt=10) 0xf13d4901
access-list outside_acl line 5 extended permit tcp any any eq 8080 (hitcnt=0) 0x63c2fd73
access-list outside_acl line 6 extended permit ip any any (hitcnt=631) 0x31f6627e
ciscoasa(config)#

[GRANT3779]

Can I see the packet tracer output for your test.
Regardless of this are you certain the ISP is forwarding ports etc for Inbound static NATs to the interfaces? I see the Outside IP is DHCP.

[GRANT3779]

What is the output from show run all sysopt?

[efreymuth_2]

Just spoke with the ISP and they claim all ports are open except for 19, 23, 53, 1900.  I am going to work on some modem configurations (again) to see if I overlooked something (doubtful).  But I put it in Bridged mode so I would expect if the ports are open like they claim it would hit either a NAT or ACL when I attempt the website from the public internet.

ciscoasa(config)# show run all sysopt
no sysopt connection timewait
sysopt connection tcpmss 1380
sysopt connection tcpmss minimum 0
sysopt connection permit-vpn
sysopt connection reclassify-vpn
no sysopt connection preserve-vpn-flows
no sysopt radius ignore-secret
no sysopt noproxyarp outside
no sysopt noproxyarp p_wired
no sysopt noproxyarp dmz
no sysopt noproxyarp private