Cisco ASA 9.1 5540 NAT statement not getting hit

[efreymuth_2]

Hello all,

I am an amateur when it comes to the true science behind some of what I am trying to configure so I love to hear explanations as to why it is not working, as well as get it fixed.  I have a Cisco 5540 running 9.1.  I have an outside, p_wired, dmz, private interfaces setup and working.  Everyone can access the internet like I would expect.  The dmz_webserver can access the outside in order to do updates but I cannot get to the website that I want to host on the dmz_webserver from the public internet.  Below is my current running config.  The immediate packet-tracer command shows a result of allow, so I am truly lost.  Any help is truly appreciated.  I have been reading and studying for almost 2 weeks because I like to try and figure things like this out myself.  

packet-tracer input outside tcp 18.218.108.31 1234 192.168.2.100 80 detailed

The p_wired interface has good internet access and I can carry out all tasks needed.  I can access the dmz interface from the p_wired as I would like because of the security-level settings are working.  The dmz has good internet access to the server and any other device I connect to it.  The private network is not a concern and is working as expected.

ASA Version 9.1(7)23
!
hostname ciscoasa
enable password [removed]
names
dns-guard
!
interface GigabitEthernet0/0
nameif outside
security-level 0
ip address dhcp setroute
!
interface GigabitEthernet0/1
nameif p_wired
security-level 50
ip address 172.16.1.1 255.255.0.0
!
interface GigabitEthernet0/2
nameif dmz
security-level 25
ip address 192.168.2.1 255.255.255.0
!
interface GigabitEthernet0/3
nameif private
security-level 100
ip address 10.0.0.1 255.0.0.0
!
interface Management0/0
shutdown
no nameif
no security-level
no ip address
!
ftp mode passive
dns domain-lookup p_wired
dns domain-lookup dmz
dns server-group DefaultDNS
name-server 8.8.8.8
name-server 4.4.2.2
same-security-traffic permit inter-interface
same-security-traffic permit intra-interface
object network dmz_webserver
host 192.168.2.100
object network outside_acl
object network dmz_acl
object service HTTP-8080
service tcp source eq 8080
object service HTTP-80
service tcp source eq www
object network dmz_subnet
subnet 192.168.2.0 255.255.255.0
access-list outside_acl extended permit tcp any4 object dmz_webserver eq www
access-list outside_acl extended permit tcp any4 object dmz_webserver eq 8080
access-list outside_acl extended permit tcp any object dmz_webserver eq www
access-list outside_acl extended permit tcp any any eq www
access-list outside_acl extended permit tcp any any eq 8080
access-list outside_acl extended permit ip any any
pager lines 24
logging enable
mtu outside 1500
mtu p_wired 1500
mtu dmz 1500
mtu private 1500
no failover
icmp unreachable rate-limit 1 burst-size 1
no asdm history enable
arp timeout 14400
arp permit-nonconnected

nat (p_wired,outside) source dynamic any interface
nat (dmz,outside) source static any dmz_webserver service HTTP-80 HTTP-80
nat (dmz,outside) source static any dmz_webserver service HTTP-8080 HTTP-8080
nat (dmz,outside) source dynamic any interface
!
object network dmz_webserver
nat (dmz,outside) static interface
access-group outside_acl in interface outside
timeout xlate 3:00:00
timeout pat-xlate 0:00:30
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02
timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00
timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00
timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute
timeout tcp-proxy-reassembly 0:01:00
timeout floating-conn 0:00:00
dynamic-access-policy-record DfltAccessPolicy
user-identity default-domain LOCAL
no snmp-server location
no snmp-server contact
crypto ipsec security-association pmtu-aging infinite
crypto ca trustpool policy
telnet timeout 5
ssh stricthostkeycheck
ssh timeout 5
ssh key-exchange group dh-group1-sha1
console timeout 0
dhcpd dns 8.8.8.8 4.4.2.2
!
dhcpd address 172.16.100.1-172.16.100.100 p_wired
dhcpd enable p_wired
!
dhcpd address 192.168.2.100-192.168.2.120 dmz
dhcpd enable dmz
!
dhcpd address 10.10.10.1-10.10.10.100 private
dhcpd enable private
!
threat-detection basic-threat
threat-detection statistics host number-of-rate 2
threat-detection statistics access-list
threat-detection statistics tcp-intercept rate-interval 30 burst-rate 400 average-rate 200
webvpn
anyconnect-essentials
cache
disable
!
!
!
policy-map global_policy
!
prompt hostname context
no call-home reporting anonymous
Cryptochecksum:[removed]
: end

---- Below is the result of NAT translation after I ran the packet-tracer command at the beginning twice.

ciscoasa(config)# show nat
Manual NAT Policies (Section 1)
1 (p_wired) to (outside) source dynamic any interface
translate_hits = 324, untranslate_hits = 5
2 (dmz) to (outside) source static any dmz_webserver service HTTP-80 HTTP-80
translate_hits = 2, untranslate_hits = 2
3 (dmz) to (outside) source static any dmz_webserver service HTTP-8080 HTTP-8080
translate_hits = 0, untranslate_hits = 0
4 (dmz) to (outside) source dynamic any interface
translate_hits = 0, untranslate_hits = 0

Auto NAT Policies (Section 2)
1 (dmz) to (outside) source static dmz_webserver interface
translate_hits = 0, untranslate_hits = 0

If you need anything else to help me out please let me know.  I know the dmz_webserver is working and the ports are listening because I have verified with the netstat command and I can access the website from either a dmz or p_wired connected device.

Thanks,

Eldon

[GRANT3779]

What is the output from show dhcpd state? 

[efreymuth_2]

ciscoasa(config)# show dhcpd state
Context Configured as DHCP Server
Interface outside, Configured for DHCP CLIENT
Interface p_wired, Configured for DHCP SERVER
Interface dmz, Configured for DHCP SERVER
Interface private, Configured for DHCP SERVER

[GRANT3779]

What is your packet tracer output now btw? I would have expected to see an UN-NAT.
I'm not sure of this is related to the DHCP on the OUTSIDE interface. Do you know if this is a public IP you are getting or a rfc1918 and then it is being NAT'd again by ISP for outbound?
I've always worked with physically assigned addresses on the ASA.

[efreymuth_2]

So, can we regroup where I am at?  The below packet tracer was successful a couple days ago.  If the below is successful that should mean that the ASA is working?  I am confident it is a public IP address because I go to whatsmyip and the noted public IP matches the IP assigned to outside (g0/0) interface.  I can confirm that the IP address assigned by my ISP is not a Class A/B/C address so it should not be an RFC1918 address.  I cannot confirm the nat'ing again by the ISP but I can confirm there seems to be an issue with outbound.  When I run wireshark I never see an ACK for the SYN packets sent to the address.  I have been using the public (outside interface g0/0) in the browser bar because I was tired of waiting for the DNS to propagate from the domains I have setup.

ciscoasa(config)# packet-tracer input outside tcp 18.218.108.31 123 [removed ip]$

Phase: 1
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
Additional Information:
MAC Access list

Phase: 2
Type: ROUTE-LOOKUP
Subtype: input
Result: ALLOW
Config:
Additional Information:
in [removed ip] 255.255.255.255 identity

Phase: 3
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 4
Type: ACCESS-LIST
Subtype:
Result: DROP
Config:
Implicit Rule
Additional Information:

Result:

input-interface: outside
input-status: up
input-line-status: up
output-interface: NP Identity Ifc
output-status: up
output-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule

[GRANT3779]

Looking at the previous packet tracer outputs I'm not convinced they were showing as successful as such. I see no UN-NAT phase and also the end result showed same input/output interface (Outside).

 So I can check something (if possible) can you remove the pwired NAT you have configured so that the only two NATs left are webserver ones. Then clear the xlates on asa using

clear xlate

Rather than test packet tracer afterwards, can you check external access over the Internet first.

Also, what is the routing table output?

Sh route

[efreymuth_2]

I am getting nothing over the external internet.  I test it often using my cell phone on 4G just so I am not on my personal network at all.  I am not to keen on removing the NAT for p_wired.  That interface runs my entire publicly wired network to include the computers I use for wired internet access.  So, I would like to save that as a last resort.

I do know that the DHCP from the ISP is a public ip.  I will post the current IP config and then I am going to go and renew the lease to get another address so there is no sensitive information.  The show route command does seem to be giving some interesting results.  Maybe the DHCP from the ISP is misconfigured?  I don't know much about these routes so I am an amateur reading this, but the first connection is not the IP given in the show IP results.  I could take the modem out of bridge mode and set up a static route there to route traffic, then I could set g0/0 to be a static IP from the ISP router.  Thanks for taking the time, no rush because I am busy with work now that the week has started.  This is more like a weekend and after work project.

ciscoasa(config-network-object)# show route

Codes: C - connected, S - static, I - IGRP, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP
i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
* - candidate default, U - per-user static route, o - ODR
P - periodic downloaded static route

Gateway of last resort is 184.62.144.1 to network 0.0.0.0

C 184.62.144.0 255.255.240.0 is directly connected, outside
C 172.16.0.0 255.255.0.0 is directly connected, p_wired
C 10.0.0.0 255.0.0.0 is directly connected, private
C 192.168.2.0 255.255.255.0 is directly connected, dmz
d* 0.0.0.0 0.0.0.0 [1/0] via 184.62.144.1, outside

ciscoasa(config-network-object)# show ip
System IP Addresses:
Interface Name IP address Subnet mask Method
GigabitEthernet0/0 outside 184.62.151.127 255.255.240.0 DHCP
GigabitEthernet0/1 p_wired 172.16.1.1 255.255.0.0 manual
GigabitEthernet0/2 dmz 192.168.2.1 255.255.255.0 manual
GigabitEthernet0/3 private 10.0.0.1 255.0.0.0 manual
Current IP Addresses:
Interface Name IP address Subnet mask Method
GigabitEthernet0/0 outside 184.62.151.127 255.255.240.0 DHCP
GigabitEthernet0/1 p_wired 172.16.1.1 255.255.0.0 manual
GigabitEthernet0/2 dmz 192.168.2.1 255.255.255.0 manual
GigabitEthernet0/3 private 10.0.0.1 255.0.0.0 manual

[GRANT3779]

I have just done a quick and dirty lab setup for this up on my own 5512  (apart from DHCP address on Outside Interface) and it works as I would expect. I just used port 22 on my ESXI for testing and mocked up an Outside Network also.

My inside ESXi host is on 10.44.0.100 and I was NAT'ing to my ASA Outside Interface which was 66.66.66.66

object network OBJ_22
host 10.44.0.100
nat (inside,outside) static interface service tcp 22 22

Packet-Tracer for traffic coming in from Internet to my esxi server on port 22 -

ROWAN-FW-01(config)# packet-tracer input outside tcp 8.8.8.8 123 66.66.66.66 22

Phase: 1
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
object network OBJ_22
nat (Inside,Outside) static interface service tcp ssh ssh
Additional Information:
NAT divert to egress interface Inside
Untranslate 66.66.66.66/22 to 10.44.0.100/22

Phase: 2
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group ACL_NAT in interface Outside
access-list ACL_NAT extended permit ip any4 any4
Additional Information:

Phase: 3
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 4
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 5
Type: NAT
Subtype: rpf-check
Result: ALLOW
Config:
object network OBJ_22
nat (Inside,Outside) static interface service tcp ssh ssh
Additional Information:

Phase: 6
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 7
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 8
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 239, packet dispatched to next module

Result:
input-interface: Outside
input-status: up
input-line-status: up
output-interface: Inside
output-status: up
output-line-status: up
Action: allow

Also - For the server going out with a source port of 22 test -

ROWAN-FW-01(config)# packet-tracer input inside tcp 10.44.0.100 22 8.8.8.8 345

Phase: 1
Type: ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
Additional Information:
found next-hop 66.66.66.1 using egress ifc Outside

Phase: 2
Type: NAT
Subtype:
Result: ALLOW
Config:
object network OBJ_22
nat (Inside,Outside) static interface service tcp ssh ssh
Additional Information:
Static translate 10.44.0.100/22 to 66.66.66.66/22

Phase: 3
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 4
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 5
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 6
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 7
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 241, packet dispatched to next module

Result:
input-interface: Inside
input-status: up
input-line-status: up
output-interface: Outside
output-status: up
output-line-status: up
Action: allow

ROWAN-FW-01(config)#

I also stuck a laptop on my mock "Dirty Internet" and tested connecting to my server via its "Public" IP (ASA Outside Interface) and it NAT'd through fine. 

What I have noticed is that when I am connecting to my server from the Internet, the untranslated counter never goes up. Even the packet tracer output never causes it to rise. It does not increment even though the NAT works and shows on the ASA. I don't know if this is normal for a port translation to the Interface address.

The Translate counter however when I am going from my server outbound increments each time.

ROWAN-FW-01(config)# sh nat
Manual NAT Policies (Section 1)
1 (Inside) to (Outside) source static any any destination static OBJ_ANYCONNECT OBJ_ANYCONNECT no-proxy-arp route-lookup
translate_hits = 0, untranslate_hits = 0

Auto NAT Policies (Section 2)
1 (Inside) to (Outside) source static OBJ_22 interface service tcp ssh ssh
translate_hits = 5, untranslate_hits = 0
ROWAN-FW-01(config)#

I am running 9.8.3 code.

Re: - your Public IP, I have never dealt with DHCP from an ISP within a Corporate setup so not sure what is/isn't normal but a /20 on your interface seems pretty huge. Don't know if this would cause problems though. Regardless, I would expect success from Packet Tracer Output.

Maybe worth sharing your config and also the full output from the packet tracer testing in both directions. Maybe someone else knows more about the DHCP side of things also.

[efreymuth_2]

So, this is the point I did want to get to in order to confirm it might be my ISP settings and/or the DHCP configurations.  Could you post your running config so I could mirror that with a setup I am experimenting with?  I think I figured out a way to remove the DHCP aspect from g0/0 but I need a trustworthy setup to experiment with.  

[efreymuth_2]

Grant,

I am not sure how missed your lab setup post.  Now that I have the port forwarding figured out on my ISP router your lab configuration poses a couple questions before I start posting more configurations for you to troubleshoot and/or review.  

1. What is this defined as in your configuration: OBJ_ANYCONNECT

2. I get an error when I try to enter the route-lookup in the NAT statement.  "Option route-lookup is only allowed for static identity case

3. What mode are you operating this configuration in?  I have single context with firewall mode: router.

I set the outside interface to a static IP and I setup a route outside.

interface GigabitEthernet0/0
mac-address 0026.9986.77af
nameif outside
security-level 0
ip address 192.168.3.4 255.255.255.0
!
interface GigabitEthernet0/1
nameif p_wired
security-level 50
ip address 172.16.1.1 255.255.0.0
!

interface GigabitEthernet0/2
nameif dmz
security-level 25
ip address 192.168.2.1 255.255.255.0
!
interface GigabitEthernet0/3
nameif private
security-level 100
ip address 10.0.0.1 255.0.0.0
!
interface Management0/0
shutdown
no nameif
no security-level
no ip address
!
ftp mode passive
dns domain-lookup p_wired
dns domain-lookup dmz
dns server-group DefaultDNS
name-server 8.8.8.8
name-server 4.4.2.2
same-security-traffic permit inter-interface
same-security-traffic permit intra-interface
object network outside_acl
object network dmz_acl
object network webserver_80
host 192.168.2.100
object network webserver_8080
host 192.168.2.100
object service HTTP-80
service tcp source eq www
object service HTTP-8080
service tcp source eq 8080
access-list outside_acl extended permit tcp interface outside host 192.168.2.100 eq www
access-list outside_acl extended permit tcp interface outside host 192.168.2.100 eq 8080
access-list dmz_acl extended permit tcp any any eq www
access-list dmz_acl extended permit tcp any any eq 8080
pager lines 24
logging enable
mtu outside 1500
mtu p_wired 1500
mtu dmz 1500
mtu private 1500
no failover
icmp unreachable rate-limit 1 burst-size 1
no asdm history enable
arp timeout 14400
arp permit-nonconnected
nat (p_wired,outside) source dynamic any interface
nat (dmz,outside) source static any any destination static webserver_80 any no-proxy-arp
nat (dmz,outside) source static any any destination static webserver_8080 any no-proxy-arp
!
object network webserver_8080
nat (dmz,outside) static interface service tcp 8080 8080
!
nat (dmz,outside) after-auto source dynamic any interface
access-group outside_acl in interface outside
access-group dmz_acl in interface dmz
route outside 0.0.0.0 0.0.0.0 192.168.3.1 1
timeout xlate 3:00:00
timeout pat-xlate 0:00:30
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02
timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00
timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00
timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute
timeout tcp-proxy-reassembly 0:01:00
timeout floating-conn 0:00:00
dynamic-access-policy-record DfltAccessPolicy
user-identity default-domain LOCAL
no snmp-server location
no snmp-server contact
crypto ipsec security-association pmtu-aging infinite
crypto ca trustpool policy
telnet timeout 5
ssh stricthostkeycheck
ssh timeout 5
ssh key-exchange group dh-group1-sha1
console timeout 0
dhcpd dns 8.8.8.8 4.4.2.2
!
dhcpd address 172.16.100.1-172.16.100.100 p_wired
dhcpd enable p_wired
!
dhcpd address 192.168.2.100-192.168.2.120 dmz
dhcpd enable dmz
!
dhcpd address 10.10.10.1-10.10.10.100 private
dhcpd enable private
!
threat-detection basic-threat
threat-detection statistics host number-of-rate 2
threat-detection statistics access-list
threat-detection statistics tcp-intercept rate-interval 30 burst-rate 400 average-rate 200
webvpn
anyconnect-essentials
cache
disable
!
!
!
policy-map global_policy
!
prompt hostname context
no call-home reporting anonymous
Cryptochecksum:9949950f150986e664605b4479778386
: end

[GRANT3779]

The anyconnect object and NAT is unrelated. That is just in place for other testing I do on the firewall. 

Routed Mode - Single Context.

Looking at your config I did not realise you had the following still

nat (dmz,outside) source static any any destination static webserver_80 any no-proxy-arp
nat (dmz,outside) source static any any destination static webserver_8080 any no-proxy-arp
!

I would remove the above and ensure it has removed from the config. Then add the following - Note that the nat below gets configured directly under that object. 

object network webserver_80

Host 192.168.2.100 
nat (dmz,outside) static interface service tcp 8080 8080

Also delete the current outside ACL lines you have and amend to following

access-list outside_acl extended permit tcp any host 192.168.2.100 eq 80

access-list outside_acl extended permit tcp any host 192.168.2.100 eq 8080

This is assuming only port 80 and 8080 are what the server is listening on. 

[efreymuth_2]

Thanks, I just wanted to ask about that other stuff because that "destination" NAT statement was something I had not seen before.  Below is my updated running-config based on your recommendations.  I also provided the capture on the outside and dmz interface for 8080.  I know the server is listening because I can see it in the netstat and I can access the site through 192.168.2.100 anywhere inside my network.

I ran Wireshark and a capture on all 8080 packets into the outside interface and the dmz interface.  I can see the ports from the public internet hitting the outside interface but nothing is hitting the dmz in the captures.

Let me know what else you want to see or try.  I feel like we are making good progress because I can finally match up packets from my external web request to the outside interface.

ciscoasa(config)# capture outside8080 interface outside match tcp any any eq 8$
ciscoasa(config)# capture dmz8080 interface dmz match tcp any any eq 8080
ciscoasa(config)# show capture outside8080

15 packets captured

1: 22:56:31.530780 192.168.3.4.54893 > [public ip removed].8080: S 1954080724:1954080724(0) win 64240 <mss 1380,nop,wscale 8,no
2: 22:56:31.530902 192.168.3.4.54894 > [public ip removed].8080: S 2263337131:2263337131(0) win 64240 <mss 1380,nop,wscale 8,no
3: 22:56:31.782079 192.168.3.4.54895 > [public ip removed].8080: S 1440162043:1440162043(0) win 64240 <mss 1380,nop,wscale 8,no
4: 22:56:34.530719 192.168.3.4.54893 > [public ip removed].8080: S 1954080724:1954080724(0) win 64240 <mss 1380,nop,wscale 8,no
5: 22:56:34.531695 192.168.3.4.54894 > [public ip removed].8080: S 2263337131:2263337131(0) win 64240 <mss 1380,nop,wscale 8,no
6: 22:56:34.782613 192.168.3.4.54895 > [public ip removed].8080: S 1440162043:1440162043(0) win 64240 <mss 1380,nop,wscale 8,no
7: 22:56:40.531771 192.168.3.4.54893 > [public ip removed].8080: S 1954080724:1954080724(0) win 64240 <mss 1380,nop,wscale 8,no
8: 22:56:40.532717 192.168.3.4.54894 > [public ip removed].8080: S 2263337131:2263337131(0) win 64240 <mss 1380,nop,wscale 8,no
9: 22:56:40.784642 192.168.3.4.54895 > [public ip removed].8080: S 1440162043:1440162043(0) win 64240 <mss 1380,nop,wscale 8,no
10: 22:56:55.199849 192.168.3.4.54896 > [public ip removed].8080: S 1947578079:1947578079(0) win 64240 <mss 1380,nop,wscale 8,no
11: 22:56:55.199986 192.168.3.4.54897 > [public ip removed].8080: S 3397323018:3397323018(0) win 64240 <mss 1380,nop,wscale 8,no
12: 22:56:58.201649 192.168.3.4.54897 > [public ip removed].8080: S 3397323018:3397323018(0) win 64240 <mss 1380,nop,wscale 8,no
13: 22:56:58.201664 192.168.3.4.54896 > [public ip removed].8080: S 1947578079:1947578079(0) win 64240 <mss 1380,nop,wscale 8,no
14: 22:57:04.215503 192.168.3.4.54896 > [public ip removed].8080: S 1947578079:1947578079(0) win 64240 <mss 1380,nop,wscale 8,no
15: 22:57:04.231097 192.168.3.4.54897 > [public ip removed].8080: S 3397323018:3397323018(0) win 64240 <mss 1380,nop,wscale 8,no
15 packets shown

ciscoasa(config)# show capture dmz8080

0 packet captured

0 packet shown
ciscoasa(config)#

ciscoasa(config)# packet-tracer input outside tcp 8.8.8.8 1234 192.168.3.4 808$

Phase: 1
Type: ROUTE-LOOKUP
Subtype: input
Result: ALLOW
Config:
Additional Information:
in 192.168.3.4 255.255.255.255 identity

Phase: 2
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x794d5ed0, priority=0, domain=nat-per-session, deny=false
hits=38677, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=6
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0, dscp=0x0
input_ifc=any, output_ifc=any

Phase: 3
Type: ACCESS-LIST
Subtype:
Result: DROP
Config:
Implicit Rule
Additional Information:
Forward Flow based lookup yields rule:
in id=0x79b82810, priority=0, domain=permit, deny=true
hits=2158, user_data=0x9, cs_id=0x0, use_real_addr, flags=0x1000, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0, dscp=0x0
input_ifc=outside, output_ifc=any

Result:
input-interface: outside
input-status: up
input-line-status: up
output-interface: NP Identity Ifc
output-status: up
output-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule

ciscoasa(config)# packet-tracer input outside tcp 8.8.8.8 1234 192.168.2.100 8$

Phase: 1
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
Additional Information:
MAC Access list

Phase: 2
Type: ROUTE-LOOKUP
Subtype: input
Result: ALLOW
Config:
Additional Information:
in 192.168.2.0 255.255.255.0 dmz

Phase: 3
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 4
Type: ACCESS-LIST
Subtype:
Result: DROP
Config:
Implicit Rule
Additional Information:

Result:
input-interface: outside
input-status: up
input-line-status: up
output-interface: dmz
output-status: up
output-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule

[efreymuth_2]

Sorry, for the running-config while prepping all the data.

interface GigabitEthernet0/0
mac-address 0026.9986.77af
nameif outside
security-level 0
ip address 192.168.3.4 255.255.255.0
!
interface GigabitEthernet0/1
nameif p_wired
security-level 50
ip address 172.16.1.1 255.255.0.0
!
interface GigabitEthernet0/2
nameif dmz
security-level 25
ip address 192.168.2.1 255.255.255.0
!
interface GigabitEthernet0/3
nameif private
security-level 100
ip address 10.0.0.1 255.0.0.0
!
interface Management0/0
shutdown
no nameif
no security-level
no ip address
!
ftp mode passive
dns domain-lookup p_wired
dns domain-lookup dmz
dns server-group DefaultDNS
name-server 8.8.8.8
name-server 4.4.2.2
same-security-traffic permit inter-interface
same-security-traffic permit intra-interface
object network outside_acl
object network dmz_acl
object network webserver_80
host 192.168.2.100
object network webserver_8080
host 192.168.2.100
object service HTTP-80
service tcp source eq www
object service HTTP-8080
service tcp source eq 8080
access-list outside_acl extended permit tcp any host 192.168.2.100 eq www
access-list outside_acl extended permit tcp any host 192.168.2.100 eq 8080
pager lines 24
logging enable
mtu outside 1500
mtu p_wired 1500
mtu dmz 1500
mtu private 1500
no failover
icmp unreachable rate-limit 1 burst-size 1
no asdm history enable
arp timeout 14400
arp permit-nonconnected
nat (p_wired,outside) source dynamic any interface
!
object network webserver_80
nat (dmz,outside) static interface service tcp www www
object network webserver_8080
nat (dmz,outside) static interface service tcp 8080 8080
!
nat (dmz,outside) after-auto source dynamic any interface
route outside 0.0.0.0 0.0.0.0 192.168.3.1 1
timeout xlate 3:00:00
timeout pat-xlate 0:00:30
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02
timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00
timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00
timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute
timeout tcp-proxy-reassembly 0:01:00
timeout floating-conn 0:00:00
dynamic-access-policy-record DfltAccessPolicy
user-identity default-domain LOCAL
no snmp-server location
no snmp-server contact
crypto ipsec security-association pmtu-aging infinite
crypto ca trustpool policy
telnet timeout 5
ssh stricthostkeycheck
ssh timeout 5
ssh key-exchange group dh-group1-sha1
console timeout 0
dhcpd dns 8.8.8.8 4.4.2.2
!
dhcpd address 172.16.100.1-172.16.100.100 p_wired
dhcpd enable p_wired
!
dhcpd address 192.168.2.100-192.168.2.120 dmz
dhcpd enable dmz
!
dhcpd address 10.10.10.1-10.10.10.100 private
dhcpd enable private
!
threat-detection basic-threat
threat-detection statistics host number-of-rate 2
threat-detection statistics access-list
threat-detection statistics tcp-intercept rate-interval 30 burst-rate 400 average-rate 200
webvpn
anyconnect-essentials
cache
disable
!
!
!
policy-map global_policy
!
prompt hostname context
no call-home reporting anonymous
Cryptochecksum:
: end

[GRANT3779]

I no longer see the access list applied. 

Can you add 

access-group outside_acl in interface outside

[efreymuth_2]

That is surely my fault.  I did remove that during some testing and forgot to add it back in.  I have corrected that and done some packet tracing based on a review of your lab and I setup port 23 to isolate traffic and hopefully make things easier.

I am able to get packet-tracer to go out on port 23, but I am having issue with the UN-NAT phase on the packet-tracer coming in.  When I try a telnet from the command line I can see the packets hitting the capture but I get connection refused and the results are the same for port 8080.

Thanks for letting me know what to try.  

--- Traffic coming in to public IP ---

ciscoasa(config)# packet-tracer input outside tcp 8.8.8.8 123 [public ip removed] 23

Phase: 1
Type: CAPTURE
Subtype:
Result: ALLOW
Config:
Additional Information:
MAC Access list

Phase: 2
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
Additional Information:
MAC Access list

Phase: 3
Type: ROUTE-LOOKUP
Subtype: input
Result: ALLOW
Config:
Additional Information:
in 0.0.0.0 0.0.0.0 outside

Phase: 4
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group outside_acl in interface outside
access-list outside_acl extended permit ip any any
Additional Information:

Phase: 5
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 6
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 7
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 8
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 9
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:

New flow created with id 40275, packet dispatched to next module

Result:
input-interface: outside
input-status: up
input-line-status: up
output-interface: outside
output-status: up
output-line-status: up
Action: allow

--- packet tracer out from webserver on port 23 ---

ciscoasa(config)# packet-tracer input dmz tcp 192.168.2.100 23 8.8.8.8 234

Phase: 1
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
Additional Information:
MAC Access list

Phase: 2
Type: ROUTE-LOOKUP
Subtype: input
Result: ALLOW
Config:
Additional Information:
in 0.0.0.0 0.0.0.0 outside

Phase: 3
Type: NAT
Subtype:
Result: ALLOW
Config:
object network webserver_23
nat (dmz,outside) static interface service tcp telnet telnet
Additional Information:
Static translate 192.168.2.100/23 to 192.168.3.4/23

Phase: 4
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 5
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 6
Type: NAT
Subtype: rpf-check
Result: ALLOW
Config:
nat (dmz,outside) after-auto source dynamic any interface
Additional Information:

Phase: 7
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 8
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 9
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:

New flow created with id 40272, packet dispatched to next module

Result:
input-interface: dmz
input-status: up
input-line-status: up
output-interface: outside
output-status: up
output-line-status: up

--- packet trace to inside using the g0/0 IP address ---

ciscoasa(config)# packet-tracer input outside tcp 8.8.8.8 123 192.168.3.4 23

Phase: 1
Type: CAPTURE
Subtype:
Result: ALLOW
Config:
Additional Information:
MAC Access list

Phase: 2
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
Additional Information:
MAC Access list

Phase: 3
Type: ROUTE-LOOKUP
Subtype: input
Result: ALLOW
Config:
Additional Information:
in 192.168.3.4 255.255.255.255 identity

Phase: 4
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 5
Type: ACCESS-LIST
Subtype:
Result: DROP
Config:
Implicit Rule
Additional Information:

Result:
input-interface: outside
input-status: up
input-line-status: up
output-interface: NP Identity Ifc
output-status: up
output-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule

--- Running config ---

ciscoasa(config)# show run
: Saved
:
: Serial Number: JMX1403L0V2
: Hardware: ASA5540, 2048 MB RAM, CPU Pentium 4 2000 MHz
:
ASA Version 9.1(7)23
!
hostname ciscoasa
enable password 8Ry2YjIyt7RRXU24 encrypted
names
dns-guard
!
interface GigabitEthernet0/0
mac-address 0026.9986.77af
nameif outside
security-level 0
ip address 192.168.3.4 255.255.255.0
!
interface GigabitEthernet0/1
nameif p_wired
security-level 50
ip address 172.16.1.1 255.255.0.0
!
interface GigabitEthernet0/2
nameif dmz
security-level 25
ip address 192.168.2.1 255.255.255.0
!
interface GigabitEthernet0/3
nameif private
security-level 100
ip address 10.0.0.1 255.0.0.0
!
interface Management0/0
shutdown
no nameif
no security-level
no ip address
!

ftp mode passive
dns domain-lookup p_wired
dns domain-lookup dmz
dns server-group DefaultDNS
name-server 8.8.8.8
name-server 4.4.2.2
same-security-traffic permit inter-interface
same-security-traffic permit intra-interface
object network outside_acl
object network dmz_acl
object network webserver_80
host 192.168.2.100
object network webserver_8080
host 192.168.2.100
object network webserver_23
host 192.168.2.100
object service HTTP-80
service tcp source eq www
object service HTTP-8080
service tcp source eq 8080
object service HTTP-23
service tcp source eq telnet
object network outside_object
subnet 192.168.3.0 255.255.255.0
access-list outside_acl extended permit tcp any host 192.168.2.100 eq www
access-list outside_acl extended permit tcp any host 192.168.2.100 eq 8080
access-list outside_acl extended permit tcp any host 192.168.2.100 eq telnet
access-list outside_acl extended permit ip any any
access-list outside_acl extended deny ip any any
pager lines 24
logging enable
mtu outside 1500
mtu p_wired 1500
mtu dmz 1500
mtu private 1500
no failover
icmp unreachable rate-limit 1 burst-size 1
no asdm history enable
arp timeout 14400
arp permit-nonconnected
nat (p_wired,outside) source dynamic any interface
!
object network webserver_80
nat (dmz,outside) static interface service tcp www www
object network webserver_8080
nat (dmz,outside) static interface service tcp 8080 8080
object network webserver_23
nat (dmz,outside) static interface service tcp telnet telnet
!
nat (dmz,outside) after-auto source dynamic any interface
access-group outside_acl in interface outside
route outside 0.0.0.0 0.0.0.0 192.168.3.1 1
timeout xlate 3:00:00
timeout pat-xlate 0:00:30
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02
timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00
timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00
timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute
timeout tcp-proxy-reassembly 0:01:00
timeout floating-conn 0:00:00
dynamic-access-policy-record DfltAccessPolicy
user-identity default-domain LOCAL
no snmp-server location
no snmp-server contact
crypto ipsec security-association pmtu-aging infinite
crypto ca trustpool policy
telnet timeout 5
ssh stricthostkeycheck
ssh timeout 5
ssh key-exchange group dh-group1-sha1
console timeout 0
dhcpd dns 8.8.8.8 4.4.2.2
!
dhcpd address 172.16.100.1-172.16.100.100 p_wired
dhcpd enable p_wired
!
dhcpd address 192.168.2.100-192.168.2.120 dmz
dhcpd enable dmz
!
dhcpd address 10.10.10.1-10.10.10.100 private
dhcpd enable private
!
threat-detection basic-threat
threat-detection statistics host number-of-rate 2
threat-detection statistics access-list
threat-detection statistics tcp-intercept rate-interval 30 burst-rate 400 average-rate 200
webvpn
anyconnect-essentials
cache
disable
!
!
!
policy-map global_policy
!
prompt hostname context
no call-home reporting anonymous

: end

[John Paul Innes]

I'm not entirely sure how you are testing the webpage. Is the server you are running https? If so you may need to do a port translation/forward as you ASA Outside Interface will be listening on port 443.

Try the following but I am running out of ideas as all my testing on my own firewall works and I have even tested this with the same image as you have also.

Remove the following NAT setup.

object network webserver_8080
nat (dmz,outside) static interface service tcp 8080 8080

and amend it to the following 

object network webserver_8080

host 192.168.2.100
nat (Inside,Outside) static interface service tcp https 8080

Now from the Outside - Access your server on https://publicIP:8080

This will forward the traffic to the server on 443 rather than you directing it at the Outside Interface. This works fine for my server listening on 443.