Solved: Port Forwarding ASA5505 packets dropping

staylor2112 · ‎01-18-2017

Outbound traffic works but inbound does not. I have checked my ACL and it looks correct, but I am still not seeing what I missed. I have attached our current running-config as well as a screenshot of the Packet Trace utility. Below is the detailed packet-tracer results.

Result of the command: "packet-tracer input outside tcp 8.8.8.8 888 192.168.1.88 888 detailed"

Phase: 1
Type: ROUTE-LOOKUP
Subtype: input
Result: ALLOW
Config:
Additional Information:
in   192.168.1.0     255.255.255.0   inside

Phase: 2
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group outside_access_in in interface outside
access-list outside_access_in extended permit object FrontelPort1 any object FRONTELSVR1
Additional Information:
Forward Flow based lookup yields rule:
in id=0xd003f6b8, priority=13, domain=permit, deny=false
   hits=7, user_data=0xca0e8a80, cs_id=0x0, use_real_addr, flags=0x0, protocol=6
   src ip/id=0.0.0.0, mask=0.0.0.0, port=888, tag=0
   dst ip/id=192.168.1.88, mask=255.255.255.255, port=888, tag=0 dscp=0x0
   input_ifc=outside, output_ifc=any

Phase: 3
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0xcba010a0, priority=0, domain=nat-per-session, deny=false
   hits=1193646144, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=6
   src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
   dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0 dscp=0x0
   input_ifc=any, output_ifc=any

Phase: 4
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0xcc08d5a0, priority=0, domain=inspect-ip-options, deny=true
   hits=612753499, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
   src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
   dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0 dscp=0x0
   input_ifc=outside, output_ifc=any

Phase: 5
Type: VPN
Subtype: ipsec-tunnel-flow
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0xcca5dc28, priority=13, domain=ipsec-tunnel-flow, deny=true
   hits=373410, user_data=0x0, cs_id=0x0, flags=0x0, protocol=0
   src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
   dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0 dscp=0x0
   input_ifc=outside, output_ifc=any

Phase: 6
Type: NAT
Subtype: rpf-check
Result: DROP
Config:
object network FRONTELSVR1
nat (inside,outside) static interface service tcp 888 888
Additional Information:
Forward Flow based lookup yields rule:
out id=0xd06d4bb0, priority=6, domain=nat-reverse, deny=false
   hits=2, user_data=0xd04266d8, cs_id=0x0, use_real_addr, flags=0x0, protocol=6
   src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
   dst ip/id=192.168.1.88, mask=255.255.255.255, port=888, tag=0 dscp=0x0
   input_ifc=outside, output_ifc=inside

Result:
input-interface: outside
input-status: up
input-line-status: up
output-interface: inside
output-status: up
output-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule

Rahul Govindan · ‎01-24-2017

It definitely looks like the ASA is dropping it although there is no reason given as to why

790: 07:23:35.273575 802.1Q vlan#2 P0 104.137.90.176.65498 > xx.xx.xx.xx.888: S 3047924272:3047924272(0) win 8192 <mss 1260,nop,wscale 8,nop,nop,sackOK>

Can you paste the packet-tracer output and the latest sanitized config again? Also, was there any syslogs that showed up on the ASA when this failed?

View solution in original post

Rahul Govindan · ‎01-18-2017

According to this statement,

object network FRONTELSVR1
 nat (inside,outside) static interface service tcp 888 888

Your server at 192.168.1.88 is translated to your outside interface ip address when tcp port 888 is used. Your packet tracer is failing because the ASA determines that there are asymmetric NAT rules being hit in the return path (hence reverse path failure or RPF). Your packet tracer should ideally be the below statement as outside users will hit the public ip address to access the server:

packet-tracer input outside tcp 8.8.8.8 888 <outside-interface-ip> 888 detailed

staylor2112 · ‎01-18-2017

Thank you for the quick reply.

Its been a while since I worked in this environment, yes I can see the un-NAT and NAT when using the packet-tracer input outside tcp 8.8.8.8 888 <outside-interface-ip> 888 detailed statement.

Why would this stop us from being able to telnet into the server using the public address on port 888? (telnet xx.xx.xx.xx 888)

Rahul Govindan · ‎01-18-2017

Telnet should work if the packet-tracer showed allow all the way through to the inside interface. Can you paste the output here?

You may want to apply a capture on the outside and inside interface to capture the entire transaction through the ASA. I would apply a capture like this:

cap capo interface outside match ip host <source ip address> any

cap capi interface inside match ip host <source ip address> any

Source ip address is the client's ip address on the internet.

staylor2112 · ‎01-18-2017

Hopefully I did this right:

Result of the command: "packet-tracer input outside tcp 8.8.8.8 888 xx.xx.xx.xx 888 detailed"

Phase: 1
Type: CAPTURE
Subtype:
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0xd0712e80, priority=13, domain=capture, deny=false
   hits=15533, user_data=0xd0712d40, cs_id=0x0, l3_type=0x0
   src mac=0000.0000.0000, mask=0000.0000.0000
   dst mac=0000.0000.0000, mask=0000.0000.0000
   input_ifc=outside, output_ifc=any

Phase: 2
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
Additional Information:
Forward Flow based lookup yields rule:
in id=0xcc087cc8, priority=1, domain=permit, deny=false
   hits=10642429583, user_data=0x0, cs_id=0x0, l3_type=0x8
   src mac=0000.0000.0000, mask=0000.0000.0000
   dst mac=0000.0000.0000, mask=0100.0000.0000
   input_ifc=outside, output_ifc=any

Phase: 3
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
object network FRONTELSVR1
nat (inside,outside) static interface service tcp 888 888
Additional Information:
NAT divert to egress interface inside
Untranslate xx.xx.xx.xx/888 to 192.168.1.88/888

Phase: 4
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group outside_access_in in interface outside
access-list outside_access_in extended permit object-group DM_INLINE_SERVICE_8 any object FRONTELSVR1
object-group service DM_INLINE_SERVICE_8
service-object object FrontelPort1
service-object tcp destination eq telnet
Additional Information:
Forward Flow based lookup yields rule:
in id=0xd003f6b8, priority=13, domain=permit, deny=false
   hits=0, user_data=0xca0e8a80, cs_id=0x0, use_real_addr, flags=0x0, protocol=6
   src ip/id=0.0.0.0, mask=0.0.0.0, port=888, tag=0
   dst ip/id=192.168.1.88, mask=255.255.255.255, port=888, tag=0 dscp=0x0
   input_ifc=outside, output_ifc=any

Phase: 5
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0xcba010a0, priority=0, domain=nat-per-session, deny=false
   hits=1197004907, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=6
   src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
   dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0 dscp=0x0
   input_ifc=any, output_ifc=any

Phase: 6
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0xcc08d5a0, priority=0, domain=inspect-ip-options, deny=true
   hits=614473460, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
   src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
   dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0 dscp=0x0
   input_ifc=outside, output_ifc=any

Phase: 7
Type: VPN
Subtype: ipsec-tunnel-flow
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0xcca5dc28, priority=13, domain=ipsec-tunnel-flow, deny=true
   hits=373713, user_data=0x0, cs_id=0x0, flags=0x0, protocol=0
   src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
   dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0 dscp=0x0
   input_ifc=outside, output_ifc=any

Phase: 8
Type: NAT
Subtype: rpf-check
Result: ALLOW
Config:
object network FRONTELSVR1
nat (inside,outside) static interface service tcp 888 888
Additional Information:
Forward Flow based lookup yields rule:
out id=0xd06d4bb0, priority=6, domain=nat-reverse, deny=false
   hits=6, user_data=0xd04266d8, cs_id=0x0, use_real_addr, flags=0x0, protocol=6
   src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
   dst ip/id=192.168.1.88, mask=255.255.255.255, port=888, tag=0 dscp=0x0
   input_ifc=outside, output_ifc=inside

Phase: 9
Type: USER-STATISTICS
Subtype: user-statistics
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
out id=0xcf077c90, priority=0, domain=user-statistics, deny=false
   hits=613755692, user_data=0xcf06e370, cs_id=0x0, reverse, flags=0x0, protocol=0
   src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
   dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0 dscp=0x0
   input_ifc=any, output_ifc=inside

Phase: 10
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
in id=0xcba010a0, priority=0, domain=nat-per-session, deny=false
   hits=1197004909, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=6
   src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
   dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0 dscp=0x0
   input_ifc=any, output_ifc=any

Phase: 11
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
in id=0xcc064370, priority=0, domain=inspect-ip-options, deny=true
   hits=614036763, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
   src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
   dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0 dscp=0x0
   input_ifc=inside, output_ifc=any

Phase: 12
Type: USER-STATISTICS
Subtype: user-statistics
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
out id=0xcf077ef8, priority=0, domain=user-statistics, deny=false
   hits=614737321, user_data=0xcf06e370, cs_id=0x0, reverse, flags=0x0, protocol=0
   src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
   dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0 dscp=0x0
   input_ifc=any, output_ifc=outside

Phase: 13
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 614835307, packet dispatched to next module
Module information for forward flow ...
snp_fp_tracer_drop
snp_fp_inspect_ip_options
snp_fp_tcp_normalizer
snp_fp_translate
snp_fp_adjacency
snp_fp_fragment
snp_ifc_stat

Module information for reverse flow ...
snp_fp_tracer_drop
snp_fp_inspect_ip_options
snp_fp_translate
snp_fp_tcp_normalizer
snp_fp_adjacency
snp_fp_fragment
snp_ifc_stat

Result:
input-interface: outside
input-status: up
input-line-status: up
output-interface: inside
output-status: up
output-line-status: up
Action: allow

Rahul Govindan · ‎01-18-2017

Packet-tracer looks good. Can you check if the capture on inside and outside interface show 2 way traffic?

staylor2112 · ‎01-18-2017

How can I check?

Rahul Govindan · ‎01-18-2017

Mentioned it in an earlier post:

You may want to apply a capture on the outside and inside interface to capture the entire transaction through the ASA. I would apply a capture like this:

cap capo interface outside match ip host <source ip address> any

cap capi interface inside match ip host <source ip address> any

Source ip address is the client's ip address on the internet. After sending traffic, check the captures using:

show capture capo
show capture capi

More info on capture here:

http://www.cisco.com/c/en/us/support/docs/security/asa-5500-x-series-next-generation-firewalls/118097-configure-asa-00.html

staylor2112 · ‎01-18-2017

I get 3 capi responses on port 888 from an internal telnet to out outside address, but nothing on the capo for port 888

Result of the command: "show capture capi"

3 packets captured

1: 08:15:29.396631       802.1Q vlan#1 P0 192.168.1.148.64584 > xx.xx.xx.xx.888: S 820795542:820795542(0) win 8192 <mss 1460,nop,wscale 8,nop,nop,sackOK>
2: 08:15:32.395853       802.1Q vlan#1 P0 192.168.1.148.64584 > xx.xx.xx.xx.888: S 820795542:820795542(0) win 8192 <mss 1460,nop,wscale 8,nop,nop,sackOK>
3: 08:15:38.395990       802.1Q vlan#1 P0 192.168.1.148.64584 > xx.xx.xx.xx.888: S 820795542:820795542(0) win 8192 <mss 1460,nop,nop,sackOK>
3 packets shown

Rahul Govindan · ‎01-18-2017

Are you trying to access the server from outside or from inside? From your captures looks like the source is an internal ip address (192.168.1.148) and destination is the ASA public ip address on the outside (xx.xx.xx.xx). This wont work, you cannot access resources on another interface of the ASA while coming in from a different interface.

Maybe I have mistaken your topology, but you had mentioned that access to the server was not working from the outside correct. Can you send some traffic from a host on the internet to the ASA's public ip address on port 888? Clear the captures using command "clear cap /all" before you test it.

staylor2112 · ‎01-18-2017

From the outside. I was testing from the inside thinking it would work. However, we cant connect even connect from another public address.

Rahul Govindan · ‎01-18-2017

Could you capture the packets when testing from outside? Your config looks ok and packet-tracer allows the packet to go all the way from outside to inside. We need to see if packets are actually being blocked by the ASA or if there is another reason it is getting dropped.

staylor2112 · ‎01-19-2017

To make sure I set up the capture correctly:

no capture capo

no capture capi

cap capo interface outside match ip host 104.137.90.176 any

cap capi interface inside match ip host 104.137.90.176 any

clear capture /all

Where 104.137.90.176 is the external address I will be testing from. (Should I put anything in the any place?)

Then I would be sending telnet xx.xx.xx.xx 888 from 104.137.90.176 to my public IP address xx.xx.xx.xx. Is this correct?

staylor2112 · ‎01-19-2017

I am not seeing any traffic on port 888. It looks like only traffic for Teamviewer from my home PC to my work PC.

I am confused as to why I am not even seeing the telnet attempt on port 888. Any additional thoughts?

Rahul Govindan · ‎01-19-2017

Not sure, we should see at least the TCP syn packet reaching the ASA from your ip address. Could it be possible that your ISP or gateway router is not allowing the port 888 across?