Cisco ASA 9.2 - Cannot get FTP to work...either way!

[djhillssc]

Hi all!  Hoping one of you can shed some light before I tear out what remains of my hair (not much, but I value it!)

Have a Cisco ASA running 9.2.  From factory reset did a quick configuration to test since I'm used to the old school PIX units and know some things are different on ASA.  Using the CLI I configured it with outside/inside interface, one test machine on the inside and one on the outside.  Few basic ACLs to allow web traffic and RDP...and...FTP.  I cannot get FTP to work - not from the inside out or the outside in...nada.  

Packet tracer shows the connection denied by an implicit rule inbound...but also shows a hitcount incrementing on the ACL *allowing* FTP.   I have no ACL on the inside interface...yet...while I can connect to other services from the inside outbound...can't get FTP!

What am I missing?

My config:

: Saved

:
: Serial Number: FCH1714J7GB
: Hardware: ASA5515, 4096 MB RAM, CPU Clarkdale 3058 MHz, 1 CPU (4 cores)
: Written by enable_15 at 22:01:23.459 UTC Thu Apr 21 2022
!
ASA Version 9.12(2)9
!
hostname ciscoasa
domain-name djhill.com
enable password ***** pbkdf2
names
no mac-address auto

!
interface GigabitEthernet0/0
nameif outside
security-level 0
ip address 169.160.35.94 255.255.255.224
!
interface GigabitEthernet0/1
nameif inside
security-level 100
ip address 10.1.1.1 255.255.255.224
!
interface GigabitEthernet0/2
shutdown
no nameif
no security-level
no ip address
!
interface GigabitEthernet0/3
shutdown
no nameif
no security-level
no ip address
!
interface GigabitEthernet0/4
shutdown
no nameif
no security-level
no ip address
!
interface GigabitEthernet0/5
shutdown
no nameif
no security-level
no ip address
!
interface Management0/0
management-only
nameif management
security-level 0
ip address 192.168.1.1 255.255.255.0
!
ftp mode passive
dns server-group DefaultDNS
domain-name djhill.com
object network ip_22
host 10.1.1.22
access-list outside_access_in extended permit tcp any object ip_22 eq www
access-list outside_access_in extended permit tcp any object ip_22 eq https
access-list outside_access_in extended permit tcp any object ip_22 eq 3389
access-list outside_access_in extended permit tcp any object ip_22 eq ftp
access-list outside_access_in extended permit tcp any host 169.160.35.80 eq ftp
pager lines 24
mtu outside 1500
mtu inside 1500
mtu management 1500
no failover
no monitor-interface service-module
icmp unreachable rate-limit 1 burst-size 1
no asdm history enable
arp timeout 14400
no arp permit-nonconnected
arp rate-limit 16384
!
object network ip_22
nat (inside,outside) static 69.160.35.80
access-group outside_access_in in interface outside
route outside 0.0.0.0 0.0.0.0 169.160.35.65 1
timeout xlate 3:00:00
timeout pat-xlate 0:00:30
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 sctp 0:02:00 icmp 0:00:02
timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00
timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00
timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute
timeout tcp-proxy-reassembly 0:01:00
timeout floating-conn 0:00:00
timeout conn-holddown 0:00:15
timeout igp stale-route 0:01:10
user-identity default-domain LOCAL
aaa authentication login-history
no snmp-server location
no snmp-server contact
crypto ipsec security-association pmtu-aging infinite
crypto ca trustpool policy
telnet timeout 5
ssh stricthostkeycheck
ssh timeout 5
ssh version 2
console timeout 0
threat-detection basic-threat
threat-detection statistics access-list
no threat-detection statistics tcp-intercept
dynamic-access-policy-record DfltAccessPolicy
!
class-map inspection_default
match default-inspection-traffic
!
!
policy-map type inspect dns preset_dns_map
parameters
message-length maximum client auto
message-length maximum 512
no tcp-inspection
policy-map global_policy
class inspection_default
inspect ftp
inspect h323 h225
inspect h323 ras
inspect ip-options
inspect netbios
inspect rsh
inspect rtsp
inspect skinny
inspect esmtp
inspect sqlnet
inspect sunrpc
inspect tftp
inspect sip
inspect xdmcp
inspect dns preset_dns_map
policy-map type inspect dns migrated_dns_map_2
parameters
message-length maximum client auto
message-length maximum 512
no tcp-inspection
policy-map type inspect dns migrated_dns_map_1
parameters
message-length maximum client auto
message-length maximum 512
no tcp-inspection
!
service-policy global_policy global
prompt hostname context
no call-home reporting anonymous
call-home
profile CiscoTAC-1
no active
destination address http https://tools.cisco.com/its/service/oddce/services/DDCEService
destination address email callhome@cisco.com
destination transport-method http
subscribe-to-alert-group diagnostic
subscribe-to-alert-group environment
subscribe-to-alert-group inventory periodic monthly
subscribe-to-alert-group configuration periodic monthly
subscribe-to-alert-group telemetry periodic daily

------------------------

Packet tracer tracing connection from outside to inside being denied by implicit rule...

packet-tracer input outside tcp 169.160.35.65 0 169.160.35.80 21 detai$

Phase: 1
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f02999fb940, priority=1, domain=permit, deny=false
hits=10, user_data=0x0, cs_id=0x0, l3_type=0x8
src mac=0000.0000.0000, mask=0000.0000.0000
dst mac=0000.0000.0000, mask=0100.0000.0000
input_ifc=outside, output_ifc=any

Phase: 2
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
object network ip_22
nat (inside,outside) static 169.160.35.80
Additional Information:
NAT divert to egress interface inside
Untranslate 169.160.35.80/21 to 10.1.1.22/21

Phase: 3
Type: ACCESS-LIST
Subtype:
Result: DROP
Config:
Implicit Rule
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f0299a05aa0, priority=501, domain=permit, deny=true
hits=0, user_data=0x8, cs_id=0x0, reverse, flags=0x0, protocol=6
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=outside, output_ifc=any

Result:
input-interface: outside
input-status: up
input-line-status: up
output-interface: inside
output-status: up
output-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule

------------------------------------

And yet making an FTP connection attempt shows the permit ACL being hit!:

ciscoasa# show access-list
access-list cached ACL log flows: total 0, denied 0 (deny-flow-max 4096)
alert-interval 300
access-list outside_access_in; 5 elements; name hash: 0x6892a938
access-list outside_access_in line 1 extended permit tcp any object ip_22 eq www (hitcnt=0) 0xffa2058b
access-list outside_access_in line 1 extended permit tcp any host 10.1.1.22 eq www (hitcnt=0) 0xffa2058b
access-list outside_access_in line 2 extended permit tcp any object ip_22 eq https (hitcnt=0) 0x54d9400f
access-list outside_access_in line 2 extended permit tcp any host 10.1.1.22 eq https (hitcnt=0) 0x54d9400f
access-list outside_access_in line 3 extended permit tcp any object ip_22 eq 3389 (hitcnt=0) 0x90e6c128
access-list outside_access_in line 3 extended permit tcp any host 10.1.1.22 eq 3389 (hitcnt=0) 0x90e6c128
access-list outside_access_in line 4 extended permit tcp any object ip_22 eq ftp (hitcnt=2) 0x2e8fcd72
access-list outside_access_in line 4 extended permit tcp any host 10.1.1.22 eq ftp (hitcnt=2) 0x2e8fcd72
access-list outside_access_in line 5 extended permit tcp any host 169.160.35.80 eq ftp (hitcnt=0) 0x4af15c96

ARGGHHH...wth??  Anyone got a tip for me?  

Many thanks in advance from a rusty but used-to-be guru lol

Dan

[djhillssc]

Interesting...if I attempt FTP connection outside->inside, and then run sh conn before the FTP client times out, it shows the following:

TCP outside 169.160.35.65:61429 inside 10.1.1.22:21, idle 0:00:01, bytes 0, flags SaAB

Does this mean that an initial connection is being made through the firewall?  Maybe control port opened on 21 but issue getting responses back through the ASA?

[MHM Cisco World]

https://www.cisco.com/c/en/us/support/docs/content-networking/file-transfer-protocol-ftp/200194-ASA-9-x-Configure-FTP-TFTP-Services.html

[djhillssc]

Thanks @MHM Cisco World but that's the documentation I used to configure and troubleshoot it...if you look at my posted config it has all of the necessary ACLs and commands...

[MHM Cisco World]

access-list outside_access_in extended permit tcp any object ip_22 eq ftp <- remove this 
access-list outside_access_in extended permit tcp any host 169.160.35.80 eq ftp <- change the IP of server with MAPPED IP since the UN_NAT is done before ACL 

[djhillssc]

@MHM Cisco World The ACL referencing object ip_22 refers to the mapped IP (the actual IP of the server, not the NAT'd).  The config shows that network object created as host 10.1.1.22, so that reference is correct.  The ACL with the external IP I added when it wasn't working in the hope I had misunderstood something and it would work.

Running sh access-list indicates incrementing hitcount on the ACL with the actual IP, so the ASA is catching that ACL correctly.

However, I did as you suggested, removed the ACL with the nat'd IP and swapped the ACL with the actual IP reference (host 10.1.1.22 instead of object ip_22).

THe issue remains...FTP into or out of the ASA does not work.  

If I enable terminal logging I can see the FTP connection attempt build a TCP connection as one would anticipate...and then it tears it down after a minute or so.  From the FTP client it simply is a connection timeout.  As mentioned above, packet-tracer shows an implicit rule denying the connection, while the ACL allowing the connection gets an incremented hitcount (bizarre)...

[MHM Cisco World]

object network ip_22
nat (inside,outside) static 69.160.35.80 <- I think this typo??

packet-tracer input outside tcp 169.160.35.65 0 169.160.35.80 21 detai$ <- I found the issue in packet the tcp port is 0 use 12345 port instead check packet-tracer again attach the result here.

[djhillssc]

Yes the NAT Ip you reference was a type-o...but not in the actual config.  I sanitized the config before posting by changing the outside IPs to random made up ones (since the outside IP is actually in use in production, I don't want to post it publicly).  So that should have said 169.x.x.x but in the actual config the IP is correct and matches all the way through.

I'll try what you suggest with the packet tracer since I'm clearly out of ideas, but:

1. My understanding of referencing a source port as 0 is that in Cisco parlance that means "any" port.  So in theory should not make a difference, though I'm willing to test it (later today, as I'm not there now).
2. Given that in fact FTP traffic is being blocked when attempted (real FTP connect attempts), I don't think we'll learn anything new from that test.

But I'll try it and post later.

Thanks for your attempts to help!

[MHM Cisco World]

Packet-tracer must enter correctly to get our goal, for example in IPSec S2S we must do two packet-tracer !! to check S2S.
please do the below packet-tracer and share output.

packet-tracer input outside tcp 200.1.1.10 12345 10.1.1.22 21 detail

[MHM Cisco World]

Any update ??

[djhillssc]

@MHM Cisco World hey sorry busy day!  Ok I tried that packet tracer.  Interestingly it got much much further in the trace process...

It did ultimately fail at Phase 8 (reversing the NAT on outside interface I believe) an "rpf-check".  Any insight into that?  Maybe that's where my FTP connects are failing, and I've been attributing it to the implicit deny since my source port 0 packet traces got dropped earlier in the process...hmmmm

Phase: 1
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7ff230e1a980, priority=1, domain=permit, deny=false
hits=0, user_data=0x0, cs_id=0x0, l3_type=0x8
src mac=0000.0000.0000, mask=0000.0000.0000
dst mac=0000.0000.0000, mask=0100.0000.0000
input_ifc=outside, output_ifc=any

Phase: 2
Type: ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
Additional Information:
found next-hop 10.1.1.22 using egress ifc inside

Phase: 3
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group outside_access_in in interface outside
access-list outside_access_in extended permit tcp any object ip_22 eq ftp
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7ff230f2a240, priority=13, domain=permit, deny=false
hits=0, user_data=0x7ff228748640, cs_id=0x0, use_real_addr, flags=0x0, protocol=6
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=10.1.1.22, mask=255.255.255.255, port=21, tag=any, dscp=0x0
input_ifc=outside, output_ifc=any

Phase: 4
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7ff230105ca0, priority=0, domain=nat-per-session, deny=false
hits=0, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=6
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=any

Phase: 5
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7ff230e22000, priority=0, domain=inspect-ip-options, deny=true
hits=0, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=outside, output_ifc=any

Phase: 6
Type: INSPECT
Subtype: inspect-ftp
Result: ALLOW
Config:
class-map inspection_default
match default-inspection-traffic
policy-map global_policy
class inspection_default
inspect ftp
service-policy global_policy global
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7ff23181e880, priority=70, domain=inspect-ftp, deny=false
hits=1, user_data=0x7ff23181e700, cs_id=0x0, use_real_addr, flags=0x0, protocol=6
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=21, tag=any, dscp=0x0
input_ifc=outside, output_ifc=any

Phase: 7
Type: VPN
Subtype: ipsec-tunnel-flow
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7ff231294430, priority=13, domain=ipsec-tunnel-flow, deny=true
hits=1, user_data=0x0, cs_id=0x0, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=outside, output_ifc=any

Phase: 8
Type: NAT
Subtype: rpf-check
Result: DROP
Config:
object network ip_22
nat (inside,outside) static 169.160.35.80
Additional Information:
Forward Flow based lookup yields rule:
out id=0x7ff230f1e740, priority=6, domain=nat-reverse, deny=false
hits=1, user_data=0x7ff22fd1fe50, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=10.1.1.22, mask=255.255.255.255, port=0, tag=any, dscp=0x0
input_ifc=outside, output_ifc=inside

Result:
input-interface: outside
input-status: up
input-line-status: up
output-interface: inside
output-status: up
output-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule

[MHM Cisco World]

packet-tracer input outside tcp 200.1.1.10 12345 169.160.35.80 21 detail <- do this change only

share the output,
rpf-check meaning the UN-NAT is different than NAT

we use Inside IP not IP access from outside the rpf-check is failed 

[djhillssc]

@MHM Cisco World that makes perfect sense.  Sorry for slow reply, I had a weekend with the Mrs.  Pretty sure if I worked this weekend she'd run off with the pool boy (and take the boat with her).  Not sure why we have a pool boy.  We don't have a pool.  But that's a question for another time.

Redid the packet-tracer as requested, and it succeeded.  However...still I can't FTP through the firewall (either way, in or out).  The connection either way times out on the client.

Using sh conn when the client is attempting to connect shows a connection open:

169.160.35.70:52901 inside 10.1.1.22:21, idle 0:00:00, bytes 0, flags SaAB

If I monitor logging (either via SSH or console) I can see the TCP connection get built, and then gets torn down a bit after the connection attempt fails at the client.

Any pointers?  I appreciate the help!

Dan

[djhillssc]

@MHM Cisco World BTW here's what the logging looks like:

%ASA-7-609001: Built local-host inside:10.1.1.22
%ASA-6-302013: Built inbound TCP connection 13 for outside:169.160.35.70/53020 (169.160.35.70/53020) to inside:10.1.1.22/21 (169.160.35.80/21)
%ASA-6-302014: Teardown TCP connection 13 for outside:169.160.35.70/53020 to inside:10.1.1.22/21 duration 0:00:30 bytes 0 SYN Timeout
%ASA-7-609002: Teardown local-host outside:169.160.35.70 duration 0:00:30
%ASA-7-609002: Teardown local-host inside:10.1.1.22 duration 0:00:30

[djhillssc]

Wondering if there's simply a routing problem from the inside -> outside.  I cannot get any connection from the inside (security 100) to outside (security 0).  It isn't being stopped by an ACL issue - packet tracer shows successful trace out...but I can't get http or anything from inside outbound.  

I have no ACL on my inside interface (only ACLs are listed in the config at the top, and apply only to inbound traffic on outside interface).

In my PIX firewall config that I'm essentially trying to replicate on the ASA, I had a few commands in that config that when I attempted to copy into the ASA, I got an error saying the command was deprecated, and use the nat command with network object defining host instead (the way it is in the config at the top).  Those commands that didn't translate but may be relevant to the issue were:

nat (inside) 1 0.0.0.0 0.0.0.0 0 0

global (outside) 1 interface

Jog any thoughts loose?