Able to Ping through ASA but no TCP traffic

[Aaron.Pittenger]

Hey everyone,

So I've been troubleshooting this for about the last 10 hours and have stripped my ACL's down to a bare minimum to avoid any confusion.

I have an ASA 5505 setup with a DMZ. I have a server connected to the DMZ (PC1, AKA Historian, 10.11.1.10, VLAN1011). I am able to remote desktop from my outside network to PC1 (via NAT) without a problem. I am now trying to get PC1 to connect to a second PC that is connected to my inside network (PC2 - 192.168.50.14, VLAN 500). There is no NAT being used here. I can ping PC2, or any other device on my inside network from PC1, but I can't RDP (I can RDP to PC2 from another device on the inside network.) When I try to RDP, it doesn't even show up in the hit count of the ASA. I have also tried telnet instead of RDP just to rule out any funnyness there.

I have a static route added to the L3 switch on my inside network to route any traffic destined for 10.11.1.0 to 192.168.15.220 (which is the address of the ASA on the 192.168.15.0 network.) (ip route 10.11.1.0 255.255.255.0 192.168.15.219) I have no static routes defined on the ASA.

Please excuse any remnants of my troubleshooting efforts - but here is a backup of my config. I have done it with a combination of the CLI and ASDM 7.5. Thanks for any help! Note: There is a rough network architecture attached at the bottom as well as a screenshot from ASDM.

: Saved
:
: Serial Number: JMX1722Z0CG
: Hardware: ASA5505, 512 MB RAM, CPU Geode 500 MHz
: Written by admin at 22:26:37.989 EDT Mon Mar 20 2017
!
ASA Version 9.2(4)
!
hostname XXXXXXXXXXXX
enable password XXX encrypted
names
!
interface Ethernet0/0
switchport access vlan 15
switchport trunk allowed vlan 15,254,400,500,600
switchport trunk native vlan 254
switchport mode trunk
!
interface Ethernet0/1
!
interface Ethernet0/2
switchport access vlan 1011
!
interface Ethernet0/3
switchport access vlan 1011
!
interface Ethernet0/4
switchport access vlan 1011
!
interface Ethernet0/5
switchport access vlan 1011
!
interface Ethernet0/6
switchport access vlan 1011
!
interface Ethernet0/7
switchport access vlan 254
!
interface Vlan1
description XXXXX Corp Interface
nameif outside
security-level 0
ip address 192.168.6.190 255.255.255.0
!
interface Vlan15
description Inside Control Network Interface VLAN15
nameif InsideVLAN15
security-level 100
ip address 192.168.15.219 255.255.255.0
!
interface Vlan400
description Inside Control Network Interface VLAN400
nameif InsideVLAN400
security-level 100
ip address 192.168.40.219 255.255.255.0
!
interface Vlan500
description Inside Control Network Interface VLAN500
nameif InsideVLAN500
security-level 100
ip address 192.168.50.219 255.255.255.0
!
interface Vlan600
description Inside Control Network Interface VLAN600
nameif InsideVLAN600
security-level 100
ip address 192.168.60.219 255.255.255.0
!
interface Vlan1011
description DMZ Interface
nameif DMZ
security-level 50
ip address 10.11.1.1 255.255.255.0
!
interface Vlan1212
nameif Temp
security-level 0
ip address 12.12.1.1 255.255.255.0
!
boot system disk0:/asa924-k8.bin
ftp mode passive
clock timezone EST -5
clock summer-time EDT recurring
same-security-traffic permit inter-interface
object network C0EAE4CE44FA
host 192.168.6.1
description XXXXX Corporate Firewall
object network PC_L3_Switch
host 192.168.15.220
description Process Control Layer 3 Switch - XXX_OLDLAB_L3S5400
object network PC_VLAN40
subnet 192.168.40.0 255.255.255.0
description Process Control VLAN40
object network PC_VLAN50
subnet 192.168.50.0 255.255.255.0
description Process Control VLAN50
object network ASA5505
host 192.168.6.190
object network ASA5505-Historian
host 192.168.6.191
object network hqCorpLAN
subnet 192.168.9.0 255.255.255.0
description hqCorpLAN
object network XXXXXCorpLAN
subnet 192.168.6.0 255.255.255.0
description XXXXXCorpLAN
object network PC_DMZ_HISTORIAN
host 10.11.1.10
description PC_DMZ_HISTORIAN
object service RDP-3389
service tcp source eq 3389 destination eq 3389
description RDP-3389
object network ASA5505-Engineering
host 192.168.6.192
description Engineering Station
object network PC_DMZ_Engineering
host 12.12.1.11
object network XXXXXAzure
subnet 10.0.1.0 255.255.255.0
description XXXXXAzure
object network ASA5505-DMZ-IP
host 10.11.1.1
object service RDP-3390
service tcp source eq 3390 destination eq 3390
description RDP-3390
object network ASA5505-INSIDE-IP_VLAN15
host 192.168.15.219
object network ASA5505-INSIDE-IP_VLAN500
host 192.168.50.219
object network ASA5505-INSIDE-IP_VLAN400
host 192.168.40.219
object network ASA5505-INSIDE-IP_VLAN600
host 192.168.60.219
object network ASA5505-INSIDE-IP_VLAN254
host 192.168.254.219
object network HistorianRDPAccess
host 10.11.1.10
description Historian
object network Temp
host 12.12.1.11
object-group network DM_INLINE_NETWORK_1
network-object object hqCorpLAN
network-object object XXXXXCorpLAN
network-object object XXXXXAzure
object-group network DM_INLINE_NETWORK_2
network-object object hqCorpLAN
network-object object XXXXXCorpLAN
network-object object XXXXXAzure
object-group network DM_INLINE_NETWORK_3
network-object object PC_DMZ_HISTORIAN
network-object object PC_DMZ_Engineering
object-group network DM_INLINE_NETWORK_4
network-object object ASA5505-Engineering
network-object object PC_DMZ_Engineering
object-group network DM_INLINE_NETWORK_5
network-object object ASA5505-Historian
network-object object PC_DMZ_HISTORIAN
object-group network DM_INLINE_NETWORK_23
network-object object PC_DMZ_HISTORIAN
network-object object PC_DMZ_Engineering
object-group network DM_INLINE_NETWORK_6
network-object object hqCorpLAN
network-object object XXXXXCorpLAN
network-object object XXXXXAzure
object-group service RSLinxTCP tcp
port-object eq 135
port-object eq 2222
port-object eq 4241
port-object eq 44818
object-group service AnyTCP tcp
port-object range 1 65535
object-group network DM_INLINE_NETWORK_12
network-object host 192.168.40.25
network-object host 192.168.50.60
network-object host 192.168.60.21
object-group service FTHistorian tcp
port-object eq 5450
port-object range 5454 5459
port-object eq 5463
port-object eq 6000
object-group network DM_INLINE_NETWORK_10
network-object host 192.168.40.25
network-object host 192.168.50.60
network-object host 192.168.60.21
object-group service DM_INLINE_TCP_3 tcp
group-object FTHistorian
group-object RSLinxTCP
object-group service AnyUDP udp
port-object range 1 65535
object-group network DM_INLINE_NETWORK_19
network-object host 192.168.40.25
network-object host 192.168.50.60
network-object host 192.168.60.21
object-group network DM_INLINE_NETWORK_22
network-object host 192.168.40.25
network-object host 192.168.50.60
network-object host 192.168.60.21
object-group service RSLinxUDP udp
port-object eq 44818
object-group protocol TCPUDP
protocol-object udp
protocol-object tcp
access-list outside_access_in extended permit tcp object-group DM_INLINE_NETWORK_6 object PC_DMZ_Engineering eq 3389
access-list outside_access_in extended permit tcp object-group DM_INLINE_NETWORK_2 object PC_DMZ_HISTORIAN eq 3389
access-list outside_access_in extended permit tcp 192.168.6.0 255.255.255.0 object ASA5505 eq https
access-list outside_access_in extended permit icmp 192.168.6.0 255.255.255.0 object ASA5505
access-list dmz_access_in extended permit icmp any any
access-list dmz_access_in extended permit ip any any
access-list dmz_access_in extended permit object RDP-3389 any object-group DM_INLINE_NETWORK_23 inactive
access-list dmz_access_in extended permit tcp object PC_DMZ_HISTORIAN object-group DM_INLINE_NETWORK_10 object-group DM_INLINE_TCP_3 inactive
access-list dmz_access_in extended permit udp object PC_DMZ_HISTORIAN object-group DM_INLINE_NETWORK_22 object-group RSLinxUDP inactive
access-list inside_access_in extended permit object RDP-3389 any any
access-list inside_access_in extended permit icmp any any
access-list outside_access_in_1 extended permit ip any any
access-list InsideVLAN500_access_in_1 extended permit object RDP-3389 any any
access-list InsideVLAN500_access_in_1 extended permit icmp any any
access-list dmz_access_out extended permit icmp any any
access-list dmz_access_out extended permit ip any any
access-list DMZ_1_access_in extended permit object RDP-3389 any any
access-list DMZ_1_access_in extended permit icmp any any
pager lines 24
logging enable
logging asdm informational
mtu outside 1500
mtu InsideVLAN15 1500
mtu InsideVLAN400 1500
mtu InsideVLAN500 1500
mtu InsideVLAN600 1500
mtu DMZ 1500
mtu Temp 1500
no failover
icmp unreachable rate-limit 1 burst-size 1
asdm image disk0:/asdm-751.bin
no asdm history enable
arp timeout 14400
no arp permit-nonconnected
!
object network HistorianRDPAccess
nat (DMZ,outside) static ASA5505-Historian net-to-net service tcp 3389 3389
access-group outside_access_in_1 in interface outside control-plane
access-group outside_access_in in interface outside
access-group inside_access_in in interface InsideVLAN15
access-group InsideVLAN500_access_in_1 in interface InsideVLAN500
access-group DMZ_1_access_in in interface DMZ
timeout xlate 3:00:00
timeout pat-xlate 0:00:30
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02
timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00
timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00
timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute
timeout tcp-proxy-reassembly 0:01:00
timeout floating-conn 0:00:00
dynamic-access-policy-record DfltAccessPolicy
user-identity default-domain LOCAL
aaa authentication enable console LOCAL
aaa authentication http console LOCAL
aaa authentication serial console LOCAL
aaa authentication ssh console LOCAL
aaa authentication telnet console LOCAL
aaa authorization command LOCAL
http server enable
http 10.0.1.0 255.255.255.0 outside
http 192.168.6.0 255.255.255.0 outside
http 192.168.9.0 255.255.255.0 outside
no snmp-server location
no snmp-server contact
crypto ipsec security-association pmtu-aging infinite
crypto ca trustpool policy
telnet 192.168.6.0 255.255.255.0 outside
telnet timeout 5
ssh stricthostkeycheck
ssh 10.0.1.0 255.255.255.0 outside
ssh 192.168.6.0 255.255.255.0 outside
ssh 192.168.9.0 255.255.255.0 outside
ssh 10.11.1.0 255.255.255.0 DMZ
ssh timeout 5
ssh key-exchange group dh-group1-sha1
console timeout 0

threat-detection basic-threat
threat-detection statistics port
threat-detection statistics protocol
threat-detection statistics access-list
no threat-detection statistics tcp-intercept
ntp server 10.0.1.12 source outside prefer
ssl server-version any
ssl client-version any
username XXX password XXX encrypted privilege 15
!
!
prompt hostname context
no call-home reporting anonymous
call-home
profile CiscoTAC-1
no active
destination address http https://tools.cisco.com/its/service/oddce/services/DDCEService
destination address email callhome@cisco.com
destination transport-method http
subscribe-to-alert-group diagnostic
subscribe-to-alert-group environment
subscribe-to-alert-group inventory periodic monthly
subscribe-to-alert-group configuration periodic monthly
subscribe-to-alert-group telemetry periodic daily
Cryptochecksum:da62513220919118a3de2b93261af7c7
: end

[Marvin Rhoads · Accepted solution]

You have asymmetric routing. That's not an issue for ping as icmp is stateless. However rdp uses tcp and requires a 3-way handshake to establish a connection.

Your incoming traffic from PC2 will hit the static route on the core switch and go into the ASA via VLAN 15.

The return traffic will see that the ASA has a direct connection to VLAN 500 (192.168.50.0/24) via the trunk into Eth0/0 and the VLAN interface that you have defined. A direct connection installs a conected route (administrative distance = 0) in the ASA's routing table and thus VLAN 500 be selected as the path for the egress traffic.

However, the stateful firewall is waiting for the 3-way handshake to complete via return traffic going out the same interface that it arrived on (VLAN 15). If you shutdown the VLAN 500 interface, the situation will be remedied and that flow should work for you.

Alternatively, you could make the ASA's VLAN 500 interfaces the default gateway for hosts on that subnet (but that would probably break other internal communications between your VLANs unless you removed the inbound ACLs and added "same-security-traffic permit inter-interface").

[Aaron.Pittenger]

Marvin,

Thanks so much for the reply (I have read many of your replies on the forum and you are always super helpful.)

I deleted those interfaces and have changed the port mode from trunk to access on the ASA (and assigned it as access VLAN 15).  I did the same for the other end of the connection, on the inside L3 switch.  

I assumed that I would need to add a static route to the ASA in order for the 10.11.1.0 traffic to 'know' where to go (via VLAN 15.)  I first created a static route for Interface: DMZ, routing 192.168.50.0 to a gateway of 192.168.15.220.  This didn't work, even for icmp ping - however, when I changed the interface to my Inside VLAN 15, icmp began working again.

So question 1) Why am I having to make this route for the inside interface instead of the DMZ?  Am I thinking about something backwards?

Question 2)  I am still not able to use any TCP protocols, just ICMP.  Did I miss something that I should have done along with the above changes?

Thanks for the help,

Aaron

[Aaron.Pittenger]

Update:

Marvin,

It seems I have answered my own question #2.  (Still a bit confused on #1).  For some reason RDP was trying to initiate the connection from a random port (not sure why.)  When I tried ssh through it was successful.  So, in short, the routing changes you proposed definitely fixed the problem and now I just need to sort out why RDP is being funny.

Thank you for the help, it is much appreciated!

Aaron

[Marvin Rhoads]

Glad to see the update.

TCP connections and UDP flows work that way - they use ephemeral ports as the source (i.e. something >1024 for most Windows PCs) and the destination port is the well-known port for the protocol.

https://en.wikipedia.org/wiki/Ephemeral_port

[Aaron.Pittenger]

Marvin,

I guess I never knew that!  Learn something new every day.  What's the typical methodology for dealing with that then?  I really don't want to leave all those ports open between my DMZ and my inside - is there a workaround, or do most people just open all of those?

Thanks,

Aaron

[Marvin Rhoads]

A stateful firewall (such as the ASA) uses a 5-tuple in the ACL. (not including newer things like SGTs). The protocol, source and destination addresses and ports are part of a given access control list entry.

If you allow the lower security DMZ hosts to initiate RDP to the higher security Inside hosts, that's generally all you need. The return traffic is allowed automatically as it is then part of an allowed and established TCP connection (i.e. it has "state"). It bypasses any input ACL on the inside interface since the first thing checked (even before the interface ACL or security level) is whether or not there's an existing connection.

http://www.cisco.com/c/en/us/support/docs/security/asa-5500-x-series-next-generation-firewalls/113396-asa-packet-flow-00.html

[Aaron.Pittenger]

Marvin,
Thank you for sticking with me here - I'm learning.

I think I understand what you are saying - I actually did intended to tighten up the ACL once I got things working.  So right now, here's a screenshot of what I've got.

When you say I should 'allow the lower security DMZ hosts to initiate RDP to the higher security Inside hosts' what exactly do you mean?  How do I allow that through without allowing giant range of ports (since I don't know exactly which port on which it will initiate RDP.)  I've allowed 3389 but, since RDP is using an ephemeral port to initiate the connection, it is getting dropped.)  It seems in the example link you posted, a similar thing is happening, but I'm not following how they are making it work.

Thank you again,

Aaron

[Marvin Rhoads]

The ACL in your screenshot appears correct. There's a areason why Service only appears once in the ACL - the destination port is the service and the source port will vary or each separate flow. That's the nature of TCP/IP.

Your DMZ is security-level 50. InsideVLAN15 is security-level 100. You can only initiate communications from a lower security level (50) to a higher (100) when explicitly allowed by an ACL. When initiating from higher to lower, it is allowed by defualt - UNLESS there is any kind of ACL in which case you then need to explicitly define what is allowed.

When in doubt about your logic or if/why an ASA is dropping a certain flow, use the packet-tracer command (or graphical version in ASDM). If will walk you through the ASA logic step-by-step and give the exact disposition of a given flow.

[Marvin Rhoads]

You're welcome Aaron.

ASA routes are egress routes. That's why the route statement needs to specifiy the InsideVLAN15 interface.

Since you've made a few changes, it would be a good time to test the logic of your flow with packet-tracer. Try running this command:

packet-tracer input DMZ tcp 10.11.1.10 1025 192.168.50.14 3389

..and please share the resultant output.