Firepower 1010, Cisco Anyconnect VPN, RDP has stopped working

rzmzr · ‎06-14-2021

Hi,

I'm in need of assitance.
The issue is as follows:
Last week, friday, everything was worknig fine for everyone. Come monday, and some users can't reach interal resources, as well as can't connect to their workstations through Remote Desktop Connection from their home computers.
Users can connect through AnyConnect application.

No changes have been made, this sudden issue has caught me off-guard.

I can ping VPN connected computers from internal network.
I can ping internal network computers from VPN connected device.

The tunnel statistics show traffic (Sent/Received)
I can also access shared folders from VPN connected device (which is strange, because other users can't)

What I can't do (and which is most important in all this) is, I can't establish an RDP to internal computers. After attempting to RDP, the following error is received:

"This computer can't connect to the remote computer.

Try connecting again. If the problem continues, contact the owner of the remote computer or your network administrator"

We don't have a network guy, and my knowledge is also quite limited regarding networks - Any help would be appreciated.

Thanks!

Rob Ingram · ‎06-14-2021

@rzmzr

If no changes have been made to the firewall, perhaps the issue lies with the laptop or the destination server. Is there a local firewall on either the laptop or the rdp server that could be blocking communication from the Remote Access VPN IP address pool? Check the logs on the server to see if there is even an attempt at communication.

rzmzr · ‎06-14-2021

I've checked the logs on a computer that I am attempting to connect to from my test laptop, and indeed it does not record any attempt at connection.

Where I checked:
Event Viewer -> App and Service Logs -> MS -> TerminalServices-RemoteConectionManager -> Operational

If I attempt the connection internally (from internal network PC, it shows up in the log)

We have Firewall enabled on computers with GPO, yes.
Remote Desktop is allowed.

I also tried checking if maybe, somehow, the RDP port is blocked.
I've launched a powershell command: tnc *destination PC IP* -port 3389 from my test laptop that is connected to VPN (using external network) and the test succeeded.

It starts to seem that the VPN tunnel is not allowing RDP through, questoin is, why?

Rob Ingram · ‎06-14-2021

Run packet-tracer from the CLI of the FTD and provide the full output, example:-

packet-tracer input outside tcp <a free ravpn ip address> 3000 <rdp server ip> 3389

rzmzr · ‎06-14-2021

This is what I got (source is the test laptop with enabled VPN and destination is a computer I'd like to RDP to)

Phase: 1
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
Additional Information:
MAC Access list

Phase: 2
Type: ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
Additional Information:
found next-hop 192.168.10.7 using egress ifc inside

Phase: 3
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
nat (inside,outside) source static NGFW-Remote-Access-VPN|natIpv4Grp NGFW-Remote-Access-VPN|natIpv4Grp destination static NGFW-Remote-Access-VPN|natIpv4PoolGrp NGFW-Remote-Access-VPN|natIpv4PoolGrp no-proxy-arp route-lookup
Additional Information:
NAT divert to egress interface inside
Untranslate 192.168.10.7/3389 to 192.168.10.7/3389

Phase: 4
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group NGFW_ONBOX_ACL global
access-list NGFW_ONBOX_ACL advanced permit object-group |acSvcg-268435463 ifc outside object VPN ifc inside object Kalps7 rule-id 268435463 event-log both
access-list NGFW_ONBOX_ACL remark rule-id 268435463: ACCESS POLICY: NGFW_Access_Policy
access-list NGFW_ONBOX_ACL remark rule-id 268435463: L7 RULE: VPN-AD
object-group service |acSvcg-268435463
service-object ip
Additional Information:
This packet will be sent to snort for additional processing where a verdict will be reached

Phase: 5
Type: NAT
Subtype:
Result: ALLOW
Config:
nat (inside,outside) source static NGFW-Remote-Access-VPN|natIpv4Grp NGFW-Remote-Access-VPN|natIpv4Grp destination static NGFW-Remote-Access-VPN|natIpv4PoolGrp NGFW-Remote-Access-VPN|natIpv4PoolGrp no-proxy-arp route-lookup
Additional Information:
Static translate 192.168.11.13/3000 to 192.168.11.13/3000

Phase: 6
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 7
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 8
Type: CP-PUNT
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 9
Type: WEBVPN-SVC
Subtype: in
Result: DROP
Config:
Additional Information:

Result:
input-interface: outside
input-status: up
input-line-status: up
output-interface: inside
output-status: up
output-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule, Drop-location: frame 0x0000556d720add37 flow (svc-spoof-detect)/snp_sp_action_cb:1802

Rob Ingram · ‎06-14-2021

Were you connected to the VPN when you run that packet-tracer? You need to use a source IP address that is not currently in use.

Provide the output of "show nat detail" as well.

rzmzr · ‎06-14-2021

Yes, the VPN was connected when I ran previous packet-tracer.

This should be from a unused IP:

Phase: 1
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
Additional Information:
MAC Access list

Phase: 2
Type: ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
Additional Information:
found next-hop 192.168.10.7 using egress ifc inside

Phase: 3
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
nat (inside,outside) source static NGFW-Remote-Access-VPN|natIpv4Grp NGFW-Remote-Access-VPN|natIpv4Grp destination static NGFW-Remote-Access-VPN|natIpv4PoolGrp NGFW-Remote-Access-VPN|natIpv4PoolGrp no-proxy-arp route-lookup
Additional Information:
NAT divert to egress interface inside
Untranslate 192.168.10.7/3389 to 192.168.10.7/3389

Phase: 4
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group NGFW_ONBOX_ACL global
access-list NGFW_ONBOX_ACL advanced permit object-group |acSvcg-268435463 ifc outside object VPN ifc inside object Kalps7 rule-id 268435463 event-log both
access-list NGFW_ONBOX_ACL remark rule-id 268435463: ACCESS POLICY: NGFW_Access_Policy
access-list NGFW_ONBOX_ACL remark rule-id 268435463: L7 RULE: VPN-AD
object-group service |acSvcg-268435463
service-object ip
Additional Information:
This packet will be sent to snort for additional processing where a verdict will be reached

Phase: 5
Type: NAT
Subtype:
Result: ALLOW
Config:
nat (inside,outside) source static NGFW-Remote-Access-VPN|natIpv4Grp NGFW-Remote-Access-VPN|natIpv4Grp destination static NGFW-Remote-Access-VPN|natIpv4PoolGrp NGFW-Remote-Access-VPN|natIpv4PoolGrp no-proxy-arp route-lookup
Additional Information:
Static translate 192.168.11.140/3000 to 192.168.11.140/3000

Phase: 6
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 7
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 8
Type: NAT
Subtype: rpf-check
Result: ALLOW
Config:
nat (inside,outside) source static NGFW-Remote-Access-VPN|natIpv4Grp NGFW-Remote-Access-VPN|natIpv4Grp destination static NGFW-Remote-Access-VPN|natIpv4PoolGrp NGFW-Remote-Access-VPN|natIpv4PoolGrp no-proxy-arp route-lookup
Additional Information:

Phase: 9
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 10
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 11
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 76914, packet dispatched to next module

Phase: 12
Type: EXTERNAL-INSPECT
Subtype:
Result: ALLOW
Config:
Additional Information:
Application: 'SNORT Inspect'

Phase: 13
Type: SNORT
Subtype:
Result: ALLOW
Config:
Additional Information:
Snort Trace:
Firewall: starting rule matching, zone 1 -> 2, geo 0 -> 0, vlan 0, sgt 0, src sgt type 0, dest_sgt_tag 0, dest sgt type 0, user 9999999, icmpType 184, icmpCode 61
Packet: TCP, SYN, seq 1561657734
Session: new snort session
AppID: service unknown (0), application unknown (0)
Firewall: starting rule matching, zone 1 -> 2, geo 0 -> 0, vlan 0, sgt 0, src sgt type 0, dest_sgt_tag 0, dest sgt type 0, user 9999999, icmpType 0, icmpCode 0
Firewall: pending rule-matching, id 268435463, pending AppID
Snort id 1, NAP id 1, IPS id 0, Verdict PASS
Snort Verdict: (pass-packet) allow this packet

Phase: 14
Type: ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
Additional Information:
found next-hop 192.168.10.7 using egress ifc inside

Phase: 15
Type: ADJACENCY-LOOKUP
Subtype: next-hop and adjacency
Result: ALLOW
Config:
Additional Information:
adjacency Active
next-hop mac address b026.28bd.4c04 hits 4805 reference 133

Result:
input-interface: outside
input-status: up
input-line-status: up
output-interface: inside
output-status: up
output-line-status: up
Action: allow

This is "show nat detail"

Manual NAT Policies (Section 1)
1 (inside) to (outside) source static NGFW-Remote-Access-VPN|natIpv4Grp NGFW-Remote-Access-VPN|natIpv4Grp destination static NGFW-Remote-Access-VPN|natIpv4PoolGrp NGFW-Remote-Access-VPN|natIpv4PoolGrp no-proxy-arp route-lookup
translate_hits = 12564, untranslate_hits = 13597
Source - Origin: 192.168.10.0/24, Translated: 192.168.10.0/24
Destination - Origin: 192.168.11.0/24, Translated: 192.168.11.0/24
2 (inside) to (outside) source static ArsicoServer interface service _|NatOrigSvc_0df0679f-0ee5-11ea-bccd-31d6102cf36a _|NatMappedSvc_0df0679f-0ee5-11ea-bccd-31d6102cf36a
translate_hits = 0, untranslate_hits = 0
Source - Origin: 192.168.10.5/32, Translated: 195.13.194.66/28
Service - Origin: tcp source eq 3389 , Translated: tcp source eq 9753
3 (inside) to (outside) source static ArsicoServer interface service _|NatOrigSvc_0d5eb9a3-0ee6-11ea-bccd-797ea53f64f9 _|NatMappedSvc_0d5eb9a3-0ee6-11ea-bccd-797ea53f64f9
translate_hits = 0, untranslate_hits = 0
Source - Origin: 192.168.10.5/32, Translated: 195.13.194.66/28
Service - Origin: tcp source eq 1036 , Translated: tcp source eq 1036
4 (inside) to (outside) source static Kalps3 interface service _|NatOrigSvc_4d968b71-0ee6-11ea-bccd-8160aa4f670e _|NatMappedSvc_4d968b71-0ee6-11ea-bccd-8160aa4f670e
translate_hits = 0, untranslate_hits = 0
Source - Origin: 192.168.10.3/32, Translated: 195.13.194.66/28
Service - Origin: tcp source eq 3389 , Translated: tcp source eq 3333

Auto NAT Policies (Section 2)
1 (nlp_int_tap) to (inside) source static nlp_server_0_http_intf2 interface service tcp https https
translate_hits = 0, untranslate_hits = 0
Source - Origin: 169.254.1.3/32, Translated: 192.168.10.2/24
Service - Protocol: tcp Real: https Mapped: https
2 (nlp_int_tap) to (inside) source static nlp_server_0_ssh_intf2 interface service tcp ssh ssh
translate_hits = 0, untranslate_hits = 0
Source - Origin: 169.254.1.3/32, Translated: 192.168.10.2/24
Service - Protocol: tcp Real: ssh Mapped: ssh
3 (inside) to (outside) source static Noliktava Public-Noliktava dns
translate_hits = 0, untranslate_hits = 918
Source - Origin: 192.168.10.33/32, Translated: 195.13.194.68/32
4 (inside) to (outside) source static MFiles Public-MFiles dns
translate_hits = 0, untranslate_hits = 1653
Source - Origin: 192.168.10.34/32, Translated: 195.13.194.67/32
5 (nlp_int_tap) to (inside) source dynamic nlp_client_0_intf2 interface
translate_hits = 0, untranslate_hits = 0
Source - Origin: 169.254.1.3/32, Translated: 192.168.10.2/24
6 (nlp_int_tap) to (outside) source dynamic nlp_client_0_intf3 interface
translate_hits = 0, untranslate_hits = 0
Source - Origin: 169.254.1.3/32, Translated: 195.13.194.66/28
7 (nlp_int_tap) to (diagnostic) source dynamic nlp_client_0_intf4 interface
translate_hits = 0, untranslate_hits = 0
Source - Origin: 169.254.1.3/32, Translated: 0.0.0.0/32
8 (inside) to (outside) source dynamic LAN interface
translate_hits = 58841, untranslate_hits = 133
Source - Origin: 192.168.10.0/24, Translated: 195.13.194.66/28
9 (nlp_int_tap) to (inside) source dynamic nlp_client_0_ipv6_intf2 interface ipv6
translate_hits = 0, untranslate_hits = 0
Source - Origin: fd00:0:0:1::3/128, Translated:
10 (nlp_int_tap) to (outside) source dynamic nlp_client_0_ipv6_intf3 interface ipv6
translate_hits = 0, untranslate_hits = 0
Source - Origin: fd00:0:0:1::3/128, Translated:
11 (nlp_int_tap) to (diagnostic) source dynamic nlp_client_0_ipv6_intf4 interface ipv6
translate_hits = 0, untranslate_hits = 0
Source - Origin: fd00:0:0:1::3/128, Translated:

Rob Ingram · ‎06-14-2021

The outcome of that simulated packet flow is "allow", I suggest double checking your local firewall configuration. Perhaps disable the firewall for testing and try again.

rzmzr · ‎06-14-2021

Pardon the question, but is there a difference in firewalls (rules that are applied) when the comptuer is connected through a VPN? It uses the domain profile still, no?

The two use cases experience the same problem:

- Personal computers with anyconnect and cert installed remoting to work computers

- Company issued computers, that are part of the company domain, being used in external network through a VPN tunnel

My understanding is that - if firewall was at fault, it would be either scenario, but not both? (Train of thought being, personal computers have no idea about the domain firewall policies?)
Besides, no settings have been changed or adjusted recently in the Firewall - I refuse to believe that something like this just randomly happens (Yes, I am getting frustrated, I am sorry)

Users in office (internal network) have no problem to RDP.

Another issue that's cropped up (or someone noticed it just now) is we have a software licensing service running on one of the servers, and users running the software from home (on Company issued devices - through the VPN) can't activate the software, meaning, it's not just RDP traffic that is dissapearing in thin air.

We had experienced a power-loss during the weekend, and I do understand that it might cause some issues for all of this.

Thank you very much for the help so far, Rob, really appreciate it.