AnyConnect split-tunnel to MPLS not working

the-lebowski · ‎06-23-2022

Have a long working RA-VPN environment with a request to route AnyConnect client traffic through the VPN and then across an MPLS connection a specific set of IPs. I have added those networks/IPs to the split-tunnel ACL, exempted them from NAT and made sure the ASA (FP running ASA code) has the correct routes. I cannot pass traffic from the RA VPN client to any of the IPs on the far side of the MPLS connection. The routes are there when connected, traceroute shows the core as the next hop and all LAN directed traffic works without issue.

I have no problem passing traffic from the inside interface of the ASA or from clients within the office, doing a packet capture shows ECHO requests being sent with no replies and traceroute shows a bunch of TCP retransmissions from AnyConnect clients.

Struggling to figure out why this isn't working. I have no access to the far side of the MPLS so I just have to assume there is a route back to my VPN clients but I don't know why there wouldn't be as its within the /16 of the office where all the traffic is able to pass. I asked the vendor to confirm but is there anything I can look at in the meantime?

What is strange is I see no no-NAT hits from the return traffic so I just assume that means nothing is coming back and the only reason that can be is there is no route back to my RA-VPN client subnet from the vendor side (#3 below)? Not even sure that is an issue because I can send traffic from the LAN to my vpn client and it works.

Manual NAT Policies (Section 1)
2 (outside-primary) to (inside) source static RA-VPN-Networks RA-VPN-Networks  destination static DM_INLINE_NETWORK_2 DM_INLINE_NETWORK_2 no-proxy-arp route-lookup
    translate_hits = 8444, untranslate_hits = 8991
3 (inside) to (outside-primary) source static DM_INLINE_NETWORK_8 DM_INLINE_NETWORK_8  destination static RA-VPN-Networks RA-VPN-Networks no-proxy-arp
    translate_hits = 0, untranslate_hits = 0
4 (outside-primary) to (outside-primary) source dynamic RA-VPN-Networks interface
    translate_hits = 12069, untranslate_hits = 3031

MHM Cisco World · ‎06-23-2022

https://www.cisco.com/c/en/us/support/docs/security/asa-5500-x-series-next-generation-firewalls/112182-ssl-tdg-config-example-00.html

check this doc.

Rob Ingram · ‎06-23-2022

@the-lebowski hard to tell, can you run packet-tracer from the CLI and provide the output for review. Example:

packet-tracert input <outside int nameif> icmp 10.92.29.245 8 0 198.105.200.146

Ensure the RAVPN IP address used in the packet-tracer is not in use by a client already connected to the VPN.

Packet-tracer will confirm which NAT rule is matched, ACL etc.

If that works and is allowed, that probably indicates the issue is not with the ASA.

NAT rules are bi-directional, rule #3 has no hits in either direction....assuming that is the correct NAT rule and traffic doesn't match a rule above?

Rob Ingram · ‎06-23-2022

@the-lebowski so the packet-tracer output result is "allow", therefore the ASA confirms it should work.

Check the upstream routing.

Are there any an upstream ACLs that could be blocking the traffic?

the-lebowski · ‎06-27-2022

@Rob Ingram No nothing upstream that I am aware of. Vendor says they are routing all RFC 1983 traffic towards us but I see nothing coming back from their side.

MHM Cisco World · ‎06-27-2022

please find below my note

MHM Cisco World · ‎06-27-2022

check my note if you have any Q please share here

the-lebowski · ‎06-27-2022

I didn't delete that post and no idea why it went away.

All the routes are there and correct as this is an already long time working environment. In theory I should have been able to add the routes via the inside interface (which I did) and no-nat those networks (which I did as well) and it still didn't work. I have a TAC case open but have yet to hear back from the engineer as to why it isn't working.

MHM Cisco World · ‎06-27-2022

Yes do try config LO and try ping ? check my note again

the-lebowski · ‎06-28-2022

It worked so that tells me there is connectivity between that network and the AnyConnect subnet but still not able to pass the traffic from the clients connected to the ASA/FP. Something is missing and I can't figure out what that is.

3850(config-if)#ip address 10.92.29.244 255.255.255.255
3850(config-if)#end
3850#ping 198.105.200.146 so loopback 99
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 198.105.200.146, timeout is 2 seconds:
Packet sent with a source address of 10.92.29.244
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 255/255/256 ms

Here is the packet-tracer output again:

fw1120# packet-tracer input outside-isp icmp 10.92.29.245 8 0 198.105$

Phase: 1
Type: INPUT-ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
Additional Information:
Found next-hop 10.92.2.1 using egress ifc  inside

Phase: 2
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
nat (outside-isp,inside) source static RA-VPN-Networks RA-VPN-Networks destination static DM_INLINE_NETWORK_2 DM_INLINE_NETWORK_2 no-proxy-arp route-lookup
Additional Information:
NAT divert to egress interface inside
Untranslate 198.105.200.146/0 to 198.105.200.146/0

Phase: 3
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group outside-isp_access_in in interface outside-isp
access-list outside-isp_access_in extended permit object-group DM_INLINE_SERVICE_2 any4 any4
object-group service DM_INLINE_SERVICE_2
 service-object icmp
 service-object icmp time-exceeded
 service-object icmp unreachable
Additional Information:

Phase: 4
Type: NAT
Subtype:
Result: ALLOW
Config:
nat (outside-isp,inside) source static RA-VPN-Networks RA-VPN-Networks destination static DM_INLINE_NETWORK_2 DM_INLINE_NETWORK_2 no-proxy-arp route-lookup
Additional Information:
Static translate 10.92.29.245/0 to 10.92.29.245/0

Phase: 5
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 6
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 7
Type: INSPECT
Subtype: np-inspect
Result: ALLOW
Config:
class-map inspection_default
 match default-inspection-traffic
policy-map global_policy
 class inspection_default
  inspect icmp
service-policy global_policy global
Additional Information:

Phase: 8
Type: INSPECT
Subtype: np-inspect
Result: ALLOW
Config:
Additional Information:

Phase: 9
Type: FLOW-EXPORT
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 10
Type: DEBUG-ICMP
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 11
Type: NAT
Subtype: rpf-check
Result: ALLOW
Config:
nat (outside-isp,inside) source static RA-VPN-Networks RA-VPN-Networks destination static DM_INLINE_NETWORK_2 DM_INLINE_NETWORK_2 no-proxy-arp route-lookup
Additional Information:

Phase: 12
Type: USER-STATISTICS
Subtype: user-statistics
Result: ALLOW
Config:
Additional Information:

Phase: 13
Type: DEBUG-ICMP
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 14
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 15
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 16
Type: USER-STATISTICS
Subtype: user-statistics
Result: ALLOW
Config:
Additional Information:

Phase: 17
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 1692093, packet dispatched to next module

Phase: 18
Type: INPUT-ROUTE-LOOKUP-FROM-OUTPUT-ROUTE-LOOKUP
Subtype: Resolve Preferred Egress interface
Result: ALLOW
Config:
Additional Information:
Found next-hop 10.92.2.1 using egress ifc  inside

Phase: 19
Type: ADJACENCY-LOOKUP
Subtype: Resolve Nexthop IP address to MAC
Result: ALLOW
Config:
Additional Information:
Found adjacency entry for Next-hop 10.92.2.1 on interface  inside
Adjacency :Active
MAC address b4a8.b909.7377 hits 146 reference 92

Result:
input-interface: outside-isp
input-status: up
input-line-status: up
output-interface: inside
output-status: up
output-line-status: up
Action: allow

MHM Cisco World · ‎06-28-2022

the LO is success that great and packet-tracer is ok, NAT & ACL is prefect.
there is command called capture in and capture out you can apply it to ASA to see if packet is ingress and egress from IN/OUT of ASA.

the router (CE) connect to MPLS PE with OSPF or BGP ?? are you redistribute connect or redistribute static ??

the-lebowski · ‎06-28-2022

I have to remove the routes between testing because it breaks connectivity for users when connected to the VPN. Its dynamic (EIGRP) between us and the vendor then static between the ASA/FP and our core. I will add the routes back and do a packet capture in/out and report back. Might just take me a day or so to get it done.

MHM Cisco World · ‎06-28-2022

you run EIGRP and in CE router you config static route of VPN POOL subnet toward INSIDE of ASA,
then you must redistribute this subnet into the EIGRP to make CE advertise the prefix to PE and PE will add MPLS LABEL to this prefix.

Without the MPLS LABEL there is blackhole.

the-lebowski · ‎06-29-2022

CE router is configured as 'stub connected' which shouldn't matter because the vendor says they are routing all RFC 1983 subnets towards me so why would I need to advertise the RA-VPN subnet if its within 10.0.0.0/8? I asked them for a copy of their routing table so I can see exactly what they are learning/pointing to us and will follow up.

router eigrp 9999
 network 10.0.0.0
 eigrp stub connected

MHM Cisco World · ‎06-29-2022

all RFC 1983 is forward to this router, this is prefect.

so fro CE to vendor (after add LO is success ) and vendor have return back toward CE.

so still last part

add these static route and check ping now.

ASA

ip route 198.105.200.146/32 toward Inside

CE
ip route 10.92.29.0/24 toward asa