CIsco ASA Firewall NAT services not working

vitumbiko nkhwazi · ‎06-14-2023

Hello.

We have a Cisco ASA firewall that is running ASA version14(4)23. This firewall is meant to replace our current firewall, we have done all the configurations including NAT and Access lists, when we connect the ASA into production, all outgoing traffic is working properly (We can browse the Internet), however, on the incoming services that we have NATed to internal private address, only one service is working and the rest are not able to connect. Our NAT and Access-list configurations look to be good but there is something preventing these services from connecting, we have tried upgrading the firmware but it did not solve the problem. What could be the issue and how can we troubleshoot why the incoming NATed services are not woking?

Regards.

MHM Cisco World · ‎06-14-2023

Which service we talking about?

Do add these services to inspection of asa?

So you use real ip or mapped ip in acl?

vitumbiko nkhwazi · ‎06-14-2023

Which service we talking about? the application that is working is running HTTPS on port 443, the others that are not working use TCP ports 6443, 8443, 9010, 8080

Do add these services to inspection of asa? - They are not added to inspection

So you use real ip or mapped ip in acl? - the ACL on the Outside Interface is using real (private) IPs

MHM Cisco World · ‎06-14-2023

We need to add it to inspection

If there is no such protocol then we need to bypass this protocol from asa global inspection.

vitumbiko nkhwazi · ‎06-14-2023

@MHM Cisco World do you suggest we add the protocols to the global_policy using class inspection_default?

MHM Cisco World · ‎06-14-2023

Yes and hope global inspection have these protocol

vitumbiko nkhwazi · ‎06-14-2023

@MHM Cisco World global inspection does not allow custom protocols, and these are not options in inspection.

MHM Cisco World · ‎06-14-2023

can you share the ACL/NAT config of ASA
thanks
MHM

vitumbiko nkhwazi · ‎06-14-2023

@MHM Cisco World see the config below

NBS-BT-INTERNET-ASA5525# sh run nat
nat (INSIDE,OUTSIDE) source static AlienVault 102.33.155.11 destination static ALIENVAULT_DESTINATIONS ALIENVAULT_DESTINATIONS service HTTPS HTTPS
nat (INSIDE,OUTSIDE) source static NETGUARDIANS 102.33.155.15 destination static NETGUARDIANS_ACCESS NETGUARDIANS_ACCESS service HTTPS HTTPS
nat (INSIDE,OUTSIDE) source static Internet_banking INTERNET_BANKING_GLOBAL service HTTPS HTTPS
nat (INSIDE,OUTSIDE) source static Mobile_banking SMARTAPP_GLOBAL service 6443 6443
nat (INSIDE,OUTSIDE) source static obj-10.40.130.102 obj-102.33.155.11 destination static BITCRACK_SOURCE_IPs BITCRACK_SOURCE_IPs service HTTPS HTTPS
nat (INSIDE,OUTSIDE) source static obj-10.40.130.68 obj-10.0.21.254 destination static SWIFT_SUBNET SWIFT_SUBNET
nat (INSIDE,OUTSIDE) source static obj-10.40.130.57-NI-LIVE obj-10.40.200.100-NI-TRANSLATED destination static NI-TO-NBS NI-TO-NBS
nat (INSIDE,OUTSIDE) source static obj-10.40.1.63-NI-TEST obj-10.40.200.201-NI-TEST-TRANSLATED destination static NI-TO-NBS NI-TO-NBS
nat (INSIDE,OUTSIDE) source static Internet_banking obj-10.40.200.102 destination static NI-TO-NBS NI-TO-NBS
nat (INSIDE,OUTSIDE) source static obj-10.40.1.201 obj-10.40.200.202 destination static NI-TO-NBS NI-TO-NBS
nat (INSIDE,OUTSIDE) source static obj-10.40.129.202 obj-10.40.200.203 destination static NI-TO-NBS NI-TO-NBS
nat (INSIDE,OUTSIDE) source dynamic NBS_INTERNAL obj-10.0.21.243 destination static SWIFT_SUBNET SWIFT_SUBNET
nat (INSIDE,OUTSIDE) source static REMOTE_ACCESS_IPs REMOTE_ACCESS_IPs destination static REMOTE_ACCESS_POOL REMOTE_ACCESS_POOL
nat (INSIDE,OUTSIDE) source dynamic NBS_INTERNAL obj-10.40.200.101 destination static NI-TO-NBS NI-TO-NBS
nat (INSIDE,OUTSIDE) source static NBS_INTERNAL NBS_INTERNAL destination static NETWORK_OBJ_10.51.200.0_24 NETWORK_OBJ_10.51.200.0_24 no-proxy-arp route-lookup
nat (INSIDE,OUTSIDE) source static AMEYO_SERVER 102.33.155.13 service tcp_8443 tcp_8443
nat (INSIDE,OUTSIDE) source static AMEYO_SERVER 102.33.155.13 service tcp_8080 tcp_8080
nat (INSIDE,OUTSIDE) source static AMEYO_SERVER 102.33.155.13 service tcp_7412 tcp_7412
nat (INSIDE,OUTSIDE) source static obj-10.40.1.153 SMARTAPP_GLOBAL service HTTPS HTTPS
nat (INSIDE,OUTSIDE) source static obj-10.40.129.94 obj-102.33.155.11 service tcp_81 tcp_81
nat (INSIDE,OUTSIDE) source static obj-10.40.129.138 obj-102.33.155.143 service tcp_81 tcp_81
nat (INSIDE,OUTSIDE) source static obj-10.40.129.138 obj-102.33.155.143 service tcp_83 tcp_83
nat (INSIDE,OUTSIDE) source static obj-10.40.129.138 obj-102.33.155.143 service tcp_86 tcp_86
nat (INSIDE,OUTSIDE) source static obj-10.40.129.205 obj-102.33.155.12 service tcp_81 tcp_81
nat (INSIDE,OUTSIDE) source static obj-10.40.129.205 obj-102.33.155.12 service tcp_84 tcp_84
nat (INSIDE,OUTSIDE) source static obj-10.40.129.205 obj-102.33.155.12 service tcp_85 tcp_85
nat (INSIDE,OUTSIDE) source static obj-10.40.129.205 obj-102.33.155.12 service tcp_86 tcp_86
nat (INSIDE,OUTSIDE) source static obj-10.40.129.205 obj-102.33.155.12 service tcp_88 tcp_88
nat (INSIDE,OUTSIDE) source static obj-10.40.129.205 obj-102.33.155.12 service tcp_90 tcp_90
nat (INSIDE,OUTSIDE) source static obj-10.40.129.205 obj-102.33.155.12 service tcp_91 tcp_91
nat (INSIDE,OUTSIDE) source static obj-10.40.129.202 obj-102.33.155.10
nat (INSIDE,OUTSIDE) source static obj-10.40.129.152 obj-102.33.155.9
!
object network obj-192.168.111.0
nat (INSIDE,OUTSIDE) dynamic interface
object network obj-192.168.222.0
nat (INSIDE,OUTSIDE) dynamic interface
!
nat (INSIDE,OUTSIDE) after-auto source dynamic NBS_INTERNAL interface
nat (INSIDE,OUTSIDE) after-auto source dynamic NBS_OLD_SUBNET interface

access-list OUTSIDE remark MANAGEMENT TRAFFIC
access-list OUTSIDE extended permit udp 102.33.155.0 255.255.255.0 object NMS object-group DM_INLINE_UDP_1
access-list OUTSIDE remark BitCrack Access
access-list OUTSIDE extended permit tcp object-group BITCRACK_SOURCE_IPs object obj-10.40.130.102 eq https
access-list OUTSIDE remark AML SFTP Access
access-list OUTSIDE extended permit tcp object-group DM_INLINE_NETWORK_1 object-group DM_INLINE_NETWORK_2 eq ssh
access-list OUTSIDE remark NI TO NBS TEST - Port 9010
access-list OUTSIDE extended permit tcp object-group NI-TO-NBS object NBS_NI_TEST eq 9010
access-list OUTSIDE remark NI TO NBS LIVE - Port 9010
access-list OUTSIDE extended permit tcp object-group NI-TO-NBS object obj-10.40.130.57-NI-LIVE eq 9010
access-list OUTSIDE extended permit tcp object-group NI-TO-NBS object obj-10.40.1.63-NI-TEST eq 10024
access-list OUTSIDE extended permit tcp object-group NI-TO-NBS object obj-10.40.1.63-NI-TEST eq 10030
access-list OUTSIDE extended permit tcp any host 102.33.155.93 eq 81
access-list OUTSIDE extended permit tcp any object INTERNET_BANKING_GLOBAL eq https
access-list OUTSIDE remark Internet Banking
access-list OUTSIDE extended permit tcp any object Internet_banking eq https
access-list OUTSIDE remark INTERNET BANKING
access-list OUTSIDE extended permit tcp any object obj-10.40.129.138_81 eq 81
access-list OUTSIDE extended permit tcp any object obj-10.40.129.138_86 eq 86
access-list OUTSIDE extended permit tcp any object obj-10.40.129.138_83 eq 83
access-list OUTSIDE remark Mobile Banking
access-list OUTSIDE extended permit tcp any object Mobile_banking eq 6443
access-list OUTSIDE remark FTP DATA TO NETMONITOR FROM NBS INTERNET HOSTS
access-list OUTSIDE extended permit tcp object NBS_INT_PUBLIC object NET_MONITOR object-group DM_INLINE_TCP_1
access-list OUTSIDE remark Netflow, SNMP, FTP, and SYSLOG
access-list OUTSIDE extended permit object-group DM_INLINE_SERVICE_1 object NBS_INT_PUBLIC object-group NET_MGNT_STATIONS
access-list OUTSIDE extended permit tcp any object obj-10.40.129.202 eq 6443
access-list OUTSIDE extended permit tcp any object DEV_TEST object-group DM_INLINE_TCP_3
access-list OUTSIDE extended permit tcp any object obj-10.40.129.94 eq 81
access-list OUTSIDE remark PAYDAY LOAN APPLICATION
access-list OUTSIDE extended permit tcp any host 10.40.129.205 object-group INSTANT_LOANS
access-list OUTSIDE remark AMEYO CALL CENTER
access-list OUTSIDE extended permit object-group DM_INLINE_SERVICE_2 any object AMEYO_SERVER
access-list OUTSIDE extended permit object HTTPS object-group DM_INLINE_NETWORK_5 object AlienVault
access-list OUTSIDE remark NETGUARDIANS
access-list OUTSIDE extended permit object HTTPS object-group NETGUARDIANS_ACCESS object NETGUARDIANS

MHM Cisco World · ‎06-14-2023

packet tracer input OUTSIDE tcp <select any IP from the outside subnet> 1234 <mapped IP of server> 8080 detail
packet tracer input OUTSIDE tcp <select any IP from the outside subnet> 1234 <mapped IP of server> 8443 detail
packet tracer input OUTSIDE tcp <select any IP from the outside subnet> 1234 <mapped IP of server> 7412 detail

please share the output are the packet drop in NAT rpf-check

vitumbiko nkhwazi · ‎06-15-2023

See below the packet tracer output, there is no drop in NAT rpf-check

NBS-BT-INTERNET-ASA5525# packet-tracer input OUTSIDE tcp 82.54.45.8 1234 102.36.145.13 8080 detail

Phase: 1
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
nat (INSIDE,OUTSIDE) source static AMEYO_SERVER 102.36.145.13 service tcp_8080 tcp_8080
Additional Information:
NAT divert to egress interface INSIDE
Untranslate 102.36.145.13/8080 to 10.40.129.50/8080

Phase: 2
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group OUTSIDE in interface OUTSIDE
access-list OUTSIDE remark AMEYO CALL CENTER
access-list OUTSIDE extended permit tcp any object AMEYO_SERVER object-group AMEYO_PORTS
object-group service AMEYO_PORTS tcp
port-object eq 7412
port-object eq 8443
port-object eq 8080
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f5cdc58d2b0, priority=13, domain=permit, deny=false
hits=0, user_data=0x7f5cd13b2a40, cs_id=0x0, use_real_addr, flags=0x0, protocol=6
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=10.40.129.50, mask=255.255.255.255, port=8080, tag=any, dscp=0x0
input_ifc=OUTSIDE, output_ifc=any

Phase: 3
Type: NAT
Subtype:
Result: ALLOW
Config:
nat (INSIDE,OUTSIDE) source static AMEYO_SERVER 102.36.145.13 service tcp_8080 tcp_8080
Additional Information:
Static translate 82.54.45.8/1234 to 82.54.45.8/1234
Forward Flow based lookup yields rule:
in id=0x7f5cdc672780, priority=6, domain=nat, deny=false
hits=0, user_data=0x7f5cd6009280, cs_id=0x0, flags=0x0, protocol=6
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=102.36.145.13, mask=255.255.255.255, port=8080, tag=any, dscp=0x0
input_ifc=OUTSIDE, output_ifc=INSIDE

Phase: 4
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f5cdb5891a0, priority=0, domain=nat-per-session, deny=false
hits=93539, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=6
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=any

Phase: 5
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f5cdc439a80, priority=0, domain=inspect-ip-options, deny=true
hits=69558, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=OUTSIDE, output_ifc=any

Phase: 6
Type: VPN
Subtype: ipsec-tunnel-flow
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f5cdeccc400, priority=13, domain=ipsec-tunnel-flow, deny=true
hits=9693, user_data=0x0, cs_id=0x0, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=OUTSIDE, output_ifc=any

Phase: 7
Type: NAT
Subtype: rpf-check
Result: ALLOW
Config:
nat (INSIDE,OUTSIDE) source static AMEYO_SERVER 102.36.145.13 service tcp_8080 tcp_8080
Additional Information:
Forward Flow based lookup yields rule:
out id=0x7f5cdc672b60, priority=6, domain=nat-reverse, deny=false
hits=1, user_data=0x7f5cdc662e60, cs_id=0x0, use_real_addr, flags=0x0, protocol=6
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=10.40.129.50, mask=255.255.255.255, port=8080, tag=any, dscp=0x0
input_ifc=OUTSIDE, output_ifc=INSIDE

Phase: 8
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
in id=0x7f5cdb5891a0, priority=0, domain=nat-per-session, deny=false
hits=93541, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=6
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=any

Phase: 9
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
in id=0x7f5cdc3d1ba0, priority=0, domain=inspect-ip-options, deny=true
hits=65726, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=INSIDE, output_ifc=any

Phase: 10
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 73343, packet dispatched to next module
Module information for forward flow ...
snp_fp_inspect_ip_options
snp_fp_tcp_normalizer
snp_fp_translate
snp_fp_adjacency
snp_fp_fragment
snp_fp_tracer_drop
snp_ifc_stat

Module information for reverse flow ...
snp_fp_inspect_ip_options
snp_fp_translate
snp_fp_tcp_normalizer
snp_fp_adjacency
snp_fp_fragment
snp_fp_tracer_drop
snp_ifc_stat

Phase: 11
Type: INPUT-ROUTE-LOOKUP-FROM-OUTPUT-ROUTE-LOOKUP
Subtype: Resolve Preferred Egress interface
Result: ALLOW
Config:
Additional Information:
Found next-hop 10.40.139.81 using egress ifc INSIDE

Phase: 12
Type: ADJACENCY-LOOKUP
Subtype: Resolve Nexthop IP address to MAC
Result: ALLOW
Config:
Additional Information:
Found adjacency entry for Next-hop 10.40.139.81 on interface INSIDE
Adjacency :Active
MAC address 0000.0c07.ac82 hits 2823 reference 2

Result:
input-interface: OUTSIDE
input-status: up
input-line-status: up
output-interface: INSIDE
output-status: up
output-line-status: up
Action: allow

MHM Cisco World · ‎06-15-2023

this phase can indicate that traffic hit the acl of s2s VPN, after you clear this do clear packet tracer again and check the result
NOTE:- check the hits of acl of s2s VPN

Phase: 6
Type: VPN
Subtype: ipsec-tunnel-flow
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f5cdeccc400, priority=13, domain=ipsec-tunnel-flow, deny=true
hits=9693, user_data=0x0, cs_id=0x0, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=OUTSIDE, output_ifc=any

vitumbiko nkhwazi · ‎06-16-2023

hello @MHM Cisco World

Do you mean the traffic is matching a Site-to-Site VPN access-list thats why its not working?

i have checked the ACL for the VPN and there are no matches

However i we have the below crypto map configuration, is the highlighted line necessary? or could it be the one matching the traffic?

crypto map OUTSIDE_map2 1 match address OUTSIDE_cryptomap
crypto map OUTSIDE_map2 1 set pfs
crypto map OUTSIDE_map2 1 set peer 196.26.195.234
crypto map OUTSIDE_map2 1 set ikev1 transform-set Trustlink_Prod
crypto map OUTSIDE_map2 1 set security-association lifetime seconds 3600
crypto map OUTSIDE_map2 65535 ipsec-isakmp dynamic SYSTEM_DEFAULT_CRYPTO_MAP
crypto map OUTSIDE_map2 interface OUTSIDE

MHM Cisco World · ‎06-16-2023

crypto map OUTSIDE_map2 65535 ipsec-isakmp dynamic SYSTEM_DEFAULT_CRYPTO_MAP

No for IPSec vpn s2s this line not need'

Remove it do packet-tracer again and see if phase 6 appear or not.

vitumbiko nkhwazi · ‎06-16-2023

I have removed the line but packet tracer still is going through stage 6

I tried to do the same packet tracer on a firewall that is currently in production and has no issues and it is also showing the same phase.