Communication behind NAT devices Issue

[milks126]

Hello all,

I created a simple lab and encountered an issue. Could you tell me what I'm missing?

I simply want that PC1 and PC2 can ping each other. To make it work, I configured the route, the access-list, and the access-group. Also, I enabled "inspect icmp" under the global policy so that ping works among the network. As you can see the diagram that I attached. P1 and P2 are behind the ASAs. PC1 can ping the outside interface of ASA2, but PC2. PC2 is the same. PC2 can ping the outside interface of ASA1, but PC1. Why is it happening?

ASA1 config:

interface GigabitEthernet0/0
nameif outside
security-level 0
ip address 100.1.1.1 255.255.255.0
!
interface GigabitEthernet0/1
nameif inside
security-level 100
ip address 10.1.1.1 255.255.255.0
!
same-security-traffic permit inter-interface
same-security-traffic permit intra-interface
object network lan
subnet 10.1.1.0 255.255.255.0
access-list test extended permit ip any any log
!
object network lan
nat (inside,outside) dynamic interface
access-group test in interface outside
!
route outside 0.0.0.0 0.0.0.0 100.1.1.2 1
!
policy-map global_policy
class inspection_default
inspect ip-options
inspect netbios
inspect rtsp
inspect sunrpc
inspect tftp
inspect dns preset_dns_map
inspect ftp
inspect h323 h225
inspect h323 ras
inspect rsh
inspect esmtp
inspect sqlnet
inspect sip
inspect skinny
inspect snmp
inspect icmp

ASA1 routing table:

Gateway of last resort is 100.1.1.2 to network 0.0.0.0

S* 0.0.0.0 0.0.0.0 [1/0] via 100.1.1.2, outside
C 10.1.1.0 255.255.255.0 is directly connected, inside
L 10.1.1.1 255.255.255.255 is directly connected, inside
C 100.1.1.0 255.255.255.0 is directly connected, outside
L 100.1.1.1 255.255.255.255 is directly connected, outside

ASA2 config:

interface GigabitEthernet0/0
nameif outside
security-level 0
ip address 200.1.1.1 255.255.255.0
!
interface GigabitEthernet0/1
nameif inside
security-level 100
ip address 172.16.1.1 255.255.255.0
!
same-security-traffic permit inter-interface
same-security-traffic permit intra-interface
object network lan
subnet 172.16.1.0 255.255.255.0
access-list test extended permit ip any any log
!
object network lan
nat (inside,outside) dynamic interface
access-group test in interface outside
route outside 0.0.0.0 0.0.0.0 200.1.1.2 1
!
policy-map global_policy
class inspection_default
inspect ip-options
inspect netbios
inspect rtsp
inspect sunrpc
inspect tftp
inspect dns preset_dns_map
inspect ftp
inspect h323 h225
inspect h323 ras
inspect rsh
inspect esmtp
inspect sqlnet
inspect sip
inspect skinny
inspect snmp
inspect icmp

ASA2 routing table:

Gateway of last resort is 200.1.1.2 to network 0.0.0.0

S* 0.0.0.0 0.0.0.0 [1/0] via 200.1.1.2, outside
C 172.16.1.0 255.255.255.0 is directly connected, inside
L 172.16.1.1 255.255.255.255 is directly connected, inside
C 200.1.1.0 255.255.255.0 is directly connected, outside
L 200.1.1.1 255.255.255.255 is directly connected, outside

R1 routing table

Gateway of last resort is not set

100.0.0.0/24 is subnetted, 1 subnets
C 100.1.1.0 is directly connected, GigabitEthernet0/0
C 200.1.1.0/24 is directly connected, GigabitEthernet1/0
172.16.0.0/24 is subnetted, 1 subnets
S 172.16.1.0 [1/0] via 200.1.1.1
8.0.0.0/32 is subnetted, 1 subnets
C 8.8.8.8 is directly connected, Loopback0
10.0.0.0/24 is subnetted, 1 subnets
S 10.1.1.0 [1/0] via 100.1.1.1

[MHM Cisco World]

Your config is correct.

Only check if PC have GW config point to inside interface of asa

MHM

[milks126]

@MHM Cisco World, that's what i wanted to hear. GW for PC1 and PC2 has been configured correctly. It could be gns3 issue. thank you so much for validating.

[MHM Cisco World]

Do ping and capture traffic in outside and inside of each ASA share result here

For capture

https://www.cisco.com/c/en/us/support/docs/security/asa-5500-x-series-next-generation-firewalls/118097-configure-asa-00.html

MHM

[milks126]

@MHM Cisco World, first of all, i applied these to both ASAs.

cap cap_inside interface inside match ip any any
cap cap_outside interface outside match ip any any

ASA1:

asa1# show cap cap_inside

5 packets captured

1: 21:58:19.384745 10.1.1.2 > 172.16.1.2 icmp: echo request
2: 21:58:21.385050 10.1.1.2 > 172.16.1.2 icmp: echo request
3: 21:58:23.386118 10.1.1.2 > 172.16.1.2 icmp: echo request
4: 21:58:25.386378 10.1.1.2 > 172.16.1.2 icmp: echo request
5: 21:58:27.386652 10.1.1.2 > 172.16.1.2 icmp: echo request

asa1# show cap cap_outside

5 packets captured

1: 21:58:19.385188 100.1.1.1 > 172.16.1.2 icmp: echo request
2: 21:58:21.385188 100.1.1.1 > 172.16.1.2 icmp: echo request
3: 21:58:23.386271 100.1.1.1 > 172.16.1.2 icmp: echo request
4: 21:58:25.386561 100.1.1.1 > 172.16.1.2 icmp: echo request
5: 21:58:27.386836 100.1.1.1 > 172.16.1.2 icmp: echo request

ASA2:

asa2# show cap cap_inside

0 packet captured

0 packet shown
asa2# show cap cap_outside

5 packets captured

1: 21:58:19.385523 100.1.1.1 > 172.16.1.2 icmp: echo request
2: 21:58:21.380488 100.1.1.1 > 172.16.1.2 icmp: echo request
3: 21:58:23.383525 100.1.1.1 > 172.16.1.2 icmp: echo request
4: 21:58:25.386637 100.1.1.1 > 172.16.1.2 icmp: echo request
5: 21:58:27.390101 100.1.1.1 > 172.16.1.2 icmp: echo request

I also checked the hit count of the ACP in ASA1 and ASA2. It's shown below:

asa1# show access-list
access-list cached ACL log flows: total 1, denied 0 (deny-flow-max 4096)
alert-interval 300
access-list test; 1 elements; name hash: 0xcb4257a3
access-list test line 1 extended permit ip any any log informational interval 300 (hitcnt=0) 0xe9e23c89

asa2# show access-list
access-list cached ACL log flows: total 1, denied 0 (deny-flow-max 4096)
alert-interval 300
access-list test; 1 elements; name hash: 0xcb4257a3
access-list test line 1 extended permit ip any any log informational interval 300 (hitcnt=5) 0xe9e23c89

Based on the findings, it looks like ASA2 drops the traffic when ICMP from PC1 hits its ASA2. However, the ACP for ASA2 shows it allowed the traffic.

[milks126]

@MHM Cisco World,

This is the output from packet-tracer.

packet-tracer input outside icmp 10.1.1.2 8 0 172.16.1.2 detailed

Phase: 1
Type: CAPTURE
Subtype:
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7fd465467d10, priority=13, domain=capture, deny=false
hits=1, user_data=0x7fd465496640, cs_id=0x0, l3_type=0x0
src mac=0000.0000.0000, mask=0000.0000.0000
dst mac=0000.0000.0000, mask=0000.0000.0000
input_ifc=outside, output_ifc=any

Phase: 2
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7fd465355ca0, priority=1, domain=permit, deny=false
hits=12, user_data=0x0, cs_id=0x0, l3_type=0x8
src mac=0000.0000.0000, mask=0000.0000.0000
dst mac=0000.0000.0000, mask=0100.0000.0000
input_ifc=outside, output_ifc=any

Phase: 3
Type: INPUT-ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
Additional Information:
Found next-hop 100.1.1.2 using egress ifc outside

Phase: 4
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group test in interface outside
access-list test extended permit ip any any log
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7fd465359050, priority=13, domain=permit, deny=false
hits=0, user_data=0x7fd451eb1d00, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=outside, output_ifc=any

Phase: 5
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7fd464feaf60, priority=0, domain=nat-per-session, deny=true
hits=50, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=any

Phase: 6
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7fd46535d210, priority=0, domain=inspect-ip-options, deny=true
hits=22, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=outside, output_ifc=any

Phase: 7
Type: INSPECT
Subtype: np-inspect
Result: ALLOW
Config:
class-map inspection_default
match default-inspection-traffic
policy-map global_policy
class inspection_default
inspect icmp
service-policy global_policy global
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7fd46544e3d0, priority=70, domain=inspect-icmp, deny=false
hits=2, user_data=0x7fd46544df00, cs_id=0x0, use_real_addr, flags=0x0, protocol=1
src ip/id=0.0.0.0, mask=0.0.0.0, icmp-type=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, icmp-code=0, tag=any, dscp=0x0
input_ifc=outside, output_ifc=any

Phase: 8
Type: QOS
Subtype:
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7fd465465e80, priority=70, domain=qos-per-class, deny=false
hits=44, user_data=0x7fd46545a7d0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=any

Phase: 9
Type: INSPECT
Subtype: np-inspect
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7fd46535ca20, priority=66, domain=inspect-icmp-error, deny=false
hits=2, user_data=0x7fd46535c6b0, cs_id=0x0, use_real_addr, flags=0x0, protocol=1
src ip/id=0.0.0.0, mask=0.0.0.0, icmp-type=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, icmp-code=0, tag=any, dscp=0x0
input_ifc=outside, output_ifc=any

Phase: 10
Type: QOS
Subtype:
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
in id=0x7fd465465e80, priority=70, domain=qos-per-class, deny=false
hits=45, user_data=0x7fd46545a7d0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=any

Phase: 11
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
in id=0x7fd464feaf60, priority=0, domain=nat-per-session, deny=true
hits=52, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=any

Phase: 12
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
in id=0x7fd46535d210, priority=0, domain=inspect-ip-options, deny=true
hits=24, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=outside, output_ifc=any

Phase: 13
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 33, packet dispatched to next module
Module information for forward flow ...
snp_fp_inspect_ip_options
snp_fp_inspect_icmp
snp_fp_adjacency
snp_fp_fragment
snp_fp_tracer_drop
snp_ifc_stat

Module information for reverse flow ...
snp_fp_inspect_ip_options
snp_fp_inspect_icmp
snp_fp_adjacency
snp_fp_fragment
snp_fp_tracer_drop
snp_ifc_stat

Phase: 14
Type: INPUT-ROUTE-LOOKUP-FROM-OUTPUT-ROUTE-LOOKUP
Subtype: Resolve Preferred Egress interface
Result: ALLOW
Config:
Additional Information:
Found next-hop 100.1.1.2 using egress ifc outside

Phase: 15
Type: ADJACENCY-LOOKUP
Subtype: Resolve Nexthop IP address to MAC
Result: ALLOW
Config:
Additional Information:
Found adjacency entry for Next-hop 100.1.1.2 on interface outside
Adjacency :Active
MAC address ca01.556d.0008 hits 21 reference 1

Result:
input-interface: outside
input-status: up
input-line-status: up
output-interface: outside
output-status: up
output-line-status: up
Action: allow

[MHM Cisco World · Accepted solution]

ok capture explain issue, 

dynamic NAT is unidirectional not like static NAT which is bidirectional.
so here both side (asa) use dynamic NAT will not work.
assume PC2 is server and PC1 client try to connect from outside what you need is config asa2 to static NATing PC2 

note:- keep other config as it

MHM 

[Jonatan Jonasson · Accepted solution]

To clarify, when you're trying to ping between the PC's, for example from PC1 to PC2, you're pinging 172.16.1.2 from PC1?

With this in mind, I can see that on both ASA's, you have a dynamic auto nat (PAT) on both ASA's.

So traffic from PC1 will be NAT'ed to 100.1.1.1 when exiting the ASA, and traffic from PC2 will be natted to 200.1.1.1.

And I don't see any other manual NAT config for any exemptions.

My initial guess is that when pinging from PC1 to PC2, on ASA2 you might see the following error:
"ASA-5-305013: Asymmetric NAT rules matched for forward and reverse flows [...]"

This doc, https://www.cisco.com/c/en/us/td/docs/security/asa/syslog/b_syslog/syslogs3.html, says this:

Explanation An attempt to connect to a mapped host using its actual address was rejected.

Recommended Action When not on the same interface as the host using NAT, use the mapped address instead of the actual address to connect to the host. In addition, enable the inspect command if the application embeds the IP address.

In your case, if you want PC1 to ping PC2 using the real IPs, you would need additional NAT statements on one or both ASAs.

[milks126]

@MHM Cisco World, thank you for explaining.I did not know about "dynamic NAT is unidirectional not like static NAT which is bidirectional"

Once I configured the following on the ASA2, PC1 can ping PC by using the real IP.

object network obj-10.1.1.2_32
host 10.1.1.2
object network obj-100.1.1.1_32
host 100.1.1.1

asa2(config)# sh run nat
nat (outside,inside) source static obj-100.1.1.1_32 obj-10.1.1.2_32

PC1> ping 172.16.1.2

84 bytes from 172.16.1.2 icmp_seq=1 ttl=63 time=12.932 ms
84 bytes from 172.16.1.2 icmp_seq=2 ttl=63 time=16.335 ms
84 bytes from 172.16.1.2 icmp_seq=3 ttl=63 time=15.578 ms
84 bytes from 172.16.1.2 icmp_seq=4 ttl=63 time=15.978 ms
84 bytes from 172.16.1.2 icmp_seq=5 ttl=63 time=15.489 ms

@Jonatan Jonasson, thank you for your input. how can I check/enable the log like that? "show logging" does not show anything except the below.

asa2(config)# show logging
Syslog logging: disabled
Facility: 20
Timestamp logging: disabled
Hide Username logging: enabled
Standby logging: disabled
Debug-trace logging: disabled
Console logging: disabled
Monitor logging: disabled
Buffer logging: disabled
Trap logging: disabled
Permit-hostdown logging: disabled
History logging: disabled
Device ID: disabled
Mail logging: disabled
ASDM logging: disabled

Also, your link for doc gives me 404...

[Jonatan Jonasson]

I accidentally added a comma at the end of the link.

The link is: https://www.cisco.com/c/en/us/td/docs/security/asa/syslog/b_syslog/syslogs3.html

And @Devaa responded regarding the syslog.

Regarding the dynamic and static nat, and different types of nat, the configuration guides provide pretty decent info as well:

https://www.cisco.com/c/en/us/td/docs/security/asa/asa922/configuration/firewall/asa-922-firewall-config/nat-basics.html

It would be valuable to practice with different types of NAT, and verify with both testing, packet captures and syslog.

[Devaa]

Basic Syslog

Enter these commands in order to enable logging, view logs, and view configuration settings.

• logging enable - Enables the transmission of syslog messages to all output locations.

• no logging enable - Disables logging to all output locations.

• show logging - Lists the contents of the syslog buffer as well as information and statistics that pertain to the current configuration.

The ASA can send syslog messages to various destinations. Enter the commands in these sections in order to specify the locations you would like the syslog information to be sent:

Send Logging Information to the Internal Buffer

logging buffered severity_level

https://www.cisco.com/c/en/us/support/docs/security/pix-500-series-security-appliances/63884-config-asa-00.html

[milks126]

@Jonatan Jonasson+ @Devaa, thank you guys so much! I could validate it after enabling the logging. I will definitely go over the link you guys provide and have a better understanding how different types of NAT work.

%ASA-5-305013: Asymmetric NAT rules matched for forward and reverse flows; Connection for icmp src outside:100.1.1.1 dst inside:172.16.1.2 (type 8, code 0) denied due to NAT reverse path failure