Simple Dynamic Nat odd behavior

[m-augustine]

Hello,

I'm working to configure a simple Dynamic NAT setup to allow inside hosts to reach the internet.

When I apply the config, the host1 from inside the network gets 1 ping response, then it stops receiving responses. I'm pinging a host (host2) that I control on the internet, and using tcpdump on host2 I can see that icmp requests are coming in from the correct IP and then host2 is sending its response to the correct IP.

Using ethanalyzer i can see that the reply packets hit Et 1/48 from host2, but they don't seem to get any further. 

Full config file is attached. Here is the basic NAT setup:

version 10.3(1) Bios:version 05.47
feature nat

ip nat inside source list mgmt-nat interface Ethernet1/48 overload add-route

interface Vlan200
ip nat inside

interface Ethernet1/48
ip nat outside

switch-01(config)# sh ip access-lists mgmt-nat

IP access list mgmt-nat
10 permit ip any any
20 permit icmp any any
30 permit tcp any any

Software
BIOS: version 05.47
NXOS: version 10.3(1) [Feature Release]
BIOS compile time: 04/28/2022
NXOS image file is: bootflash:///nxos64-cs.10.3.1.F.binq
NXOS compile time: 8/18/2022 15:00:00 [08/19/2022 02:44:02]

Hardware
cisco Nexus9000 C93180YC-FX Chassis
Intel(R) Xeon(R) CPU D-1528 @ 1.90GHz with 32802108 kB of memory.
Processor Board ID FLM26400J37

Does anyone have any insight into what may be happening?

P.S. The license smart license registration has not completed. Is a license required for this to work?

switch-01# sh license sum
License Usage:
License Entitlement tag Count Status
-------------------------------------------------------------------------
LAN license for Nexus 9... (LAN_ENTERPRISE_SERVICES_PKG) 1 IN USE

thanks in advance

[MHM Cisco World]

Can I see ethanylzer of 10 packet when you ping from host1 to host2

[m-augustine]

Thanks for taking the time. Here is the ethanalyzer output for the ping from host1 => host2
You can see that the first icmp request gets sent and the reply makes its way back to host1. The rest do not

switch-01# ethanalyzer local interface inband capture-filter "host 162.xxx.xxx.4"

Capturing on 'ps-inb'

1 2023-04-06 18:25:24.089350732 172.36.2.2 → 162.xxx.xxx.4 ICMP 98 Echo (ping) request id=0x0036, seq=1/256, ttl=64

2 2023-04-06 18:25:24.090179663 103.xxx.xxx.110 → 162.xxx.xxx.4 ICMP 98 Echo (ping) request id=0xecc9, seq=1/256, ttl=63

3 2023-04-06 18:25:24.266108463 162.xxx.xxx.4 → 103.xxx.xxx.110 ICMP 98 Echo (ping) reply id=0xecc9, seq=1/256, ttl=56 (request in 2)

4 4 2023-04-06 18:25:24.266563437 162.xxx.xxx.4 → 172.36.2.2 ICMP 98 Echo (ping) reply id=0x0036, seq=1/256, ttl=55 (request in 1)

5 2023-04-06 18:25:25.090870636 172.36.2.2 → 162.xxx.xxx.4 ICMP 98 Echo (ping) request id=0x0036, seq=2/512, ttl=64

6 6 2023-04-06 18:25:25.091439961 103.xxx.xxx.110 → 162.xxx.xxx.4 ICMP 98 Echo (ping) request id=0xecc9, seq=2/512, ttl=63

7 7 2023-04-06 18:25:25.267777295 162.xxx.xxx.4 → 103.xxx.xxx.110 ICMP 98 Echo (ping) reply id=0xecc9, seq=2/512, ttl=56 (request in 6)

8 8 2023-04-06 18:25:26.292229103 162.xxx.xxx.4 → 103.xxx.xxx.110 ICMP 98 Echo (ping) reply id=0x0036, seq=3/768, ttl=56

9 9 2023-04-06 18:25:27.315883839 162.xxx.xxx.4 → 103.xxx.xxx.110 ICMP 98 Echo (ping) reply id=0x0036, seq=4/1024, ttl=56

10 10 2023-04-06 18:25:28.342762588 162.xxx.xxx.4 → 103.xxx.xxx.110 ICMP 98 Echo (ping) reply id=0x0036, seq=5/1280, ttl=56

[Richard Burts]

The original post says full config is attached but I do not see any attachment. Seeing the full config would be very helpful. 
The very partial config in the post shows several permit any any statements. I have seen situations where permit any any has caused issues. I do not think that is necessarily the issue here, but strongly suggest revising the nat acl. In my experience  where you have an inside network(s) access the Internet the nat can usually be worked out using standard acl rather than extended. And with standard acl it is easier to avoid permit any.

[MHM Cisco World]

THIS  1-CYCLE START 

1 2023-04-06 18:25:24.089350732 172.36.2.2 → 162.xxx.xxx.4 ICMP 98 Echo (ping) request id=0x0036, seq=1/256, ttl=64

2 2023-04-06 18:25:24.090179663 103.xxx.xxx.110 → 162.xxx.xxx.4 ICMP 98 Echo (ping) request id=0xecc9, seq=1/256, ttl=63

3 2023-04-06 18:25:24.266108463 162.xxx.xxx.4 → 103.xxx.xxx.110 ICMP 98 Echo (ping) reply id=0xecc9, seq=1/256, ttl=56 (request in 2)

4 4 2023-04-06 18:25:24.266563437 162.xxx.xxx.4 → 172.36.2.2 ICMP 98 Echo (ping) reply id=0x0036, seq=1/256, ttl=55 (request in 1)
OTHER 2-CYCLE START 

5 2023-04-06 18:25:25.090870636 172.36.2.2 → 162.xxx.xxx.4 ICMP 98 Echo (ping) request id=0x0036, seq=2/512, ttl=64

6 6 2023-04-06 18:25:25.091439961 103.xxx.xxx.110 → 162.xxx.xxx.4 ICMP 98 Echo (ping) request id=0xecc9, seq=2/512, ttl=63

7 7 2023-04-06 18:25:25.267777295 162.xxx.xxx.4 → 103.xxx.xxx.110 ICMP 98 Echo (ping) reply id=0xecc9, seq=2/512, ttl=56 (request in 6)
HERE THE 2-CYCLE STOP NO NAT <<- 

OTHER PING FROM HOST TO NSK THIS NOT NATing at all 

8 8 2023-04-06 18:25:26.292229103 162.xxx.xxx.4 → 103.xxx.xxx.110 ICMP 98 Echo (ping) reply id=0x0036, seq=3/768, ttl=56

9 9 2023-04-06 18:25:27.315883839 162.xxx.xxx.4 → 103.xxx.xxx.110 ICMP 98 Echo (ping) reply id=0x0036, seq=4/1024, ttl=56

10 10 2023-04-06 18:25:28.342762588 162.xxx.xxx.4 → 103.xxx.xxx.110 ICMP 98 Echo (ping) reply id=0x0036, seq=5/1280, ttl=56

can you ping from hostA to hostB with repeat 100, I need to see the arrythmia of success failed 

[paul driver]

Hello
change your nat acl to be specific to the network your wish to translate- using any any isnt recommended as such cannot be deterministic in the results of the translation.

[m-augustine]

Thank you all for the suggestions. I added a more specific ACL and adjust the ip nat command

ip nat inside source list internet-access interface Ethernet1/48 overload add-route

ip access-list internet-access
10 permit ip 172.36.0.0/16 any
20 permit tcp 172.36.0.0/16 any
30 permit udp 172.36.0.0/16 any

Ive also attached the switch config (hopefully)

I will follow up with the ping repeat 100 results shortly

[m-augustine]

Attached is the ping with repeat 100

It's interesting to note that if I restart ping Host1 (inside) => Host2 (public ip) that the first icmp request will always receive a reply

[MHM Cisco World]

So sorry I see you post now 
I see one request and many many reply!! 
what is success rate of this ping 
I think there is asymmetric traffic here 

[m-augustine]

It is odd that ethanalyzer doesnt show the outbound ping request from hostA => hostB

attached is a tcpdump from hostB where we can see new requests coming in

When i ping with -c 100, its 99% packet drop. only the first request gets a reply

[root@host-A ~]$ ping 162.xxx.xxx.4 -c 100
PING 162.xxx.xxx.4 (162.xxx.xxx.4) 56(84) bytes of data.
64 bytes from 162.xxx.xxx.4: icmp_seq=1 ttl=55 time=178 ms

--- 162.142.125.4 ping statistics ---
100 packets transmitted, 1 received, 99% packet loss, time 101381ms
rtt min/avg/max/mdev = 177.608/177.608/177.608/0.000 ms

Here is another oddity, when I decrease the ping interval, the first several requests get replies. It's like after 1s, something happens that drops in the incoming reply

[root@host-A ~]$ ping 162.xxx.xxx.4 -i .25 -c 40

PING 162.xxx.xxx.4 (162.xxx.xxx.4) 56(84) bytes of data.

64 bytes from 162.xxx.xxx.4: icmp_seq=1 ttl=55 time=177 ms

64 bytes from 162.xxx.xxx.4: icmp_seq=2 ttl=55 time=177 ms

64 bytes from 162.xxx.xxx.4: icmp_seq=3 ttl=55 time=177 ms

64 bytes from 162.xxx.xxx.4: icmp_seq=4 ttl=55 time=177 ms

--- 162.xxx.xxx.4 ping statistics ---

40 packets transmitted, 4 received, 90% packet loss, time 9960ms

rtt min/avg/max/mdev = 176.827/177.024/177.164/0.142 ms

one thing that stuck out in the documentation to me was this line, but Im not sure of its impact on my situation

When creating a new translation on a Cisco Nexus 9000 Series switch, the flow is software forwarded until the translation is programmed in the hardware, which might take a few seconds. During this period, there is no translation entry for the inside global address. Therefore, returning traffic is dropped

---

Thanks, again, for the help!

[MHM Cisco World]

show policy-map interface control-plane <<- please share this 

Understanding Control Plane Packet Loss due to CoPP | Christopher Hart (chrisjhart.com)

[m-augustine]

Attached is the output. 
I see the nat flow shows drops. Interested to know your interpretation 
Thanks again!

[Richard Burts]

show policy-map was a good suggestion and does show that some nat traffic is being dropped.

I am interested in this statement "when I decrease the ping interval, the first several requests get replies" That would be consistent with the understanding that when you run the ping full bore it exceeds the permitted rate and when you slow the ping down then more pings are permitted.

[m-augustine]

The issue I'm seeing is that when I use the default interval (1/s) for pings, I only see 1 single ping request get a reply, the rest seem to get dropped

When I do a lower interval, i.e: .2 (5 pings/s) i see the first 5 pings get replies, then the rest get dropped

[MHM Cisco World]

It seem that NSK assume the reply direct to it not to host behind NAT.

So packet

Frrist packet Request send to control plane 

Second packet request pass through data plane not via control plane and hence not appear in ethanalyzer.

Now reply retrun always pass through control not data plane and this make CoPP drop any traffic higher than specific rate.

So we now test NAT with icmp please try use other traffic I think it ok 

OR

use NATing to other IP other than NSK interface IP.