Simple Dynamic Nat odd behavior

[m-augustine]

Hello,

I'm working to configure a simple Dynamic NAT setup to allow inside hosts to reach the internet.

When I apply the config, the host1 from inside the network gets 1 ping response, then it stops receiving responses. I'm pinging a host (host2) that I control on the internet, and using tcpdump on host2 I can see that icmp requests are coming in from the correct IP and then host2 is sending its response to the correct IP.

Using ethanalyzer i can see that the reply packets hit Et 1/48 from host2, but they don't seem to get any further. 

Full config file is attached. Here is the basic NAT setup:

version 10.3(1) Bios:version 05.47
feature nat

ip nat inside source list mgmt-nat interface Ethernet1/48 overload add-route

interface Vlan200
ip nat inside

interface Ethernet1/48
ip nat outside

switch-01(config)# sh ip access-lists mgmt-nat

IP access list mgmt-nat
10 permit ip any any
20 permit icmp any any
30 permit tcp any any

Software
BIOS: version 05.47
NXOS: version 10.3(1) [Feature Release]
BIOS compile time: 04/28/2022
NXOS image file is: bootflash:///nxos64-cs.10.3.1.F.binq
NXOS compile time: 8/18/2022 15:00:00 [08/19/2022 02:44:02]

Hardware
cisco Nexus9000 C93180YC-FX Chassis
Intel(R) Xeon(R) CPU D-1528 @ 1.90GHz with 32802108 kB of memory.
Processor Board ID FLM26400J37

Does anyone have any insight into what may be happening?

P.S. The license smart license registration has not completed. Is a license required for this to work?

switch-01# sh license sum
License Usage:
License Entitlement tag Count Status
-------------------------------------------------------------------------
LAN license for Nexus 9... (LAN_ENTERPRISE_SERVICES_PKG) 1 IN USE

thanks in advance

[Richard Burts]

If I understand the issue correctly (and it is quite possible that I do not) the issue is about traffic that needs nat. If that is the case I would expect that traffic other than icmp would also have this difficulty.

[MHM Cisco World]

If that is the case I would expect that traffic other than icmp would also have this difficulty. 
I want to make sure that other traffic not face same drop when NATing by NSK

[m-augustine]

Thank you for the replies. Sorry for my late reply, it's been a very busy week at work.

All traffic fails, which is to say that the host machines have no access to any internet resources. I assume its the same reason that ICMP gets dropped.

I have studied up on CoPP a little (all new to me). It appears it can't be disabled (nor would I want to disable it), but it seems we can adjust the policies?

[MHM Cisco World]

Let me make double review may be we can solve it without change secure CoPP.

[MHM Cisco World]

I make deep and find why the HW not handle the NAT and forward it to CPU and icmp drop after few success 
there is by default no room for NAT in TCAM so you need to Carve the TCAM and add some room for NAT 
I think after that the NAT will work.

• By default no TCAM entries are allocated for the NAT feature. You allocate the TCAM size for the NAT feature by adjusting the TCAM size of other features. The TCAM can be allocated with the hardware access-list tcam region nat tcam-size command.

Understand how to Carve TCAM Space - Nexus 9000 - Cisco

[paul driver]

Hello
I remember having a similar issue when using a 9k as a nat rtr on eve-ng.
I had to set the tcam nat value to other than 0 to allow nat to work correctly and then it still timed out after 100 + pings etc and the only why it began to work there after that was performing a manual force clearing of the translation table

Clear ip nat translation all

conf t
hardware access-list tcam region nat xxx 
exit
copy running-config start-up config

[m-augustine]

Thanks for following up.

I have had these set since beginning of this thread 

---

hardware access-list tcam region ing-racl 1792

hardware access-list tcam region nat 512

I had to adjust the `ing-racl` down to make room for the nat. Perhaps 512 is not enough? I believe it was the minimum it would let me set for nat

[MHM Cisco World]

show hardware access-list tcam reg

share this

[m-augustine]

switch-01# show hardware access-list tcam reg
NAT ACL[nat] size = 512
Ingress PACL [ing-ifacl] size = 0
VACL [vacl] size = 0
Ingress RACL [ing-racl] size = 1792
Ingress L2 QOS [ing-l2-qos] size = 256
Ingress L3/VLAN QOS [ing-l3-vlan-qos] size = 512
Ingress SUP [ing-sup] size = 512
Ingress L2 SPAN filter [ing-l2-span-filter] size = 256
Ingress L3 SPAN filter [ing-l3-span-filter] size = 256
Ingress FSTAT [ing-fstat] size = 0
span [span] size = 512
Egress RACL [egr-racl] size = 1792
Egress SUP [egr-sup] size = 256
Ingress Redirect [ing-redirect] size = 0
Egress L2 QOS [egr-l2-qos] size = 0
Egress L3/VLAN QOS [egr-l3-vlan-qos] size = 0
Ingress Netflow/Analytics [ing-netflow] size = 512
Ingress NBM [ing-nbm] size = 0
TCP NAT ACL[tcp-nat] size = 0
Egress sup control plane[egr-copp] size = 0
Ingress Flow Redirect [ing-flow-redirect] size = 0
Ingress CNTACL [ing-cntacl] size = 0
Egress CNTACL [egr-cntacl] size = 0
MCAST NAT ACL[mcast-nat] size = 0
Ingress DACL [ing-dacl] size = 0
Ingress PACL Super Bridge [ing-pacl-sb] size = 0
Ingress Storm Control [ing-storm-control] size = 0
Ingress VACL redirect [ing-vacl-nh] size = 0
Egress PACL [egr-ifacl] size = 0

[MHM Cisco World]

[SUCCESS] New tcam size will be applicable only at boot time.
You need to 'copy run start' and 'reload'

did you done reload when apply new TCAM value ? 

[MHM Cisco World]

show system internal access-list globals 

please share this let us double check the TCAM have a room for NAT 

[m-augustine]

I thought I had reloaded, but after another reload, the NAT appears to be working

I will verify with more hosts, then I will accept the NAT TCAM info as the solution to the thread

I appreciate the time you invested here and I feel foolish it was something so simple, but I did learn a lot about this line of switches in the process

Thank you!

[MHM Cisco World]

OMG finally happy news 
thanks a lot for update us 
have a nice day 
thanks again 
MHM 

[m-augustine]

Hello,

I'm back with some bad news
I schedule some time to work with the Cisco Nexus switch today and noticed that I'm still experiencing the same issues with NATing

This time, however, the NAT class-map in CoPP is not the one dropping packets, it is entirely in  "class-map copp-system-p-class-l3uc-data (match-any)"
(see attached output)

I've been looking into adjusting the CIR and BC. Would this be a good way to confirm that this policy is the issue?

Thanks