NAT: Failed to allocate more ports for address, no ports available

[datacenter]

Hello Guys!

Having issues with intermittent internet services connectivity, and if I change outgoing link (there are two links of different providers) it works for a while but then it starts again.

All that I have from logs is this:

Nov 2 06:10:02.420: NAT: Received request to allocate 1024 ports for address X.X.X.X, start_port 5062 end_port 65000
Nov 2 06:10:02.420: NAT: Failed to allocate more ports for address X.X.X.X, no ports available for the range asked

Repeatedly...

We have many ISR 4000 routers and configs do not vary that much, but these logs I've never seen.

It is a CISCO ISR4451 with additional MEM/CPU running IOS Amsterdam Version 17.3.5.

Configs for NAT:

ip nat inside source list ACL_NAT interface GigabitEthernet0/0/1 overload
ip nat inside source list ACL_NAT interface GigabitEthernet0/0/1 vrf INET1 overload
ip nat inside source list ACL_NAT interface GigabitEthernet0/0/2 vrf INET2 overload

There are also other static 1:1 NATs for specific exposed services and some static PATs too.

Extended IP access list ACL_NAT
10 deny ip 10.0.0.0 0.255.255.255 10.0.0.0 0.255.255.255
12 deny ip 10.0.0.0 0.255.255.255 172.16.0.0 0.15.255.255
20 permit ip 10.0.0.0 0.255.255.255 any
22 deny ip 172.16.0.0 0.15.255.255 10.0.0.0 0.255.255.255
23 deny ip 172.16.0.0 0.15.255.255 172.16.0.0 0.15.255.255
24 permit ip 172.16.0.0 0.15.255.255 any
52 permit ip 192.168.0.0 0.0.0.255 any
62 permit ip 192.168.1.0 0.0.0.255 any

# PROVIDER 1

interface GigabitEthernet0/0/1

 ip vrf forwarding INET1

ip nat outside

# PROVIDER 2

interface GigabitEthernet0/0/2

ip vrf forwarding INET2

ip nat outside

!

XXXXXXX#sh ip nat translations total
Total number of translations: 76643

This is a medium size branch (~350 users) with some good load over internet, but we have other branches with similar profile and we have never seen this logs before.

Thanks!

[M02@rt37]

Hello @datacenter,

Do you try to clear the NAT table ?

clear ip nat translation *

[datacenter]

No because they are running production now, trying to avoid impacts.

But I'm pretty sure that clearing the NAT table would probably work and would serve as a workaround solution.

[MHM Cisco World]

can I see 
show nat statistics 

Thanks A Lot
MHM

[datacenter]

RTZAJNB022#sh ip nat stat
Total active translations: 1371 (14 static, 1357 dynamic; 1343 extended)
Outside interfaces:
GigabitEthernet0/0/1, GigabitEthernet0/0/2, Tunnel100, Tunnel200
Inside interfaces:
GigabitEthernet0/0/0.901, GigabitEthernet0/0/0.1000
GigabitEthernet0/0/0.2000
Hits: 53701972088 Misses: 111288959
Expired translations: 110421075
Dynamic mappings:
-- Inside Source
[Id: 1] access-list ACL_NAT interface GigabitEthernet0/0/1 refcount 0
[Id: 2] access-list ACL_NAT interface GigabitEthernet0/0/1 refcount 916
[Id: 3] access-list ACL_NAT interface GigabitEthernet0/0/2 refcount 10
nat-limit statistics:
max entry: max allowed 0, used 0, missed 0
In-to-out drops: 32651651 Out-to-in drops: 2829472
Pool stats drop: 0 Mapping stats drop: 0
Port block alloc fail: 2052136
IP alias add fail: 0
Limit entry add fail: 0

Issue is not happening right now, but when facing problem I saw stats over 65536 (close to 80k) that is probably why it was showing the logs.

I've reduced NAT timeout time and been monitoring last days.

ip nat translation timeout 300
ip nat translation tcp-timeout 300

[Richard Burts]

I do not understand what is the cause of the issue here and have a couple of things that puzzle me:

- why the request for so many ports "Received request to allocate 1024 ports for address X.X.X.X, start_port 5062 end_port 65000"

- why are there 2 nat statements for G0/0/1

ip nat inside source list ACL_NAT interface GigabitEthernet0/0/1 overload
ip nat inside source list ACL_NAT interface GigabitEthernet0/0/1 vrf INET1 overload

I suggest removing the first nat statement and see if the behavior changes.

[datacenter]

- why the request for so many ports "Received request to allocate 1024 ports for address X.X.X.X, start_port 5062 end_port 65000"

This is my main question too... By the message it looks like someone is trying to allocate a wide range of ports, but I think this is a tricky message... it is probably saying that someone tried to allocato ONE port and this whole range is allocated already (I'm assuming this).

- why are there 2 nat statements for G0/0/1

This is the main breakout interface, so I need one NAT for VRF INET1 but also one for NO VRF for router internal traffic to be natted (as there is no VRF to internal traffic).

What I found very awkward is during the problem I took some outputs and found a bunch of NATs establisheds for days, and a ton of NATs from same few devices (and to add more to this they were Android Barcode Scanners).

As I said this is pretty standard config for more than 100 branches we have (running isr4k with same IOS), and this is the first time we face this issue/messages

[MHM Cisco World]

Router(config)#ip nat translation tcp-timeout 500 Router(config)#ip nat translation udp-timeout 30 Router(config)#ip nat translation dns-timeout 30 Router(config)#ip nat translation icmp-timeout 30 Router(config)#ip nat translation finrst-timeout 30 Router(config)#ip nat translation syn-timeout 30

use this timeout and remove the two command you use before.
the port is not timeout and hence the NAT can not allocated new port for new NAT. 

Thanks A Lot
MHM