Re: IPNAT-4-ADDR_ALLOC_FAILURE - Any Ideas?

asaykao73 · ‎07-19-2009

Hi All,

We are seeing this repeated in our router logs. Running Cisco 7606 with IOS 12.2(18)SXF11.

Jul 20 13:12:35.045 AEST: %IPNAT-4-ADDR_ALLOC_FAILURE: Address allocation failed for 172.16.75.89, pool NAT-POOL might be exhausted

I googled it but it turns up every little information.

-----------------------------------------

Error Message IPNAT-4-ADDR_ALLOC_FAILURE: Address allocation failed for [IP_address], pool [chars] might be exhausted

Explanation: This message indicates that an address could not be allocated from the IP NAT pool. This condition can cause a translation failure and might result in packets being dropped. The counter for missed packets will be incremented.

Recommended Action: Determine if the NAT pool has been exhausted. To reuse any existing addresses in the NAT pool for new packet flows, clear the current NAT entries using the clear ip nat translation command. "

-----------------------------------------

When the error message appears, there isn't much translation taking place for the IP.

Jul 20 13:12:35.045 AEST: %IPNAT-4-ADDR_ALLOC_FAILURE: Address allocation failed for 172.16.75.89, pool NAT-POOL might be exhausted

core2#sh ip nat trans | inc 172.16.75.89

tcp 210.15.240.6:1026 172.16.75.89:58254 210.15.254.237:443 210.15.254.237:443

tcp 210.15.240.6:1033 172.16.75.89:58259 210.15.254.237:443 210.15.254.237:443

tcp 210.15.240.6:1035 172.16.75.89:58261 210.15.254.237:443 210.15.254.237:443

When the error mesg appeared a few minutes later, this time no current nat translation taking place for it.

Jul 20 13:18:16.995 AEST: %IPNAT-4-ADDR_ALLOC_FAILURE: Address allocation failed for 172.16.75.89, pool NAT-POOL might be exhausted

core2#sh ip nat trans | inc 172.16.75.89

I don't know what's causing this. It's happening for other DHCP clients as well. I can't replicate it on my work station.

Sometimes the host with that IP will lose data flow and we have to do a "clear ip nat tran" to restore it.

Here's the show ip nat statistics.

core2#sh ip nat statistics

Total active translations: 25 (0 static, 25 dynamic; 25 extended)

Outside interfaces:

Vlan11, Vlan111

Inside interfaces:

Vlan22, Vlan63, Vlan69, Vlan512

Hits: 4717 Misses: 450

Expired translations: 457

Dynamic mappings:

-- Inside Source

[Id: 1] route-map nonat pool NAT-POOL refcount 25

pool NAT-POOL: netmask 255.255.255.224

start 210.15.240.6 end 210.15.240.6

type generic, total addresses 1, allocated 1 (100%), misses 62

longest chain in pool: NAT-POOL's addr-hash: 1, average len 0,chains 1/256

longest chain in local hash: 1, average length 0, chains 25/2048

longest chain in global hash: 1, average length 0, chains 25/2048

Has anyone come across this problem before? Is it an IOS bug? Anything else I should look for?

Thanks.

Andy

Giuseppe Larosa · ‎07-19-2009

Hello Andy,

>> Hits: 4717 Misses: 450

it looks like that there is almost a 10% probability of NAT failure so there is impact.

And 25 concurrent translations are not many.

I did some search on bug toolkit with no exact match for your case.

However, the following global command might be useful

ip nat translation max-entries

Hope to help

Giuseppe

asaykao73 · ‎07-19-2009

Hi Giuseppe ,

We don't do very many nat translations on this router. The nat network on the router is there to support those who bring a wireless laptop to work (which isn't many). It also supports people with iphones who connect to the wireless access point to download their email.

Apart from lodging a TAC case I'm not sure what else I can do.

Cheers.

Andy

asaykao73 · ‎07-19-2009

I've increased the NAT-POOL now from one public IP to four.

Looks like this might have fixed the problem although I'm not sure why it was apparent in the first place.

Config change:

no ip nat pool NAT-POOL 210.15.240.6 210.15.240.6 netmask 255.255.255.224

ip nat pool NAT-POOL 210.15.240.6 210.15.240.9 netmask 255.255.255.224

core2#sh ip nat trans

Pro Inside global Inside local Outside local Outside global

--- 210.15.240.9 172.16.75.111 --- ---

tcp 210.15.240.8:1024 172.16.75.113:49843 66.102.11.193:443 66.102.11.193:443

What I don't get is how you can get a translation appear in the NAT table without any protocol, outside local and outside globals as seen above for 172.16.75.111.

I believe this is what's causing the error messages in the logs. 172.16.75.111 appears to be doing something strange and causing a weird entry in the NAT table. Now if there are existing translations in place, for whatever reason it can't use the public IP as the inside local and starts complaining that the NAT-POOL has been exhausted. This was the case with just a single public IP to NAT to. With multiple public IPs to NAT to now, it doesn't complain because it can just grab the next available public IP to use.

How on earth you get that weird entry from 172.16.75.111 is unknown to me.

tataravatu · ‎09-06-2012

Hi Andy,

I realise it's been 4 years since your last post to this issue, but it's exactly the problem I am having. Did you manage to find a solution for it?

That is, although nat pool size has been increased, the first public IP is patting correctly however the three other public IPs are only natting for 1 private IP each.

Apprecaite your help.

Regards,

AT

Sasha Tchepourko · ‎04-01-2013

Yep same here, was there a resolution? Looks more like a bug with IOS based NAT...

Those NAT entries without protocol seem to have a lifetime of 24 hours too which is not very efficient.

Neeraj Arora · ‎04-02-2013

Hi,

This is an expected behaviour when you use an IP POOL for NATing.

Lets say, if the pool is for 4 Ip addresses, then first 3 will create one-2-one NATing, which you can see as:

Pro Inside global Inside local Outside local Outside global

--- 210.15.240.9 172.16.75.111 --- ---

And the last ip would be used for PAT

this is sticky in nature i.e till the time it timeouts, this Public ip address would not be available for others and 172.16.75.111 will always be translated into 210.15.240.9 till the time this entry is in NAT translation table.

See Bug id: CSCdm68899

http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCdm68899

So a better solution for this would be use an interface ip for PAT or a single Ip in the pool. An eg. of the config:

-----------------------------------------------------------

int gig0/0

ip nat outside

int gig0/1

ip nat inside

ip nat inside source list XXX interface gig0/0 overload

-----------------------------------------------------------

Hope it helps

Neeraj

Sasha Tchepourko · ‎04-02-2013

Thanks Neeraj.

How would the NAT overload process know not to use the router-router IPs that are assigned to gig0/0 and the other end?

Also this bug has been "Terminated (Junked)" and is almost 13 years old, surely recent router IOS software has been fixed by now

rojesara.prashant · ‎01-30-2019

Did you find any issue ?

We are also getting below messages.

%IOSXE-6-PLATFORM:cpp_cp: QFP:0.0 Thread:001 TS:00016130839432809296 %NAT-6-ADDR_ALLOC_FAILURE: Address allocation failed; pool 10 may be exhausted model ISR4431/K9 with running Denali version.

rojesara.prashant · ‎01-30-2019

Did you find any issue ?

We are also getting below messages.

%IOSXE-6-PLATFORM:cpp_cp: QFP:0.0 Thread:001 TS:00016130839432809296 %NAT-6-ADDR_ALLOC_FAILURE: Address allocation failed; pool 10 may be exhausted model ISR4431/K9 with running Denali version.