PAT pool exhaustion / Portmap translation creation failures

[kpieckiel]

I'm getting some very strange (to me) logs from my ASA, and I'm out of ideas on how to figure it out.

System: ASA-5520

Software: 8.4(3)

Flash: 256 MB

Memory: 2048 MB

Mode: Routed / Single

License: VPN Plus

My Tunnels interface has address 192.168.6.1/24 and has restricted access to my internal network (named ATMS):  a handful of servers are NATted from ATMS to Tunnels with no translation and access from Tunnels to these servers on ATMS is allowed for the services they provide.  All other traffic across the Tunnels interface is outbound and initiated from some host on the ATMS interface (which isn't all that much).

In the syslogs (see attached screen cap) I see plethora dynamic UDP xlates being built from ATMS:192.168.6.1 to Tunnels:192.168.6.1 (first, I don't understand why this IP address is the same on both interfaces, as the ATMS interface is on a different subnet).  Each mapping is chained with the previous with each successive source port being the destination port from the previous map.  An excerpt from the attachment:

192.168.6.1:134 -> 192.168.6.1:149

192.168.6.1:149 -> 192.168.6.1:151

192.168.6.1:151 -> 192.168.6.1:271

192.168.6.1:271 -> 192.168.6.1:272

192.168.6.1:272 -> 192.168.6.1:281

and so on.  Then I get ID 305006 messages saying portmap translation creation failed and I get an ID 202010 message saying the NAT/PAT pool is exhausted (ID 202010 is not shown in the screen capture due to my display filter settings).

Why are there so many chained xlates?  Why are they being chained in the first place?  How do I keep this from occurring?

Some relevant config excerpts follow.  In fact, this is every line related to the Tunnels interface and the 192.168.6.0/24 subnet with the exception of the details of the object or service groups being used:

...

interface Port-channel1.6

vlan 6

nameif Tunnels

security-level 30

ip address 192.168.6.1 255.255.255.0

...

nat (ATMS,Tunnels) source static Domain-Controllers Domain-Controllers description AMMS computers at tunnels are domain computers

nat (ATMS,Tunnels) source static SQL2-Cluster SQL2-Cluster

nat (ATMS,Tunnels) source static HR-ATMS-SQL1 HR-ATMS-SQL1 description Tunnels use AMMS V7

nat (ATMS,Tunnels) source static HR-ATMS-FILE1 HR-ATMS-FILE1 description Document repository for AMMS at tunnels

nat (ATMS,Tunnels) source static hr-atms-media hr-atms-media description Symantec Endpoint Protection access for clients

...

access-list Tunnels_access_in extended permit object-group Active-Directory-Services 192.168.6.0 255.255.255.0 object-group Domain-Controllers

access-list Tunnels_access_in extended permit object ms-sql-s 192.168.6.0 255.255.255.0 object HR-ATMS-SQL1

access-list Tunnels_access_in extended permit tcp 192.168.6.0 255.255.255.0 object HR-ATMS-SQL1 object-group DM_INLINE_TCP_1

access-list Tunnels_access_in extended permit object-group ms-sharing 192.168.6.0 255.255.255.0 object HR-ATMS-FILE1

access-list Tunnels_access_in extended permit tcp 192.168.6.0 255.255.255.0 object hr-atms-media eq 8014

...

nat (ATMS,Tunnels) after-auto source dynamic any interface

access-group Tunnels_access_in in interface Tunnels

...

I don't know what other info you'd need to help me figure this out.  If anyone can aid me I would greatly appreciate it.

Thank you,

Kevin

[Jouni Forss]

Hi,

Just some guessing on my part but heres some thoughts. Hopefully they help even abit.

Can you include the full configurations of the object-groups / objects used in the NAT configurations.

Is the IP address 192.168.6.1 included in any of the groups?

I'm not totally sure about why you are seeing the same IP address on the src/dst fields but I'd guess its related to the static NAT configurations you have. As every statement has the same object twice it should mean that the address stays the same even when passing the interfaces mentioned in the configurations.

The red "portmap" translation failing is probably due to the fact that the destination address is a network broadcast address.

I can't really say anything about the ports you are seeing in the messages. To my understanding the src/dst port should be the same in the sort of NATs you have configured.

Is there some actual problem with traffic passing through the ASA or are you just wondering why you are seeing the mentioned log messages?

Is there any need to do any translations between ATMS and Tunnels interfaces? I mean if you just removed the NAT configurations alltogether ASA would pass traffic between those interfaces without NATing the traffic.

In problem situations I usually use ASAs "packet-tracer" command to check how the NAT configurations are behaving though I'm not sure if that would help you in this situation.

I have a feeling that theres somekind of overlap in your NAT configurations that might be causing this but thats just a guess at this point with the information provided.

[kpieckiel]

I'm hesitant to post too much config info, because at some point the risk of posting too much of my security configuration becomes unacceptable... I don't know what best practice really is for forums like this with the exception of changing IP addresses.  Nevertheless, you can't help if I don't tell you at least some stuff.  I've attached a text file of all object nat and nat statements as requested.  AFAIK, this is every statement directly related to NAT.

The IP address 192.168.6.1 is only included in the NAT process in translating access from the higher security iface ATMS to the Tunnels iface:

nat (ATMS,Tunnels) after-auto source dynamic any interface

I haven't noticed that this particular issue is causing any problems.  I've had a few connectivity issues with clients on the Tunnels network, but as best as I've been able to determine they've all been issues on the client PCs.  I'm mainly concerned that this will become a problem or that it will indirectly lead to other problems (memory exhaustion, excessive CPU utilization, etc.) and be difficult to track later.

The specific NAT translations were added because clients on the Tunnels network are domain members, so seeing the domain controllers (who live on the ATMS network) with their real IPs instead of translated IPs is necessary.  I don't completely trust this network and don't have control over physical security of the network segment, so I opted for putting it on a separate interface and VLAN on the ASA and translating addresses except where this is not possible.  I'm open to suggestions for better ways of handling this if anyone has any to offer.  Maybe I ought to just use ACLs to restrict traffic and ignore translation altogether?

I've never used the packet tracer function of the ASA, so I'll do some research on that and see if it can be of aid to me.

Thank you very much for your time and questions/expertise.  I appreciate it.

Kevin