portmap translation creation failed for tcp src inside

zhuffines · ‎03-25-2013

We have an ASA 5540 with 8.2(5)

Last three days in early afternoon we start getting these errors in the log and webpages either won't load or pages only half load.

3|Mar 22 2013|13:22:24|305006|184.73.105.115|443|||portmap translation creation failed for tcp src inside:10.10.176.114/58217 dst outside:184.73.105.115/443

3|Mar 22 2013|13:22:24|305006|54.243.129.71|80|||portmap translation creation failed for tcp src inside:10.35.54.37/1517 dst outside:54.243.129.71/80

3|Mar 22 2013|13:22:24|305006|74.125.227.70|80|||portmap translation creation failed for tcp src inside:10.110.22.50/3968 dst outside:74.125.227.70/80

3|Mar 22 2013|13:22:24|305006|54.243.129.71|80|||portmap translation creation failed for tcp src inside:10.35.54.37/1516 dst outside:54.243.129.71/80

3|Mar 22 2013|13:22:24|305006|54.243.129.71|80|||portmap translation creation failed for tcp src inside:10.35.54.37/1515 dst outside:54.243.129.71/80

3|Mar 22 2013|13:22:24|305006|74.125.139.125|5222|||portmap translation creation failed for tcp src inside:10.160.230.91/49926 dst outside:74.125.139.125/5222

3|Mar 22 2013|13:22:24|305006|199.7.59.72|80|||portmap translation creation failed for tcp src inside:10.100.22.214/49988 dst outside:199.7.59.72/80

3|Mar 22 2013|13:22:24|305006|68.67.151.213|80|||portmap translation creation failed for tcp src inside:10.50.183.3/1420 dst outside:68.67.151.213/80

3|Mar 22 2013|13:22:24|305006|98.139.50.175|80|||portmap translation creation failed for tcp src inside:10.195.38.27/2259 dst outside:98.139.50.175/80

3|Mar 22 2013|13:22:24|305006|216.252.124.30|80|||portmap translation creation failed for tcp src inside:10.195.38.27/2258 dst outside:216.252.124.30/80

3|Mar 22 2013|13:22:24|305006|98.137.51.1|443|||portmap translation creation failed for tcp src inside:10.160.230.92/49984 dst outside:98.137.51.1/443

3|Mar 22 2013|13:22:24|305006|74.125.227.69|80|||portmap translation creation failed for tcp src inside:10.110.22.50/3966 dst outside:74.125.227.69/80

3|Mar 22 2013|13:22:24|305006|74.125.227.11|443|||portmap translation creation failed for tcp src inside:10.100.52.7/55758 dst outside:74.125.227.11/443

3|Mar 22 2013|13:22:24|305006|63.111.11.175|5222|||portmap translation creation failed for tcp src inside:10.10.184.106/52130 dst outside:63.111.11.175/5222

Here's some of the config:

!

interface GigabitEthernet0/0

speed 1000

duplex full

nameif outside

security-level 0

ip address 72.xxx.xxx.2 255.255.255.0

!

interface GigabitEthernet0/1

speed 1000

duplex full

nameif inside

security-level 100

ip address 10.1.1.2 255.255.240.0

!

interface GigabitEthernet0/2

speed 100

duplex full

nameif DMZ1

security-level 0

no ip address

!

interface GigabitEthernet0/3

shutdown

nameif DMZ2

security-level 0

no ip address

!

interface Management0/0

nameif management

security-level 100

ip address 192.168.1.1 255.255.255.0

management-only

!

boot system disk0:/asa825-k8.bin

ftp mode passive

clock timezone EST -5

clock summer-time EDT recurring

same-security-traffic permit intra-interface

pager lines 42

logging enable

logging monitor notifications

logging trap notifications

logging asdm informational

logging host inside 10.10.3.35

mtu outside 1500

mtu inside 1500

mtu DMZ1 1500

mtu DMZ2 1500

mtu management 1500

ip local pool attsupport 172.xxx.xxx.10-172.xxx.xxx.254 mask 255.255.255.0

ip verify reverse-path interface outside

ip verify reverse-path interface inside

no failover

failover polltime unit 15 holdtime 45

icmp unreachable rate-limit 1 burst-size 1

asdm image disk0:/asdm-645-106.bin

asdm history enable

arp timeout 14400

global (outside) 1 72.xxx.xxx.254

global (outside) 1 interface

nat (outside) 1 172.xxx.xxx.0 255.255.255.0

nat (inside) 0 access-list nonat

nat (inside) 1 0.0.0.0 0.0.0.0

access-group acl-out in interface outside

access-group email_egress in interface inside

route outside 0.0.0.0 0.0.0.0 72.xxx.xxx.1 1

route inside 10.0.0.0 255.0.0.0 10.1.1.1 1

route inside 207.144.48.0 255.255.255.0 10.1.1.1 1

Any ideas?

Jouni Forss · ‎03-25-2013

Hi,

Have you checked the "show xlate count" to see how many translations are active at that moment?

Maybe also check the "show conn count" output at the same time.

The command "show perfmon" is also something that tells the current rate of connections and translations

You have a very basic Dynamic PAT configuration other than that you have 2 public IP addresses configured so you should have plenty of resource for PAT translations but I cant think of any other limitation at the moment.

Just seem that this might be one possible reason. It might also explain why the pages load partially. When you load a web page there are most probably several connections involved to load the whole page. Some connections might not succeed because the firewall has exhausted all available translations. But still one would imagine that this would require some really heavy Internet use on your network.

Too bad your software level doesnt have the command "show nat pool" available

- Jouni

zhuffines · ‎03-25-2013

Not expereincing any problems when I took these stats. Could we be reaching the limit for sessions behind one NAT IP address?

LEX5-Firewall# sho xlate count

42710 in use, 48835 most used

LEX5-Firewall# sho conn count

36936 in use, 41027 most used

LEX5-Firewall# sho perfmon

PERFMON STATS: Current Average

Xlates 338/s 347/s

Connections 388/s 431/s

TCP Conns 323/s 338/s

UDP Conns 62/s 90/s

URL Access 181/s 184/s

URL Server Req 0/s 0/s

TCP Fixup 21936/s 0/s

TCP Intercept Established Conns 0/s 0/s

TCP Intercept Attempts 0/s 0/s

TCP Embryonic Conns Timeout 3/s 3/s

HTTP Fixup 21936/s 618/s

FTP Fixup 0/s 0/s

AAA Authen 0/s 0/s

AAA Author 0/s 0/s

AAA Account 0/s 0/s

VALID CONNS RATE in TCP INTERCEPT: Current Average

N/A 96.00%

Jouni Forss · ‎03-25-2013

Hi,

It does seem you have quite abit of connections active on the ASA.

Though that being said it still to my understanding shouldnt even be close to reaching the limit.

Also the fact that you have 2 Public IP address configured with the "global" command using the same "ID" number of the NAT configuration. This should mean that you should have more than enough resources.

I guess if you have some spare public IP addresses you could try adding a third PAT IP address with the "global" configuration command and see if theres any change in the situation.

I fear that this is one of the things where a person outside of Cisco might have problems troubleshooting wihtout trying to lab and reproduce the situation. But to be honest I havent looked at the this kind of situations that much and the few times I have, it have run into the problem that the firewall has been running too old software to actually get some clear output. Thats why I mentioned the "show nat pool" command from newer software.

For example output from my little ASA5505 running a newer software

ASA# show nat pool | inc WAN

TCP PAT pool WAN, address x.x.x.x, range 1-511, allocated 3

TCP PAT pool WAN, address x.x.x.x4, range 512-1023, allocated 0

TCP PAT pool WAN, address x.x.x.x, range 1024-65535, allocated 46

UDP PAT pool WAN, address x.x.x.x, range 1-511, allocated 17

UDP PAT pool WAN, address x.x.x.x, range 512-1023, allocated 0

UDP PAT pool WAN, address x.x.x.x, range 1024-65535, allocated 16

Where

WAN = My "outside" interface
x.x.x.x = My "WAN" interface IP address

I dont know if there is a similiar command on your software. I would check the available parameters on the "show xlate" and "show nat" commands.

Also the command reference for your software should list the command parameters if you want to check.

But as I said I would probably see if I could add a third PAT IP address and see if it has any kind of effect on the problem. Or perhaps wait if someone from Cisco has run into this before and would have good commands to troubleshoot this issue.

- Jouni

zhuffines · ‎03-25-2013

We have over 12,000 devices on our network. Should we lower these timeout values?

Connections are going down. After school hours now, I don't have values for what they were during school hours.

LEX5-Firewall# sho xlate count

33200 in use, 48835 most used

LEX5-Firewall# sho conn count

31056 in use, 41027 most used

LEX5-Firewall# sho run | i timeout

arp timeout 14400

timeout xlate 24:00:00

timeout conn 12:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02

timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00

timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00

timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute

timeout tcp-proxy-reassembly 0:01:00

timeout floating-conn 0:00:00

telnet timeout 10

ssh timeout 5

console timeout 0

vpn-idle-timeout 45

vpn-session-timeout none

Jouni Forss · ‎03-25-2013

Hi,

Well just to compare the stats that we have at our customer range from 3 hours to 9 hours. (for xlate)

In the normal situation I would imagine that the translation is torn down after the connection is removed but I guess it wouldnt really hurt to lower the xlate timeout to perhaps closer to the connection timeout.

Though as I said if you would want to test with least amount of impact to current operation you could try adding additional PAT IP addresses using the same in the "global" configuration and monitor of the problem continues.

Then again the "show xlate count" should already tell the "most used" value for the firewall unless they firewall has rebooted (but I doubt it?)

- Jouni

zhuffines · ‎03-25-2013

We did reboot the firewall because that was the "fix".

:/

So all the counts were reset. That most used count above was within 15 minutes of rebooting this afternoon.

I'll find out tomorrow if it happens again. Thanks for the info!