Re: Network Address Translation not translating TCP packets

Luke Smith · ‎02-02-2021

I have an interesting issue. I sporadically have a customer who reports no internet. When I look at their NAT translations in our ASR1002 it shows 1 or 2 TCP translations and the rest are UDP. The moment they start working again it shows many more NAT translations. I'm at a loss I have tried to debug NAT packets but I don't see anything happening.

#sh ip nat trans in 2.2.2.2
Pro Inside global Inside local Outside local Outside global
udp 1.1.1.1:60909 2.2.2.2:60909 --- ---
udp 1.1.1.1:41106 2.2.2.2:41106 --- ---
udp 1.1.1.1:35818 2.2.2.2:35818 --- ---
udp 1.1.1.1:40247 2.2.2.2:40247 --- ---
udp 1.1.1.1:45731 2.2.2.2:45731 --- ---
udp 1.1.1.1:63649 2.2.2.2:33020 --- ---
udp 1.1.1.1:38240 2.2.2.2:57829 --- ---
udp 1.1.1.1:48664 2.2.2.2:48664 --- ---
udp 1.1.1.1:33014 2.2.2.2:33014 --- ---
udp 1.1.1.1:40186 2.2.2.2:40186 --- ---
udp 1.1.1.1:56641 2.2.2.2:48127 --- ---
udp 1.1.1.1:36594 2.2.2.2:36594 --- ---
udp 1.1.1.1:33614 2.2.2.2:33614 --- ---
udp 1.1.1.1:33484 2.2.2.2:33484 --- ---
udp 1.1.1.1:60441 2.2.2.2:60441 --- ---
udp 1.1.1.1:49148 2.2.2.2:49148 --- ---
udp 1.1.1.1:51314 2.2.2.2:51314 --- ---
udp 1.1.1.1:48106 2.2.2.2:48106 --- ---
udp 1.1.1.1:59228 2.2.2.2:59228 --- ---
udp 1.1.1.1:47106 2.2.2.2:47106 --- ---
udp 1.1.1.1:48124 2.2.2.2:48124 --- ---
udp 1.1.1.1:35738 2.2.2.2:35738 --- ---
udp 1.1.1.1:53389 2.2.2.2:53389 --- ---
udp 1.1.1.1:28064 2.2.2.2:58991 --- ---
udp 1.1.1.1:59392 2.2.2.2:50689 --- ---
udp 1.1.1.1:42240 2.2.2.2:56158 --- ---
udp 1.1.1.1:39130 2.2.2.2:39130 --- ---
udp 1.1.1.1:45912 2.2.2.2:45912 --- ---
udp 1.1.1.1:59973 2.2.2.2:59973 --- ---
udp 1.1.1.1:61536 2.2.2.2:49842 --- ---
udp 1.1.1.1:57243 2.2.2.2:57243 --- ---
udp 1.1.1.1:42606 2.2.2.2:42606 --- ---
udp 1.1.1.1:56412 2.2.2.2:56412 --- ---
udp 1.1.1.1:58990 2.2.2.2:58990 --- ---
udp 1.1.1.1:55349 2.2.2.2:55349 --- ---
udp 1.1.1.1:51574 2.2.2.2:51574 --- ---
udp 1.1.1.1:59552 2.2.2.2:36452 --- ---
udp 1.1.1.1:57027 2.2.2.2:57027 --- ---
udp 1.1.1.1:42314 2.2.2.2:42314 --- ---
udp 1.1.1.1:49931 2.2.2.2:49931 --- ---
udp 1.1.1.1:37997 2.2.2.2:37997 --- ---
udp 1.1.1.1:17216 2.2.2.2:35114 --- ---
udp 1.1.1.1:49129 2.2.2.2:49129 --- ---
udp 1.1.1.1:41176 2.2.2.2:41176 --- ---
udp 1.1.1.1:51185 2.2.2.2:51185 --- ---
udp 1.1.1.1:41791 2.2.2.2:41791 --- ---
udp 1.1.1.1:41322 2.2.2.2:41322 --- ---
udp 1.1.1.1:37280 2.2.2.2:54094 --- ---
udp 1.1.1.1:38472 2.2.2.2:38472 --- ---
udp 1.1.1.1:59411 2.2.2.2:59411 --- ---
udp 1.1.1.1:36291 2.2.2.2:34643 --- ---
udp 1.1.1.1:62753 2.2.2.2:55718 --- ---
udp 1.1.1.1:58748 2.2.2.2:58748 --- ---
udp 1.1.1.1:40096 2.2.2.2:38171 --- ---
udp 1.1.1.1:48032 2.2.2.2:38465 --- ---
udp 1.1.1.1:36158 2.2.2.2:36158 --- ---
udp 1.1.1.1:36308 2.2.2.2:36308 --- ---
udp 1.1.1.1:56613 2.2.2.2:56613 --- ---
udp 1.1.1.1:60911 2.2.2.2:60911 --- ---
udp 1.1.1.1:44222 2.2.2.2:44222 --- ---
udp 1.1.1.1:48932 2.2.2.2:48932 --- ---
udp 1.1.1.1:39829 2.2.2.2:39829 --- ---
udp 1.1.1.1:7840 2.2.2.2:43812 --- ---
udp 1.1.1.1:57094 2.2.2.2:57094 --- ---
udp 1.1.1.1:43144 2.2.2.2:43144 --- ---
udp 1.1.1.1:46032 2.2.2.2:46032 --- ---
udp 1.1.1.1:55102 2.2.2.2:55102 --- ---
udp 1.1.1.1:46674 2.2.2.2:46674 --- ---
udp 1.1.1.1:52943 2.2.2.2:52943 --- ---
udp 1.1.1.1:57652 2.2.2.2:57652 --- ---
udp 1.1.1.1:60967 2.2.2.2:60967 --- ---
udp 1.1.1.1:47619 2.2.2.2:47619 --- ---
udp 1.1.1.1:50824 2.2.2.2:50824 --- ---
udp 1.1.1.1:37610 2.2.2.2:37610 --- ---
udp 1.1.1.1:9152 2.2.2.2:36067 --- ---
udp 1.1.1.1:59808 2.2.2.2:59203 --- ---
udp 1.1.1.1:42839 2.2.2.2:42839 --- ---
udp 1.1.1.1:52803 2.2.2.2:37577 --- ---
udp 1.1.1.1:49488 2.2.2.2:49488 --- ---
udp 1.1.1.1:54300 2.2.2.2:54300 --- ---
udp 1.1.1.1:50280 2.2.2.2:50280 --- ---
udp 1.1.1.1:37848 2.2.2.2:37848 --- ---
udp 1.1.1.1:44221 2.2.2.2:44221 --- ---
udp 1.1.1.1:57726 2.2.2.2:57726 --- ---
udp 1.1.1.1:45656 2.2.2.2:45656 --- ---
udp 1.1.1.1:40412 2.2.2.2:40412 --- ---
udp 1.1.1.1:41196 2.2.2.2:41196 --- ---
udp 1.1.1.1:57640 2.2.2.2:57640 --- ---
udp 1.1.1.1:51515 2.2.2.2:51515 --- ---
udp 1.1.1.1:33345 2.2.2.2:42537 --- ---
udp 1.1.1.1:49657 2.2.2.2:49657 --- ---
udp 1.1.1.1:55937 2.2.2.2:51095 --- ---
udp 1.1.1.1:33738 2.2.2.2:33738 --- ---
udp 1.1.1.1:46442 2.2.2.2:46442 --- ---
udp 1.1.1.1:46273 2.2.2.2:46273 --- ---
udp 1.1.1.1:38534 2.2.2.2:38534 --- ---
udp 1.1.1.1:56208 2.2.2.2:56208 --- ---
udp 1.1.1.1:34392 2.2.2.2:34392 --- ---
tcp 1.1.1.1:60390 2.2.2.2:60390 --- ---
udp 1.1.1.1:59702 2.2.2.2:59702 --- ---
udp 1.1.1.1:37291 2.2.2.2:37291 --- ---
udp 1.1.1.1:36864 2.2.2.2:1024 --- ---
udp 1.1.1.1:55616 2.2.2.2:50662 --- ---
udp 1.1.1.1:37373 2.2.2.2:37373 --- ---
udp 1.1.1.1:40484 2.2.2.2:40484 --- ---
udp 1.1.1.1:60097 2.2.2.2:58600 --- ---
udp 1.1.1.1:45862 2.2.2.2:45862 --- ---
udp 1.1.1.1:49536 2.2.2.2:56039 --- ---
udp 1.1.1.1:39241 2.2.2.2:39241 --- ---
Total number of translations: 109

TJ-20933766 · ‎02-02-2021

Are they using Port Address Translation to a single IP address? Where I'm going with this is I'm wondering if they are exhausting the number of ports available to overload that single IP. Might try going with a pool of addresses if that's an option.

Alternatively, it could be a software bug although I've not looked around to see if I could find one. Updating the router's software might resolve the bug if one exists and is causing this issue.

Richard Burts · ‎02-02-2021

If I am understanding the original post this router begins to have problems, that seem to be related to translation (or failure to translate) of tcp packets. And then starts to work again. Any clue what happened that begins the issue? Any clue what happened that resolved the issue (reboot, shut/no shut, something else)?

What is the logging level for this router? If you look in the logs around the time that the problem starts are there any messages that might shed light on this? If you look in the logs around the time that the problem is resolved are there any messages that might shed light on this?

Exhausting the available ports is an interesting possibility.

HTH

Rick

Luke Smith · ‎02-02-2021

This is the best part of it all ...

What provoked the problem? Unknown, no changes were made.

What resolved the problem? Unknown, no changes were made.

As for logging level, it is set to debug and all I see are TCP session creation/deletion messages as there are over 200k NAT translations going on in the ASR at any given time.

Georg Pauwen · ‎02-02-2021

Hello,

in addition to Tyler's remarks, you might want to change the global NAT timeout:

--> ip nat translation 300

A pool is certainly a good idea, in case you have more than one public IP address. The 'rotary' keyword would ensure that each new TCP session would be translated into a session with a new public IP address:

--> ip nat pool name start-ip end-ip {netmask netmask | prefix-length prefix-length} type rotary

-->

Luke Smith · ‎02-02-2021

I had tried the timeout of 300 and it caused more damage then good.

I will look into the rotary command, however the pools right now are single IP addresses so I don't think rotary would cause any difference right now.

Richard Burts · ‎02-03-2021

I had hoped that there would be some type of log message as the problem began (and hoped for some type of log message as things started to work again). Sorry that does not seem to be the case. I understand that currently your pools are single address. Is there any possibility of making changes and have additional address in the pool?

HTH

Rick