Hi Ken,It is good to hear

ken129gem · ‎10-08-2015

Hi,

I have a Cisco 1811 ISR running IOS version 15.1(4)M10. I'm using NAT NVI for port forwarding to a couple of servers on my local network:

ip nat source list 1 interface FastEthernet0 overload
ip nat source static tcp 10.0.0.15 22 interface FastEthernet0 22
ip nat source static tcp 10.0.0.20 80 interface FastEthernet0 80

NAT NVI is being used rather than traditional NAT because it supports hairpinning, which is necessary in my network setup.

When I first entered these commands, everything was working fine. The SSH and HTTP services were accessible through the public IP address (assigned by the ISP with DHCP).

However, it seems that these two entries became disabled after a week or so for no apparent reason. When this happened, I was unable to access the SSH and HTTP services. Ports 22 and 80 appeared to be inaccessible when I ran an Nmap scan on the public IP address. The strange thing is that that sh run showed that the source static lines were still in the configuration. I was able to resolve the problem by re-entering the two commands in configuation mode (conf t):

ip nat source static tcp 10.0.0.15 22 interface FastEthernet0 22
ip nat source static tcp 10.0.0.20 80 interface FastEthernet0 80

When I re-entered the lines, port forwarding began working properly. However, the problem happened again after about a week, so I had to re-enter the lines once again.

Does anyone have an idea what may be causing this issue or how it can be resolved?

Thanks.

Peter Paluch · ‎10-08-2015

Hi Ken,

I hope you don't mind joining this thread again even though I could be called the source of your problems, as the configuration you're currently using is authored by me.

The behavior is most suspicious, to be entirely honest. I have not experienced it myself. The fact that it requires a longer time to occur suggests that there is something related to the operation rather than configuration. This could indeed be a bug. Is there a possibility of opening a TAC case on this one?

Ideally, if this happens again, it would be ideal to capture the output of show tech-support while the NAT is still inoperative and post it here even though this output is going to be very long and extremely detailed.

Do you have an option of trying a different IOS version, perhaps the 15.1(3)T4?

Best regards,
Peter

ken129gem · ‎10-08-2015

Hi Peter,

Thanks for your response. It's good to hear from you again.

I may look into opening a TAC case eventually, but first I will take your advice about checking show tech-support when the problem happens again. I have a feeling that the issue may take a little while to diagnose since it has only occurred twice in the past couple of weeks.

If I am unable to find a solution with the 15.1(4)M10 version, I will try installing a different version of IOS to see whether that has any effect.

Also, after a bit of Google searching, I came across a thread about a problem that sounds kind of similar:

https://supportforums.cisco.com/discussion/11685796/cisco-881-loses-nat-nvi-mappings-after-reload

The only difference is that they are using a Cisco 881 router, and their NAT NVI mappings are lost after a reload, not while the router is running. It looks like the problem can be resolved by switching back to traditional NAT, but that would prevent me from using the hairpinning feature.

Peter Paluch · ‎10-14-2015

Hi Ken,

It is good to hear from you as well!

Any success or luck with this one? It's actually frustrating... Cisco keeps emphasizing in all official documents that the use of this NVI style of NAT is only intended to be used in VRF environments, something you're obviously not using. We could theoretically try to convert things into VRFs just to make NVI happy, but that would complicate your things just for the sake of having your NAT working. The same Cisco documents still point to an ancient document using the PBR-based approach for NAT on a stick, but as we've discussed earlier, that approach no longer works with newer IOSes - so I wonder what is now the Cisco's endorsed solution for NAT-on-stick if there is any!

Best regards,
Peter

ken129gem · ‎10-14-2015

Hi Peter,

Unfortunately I have not been able to (permanently) solve this problem yet.

However, I did write a script that automatically fixes the issue when it arises. At a set interval of time, it probes the public IP address on port 80 to determine whether NAT is working as it should. If the server is not accessible through port 80, the script connects to the router via SSH and sends the necessary commands to remove and re-add the NAT entries.

By the way, I was wrong about only entering the two commands to fix the problem. First, each line must be prefixed with no to remove it, and then they can be added again:

no ip nat source static tcp 10.0.0.15 22 interface FastEthernet0 22
no ip nat source static tcp 10.0.0.20 80 interface FastEthernet0 80
ip nat source static tcp 10.0.0.15 22 interface FastEthernet0 22
ip nat source static tcp 10.0.0.20 80 interface FastEthernet0 80

The script has been working, but it doesn't seem like a very good permanent solution.

Do you think converting things to VRF will solve the problem? As long as I don't lose any functionality, I would be willing to try it.

It puzzles me why NAT hairpinning is so convoluted in Cisco's IOS platform. Sure, some people use the split DNS method, but that isn't always the ideal option. Those $5 Linksys consumer routers do hairpinning right out of the box...

milan.kulik · ‎10-15-2015

Hi Ken,

it might be interesting to run

sh ip nat nvi translations

command at the moment your router is in trouble.

Isn't it possible the NAT translation table is full for some reason?

Best regards,

Milan

Peter Paluch · ‎10-16-2015

Hi Milan,

A good point. Ken, can you temporarily disable the script you have written to automatically remove and reinstate the NVI NAT configuration to have the router eventually fail the NAT? After that, these two commands would be of importance:

show ip nat nvi statistics
show ip nat nvi translations

Best regards,
Peter

P.S.: About the Cisco's inability to do simple hairpin NAT... no comment (sigh).

ken129gem · ‎10-17-2015

Hi Milan and Peter,

Thanks for the suggestions.

Does it matter when I run the commands? Seconds, minutes, hours after the problem begins?

I'm not sure how much time had elapsed before I captured the outputs of the commands.

Here is the output of show ip nat nvi statistics while ports 22 and 80 were inaccessible:

Total active translations: 159 (0 static, 159 dynamic; 159 extended)
NAT Enabled interfaces:
  FastEthernet0, Vlan1
Hits: 3631725  Misses: 30945
CEF Translated packets: 1480410, CEF Punted packets: 1728
Expired translations: 30931
Dynamic mappings:
-- Source [Id: 1] access-list 1 interface FastEthernet0 refcount 157

Here is the output of the same command shortly after I applied the fix:

Total active translations: 90 (0 static, 90 dynamic; 90 extended)
NAT Enabled interfaces:
  FastEthernet0, Vlan1
Hits: 3718993  Misses: 32182
CEF Translated packets: 1516707, CEF Punted packets: 1733
Expired translations: 32239
Dynamic mappings:
-- Source [Id: 1] access-list 1 interface FastEthernet0 refcount 86

Here are the first few lines of show ip nat nvi translations while ports 22 and 80 were inaccessible:

Note: xx.xxx.xxx.xx is the public IP address assigned by the ISP through DHCP.

Pro Source global      Source local       Destin  local      Destin  global
udp xx.xxx.xxx.xx:27   10.0.0.15:138      10.0.0.255:138     10.0.0.255:138
tcp xx.xxx.xxx.xx:80   10.0.0.20:80       76.xx.xxx.137:50232 76.xx.xxx.137:50232
tcp xx.xxx.xxx.xx:80   10.0.0.20:80       86.xxx.xxx.180:36791 86.xxx.xxx.180:36791
udp xx.xxx.xxx.xx:138  10.0.0.20:138      10.0.0.255:138     10.0.0.255:138
tcp xx.xxx.xxx.xx:52735 10.0.0.20:52735   91.xxx.xx.36:80    91.xxx.xx.36:80
...

Notice how the NAT NVI entries for ports 22 and 80 are not present in the output of this command, but they are still shown in show running-config:

ip nat source static tcp 10.0.0.15 22 interface FastEthernet0 22
ip nat source static tcp 10.0.0.20 80 interface FastEthernet0 80

How can remote clients such as 76.xx.xxx.137:50232 and 86.xxx.xxx.180:36791 be connected to the web server on port 80 if no NAT NVI entries are present for ports 22 and 80?

Here are the first few lines of the same command shortly after I applied the fix:

Pro Source global         Source local          Destin  local         Destin  global
tcp xx.xxx.xxx.xx:22      10.0.0.15:22          ---                   ---
tcp xx.xxx.xxx.xx:80      10.0.0.20:80          76.xx.xxx.137:50232   76.xx.xxx.137:50232
tcp xx.xxx.xxx.xx:80      10.0.0.20:80          86.xxx.xxx.180:36791  86.xxx.xxx.180:36791
tcp xx.xxx.xxx.xx:80      10.0.0.20:80          ---                   ---
udp xx.xxx.xxx.xx:36      10.0.0.20:138         10.0.0.255:138        10.0.0.255:138
...

Quick question unrelated to the NAT NVI issue: What are those UDP ports (27, 36, and 138) on the public IP address? 10.0.0.255 is the network's broadcast address and port 138 is NetBIOS, right?

milan.kulik · ‎10-19-2015

Hi Ken,

let me start from the end of your last post:

IMHO, entries like

udp xx.xxx.xxx.xx:36 10.0.0.20:138 10.0.0.255:138 10.0.0.255:138
are caused by your server sending NetBIOS packets to 10.0.0.255 subnet broadcast address. They are NATed by your command

ip nat source list 1 interface FastEthernet0 overload
possibly?

Which is leading to a question though:

What is the subnet mask used on the server and your router port?

Regarding the outputs generally:

Do you see the entries like

tcp xx.xxx.xxx.xx:22 10.0.0.15:22 --- ---

at the moment your NAT is in trouble?

Best regards,

Milan

ken129gem · ‎10-19-2015

Hi Milan,

Thanks for your response.

I was curious as to why those NetBIOS packets were appearing in the NAT NVI translation table. If they are being broadcast throughout the network, where does xx.xxx.xxx.xx:36 (the public IP address) come into place?

The subnet mask is 255.255.255.0, so the network is 10.0.0.0/24.

Entries like

tcp xx.xxx.xxx.xx:22      10.0.0.15:22          ---                   ---

do not appear in show ip nat nvi translations while the problem is occurring. They reappear after the NAT NVI commands are removed and re-added from configuration mode. Please see my previous post for more information.

milan.kulik · ‎10-20-2015

Hi Ken,

I guess the NetBIOS packets were appearing in the NAT NVI translation tablke due to NAT hairpinning used in your network.

How dos the list 1 used in your command

ip nat source list 1 interface FastEthernet0 overload
look like exactly?

I guess entries like

tcp xx.xxx.xxx.xx:22      10.0.0.15:22          ---                   ---

should be present within the NAT translation table all the time - actually those are the static entries created by the

ip nat source static ...

command.

And I believe these entries missing are a symptom of your problem.

Best regards,

Milan

ken129gem · ‎10-21-2015

Hi Milan,

Yes, this is the exact command:

ip nat source list 1 interface FastEthernet0 overload

I agree, those entries should be shown in the NAT NVI translation table during normal operation. They disappear when the problem occurs, and they reappear when the NAT NVI commands are removed and re-added.

Do you have any idea what may be causing this problem or how it can be resolved?

milan.kulik · ‎10-22-2015

Hi Ken,

to be honest, I've got no idea.

I'm running NVI NAT within a VRF on my router without any problem.

Best regards,

Milan

dastrix80 · ‎08-11-2018

I have this exact same issue, were you able to solve this problem?

Regards

Kris

Ivan Sokirko · ‎04-17-2020

I have the same bug. NVI NAT static rule dissapeared after some time from running config.

Cisco 3945e ios c3900e-universalk9-mz.SPA.157-3.M6.bin

NAT NVI entries are being disabled randomly? IOS bug?