cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
6349
Views
10
Helpful
14
Replies

DMVPN and INTERNET VIA HUB LOCATION ISSUES

lap
Explorer
Explorer

Hello everyboby,

I really wish that you can help me with the issue I have.

I explain. I have to test a Dual Hub - Dual DMVPN Layout for a customer before we configure it in real production.
The customer has some sites where routers are behind some ISP routers which are doing NAT.

Drawing-Lab-Setup.jpeg

How things are configured:

- All the traffic from spokes has to go via the Hub location so no local internet traffic on spokes.
- Hub 1 and Hub 2 sends a default route to spokes via EIGRP. But only Hub 1 is used.
- Hub 1 is the primary router for DMVPN. In case of hardware/Connection to Internet failure Hub 2 become active for DMVPN and Internet.
- Hub 1 and Hub 2 are both connected to one ISP and are Internet Gateway for spokes.
- Hub 1 and Hub 2 are configured with IOS Firewall.
- On spokes I have used VRF to seperate DMVPN routning table from Global routning table so I could receive a default route from Hub 1 and Hub 2 to route traffic from spokes to Internet via Hub location

What is working:

- All spokes can have access to local LAN at Hub location.
- All spokes can do spoke to spoke
- Failover working for DMVPN
- Spokes NOT behind NAT ISP router (that is to say having the public IP address directly attached at their outside interface) can go to Internet via hub location and all packets are inspected correctly by the IOS firewall and Nat correctly
 
What is not working:

- Spokes behind NAT ISP router cannot reach the Internet through Hub location. They can only reach local LAN at Hub location and do spoke to spoke.
  On hub router the IOS firewall sees the packets comming from theses spokes (behind NAT) with a source IP address which is the public IP address og the ISP router outside interface. Not the LAN private IP address behind spoke.
  Moreover packets are never natted. If I do some snifing on an Internet server the source private IP address is the LAN IP address of the LAN behind the spoke. That means that the Hub router never nat these packets.

 
How to solve this problem?

Well I don't know that is why I need your help/advices :-)

I don't know if I should configure a VRF on the hub location also as maybe things gets mess up.

The problem seems to be coming from NAT-T as the spokes which aren't behind NAT can find go on the Internet through Hub and both Cisco IOS inspection and NAT are working find.

As I was testing today with the customer at the begining the spoke behind nat could ping different server on the Internet but not open a HTTP session. DNS was working find. The IOS Firewall was actually

inspecting packets with the real private IP addresse. Then I thought that it was a MTU issue so I decided to ping out the Internet with bigger MTU size and suddenly the pings were not going through anymore.

I could see on the Hub1 router that the IOS firewall was inspecting the public IP address again of the ISP NAT router at spoke side and not anymore the real private IP address. Really strange!

Attached files:

I attach the following files: a drawing of the setup called Drawing-Lab-Setup.jpeg | All the configs files for HUB1, BRANCH1, BRANCH2 and ISP-ROUTER named respectively: HUB1.txt, BRANCH1.txt, BRANCH2.txt and ISP-ROUTER.txt

Logs from Hub1 When pinging host 200.200.200.200 on the Internet from Branch2 (behind NAT ISP router):

BRANCH2#ping vrf DMVPN-VRF 200.200.200.200 source vlan 100

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 200.200.200.200, timeout is 2 seconds:
Packet sent with a source address of 192.168.110.1
.....
Success rate is 0 percent (0/5)

*Jul 15 06:04:51.017 UTC: %FW-6-SESS_AUDIT_TRAIL_START: Start icmp session: initiator (110.10.10.2:8) -- responder (200.200.200.200:0)

So the IOS firewall doesn't inspect the real source private IP adresse which should be in this case: 192.168.110.2. It sess on the public IP address.

HUB1#sh ip nat translations
Pro Inside global      Inside local       Outside local      Outside global
icmp 80.10.10.2:1      80.10.10.2:1       100.10.10.2:1      100.10.10.2:1
icmp 80.10.10.2:2      80.10.10.2:2       110.10.10.2:2      110.10.10.2:2
udp 80.10.10.2:4500    80.10.10.2:4500    110.10.10.2:4500   110.10.10.2:4500

There is no NAT entries present for thes packets

Snifing on Tunnel 1 interface on Hub1 (packets comming in):

7    7.355997    192.168.110.1    200.200.200.200    ICMP    Echo (ping) request
So although the IOS firewall inspect on the public IP 110.10.10.2:8 the sniffing capture says that the packet are comming from the real private IP address

Sniffing on the server (200.200.200.200) with wireshark:

114    14.123552    192.168.110.1    200.200.200.200    ICMP    Echo (ping) request

So the source private IP address from BRANCH2 LAN is never natted by HUB1

So the server sees the source private IP address not natted although Hub1 IOS firewall inspect the public IP address 110.10.10.2:8


Logs from Hub1 When pinging host 200.200.200.200 on the Internet from Branch1 (Not behind NAT ISP router):

BRANCH1#ping vrf DMVPN-VRF 200.200.200.200 source vlan 100

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 200.200.200.200, timeout is 2 seconds:
Packet sent with a source address of 192.168.100.1
!!!!!

*Jul 15 06:05:18.217 UTC: %FW-6-SESS_AUDIT_TRAIL_START: Start icmp session: initiator (192.168.100.1:8) -- responder (200.200.200.200:0)

So here the firewall sees the real private IP address which is 192.168.100.1

HUB1#sh ip nat translations
Pro Inside global      Inside local       Outside local      Outside global
icmp 80.10.10.2:1      80.10.10.2:1       100.10.10.2:1      100.10.10.2:1
icmp 80.10.10.2:2      80.10.10.2:2       110.10.10.2:2      110.10.10.2:2
udp 80.10.10.2:4500    80.10.10.2:4500    110.10.10.2:4500   110.10.10.2:4500
icmp 80.10.10.2:22     192.168.100.1:22   200.200.200.200:22 200.200.200.200:22

The source real private IP address is also find natted to Hub 1 outside public IP address

Snifing on Tunnel 1 interface on Hub1 (packets comming in):

8    7.379997    192.168.100.1    200.200.200.200    ICMP    Echo (ping) request

Same Real private IP address as inspected by IOS firewall so everything is find there.

Sniffing on the server (200.200.200.200) with wireshark:

67                  10.441153     80.10.10.2     200.200.200.200                 ICMP            Echo (ping) request

So here everything is fine. The address is natted correctly.

__________________________________________________________________________________________

Best Regards,

Laurent

3 Accepted Solutions

Accepted Solutions

Hi,

Just saw your post, hopefully it is not too late.

I am not sure what your exact problem is, but I think we can work through it to figure it out.

One thing I did notice was that your NAT ACL is too general. You need to make it more

specific.  In particular you want to make sure that it doesn't match the VPN traffic coming

in to / going out of the router.

For example you shouldn't really have any of these entries in your NAT translation table.

HUB1#sh ip nat translations
Pro Inside global      Inside local       Outside local      Outside global
icmp 80.10.10.2:1      80.10.10.2:1       100.10.10.2:1      100.10.10.2:1
icmp 80.10.10.2:2      80.10.10.2:2       110.10.10.2:2      110.10.10.2:2
udp 80.10.10.2:4500    80.10.10.2:4500    110.10.10.2:4500   110.10.10.2:4500

Instead of using:

ip access-list extended Nat
deny   ip any 192.168.0.0 0.0.255.255 log
permit ip any any
deny   ip any any log

Can you use:

ip access-list extended Nat
  deny   ip 192.168.0.0 0.0.255.255 192.168.0.0 0.0.255.255 log
  permit ip 192.168.0.0 0.0.255.255 any
  deny   ip any any log

Also I would be very careful with using the 'log' keyword in a NAT ACL.

I have seen it cause problems.

What IOS versions are you using?

Try making the changes to NAT so that you no longer see NAT translation entries

for the NAT-T (UDP 4500) packets in the NAT translation table on the hub. It may be

that this is setting a flag on the packet structure, that IOS-Firewall and then NAT is

picking up on and then doing the wrong thing in this case.

If this doesn't work then let me know.

This may be something for which you will have to open a TAC case so that we can

debug this directly on your setup.

Mike.

View solution in original post