Issue routing across GRE Tunnel

[clamasters]

My company uses GRE between our POP routers for management purposes.  We have a NMS that sits on a public IP at our head end to monitor all remote nodes.  The issue I'm having is with 3620 routers, for some reason, I do not get any return traffic from the remote nodes to our NMS, therefore we are showing an outage.  My configuration and details are below.  I'm stumped.

## Remote Router ##

interface Tunnel 100

ip address 172.16.100.2 255.255.255.252

tunnel source serial 0/0

tunnel destination 72.x.x.250

interface Serial 0/0

ip address 12.x.x.100 255.255.255.252

interface FastEthernet 0/0

ip address 10.0.0.1 255.255.255.0

ip route 72.x.x.220 255.255.255.255 tunnel 100 name NMS

ip route 72.x.x.221 255.255.255.255 tunnel 100 name NMS

ip route 10.254.254.0 255.255.255.0 tunnel 100 name NOC

## Head End Router ##

interface Tunnel 100

ip address 172.16.100.1 255.255.255.252

tunnel source f0/0

tunnel destination 12.x.x.100

interface FastEthernet 0/0

ip address 72.x.x.250 255.255.255.0

interface FastEthernet 0/1

ip address 10.254.254.1 255.255.255.0

ip route 10.0.0.0 255.255.255.0 tunnel 100 name Remote_Nodes

--------------------

The issue is that from our internal network we can connect to the remote nodes without a problem.  However, the NMS which is at 72.x.x.220 is not able to.  From the remote router I can ping the NMS and a traceroute shows that it goes through the tunnel like it's supposed to.

I'm hoping someone has ran into this before.  If you need more information, just let me know.

[clamasters]

Oh yeah, IOS 12.1(5)T12 if that helps.

[clamasters]

Nobody huh.  I have now tried an IOS update without success.  I guess I'll be buying 23 routers to replace these.

[Edison Ortiz]

What's the NMS's gateway? Is it 72.x.x.250?

From the remote location devices, the packet is sourced by the internal subnet 10.0.0.x and it knows via the routing table from the edge router to go via the tunnel.

However, at the Head End Router, the NMS must send the packet destined to 10.0.0.x back to your Head End Router. If the NMS's gateway is another router then the return packet is lost.

You can ping from the remote router but the response is asymmetrical. The remote router sends the packet via the tunnel because it has it on the routing table, however - the source ip is not the 10.0.0.x subnet - it uses the routable IP 12.x.x.100. The NMS uses that IP on the ping reply via the internet.

[clamasters]

NMS is 72.x.x.220.  Path is below.

72.x.x.220 (NMS) --> 72.x.x.250 (Cisco 3745 f0/0) --> 172.16.100.1 (Cisco 3745 Tun100) --> 172.16.100.2 (Cisco 3620 Tun100) --> 10.0.0.1 (Cisco 3620 f0/0) --> 10.0.0.13 (Remote Node)

NMS default gateway as shown above is .250.  On the Cisco 3745 there is a route for the 10.0.0.0/24 network to go to 172.16.100.2 which is the remote router GRE interface.  Then the 10.0.0.0/24 network is directly connected to the 3620.

Coming the other direction, the remote node (IP 10.0.0.13) has a default gateway of 10.0.0.1.  The router has a route to the 72.x.x.220/32 host to go to 172.16.100.1 which is the 3745 Tun 100 IP.

In summary, all gateway's are symetrical.

[Edison Ortiz]

Can you provide full configs

(sanitized of course) and the routing table from both routers?

[clamasters]

Please see attached.  They have been sanitized, but should be an accurate representation.  As stated, this issue is only with 3620 routers.  All other routers seem to be working with this configuration. 

Thanks

[Edison Ortiz]

I don't see any issues with the config. You mentioned is only happening with the 3620 routers and others are working fine.

Try upgrading the IOS to the latest supported version for such hardware or just replace the hardware.

[clamasters]

Thanks.  I'm going to be ordering a few today and getting these replaced.

[timothy.tam]

The problem looks like related to NAT.

Can try to amend the ACL in remote site as follow to try again?

- Amend the ip nat command ACL to 101.

- Add ACL 101 as follow.

Good luck!!

Command for reference:

ip nat inside source list 101 pool NAT overload

access-list 101 deny ip 10.0.0.0 0.0.0.255 72.x.x.0 0.0.0.255

access-list 101 permit ip 10.0.0.0 0.0.0.255 any any