GRE Tunnel w/IPSEC over Internet as backup link to PTP - Every other ping fails.

[tkhan]

Hello

We have two offices connected via a T1 point to point.  We have had the link fail, and now would like a backup link over the internet using GRE/IPSEC tunnel.

The tunnel is in place, but every other packet fails. With a ping, 50% packet loss.  Ideally we want a backup route if the PTP fails.  He are the configs for RTR1 and RTR2 as well as the output on the ping.  I get a icmp port destitnation unreachable (something close to that).

Is it possible to have a backup route, that when the ptp fails, traffic automatically routes over the tunnel?

rtr2

!
! Last configuration change at 13:32:43 EDT Tue Jul 20 2010 by admin
! NVRAM config last updated at 11:06:30 EDT Tue Jul 20 2010 by admin
!
version 12.3
no service pad
service timestamps debug datetime msec show-timezone year
service timestamps log datetime msec show-timezone year
service password-encryption
!
hostname RTR2
!

<snip>
!

ip subnet-zero
no ip source-route
no ip gratuitous-arps
ip flow-cache timeout active 1
ip cef
!
<snip>

crypto isakmp policy 1
authentication pre-share
crypto isakmp key <snip> address 1.1.1.1
!
!
crypto ipsec transform-set TRANSFORMS_CRYPTO_192.168.2.0_192.168.3.0 esp-aes esp-md5-hmac
mode transport
!
crypto map CRYPTO_MAP_PTPVPN_<snip> 10 ipsec-isakmp
set peer 1.1.1.1
set transform-set TRANSFORMS_CRYPTO_192.168.2.0_192.168.3.0
match address ACL_CRYPTO_192.168.2.0_192.168.3.0
!
!
<snip>

interface Tunnel0
ip address 172.16.1.1 255.255.255.252
ip tcp adjust-mss 1402
tunnel source FastEthernet0/0
tunnel destination 1.1.1.1
!
interface FastEthernet0/0
description <snip> --> Firewall
ip address 192.168.4.254 255.255.255.0
ip nat outside
ip policy route-map RMAP_1
speed 100
full-duplex
crypto map CRYPTO_MAP_PTPVPN_<snip>
!
interface Serial0/0
bandwidth 1536
ip address 172.16.2.1 255.255.255.252
no ip redirects
no ip proxy-arp
ip nat inside
encapsulation ppp
ip ospf network point-to-point
logging event subif-link-status
down-when-looped
fair-queue
service-module t1 timeslots 1-24
!
<snip>

!
router ospf 100
log-adjacency-changes
passive-interface Loopback0
<snip>
network 172.16.2.0 0.0.0.255 area 0.0.0.0
network 192.168.3.0 0.0.0.255 area 0.0.0.0
<snip>
!
<snip>

ip route 0.0.0.0 0.0.0.0 192.168.4.1 10
ip route 192.168.2.0 255.255.255.0 Tunnel0 200
<snip>
!
!
!
ip access-list extended ACL_CRYPTO_192.168.2.0_192.168.3.0
remark #######################################################
remark Permit backup VPN traffic 192.168.3.0 and 192.168.2.0
remark #######################################################
remark
permit gre host 172.16.1.1 host 172.16.1.2
permit ip 192.168.3.0 0.0.0.255 192.168.2.0 0.0.0.255
remark
remark #######################################################
remark Deny All
remark #######################################################
deny   ip any any
remark

==========

rtr1

!
! Last configuration change at 13:32:49 EDT Tue Jul 20 2010 by admin
! NVRAM config last updated at 11:06:28 EDT Tue Jul 20 2010 by admin
!
version 12.3
service timestamps debug datetime msec show-timezone year
service timestamps log datetime msec show-timezone year
service password-encryption
!
hostname RTR1
!
boot-start-marker
boot-end-marker

!
memory-size iomem 15

!
!
<snip>
no ip subnet-zero
no ip source-route
no ip gratuitous-arps
ip flow-cache timeout active 5
!
!
ip cef
<snip>
!
<snip>
controller T1 0/0
framing esf
linecode ami
!
controller T1 0/1
framing sf
linecode ami
!
ip ssh time-out 60
ip ssh source-interface FastEthernet0/0
!
!
crypto isakmp policy 1
authentication pre-share
crypto isakmp key <snip>address 2.2.2.2
!
!
crypto ipsec transform-set TRANSFORMS_CRYPTO_192.168.2.0_192.168.3.0 esp-aes esp-md5-hmac
mode transport
!
crypto map CRYPTO_MAP_PTPVPN_<snip> 10 ipsec-isakmp
set peer 2.2.2.2
set transform-set TRANSFORMS_CRYPTO_192.168.2.0_192.168.3.0
match address ACL_CRYPTO_192.168.2.0_192.168.3.0
!
!
!
!
interface Loopback0
ip address 2.2.16.172 255.255.255.0
!
interface Tunnel0
ip address 172.16.1.2 255.255.255.252
ip tcp adjust-mss 1402
tunnel source FastEthernet0/1
tunnel destination 2.2.2.2
!
<snip>
!
interface FastEthernet0/1
<snip>
ip address 192.168.0.254 255.255.255.0
ip nat outside
ip flow ingress
ip route-cache flow
speed 100
full-duplex
crypto map CRYPTO_MAP_PTPVPN_<snip>
!
interface Serial0/2
<snip>
bandwidth 1536
ip address 172.16.2.2 255.255.255.252
no ip redirects
no ip proxy-arp
ip nat inside
ip flow ingress
encapsulation ppp
ip route-cache flow
ip ospf network point-to-point
logging event subif-link-status
fair-queue
!
<snip>
router ospf 100
log-adjacency-changes
network 172.16.2.0 0.0.0.255 area 0.0.0.0
network 192.168.0.0 0.0.0.255 area 0.0.0.0
network 192.168.2.0 0.0.0.255 area 0.0.0.0

!
<snip>
!
<snip>
ip route 0.0.0.0 0.0.0.0 192.168.0.1 20
ip route 0.0.0.0 0.0.0.0 192.168.3.1 100
ip route 192.168.3.0 255.255.255.0 Tunnel0 200
ip route 192.168.32.0 255.255.255.0 192.168.3.1
!
!
!
ip access-list extended ACL_CRYPTO_192.168.2.0_192.168.3.0
remark #######################################################
remark Permit backup VPN traffic 192.168.3.0 and 192.168.2.0
remark #######################################################
remark
permit gre host 172.16.1.2 host 172.16.1.1
permit ip 192.168.2.0 0.0.0.255 192.168.3.0 0.0.0.255
remark
remark #######################################################
remark Deny All
remark #######################################################
deny   ip any any
remark

Output of Ping

Reply from 172.16.1.2: bytes=32 time=17ms TTL=254
Request timed out.
Reply from 172.16.1.2: bytes=32 time=17ms TTL=254
Request timed out.
Reply from 172.16.1.2: bytes=32 time=17ms TTL=254
Request timed out.
Reply from 172.16.1.2: bytes=32 time=17ms TTL=254
Request timed out.
Reply from 172.16.1.2: bytes=32 time=17ms TTL=254
Request timed out.
Reply from 172.16.1.2: bytes=32 time=17ms TTL=254
Request timed out.
Reply from 172.16.1.2: bytes=32 time=21ms TTL=254
Request timed out.
Reply from 172.16.1.2: bytes=32 time=17ms TTL=254
Request timed out.
Reply from 172.16.1.2: bytes=32 time=17ms TTL=254
Request timed out.
Reply from 172.16.1.2: bytes=32 time=17ms TTL=254
Request timed out.
Reply from 172.16.1.2: bytes=32 time=17ms TTL=254
Request timed out.
Reply from 172.16.1.2: bytes=32 time=17ms TTL=254
Request timed out.
Reply from 172.16.1.2: bytes=32 time=17ms TTL=254
Request timed out.
Reply from 172.16.1.2: bytes=32 time=17ms TTL=254
Request timed out.
Reply from 172.16.1.2: bytes=32 time=18ms TTL=254
Request timed out.
Reply from 172.16.1.2: bytes=32 time=19ms TTL=254
Request timed out.
Reply from 172.16.1.2: bytes=32 time=17ms TTL=254
Request timed out.
Reply from 172.16.1.2: bytes=32 time=17ms TTL=254
Request timed out.
Reply from 172.16.1.2: bytes=32 time=17ms TTL=254
Request timed out.
Reply from 172.16.1.2: bytes=32 time=17ms TTL=254
Request timed out.
Reply from 172.16.1.2: bytes=32 time=17ms TTL=254
Request timed out.
Reply from 172.16.1.2: bytes=32 time=17ms TTL=254

[vmiller]

Try this. change the admin distance on your backup link to

20. Your symptoms indicate asynchronus routing.

[tkhan]

I tried that, and then more traffic went over that link, and timed out as well.

The reason I set it at admin dist 200 is to be higher than the ospf route that is advertised over the serial link.

I tried it again, and still have the 50% failure.

[Mohamed Sobair]

Hi,

There are multiple points that I would correct in your configuration after analyzing it:

1- if the primary link fails, the tunnels destination in both sides are learned via the tunnel itself, this would result in recursive routing lookup and you will be having flapping on the second link (Tunnel link).

   I recommend you set 2 static routes with higher AD value than the OSPF as bellow:

   a- ip route 1.1.1.1 255.255.255.255 F0/1 120 (on the second router)

   b- ip route 2.2.2.2 255.255.255.255 f0/0 120 (on the 1st router)

2- I also recommend setting the command (IP MTU 1400) on both tunnel interfaces on both routers to prevent fragmentation.

    Fragmentation is not desired and recommended, fragmentation can cause delays , the default timeout in a Cisco router is 2 sec which means if the delay is larger than those 2 sec , you will see .... as ping result instead of !!! as asuccessful ping result.

let us know if the suggested points solves your problem,

Mohamed

[Richard Burts]

When every other ping fails as in your case the usual cause is that there are 2 routes in the routing table for the destination and only one of them works. So the first thing that I would suggest would be to post the output show how ip route 172.16.1.2 and we will see if there are 2 paths and what they are.

I also see several things in your config that need to be fixed.

- with GRE tunnels the source and destination addresses need to be the inverse of the other side (the source of one needs to be the destination of the other) but your tunnels do not do that. rtr2 has the source as 192.168.4.254and destination as 1.1.1.1 while rtr1 has the source as  192.168.0.254 and the destination as 2.2.2.2

- for the tunnel to come up the router must know a route to the tunnel destination that is independent of the tunnel iteslf. It is not clear from the posted config that the routers do know routes to 1.1.1.1 and 2.2.2.2.

- the access list used to identify traffic to be protected by VPN is flawed. You have:

permit gre host 172.16.1.1 host 172.16.1.2
permit ip 192.168.3.0  0.0.0.255 192.168.2.0 0.0.0.255

but the first permit gre should specify the tunnel source and destination not the tunnel addresses.

and you do not need the second permit since going through the tunnel neither the source nor destination will be 192.168.2.0 or 192.168.3.0 (they would be the source and destination of the original packet that is encapsulated by GRE).

- the set peer statement in the crypto map indicates that the peers are 1.1.1.1 and 2.2.2.2 but there is no indication that these are addresses on the routers.

HTH

Rick

[tkhan]

Rick,

Thanks so much.  Let me clarify a bit.

* The 1.1.1.1 address is an external IP that is NAT'd when it it passed by firewall at remote end.  THe same with 192.168.4.254 (It gets NAT'd to a public IP after leaving the Firewall)

* There is a 1 to 1 NAT for 1.1.1.1 to a public IP, and 2.2.2.2 to a public IP on my firewalls.

   - 2.2.2.2 gets nat'd to 192.168.4.254 and 1.1.1.1 gets nat'd to 192.168.0.254

* Since 1.1.1.1 a public IP, it goes out the def gw, so I assume no route needed

* I fixed the ACL to be permit GRE host 192.168.4.254 host 1.1.1.1 (Public IP)

* I fixed the ACL to be permit GRE HOST 192.168.0.254 host 2.2.2.2 (Public IP)

Questions:

* Do I have to add a static route for the 192.168.2/3.0 networks over the tunnel? Currently via the PTP link OSPF advertises these networks, is it possible to have the routing protocol advertise a second, less preferable route instead of static routes?

RTR2

crypto isakmp policy 1
authentication pre-share
crypto isakmp  key address 1.1.1.1 (This is the external IP of the other router via the Internet)
!
!
crypto ipsec transform-set  TRANSFORMS_CRYPTO_192.168.2.0_192.168.3.0 esp-aes esp-md5-hmac
mode transport
!
crypto map CRYPTO_MAP_PTPVPN_ 10  ipsec-isakmp
set peer 1.1.1.1 (This is the external IP of the other router via the Internet)
set transform-set  TRANSFORMS_CRYPTO_192.168.2.0_192.168.3.0
match address  ACL_CRYPTO_192.168.2.0_192.168.3.0
!
!

interface  Tunnel0

ip address 172.16.1.1 255.255.255.252

ip tcp adjust-mss  1402

tunnel source FastEthernet0/0

tunnel destination 1.1.1.1 (This is the external IP of the other router via the  Internet)

!

interface  FastEthernet0/0

description --> Firewall

ip  address 192.168.4.254 255.255.255.0

ip nat outside

ip policy  route-map RMAP_1

speed 100

full-duplex

crypto map  CRYPTO_MAP_PTPVPN_

[tkhan]

I have actually slimmed it down and now and just trying to get the GRE tunnel to work without IPSEC.  Just for clarification I am going to past the tunnel info, as well as the NAT'd addresses.

rtr 1

interface Tunnel0
ip address 172.16.1.2 255.255.255.2
ip mtu 1400
tunnel source FastEthernet0/1
tunnel destination 65.220.77.1

65.220.77.1

interface FastEthernet0/1
description External Interface --> Firewall
ip address 192.168.0.254 255.255.255.0
ip nat outside
ip flow ingress
ip route-cache flow
speed 100
full-duplex

FW NAT RULES

Source=65.220.77.1 - dest=173.17.122.1 --> Src=orig - dest=192.168.0.254

Source=192.168.0.254 - dest=65.220.77.1 --> src=173.17.122.1 - dest=orig

FW ALLOW GRE, IKE,IKE_NAT_TRAVERSAL(udp4500)

rtr 2

interface Tunnel0
ip address 172.16.1.1 255.255.255.252
ip mtu 1400
tunnel source FastEthernet0/0
tunnel destination 173.17.122.1

interface FastEthernet0/0
description External Interface --> Firewall
ip address 192.168.4.254 255.255.255.0
ip nat outside
speed 100
full-duplex

FW NAT RULES

Source=173.17.122.1 - dest=65.220.77.1 --> Src=orig -  dest=192.168.4.254

Source=192.168.4.254 - dest=173.17.122.1 --> src=65.220.77.11 - dest=orig

FW ALLOW GRE, IKE,IKE_NAT_TRAVERSAL(udp4500)

EDIT - The GRE Tunnel works just fine.  I added the crypto maps back

[Mohamed Sobair]

Rick,

Point 1:

You are indeed correct if he has more than one path to the 172.16 Network, however, from his config.. he should only have one path through the OSPF as primary and floating static with higher AD as asecondary.

Point 2:

If his primary link fails, the tunnel destination would be learned via the tunnel itself , this will result in recursive routing lookup and flapping on th tunnel , So it must be corrected , Dont you agree?

Point 3:

I have suggested setting the IP MTU on the tunnel, Dont you agree also this could be a source of a problem?

HTH

Mohamed

[Richard Burts]

Mohamed

1) It would be logical to assume that OSPF would have a single route. But we do not know and that is why I asked that he post output of show ip route so that we would know for sure what the router has got for routes to that destination.

2) the tunnel destination must always be reached by a path independent of the tunnel. If the route to the tunnel destination is ever through the tunnel then it does cause recursive problems and IOS will shut down the tunnel.

3) I believe that IP MTU is nice to have. But I think that the tunnel could work without it. And in his recent post he has added IP MTU on the tunnels.

HTH

Rick

[tkhan]

Output of show ip route:

rtr2

     1.0.0.0/32 is subnetted, 1 subnets
C       1.2.16.172 is directly connected, Loopback0
C    192.168.31.0/24 is directly connected, Ethernet1/0
     65.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
O       88.xx.xx.232/30 [110/67] via 172.16.2.2, 00:02:15, Serial0/0
S       88.xx.xx.1/32 [1/0] via 192.168.4.1
     172.16.0.0/16 is variably subnetted, 4 subnets, 2 masks
C       172.16.2.2/32 is directly connected, Serial0/0
C       172.16.1.0/30 is directly connected, Tunnel0
C       172.16.2.0/30 is directly connected, Serial0/0
C       172.16.100.0/30 is directly connected, Ethernet1/3
O    192.168.200.0/24 [110/2] via 192.168.4.1, 00:02:15, FastEthernet0/0
C    192.168.4.0/24 is directly connected, FastEthernet0/0
O    192.168.20.0/24 [110/66] via 172.16.2.2, 00:02:15, Serial0/0
     209.xx.xx.0/24 is variably subnetted, 2 subnets, 2 masks
S       209.xx.xx.0/24 [10/0] via 192.168.0.254
S       209.xx.xx.1/32 [1/0] via 192.168.4.1
O    192.168.0.0/24 [110/66] via 172.16.2.2, 00:02:15, Serial0/0
C    192.168.1.0/24 is directly connected, Ethernet1/2
O    192.168.2.0/24 [110/66] via 172.16.2.2, 00:02:15, Serial0/0 (Not the tunnel)
O    192.168.32.0/24 [110/20] via 172.16.100.2, 00:02:15, Ethernet1/3
C    192.168.3.0/24 is directly connected, FastEthernet0/1
O    192.168.33.0/24 [110/20] via 172.16.100.2, 00:02:15, Ethernet1/3
O    192.168.101.0/24 [110/67] via 172.16.2.2, 00:02:15, Serial0/0
S*   0.0.0.0/0 [10/0] via 192.168.4.1

rtr1

     2.0.0.0/24 is subnetted, 1 subnets
C       2.2.16.0 is directly connected, Loopback0
O    192.168.31.0/24 [110/75] via 172.16.2.1, 00:03:29, Serial0/2
    88.0.0.0/30 is subnetted, 1 subnets
O       88.xx.108.232 [110/2] via 192.168.0.1, 00:03:29, FastEthernet0/1
     172.16.0.0/16 is variably subnetted, 4 subnets, 2 masks
C       172.16.1.0/30 is directly connected, Tunnel0
C       172.16.2.0/30 is directly connected, Serial0/2
C       172.16.2.1/32 is directly connected, Serial0/2
O       172.16.100.0/30 [110/75] via 172.16.2.1, 00:03:29, Serial0/2
O    192.168.200.0/24 [110/67] via 172.16.2.1, 00:03:29, Serial0/2
O    192.168.4.0/24 [110/66] via 172.16.2.1, 00:03:29, Serial0/2
C    192.168.20.0/24 is directly connected, FastEthernet1/1
C    192.168.0.0/24 is directly connected, FastEthernet0/1
O    192.168.1.0/24 [110/75] via 172.16.2.1, 00:03:29, Serial0/2
C    192.168.2.0/24 is directly connected, FastEthernet0/0
S    192.168.32.0/24 [1/0] via 192.168.3.1
O    192.168.3.0/24 [110/66] via 172.16.2.1, 00:03:29, Serial0/2 (not the tunnel)
O    192.168.33.0/24 [110/85] via 172.16.2.1, 00:03:29, Serial0/2
O    192.168.101.0/24 [110/2] via 192.168.0.1, 00:03:29, FastEthernet0/1
S*   0.0.0.0/0 [20/0] via 192.168.0.1

Anything else I forgot to post?

Just an FYI, I am now getting no reply back from a ping, maybe it means I am closer

[Richard Burts]

If you are now getting no response then the symptoms have certainly changed. The posted routing table is clear that there is only a single path to 172.16.1.1-2 and that path is through the tunnel which sounds like an improvement.

What does the output of show crypto map look like?

HTH

Rick

[tkhan]

Crypto Map "CRYPTO_MAP_PTPVPN_" 10 ipsec-isakmp
        Peer =  173.17.122.1
        Extended IP access list ACL_CRYPTO_192.168.2.0_192.168.3.0
            access-list ACL_CRYPTO_192.168.2.0_192.168.3.0 permit gre host 192.1
68.4.254 host  173.17.122.1
            access-list ACL_CRYPTO_192.168.2.0_192.168.3.0 deny ip any any
        Current peer:  173.17.122.1
        Security association lifetime: 4608000 kilobytes/3600 seconds
        PFS (Y/N): N
        Transform sets={
                TRANSFORMS_CRYPTO_192.168.2.0_192.168.3.0,
        }
        Interfaces using crypto map CRYPTO_MAP_PTPVPN_:
                Tunnel0
                FastEthernet0/0

====

Crypto Map "CRYPTO_MAP_PTPVPN_" 10 ipsec-isakmp
        Peer = 65.220.77.1
        Extended IP access list ACL_CRYPTO_192.168.2.0_192.168.3.0
            access-list ACL_CRYPTO_192.168.2.0_192.168.3.0 permit gre host 192.1
68.0.254 host 65.220.77.1
            access-list ACL_CRYPTO_192.168.2.0_192.168.3.0 deny ip any any
        Current peer: 65.220.77.1
        Security association lifetime: 4608000 kilobytes/3600 seconds
        PFS (Y/N): N
        Transform sets={
                TRANSFORMS_CRYPTO_192.168.2.0_192.168.3.0,
        }
        Interfaces using crypto map CRYPTO_MAP_PTPVPN_:
                Tunnel0
                FastEthernet0/1

[Richard Burts]

Would I assume from the output that you have configured the crypto map on both the physical interface and the tunnel interface? On old versions of IOS this was required but on more recent IOS the crypto map should be only on the physical interface (not on the tunnel). Try removing the crypto map from the tunnel and see if the behavior changes.

HTH

Rick

[tkhan]

Tried that before. Just tried it again.  All to no avail.

When I shut down the s0/0 interface (point to point) I do see the route come up for the 192.168.2.0 network.  I have the route as:

ip route 192.168.2.0 255.255.255.0 tunnel 0 200

should it be to the fast 0/0 interface?  I am sure I tried this, but will again.  Also, when I shut the interface down (s0/0) and with a debug of ipsec, I don't see anything.  It doesn't appear that the traffic is interesting enough. I assume that when I set the route between the tunnel interfaces, that should set off the ipsec, correct?

Added the access list:

Extended IP access list ACL_CRYPTO_192.168.2.0_192.168.3.0
    10 permit gre host 192.168.4.254 host 173.17.122.1 (1110 matches) (This number does not change when the int s0/0 is dropped)

I was hoping it would set off with the traffic.

Do I have the correct ACL?

[Richard Burts · Accepted solution]

I am finding it a bit difficult to keep up with how things have changed. So let me suggest a few things:

- verify that the GRE tunnels do work. If you take away the crypto map and are just sending traffic in the clear do the GRE tunnels work. I would suggest one good way to check is to do traceroute on one router to the tunnel interface address of the other router. (traceroute 172.16.1.2). If you get a response and if the response shows only one hop and if the responding  address is the other router tunnel interface then you have verified that the GRE tunnel is working.

- once you are sure that the GRE is working without encryption then add the crypto map back into the config.

- the crypto map should go on the physical interface and not on the tunnel interface (unless you are running quite old code).

- the access list for crypto should essentially say permit gre host host . It gets tricky figuring how the address translation works with this. So I might configure multiple lines in the access list trying both the translated and the non-translated versions of the addresses. Then look to see which one is getting hits.

- when the crypto config is in place do show crypto isakmp sa. Look to see if SAs are negotiated.

- there might not be an ISAKMP SA yet, since it is negotiated when there is interesting traffic. So generate some traffic that will go through the tunnel. The easy way is to ping the tunnel address of the other router (ping 172.16.1.2).

- check for crypto isakmp sa again. If the ISAKMP SA is negotiated that is good and move on to check for IPSec. If the ISAKMP SA is still not negotiated there was some problem in the negotiation. try debug crypto isakmp to try to find the problem.

- if ISAKMP SA was negotiated then check for IPSec SA (show crypto ipsec sa).

HTH

Rick

[edit] and the static route should point to the tunnel not to the physical interface.