GRE Over ISPEC not working

[Infrastructure Team]

Hi all, Hope you can help.

I'm trying to get GRE for eigrp working over an ipsec tunnel.  There's an IPSEC tunnel from a 1921 to an ASA.  Then a GRE tunnel over this from loopback on the 1921 to loopback on a 6800.  Debug's for GRE show packets going both ways.  Tunnel on the 6800 shows up/up, Tunnel on 1921 shows up/down.  

6800 pings local loopback and tunnel, but not the remote

1921 pings it's local loopback but not the tunnel (because it's down) and not the remote.  I think I just need to get the tunnel interface up, but can't figure out how.  Simulation in GNS3 (different IOS unfortunately) brings the tunnel interface up as soon as I put a static route for the tunnel destination.

Thanks for any help.

Diagram and configs as follows.

1921 CONFIG/Tu-INT /GRE debug/CRYPTO IPSEC is below

version 15.4
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname r-1921
!
boot-start-marker
boot-end-marker
!
!
logging buffered 51200 warnings
!
aaa new-model
!
!
!
!
!
!
!
aaa session-id common
memory-size iomem 5
!
!
!
!
!
!
no ip icmp rate-limit unreachable
!
!
!
!
!
!
!
!
ip domain name ??????
ip name-server ??????
ip name-server ??????
ip cef
no ipv6 cef
!
multilink bundle-name authenticated
!
cts logging verbose
!
crypto pki trustpoint TP-self-signed-600784525
enrollment selfsigned
subject-name cn=IOS-Self-Signed-Certificate-600784525
revocation-check none
rsakeypair TP-self-signed-600784525
!
!
crypto pki certificate chain TP-self-signed-600784525
certificate self-signed 01
??????
quit
license udi pid CISCO1921/K9 sn ??????
!
!
username ??????
!
redundancy
!
!
!
!
no cdp log mismatch duplex
!
ip tcp synwait-time 5
ip ssh version 1
!
!
crypto isakmp policy 10
encr aes 256
authentication pre-share
group 5
lifetime 28800
crypto isakmp key ?????? address asa-outside-address
crypto isakmp keepalive 10
!
!
crypto ipsec transform-set s2s-tset esp-aes 256 esp-sha-hmac
mode tunnel
no crypto ipsec nat-transparency udp-encapsulation
!
!
!
crypto map s2s-map 10 ipsec-isakmp
set peer asa-outside-address
set transform-set s2s-tset
set pfs group5
match address gre-tun
!
!
!
!
!
interface Loopback0
ip address 192.168.2.4 255.255.255.255
!
interface Tunnel5
ip address 192.168.3.10 255.255.255.252
ip mtu 1400
ip tcp adjust-mss 1300
keepalive 10 3
tunnel source Loopback0
tunnel destination 192.168.2.1
tunnel path-mtu-discovery
!
interface Embedded-Service-Engine0/0
no ip address
shutdown
!
interface GigabitEthernet0/0
description outside
ip address dhcp
ip access-group acl-in in
ip nat outside
ip virtual-reassembly in
duplex auto
speed auto
crypto map s2s-map
!
interface GigabitEthernet0/1
description inside
ip address 10.45.0.254 255.255.255.128
ip nat inside
ip virtual-reassembly in
duplex auto
speed auto
!
!
router eigrp 6669
network 10.45.0.128 0.0.0.127
network 192.168.3.8 0.0.0.3
eigrp stub connected
no eigrp log-neighbor-changes
!
ip forward-protocol nd
!
no ip http server
ip http access-class 23
ip http authentication local
no ip http secure-server
ip http timeout-policy idle 60 life 86400 requests 10000
!
ip nat inside source list nat-list interface GigabitEthernet0/0 overload
ip route 0.0.0.0 0.0.0.0 192.168.0.1
ip route 192.168.2.1 255.255.255.255 192.168.0.1
!
ip access-list extended gre-tun
permit gre host 192.168.2.4 host 192.168.2.1
ip access-list extended inside-out
permit gre host 192.168.2.4 host 192.168.2.1
permit gre host 192.168.2.1 host 192.168.2.4
permit ip any any
ip access-list extended acl-in
permit icmp any any echo-reply
permit esp host asa-outside-address any
permit udp any any eq isakmp
permit udp any any eq non500-isakmp
permit ahp any any
permit gre any any
ip access-list extended nat-list
deny ip any any
!
!
!
!
!
!
control-plane
!
!
!end


Tunnel5 is up, line protocol is down
Hardware is Tunnel
Internet address is 192.168.3.10/30
MTU 17916 bytes, BW 100 Kbit/sec, DLY 50000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation TUNNEL, loopback not set
Keepalive set (10 sec), retries 3
Tunnel linestate evaluation down - keepalive down
Tunnel source 192.168.2.4 (Loopback0), destination 192.168.2.1
Tunnel Subblocks:
src-track:
Tunnel5 source tracking subblock associated with Loopback0
Set of tunnels with source Loopback0, 1 member (includes iterators), on interface <OK>
Tunnel protocol/transport GRE/IP
Key disabled, sequencing disabled
Checksumming of packets disabled
Tunnel TTL 255, Fast tunneling enabled
Path MTU Discovery, ager 10 mins, min MTU 92
Tunnel transport MTU 1476 bytes
Tunnel transmit bandwidth 8000 (kbps)
Tunnel receive bandwidth 8000 (kbps)
Last input 00:00:00, output 00:00:04, output hang never
Last clearing of "show interface" counters 1d04h
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 542
Queueing strategy: fifo
Output queue: 0/0 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
5754 packets input, 415548 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
10145 packets output, 486960 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 output buffer failures, 0 output buffers swapped out

*Nov 29 15:01:30.076: Tunnel5: IPv6 PAK flags reset

*Nov 29 15:01:30.076: Tunnel5: GRE decapsulated IP packet (linktype=7, len=60)
*Nov 29 15:01:30.076: Tunnel5: GRE decapsulated IP packet (linktype=7, len=60)
r-1921#no deb
*Nov 29 15:01:34.380: Tunnel5: GRE/IP (PS) to decaps 192.168.2.1->192.168.2.4 (tbl=0,"default" len=84 ttl=254)
*Nov 29 15:01:34.380: Tunnel5: Pak Decapsulated on GigabitEthernet0/0, ptype 0x800, nw start 0xF54B5F0, mac start 0x0, datagram size 60 link type 0x7

*Nov 29 15:01:34.380: Tunnel5: IPv6 PAK flags reset

*Nov 29 15:01:34.380: Tunnel5: GRE decapsulated IP packet (linktype=7, len=60)
*Nov 29 15:01:34.380: Tunnel5: GRE decapsulated IP packet (linktype=7, len=60)
*Nov 29 15:01:34.408: Tunnel5
r-1921#no debug all: GRE/IP encapsulated 192.168.2.4->192.168.2.1 (linktype=7, len=48)
*Nov 29 15:01:34.408: Tunnel5: count tx, adding 0 encap bytes

r-1921#sh crypto ipsec sa

interface: GigabitEthernet0/0
Crypto map tag: s2s-map, local addr 192.168.0.66

protected vrf: (none)
local ident (addr/mask/prot/port): (192.168.2.4/255.255.255.255/47/0)
remote ident (addr/mask/prot/port): (192.168.2.1/255.255.255.255/47/0)
current_peer asa-outside port 500
PERMIT, flags={origin_is_acl,}
#pkts encaps: 2234, #pkts encrypt: 2234, #pkts digest: 2234
#pkts decaps: 3526, #pkts decrypt: 3526, #pkts verify: 3526
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 0, #pkts compr. failed: 0
#pkts not decompressed: 0, #pkts decompress failed: 0
#send errors 0, #recv errors 0

local crypto endpt.: 192.168.0.66, remote crypto endpt.: asa-outside
plaintext mtu 1438, path mtu 1500, ip mtu 1500, ip mtu idb GigabitEthernet0/0
current outbound spi: 0xC3035550(3271775568)
PFS (Y/N): Y, DH group: group5

inbound esp sas:
spi: 0xE65FDA97(3865041559)
transform: esp-256-aes esp-sha-hmac ,
in use settings ={Tunnel, }
conn id: 2057, flow_id: Onboard VPN:57, sibling_flags 80000040, crypto map: s2s-map
sa timing: remaining key lifetime (k/sec): (4291718/2143)
IV size: 16 bytes
replay detection support: Y
Status: ACTIVE(ACTIVE)

inbound ah sas:

inbound pcp sas:

outbound esp sas:
spi: 0xC3035550(3271775568)
transform: esp-256-aes esp-sha-hmac ,
in use settings ={Tunnel, }
conn id: 2058, flow_id: Onboard VPN:58, sibling_flags 80000040, crypto map: s2s-map
sa timing: remaining key lifetime (k/sec): (4291751/2143)
IV size: 16 bytes
replay detection support: Y
Status: ACTIVE(ACTIVE)

outbound ah sas:

outbound pcp sas:

6800 CONFIG/Tu-INT /GRE debug is below

!
interface Loopback0
ip address 192.168.2.1 255.255.255.255
end
!
!
interface Tunnel4
description CBC Gateshead
ip address 192.168.3.9 255.255.255.252
no ip redirects
ip mtu 1400
ip tcp adjust-mss 1300
keepalive 10 3
tunnel source Loopback0
tunnel destination 192.168.2.4
tunnel path-mtu-discovery
end
!
!
ip route 192.168.2.4 255.255.255.255 10.132.27.234
!
!

6800#sh int t4
Tunnel4 is up, line protocol is up
Hardware is Tunnel
Description: CBC Gateshead
Internet address is 192.168.3.9/30
MTU 17868 bytes, BW 100 Kbit/sec, DLY 50000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation TUNNEL, loopback not set
Keepalive set (10 sec), retries 3
Tunnel linestate evaluation up
Tunnel source 192.168.2.1 (Loopback0), destination 192.168.2.4
Tunnel Subblocks:
src-track:
Tunnel4 source tracking subblock associated with Loopback0
Set of tunnels with source Loopback0, 1 member (includes iterators), on interface <OK>
Tunnel protocol/transport GRE/IP
Key disabled, sequencing disabled
Checksumming of packets disabled
Tunnel TTL 255, Fast tunneling enabled
Path MTU Discovery, ager 10 mins, min MTU 92
Tunnel transport MTU 1476 bytes
Tunnel transmit bandwidth 8000 (kbps)
Tunnel receive bandwidth 8000 (kbps)
Last input never, output 00:00:00, output hang never
Last clearing of "show interface" counters 3w4d
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/0 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
1948 packets input, 93504 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
5961 packets output, 430596 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 output buffer failures, 0 output buffers swapped out

>192.168.2.4 (linktype=7, len=48)
002086: .Nov 29 15:02:20.291 UTC: SW1: Tunnel4: GRE/IP (PS) to decaps 192.168.2.4->192.168.2.1 (tbl=0,"default" len=24 ttl=253)
002087: .Nov 29 15:02:20.703 UTC: SW1: Tunnel4: GRE/IP encapsulated 192.168.2.1->192.168.2.4 (linktype=7, len=84)
central#
002088: .Nov 29 15:02:25.411 UTC: SW1: Tunnel4: GRE/IP encapsulated 192.168.2.1->192.168.2.4 (linktype=7, len=84)
central#
002089: .Nov 29 15:02:29.923 UTC: SW1: Tunnel4: GRE/IP encapsulated 192.168.2.1->192.168.2.4 (linktype=7, len=84)
002090: .Nov 29 15:02:30.291 UTC: SW1: Tunnel4: GRE/IP encapsulated 192.168.2.1->192.168.2.4 (linktype=7, len=48)
002091: .Nov 29 15:02:30.291 UTC: SW1: Tunnel4: GRE/IP (PS) to decaps 192.168.2.4->192.168.2.1 (tbl=0,"default" len=24 ttl=253)
central#
002092: .Nov 29 15:02:34.699 UTC: SW1: Tunnel4: GRE/IP encapsulated 192.168.2.1->192.168.2.4 (linktype=7, len=84)

[Infrastructure Team]

Addition.

If I shut down L0 and T5 on the 1921, and then re-enable them in that order (after a short period down).  Everything comes up, and I populate with routing tables.

Then after a short while the tunnel interface goes protocol down.

[cofee]

Hi there,

can you try configuring both side of IPSec tunnel in transport mode and not tunnel mode?

crypto ipsec transform-set s2s-tset esp-aes 256 esp-sha-hmac
mode transport 

you will need to replace mode from tunnel to transport on both sides and then clear the IPSec sa and bring the tunnel back up. 

[Infrastructure Team]

Hi Cofee,

I've done this, and still have the issue.

It appears to be timeouts on the tunnel.

If I increase the keepalives to 3 20 I get 1 minute.

if I have the keepalives to just 3, the tunnel stays up for 3 s.

If I put a client on the remote end and start a continuous ping through the tunnel, it stays up.

So how to get the keep alives to stay permanent....

[Infrastructure Team]

I think this may be resolved.

I've added config to test (such as forcing eigrp adverts down tunnels, and removed them.

added static routes and removed.

changed keep alives, and removed them and re-added them.

rebooted

then after lunch I re-enabled the interface looking to routing loops being the issue, and it stayed up.

hope this is it.

thanks for looking

[cofee · Accepted solution]

Good that you were able to figure it out.

Based on point-to-point GRE tunnel

* On demand VPN (usually triggered by routing protocol)
* Tunnel is up as long as route present for destination
* Optionally tunnel keepalives sets the tunnel state UP/Down

[Infrastructure Team]

yeah.  I couldn't figure out why the gre wouldn't stay up or come up. despite having a 32bit route specifically for the end point.  And would only stay active for the duration of 1 round of timers :s

Still not sure, but it's up.

Thanks

[muhammadadil12]

Hello

I am having a quite similar problem, I am running on Cisco IOS.

I deploying a hub and spoke site-to-site IPSec VPN.

So I have two GRE tunnels to the hub

The problem is only one Tunnel establishes adjacency with the hub, however, if i shut down this tunnel, the other tunnel comes up.

It is not keeping both tunnels up at the same time.

Do I need different transform-set per site?

These are my config in the hub:

crypto isakmp policy 10

authentication pre-share

encryption 3des

hash md5

group 1

lifetime 34000

exit

crypto isakmp key uos123 address 20.5.8.8

crypto isakmp key uos123 address 20.7.10.10

crypto ipsec transform-set VPN-to-Spokes esp-3des esp-sha-hmac

mode tunnel

exit

crypto map VPN 10 ipsec-isakmp

set peer 20.5.8.8

match address VPN-traffic

set transform-set VPN-to-Spokes

set pfs group2

set security-association lifetime seconds 44000

exit

crypto map VPN 20 ipsec-isakmp

set peer 20.7.10.10

match address VPN-traffic

set transform-set VPN-to-Spokes

set pfs group2

set security-association lifetime seconds 44000

exit

ip access-list extended VPN-traffic

 permit ip any any

 permit gre any any

interface GigabitEthernet0/1

crypto map VPN

exit

Thanks in advance.

[Infrastructure Team]

I now run ipsec mode as transport instead of tunnel.

also there is for multipoint vpns, something that needs to specify this.  I know both ipsec and gre have their own methods for this.  The GRE which I suspect you're questioning is mGRE

I'm not sure of the actual config without a bit of research.

[muhammadadil12]

I actually asking for the traditional hub-and-spoke site-to-site VPN where spokes traffic traverses the hub.

do i need a different ipsec mode per spoke or i can use the same in the crypto maps for each spoke?

[Infrastructure Team]

you can use the same crypto maps.

[muhammadadil12]

Thanks,

I have sorted out.

The problem was with ACL. In the hub it was any to any.