GRE tunnels will not come up on IPsec/GRE VPN

[mark.levy]

Hi All,

We've got 400+ remote sites that connect to our central location (and a hot site) using Cisco routers with IPSec/GRE vpn tunnels.  We use a basic template for creating the tunnels, so there is very little chance of a misconfiguration on either router.  The remote sites use Cisco 831s, the central sites use Cisco 2821s.  There is one site where the GRE tunnels just refuse to come up.

The routers are able to ping their public IP addresses, so it's not a routing issue, but the gre endpoints can not ping.  There is no NATing involved, both routers directly access the Internet.  The assorded show commands seem to indicate that the SAs are being properly built, but from the logs, it seems like the last part just doesn't get completed, and the GRE tunnels just don't come up.  

From the attached log file, it appears that both ISAKMP & IPSEC SAs are created @ 00:25:14, then QM_PHASE2 completes @ 00:25:15.

00:25:15: ISAKMP:(0:10:HW:2):deleting node 1891573546 error FALSE reason "QM done (await)"
00:25:15: ISAKMP:(0:10:HW:2):Node 1891573546, Input = IKE_MESG_FROM_PEER, IKE_QM_EXCH
00:25:15: ISAKMP:(0:10:HW:2):Old State = IKE_QM_R_QM2  New State = IKE_QM_PHASE2_COMPLETE
00:25:15: ISAKMP (0:268435467): received packet from 208.XX.YY.11 dport 500 sport 500 Global (I) QM_IDLE   
   
00:25:15: IPSEC(key_engine): got a queue event with 1 kei messages
00:25:15: IPSEC(key_engine_enable_outbound): rec'd enable notify from ISAKMP
00:25:15: IPSEC(key_engine_enable_outbound): enable SA with spi 1572231461/50 
00:25:15: ISAKMP:(0:11:HW:2):deleting node -1931380074 error FALSE reason "QM done (await)"
00:25:15: ISAKMP:(0:11:HW:2):Node -1931380074, Input = IKE_MESG_FROM_PEER, IKE_QM_EXCH
00:25:15: ISAKMP:(0:11:HW:2):Old State = IKE_QM_R_QM2  New State = IKE_QM_PHASE2_COMPLETE
00:25:15: IPSEC(key_engine): got a queue event with 1 kei messages
00:25:15: IPSEC(key_engine_enable_outbound): rec'd enable notify from ISAKMP
00:25:15: IPSEC(key_engine_enable_outbound): enable SA with spi 310818168/50 

I do have the log file from the remote router, and it's very long, so I've attached it.  Before I captured the log file, I enabled debug on isakmp & ipsec, and immediately cleared the SAs.  

Assorted useful details and the results of the assorted show commands are:

Cisco IOS Software, C831 Software (C831-K9O3SY6-M), Version 12.4(25d), RELEASE SOFTWARE (fc1)

There are 2 IPSEC/GRE tunnel connections:

Tunnel101:  KC ( 208.YY.ZZ.11 )  -  Remote  ( 74.WW.XX.35 )
Tunnel201:  Dallas ( 208.XX.YY.11 )  -  Remote  ( 74.WW.XX.35 )

Site-382-831#sho ip int br
Interface                  IP-Address      OK? Method Status                Protocol
FastEthernet1              unassigned      YES unset  down                  down    
FastEthernet2              unassigned      YES unset  up                    up      
FastEthernet3              unassigned      YES unset  up                    up      
FastEthernet4              unassigned      YES unset  up                    up      
Ethernet0                  10.3.82.10      YES NVRAM  up                    up      
Ethernet1                  74.WW.XX.35    YES NVRAM  up                    up      
Ethernet2                  172.16.1.10     YES NVRAM  up                    up      
Tunnel101                  1.3.82.46       YES NVRAM  up                    down     <<<<====
Tunnel201                  1.3.82.62       YES NVRAM  up                    down     <<<<====    
NVI0                       unassigned      NO  unset  up                    up      

Site-382-831#
Site-382-831#sho run int tunnel101
Building configuration...

Current configuration : 277 bytes
!
interface Tunnel101
 description %%%%%%%%%%%%%%%%   connected to 2nd KC BGP 2821 - PRI-B
 ip address 1.3.82.46 255.255.255.252
 ip mtu 1500
 ip virtual-reassembly
 ip tcp adjust-mss 1360
 keepalive 3 3
 tunnel source Ethernet1
 tunnel destination 208.YY.ZZ.11
end

Site-382-831#


Site-382-831#show crypto isakmp sa
dst             src             state          conn-id slot status
208.XX.YY.11   74.WW.XX.35    QM_IDLE             11    0 ACTIVE
208.YY.ZZ.11   74.WW.XX.35    QM_IDLE             10    0 ACTIVE
Site-382-831#

Site-382-831#
Site-382-831#show crypto isakmp sa detail
Codes: C - IKE configuration mode, D - Dead Peer Detection
       K - Keepalives, N - NAT-traversal
       X - IKE Extended Authentication
       psk - Preshared key, rsig - RSA signature
       renc - RSA encryption

C-id  Local           Remote          I-VRF    Status Encr Hash Auth DH Lifetime Cap.
11    74.WW.XX.35    208.XX.YY.11            ACTIVE 3des sha  psk  1  23:56:09    
       Connection-id:Engine-id =  11:2(hardware)
10    74.WW.XX.35    208.YY.ZZ.11            ACTIVE 3des sha  psk  1  23:56:09    
       Connection-id:Engine-id =  10:2(hardware)
Site-382-831#

Site-382-831#
Site-382-831#show crypto ipsec sa

interface: Ethernet1
    Crypto map tag: IPVPN_MAP, local addr 74.WW.XX.35

   protected vrf: (none)
   local  ident (addr/mask/prot/port): (74.WW.XX.35/255.255.255.255/47/0)
   remote ident (addr/mask/prot/port): (208.YY.ZZ.11/255.255.255.255/47/0)
   current_peer 208.YY.ZZ.11 port 500
     PERMIT, flags={origin_is_acl,}
    #pkts encaps: 2333, #pkts encrypt: 2333, #pkts digest: 2333
    #pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
    #send errors 21, #recv errors 0

     local crypto endpt.: 74.WW.XX.35, remote crypto endpt.: 208.YY.ZZ.11
     path mtu 1500, ip mtu 1500, ip mtu idb Ethernet1
     current outbound spi: 0x45047D1D(1157922077)

     inbound esp sas:
      spi: 0x15B97AEA(364477162)
        transform: esp-3des esp-sha-hmac ,
        in use settings ={Tunnel, }
        conn id: 2004, flow_id: C83X_MBRD:4, crypto map: IPVPN_MAP
        sa timing: remaining key lifetime (k/sec): (4486831/1056)
        IV size: 8 bytes
        replay detection support: Y
        Status: ACTIVE

     inbound ah sas:

     inbound pcp sas:

     outbound esp sas:
      spi: 0x45047D1D(1157922077)
        transform: esp-3des esp-sha-hmac ,
        in use settings ={Tunnel, }
        conn id: 2003, flow_id: C83X_MBRD:3, crypto map: IPVPN_MAP
        sa timing: remaining key lifetime (k/sec): (4486744/1056)
        IV size: 8 bytes
        replay detection support: Y
        Status: ACTIVE

     outbound ah sas:

     outbound pcp sas:

   protected vrf: (none)
   local  ident (addr/mask/prot/port): (74.WW.XX.35/255.255.255.255/47/0)
   remote ident (addr/mask/prot/port): (208.XX.YY.11/255.255.255.255/47/0)
   current_peer 208.XX.YY.11 port 500
     PERMIT, flags={origin_is_acl,}
    #pkts encaps: 2333, #pkts encrypt: 2333, #pkts digest: 2333
    #pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
    #send errors 21, #recv errors 0

     local crypto endpt.: 74.WW.XX.35, remote crypto endpt.: 208.XX.YY.11
     path mtu 1500, ip mtu 1500, ip mtu idb Ethernet1
     current outbound spi: 0xE82A86BC(3895101116)

     inbound esp sas:
      spi: 0x539697CA(1402378186)
        transform: esp-3des esp-sha-hmac ,
        in use settings ={Tunnel, }
        conn id: 2008, flow_id: C83X_MBRD:8, crypto map: IPVPN_MAP
        sa timing: remaining key lifetime (k/sec): (4432595/1039)
        IV size: 8 bytes
        replay detection support: Y
        Status: ACTIVE

     inbound ah sas:

     inbound pcp sas:

     outbound esp sas:
      spi: 0xE82A86BC(3895101116)
        transform: esp-3des esp-sha-hmac ,
        in use settings ={Tunnel, }
        conn id: 2001, flow_id: C83X_MBRD:1, crypto map: IPVPN_MAP
        sa timing: remaining key lifetime (k/sec): (4432508/1039)
        IV size: 8 bytes
        replay detection support: Y
        Status: ACTIVE

     outbound ah sas:

     outbound pcp sas:
Site-382-831#

Site-382-831#
Site-382-831#show crypto ipsec sa | inc pkts|lifetime
    #pkts encaps: 2397, #pkts encrypt: 2397, #pkts digest: 2397
    #pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
        sa timing: remaining key lifetime (k/sec): (4486831/862)
        sa timing: remaining key lifetime (k/sec): (4486738/862)
    #pkts encaps: 2397, #pkts encrypt: 2397, #pkts digest: 2397
    #pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
        sa timing: remaining key lifetime (k/sec): (4432595/846)
        sa timing: remaining key lifetime (k/sec): (4432501/846)
Site-382-831#    


Site-382-831#
Site-382-831#show crypto isakmp policy

Global IKE policy
Protection suite of priority 10
        encryption algorithm:   Three key triple DES
        hash algorithm:         Secure Hash Standard
        authentication method:  Pre-Shared Key
        Diffie-Hellman group:   #1 (768 bit)
        lifetime:               86400 seconds, no volume limit
Default protection suite
        encryption algorithm:   DES - Data Encryption Standard (56 bit keys).
        hash algorithm:         Secure Hash Standard
        authentication method:  Rivest-Shamir-Adleman Signature
        Diffie-Hellman group:   #1 (768 bit)
        lifetime:               86400 seconds, no volume limit
Site-382-831#

Site-382-831#show crypto map
Crypto Map "IPVPN_MAP" 101 ipsec-isakmp
        Description: to 2nd KC BGP 2821 - PRI-B
        Peer = 208.YY.ZZ.11
        Extended IP access list PRI-B
            access-list PRI-B permit gre host 74.WW.XX.35 host 208.YY.ZZ.11
        Current peer: 208.YY.ZZ.11
        Security association lifetime: 4608000 kilobytes/3600 seconds
        PFS (Y/N): N
        Transform sets={
                IPVPN,
        }

Crypto Map "IPVPN_MAP" 201 ipsec-isakmp
        Description: to 2nd Dallas BGP 2821 - SEC-B
        Peer = 208.XX.YY.11
        Extended IP access list SEC-B
            access-list SEC-B permit gre host 74.WW.XX.35 host 208.XX.YY.11
        Current peer: 208.XX.YY.11
        Security association lifetime: 4608000 kilobytes/3600 seconds
        PFS (Y/N): N
        Transform sets={
                IPVPN,
        }
        Interfaces using crypto map IPVPN_MAP:
                Ethernet1
Site-382-831#


The tunnel configuration between KC & the remote site is:

***  Remote  c831  - KC  ***

crypto isakmp policy 10
 encr 3des
 authentication pre-share
!
crypto isakmp key PRI-B-382 address 208.YY.ZZ.11
!
crypto ipsec transform-set IPVPN esp-3des esp-sha-hmac
 mode transport
!
crypto map IPVPN_MAP 101 ipsec-isakmp
 description to 2nd KC BGP 2821 - PRI-B
 set peer 208.YY.ZZ.11
 set transform-set IPVPN
 match address PRI-B
!
interface Tunnel101
 description %%%%%%%%%%%%% connected to 2nd KC BGP 2821 - PRI-B
 ip address 1.3.82.46 255.255.255.252
 ip mtu 1500
 keepalive 3 3
 ip virtual-reassembly
 ip tcp adjust-mss 1360
 tunnel source Ethernet1
 tunnel destination 208.YY.ZZ.11
!
interface Ethernet0
 description Private network
 ip address 10.3.82.10 255.255.255.0
 ip mtu 1500
 no shutdown
!
interface Ethernet1
 ip address 74.WW.XX.35 255.255.255.248
 ip mtu 1500
 duplex auto
 ip virtual-reassembly
 crypto map IPVPN_MAP
 no shutdown
!
ip access-list extended PRI-B
 permit gre host 74.WW.XX.35 host 208.YY.ZZ.11
!


***  KC-2821  ***

crypto isakmp key PRI-B-382 address 74.WW.XX.35
!
ip access-list extended PRI-B-382
 permit gre host 208.YY.ZZ.11 host 74.WW.XX.35
!
crypto map IPVPN_MAP 382 ipsec-isakmp
 description %%%%%%%%%%%  Connected to 2nd KC BGP 2821
 set peer 74.WW.XX.35
 set transform-set IPVPN
 match address PRI-B-382
!
interface Tunnel382
 description %%%%%%%%%%%%%%
 ip address 1.3.82.45 255.255.255.252
 keepalive 3 3
 ip virtual-reassembly
 ip tcp adjust-mss 1360
 IP MTU 1400
 delay 40000
 tunnel source 208.YY.ZZ.11
 tunnel destination 74.WW.XX.35
!
end

Any help would be very much appreciated!

Mark                             

[Abaji Rawool · Accepted solution]

Hi,

logs on Site-382-831, show only encrypts but no decrypts, could you check corresponding entry on the peer and see if has any issues sending the traffic back?

Site-382-831#show crypto ipsec sa | inc pkts|lifetime
    #pkts encaps: 2397, #pkts encrypt: 2397, #pkts digest: 2397
    #pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
        sa timing: remaining key lifetime (k/sec): (4486831/862)
        sa timing: remaining key lifetime (k/sec): (4486738/862)
    #pkts encaps: 2397, #pkts encrypt: 2397, #pkts digest: 2397
    #pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
        sa timing: remaining key lifetime (k/sec): (4432595/846)
        sa timing: remaining key lifetime (k/sec): (4432501/846)
Site-382-831#    

Regards,

Abaji.

[mark.levy]

Thank you very much for your reply.  I had thought I had originally included this information, but it seems I didn't.  At this moment, the remote Cisco 831 is down, as the location is currently using an alternate means to do business, so I can't get the most recent IPsec SAs.  However, this is the current output of "show crypt ipsec sa" on the KC 2821:

   local  ident (addr/mask/prot/port): (208.YY.ZZ.11/255.255.255.255/47/0)

   remote ident (addr/mask/prot/port): (74.WW.XX.35/255.255.255.255/47/0)

   current_peer 74.WW.XX.35 port 500

     PERMIT, flags={origin_is_acl,ipsec_sa_request_sent}

    #pkts encaps: 23839, #pkts encrypt: 23839, #pkts digest: 23839

    #pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0

    #pkts compressed: 0, #pkts decompressed: 0

    #pkts not compressed: 0, #pkts compr. failed: 0

    #pkts not decompressed: 0, #pkts decompress failed: 0

    #send errors 119712, #recv errors 0

     local crypto endpt.: 208.YY.ZZ.11, remote crypto endpt.: 74.WW.XX.35

     path mtu 1500, ip mtu 1500, ip mtu idb GigabitEthernet0/0
     
     current outbound spi: 0x0(0)
     
     esp sas:

     inbound ah sas:

     inbound pcp sas

     outbound esp sas:

     outbound ah sas:

     outbound pcp sas:

   protected vrf: (none)

It appears that there is no decryption at either end.  What's bothersome from the log was that it appears that phase 2 is complete, but goes no further.:

As I had mentinoed before, I've used 2 different "known good" templates to build the configuration, and I even re-entered the pre-shared key by hand on two occasions.

Thanks again!

Mark

[mark.levy · Accepted solution]

Thank you so much!  Your message pointed me in a different direction, and I found the problem I was having that I had shipped the remote location a replacement c831, which would build the VPN tunnel with the SECOND set of 2821s (the original c831 connected to the first set of 2821s), and I based all of my troubleshooting on that.  However, the remote location NEVER swapped the c831s, and that completely explains why both routers were sending encrypted data, but never decrypted any! 

In effect, the remote c831 would begin the IKE with the 1st 2821, but because the 2nd 2821 was configured for the VPN, the 1st 2821 would just start the IKE process over again.  The c831 never even got as far as the exchange of the IPsec PSKs.

Thanks again for pointing me in a direction I hadn't thought to check, even though looking back on it, it was very obvious.

Mark