Tunnel not coming up

[abc1235]

Hallo,

I have configured an ISR1100 router to communicate with a remote site. However, the tunnels are not coming up. 

R1-1#sh ip int brief
Interface IP-Address OK? Method Status Protocol
GigabitEthernet0/0/0 192.168.1.1 YES NVRAM down down
GigabitEthernet0/1/0 unassigned YES unset up up
GigabitEthernet0/1/1 unassigned YES unset down down
GigabitEthernet0/1/2 unassigned YES unset down down
GigabitEthernet0/1/3 unassigned YES unset down down
Wl0/1/4 unassigned YES unset administratively down down
Cellular0/2/0 10.x.x.x YES IPCP up up
Cellular0/2/1 unassigned YES NVRAM administratively down down
ATM0/3/0 unassigned YES NVRAM administratively down down
Ethernet0/3/0 unassigned YES NVRAM down down
Loopback0 172.x.x.x YES manual up up
Tunnel100 172.x.x.x YES manual up down
Tunnel200 172.x.x.x YES manual up down
Vlan1 unassigned YES unset administratively down down
Vlan100 x.x.x.x YES manual up up
Vlan251 x.x.x.x YES manual up up
Vlan300 x.x.x.x YES manual up up
Vlan804 x.x.x.x YES manual up up
Vlan805 x.x.x.x YES manual up up
Vlan806 x.x.x.x YES manual up up

The running configs for the tunnel are as below:

R1#sh int tunnel100
Tunnel100 is up, line protocol is down
Hardware is Tunnel
Internet address is 172.x.x.x/24
MTU 9972 bytes, BW 100 Kbit/sec, DLY 10000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation TUNNEL, loopback not set
Keepalive not set
Tunnel linestate evaluation down - linestate mode reg down
Tunnel source 10.x.x.x (Cellular0/2/0)
Tunnel Subblocks:
src-track:
Tunnel100 source tracking subblock associated with Cellular0/2/0
Set of tunnels with source Cellular0/2/0, 2 members (includes iterators), on interface <OK>
Tunnel protocol/transport multi-GRE/IP
Key 0x64, sequencing disabled
Checksumming of packets disabled
Tunnel TTL 255, Fast tunneling enabled
Tunnel transport MTU 1472 bytes
Tunnel transmit bandwidth 8000 (kbps)
Tunnel receive bandwidth 8000 (kbps)
Tunnel protection via IPSec (profile "DMVPN-PROFILE-1")
Last input never, output 00:00:01, output hang never
Last clearing of "show interface" counters 16:35:32
Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/0 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
0 packets input, 0 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
3407 packets output, 463352 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 output buffer failures, 0 output buffers swapped out
R1#

The cellular interface is as follows:

R1#sh int cellular 0/2/0
Cellular0/2/0 is up, line protocol is up
Hardware is LTE Adv CAT6 - Multimode LTE/DC-HSPA+/HSPA+/HSPA/UMTS/EDGE/GPRS
Internet address is 10.x.x.x/32
MTU 1500 bytes, BW 50000 Kbit/sec, DLY 20000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation HDLC, loopback not set
Keepalive not supported
DTR is pulsed for 1 seconds on reset
Last input 00:00:03, output 00:00:04, output hang never
Last clearing of "show interface" counters never
Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
7608 packets input, 2025268 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
8008 packets output, 1795372 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions

R1#sh run int cellular 0/2/0
Building configuration...

Current configuration : 139 bytes
!
interface Cellular0/2/0
ip address negotiated
ip nat outside
dialer in-band
dialer idle-timeout 0
dialer-group 1
pulse-time 1
end

R1#

Kindly let me know what the issue could be. Thanks.

[balaji.bandi]

we do not see any VPN config here? post config here also enable debugging and checking what logs you see ?

you can find example guide:

https://www.cisco.com/c/en/us/td/docs/routers/access/interfaces/NIM/software/configuration/guide/4GLTENIM_SW.html#pgfId-1334940

[abc1235]

Hallo BB,

I did a quick Tshoot by comparing with a similar ISR and the only difference is that for the problematic router is using cellular interface as the tunnel source while the working one is using a dialer. Below the DMVPN details:

R1#sh dmvpn detail
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
N - NATed, L - Local, X - No Socket
T1 - Route Installed, T2 - Nexthop-override
C - CTS Capable, I2 - Temporary
# Ent --> Number of NHRP entries with same NBMA peer
NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting
UpDn Time --> Up or Down Time for a Tunnel
==========================================================================

Interface Tunnel100 is up/down, Addr. is 172.x.x.x, VRF ""
Tunnel Src./Dest. addr: 10.x.x.x/Multipoint, Tunnel VRF "INTERNET"
Protocol/Transport: "multi-GRE/IP", Protect "DMVPN-PROFILE-1"
Interface State Control: Enabled
nhrp event-publisher : Disabled
IPv4 Registration Timer: 60 seconds

IPv4 NHS:
172.x.x.x.x E NBMA Address: 109.x.x.x.x priority = 0 cluster = 0
Type:Spoke, Total NBMA Peers (v4/v6): 1

# Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb Target Network
----- --------------- --------------- ----- -------- ----- -----------------
1 109.x.x.x 172.x.x.x NHRP 16:31:26 S 172.x.x.x/32

Interface Tunnel200 is up/down, Addr. is 172.x.x.x, VRF ""
Tunnel Src./Dest. addr: 10.x.x.x/Multipoint, Tunnel VRF "INTERNET"
Protocol/Transport: "multi-GRE/IP", Protect "DMVPN-PROFILE-2"
Interface State Control: Enabled
nhrp event-publisher : Disabled
IPv4 Registration Timer: 60 seconds

IPv4 NHS:
172.x.x.x E NBMA Address: 109.x.x.x priority = 0 cluster = 0
Type:Spoke, Total NBMA Peers (v4/v6): 1

# Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb Target Network
----- --------------- --------------- ----- -------- ----- -----------------
1 109.x.x.x 172.x.x.x NHRP 16:31:23 S 172.x.x.x/32

Crypto Session Details:
--------------------------------------------------------------------------------

Interface: Tunnel100
Session: [0x7F4CF77318]
Session ID: 2
IKEv2 SA: local 10.x.x.x/4500 remote 109.x.x.x/4500 Active
Capabilities:DNU connid:2 lifetime:07:28:30
Crypto Session Status: UP-ACTIVE
fvrf: INTERNET, Phase1_id: 109.x.x.x
IPSEC FLOW: permit 47 host 10.x.x.x host 109.x.x.x
Active SAs: 2, origin: crypto map
Inbound: #pkts dec'ed 0 drop 0 life (KB/Sec) 4608000/1819
Outbound: #pkts enc'ed 540 drop 0 life (KB/Sec) 4607994/1819
Outbound SPI : 0xC1224905, transform : esp-256-aes esp-sha256-hmac
Socket State: Open

Interface: Tunnel200
Session: [0x7F4CF77198]
Session ID: 1
IKEv2 SA: local 10.x.x.x/4500 remote 109.x.x.x/4500 Active
Capabilities:DNU connid:1 lifetime:16:40:39
Crypto Session Status: UP-ACTIVE
fvrf: INTERNET, Phase1_id: 109.x.x.x
IPSEC FLOW: permit 47 host 10.x.x.x host 109.x.x.x
Active SAs: 2, origin: crypto map
Inbound: #pkts dec'ed 0 drop 0 life (KB/Sec) 4608000/3325
Outbound: #pkts enc'ed 488 drop 0 life (KB/Sec) 4607999/3325
Outbound SPI : 0xDD6A9FF5, transform : esp-256-aes esp-sha256-hmac
Socket State: Open

Pending DMVPN Sessions:

R1#

[Georg Pauwen]

Hello, 

since you use the cellular interface as the tunnel source, you (most likely) have a dynamic IP address. Under the tunnel interface, try and configure 'tunnel source dynamic'...

What is connected on the other side ?

[balaji.bandi]

I would insists to see your fulll config and and debug logs

[abc1235]

Hallo Georg,

I have done that but there is not change. Both tunnels are still in up/down state.

[abc1235]

Oh, and on the other side is a hub that is connected.

Joyce

[Georg Pauwen · Accepted solution]

Hello,

the tunnel is in the "INTERNET' VRF, is that right ?

Turn on debugging:

debug crypto ipsec

debug crypto isakamp

and post the output...

Seeing the full running configs of both sides would help, too...

[abc1235]

Hallo Georg,

Yes, the tunnel is in vrf INTERNET. Here are the running configs of spoke and hub:

[abc1235]

Hallo Georg,

The the debug messages received are as below:

*Oct 22 2021 10:32:42.640 CEST: IPSEC(key_engine): got a queue event with 1 KMI message(s)
*Oct 22 2021 10:32:42.641 CEST: IPSEC(validate_proposal_request): proposal part #1
*Oct 22 2021 10:32:42.641 CEST: IPSEC(validate_proposal_request): proposal part #1,
(key eng. msg.) INBOUND local= 10.x.x.x:0, remote= 109.x.x.x,
local_proxy= 46.x.x.x/255.255.255.255/47/0,
remote_proxy= 109.x.x.x/255.255.255.255/47/0,
protocol= ESP, transform= esp-aes 256 esp-sha256-hmac (Transport-UDP),
lifedur= 0s and 0kb,
spi= 0x0(0), conn_id= 0, keysize= 256, flags= 0x0
*Oct 22 2021 10:32:42.641 CEST: map_db_check_isakmp_profile profile did not match,
ike passed profile : FVRF-IKEv2-IWAN-TRANSPORT-2,
map_ike_profile: FVRF-IKEv2-IWAN-TRANSPORT-1,
head_ike_profile: FVRF-IKEv2-IWAN-TRANSPORT-1
*Oct 22 2021 10:32:42.641 CEST: map_db_check_isakmp_profile profile did not match,
ike passed profile : FVRF-IKEv2-IWAN-TRANSPORT-2,
map_ike_profile: FVRF-IKEv2-IWAN-TRANSPORT-1,
head_ike_profile: FVRF-IKEv2-IWAN-TRANSPORT-1
*Oct 22 2021 10:32:42.641 CEST: map_db_find_best did not find matching map
*Oct 22 2021 10:32:42.642 CEST: Crypto mapdb : proxy_match
src addr : 10.x.x.x
dst addr : 109.x.x.x
protocol : 47
src port : 0
dst port : 0
*Oct 22 2021 10:32:42.642 CEST: (ipsec_process_proposal)Map Accepted: Tunnel200-head-0, 65537
*Oct 22 2021 10:32:42.643 CEST: IPSEC(key_engine): got a queue event with 1 KMI message(s)
*Oct 22 2021 10:32:42.643 CEST: Crypto mapdb : proxy_match
src addr : 10.x.x.x
dst addr : 109.x.x.x
protocol : 47
src port : 0
dst port : 0
*Oct 22 2021 10:32:42.643 CEST: IPSEC:(SESSION ID = 1) (crypto_ipsec_create_ipsec_sas) Map found Tunnel200-head-0, 65537TBAR_DBG ident_prep_create_sa: after initilize settings for time-based antireplay: do_ipd3p=0, ipd3p_type=0, win-size=0, do_tbar=0

*Oct 22 2021 10:32:42.644 CEST: IPSEC:(SESSION ID = 1) (create_sa) sa created,
(sa) sa_dest= 10.131.228.28, sa_proto= 50,
sa_spi= 0x89D1B5FD(2312222205),
sa_trans= esp-aes 256 esp-sha256-hmac , sa_conn_id= 2089
sa_lifetime(k/sec)= (4608000/3600),
(identity) local= 10.x.x.x:0, remote= 109.x.x.x,
local_proxy= 10.x.x.x/255.255.255.255/47/0,
remote_proxy= 109.x.x.x/255.255.255.255/47/0
*Oct 22 2021 10:32:42.645 CEST: ipsec_out_sa_hash_idx: sa=0x7F48ADD7A0, hash_idx=990, port=4500/4500, addr=0x0A83E41C/0x6D46C005
*Oct 22 2021 10:32:42.645 CEST: crypto_ipsec_hook_out_sa: ipsec_out_sa_hash_array[990]=0x7F48ADD7A0
*Oct 22 2021 10:32:42.645 CEST: IPSEC:(SESSION ID = 1) (create_sa) sa created,
(sa) sa_dest= 109.x.x.x, sa_proto= 50,
sa_spi= 0x430F4014(1125072916),
sa_trans= esp-aes 256 esp-sha256-hmac , sa_conn_id= 2090
sa_lifetime(k/sec)= (4608000/3600),
(identity) local= 10.x.x.x:0, remote= 109.x.x.x,
local_proxy= 10.x.x.x.28/255.255.255.255/47/0,
remote_proxy= 109.x.x.x/255.255.255.255/47/0
*Oct 22 2021 10:32:42.714 CEST: IPSEC(key_engine): got a queue event with 1 KMI message(s)
*Oct 22 2021 10:32:42.714 CEST: IDB is NULL : in crypto_ipsec_key_engine_delete_sas (), 5733
*Oct 22 2021 10:32:42.714 CEST: IPSEC:(SESSION ID = 1) (key_engine_delete_sas) rec'd delete notify from ISAKMP
*Oct 22 2021 10:32:42.715 CEST: IPSEC: still in use sa: 0x7F48ADD6A0
*Oct 22 2021 10:32:42.715 CEST: IPSEC:(SESSION ID = 1) (update_current_outbound_sa) updated peer 109.x.x.x current outbound sa to SPI 430F4014
*Oct 22 2021 10:32:42.723 CEST: IPSEC:(SESSION ID = 1) (key_engine_delete_sas) delete SA with spi 0xBADF0E2F proto 50 for 10.x.x.x
*Oct 22 2021 10:32:42.723 CEST: IPSEC:(SESSION ID = 1) (delete_sa) deleting SA,
(sa) sa_dest= 10.x.x.x, sa_proto= 50,
sa_spi= 0xBADF0E2F(3135180335),
sa_trans= esp-aes 256 esp-sha256-hmac , sa_conn_id= 2085
sa_lifetime(k/sec)= (4608000/3600),
(identity) local= 10.x.x.x:0, remote= 109.70.192.5:0,
local_proxy= 10.x.x.x/255.255.255.255/47/0,
remote_proxy= 109.x.x.x/255.255.255.255/47/0
*Oct 22 2021 10:32:42.723 CEST: IPSEC:(SESSION ID = 1) (delete_sa) deleting SA,
(sa) sa_dest= 109.x.x.x, sa_proto= 50,
sa_spi= 0x48E911D1(1223234001),
sa_trans= esp-aes 256 esp-sha256-hmac , sa_conn_id= 2086
sa_lifetime(k/sec)= (4608000/3600),
(identity) local= 10.x.x.x:0, remote= 109.70.192.5:0,
local_proxy= 10.x.x.x/255.255.255.255/47/0,
remote_proxy= 109.x.x.x/255.255.255.255/47/0
*Oct 22 2021 10:32:42.724 CEST: IPSEC(send_delete_notify_kmi): not sending KEY_ENGINE_DELETE_SAS
*Oct 22 2021 10:32:42.724 CEST: ipsec_out_sa_hash_idx: sa=0x7F48ADD5A0, hash_idx=990, port=4500/4500, addr=0x0A83E41C/0x6D46C005
*Oct 22 2021 10:32:42.729 CEST: IPSEC:(SESSION ID = 1) (ident_delete_notify_kmi) Failed to send KEY_ENG_DELETE_SAS
*Oct 22 2021 10:32:42.729 CEST: IPSEC:(SESSION ID = 1) (ident_update_final_flow_stats) Collect Final Stats and update MIB
IPSEC get IKMP peer index from peer 0x7F48ADB360 ikmp handle 0x0
[ident_update_final_flow_stats] : Flow delete complete event received for flow id 0x24000055,peer index 0

R1#

[Georg Pauwen]

Hello,

since there is a ZBF involved, I think you need an outside to self allowing all ISAKMP. Try and add the below:

zone-pair security DMVPN-TO-SELF source DMVPN destination self
service-policy type inspect DMVPN-TO-SELF-PM
!
policy-map type inspect DMVPN-TO-SELF-PM
class type inspect DMVPN-TO-SELF-CM
pass
class class-default
!
class-map type inspect match-any DMVPN-TO-SELF-CM
match access-group name ISAKMP-ACL
!
ip access-list extended ISAKMP-ACL
permit udp any any eq isakmp
permit ahp any any
permit esp any any
permit udp any any eq non500-isakmp

If that doesn't work, I'll need to lab this up...

[pman]

base on this lines you wrote:

"The cellular interface is as follows:

R1#sh int cellular 0/2/0
Cellular0/2/0 is up, line protocol is up
Hardware is LTE Adv CAT6 - Multimode LTE/DC-HSPA+/HSPA+/HSPA/UMTS/EDGE/GPRS
Internet address is 10.x.x.x/32"

I have a question, and maybe I'm wrong, is there a chance you're trying to open connection (ipsec for example) with external address from an internal address (10.x.x.x)?
Is NAT configured from an internal address to an external address somewhere?

I guess you know, the remote-address(remote IPSEC in this case) should recognize your external address for IPSEC and not the internal address (10.x.x.x)

[abc1235]

Hallo Pman,

Cellular 0/2/0 interface is configured to IP NAT outside. But thanks for the observation.

[abc1235]

@Georg Pauwen I have tried the configs you have sent but I get the following errors:

R1(config-sec-zone-pair)#$icy type inspect DMVPN-TO-SELF-PM
Policy DMVPN-TO-SELF-PM does not exist
Inspect service-policy attachment failed

...

R1(config-pmap)#class type inspect DMVPN-TO-SELF-CM
class map DMVPN-TO-SELF-CM not configured
R1(config-pmap)#pass
^
% Invalid input detected at '^' marker.

...

R1(config-pmap-c)#match access-group name ISAKMP-ACL
^
% Invalid input detected at '^' marker.

This is Cisco C1116-4PLTEEAWE Chassis router.

Let me know if there is anything else I should change. Thanks.

[Georg Pauwen]

Hello,

you need to create the access list first:

ip access-list extended ISAKMP-ACL
permit udp any any eq isakmp
permit ahp any any
permit esp any any
permit udp any any eq non500-isakmp

then the class map:

class-map type inspect match-any DMVPN-TO-SELF-CM
match access-group name ISAKMP-ACL

then the policy map:

policy-map type inspect DMVPN-TO-SELF-PM
class type inspect DMVPN-TO-SELF-CM
pass
class class-default

then the zone pair:

zone-pair security DMVPN-TO-SELF source DMVPN destination self
service-policy type inspect DMVPN-TO-SELF-PM

Where do you get the 'invalid input' ? It usually tells you why the input is invalid...