Solved: Re: problem when applying IPSEC to DMVPN

Abdelilah Benrida · ‎05-14-2013

Hi i have some trouble with DMVPN

i configured NHRP between a HUB and aSPOKE:

HUB

tu0 tu1

| |

ISP

|

tu0,tu1

SPOKE

the HUB has two physical interfaces and two logical interfaces.

The SPOKE has one physical interface and two logical interfaces.

in configured NHRP correctly, the tunnels are detected in the HUB and the SPOKE.

when i add the profile IPSEC to the intefaces i lose tunnel1.

SPOKE1#sh ip nhrp

10.1.1.4/32 via 10.1.1.4, Tunnel0 created 02:22:01, never expire

Type: static, Flags: authoritative used

NBMA address: 190.1.1.1

10.2.2.4/32 via 10.2.2.4, Tunnel1 created 02:18:21, never expire

Type: static, Flags: authoritative used

NBMA address: 190.1.2.1

SPOKE1#debug ip nhrp

tunnel0

*Mar 1 03:50:09.399: NHRP: Attempting to send packet via DEST 10.1.1.4

*Mar 1 03:50:09.399: NHRP: Encapsulation succeeded. Tunnel IP addr 190.1.1.1

*Mar 1 03:50:09.399: NHRP: Send Registration Request via Tunnel0 vrf 0, packet size: 82

*Mar 1 03:50:09.403: src: 10.1.1.1, dst: 10.1.1.4

*Mar 1 03:50:09.403: NHRP: 82 bytes out Tunnel0

*Mar 1 03:50:09.519: NHRP: Receive Registration Reply via Tunnel0 vrf 0, packet size: 102

*Mar 1 03:50:09.519: NHRP: netid_in = 0, to_us = 1

tunnel 1

*Mar 1 03:50:30.575: NHRP: Attempting to send packet via DEST 10.2.2.4

*Mar 1 03:50:30.575: NHRP: Encapsulation succeeded. Tunnel IP addr 190.1.2.1

*Mar 1 03:50:30.575: NHRP: Send Registration Request via Tunnel1 vrf 0, packet size: 82

*Mar 1 03:50:30.579: src: 10.2.2.1, dst: 10.2.2.4

*Mar 1 03:50:30.579: NHRP: 82 bytes out Tunnel1

*Mar 1 03:50:30.579: NHRP: Resetting retransmit due to hold-timer for 10.2.2.4

no reply from the HUB.

HUB#sh ip nhrp

10.1.1.1/32 via 10.1.1.1, Tunnel0 created 00:05:05, expire 00:08:29

Type: dynamic, Flags: authoritative unique registered

NBMA address: 191.1.1.11

just tunnel0 is there !

i have also this on the HUB :

*Mar 1 03:58:54.519: %CRYPTO-6-IKMP_MODE_FAILURE: Processing of Quick mode failed with peer at 191.1.1.11 (physical adress of SPOKE1)

configs :

HUB :

!

crypto isakmp policy 10

encr aes

hash md5

authentication pre-share

group 2

crypto isakmp key techservices address 0.0.0.0 0.0.0.0

!

crypto ipsec transform-set AES_MD5 esp-aes esp-md5-hmac

!

crypto ipsec profile DMVPN

set transform-set AES_MD5

!

interface Tunnel0

bandwidth 10000

ip address 10.1.1.4 255.255.255.0

no ip redirects

ip mtu 1400

no ip next-hop-self eigrp 123

ip nhrp authentication dmvpn1

ip nhrp map multicast dynamic

ip nhrp network-id 123

no ip split-horizon eigrp 123

tunnel source FastEthernet0/0

tunnel mode gre multipoint

tunnel key 123

tunnel protection ipsec profile DMVPN

!

interface Tunnel1

bandwidth 10000

ip address 10.2.2.4 255.255.255.0

no ip redirects

ip mtu 1400

no ip next-hop-self eigrp 124

ip nhrp authentication dmvpn2

ip nhrp map multicast dynamic

ip nhrp network-id 124

no ip split-horizon eigrp 124

tunnel source FastEthernet1/0

tunnel mode gre multipoint

tunnel key 124

tunnel protection ipsec profile DMVPN

!

router eigrp 123

network 10.1.1.0 0.0.0.255

network 172.16.4.0 0.0.0.255

no auto-summary

!

router eigrp 124

network 10.2.2.0 0.0.0.255

network 172.16.4.0 0.0.0.255

no auto-summary

!

SPOKE1:

!

crypto isakmp policy 10

encr aes

hash md5

authentication pre-share

group 2

crypto isakmp key techservices address 0.0.0.0 0.0.0.0

!

crypto ipsec transform-set AES_MD5 esp-aes esp-md5-hmac

!

crypto ipsec profile DMVPN

set transform-set AES_MD5

!

interface Tunnel0

bandwidth 10000

ip address 10.1.1.1 255.255.255.0

ip mtu 1400

ip nhrp authentication dmvpn1

ip nhrp map multicast 190.1.1.1

ip nhrp map 10.1.1.4 190.1.1.1

ip nhrp network-id 123

ip nhrp holdtime 600

ip nhrp nhs 10.1.1.4

ip nhrp registration timeout 300

tunnel source FastEthernet0/0

tunnel mode gre multipoint

tunnel key 123

tunnel protection ipsec profile DMVPN

!

interface Tunnel1

bandwidth 10000

ip address 10.2.2.1 255.255.255.0

ip mtu 1400

ip nhrp authentication dmvpn2

ip nhrp map multicast 190.1.2.1

ip nhrp map 10.2.2.4 190.1.2.1

ip nhrp network-id 124

ip nhrp holdtime 600

ip nhrp nhs 10.2.2.4

ip nhrp registration timeout 300

tunnel source FastEthernet0/0

tunnel mode gre multipoint

tunnel key 124

tunnel protection ipsec profile DMVPN

!

router eigrp 123

network 10.1.1.0 0.0.0.255

network 172.16.1.0 0.0.0.255

no auto-summary

!

router eigrp 124

network 10.2.2.0 0.0.0.255

network 172.16.1.0 0.0.0.255

no auto-summary

!

regards

Gustavo Medina · ‎05-15-2013

Good to hear. Looks like it could've been a timing issue. On recent releases a logic to restart the registration timer during some timing conditions caused by configuration sequence was added. Since you are using an old code that could be the reason why it worked after reconfiguring the tunnel interface.

P.D Make sure you mark this thread as answered so it can help others.

View solution in original post

Gustavo Medina · ‎05-14-2013

Hello Abdelilah,

The problem here is on the Spoke; when multiple GRE over IPSec tunnels are sourced from the same interface, and at least one is an mGRE tunnel it becomes impossible to guarantee proper Tunnel interface selection. This means that an incoming IPSec encrypted packet may be sent to the wrong Tunnel interface and then be unable to be decrypted as that Tunnel does not have the encryption/decryption information for that tunnel. We will find that one of the tunnels may work and all others may fail.

Encryption/Decryption information is stored in a data structure that we call the SADB (Security Association Database). The SADB is associated on a per interface basis and stores the encryption/decryption rules and keying data. If a packet comes in to the wrong interface, and therefore the wrong SADB, the packet will be dropped because we cannot find the SPI (Security Parameter Index) in that SADB.

When it is not possible to select the proper interface, we share the same SADB across the tunnel interfaces associated with the same tunnel source. Now we can be guaranteed that a packet can be decrypted and after decryption we have some more information to select the appropriate tunnel interface to drop the decrypted traffic in.

So that's why in your case we need to use Shared Tunnel Protection

interface Tunnel0

bandwidth 10000

ip address 10.1.1.1 255.255.255.0

ip mtu 1400

ip nhrp authentication dmvpn1

ip nhrp map multicast 190.1.1.1

ip nhrp map 10.1.1.4 190.1.1.1

ip nhrp network-id 123

ip nhrp holdtime 600

ip nhrp nhs 10.1.1.4

ip nhrp registration timeout 300

tunnel source FastEthernet0/0

tunnel mode gre multipoint

tunnel key 123

tunnel protection ipsec profile DMVPN shared

!

interface Tunnel1

bandwidth 10000

ip address 10.2.2.1 255.255.255.0

ip mtu 1400

ip nhrp authentication dmvpn2

ip nhrp map multicast 190.1.2.1

ip nhrp map 10.2.2.4 190.1.2.1

ip nhrp network-id 124

ip nhrp holdtime 600

ip nhrp nhs 10.2.2.4

ip nhrp registration timeout 300

tunnel source FastEthernet0/0

tunnel mode gre multipoint

tunnel key 124

tunnel protection ipsec profile DMVPN shared

Important notes:

a) The shared keyword must be applied to the tunnel protection command

b) The tunnel protection on each of the tunnels that use the shared interface must Use the same ipsec profile. This profile should be different than other GRE/IPSec interfaces that are not sharing the SADB.

c) Each of the tunnels must have a unique tunnel key used to identify the traffic after it is decrypted. The tunnel key must match the key defined on the other side of the tunnel.

Abdelilah Benrida · ‎05-14-2013

thank you very much

Abdelilah Benrida · ‎05-14-2013

Hello Gustavo

thank you it works

Abdelilah Benrida · ‎05-15-2013

but when i add an other SPOKE there is a problem :

HUB

| |

SPOKE1___ ISP__SPOKE2

HUB:

!

crypto isakmp policy 10

encr aes

hash md5

authentication pre-share

group 2

crypto isakmp key cisco address 0.0.0.0 0.0.0.0

!

crypto ipsec transform-set AES_MD5 esp-aes esp-md5-hmac

!

crypto ipsec profile DMVPN

set transform-set AES_MD5

!

interface Tunnel0

bandwidth 1000

ip address 10.1.1.4 255.255.255.0

no ip redirects

ip mtu 1400

no ip next-hop-self eigrp 123

ip nhrp authentication dmvpn1

ip nhrp map multicast dynamic

ip nhrp network-id 123

no ip split-horizon eigrp 123

tunnel source FastEthernet0/0

tunnel mode gre multipoint

tunnel key 123

tunnel protection ipsec profile DMVPN

!

interface Tunnel1

bandwidth 1000

ip address 10.2.2.4 255.255.255.0

no ip redirects

ip mtu 1400

no ip next-hop-self eigrp 124

ip nhrp authentication dmvpn2

ip nhrp map multicast dynamic

ip nhrp network-id 124

no ip split-horizon eigrp 124

tunnel source FastEthernet1/0

tunnel mode gre multipoint

tunnel key 124

tunnel protection ipsec profile DMVPN

!

router eigrp 123

network 10.1.1.0 0.0.0.255

network 172.16.4.0 0.0.0.255

no auto-summary

!

router eigrp 124

network 10.2.2.0 0.0.0.255

network 172.16.4.0 0.0.0.255

no auto-summary

!

SPOKE1 :

!

crypto isakmp policy 10

encr aes

hash md5

authentication pre-share

group 2

crypto isakmp key cisco address 0.0.0.0 0.0.0.0

!

crypto ipsec transform-set AES_MD5 esp-aes esp-md5-hmac

!

crypto ipsec profile DMVPN

set transform-set AES_MD5

!

interface Tunnel0

bandwidth 1000

ip address 10.1.1.1 255.255.255.0

no ip redirects

ip mtu 1400

ip nhrp authentication dmvpn1

ip nhrp map multicast 190.1.1.1

ip nhrp map 10.1.1.4 190.1.1.1

ip nhrp network-id 123

ip nhrp holdtime 600

ip nhrp nhs 10.1.1.4

ip nhrp registration timeout 300

tunnel source FastEthernet0/0

tunnel mode gre multipoint

tunnel key 123

tunnel protection ipsec profile DMVPN shared

!

interface Tunnel1

bandwidth 1000

ip address 10.2.2.1 255.255.255.0

no ip redirects

ip mtu 1400

ip nhrp authentication dmvpn2

ip nhrp map multicast 190.1.2.1

ip nhrp map 10.2.2.4 190.1.2.1

ip nhrp network-id 124

ip nhrp holdtime 600

ip nhrp nhs 10.2.2.4

ip nhrp registration timeout 300

tunnel source FastEthernet0/0

tunnel mode gre multipoint

tunnel key 124

tunnel protection ipsec profile DMVPN shared

!

router eigrp 123

network 10.1.1.0 0.0.0.255

network 172.16.1.0 0.0.0.255

no auto-summary

!

router eigrp 124

network 10.2.2.0 0.0.0.255

network 172.16.1.0 0.0.0.255

no auto-summary

!

SPOKE2 :

!

crypto isakmp policy 10

encr aes

hash md5

authentication pre-share

group 2

crypto isakmp key cisco address 0.0.0.0 0.0.0.0

!

crypto ipsec transform-set AES_MD5 esp-aes esp-md5-hmac

!

crypto ipsec profile DMVPN

set transform-set AES_MD5

!

interface Tunnel0

bandwidth 1000

ip address 10.1.1.2 255.255.255.0

no ip redirects

ip mtu 1400

ip nhrp authentication dmvpn1

ip nhrp map multicast 190.1.1.1

ip nhrp map 10.1.1.4 190.1.1.1

ip nhrp network-id 123

ip nhrp holdtime 600

ip nhrp nhs 10.1.1.4

ip nhrp registration timeout 300

tunnel source FastEthernet0/0

tunnel mode gre multipoint

tunnel key 123

tunnel protection ipsec profile DMVPN shared

!

interface Tunnel1

bandwidth 1000

ip address 10.2.2.2 255.255.255.0

no ip redirects

ip mtu 1400

ip nhrp authentication dmvpn2

ip nhrp map multicast 190.1.2.1

ip nhrp map 10.2.2.4 190.1.2.1

ip nhrp network-id 124

ip nhrp holdtime 600

ip nhrp nhs 10.2.2.4

ip nhrp registration timeout 300

tunnel source FastEthernet0/0

tunnel mode gre multipoint

tunnel key 124

tunnel protection ipsec profile DMVPN shared

!

router eigrp 123

network 10.1.1.0 0.0.0.255

network 172.16.2.0 0.0.0.255

no auto-summary

!

router eigrp 124

network 10.2.2.0 0.0.0.255

network 172.16.2.0 0.0.0.255

no auto-summary

!

HUB:

HUB#sh ip nhrp

10.1.1.1/32 via 10.1.1.1, Tunnel0 created 00:15:17, expire 00:09:21

Type: dynamic, Flags: authoritative unique registered

NBMA address: 191.1.1.11

10.1.1.2/32 via 10.1.1.2, Tunnel0 created 00:12:09, expire 00:07:50

Type: dynamic, Flags: authoritative unique registered

NBMA address: 191.1.1.12

10.2.2.1/32, Tunnel1 created 00:02:57, expire 00:00:07

Type: incomplete, Flags: negative

Cache hits: 7

10.2.2.2/32 via 10.2.2.2, Tunnel1 created 00:12:00, expire 00:07:58

Type: dynamic, Flags: authoritative unique registered

NBMA address: 191.1.1.12

HUB can't have the NBMA adress for 10.2.2.1 for SPOKE1

HUB#ping 10.2.2.1

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 10.2.2.1, timeout is 2 seconds:

.....

Success rate is 0 percent (0/5)

*Mar 1 00:45:18.431: NHRP: MACADDR: if_in null netid-in 0 if_out Tunnel1 netid-out 124

*Mar 1 00:45:18.435: NHRP: Checking for delayed event 0.0.0.0/10.2.2.1 on list (Tunnel1).

*Mar 1 00:45:18.435: NHRP: No node found..

*Mar 1 00:45:07.131: NHRP: MACADDR: if_in null netid-in 0 if_out Tunnel1 netid-out 124

*Mar 1 00:45:07.131: NHRP: Checking for delayed event 0.0.0.0/10.2.2.1 on list (Tunnel1).

*Mar 1 00:48:30.759: NHRP: Checking for delayed event 0.0.0.0/10.2.2.1 on list (Tunnel1).

*Mar 1 00:48:30.763: NHRP: No node found.

*Mar 1 00:48:30.763: NHRP: Attempting to send packet via DEST 10.2.2.1

*Mar 1 00:48:30.767: NHRP: Send Resolution Request via Tunnel1 vrf 0, packet size: 82

*Mar 1 00:48:30.771: src: 10.2.2.4, dst: 10.2.2.1

*Mar 1 00:48:30.771: NHRP: Encapsulation failed for destination 10.2.2.1 out Tunnel1

SPOKE1#

*Mar 1 00:53:38.695: NHRP: Setting retrans delay to 64 for nhs dst 10.2.2.4

*Mar 1 00:53:38.699: NHRP: Attempting to send packet via DEST 10.2.2.4

*Mar 1 00:53:38.699: NHRP: Encapsulation succeeded. Tunnel IP addr 190.1.2.1

*Mar 1 00:53:38.703: NHRP: Send Registration Request via Tunnel1 vrf 0, packet size: 82

*Mar 1 00:53:38.711: src: 10.2.2.1, dst: 10.2.2.4

*Mar 1 00:53:38.715: NHRP: 82 bytes out Tunnel1

no reply from the HUB

SPOKE1#ping 10.2.2.4

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 10.2.2.4, timeout is 2 seconds:

.....

Success rate is 0 percent (0/5)

the SPOKE can't reach 10.2.2.4

after a few time :

HUB#sh ip nhrp

10.1.1.1/32 via 10.1.1.1, Tunnel0 created 00:25:03, expire 00:09:35

Type: dynamic, Flags: authoritative unique registered used

NBMA address: 191.1.1.11

10.1.1.2/32 via 10.1.1.2, Tunnel0 created 00:21:55, expire 00:08:03

Type: dynamic, Flags: authoritative unique registered

NBMA address: 191.1.1.12

10.2.2.2/32 via 10.2.2.2, Tunnel1 created 00:21:47, expire 00:08:12

Type: dynamic, Flags: authoritative unique registered

NBMA address: 191.1.1.12

only 3 tunnels

Abdelilah Benrida · ‎05-15-2013

i need help please.

Gustavo Medina · ‎05-15-2013

Hello,

What version are you running on the failing spoke?

Can you try removing the entire tunnel config and pasting it again after a couple of minutes?

Abdelilah Benrida · ‎05-15-2013

Hello

I configured everything again but the problem persists

if i use just one tunnel interface on spoke and change the adresse 10.2.2.4 to 10.1.1.5 this will be a good choice?

i will put the two mapping for the two HUB interfaces in that tunnel interface is that correct?

Gustavo Medina · ‎05-15-2013

What is the version you are using? could you provide the debugs outputs from HUB and Failing Spoke so I can take a look?

Abdelilah Benrida · ‎05-15-2013

HUB#sh version

Cisco IOS Software, 3600 Software (C3640-IK9O3S-M), Version 12.4(3), RELEASE SOFTWARE (fc2)

Technical Support: http://www.cisco.com/techsupport

Compiled Fri 22-Jul-05 07:14 by hqluong

ROM: ROMMON Emulation Microcode

ROM: 3600 Software (C3640-IK9O3S-M), Version 12.4(3), RELEASE SOFTWARE (fc2)

HUB uptime is 0 minutes

System returned to ROM by unknown reload cause - suspect boot_data[BOOT_COUNT] 0x0, BOOT_COUNT 0, BOOTDATA 19

System image file is "tftp://255.255.255.255/unknown"

This product contains cryptographic features and is subject to United

States and local country laws governing import, export, transfer and

use. Delivery of Cisco cryptographic products does not imply

third-party authority to import, export, distribute or use encryption.

Importers, exporters, distributors and users are responsible for

compliance with U.S. and local country laws. By using this product you

agree to comply with applicable laws and

*Mar 1 00:00:29.943: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel0, changed state to up

*Mar 1 00:00:30.563: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel1, changed state to upregulations. If you are unable

to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:

http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to

export@cisco.com.

Cisco 3640 (R4700) processor (revision 0xFF) with 124928K/6144K bytes of memory.

Processor board ID FF1045C5

R4700 CPU at 100MHz, Implementation 33, Rev 1.2

4 FastEthernet interfaces

DRAM configuration is 64 bits wide with parity enabled.

125K bytes of NVRAM.

8192K bytes of processor board System flash (Read/Write)

Configuration register is 0x2102

SPOKE1#sh version

Cisco IOS Software, 3600 Software (C3640-IK9O3S-M), Version 12.4(3), RELEASE SOFTWARE (fc2)

Technical Support: http://www.cisco.com/techsupport

Compiled Fri 22-Jul-05 07:14 by hqluong

ROM: ROMMON Emulation Microcode

ROM: 3600 Software (C3640-IK9O3S-M), Version 12.4(3), RELEASE SOFTWARE (fc2)

SPOKE1 uptime is 1 minute

System returned to ROM by unknown reload cause - suspect boot_data[BOOT_COUNT] 0x0, BOOT_COUNT 0, BOOTDATA 19

System image file is "tftp://255.255.255.255/unknown"

This product contains cryptographic features and is subject to United

States and local country laws governing import, export, transfer and

use. Delivery of Cisco cryptographic products does not imply

third-party authority to import, export, distribute or use encryption.

Importers, exporters, distributors and users are responsible for

compliance with U.S. and local country laws. By using this product you

agree to comply with applicable laws and regulations. If you are unable

to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:

http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to

export@cisco.com.

Cisco 3640 (R4700) processor (revision 0xFF) with 124928K/6144K bytes of memory.

Processor board ID FF1045C5

R4700 CPU at 100MHz, Implementation 33, Rev 1.2

1 FastEthernet interface

DRAM configuration is 64 bits wide with parity enabled.

125K bytes of NVRAM.

8192K bytes of processor board System flash (Read/Write)

Configuration register is 0x2102

Gustavo Medina · ‎05-15-2013

wow that's old. I will need the complete configurations and debug outputs to find out what is the problem.

Abdelilah Benrida · ‎05-15-2013

i configured it one last time and it worked i really don't know why!

thank you Gustavo, very much for your help ,your comments were very informative

Gustavo Medina · ‎05-15-2013

Good to hear. Looks like it could've been a timing issue. On recent releases a logic to restart the registration timer during some timing conditions caused by configuration sequence was added. Since you are using an old code that could be the reason why it worked after reconfiguring the tunnel interface.

P.D Make sure you mark this thread as answered so it can help others.