Solved: Cant ping spoke to spoke IPSEC/GRE Deployment

Mark Rigby · ‎02-21-2010

Greetings, i have a hub and spoke IPSEC/GRE configuration connecting together 5 routers. I can ping from spoke to spoke when connected to each spoke router but if i add the inside interface as the source address i cant ping between spokes.

I think it may be a NAT or Crypto issue but any assistance would be much appreciated, yes i could create seperate tunnels between spokes or use DMVPN but the customer wants all traffic to go via the HUB site.

Ping another spoke router

#ping 192.168.20.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.20.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 120/123/124 ms

Traceroute to another spoke router

#traceroute 192.168.20.1

Type escape sequence to abort.
Tracing the route to 192.168.20.1

1 10.255.255.13 56 msec 56 msec 52 msec
2 10.255.255.2 116 msec * 116 msec

Ping another spoke router with source-interface statement

#ping 192.168.20.1 source vlan 1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.20.1, timeout is 2 seconds:
Packet sent with a source address of 192.168.50.1
.....
Success rate is 0 percent (0/5)

Show IP Route on spoke router with LAN IP 192.168.50.1

bhx-ro-877#sh ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route

Gateway of last resort is 0.0.0.0 to network 0.0.0.0

D    192.168.30.0/24 [90/2882560] via 10.255.255.13, 01:35:17, Tunnel0
     81.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C       81.**.**.**/32 is directly connected, Dialer1
C       81.**.**.**/24 is directly connected, Dialer1
D    192.168.10.0/24 [90/1602560] via 10.255.255.13, 02:46:54, Tunnel0
D    192.168.40.0/24 [90/2882560] via 10.255.255.13, 02:46:54, Tunnel0
D    192.168.20.0/24 [90/2882560] via 10.255.255.13, 02:46:54, Tunnel0
     10.0.0.0/30 is subnetted, 4 subnets
D       10.255.255.8 [90/2880000] via 10.255.255.13, 02:46:54, Tunnel0
C       10.255.255.12 is directly connected, Tunnel0
D       10.255.255.0 [90/2880000] via 10.255.255.13, 02:46:54, Tunnel0
D       10.255.255.4 [90/2880000] via 10.255.255.13, 02:46:54, Tunnel0
C    192.168.50.0/24 is directly connected, Vlan1
S*   0.0.0.0/0 is directly connected, Dialer1

Crypto Map and Tunnel Interface on Spoke Router - 192.168.50.1

interface Tunnel0
bandwidth 8000
ip address 10.255.255.14 255.255.255.252
ip mtu 1420
ip nbar protocol-discovery
qos pre-classify
tunnel source Dialer1
tunnel destination 81.**.**.**
tunnel path-mtu-discovery
crypto map cryptomap

!

crypto map cryptomap 1 ipsec-isakmp
description ****** Link to HUB ******
set peer 81.**.**.**
set security-association lifetime seconds 86400
set transform-set 3DesL2L
set pfs group2
match address 101

!

access-list 100 remark ****** NAT ACL ******
access-list 100 deny ip 192.168.50.0 0.0.0.255 192.168.0.0 0.0.255.255
access-list 100 deny ip 192.168.50.0 0.0.0.255 172.31.255.0 0.0.0.255
access-list 100 permit ip 192.168.50.0 0.0.0.255 any
access-list 101 remark ****** Link to hqhnswth-ro-877 ******
access-list 101 permit ip 192.168.50.0 0.0.0.255 192.168.0.0 0.0.255.255

!

route-map nonat permit 1
match ip address 100

!

ip nat inside source route-map nonat interface Dialer1 overload

!

router eigrp 1
passive-interface Dialer1
passive-interface Vlan1
network 10.255.255.0 0.0.0.255
network 192.168.50.0
no auto-summary
eigrp stub connected summary

Crypto Map and Tunnel Interface on HUB Router - 192.168.10.1

interface Tunnel3
bandwidth 8000
ip address 10.255.255.13 255.255.255.252
ip mtu 1420
ip nbar protocol-discovery
qos pre-classify
tunnel source Dialer1
tunnel destination 81.**.**.**
tunnel path-mtu-discovery
crypto map hqmap

!

interface Tunnel0
bandwidth 8000
ip address 10.255.255.1 255.255.255.252
ip mtu 1420
tunnel source Dialer1
tunnel destination 217.**.**.**
tunnel path-mtu-discovery
crypto map hqmap

!

crypto map hqmap 1 ipsec-isakmp
description ****** Link to 192.168.20.0/24******
set peer 217.**.**.**
set security-association lifetime seconds 86400
set transform-set 3DesL2L
set pfs group2
match address 101
!
crypto map hqmap 4 ipsec-isakmp
description ****** Link to 192.168.50.0/24 ******
set peer 81.**.**.**
set security-association lifetime seconds 86400
set transform-set 3DesL2L
set pfs group2
match address 104

!

access-list 100 remark ****** NAT ACL ******
access-list 100 deny ip 192.168.10.0 0.0.0.255 192.168.0.0 0.0.255.255
access-list 100 permit ip 192.168.10.0 0.0.0.255 any

!
access-list 101 remark ****** Link to 192.168.20.0/24 ******
access-list 101 permit ip 192.168.10.0 0.0.0.255 192.168.20.0 0.0.0.255

!
access-list 104 remark ****** Link to 192.168.50.0/24 ******
access-list 104 permit ip 192.168.10.0 0.0.0.255 192.168.50.0 0.0.0.255

!

ip nat inside source route-map nonat interface Dialer1 overload

!

router eigrp 1
passive-interface Dialer1
passive-interface Vlan1
network 10.255.255.0 0.0.0.255
network 192.168.0.0 0.0.255.255
no auto-summary

Show IP Route on HUB Router

#sh ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route

Gateway of last resort is 0.0.0.0 to network 0.0.0.0

D    192.168.30.0/24 [90/1602560] via 10.255.255.6, 01:45:27, Tunnel1
     81.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C       81.**.**.**/24 is directly connected, Dialer1
C       81.**.**.**//32 is directly connected, Dialer1
C    192.168.10.0/24 is directly connected, Vlan1
D    192.168.40.0/24 [90/1602560] via 10.255.255.10, 1d23h, Tunnel2
D    192.168.20.0/24 [90/1602560] via 10.255.255.2, 06:11:01, Tunnel0
     10.0.0.0/30 is subnetted, 4 subnets
C       10.255.255.8 is directly connected, Tunnel2
C       10.255.255.12 is directly connected, Tunnel3
C       10.255.255.0 is directly connected, Tunnel0
C       10.255.255.4 is directly connected, Tunnel1
D    192.168.50.0/24 [90/1602560] via 10.255.255.14, 02:57:08, Tunnel3
S*   0.0.0.0/0 is directly connected, Dialer1

Ping spoke from HUB Router

#ping 192.168.20.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.20.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 72/72/76 ms

Ping spoke from HUB Router using "source-interface" statement

#ping 192.168.20.1 source vlan 1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.20.1, timeout is 2 seconds:
Packet sent with a source address of 192.168.10.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 72/72/76 ms

Lei Tian · ‎02-22-2010

Hi Mark,

It looks about right. Just 2 things need to change

1, change the ipsec mode to transport instead of tunnel.

2, remove network 0.0.0.0 statement from eigrp process, you don’t want advertise your tunnel end point thought tunnel; otherwise you will have recursive lookup.

HTH,

Lei Tian

View solution in original post

dbass · ‎02-22-2010

It looks like your original problem is with your crypto ACL. What you are trying to do is encrypt GRE traffic from the remote site to the hub and send the actual user data within the GRE tunnel. So, this is what needs to happen: the router gets a packet from 192.168.50.x that is destined to 192.168.20.x. It looks up the route and determines the next hop is the other end of the GRE tunnel, and should forward it out that interface. When it forwards it out that interface it encapsulates the original packet with GRE and sends it to the physical interface to be transmitted on to the wire. That is when the crypto engine examines the packet and determines that it should be encrypted, which it then encrypts and transmits the packet. Now, because your crypto ACL is wrong the router is conflicted in whether to encrypt the original packet in IPSEC or GRE (then IPSEC), and dropping it. I would be willing to bet that if you do a "sh cry ips sa" on both ends that you will see incrementing send errors or encap/decap errors if there is even an SA established. Hope I explained it in a manner that helps you understand it better.

The reason why doing a straight ping works, but an extended (with sourced from the internal network) doesn't also directly ties to the incorrect crypto ACL. The regular ping will be sent with a source address of 10.255.255.14, which is not referenced in the crypto ACL so there is no conflict.

The correct crypto ACL would have a source address of the public address on the dialer interface (outside interface) and destination of the public address on the hub site, and the opposite for the hub site crypto ACL. You can specify GRE as the protocol (ie: permit gre host x.x.x.x host x.x.x.x), or all IP (ie: permit ip host x.x.x.x host x.x.x.x) depending on what you prefer. Sometime's it helps to be able to ping the public address with it not being encrypted, which is why I prefer to just specifically do the GRE only traffic.

One thing to remember for future use as well...GRE/IPSEC and DMVPN is the same technology at heart, so if you cannot get the former working you will run in to the same problems with the later. In fact the later is more complicated of a config and technology, so more likely it will be harder to get up without a thorough understanding of the technology. This goes for any technology...ie, if you can't get EIGRP running between 2 routers switching to BGP is probably not going to get you very far.

View solution in original post

Lei Tian · ‎02-23-2010

Hi Mark,

As I mentioned you still have "no ip next-hop-self eigrp 1" on your hub; you need remove that statement for phase 3 to work properly.

If i remove the statement " no ip next-hop-self eigrp 1" ie change it to " ip next-hop-self eigrp 1" the configuration becomes Phase2 and all spokes talk to one another via the HUB. Although when i removed the statement i dont think i allowed time for the tunnel interface to be reset so ill try that later.

Actually, it is the other way around. with "no ip next-hop-self eigrp 1" it will work as phase 2, and phase 3 do not need that statement on hub. After established the dynamic tunnel, traffic from spoke to spoke will be sent via the ipsec tunnel and will not process by hub router. Be aware remove "no ip next-hop-self eigrp 1" on hub router will reset all eigrp neighbors.

These are little difference between phase 3 and phase 2. When you send traffic from spoke 1 to spoke 2,in phase 2 ipsec tunnel is initialed by spoke 1; in phase 3 ipsec tunnel is initialed by spoke 2. From user point of view, you might not even notice the difference. What matter is in phase 2, before spoke to spoke tunnel established, traffic will be process switched by hub router; in phase 3 all traffic are cef switched. That gives phase 3 a big plus when you have lot spokes. Phase 3 also build up spoke to spoke tunnel quicker than phase 2. You can verify that by doing ping from spoke 1 to spoke 2 with phase 2 and phase 3, and compare the number of pkt been encrypted from "show crypto ipsec sa".

In term of best practice, it doesn’t hurt to put spoke as eigrp stub network, but it wont help anything unless you have dual hub. For dual hub situation, you want put spoke as stub network so it won’t become transit site.

This is a good document for migrating DMVPN phase 2 to phase 3; you can find some best practices in the doc as well.

http://www.cisco.com/en/US/prod/collateral/iosswrel/ps6537/ps6586/ps6660/ps6808/prod_white_paper0900aecd8055c34e_ps6658_Products_White_Paper.html

HTH,

Lei Tian

View solution in original post

Lei Tian · ‎02-24-2010

Hi Mark,

What you have seen is the correct behavior. In Phase 3, spoke will always point to hub as the next-hop in the routing table; what changed is the nhrp map. If you goto the spoke router and show ip nhrp brief, you will notice there will be one entry for remote spoke tunnel's (10.255.255.2 in your case), and another entry for remote spoke's LAN subnet (192.168.20.0 in your case). The later entry will be used when traffic send to 192.168.20.0 subnet.

And because of this feature, phase 3 allows you on hub router only advertise summary route to spokes.

HTH,

Lei Tian

View solution in original post

Lei Tian · ‎02-24-2010

Hi Mark,

Yes, you can apply ACL on tunnel interface to deny certain type of traffic on either direction.

After read your requirement, I think reflexive acl might be a good fit here. Say provider site is spoke 1 and the customer site is spoke 2; any traffic originate by spoke 1 and return traffic can pass, but traffic originate by spoke 2 would be blocked. Here is the configuration guide if you are interested with reflexive acl.

http://www.cisco.com/en/US/docs/ios/sec_data_plane/configuration/guide/sec_cfg_ip_filter_ps6350_TSD_Products_Configuration_Guide_Chapter.html#wp1000897

HTH,

Lei Tian

View solution in original post

Lei Tian · ‎02-21-2010

Hi Mark,

Can you post "show ip route" output from the other spoke?

Thanks

Lei Tian

Mark Rigby · ‎02-21-2010

Thank you for your post, please see the output from a second spoke, this is the one im pinging in the above examples with LAN IP 192.168.20.1.

Regards

#sh ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route

Gateway of last resort is 0.0.0.0 to network 0.0.0.0

D    192.168.30.0/24 [90/2882560] via 10.255.255.1, 02:06:42, Tunnel0
     81.0.0.0/32 is subnetted, 1 subnets
C       81.**.**.** is directly connected, Dialer1
D    192.168.10.0/24 [90/1602560] via 10.255.255.1, 06:32:16, Tunnel0
D    192.168.40.0/24 [90/2882560] via 10.255.255.1, 06:32:16, Tunnel0
C    192.168.20.0/24 is directly connected, Vlan1
     10.0.0.0/30 is subnetted, 4 subnets
D       10.255.255.8 [90/2880000] via 10.255.255.1, 06:32:16, Tunnel0
D       10.255.255.12 [90/2880000] via 10.255.255.1, 06:32:16, Tunnel0
C       10.255.255.0 is directly connected, Tunnel0
D       10.255.255.4 [90/2880000] via 10.255.255.1, 06:32:16, Tunnel0
D    192.168.50.0/24 [90/2882560] via 10.255.255.1, 03:18:22, Tunnel0
C    217.**.**.**24 is directly connected, Dialer1
S*   0.0.0.0/0 is directly connected, Dialer1

Lei Tian · ‎02-21-2010

Hi Mark,

The routing looks fine. I suspect your ICMP traffic is not been encrypted at all. If you issue "show crypto ipsec sa" on your spoke do you see number of packets been encrypted and decrypted?

Thanks

Lei Tian

Lei Tian · ‎02-21-2010

Hi Mark,

Just noticed one thing, the interested ACL on your hub site is not same as the ACL on your spoke router.

Please change the ACL 101 to

access-list 101 per ip 192.168.0.0 0.0.255.255 192.168.20.0 0.0.0.255

ACL 104 to

access-list 104 per ip 192.168.0.0 0.0.255.255 192.168.50.0 0.0.0.255

HTH,

Lei Tian

Mark Rigby · ‎02-22-2010

Thank you Lei, i shall try changing the statements on the HUB router later on today and let you know how we get on.

Regards

paolo bevilacqua · ‎02-22-2010

True DMVPN connects spokes w/o hub involved

Lei Tian · ‎02-22-2010

You welcome Mark.

As a note here, when using GRE+IPSEC, it is easier apply crypto-map on the physical interface and configure the ACL to permit GRE traffic between tunnel endpoints.

Another way to do it is as p.bevilacqua mentioned uses phase 1 DMVPN for hub and spoke only network.

HTH,

Lei Tian

Mark Rigby · ‎02-22-2010

Thank for the replies guys, yes i think it's time to go down the DMVPN route rather than trying to get a non standard configuration working properly.

Can i ask what the difference is between DMVPN Hub and Spoke and DMVPN Full Mesh? Would i be correct in making the following assumptions.

DMVPN Hub and Spoke - All traffic goes via the hub site?
DMVPN Full Mesh - Spoke routers can initiate runnels with other spoke routers using the hub to determine the IP address of the second spoke router?

Regards

Lei Tian · ‎02-22-2010

Yes Mark, that is correct.

Mark Rigby · ‎02-22-2010

Cheers this is my full mesh config so far for the HUB site and a single spoke so far. Just need to add a dynamic cryto map for VPN clients.

! HUB SITE !

crypto isakmp policy 1
hash sha
authentication pre-share
encryption 3des
group 2
lifetime 86400
!
crypto isakmp key hhDT8Fv1ZcM3T address 0.0.0.0 0.0.0.0
!
crypto ipsec transform-set 3DesSecure esp-3des esp-sha-hmac
mode tunnel
!
crypto ipsec profile dmvpn
set transform-set 3DesSecure
set security-association lifetime seconds 86400
set security-association lifetime kilobytes 4608000
!
interface Tunnel 0
description ****** DMVPN GRE Tunnel ******
ip address 10.255.255.1 255.255.255.0
bandwidth 1000
delay 1000
ip nhrp holdtime 360
ip nhrp network-id 100000
ip nhrp authentication 1zhGo5W3m26Cr
ip mtu 1400
ip tcp adjust-mss 1360
ip nhrp map multicast dynamic
tunnel source Dialer1
tunnel mode gre multipoint
tunnel key 100000
tunnel protection ipsec profile dmvpn
no ip split-horizon eigrp 1
no ip next-hop-self eigrp 1
!
router eigrp 1
network 10.255.255.1 0.0.0.0
network 0.0.0.0 255.255.255.255
no auto-summary
!
interface Dialer1
description ****** Outside Interface ******
ip nat outside
!
interface Vlan1
description ****** Inside Interface ******
ip nat inside
!
ip nat inside source list 100 interface Dialer1 overload
!
access-list 100 remark ****** NAT ACL ******
access-list 100 deny ip 192.168.0.0 0.0.255.255 192.168.0.0 0.0.255.255
access-list 100 permit ip 192.168.10.0 0.0.0.255

!

! SPOKE SITE !

crypto isakmp policy 10
hash sha
authentication pre-share
encryption 3des
group 2
lifetime 86400
!
crypto isakmp key hhDT8Fv1ZcM3T address 0.0.0.0 0.0.0.0
!
crypto ipsec transform-set 3DesSecure esp-3des esp-sha-hmac
mode tunnel
!
crypto ipsec profile dmvpn
set transform-set 3DesSecure
set security-association lifetime seconds 86400
set security-association lifetime kilobytes 4608000
!
interface Tunnel 0
description ****** DMVPN GRE Tunnel ******
ip address 10.255.255.2 255.255.255.0
bandwidth 1000
delay 1000
ip nhrp holdtime 360
ip nhrp network-id 100000
ip nhrp authentication 1zhGo5W3m26Cr
ip mtu 1400
ip tcp adjust-mss 1360
ip nhrp nhs 10.255.255.1
ip nhrp map multicast **HUB External IP Address **
ip nhrp map 10.255.255.1 **HUB External IP Address **
tunnel source Dialer1
tunnel mode gre multipoint
tunnel key 100000
tunnel protection ipsec profile dmvpn
!
router eigrp 1
network 10.255.255.2 0.0.0.0
network 0.0.0.0 255.255.255.255
no auto-summary
!
interface Dialer1
description ****** Outside Interface ******
ip nat outside
!
interface Vlan1
description ****** Inside Interface ******
ip nat inside
!
ip nat inside source list 100 interface Dialer1 overload
!
access-list 100 remark ****** NAT ACL ******
access-list 100 deny ip 192.168.0.0 0.0.255.255 192.168.0.0 0.0.255.255
access-list 100 permit ip 192.168.20.0 0.0.0.255

Lei Tian · ‎02-22-2010

Hi Mark,

It looks about right. Just 2 things need to change

1, change the ipsec mode to transport instead of tunnel.

2, remove network 0.0.0.0 statement from eigrp process, you don’t want advertise your tunnel end point thought tunnel; otherwise you will have recursive lookup.

HTH,

Lei Tian

Mark Rigby · ‎02-22-2010

Thank you for the recommendations, would you suggest advertising the network 192.168.0.0/16 as all the sites fall under this range?

Regards

Lei Tian · ‎02-22-2010

Hi Mark,

192.168.0.0/16 is fine. The bottom line is only advertising the protected LAN segment, 192.168.0.0 on your case and tunnel IP 10.255.255.0 on your case through tunnel.

HTH,

Lei Tian

dbass · ‎02-22-2010

It looks like your original problem is with your crypto ACL. What you are trying to do is encrypt GRE traffic from the remote site to the hub and send the actual user data within the GRE tunnel. So, this is what needs to happen: the router gets a packet from 192.168.50.x that is destined to 192.168.20.x. It looks up the route and determines the next hop is the other end of the GRE tunnel, and should forward it out that interface. When it forwards it out that interface it encapsulates the original packet with GRE and sends it to the physical interface to be transmitted on to the wire. That is when the crypto engine examines the packet and determines that it should be encrypted, which it then encrypts and transmits the packet. Now, because your crypto ACL is wrong the router is conflicted in whether to encrypt the original packet in IPSEC or GRE (then IPSEC), and dropping it. I would be willing to bet that if you do a "sh cry ips sa" on both ends that you will see incrementing send errors or encap/decap errors if there is even an SA established. Hope I explained it in a manner that helps you understand it better.

The reason why doing a straight ping works, but an extended (with sourced from the internal network) doesn't also directly ties to the incorrect crypto ACL. The regular ping will be sent with a source address of 10.255.255.14, which is not referenced in the crypto ACL so there is no conflict.

The correct crypto ACL would have a source address of the public address on the dialer interface (outside interface) and destination of the public address on the hub site, and the opposite for the hub site crypto ACL. You can specify GRE as the protocol (ie: permit gre host x.x.x.x host x.x.x.x), or all IP (ie: permit ip host x.x.x.x host x.x.x.x) depending on what you prefer. Sometime's it helps to be able to ping the public address with it not being encrypted, which is why I prefer to just specifically do the GRE only traffic.

One thing to remember for future use as well...GRE/IPSEC and DMVPN is the same technology at heart, so if you cannot get the former working you will run in to the same problems with the later. In fact the later is more complicated of a config and technology, so more likely it will be harder to get up without a thorough understanding of the technology. This goes for any technology...ie, if you can't get EIGRP running between 2 routers switching to BGP is probably not going to get you very far.