Re: DMVPN - spoke to spoke comm not working

mikepro · ‎02-01-2019

Hi,

I'm in the process of designing a spoke to spoke DMVPN solution with iBGP routing.

The spoke to hub communication works fine (PC1 & PC2 can successfully ping 10.10.1.1) but not the spoke to spoke's (PC1 & PC2 can't ping each other).

Any idea why?

Hardware:

Spokes: Cellular routers connecting through SIMs.

Hub: Cisco 2900 Series connecting through ADSL.

Diagram:

GRE addressing:

Hub: 192.168.1.1

Spoke1: 192.168.1.2

Spoke2: 192.168.1.3

Hub's running conf:

crypto isakmp policy 10

encr 3des

hash md5

authentication pre-share

group 2

lifetime 3600

crypto isakmp key xxxx address 0.0.0.0

crypto isakmp invalid-spi-recovery

!

crypto ipsec transform-set DMVPN esp-3des esp-md5-hmac

mode transport

!

crypto ipsec profile DMVPN

set security-association lifetime seconds 86400

set transform-set DMVPN

!

interface Loopback0

ip address 10.10.1.1 255.255.255.0

!

interface Tunnel0

ip address 192.168.1.1 255.255.255.0

no ip redirects

ip nhrp authentication xxxx

ip nhrp map multicast dynamic

ip nhrp network-id 1

no ip nhrp record

no ip nhrp cache non-authoritative

keepalive 20 3

tunnel source GigabitEthernet0/0

tunnel mode gre multipoint

tunnel key 0

tunnel protection ipsec profile DMVPN

!

interface Embedded-Service-Engine0/0

no ip address

shutdown

!

interface GigabitEthernet0/0

ip address 86.x.x.x 255.255.255.240

duplex auto

speed auto

!

interface GigabitEthernet0/1

no ip address

shutdown

duplex auto

speed auto

!

router ospf 1

!

router bgp 65500

bgp router-id 192.168.1.1

bgp log-neighbor-changes

bgp listen range 192.168.1.0/24 peer-group DMVPN

bgp listen limit 500

neighbor DMVPN peer-group

neighbor DMVPN remote-as 65500

neighbor DMVPN timers 30 180

!

address-family ipv4

network 10.10.1.0 mask 255.255.255.0

network 192.168.1.0

neighbor DMVPN activate

neighbor DMVPN next-hop-self

exit-address-family

!

ip forward-protocol nd

!

no ip http server

no ip http secure-server

!

ip route 0.0.0.0 0.0.0.0 86.x.x.x

Hub's routing table:

Gateway of last resort is 86.x.x.x to network 0.0.0.0

S* 0.0.0.0/0 [1/0] via 86.x.x.x
10.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C 10.10.1.0/24 is directly connected, Loopback0
L 10.10.1.1/32 is directly connected, Loopback0
86.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C 86.x.x.x/28 is directly connected, GigabitEthernet0/0
L 86.x.x.x/32 is directly connected, GigabitEthernet0/0
B 192.168.0.0/24 [200/0] via 192.168.1.2, 01:51:54
192.168.1.0/24 is variably subnetted, 2 subnets, 2 masks
C 192.168.1.0/24 is directly connected, Tunnel0
L 192.168.1.1/32 is directly connected, Tunnel0
B 192.168.2.0/24 [200/0] via 192.168.1.3, 00:00:43

I thought it could be because the spokes don't know where to send the traffic toward one another so I added the following IP range to the Hub’s iBGP routing address-family so it advertises it to the spokes:

Network 192.168.0.0 mask 255.255.0.0

But it didn’t work… The spokes won't take it for some reasons.

Whereas they do take:

Network 10.10.1.0 mask 255.255.255.0

Which is also advertised by the Hub…

KR

Mike

grabonlee · ‎02-01-2019

Hello Mike,

Since I don't have a view of your Spoke configuration, there a few lines I suggest you add to the Hub and Spoke.

1. Add ip nhrp redirect on the Hub tunnel interface

2. Add neighbor peer-group route-reflector-client on the Hub BGP

3. On the Spokes add ip nhrp redirect.

Cheers

mikepro · ‎02-01-2019

Thanks!

I'll try this and come back to you.

grabonlee · ‎02-01-2019

Mike,

Another thing to note is that 192.168.0.0 and 192.168.2.0 are known networks by the Spokes and not the Hub, so those networks should be advertised from the Spokes and not the Hub.

mikepro · ‎02-04-2019

Hi,

They are.

ngkin2010 · ‎02-01-2019

Hi Mike,

As I remember, the "network" command used by BGP is referring to routing table, looking for the exact network learnt via different source of routing protocol (E.g static route / OSPF / directly connected). Otherwise, it won't advertise to peer.

That's the reason why 10.10.1.0/24 has advertised because it is the directly connected route in the routing table.
Neither 192.168.0.0/24 or 192.168.2.0/24 would be advertised because it learnt via BGP.

May I know why would you choose to use iBGP only rather than other protocol like OSPF? Anyway, try to fix the routing issue first.

Or if you would like to bypass the routing issue, could you place static routes on both spokes, and verify the connectivity between 2 spokes?

Last but not lease, would you consider to enable DMVPN phase 3 for better 'spoke-to-spoke' connection? Copy from others:

- "ip nhrp redirect" should be configured on the hub, which informs to the spoke that it can communicate to other intended spoke directly.
- "ip nhrp shortcut" should be configured on the spoke which is responsible to rewrite the CEF entry after getting the redirect message from hub.

mikepro · ‎02-04-2019

Hi,

Makes sense, thanks for the explanation.

I was asked to use BGP, it was part of the project specifications.

I wish I could.

Unfortunately the spokes (cellular routers) won't let me add a static route with the GRE as interface, it's not even in the dropbox...

I've informed our supplier about this.

I've added "ip nhrp redirect" on both the hub and spokes and "ip nhrp shortcut" on the spokes but no joy :/

Same with the route-reflection-client setting.

I thought it could simply be my PCs (behind the spoke routers) firewalls blocking the traffic but they are disabled...

Thanks all for your help so far.

mikepro · ‎02-04-2019

Question:

Is it normal that I can't ping the GRE client IPs from a spoke to another?

Yet the spokes do have a route for 192.168.1.0/24 through 192.168.1.1 (hub's GRE IP)

192.168.0.100 (behind spoke 1 - GRE IP is 192.168.1.2) can't ping spoke 2's GRE IP (192.168.1.3) for example.

grabonlee · ‎02-04-2019

I think it's best you post the complete configs for Hub and Spokes, and not just the Hub, so that we can have a better insight.

Secondly, whatever routing device you ping from must have the destination network in it's routing table or learn a default through a dynamic protocol or statically configured route.

mikepro · ‎02-04-2019

You're right.

Re-posting hub's as it changed slightly.

Hub's conf (current):

crypto isakmp policy 10

encr 3des

hash md5

authentication pre-share

group 2

lifetime 3600

crypto isakmp key xxxx address 0.0.0.0

crypto isakmp invalid-spi-recovery

!

crypto ipsec transform-set DMVPN esp-3des esp-md5-hmac

mode transport

!

crypto ipsec profile DMVPN

set security-association lifetime seconds 86400

set transform-set DMVPN

!

interface Loopback0

ip address 10.10.1.1 255.255.255.0

!

interface Tunnel0

ip address 192.168.1.1 255.255.255.0

no ip redirects

ip nhrp authentication xxxx

ip nhrp map multicast dynamic

ip nhrp network-id 1

ip nhrp redirect

keepalive 20 3

tunnel source GigabitEthernet0/0

tunnel mode gre multipoint

tunnel key 0

tunnel protection ipsec profile DMVPN

!

interface Embedded-Service-Engine0/0

no ip address

shutdown

!

interface GigabitEthernet0/0

ip address 86.x.x.x 255.255.255.240

duplex auto

speed auto

!

interface GigabitEthernet0/1

no ip address

shutdown

duplex auto

speed auto

!

router bgp 65500

bgp router-id 192.168.1.1

bgp log-neighbor-changes

bgp listen range 192.168.1.0/24 peer-group DMVPN

bgp listen limit 500

neighbor DMVPN peer-group

neighbor DMVPN remote-as 65500

neighbor DMVPN timers 30 180

!

address-family ipv4

network 10.10.1.0 mask 255.255.255.0

network 192.168.1.0

neighbor DMVPN activate

neighbor DMVPN route-reflector-client

neighbor DMVPN next-hop-self

exit-address-family

!

ip forward-protocol nd

!

no ip http server

no ip http secure-server

!

ip route 0.0.0.0 0.0.0.0 86.x.x.x

Hub's routing table:

Gateway of last resort is 86.x.x.x to network 0.0.0.0

S* 0.0.0.0/0 [1/0] via 86.x.x.x
10.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C 10.10.1.0/24 is directly connected, Loopback0
L 10.10.1.1/32 is directly connected, Loopback0
86.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C 86.x.x.x/28 is directly connected, GigabitEthernet0/0
L 86.x.x.x/32 is directly connected, GigabitEthernet0/0
B 192.168.0.0/24 [200/0] via 192.168.1.2, 01:51:54
192.168.1.0/24 is variably subnetted, 2 subnets, 2 masks
C 192.168.1.0/24 is directly connected, Tunnel0
L 192.168.1.1/32 is directly connected, Tunnel0
B 192.168.2.0/24 [200/0] via 192.168.1.3, 00:00:43

Spokes' conf (its the same for both, excepted, obviously, the GRE IPs and the LANs) :

As a reminder:

GRE addressing:

Spoke1: 192.168.1.2

Spoke2: 192.168.1.3

Spokes' LANs:

Spoke1: 192.168.0.0/24

Spoke2: 192.168.2.0/24

Spokes routing table:

This is Spoke1's.

Spoke2's is the same apart from the br0 interface which is 192.168.2.0/24.

Some possibly helpful info from the hub:

mikepro · ‎02-04-2019

"Secondly, whatever routing device you ping from must have the destination network in it's routing table or learn a default through a dynamic protocol or statically configured route."

I agree completely.

How come I can't ping Spoke2's GRE IP from Spoke1 though?

Yet, both spokes have got a route to 192.168.1.0/24 and the hub's got routes to both spokes...

And it can't be down to ICMP being disabled at any end as Spoke2's GRE IP is pingable from the hub and the hub's GRE IP is pingable from both spokes...

I thought it could be caused by the "no ip redirects" setting on the GRE tunnel interface (Tunnel0) so I removed it but it didn't help unfortunately :(

grabonlee · ‎02-04-2019

Hi,

What source interface or IP did you put in the ping command from Spoke 1?

Is the issue still that devices behind Spokes can't communicate across the tunnel?

Could you also do the following:

1. Remove network 192.168.1.0 from the Hub BGP. This is a no-no. It's just like redistributing connected interfaces, which would include the DMVPN tunnel interface, and this should never be done.

2. Set the Weights for the routes learned via DMVPN tunnel higher than 32768 (locally injected route weight). For example, on the HUB, neighbor peer-group-name weight 50000. I don't know what type of Spoke router you have and if it would support the command, as it also needs to be done at the Spokes.

mikepro · ‎02-04-2019

Hi,

Thanks for your reply.

What source interface or IP did you put in the ping command from Spoke 1?

I pinged Spoke2 from PC1, which is behind/connected to Spoke1.

Is the issue still that devices behind Spokes can't communicate across the tunnel?

Yes.

I've removed 192.168.0.1/24 from the Hub BGP as advised.

I've increased the peer-group weight to 50000 as suggested but no joy :/

grabonlee · ‎02-04-2019

Maybe it's a typo from you, but I said remove 192.168.1.0 and not 192.168.0.0 from the Hub. Also increasing the weight on the Hub only isn't helpful, if same can't be done on the Spokes.

I don't know what Spoke routers you have, but if they're not Cisco, then I don't think that they would integrate properly with Cisco router, which is your Hub router. Although DMVPN uses NHRP, mGRE, IPSEC (optional) as underlying technologies, Cisco's implementation is proprietary.

The following are what makes Spoke-to-Spoke communication functional and Cisco proprietary:

1. tunnel mode gre multipoint - Required on both Hub and Spokes. If missing on Spoke, then it's effectively a Hub-Spoke only communication (DMVPN Phase 1). That is Point-to-Point.

2. ip nhrp redirect - Required only on Hub

3. ip nhrp shortcut - Required only on Spokes

When a ping is done from a Spoke and the ip routing table is checked immediately, the flag

% appears after the routing protocol value, which indicates an override and confirms Spoke-to-Spoke communication.

mikepro · ‎02-05-2019

Hi,

Thanks for your reply.

It's a typo indeed.

I'm only missing the tunnel mode gre multipoint on the spokes then.

I'll see with our device supplier how to add it.