DMVPN and number of spokes

communication.boy · ‎07-25-2011

I have heard from a juniper guy that DMVPN is not recommended for large networks due to the fact that it works using GRE and it has extra overhead . How many branches have you guys seen in maximum using DMVPN technology . I have worked with an organization which has 50 branches and 1 spoke using DMVPN connectivity . He told me that this cannot go more than 100 branches and I need to use some other technology .

Is this true ?

RYAN BARNES · ‎07-25-2011

I can't speak of maximum numbers, however my last project had 125+ branch site dual-hub configuration connected over DMVPN using 881's at the spokes and 3845's as the hubs. With the AIM encryption modules the encryption is done at the hardware layer, however GRE is done in software and does have an impact on overall throughput. If I remember correctly we weren't running the CPU too hot 30-40%, but don't quote me on that since it's been a few years. Will also depend on the average throughput you think will be traversing each tunnel. If it's point-of-sale type stuff which is low volume, then you can probably support a significantly larger number of tunnels vs. higher throughput clients with traffic hungry apps.

In addition to CPU overhead having to encapsulate traffic, there is also potential fragmentation and reassembly overhead if you aren't careful about your MTU settings and adjustments. This is due to the header sizes of both IPSec and GRE. You definitely want to avoid fragmentation wherever possible when attempting to scale a DMVPN as that will effectively cut your potential throughput in half (likely more than half) if you're doing lots of reassembly.

If you want to scale you will need to switch to hardware GRE platforms like the 7600/Cat6500. There's an excellent white paper here which you might find useful. Keep in mind there are no other options enabled when they do these lab tests (QoS, etc) so these would be best case numbers...

http://www.cisco.com/en/US/docs/solutions/Enterprise/WAN_and_MAN/P2P_GRE_IPSec/4_p2pGRE_Phase2.html

communication.boy · ‎07-25-2011

I actually proposed Dual Hub ASR 1002 to a client for 250 branches nationwide . I think the CPU would be happy to handle the load but I wanted to know if GRE or mGRE as a technology is good for this or not ?

RYAN BARNES · ‎07-25-2011

Seems reasonable to me... although to quote Cisco speak. It depends. :-) If you need to ensure you can support 10Mbps per site during peak hours then you'd be looking at 2.5Gbps of encrypted traffic which might be a stretch for that platform. If however, you're talking 1-2Mbps (PoS transactions, etc) then you'd probably be OK.

I doubt you'll have any issues sustaining that number of tunnels - so it would really depend on prefrag, and expected throughput. The ASR is optimized to handle GRE, however it's still CPU bound from what I understand.

If you're doing Hub-Spoke deployment then you would use mGRE on the hubs and standard GRE on the spokes. (PhaseI) Unless you want spoke-spoke traffic, in which case mGRE on all the interfaces (Phase II).

PhaseIII DMVPN gets rid of the NHS issues and allows even greater scalability when deploying large numbers of spokes.

http://www.cisco.com/en/US/prod/collateral/iosswrel/ps6537/ps6586/ps6660/ps6808/prod_white_paper0900aecd8055c34e_ps6658_Products_White_Paper.html

communication.boy · ‎07-25-2011

Thanks for the reply . The point in this case is that all servers are located in Headoffice and there is no IP Telephony setup which means that there is no need for spoke to spoke communication . In a case where there is no need for spoke to spoke communication should I go for DMVPN Phase 3 ? or stick around Phase 1 only . In phase 1 if by change it requires to connect to any other branch user it will use headoffice link to do so . As far as link goes each branch will have about 512 - 2 MB of link depending on the number of users so I think this would be resonable for ASR to handle .

RYAN BARNES · ‎07-26-2011

Definitely sounds perfectly reasonable to me as well... The ASR should be able to handle that with no issues. As I mentioned, I was running about the same load (100+ sites, 1-2Mbps of traffic each) on a 3845 with an AIM module over 5 years ago, so that should be a breeze for the newer ASR.

Someone else on the site might have some additional input though...

Personally, I would stick with Phase I - running source/destination GRE on the spokes and mGRE on the hubs if there no reason in the future for spoke to spoke, or you want to specifically prevent it. Also, I would consider putting the extra work into using certificates rather than pre-shared keys for your IKE if you're running 100+ nodes. I found it makes key management much easier. (ie. you just revoke a spoke cert on your CA if one of your spoke routers is stolen, etc) rather than messing about trying to update your shared keys everywhere. This also helps with cookie cutter spoke deployments if you're not having to generate a new set of keys for each set of sites.

NIL (as usual) has an excellent well written article(s) here.

http://www.nil.com/ipcorner/IPsecVPN4/

You may also want to consider VTI deployment (sort of like a flashback from the remote access days) using static VTI's on the spokes and dynamic VTI's on the hub. You'll avoid the GRE overhead, and have a bit better control over QoS which may be a bit of a bonus. The advantages/disadvantages are listed at the bottom of this link.

http://www.nil.com/ipcorner/IPsecVPN3/

Personally, I have yet to deploy a VTI based network, so unfortunately I can't give you any personal experience with it.

-Ryan

Jason Gervia · ‎07-26-2011

Actually, if I remember some discussions back at Cisco the limiting factor for DMVPN was more the number of routing protocol peers. It was different for EIGRP vs OSPF.

DMVPN is *not* limited because of the use of GRE.

This is old:

http://www.cisco.com/en/US/prod/collateral/iosswrel/ps6537/ps6586/ps6635/ps6658/prod_presentation0900aecd80313ca3.pdf

One of the Cisco folks here could probably answer it better as they have access to engineering docs. I'll see if I can get someone to look at this thread.

communication.boy · ‎07-26-2011

Excellent Link . I was thinking about SLB earlier when I was going through Yusof book but then i thought what if the Load Balancer fails . What kind of router would you recommend for SLB ? Would a 3845 be fine which I have in stock ?