Hybrid IWAN Question

[Chad Parish]

I am currently working to design a hybrid IWAN network between three sites.  We currently have at each site 2 edge routers for MPLS (peering to a primary ISP and backup ISP) and two edge routers for INET (peering to a primary ISP and backup ISP).  We want to consolidate using two new ISR 4331 routers at each site, wherein we would have in each a MPLS link and an INET link out to our ISPs.  The idea is that if we lost one edge router we would still have both MPLS and INET provided through the remaining edge router.

We will use DMVPN as a transport overlay and separate the routing tables in each router using VRFs for both EIGRP and eBGP.

I have attached a diagram showing one of the edge routers at each site (the other edge ISR would be exactly the same but to a different INET and MPLS ISP for redundancy).

I can build the VRFs (at least the EIGRP VRFs), and the tunnels for DMVPN.  But I am stumped at the eBGP portion as BGP allows only a single ASN per router.  But we use BGP 64444 with our INET links and BGP 36582 to peer over to our MPLS ISPs.

Is there a way to allow for a single edge router to peer to two ISPs using a different BGP ASN for MPLS and INET?

I have attached what I have so far configured for our EV site.  If I can figure this out, then the config can be used as a template for the other sites, barring a few changes for spoke tunnels versus the EV hub tunnel configs).

!
vrf definition MPLS
!
address-family ipv4
exit-address-family
!
vrf definition INET
!
address-family ipv4
exit-address-family
!
!
ip vrf INET
rd 36582:20
!
ip vrf MPLS
rd 64444:10
!
!
!
crypto ikev2 keyring DMVPN-KEYRING-MPLS
peer ANY
address 0.0.0.0 0.0.0.0
pre-shared-key c1sco123
!
crypto ikev2 profile DMVPN-IKE-PROFILE-MPLS
match fvrf MPLS
match identity remote address 0.0.0.0
authentication remote pre-share
authentication local pre-share
keyring local DMVPN-KEYRING-MPLS
!
crypto ipsec security-association replay window-size 512
!
!
crypto ipsec transform-set AES256/SHA/TRANSPORT esp-aes 256 esp-sha-hmac
mode transport
!
crypto ipsec profile DMVPN-IPSEC-PROFILE-MPLS
set transform-set AES256/SHA/TRANSPORT
set ikev2-profile DMVPN-IKE-PROFILE-MPLS
!
crypto ikev2 keyring DMVPN-KEYRING-INET
peer ANY
address 0.0.0.0 0.0.0.0
pre-shared-key CISCO123
!
!
crypto ikev2 profile DMVPN-IKE-PROFILE-INET
match fvrf INET
match identity remote address 0.0.0.0
authentication remote pre-share
authentication local pre-share
keyring local DMVPN-KEYRING-INET
!
crypto ipsec security-association replay window-size 512
!
crypto ipsec transform-set AES256/SHA/TRANSPORT esp-aes 256 esp-sha-hmac
mode transport
!
crypto ipsec profile DMVPN-IPSEC-PROFILE-INET
set transform-set AES256/SHA/TRANSPORT
set ikev2-profile DMVPN-IKE-PROFILE-INET
!
!
!
interface GigabitEthernet0/0/0
description MPLS-TRANSPORT
ip vrf forwarding MPLS
ip address 172.21.8.46 255.255.255.252
!
ip route vrf MPLS 0.0.0.0 0.0.0.0 172.21.8.45
!
!
interface GigabitEthernet0/1/0
description INET-TRANSPORT
ip vrf forwarding INET
ip address 209.160.200.78 255.255.255.252
!
ip route vrf INET 0.0.0.0 0.0.0.0 209.160.200.77
!
interface Tunnel10
description MPLS DMVPN-TUNNEL
bandwidth 1000
ip vrf forwarding MPLS
ip address 192.168.10.1 255.255.255.0
no ip redirects
ip hold-time eigrp 1 35
ip mtu 1400
no ip next-hop-self eigrp 150
no ip split-horizon eigrp 150
ip nhrp authentication cisco123
ip nhrp map multicast dynamic
ip nhrp network-id 10
ip nhrp holdtime 300
ip nhrp nhs 192.168.10.1
ip tcp adjust-mss 1360
delay 1000
tunnel source GigabitEthernet0/0/0
tunnel mode gre multipoint
tunnel key 12345
tunnel vrf MPLS
tunnel protection ipsec profile DMVPN
!
interface Tunnel20
description INET DMVPN-TUNNEL
bandwidth 1000
ip vrf forwarding INET
ip address 192.168.20.1 255.255.255.0
no ip redirects
ip mtu 1400
no ip next-hop-self eigrp 1
no ip split-horizon eigrp 1
ip nhrp authentication cisco123
ip nhrp map multicast dynamic
ip nhrp network-id 20
ip nhrp holdtime 300
ip nhrp nhs 192.168.20.1
ip tcp adjust-mss 1360
delay 1000
tunnel source GigabitEthernet0/1/0
tunnel mode gre multipoint
tunnel key 12345
tunnel vrf INET
tunnel protection ipsec profile DMVPN
!
router eigrp 150
network 10.1.255.150 0.0.0.0
network 10.150.1.8 0.0.0.7
network 10.150.1.16 0.0.0.7
network 172.21.8.44 0.0.0.3
network 192.168.10.0
network 192.168.20.0
network 209.160.200.76 0.0.0.3
no auto-summary
!
address-family ipv4 vrf MPLS
redistribute bgp 64444 metric 44000 43 255 1 1500 route-map bgp-eigrp
network 10.1.255.150 0.0.0.0
network 10.150.1.8 0.0.0.7
network 10.150.1.16 0.0.0.7
network 172.21.8.44 0.0.0.3
network 192.168.10.0
maximum-paths 1
no auto-summary
autonomous-system 150
exit-address-family
!
address-family ipv4 vrf INET
network 192.168.20.0
network 209.160.200.76 0.0.0.3
autonomous-system 150
no auto-summary
exit-address-family
!
router bgp 64444
bgp log-neighbor-changes
timers bgp 10 60
neighbor 74.113.151.4 remote-as 36582
neighbor 209.160.200.77 remote-as 46887
neighbor 172.21.8.45 remote-as 1803
neighbor 172.21.8.45 next-hop-self
neighbor 172.21.8.45 send-community both
neighbor 172.21.8.45 route-map bgpcomm-spr out
!
address-family ipv4 vrf MPLS autonomous-system 64444
no synchronization
bgp log-neighbor-changes
network 10.1.255.150 mask 255.255.255.255
network 10.150.1.8 mask 255.255.255.248
network 172.21.8.44 mask 255.255.255.252
redistribute eigrp 150 route-map eigrp-bgp
neighbor 172.21.8.45 remote-as 1803
neighbor 172.21.8.45 next-hop-self
neighbor 172.21.8.45 send-community both
neighbor 172.21.8.45 route-map bgpcomm-spr out
no auto-summary
exit-address-family
!
address-family ipv4 vrf INET autonomous-system 36582
redistribute static
timers bgp 10 60
neighbor 74.113.151.4 remote-as 36582
neighbor 74.113.151.4 local-as 36582 no-prepend replace-as
neighbor 74.113.151.4 activate
neighbor 74.113.151.4 next-hop-self
neighbor 209.160.200.77 remote-as 46887
neighbor 209.160.200.77 local-as 36582 no-prepend replace-as
neighbor 209.160.200.77 activate
neighbor 209.160.200.77 next-hop-self
maximum-paths 6
maximum-paths ibgp 6
no auto-summary
no synchronization
bgp dmzlink-bw
network 74.113.151.0 mask 255.255.255.0
network 209.160.200.76 mask 255.255.255.252
exit-address-family

[Paul Chapman · Accepted solution]

Hi Chad -

First, I suggest going through the current IWAN CVD (Oct 2016) and sample configurations in detail.

From the perspective of a standardized deployment, you must have 1 router per DMVPN cloud at the hub site.  This is mandatory based on all the documents I've seen.  Your remote sites (spokes) can have a single router.

You can overcome the BGP instance limitation by using the "local-as" command in the address-family sections for each VRF in your BGP configuration.

Last, since you're using IKEv2, then I strongly recommend that you use newer encryption, hashing, and Diffie-Hellman functions. (AES-256 / SHA-512 / DH14) (export rules permitting)

PSC

[Chad Parish]

Thanks Paul, so at the hub site, I cannot have a single router handling the DMVPN of the INET and the WAN?  Or in the case of a hybrid IWAN setup, would it be a single DMVPN for both INET and WAN?

[Paul Chapman]

Hi Chad -

Unless Cisco has modified the PfR code significantly, the answer is no.  I worked an IWAN project when the product was first introduced (2015) and we tried to host dual DMVPNs on the hub routers and had problems.  TAC told us to tear down the secondary DMVPNs and go with a single per hub.

Oh... I forgot to mention that you can only run the PfR Master Hub role on a ISR4451, ASR1001-X, or CSR1000V.  (see the CVD for details)  In a small deployment like yours, you would typically host this role on one of your Hub routers.  If you don't have a 4451, and you don't have budget to get one, then you'll need to go with the CSR1000V (I recommend the 10Mb license as it should be enough for a small deployment).

If you have time to experiment, go ahead and try.  I'd like to know if you can get it working.

PSC

[Chad Parish]

Good to know Paul.  We do have a pair of ISR 4351's but I gather those wont work as the hub for PfR.  It sounds like I'll have to use one ISR 4351 as a hub for my INET DMVPN and the other to handle our two MPLS providers, which we use the same BGP AS #.

[Chad Parish]

so I went back to the drawing board and will now be just doing DMVPN for my INET with my two INET providers being handled from one of my ISR 4351's and the two MPLS providers out of the other edge router.  My question is, when you have two ISP providers for INET, such as Sprint and Level3, do you need to set up VRFs?  I would assume that with INET traffic I could simply set up BGP without VRFs and configure the BGP as normal with teo neighbors.

[Paul Chapman]

Hi Chad -

The answer to whether you need VRFs or not depends on how you want routing to work.

1. Do you want the default route (0.0.0.0/0) sent directly inbound? (instead of going to a firewall)
2. What is the impact on your network if multiple default routes are advertised from different points?
3. Do you want internet services to be backhauled across your DMVPN to a central site for policy enforcement?
4. How are you protecting your internal routes from the internet?

You don't have to answer these questions on the forum.  They are intended to give you areas to think about while you're doing your design.

PSC