cancel
Showing results for 
Search instead for 
Did you mean: 
cancel

SD-WAN Configuration Example: Multiple transports and traffic engineering with centralized control policy and app route policy

741
Views
0
Helpful
0
Comments

This article is also available on cisco.com now as well: https://www.cisco.com/c/en/us/support/docs/routers/sd-wan/214149-configure-multiple-transports-and-traffi.html

 

Introduction

 

This article is intended to serve as a configuration example on how to configure centralized control policy and app route policy to achieve traffic engineering between sites. It might be considered as a specific design guide for the particular use case as well.

Configuration

 

For the purpose of demonstration and better understanding of the problem described later, please consider a topology diagram below:

Screen Shot 2018-11-27 at 19.59.04.png

 

 

Please note, that in general between vedge1 and vedge3 we should have second link/subinterface for "biz-internet" TLOC extension as well, but here for sake of simplicty it was not configured.

 

Here are corresponding system settings for vEdges/vSmart (vedge2 represent all other sites):

 

hostname site-id system-ip
vedge1 13 192.168.30.4
vedge3 13 192.168.30.6
vedge4 4 192.168.30.7
vedgex X 192.168.30.5
vsmart1 192.168.30.3

 

Below you can find a transport side configurations for reference.

 

vedge1:

vedge1# show running-config vpn 0
vpn 0
 interface ge0/0
  description "ISP_1"
  ip address 192.168.109.4/24
  nat
   respond-to-ping
  !
  tunnel-interface
   encapsulation ipsec
   color biz-internet
   no allow-service bgp
   allow-service dhcp
   allow-service dns
   allow-service icmp
   allow-service sshd
   no allow-service netconf
   no allow-service ntp
   no allow-service ospf
   allow-service stun
  !
  no shutdown
 !
 interface ge0/3
  description "TLOC-extension via vedge3 to ISP_2"
  ip address 192.168.80.4/24
  tunnel-interface
   encapsulation ipsec
   color public-internet
   no allow-service bgp
   allow-service dhcp
   allow-service dns
   allow-service icmp
   no allow-service sshd
   no allow-service netconf
   no allow-service ntp
   no allow-service ospf
   allow-service stun
  !
  no shutdown
 !
 !
 ip route 0.0.0.0/0 192.168.80.6
 ip route 0.0.0.0/0 192.168.109.10
!

vedge3:

vpn 0
 interface ge0/0
  description "ISP_2"
  ip address 192.168.110.6/24
  nat
   respond-to-ping
  !
  tunnel-interface
   encapsulation ipsec
   color public-internet
   carrier carrier3
   no allow-service bgp
   allow-service dhcp
   allow-service dns
   allow-service icmp
   no allow-service sshd
   no allow-service netconf
   no allow-service ntp
   no allow-service ospf
   no allow-service stun
  !
  no shutdown
 !
 interface ge0/3
  ip address 192.168.80.6/24
  tloc-extension ge0/0
  no shutdown
 !
 ip route 0.0.0.0/0 192.168.110.10

vedge4:

vpn 0
 interface ge0/1
  ip address 192.168.103.7/24
  tunnel-interface
   encapsulation ipsec
   color public-internet
   no allow-service bgp
   allow-service dhcp
   allow-service dns
   allow-service icmp
   no allow-service sshd
   no allow-service netconf
   no allow-service ntp
   allow-service ospf
   no allow-service stun
  !
  no shutdown
 !
 ip route 0.0.0.0/0 192.168.103.10
!

Problem

User want to achieve the following goals: 

- Internet service provide "ISP 2" should be preferred to communicate between "site 13" and "site 4" because of some reasons. For example, It's quiet common use case and a scenario when connection/connectivity quality within an ISP between it's own clients is very good, but toward the rest of the Internet connectivity quality does not meet company's SLA because of some troubles or congestion on an ISP uplink and and hence this ISP ("ISP 2" in our case) should be avoided in general.  

- Site "site 13" should prefer "public-internet" uplink to connect to site "site 4", but still maintain redundancy and should be able to reach "site 4" if "public-internet" fails.

- Site "site 4" should still maintain best-effort connectivity with all other sites directly (hence we can't use restrict keyword here on vedge4 to achieve that goal).

- Site "site 13" should use better quality link with "biz-internet" color to reach all other sites (represented by "site X" on topology diagram).

 

Another reason might be costs/pricing issues when traffic within ISP is free of charge, but much more expensive when a traffic exiting a provider network (autonomous system). 

 

Some users who are not experienced with SD-WAN approach and get used to "classis" routing may start ro configure static routing to force traffic from vedge1 to vedge4 public interface address via TLOC-extension interface between vedge1 and vedge3, but it won't give desired result and moreover will be confusing for a user because:

 

- management plane traffic (e.g. ping, traceroute utilities packets) will follow a desired route 

- at the same time SD-WAN data plane tunnels (ipsec or gre transport tunnels) will ignore routing table information and will form connections based on TLOCs "colors"

- since a static route has no intelligence, if "public-internet" TLOC is down on vedge3 (uplink to "ISP 2"), then vedge1 won't notice this and connectivity to vedge4 will fail despite the fact that vedge1 still has "biz-internet" available.

 

Hence this approach should be avoided and not usable. 

Solution

1. Use of centralized control policy to set preference for "public-internet" TLOC on vSmart controller when announcing corresponding OMP routes to vedge4. It will help to archive desired traffic path from "site 4" to "site 13".

2. To achieve desired traffic path in reverse direction from "site 13" to "site 4" we can't use centralized control policy because vedge4 has only one TLOC available, hence we can't set preference to anything, but we can use app route policy to achieve this result for egress traffic from "site 13".

 

Here is how centralized control policy may look like on vSmart controller to prefer "public-internet" TLOC to reach "site 13":

policy
 control-policy S4_S13_via_PUB
  sequence 10
   match tloc
    color   public-internet
    site-id 13
   !
   action accept
    set
     preference 333
    !
   !
  !
  default-action accept
 !
!
 

And here is an example of app route policy to prefer "public-internet" uplink as an exit point for egress traffic from "site 13" to "site 4" :

policy
 app-route-policy S13_S4_via_PUB
  vpn-list CORP_VPNs
   sequence 10
    match
     destination-data-prefix-list SITE4_PREFIX
    !
    action
     count                      COUNT_PKT
     sla-class SLA_CL1 preferred-color public-internet
    !
   !
  !
 !
policy
 lists
  site-list S13
   site-id 13
  !
  site-list S40
   site-id 4
  !
  data-prefix-list SITE4_PREFIX
   ip-prefix 192.168.60.0/24
  !
  vpn-list CORP_VPNs
   vpn 40
  !
 !
 sla-class SLA_CL1
  loss    1
  latency 100
  jitter  100
 !

Policies should be applied appropriately on vSmart controller:

apply-policy
 site-list S13
  app-route-policy S13_S4_via_PUB
 !
 site-list S4
  control-policy S4_S13_via_PUB out
 !
!

Please remember also that app-route policies can't be configured as a localized policy and should be applied on vSmart only. 

Verification 

Please note app route policy won't be applied to vEdge locally generated traffic, hence to verify if traffic flows steered according to desired path it's recommended to generate some traffic from LAN segments of corresponding sites. As a high-level testing scenario case you can use iperf to generate traffic between hosts in LAN segments of "site 13" and "site 4" and then check an interface statistics. For example, in my case there was no traffic besides system generated and hence we can see that major amount of traffic passed through ge0/3 interface towards TLOC extension on vedge3:

 

vedge1# show interface statistics

                                                                                                                              PPPOE  PPPOE  DOT1X  DOT1X  
                  AF    RX                   RX      RX     TX                   TX      TX     RX     RX      TX     TX      TX     RX     TX     RX     
VPN  INTERFACE    TYPE  PACKETS  RX OCTETS   ERRORS  DROPS  PACKETS  TX OCTETS   ERRORS  DROPS  PPS    Kbps    PPS    Kbps    PKTS   PKTS   PKTS   PKTS   
----------------------------------------------------------------------------------------------------------------------------------------------------------
0    ge0/0        ipv4  1832     394791      0       167    1934     894680      0       0      26     49      40     229     -      -      0      0      
0    ge0/2        ipv4  0        0           0       0      0        0           0       0      0      0       0      0       -      -      0      0      
0    ge0/3        ipv4  3053034  4131607715  0       27     2486248  3239661783  0       0      51933  563383  41588  432832  -      -      0      0      
0    ge0/4        ipv4  0        0           0       0      0        0           0       0      0      0       0      0       -      -      0      0      

Troubleshooting

First of all make sure that corresponding BFD sessions are established (we are not using "restrict" keyword anywhere):

 

vedge1# show bfd sessions 
                                      SOURCE TLOC      REMOTE TLOC                                      DST PUBLIC                      DST PUBLIC         DETECT      TX                              
SYSTEM IP        SITE ID  STATE       COLOR            COLOR            SOURCE IP                       IP                              PORT        ENCAP  MULTIPLIER  INTERVAL(msec) UPTIME          TRANSITIONS 
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
192.168.30.5     2        up          public-internet  public-internet  192.168.80.4                    192.168.109.5                   12386       ipsec  7           1000           0:02:10:54      3           
192.168.30.5     2        up          biz-internet     public-internet  192.168.109.4                   192.168.109.5                   12386       ipsec  7           1000           0:02:10:48      3           
192.168.30.7     4        up          public-internet  public-internet  192.168.80.4                    192.168.103.7                   12366       ipsec  7           1000           0:02:11:01      2           
192.168.30.7     4        up          biz-internet     public-internet  192.168.109.4                   192.168.103.7                   12366       ipsec  7           1000           0:02:10:56      2           
vedge3# show bfd sessions 
                                      SOURCE TLOC      REMOTE TLOC                                      DST PUBLIC                      DST PUBLIC         DETECT      TX                              
SYSTEM IP        SITE ID  STATE       COLOR            COLOR            SOURCE IP                       IP                              PORT        ENCAP  MULTIPLIER  INTERVAL(msec) UPTIME          TRANSITIONS 
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
192.168.30.5     2        up          public-internet  public-internet  192.168.110.6                   192.168.109.5                   12386       ipsec  7           1000           0:02:11:05      1           
192.168.30.7     4        up          public-internet  public-internet  192.168.110.6                   192.168.103.7                   12366       ipsec  7           1000           0:02:11:13      2           
vedge4# show bfd sessions 
                                      SOURCE TLOC      REMOTE TLOC                                      DST PUBLIC                      DST PUBLIC         DETECT      TX                              
SYSTEM IP        SITE ID  STATE       COLOR            COLOR            SOURCE IP                       IP                              PORT        ENCAP  MULTIPLIER  INTERVAL(msec) UPTIME          TRANSITIONS 
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
192.168.30.4     13       up          public-internet  biz-internet     192.168.103.7                   192.168.109.4                   12346       ipsec  7           1000           0:02:09:11      2           
192.168.30.4     13       up          public-internet  public-internet  192.168.103.7                   192.168.110.6                   63084       ipsec  7           1000           0:02:09:16      2           
192.168.30.5     2        up          public-internet  public-internet  192.168.103.7                   192.168.109.5                   12386       ipsec  7           1000           0:02:09:10      3           
192.168.30.6     13       up          public-internet  public-internet  192.168.103.7                   192.168.110.6                   12386       ipsec  7           1000           0:02:09:07      2           

 If you can't achieve desired result with traffic engineering, then check that policies were applied properly:

 

1. On vedge4 you should check that for prefixes originated from "site 13" appropriate TLOC was selected:

vedge4# show omp routes 192.168.40.0/24 detail 

---------------------------------------------------
omp route entries for vpn 40 route 192.168.40.0/24
---------------------------------------------------
            RECEIVED FROM:                   
peer            192.168.30.3
path-id         72
label           1002
status          R
loss-reason     tloc-preference
lost-to-peer    192.168.30.3
lost-to-path-id 74
    Attributes:
     originator       192.168.30.4
     type             installed
     tloc             192.168.30.4, biz-internet, ipsec
     ultimate-tloc    not set
     domain-id        not set
     overlay-id        1
     site-id          13
     preference       not set
     tag              not set
     origin-proto     connected
     origin-metric    0
     as-path          not set
     unknown-attr-len not set
            RECEIVED FROM:                   
peer            192.168.30.3
path-id         73
label           1002
status          C,I,R
loss-reason     not set
lost-to-peer    not set
lost-to-path-id not set
    Attributes:
     originator       192.168.30.4
     type             installed
     tloc             192.168.30.4, public-internet, ipsec
     ultimate-tloc    not set
     domain-id        not set
     overlay-id        1
     site-id          13
     preference       not set
     tag              not set
     origin-proto     connected
     origin-metric    0
     as-path          not set
     unknown-attr-len not set
            RECEIVED FROM:                   
peer            192.168.30.3
path-id         74
label           1002
status          C,I,R
loss-reason     not set
lost-to-peer    not set
lost-to-path-id not set
    Attributes:
     originator       192.168.30.6
     type             installed
     tloc             192.168.30.6, public-internet, ipsec
     ultimate-tloc    not set
     domain-id        not set
     overlay-id        1
     site-id          13
     preference       not set
     tag              not set
     origin-proto     connected
     origin-metric    0
     as-path          not set
     unknown-attr-len not set

2. On vedge1 and vedge3 we should make sure that appropriate policy from vSmart was installed and packets are matching and being counted:

vedge1# show policy from-vsmart 
from-vsmart sla-class SLA_CL1
 loss    1
 latency 100
 jitter  100
from-vsmart app-route-policy S13_S4_via_PUB
 vpn-list CORP_VPNs
  sequence 10
   match
    destination-data-prefix-list SITE4_PREFIX
   action
    count                      COUNT_PKT
    backup-sla-preferred-color biz-internet
    sla-class       SLA_CL1
    no sla-class strict
    sla-class preferred-color public-internet
from-vsmart lists vpn-list CORP_VPNs
 vpn 40
from-vsmart lists data-prefix-list SITE4_PREFIX
 ip-prefix 192.168.60.0/24

vedge1# show policy app-route-policy-filter 

                COUNTER                          
NAME      NAME  NAME     PACKETS   BYTES         
-------------------------------------------------
S13_S4_via_PUB CORP_VPNs  COUNT_PKT     81126791  110610503611  

 Besides that you should see much more packets sent via "public-internet" color from "site 13" (during my testing there was no traffic via "biz-internet" TLOC):

vedge1# show app-route stats remote-system-ip 192.168.30.7 
app-route statistics 192.168.80.4 192.168.103.7 ipsec 12386 12366
 remote-system-ip 192.168.30.7
 local-color      public-internet
 remote-color     public-internet
 mean-loss        0
 mean-latency     1
 mean-jitter      0
 sla-class-index  0,1

       TOTAL          AVERAGE  AVERAGE  TX DATA  RX DATA  
INDEX  PACKETS  LOSS  LATENCY  JITTER   PKTS     PKTS     
----------------------------------------------------------
0      600      0     0        0        0        0        
1      600      0     1        0        5061061  6731986  
2      600      0     0        0        3187291  3619658  
3      600      0     0        0        0        0        
4      600      0     2        0        9230960  12707216 
5      600      0     1        0        9950840  4541723  

app-route statistics 192.168.109.4 192.168.103.7 ipsec 12346 12366
 remote-system-ip 192.168.30.7
 local-color      biz-internet
 remote-color     public-internet
 mean-loss        0
 mean-latency     0
 mean-jitter      0
 sla-class-index  0,1

       TOTAL          AVERAGE  AVERAGE  TX DATA  RX DATA  
INDEX  PACKETS  LOSS  LATENCY  JITTER   PKTS     PKTS     
----------------------------------------------------------
0      600      0     0        0        0        0        
1      600      0     1        0        0        0        
2      600      0     0        0        0        0        
3      600      0     0        0        0        0        
4      600      0     2        0        0        0        
5      600      0     0        0        0        0        

 

References

https://sdwan-docs.cisco.com/Product_Documentation/Software_Features/Release_18.3/07Policy_Applications/01Application-Aware_Routing/01Configuring_Application-Aware_Routing

https://sdwan-docs.cisco.com/Product_Documentation/Software_Features/Release_18.3/02System_and_Interfaces/06Configuring_Network_Interfaces

https://sdwan-docs.cisco.com/Product_Documentation/Command_Reference/Configuration_Commands/color

 

 

CreatePlease to create content
Content for Community-Ad
July's Community Spotlight Awards