This article is also available on cisco.com now as well: https://www.cisco.com/c/en/us/support/docs/routers/sd-wan/214149-configure-multiple-transports-and-traffi.html
Introduction
This article is intended to serve as a configuration example on how to configure centralized control policy and app route policy to achieve traffic engineering between sites. It might be considered as a specific design guide for the particular use case as well.
Configuration
For the purpose of demonstration and better understanding of the problem described later, please consider a topology diagram below:
Please note, that in general between vedge1 and vedge3 we should have second link/subinterface for "biz-internet" TLOC extension as well, but here for sake of simplicty it was not configured.
Here are corresponding system settings for vEdges/vSmart (vedge2 represent all other sites):
| hostname | site-id | system-ip |
| vedge1 | 13 | 192.168.30.4 |
| vedge3 | 13 | 192.168.30.6 |
| vedge4 | 4 | 192.168.30.7 |
| vedgex | X | 192.168.30.5 |
| vsmart1 | 1 | 192.168.30.3 |
Below you can find a transport side configurations for reference.
vedge1:
vedge1# show running-config vpn 0 vpn 0 interface ge0/0 description "ISP_1" ip address 192.168.109.4/24 nat respond-to-ping ! tunnel-interface encapsulation ipsec color biz-internet no allow-service bgp allow-service dhcp allow-service dns allow-service icmp allow-service sshd no allow-service netconf no allow-service ntp no allow-service ospf allow-service stun ! no shutdown ! interface ge0/3 description "TLOC-extension via vedge3 to ISP_2" ip address 192.168.80.4/24 tunnel-interface encapsulation ipsec color public-internet no allow-service bgp allow-service dhcp allow-service dns allow-service icmp no allow-service sshd no allow-service netconf no allow-service ntp no allow-service ospf allow-service stun ! no shutdown ! ! ip route 0.0.0.0/0 192.168.80.6 ip route 0.0.0.0/0 192.168.109.10 !
vedge3:
vpn 0 interface ge0/0 description "ISP_2" ip address 192.168.110.6/24 nat respond-to-ping ! tunnel-interface encapsulation ipsec color public-internet carrier carrier3 no allow-service bgp allow-service dhcp allow-service dns allow-service icmp no allow-service sshd no allow-service netconf no allow-service ntp no allow-service ospf no allow-service stun ! no shutdown ! interface ge0/3 ip address 192.168.80.6/24 tloc-extension ge0/0 no shutdown ! ip route 0.0.0.0/0 192.168.110.10
vedge4:
vpn 0 interface ge0/1 ip address 192.168.103.7/24 tunnel-interface encapsulation ipsec color public-internet no allow-service bgp allow-service dhcp allow-service dns allow-service icmp no allow-service sshd no allow-service netconf no allow-service ntp allow-service ospf no allow-service stun ! no shutdown ! ip route 0.0.0.0/0 192.168.103.10 !
Problem
User want to achieve the following goals:
- Internet service provide "ISP 2" should be preferred to communicate between "site 13" and "site 4" because of some reasons. For example, It's quiet common use case and a scenario when connection/connectivity quality within an ISP between it's own clients is very good, but toward the rest of the Internet connectivity quality does not meet company's SLA because of some troubles or congestion on an ISP uplink and and hence this ISP ("ISP 2" in our case) should be avoided in general.
- Site "site 13" should prefer "public-internet" uplink to connect to site "site 4", but still maintain redundancy and should be able to reach "site 4" if "public-internet" fails.
- Site "site 4" should still maintain best-effort connectivity with all other sites directly (hence we can't use restrict keyword here on vedge4 to achieve that goal).
- Site "site 13" should use better quality link with "biz-internet" color to reach all other sites (represented by "site X" on topology diagram).
Another reason might be costs/pricing issues when traffic within ISP is free of charge, but much more expensive when a traffic exiting a provider network (autonomous system).
Some users who are not experienced with SD-WAN approach and get used to "classis" routing may start ro configure static routing to force traffic from vedge1 to vedge4 public interface address via TLOC-extension interface between vedge1 and vedge3, but it won't give desired result and moreover will be confusing for a user because:
- management plane traffic (e.g. ping, traceroute utilities packets) will follow a desired route
- at the same time SD-WAN data plane tunnels (ipsec or gre transport tunnels) will ignore routing table information and will form connections based on TLOCs "colors"
- since a static route has no intelligence, if "public-internet" TLOC is down on vedge3 (uplink to "ISP 2"), then vedge1 won't notice this and connectivity to vedge4 will fail despite the fact that vedge1 still has "biz-internet" available.
Hence this approach should be avoided and not usable.
Solution
1. Use of centralized control policy to set preference for "public-internet" TLOC on vSmart controller when announcing corresponding OMP routes to vedge4. It will help to archive desired traffic path from "site 4" to "site 13".
2. To achieve desired traffic path in reverse direction from "site 13" to "site 4" we can't use centralized control policy because vedge4 has only one TLOC available, hence we can't set preference to anything, but we can use app route policy to achieve this result for egress traffic from "site 13".
Here is how centralized control policy may look like on vSmart controller to prefer "public-internet" TLOC to reach "site 13":
policy
control-policy S4_S13_via_PUB
sequence 10
match tloc
color public-internet
site-id 13
!
action accept
set
preference 333
!
!
!
default-action accept
!
!
And here is an example of app route policy to prefer "public-internet" uplink as an exit point for egress traffic from "site 13" to "site 4" :
policy
app-route-policy S13_S4_via_PUB
vpn-list CORP_VPNs
sequence 10
match
destination-data-prefix-list SITE4_PREFIX
!
action
count COUNT_PKT
sla-class SLA_CL1 preferred-color public-internet
!
!
!
!
policy
lists
site-list S13
site-id 13
!
site-list S40
site-id 4
!
data-prefix-list SITE4_PREFIX
ip-prefix 192.168.60.0/24
!
vpn-list CORP_VPNs
vpn 40
!
!
sla-class SLA_CL1
loss 1
latency 100
jitter 100
!
Policies should be applied appropriately on vSmart controller:
apply-policy site-list S13 app-route-policy S13_S4_via_PUB ! site-list S4 control-policy S4_S13_via_PUB out ! !
Please remember also that app-route policies can't be configured as a localized policy and should be applied on vSmart only.
Verification
Please note app route policy won't be applied to vEdge locally generated traffic, hence to verify if traffic flows steered according to desired path it's recommended to generate some traffic from LAN segments of corresponding sites. As a high-level testing scenario case you can use iperf to generate traffic between hosts in LAN segments of "site 13" and "site 4" and then check an interface statistics. For example, in my case there was no traffic besides system generated and hence we can see that major amount of traffic passed through ge0/3 interface towards TLOC extension on vedge3:
vedge1# show interface statistics
PPPOE PPPOE DOT1X DOT1X
AF RX RX RX TX TX TX RX RX TX TX TX RX TX RX
VPN INTERFACE TYPE PACKETS RX OCTETS ERRORS DROPS PACKETS TX OCTETS ERRORS DROPS PPS Kbps PPS Kbps PKTS PKTS PKTS PKTS
----------------------------------------------------------------------------------------------------------------------------------------------------------
0 ge0/0 ipv4 1832 394791 0 167 1934 894680 0 0 26 49 40 229 - - 0 0
0 ge0/2 ipv4 0 0 0 0 0 0 0 0 0 0 0 0 - - 0 0
0 ge0/3 ipv4 3053034 4131607715 0 27 2486248 3239661783 0 0 51933 563383 41588 432832 - - 0 0
0 ge0/4 ipv4 0 0 0 0 0 0 0 0 0 0 0 0 - - 0 0
Troubleshooting
First of all make sure that corresponding BFD sessions are established (we are not using "restrict" keyword anywhere):
vedge1# show bfd sessions
SOURCE TLOC REMOTE TLOC DST PUBLIC DST PUBLIC DETECT TX
SYSTEM IP SITE ID STATE COLOR COLOR SOURCE IP IP PORT ENCAP MULTIPLIER INTERVAL(msec) UPTIME TRANSITIONS
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
192.168.30.5 2 up public-internet public-internet 192.168.80.4 192.168.109.5 12386 ipsec 7 1000 0:02:10:54 3
192.168.30.5 2 up biz-internet public-internet 192.168.109.4 192.168.109.5 12386 ipsec 7 1000 0:02:10:48 3
192.168.30.7 4 up public-internet public-internet 192.168.80.4 192.168.103.7 12366 ipsec 7 1000 0:02:11:01 2
192.168.30.7 4 up biz-internet public-internet 192.168.109.4 192.168.103.7 12366 ipsec 7 1000 0:02:10:56 2
vedge3# show bfd sessions
SOURCE TLOC REMOTE TLOC DST PUBLIC DST PUBLIC DETECT TX
SYSTEM IP SITE ID STATE COLOR COLOR SOURCE IP IP PORT ENCAP MULTIPLIER INTERVAL(msec) UPTIME TRANSITIONS
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
192.168.30.5 2 up public-internet public-internet 192.168.110.6 192.168.109.5 12386 ipsec 7 1000 0:02:11:05 1
192.168.30.7 4 up public-internet public-internet 192.168.110.6 192.168.103.7 12366 ipsec 7 1000 0:02:11:13 2
vedge4# show bfd sessions
SOURCE TLOC REMOTE TLOC DST PUBLIC DST PUBLIC DETECT TX
SYSTEM IP SITE ID STATE COLOR COLOR SOURCE IP IP PORT ENCAP MULTIPLIER INTERVAL(msec) UPTIME TRANSITIONS
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
192.168.30.4 13 up public-internet biz-internet 192.168.103.7 192.168.109.4 12346 ipsec 7 1000 0:02:09:11 2
192.168.30.4 13 up public-internet public-internet 192.168.103.7 192.168.110.6 63084 ipsec 7 1000 0:02:09:16 2
192.168.30.5 2 up public-internet public-internet 192.168.103.7 192.168.109.5 12386 ipsec 7 1000 0:02:09:10 3
192.168.30.6 13 up public-internet public-internet 192.168.103.7 192.168.110.6 12386 ipsec 7 1000 0:02:09:07 2
If you can't achieve desired result with traffic engineering, then check that policies were applied properly:
1. On vedge4 you should check that for prefixes originated from "site 13" appropriate TLOC was selected:
vedge4# show omp routes 192.168.40.0/24 detail
---------------------------------------------------
omp route entries for vpn 40 route 192.168.40.0/24
---------------------------------------------------
RECEIVED FROM:
peer 192.168.30.3
path-id 72
label 1002
status R
loss-reason tloc-preference
lost-to-peer 192.168.30.3
lost-to-path-id 74
Attributes:
originator 192.168.30.4
type installed
tloc 192.168.30.4, biz-internet, ipsec
ultimate-tloc not set
domain-id not set
overlay-id 1
site-id 13
preference not set
tag not set
origin-proto connected
origin-metric 0
as-path not set
unknown-attr-len not set
RECEIVED FROM:
peer 192.168.30.3
path-id 73
label 1002
status C,I,R
loss-reason not set
lost-to-peer not set
lost-to-path-id not set
Attributes:
originator 192.168.30.4
type installed
tloc 192.168.30.4, public-internet, ipsec
ultimate-tloc not set
domain-id not set
overlay-id 1
site-id 13
preference not set
tag not set
origin-proto connected
origin-metric 0
as-path not set
unknown-attr-len not set
RECEIVED FROM:
peer 192.168.30.3
path-id 74
label 1002
status C,I,R
loss-reason not set
lost-to-peer not set
lost-to-path-id not set
Attributes:
originator 192.168.30.6
type installed
tloc 192.168.30.6, public-internet, ipsec
ultimate-tloc not set
domain-id not set
overlay-id 1
site-id 13
preference not set
tag not set
origin-proto connected
origin-metric 0
as-path not set
unknown-attr-len not set
2. On vedge1 and vedge3 we should make sure that appropriate policy from vSmart was installed and packets are matching and being counted:
vedge1# show policy from-vsmart
from-vsmart sla-class SLA_CL1
loss 1
latency 100
jitter 100
from-vsmart app-route-policy S13_S4_via_PUB
vpn-list CORP_VPNs
sequence 10
match
destination-data-prefix-list SITE4_PREFIX
action
count COUNT_PKT
backup-sla-preferred-color biz-internet
sla-class SLA_CL1
no sla-class strict
sla-class preferred-color public-internet
from-vsmart lists vpn-list CORP_VPNs
vpn 40
from-vsmart lists data-prefix-list SITE4_PREFIX
ip-prefix 192.168.60.0/24
vedge1# show policy app-route-policy-filter
COUNTER
NAME NAME NAME PACKETS BYTES
-------------------------------------------------
S13_S4_via_PUB CORP_VPNs COUNT_PKT 81126791 110610503611
Besides that you should see much more packets sent via "public-internet" color from "site 13" (during my testing there was no traffic via "biz-internet" TLOC):
vedge1# show app-route stats remote-system-ip 192.168.30.7
app-route statistics 192.168.80.4 192.168.103.7 ipsec 12386 12366
remote-system-ip 192.168.30.7
local-color public-internet
remote-color public-internet
mean-loss 0
mean-latency 1
mean-jitter 0
sla-class-index 0,1
TOTAL AVERAGE AVERAGE TX DATA RX DATA
INDEX PACKETS LOSS LATENCY JITTER PKTS PKTS
----------------------------------------------------------
0 600 0 0 0 0 0
1 600 0 1 0 5061061 6731986
2 600 0 0 0 3187291 3619658
3 600 0 0 0 0 0
4 600 0 2 0 9230960 12707216
5 600 0 1 0 9950840 4541723
app-route statistics 192.168.109.4 192.168.103.7 ipsec 12346 12366
remote-system-ip 192.168.30.7
local-color biz-internet
remote-color public-internet
mean-loss 0
mean-latency 0
mean-jitter 0
sla-class-index 0,1
TOTAL AVERAGE AVERAGE TX DATA RX DATA
INDEX PACKETS LOSS LATENCY JITTER PKTS PKTS
----------------------------------------------------------
0 600 0 0 0 0 0
1 600 0 1 0 0 0
2 600 0 0 0 0 0
3 600 0 0 0 0 0
4 600 0 2 0 0 0
5 600 0 0 0 0 0
References
https://sdwan-docs.cisco.com/Product_Documentation/Command_Reference/Configuration_Commands/color