02-22-2019 11:34 AM - edited 03-05-2019 08:09 AM
This article is also available on cisco.com now as well: https://www.cisco.com/c/en/us/support/docs/routers/sd-wan/214149-configure-multiple-transports-and-traffi.html
This article is intended to serve as a configuration example on how to configure centralized control policy and app route policy to achieve traffic engineering between sites. It might be considered as a specific design guide for the particular use case as well.
For the purpose of demonstration and better understanding of the problem described later, please consider a topology diagram below:
Please note, that in general between vedge1 and vedge3 we should have second link/subinterface for "biz-internet" TLOC extension as well, but here for sake of simplicty it was not configured.
Here are corresponding system settings for vEdges/vSmart (vedge2 represent all other sites):
hostname | site-id | system-ip |
vedge1 | 13 | 192.168.30.4 |
vedge3 | 13 | 192.168.30.6 |
vedge4 | 4 | 192.168.30.7 |
vedgex | X | 192.168.30.5 |
vsmart1 | 1 | 192.168.30.3 |
Below you can find a transport side configurations for reference.
vedge1:
vedge1# show running-config vpn 0 vpn 0 interface ge0/0 description "ISP_1" ip address 192.168.109.4/24 nat respond-to-ping ! tunnel-interface encapsulation ipsec color biz-internet no allow-service bgp allow-service dhcp allow-service dns allow-service icmp allow-service sshd no allow-service netconf no allow-service ntp no allow-service ospf allow-service stun ! no shutdown ! interface ge0/3 description "TLOC-extension via vedge3 to ISP_2" ip address 192.168.80.4/24 tunnel-interface encapsulation ipsec color public-internet no allow-service bgp allow-service dhcp allow-service dns allow-service icmp no allow-service sshd no allow-service netconf no allow-service ntp no allow-service ospf allow-service stun ! no shutdown ! ! ip route 0.0.0.0/0 192.168.80.6 ip route 0.0.0.0/0 192.168.109.10 !
vedge3:
vpn 0 interface ge0/0 description "ISP_2" ip address 192.168.110.6/24 nat respond-to-ping ! tunnel-interface encapsulation ipsec color public-internet carrier carrier3 no allow-service bgp allow-service dhcp allow-service dns allow-service icmp no allow-service sshd no allow-service netconf no allow-service ntp no allow-service ospf no allow-service stun ! no shutdown ! interface ge0/3 ip address 192.168.80.6/24 tloc-extension ge0/0 no shutdown ! ip route 0.0.0.0/0 192.168.110.10
vedge4:
vpn 0 interface ge0/1 ip address 192.168.103.7/24 tunnel-interface encapsulation ipsec color public-internet no allow-service bgp allow-service dhcp allow-service dns allow-service icmp no allow-service sshd no allow-service netconf no allow-service ntp allow-service ospf no allow-service stun ! no shutdown ! ip route 0.0.0.0/0 192.168.103.10 !
User want to achieve the following goals:
- Internet service provide "ISP 2" should be preferred to communicate between "site 13" and "site 4" because of some reasons. For example, It's quiet common use case and a scenario when connection/connectivity quality within an ISP between it's own clients is very good, but toward the rest of the Internet connectivity quality does not meet company's SLA because of some troubles or congestion on an ISP uplink and and hence this ISP ("ISP 2" in our case) should be avoided in general.
- Site "site 13" should prefer "public-internet" uplink to connect to site "site 4", but still maintain redundancy and should be able to reach "site 4" if "public-internet" fails.
- Site "site 4" should still maintain best-effort connectivity with all other sites directly (hence we can't use restrict keyword here on vedge4 to achieve that goal).
- Site "site 13" should use better quality link with "biz-internet" color to reach all other sites (represented by "site X" on topology diagram).
Another reason might be costs/pricing issues when traffic within ISP is free of charge, but much more expensive when a traffic exiting a provider network (autonomous system).
Some users who are not experienced with SD-WAN approach and get used to "classis" routing may start ro configure static routing to force traffic from vedge1 to vedge4 public interface address via TLOC-extension interface between vedge1 and vedge3, but it won't give desired result and moreover will be confusing for a user because:
- management plane traffic (e.g. ping, traceroute utilities packets) will follow a desired route
- at the same time SD-WAN data plane tunnels (ipsec or gre transport tunnels) will ignore routing table information and will form connections based on TLOCs "colors"
- since a static route has no intelligence, if "public-internet" TLOC is down on vedge3 (uplink to "ISP 2"), then vedge1 won't notice this and connectivity to vedge4 will fail despite the fact that vedge1 still has "biz-internet" available.
Hence this approach should be avoided and not usable.
1. Use of centralized control policy to set preference for "public-internet" TLOC on vSmart controller when announcing corresponding OMP routes to vedge4. It will help to archive desired traffic path from "site 4" to "site 13".
2. To achieve desired traffic path in reverse direction from "site 13" to "site 4" we can't use centralized control policy because vedge4 has only one TLOC available, hence we can't set preference to anything, but we can use app route policy to achieve this result for egress traffic from "site 13".
Here is how centralized control policy may look like on vSmart controller to prefer "public-internet" TLOC to reach "site 13":
policy control-policy S4_S13_via_PUB sequence 10 match tloc color public-internet site-id 13 ! action accept set preference 333 ! ! ! default-action accept ! !
And here is an example of app route policy to prefer "public-internet" uplink as an exit point for egress traffic from "site 13" to "site 4" :
policy app-route-policy S13_S4_via_PUB vpn-list CORP_VPNs sequence 10 match destination-data-prefix-list SITE4_PREFIX ! action count COUNT_PKT sla-class SLA_CL1 preferred-color public-internet ! ! ! ! policy lists site-list S13 site-id 13 ! site-list S40 site-id 4 ! data-prefix-list SITE4_PREFIX ip-prefix 192.168.60.0/24 ! vpn-list CORP_VPNs vpn 40 ! ! sla-class SLA_CL1 loss 1 latency 100 jitter 100 !
Policies should be applied appropriately on vSmart controller:
apply-policy site-list S13 app-route-policy S13_S4_via_PUB ! site-list S4 control-policy S4_S13_via_PUB out ! !
Please remember also that app-route policies can't be configured as a localized policy and should be applied on vSmart only.
Please note app route policy won't be applied to vEdge locally generated traffic, hence to verify if traffic flows steered according to desired path it's recommended to generate some traffic from LAN segments of corresponding sites. As a high-level testing scenario case you can use iperf to generate traffic between hosts in LAN segments of "site 13" and "site 4" and then check an interface statistics. For example, in my case there was no traffic besides system generated and hence we can see that major amount of traffic passed through ge0/3 interface towards TLOC extension on vedge3:
vedge1# show interface statistics PPPOE PPPOE DOT1X DOT1X AF RX RX RX TX TX TX RX RX TX TX TX RX TX RX VPN INTERFACE TYPE PACKETS RX OCTETS ERRORS DROPS PACKETS TX OCTETS ERRORS DROPS PPS Kbps PPS Kbps PKTS PKTS PKTS PKTS ---------------------------------------------------------------------------------------------------------------------------------------------------------- 0 ge0/0 ipv4 1832 394791 0 167 1934 894680 0 0 26 49 40 229 - - 0 0 0 ge0/2 ipv4 0 0 0 0 0 0 0 0 0 0 0 0 - - 0 0 0 ge0/3 ipv4 3053034 4131607715 0 27 2486248 3239661783 0 0 51933 563383 41588 432832 - - 0 0 0 ge0/4 ipv4 0 0 0 0 0 0 0 0 0 0 0 0 - - 0 0
First of all make sure that corresponding BFD sessions are established (we are not using "restrict" keyword anywhere):
vedge1# show bfd sessions SOURCE TLOC REMOTE TLOC DST PUBLIC DST PUBLIC DETECT TX SYSTEM IP SITE ID STATE COLOR COLOR SOURCE IP IP PORT ENCAP MULTIPLIER INTERVAL(msec) UPTIME TRANSITIONS ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 192.168.30.5 2 up public-internet public-internet 192.168.80.4 192.168.109.5 12386 ipsec 7 1000 0:02:10:54 3 192.168.30.5 2 up biz-internet public-internet 192.168.109.4 192.168.109.5 12386 ipsec 7 1000 0:02:10:48 3 192.168.30.7 4 up public-internet public-internet 192.168.80.4 192.168.103.7 12366 ipsec 7 1000 0:02:11:01 2 192.168.30.7 4 up biz-internet public-internet 192.168.109.4 192.168.103.7 12366 ipsec 7 1000 0:02:10:56 2
vedge3# show bfd sessions SOURCE TLOC REMOTE TLOC DST PUBLIC DST PUBLIC DETECT TX SYSTEM IP SITE ID STATE COLOR COLOR SOURCE IP IP PORT ENCAP MULTIPLIER INTERVAL(msec) UPTIME TRANSITIONS ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 192.168.30.5 2 up public-internet public-internet 192.168.110.6 192.168.109.5 12386 ipsec 7 1000 0:02:11:05 1 192.168.30.7 4 up public-internet public-internet 192.168.110.6 192.168.103.7 12366 ipsec 7 1000 0:02:11:13 2
vedge4# show bfd sessions SOURCE TLOC REMOTE TLOC DST PUBLIC DST PUBLIC DETECT TX SYSTEM IP SITE ID STATE COLOR COLOR SOURCE IP IP PORT ENCAP MULTIPLIER INTERVAL(msec) UPTIME TRANSITIONS ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 192.168.30.4 13 up public-internet biz-internet 192.168.103.7 192.168.109.4 12346 ipsec 7 1000 0:02:09:11 2 192.168.30.4 13 up public-internet public-internet 192.168.103.7 192.168.110.6 63084 ipsec 7 1000 0:02:09:16 2 192.168.30.5 2 up public-internet public-internet 192.168.103.7 192.168.109.5 12386 ipsec 7 1000 0:02:09:10 3 192.168.30.6 13 up public-internet public-internet 192.168.103.7 192.168.110.6 12386 ipsec 7 1000 0:02:09:07 2
If you can't achieve desired result with traffic engineering, then check that policies were applied properly:
1. On vedge4 you should check that for prefixes originated from "site 13" appropriate TLOC was selected:
vedge4# show omp routes 192.168.40.0/24 detail --------------------------------------------------- omp route entries for vpn 40 route 192.168.40.0/24 --------------------------------------------------- RECEIVED FROM: peer 192.168.30.3 path-id 72 label 1002 status R loss-reason tloc-preference lost-to-peer 192.168.30.3 lost-to-path-id 74 Attributes: originator 192.168.30.4 type installed tloc 192.168.30.4, biz-internet, ipsec ultimate-tloc not set domain-id not set overlay-id 1 site-id 13 preference not set tag not set origin-proto connected origin-metric 0 as-path not set unknown-attr-len not set RECEIVED FROM: peer 192.168.30.3 path-id 73 label 1002 status C,I,R loss-reason not set lost-to-peer not set lost-to-path-id not set Attributes: originator 192.168.30.4 type installed tloc 192.168.30.4, public-internet, ipsec ultimate-tloc not set domain-id not set overlay-id 1 site-id 13 preference not set tag not set origin-proto connected origin-metric 0 as-path not set unknown-attr-len not set RECEIVED FROM: peer 192.168.30.3 path-id 74 label 1002 status C,I,R loss-reason not set lost-to-peer not set lost-to-path-id not set Attributes: originator 192.168.30.6 type installed tloc 192.168.30.6, public-internet, ipsec ultimate-tloc not set domain-id not set overlay-id 1 site-id 13 preference not set tag not set origin-proto connected origin-metric 0 as-path not set unknown-attr-len not set
2. On vedge1 and vedge3 we should make sure that appropriate policy from vSmart was installed and packets are matching and being counted:
vedge1# show policy from-vsmart from-vsmart sla-class SLA_CL1 loss 1 latency 100 jitter 100 from-vsmart app-route-policy S13_S4_via_PUB vpn-list CORP_VPNs sequence 10 match destination-data-prefix-list SITE4_PREFIX action count COUNT_PKT backup-sla-preferred-color biz-internet sla-class SLA_CL1 no sla-class strict sla-class preferred-color public-internet from-vsmart lists vpn-list CORP_VPNs vpn 40 from-vsmart lists data-prefix-list SITE4_PREFIX ip-prefix 192.168.60.0/24 vedge1# show policy app-route-policy-filter COUNTER NAME NAME NAME PACKETS BYTES ------------------------------------------------- S13_S4_via_PUB CORP_VPNs COUNT_PKT 81126791 110610503611
Besides that you should see much more packets sent via "public-internet" color from "site 13" (during my testing there was no traffic via "biz-internet" TLOC):
vedge1# show app-route stats remote-system-ip 192.168.30.7 app-route statistics 192.168.80.4 192.168.103.7 ipsec 12386 12366 remote-system-ip 192.168.30.7 local-color public-internet remote-color public-internet mean-loss 0 mean-latency 1 mean-jitter 0 sla-class-index 0,1 TOTAL AVERAGE AVERAGE TX DATA RX DATA INDEX PACKETS LOSS LATENCY JITTER PKTS PKTS ---------------------------------------------------------- 0 600 0 0 0 0 0 1 600 0 1 0 5061061 6731986 2 600 0 0 0 3187291 3619658 3 600 0 0 0 0 0 4 600 0 2 0 9230960 12707216 5 600 0 1 0 9950840 4541723 app-route statistics 192.168.109.4 192.168.103.7 ipsec 12346 12366 remote-system-ip 192.168.30.7 local-color biz-internet remote-color public-internet mean-loss 0 mean-latency 0 mean-jitter 0 sla-class-index 0,1 TOTAL AVERAGE AVERAGE TX DATA RX DATA INDEX PACKETS LOSS LATENCY JITTER PKTS PKTS ---------------------------------------------------------- 0 600 0 0 0 0 0 1 600 0 1 0 0 0 2 600 0 0 0 0 0 3 600 0 0 0 0 0 4 600 0 2 0 0 0 5 600 0 0 0 0 0
https://sdwan-docs.cisco.com/Product_Documentation/Command_Reference/Configuration_Commands/color
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: