cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
5439
Views
0
Helpful
5
Replies

IWAN PFRv3 Load Balancing

I am working on a 4 site IWAN - PFRv3 install.  This is currently in lab.  I thought I had it all working but when I started generating some traffic for testing I found that all traffic paths are using the same tunnel.  The domain master exist cli shows the inbalance.  Am I simply not generating enough to tip the load balance features.  I did read that the available bandwidth is the first criteria.  Ahead of explicitly stated path statements.  Bandwidth on the interfaces are the same at all points.   Thoughts?  Any help is appriceated.

CR1#sh domain IWAN master traffic-classes summary

APP - APPLICATION, TC-ID - TRAFFIC-CLASS-ID, APP-ID - APPLICATION-ID
SP - SERVICE PROVIDER, PC = PRIMARY CHANNEL ID,
BC - BACKUP CHANNEL ID, BR - BORDER, EXIT - WAN INTERFACE
UC - UNCONTROLLED, PE - PICK-EXIT, CN - CONTROLLED, UK - UNKNOWN

Dst-Site-Pfx      Dst-Site-Id       State DSCP      TC-ID APP-ID    APP           Current-Exit

10.10.40.0/24    10.1.127.5      CN    4[4]      2608  N/A       N/A  CTLMPLS(0:0|0:0)/10.1.127.1/Tu100(Ch:1373)
10.10.40.0/24    10.1.127.5      CN    cs5[40]   2581  N/A       N/A           CTLMPLS(0:0|0:0)/10.1.127.1/Tu100(Ch:1340)
10.10.40.0/24    10.1.127.5      CN    cs7[56]   2580  N/A       N/A           CTLMPLS(0:0|0:0)/10.1.127.1/Tu100(Ch:1338)
10.10.40.0/24    10.1.127.5      CN    default[0 2564  N/A       N/A           CTLMPLS(0:0|0:0)/10.1.127.1/Tu100(Ch:913)
10.10.25.5/32    10.1.127.7      CN    default[0 2284  N/A       N/A           CTLMPLS(0:0|0:0)/10.1.127.1/Tu100(Ch:438)
10.10.25.6/32    10.1.127.7      CN    default[0 2279  N/A       N/A           CTLMPLS(0:0|0:0)/10.1.127.1/Tu100(Ch:438)
10.10.40.5/32    10.1.127.5      CN    default[0 2255  N/A       N/A           CTLMPLS(0:0|0:0)/10.1.127.1/Tu100(Ch:913)
10.10.40.6/32    10.1.127.5      CN    default[0 921   N/A       N/A           CTLMPLS(0:0|0:0)/10.1.127.1/Tu100(Ch:913)
10.11.0.6/32     10.1.127.9      CN    default[0 2300  N/A       N/A           CTLMPLS(0:0|0:0)/10.1.127.1/Tu100(Ch:440)
10.10.25.0/24    10.1.127.7      CN    4[4]      2609  N/A       N/A           CTLMPLS(0:0|0:0)/10.1.127.1/Tu100(Ch:1375)
10.11.0.5/32     10.1.127.9      CN    default[0 2299  N/A       N/A           CTLMPLS(0:0|0:0)/10.1.127.1/Tu100(Ch:440)
10.11.0.0/24     10.1.127.9      CN    4[4]      2583  N/A       N/A           CTLMPLS(0:0|0:0)/10.1.127.1/Tu100(Ch:1344)
10.11.0.0/24     10.1.127.9      CN    default[0 301   N/A       N/A           CTLMPLS(0:0|0:0)/10.1.127.1/Tu100(Ch:440)
 Total Traffic Classes: 13 Site: 13  Internet: 0
CR1#sh domain IWAN master exits                  

  BR address: 10.1.127.1 | Name: Tunnel100 | type: external | Path: MPLS1 | path-id: 0 | PLR TCs: 0
      Egress capacity: 102400 Kbps | Egress BW: 17302 Kbps | Ideal:9369 Kbps | over: 7933 Kbps | Egress Utilization: 16 %
      DSCP: default[0]-Number of Traffic Classes[8]
      DSCP: 4[4]-Number of Traffic Classes[3]
      DSCP: cs5[40]-Number of Traffic Classes[1]
      DSCP: cs7[56]-Number of Traffic Classes[1]

  BR address: 10.1.127.2 | Name: Tunnel200 | type: external | Path: MPLS2 | path-id: 0 | PLR TCs: 0
      Egress capacity: 102400 Kbps | Egress BW: 1436 Kbps | Ideal:9369 Kbps | under: 7933 Kbps | Egress Utilization: 1 %
 
--------------------------------------------------------------------------------
CR1#sh run | sec domain                                

domain IWAN
 vrf default
  border
   source-interface Loopback0
   master local
  master hub
   source-interface Loopback0
   site-prefixes prefix-list DC01-SitePrefix
   monitor-interval 2 dscp af33
   monitor-interval 2 dscp cs4
   monitor-interval 2 dscp cs5
   monitor-interval 2 dscp ef
   load-balance advanced
    path-preference MPLS2 fallback MPLS1
   enterprise-prefix  prefix-list EntPrefix
   class VOICE sequence 10
    match dscp ef policy voice
    path-preference MPLS1 fallback MPLS2
   class SMB sequence 20
    match dscp default policy bulk-data
    path-preference MPLS2 fallback MPLS1
   class Background sequence 30
    match dscp cs1 policy best-effort
    path-preference MPLS2 fallback MPLS1

1 Accepted Solution

Accepted Solutions

Hello.

You are right - that is the problem:

Channel Id: 1668  Dst Site-Id: 10.1.127.7  Link Name: MPLS2  DSCP: 4 [4] pfr-label: 0:0 | 0:0 [0x0] TCs: 0  BackupTCs: 0
  Channel Created: 00:04:24 ago
  Provisional State: Initiated and open
  Operational state: Not-Available(Channel in Initial state)

To troubleshoot you need to go to the BRs (Hub and branch) and collect:

  • show domain ... border channel
  • show domain ... border channel parent-route

This, for example, may be a routing issue, when RIB (on HubBR) for remote branch does not point into the tunnel or has [unexpectedly] multiple entries; or may be QoS issue.

PS: basic diagram with IP-addresses (including overlay ip-addresses) could be beneficial.

View solution in original post

5 Replies 5

Hello.

You may run debugs for pdp process:

  • debug domain IWAN master pdp path-preference   
  • debug domain IWAN master pdp path-selection 

Also you may lower tunnel bandwidth to 20M and you won't need more traffic to observe the behaviour.

Could you also provide outputs for "show domain IWAN master traffic dscp 4" + "show domain IWAN master channel" ?

Attached debug and show outputs.

Thank you for your time and input on this. 

Hello.

You are right - that is the problem:

Channel Id: 1668  Dst Site-Id: 10.1.127.7  Link Name: MPLS2  DSCP: 4 [4] pfr-label: 0:0 | 0:0 [0x0] TCs: 0  BackupTCs: 0
  Channel Created: 00:04:24 ago
  Provisional State: Initiated and open
  Operational state: Not-Available(Channel in Initial state)

To troubleshoot you need to go to the BRs (Hub and branch) and collect:

  • show domain ... border channel
  • show domain ... border channel parent-route

This, for example, may be a routing issue, when RIB (on HubBR) for remote branch does not point into the tunnel or has [unexpectedly] multiple entries; or may be QoS issue.

PS: basic diagram with IP-addresses (including overlay ip-addresses) could be beneficial.

I was so focused on your reply from the other post I didnt see this reply.  Reposting the resolution here.

The Devil is in the details. Your reply helped me focus and discover the root cause, so thank you very much. I was only seeing RX in the Initial State. TX was Reachable. Not sure how that added up to me checking the routing config but it did. One of those 'you look are the config 100 times but still seem to miss what you missed' I have been through the CVD and many other references but you still miss things. This is my first real exposure to PfR (any version) and from what I have heard v3 is far easier the previous versions.

The root cause was that I missed a route-map on the second BR that tags all routes as the loopback / routerID IP.

router eigrp IWAN
!
address-family ipv4 unicast autonomous-system 1001
!
af-interface Tunnel200
summary-address 10.16.40.0 255.255.252.0
summary-address 10.18.0.0 255.255.252.0
hello-interval 20
hold-time 60
no next-hop-self
no split-horizon
exit-af-interface
!
af-interface GigabitEthernet0/0/0.3
passive-interface
exit-af-interface
!
af-interface GigabitEthernet0/0/0.4
passive-interface
exit-af-interface
!
af-interface GigabitEthernet0/0/0.5
passive-interface
exit-af-interface
!
topology base

##### missing corresponding route-map for distribution-list below ####

distribute-list route-map SET-TAG-ALL out Tunnel200

distribute-list route-map BLOCK-DMVPN1 in GigabitEthernet0/0/0.2
distribute-list route-map SET-TAG-DMVPN2 out GigabitEthernet0/0/0.2
exit-af-topology
network 10.18.0.0 0.0.0.255
network 10.18.1.0 0.0.0.255
network 10.18.2.0 0.0.0.255
network 10.18.3.0 0.0.0.255
network 10.31.126.0 0.0.0.255
network 10.31.127.2 0.0.0.0
eigrp router-id 10.31.127.2
exit-address-family

route-map BLOCK-DMVPN1 deny 10
match tag 10.31.127.1
!
route-map BLOCK-DMVPN1 permit 20
!

###### COMPLETELY MISSING route-map below #####

route-map SET-TAG-ALL permit 10
description tag all routes advertised through the tunnel
set tag 10.31.127.2
!
route-map LEAK-DMVPN2 permit 10
match ip address prefix-list LOCAL-ROUTES
set tag 10.31.127.2
!
route-map SET-TAG-DMVPN2 permit 10
description tag all routes advertised through the tunnel
match ip route-source DMVPN2-SPOKES
set tag 10.31.127.2
!
route-map SET-TAG-DMVPN2 permit 20
description advertise all other routes with no tag

I see in the 'show domain...channels' cmd that all of the tunnels over MPLS2 are provisional state as 'initiated and open' but the operational state is 'not-available'

Now to figure out why.

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Innovations in Cisco Full Stack Observability - A new webinar from Cisco