cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
4101
Views
0
Helpful
1
Replies

Implementing Performance Routing (PfR) BGP Inbound Optimization

kriska.martin
Level 1
Level 1

Hello everybody,

I am trying to test various use cases of Cisco Performance Routing technology ina lab environment

the devices used are 2811's with either:

15.1(3)T3 Enterprise Services PfR version 3.0

or

12.4(24)T2 Advanced IP Services PfR version 2.2

I tested successfully following PfR scenarios (with minor modifications of course)

static routing --> PfR injected static route to divert some traffic .... OK

static PBR --> PfR performed PBR for the traffic class I specified manually .... OK

PBR with NBAR/CCE --> PfR performed PBR for traffic that NBAR recognized and matched .... OK

BGP --> PfR selected different path to prefix than which was chosen by BGP path selection algorithm .... OK

I watched INE PfR/OER vSeminar which was really the best resource for me so far

I also read the PfR configuration guide which is great for theoretical background but in my opinion lacks some more complex examples.

So here I am, struggling with PfR Inbound BGP optimization

This is what my current topology is

PfR.png

R10 is ISP1

R11 is ISP2

R12 is a company that implemented PfR and wants to do BGP (outbound) optimization for traffic towards 100.100.50.0/24 and 100.100.51.0/24 prefixes based on delay.

Path via R12 -> R11 -> R10 is clocked to 4Mbps end to end and bandwidth command is set on R12 S0/0/0

Path via R12 -> R10 is clocked to 1Mbps and bandwidth command is set on R12 S0/0/1

Because of default BGP behavior, path R12 -> R10 is preferred because of shorter AS Path. Once the delay via that link rises above a threshold, PfR diverts the traffic via R12 -> R11 -> R10 and this is what I was expecting

Of course the traffic in backward direction is not aware of PfR and because of shortest AS Path, all the traffic back to LANs behind R12 passes directly via R10 --> R12

For this reason I wanted to implement Inbound Optimization via BGP using downgrade option to send community attached to eBGP advertisement towards R10 and R11 when needed.

R10 config

router bgp 300

no synchronization

bgp log-neighbor-changes

network 1.1.1.10 mask 255.255.255.255

network 100.100.50.0 mask 255.255.255.0

network 100.100.51.0 mask 255.255.255.0

neighbor 198.0.110.12 remote-as 100

neighbor 198.0.110.12 route-map BGP_COMM in

neighbor 198.0.111.11 remote-as 200

no auto-summary

!

ip bgp-community new-format

ip community-list standard BGP_COMM permit 100:90

!

route-map BGP_COMM permit 10

match community BGP_COMM

set local-preference 90

!

route-map BGP_COMM permit 20

R11 config

router bgp 200

no synchronization

bgp log-neighbor-changes

network 1.1.1.11 mask 255.255.255.255

neighbor 198.0.111.10 remote-as 300

neighbor 198.0.112.12 remote-as 100

neighbor 198.0.112.12 route-map BGP_COMM in

no auto-summary

!

ip bgp-community new-format

ip community-list standard BGP_COMM permit 100:90

!

route-map BGP_COMM permit 10

match community BGP_COMM

set local-preference 90

!

route-map BGP_COMM permit 20

R12 config

key chain PFR

key 1

key-string cisco123

!

oer master

policy-rules POLICY

logging

!

border 1.1.1.12 key-chain PFR

interface FastEthernet0/0.10 internal

interface FastEthernet0/0.11 internal

interface Serial0/0/0 external

downgrade bgp community 100:90

interface Serial0/0/1 external

downgrade bgp community 100:90

holddown 90

backoff 90 90

!

oer border

local Loopback0

master 1.1.1.12 key-chain PFR

active-probe address source interface Loopback0

!

router bgp 100

no synchronization

bgp log-neighbor-changes

network 1.1.1.12 mask 255.255.255.255

network 192.168.10.0

network 192.168.11.0

neighbor 198.0.110.10 remote-as 300

neighbor 198.0.110.10 send-community

neighbor 198.0.112.11 remote-as 200

neighbor 198.0.112.11 send-community

no auto-summary

!

ip prefix-list INSIDE_10 seq 5 permit 192.168.10.0/24

!

ip prefix-list INSIDE_11 seq 5 permit 192.168.11.0/24

!

ip prefix-list NET_50 seq 5 permit 100.100.50.0/24

!

oer-map POLICY 10

match traffic-class prefix-list NET_50

set delay threshold 100

set mode route control

set mode monitor fast

set resolve delay priority 1 variance 1

set active-probe echo 100.100.50.2

set probe frequency 2

!

oer-map POLICY 20

match traffic-class prefix-list INSIDE_10 inside

set delay threshold 50

set mode route control

set resolve delay priority 1 variance 1

set resolve range priority 3

no set resolve utilization

!

oer-map POLICY 30

match traffic-class prefix-list INSIDE_11 inside

set delay threshold 50

set mode route control

set resolve delay priority 1 variance 1

set resolve range priority 3

no set resolve utilization

full configuration is attached at the end of this post

I am flooding those serial links with traffic generated by hosts attached to switches communicating over WAN (simulated with Iperf generating TCP traffic) and the delay is ~550ms when R12 -> R10 is fully utilized

According to my policy, this delay should trigger R12 to send eBGP update with community attached that when matched lowers Local Preference and another path is preferred in the inbound direction

here is the output of

R12-PfR#show oer master prefix inside

OER Prefix Statistics:
Pas - Passive, Act - Active, S - Short term, L - Long term, Dly - Delay (ms),
P - Percentage below threshold, Jit - Jitter (ms),
MOS - Mean Opinion Score
Los - Packet Loss (packets-per-million), Un - Unreachable (flows-per-million),
E - Egress, I - Ingress, Bw - Bandwidth (kbps), N - Not applicable
U - unknown, * - uncontrolled, + - control more specific, @ - active probe all
# - Prefix monitor mode is Special, & - Blackholed Prefix
% - Force Next-Hop, ^ - Prefix is denied

Prefix (inside)         State     Time Curr BR         CurrI/F         Protocol
                      PasSDly  PasLDly   PasSUn   PasLUn  PasSLos  PasLLos
                      ActSDly  ActLDly   ActSUn   ActLUn      EBw      IBw
--------------------------------------------------------------------------------
192.168.11.0/24        INPOLICY*        0 1.1.1.12        Se0/0/1         U      
                               U        U        0        0     3438     5495
                               N        N        N        N        6      155
192.168.10.0/24        INPOLICY*        0 1.1.1.12        Se0/0/1         U      
                               U        U        0        0     1202    14359
                               N        N        N        N        4       26

Now I noticed that even though I am generating TCP traffic (Iperf and Character generation on TCP port 19 on R10) I am not seeing any delay being reported, U shows up instead as Unavailable

I guess that this is the reason why my POLICY 20 and 30 entried do not kick in

Inbound optimization happens when certain threshold of Loss is reached and inbound routing changes and I can see community attached to the route on one of ISP routers.

When I check the outside prefixes, Delay (both passive and active as I run in monitor fast) is reported and being acted upon

On the other hand I checked out the Cisco Wiki page http://docwiki.cisco.com/wiki/PfR:Solutions:InternetInboundLoadBalancing

and their PasSDly  PasLDly  columns have Unavailable value just as mine do, so this might not be it

I tried pasting the Flexible NetFlow sample config that is available on that page but it did not help me resolve the issue

Here are a few lines of debug when Loss triggered inbound optimization

*Mar  1 17:10:17.791: %OER_MC-5-NOTICE: Range OOP BR 1.1.1.12, i/f Se0/0/1, percent 32. Other BR 1.1.1.12, i/f Se0/0/0, percent 0

*Mar  1 17:10:17.791: %OER_MC-5-NOTICE: Exit 1.1.1.12 intf Se0/0/1 OOP, Tx BW 21, Rx BW 64, Tx Load 32, Rx Load 100

*Mar  1 17:10:37.307: OER MC PFX 192.168.11.0/24: PDP inside check oop, ext Se0/0/1, log 1

*Mar  1 17:10:37.307: OER MC PFX 192.168.11.0/24: Goto in policy

*Mar  1 17:10:37.307: OER MC PFX 192.168.11.0/24: PDP no start timer, prefix state = INPOLICY*

*Mar  1 17:10:37.311: OER MC PFX 192.168.11.0/24: Current exit: BR 1.1.1.12, intf 5, Se0/0/1

*Mar  1 17:10:37.311: OER MC PFX 192.168.11.0/24: PDP inside check oop, ext Se0/0/1, log 1

*Mar  1 17:10:37.311: OER MC PFX 192.168.11.0/24: Goto in policy

*Mar  1 17:10:37.311: OER MC PFX 192.168.11.0/24: PDP no start timer, prefix state = INPOLICY*

*Mar  1 17:10:37.831: %OER_MC-5-NOTICE: Range OOP BR 1.1.1.12, i/f Se0/0/1, percent 32. Other BR 1.1.1.12, i/f Se0/0/0, percent 0

*Mar  1 17:10:37.831: %OER_MC-5-NOTICE: Exit 1.1.1.12 intf Se0/0/1 OOP, Tx BW 21, Rx BW 64, Tx Load 32, Rx Load 100

*Mar  1 17:10:58.027: %OER_MC-5-NOTICE: Range OOP BR 1.1.1.12, i/f Se0/0/1, percent 32. Other BR 1.1.1.12, i/f Se0/0/0, percent 0

*Mar  1 17:10:58.027: %OER_MC-5-NOTICE: Exit 1.1.1.12 intf Se0/0/1 OOP, Tx BW 21, Rx BW 64, Tx Load 32, Rx Load 100

*Mar  1 17:11:08.307: OER MC PFX 192.168.11.0/24: PDP inside check oop, ext Se0/0/1, log 1

*Mar  1 17:11:08.307: %OER_MC-5-NOTICE: Passive REL Loss OOP Prefix 192.168.11.0/24 (inside), loss 29, relative change 11

*Mar  1 17:11:08.311: OER MC PFX 192.168.11.0/24: Current exit: BR 1.1.1.12, intf 5, Se0/0/1

*Mar  1 17:11:08.311: OER MC PFX 192.168.11.0/24: PDP choose entrance, prefix state = INPOLICY*, 7

*Mar  1 17:11:08.311: OER MC PFX 192.168.11.0/24: PDP get good entrance pas, prefix state = INPOLICY*, oop reason Loss

*Mar  1 17:11:08.311: OER MC PFX 192.168.11.0/24: PDP enforce best entr

*Mar  1 17:11:08.311: OER MC PFX 192.168.11.0/24: No way to upgrade

*Mar  1 17:11:08.311: OER MC PFX 192.168.11.0/24: PDP inside ctrl save ctx, 5000014 2 274

*Mar  1 17:11:08.311: OER MC PFX 192.168.11.0/24: PDP start timer = 15 secs, prefix state = CHOOSE

*Mar  1 17:11:08.323: OER MC PFX 192.168.11.0/24: PDP inside ctrl status 0, 5000014 2 Se0/0/1

*Mar  1 17:11:08.323: OER MC PFX 192.168.11.0/24: PDP start timer = 90 secs, prefix state = HOLDDOWN

*Mar  1 17:11:08.323: %OER_MC-5-NOTICE: Route changed Prefix 192.168.11.0/24, BR 1.1.1.12, i/f Se0/0/1, Reason None, OOP Reason Loss

*Mar  1 17:11:17.951: %OER_MC-5-NOTICE: Range OOP BR 1.1.1.12, i/f Se0/0/1, percent 100. Other BR 1.1.1.12, i/f Se0/0/0, percent 0

*Mar  1 17:11:17.951: %OER_MC-5-NOTICE: Load OOP BR 1.1.1.12, i/f Se0/0/1,  load 64 policy 48

*Mar  1 17:11:17.951: %OER_MC-5-NOTICE: Exit 1.1.1.12 intf Se0/0/1 OOP, Tx BW 64, Rx BW 64, Tx Load 100, Rx Load 100

*Mar  1 17:11:35.311: OER MC PFX 192.168.11.0/24: Current exit: BR 1.1.1.12, intf 5, Se0/0/1

*Mar  1 17:11:38.007: %OER_MC-5-NOTICE: Range OOP BR 1.1.1.12, i/f Se0/0/1, percent 100. Other BR 1.1.1.12, i/f Se0/0/0, percent 1

*Mar  1 17:11:38.007: %OER_MC-5-NOTICE: Load OOP BR 1.1.1.12, i/f Se0/0/1,  load 64 policy 48

*Mar  1 17:11:38.007: %OER_MC-5-NOTICE: Exit 1.1.1.12 intf Se0/0/1 OOP, Tx BW 64, Rx BW 64, Tx Load 100, Rx Load 100

*Mar  1 17:11:58.111: %OER_MC-5-NOTICE: Range OOP BR 1.1.1.12, i/f Se0/0/1, percent 100. Other BR 1.1.1.12, i/f Se0/0/0, percent 2

*Mar  1 17:11:58.111: %OER_MC-5-NOTICE: Load OOP BR 1.1.1.12, i/f Se0/0/1,  load 64 policy 48

*Mar  1 17:11:58.111: %OER_MC-5-NOTICE: Exit 1.1.1.12 intf Se0/0/1 OOP, Tx BW 64, Rx BW 64, Tx Load 100, Rx Load 100

*Mar  1 17:12:06.311: OER MC PFX 192.168.11.0/24: Current exit: BR 1.1.1.12, intf 5, Se0/0/1

*Mar  1 17:12:18.143: %OER_MC-5-NOTICE: Range OOP BR 1.1.1.12, i/f Se0/0/1, percent 100. Other BR 1.1.1.12, i/f Se0/0/0, percent 2

*Mar  1 17:12:18.143: %OER_MC-5-NOTICE: Load OOP BR 1.1.1.12, i/f Se0/0/1,  load 64 policy 48

*Mar  1 17:12:18.143: %OER_MC-5-NOTICE: Exit 1.1.1.12 intf Se0/0/1 OOP, Tx BW 64, Rx BW 64, Tx Load 100, Rx Load 100

*Mar  1 17:12:36.311: OER MC PFX 192.168.11.0/24: Current exit: BR 1.1.1.12, intf 5, Se0/0/1

*Mar  1 17:12:37.267: OER MC PFX 192.168.11.0/24: Prefix timeout, state HOLDDOWN

*Mar  1 17:12:37.267: OER MC PFX 192.168.11.0/24: Goto in policy

*Mar  1 17:12:37.267: OER MC PFX 192.168.11.0/24: PDP no start timer, prefix state = INPOLICY

*Mar  1 17:12:38.199: %OER_MC-5-NOTICE: Range OOP BR 1.1.1.12, i/f Se0/0/1, percent 100. Other BR 1.1.1.12, i/f Se0/0/0, percent 2

*Mar  1 17:12:38.199: %OER_MC-5-NOTICE: Load OOP BR 1.1.1.12, i/f Se0/0/1,  load 64 policy 48

*Mar  1 17:12:38.199: %OER_MC-5-NOTICE: Exit 1.1.1.12 intf Se0/0/1 OOP, Tx BW 64, Rx BW 64, Tx Load 100, Rx Load 100

*Mar  1 17:10:17.791: %OER_MC-5-NOTICE: Range OOP BR 1.1.1.12, i/f Se0/0/1, percent 32. Other BR 1.1.1.12, i/f Se0/0/0, percent 0
*Mar  1 17:10:17.791: %OER_MC-5-NOTICE: Exit 1.1.1.12 intf Se0/0/1 OOP, Tx BW 21, Rx BW 64, Tx Load 32, Rx Load 100
*Mar  1 17:10:37.307: OER MC PFX 192.168.11.0/24: PDP inside check oop, ext Se0/0/1, log 1
*Mar  1 17:10:37.307: OER MC PFX 192.168.11.0/24: Goto in policy
*Mar  1 17:10:37.307: OER MC PFX 192.168.11.0/24: PDP no start timer, prefix state = INPOLICY*
*Mar  1 17:10:37.311: OER MC PFX 192.168.11.0/24: Current exit: BR 1.1.1.12, intf 5, Se0/0/1
*Mar  1 17:10:37.311: OER MC PFX 192.168.11.0/24: PDP inside check oop, ext Se0/0/1, log 1
*Mar  1 17:10:37.311: OER MC PFX 192.168.11.0/24: Goto in policy
*Mar  1 17:10:37.311: OER MC PFX 192.168.11.0/24: PDP no start timer, prefix state = INPOLICY*
*Mar  1 17:10:37.831: %OER_MC-5-NOTICE: Range OOP BR 1.1.1.12, i/f Se0/0/1, percent 32. Other BR 1.1.1.12, i/f Se0/0/0, percent 0
*Mar  1 17:10:37.831: %OER_MC-5-NOTICE: Exit 1.1.1.12 intf Se0/0/1 OOP, Tx BW 21, Rx BW 64, Tx Load 32, Rx Load 100
*Mar  1 17:10:58.027: %OER_MC-5-NOTICE: Range OOP BR 1.1.1.12, i/f Se0/0/1, percent 32. Other BR 1.1.1.12, i/f Se0/0/0, percent 0
*Mar  1 17:10:58.027: %OER_MC-5-NOTICE: Exit 1.1.1.12 intf Se0/0/1 OOP, Tx BW 21, Rx BW 64, Tx Load 32, Rx Load 100
*Mar  1 17:11:08.307: OER MC PFX 192.168.11.0/24: PDP inside check oop, ext Se0/0/1, log 1
*Mar  1 17:11:08.307: %OER_MC-5-NOTICE: Passive REL Loss OOP Prefix 192.168.11.0/24 (inside), loss 29, relative change 11
*Mar  1 17:11:08.311: OER MC PFX 192.168.11.0/24: Current exit: BR 1.1.1.12, intf 5, Se0/0/1
*Mar  1 17:11:08.311: OER MC PFX 192.168.11.0/24: PDP choose entrance, prefix state = INPOLICY*, 7
*Mar  1 17:11:08.311: OER MC PFX 192.168.11.0/24: PDP get good entrance pas, prefix state = INPOLICY*, oop reason Loss
*Mar  1 17:11:08.311: OER MC PFX 192.168.11.0/24: PDP enforce best entr
*Mar  1 17:11:08.311: OER MC PFX 192.168.11.0/24: No way to upgrade
*Mar  1 17:11:08.311: OER MC PFX 192.168.11.0/24: PDP inside ctrl save ctx, 5000014 2 274
*Mar  1 17:11:08.311: OER MC PFX 192.168.11.0/24: PDP start timer = 15 secs, prefix state = CHOOSE
*Mar  1 17:11:08.323: OER MC PFX 192.168.11.0/24: PDP inside ctrl status 0, 5000014 2 Se0/0/1
*Mar  1 17:11:08.323: OER MC PFX 192.168.11.0/24: PDP start timer = 90 secs, prefix state = HOLDDOWN
*Mar  1 17:11:08.323: %OER_MC-5-NOTICE: Route changed Prefix 192.168.11.0/24, BR 1.1.1.12, i/f Se0/0/1, Reason None, OOP Reason Loss
*Mar  1 17:11:17.951: %OER_MC-5-NOTICE: Range OOP BR 1.1.1.12, i/f Se0/0/1, percent 100. Other BR 1.1.1.12, i/f Se0/0/0, percent 0
*Mar  1 17:11:17.951: %OER_MC-5-NOTICE: Load OOP BR 1.1.1.12, i/f Se0/0/1,  load 64 policy 48
*Mar  1 17:11:17.951: %OER_MC-5-NOTICE: Exit 1.1.1.12 intf Se0/0/1 OOP, Tx BW 64, Rx BW 64, Tx Load 100, Rx Load 100
*Mar  1 17:11:35.311: OER MC PFX 192.168.11.0/24: Current exit: BR 1.1.1.12, intf 5, Se0/0/1
*Mar  1 17:11:38.007: %OER_MC-5-NOTICE: Range OOP BR 1.1.1.12, i/f Se0/0/1, percent 100. Other BR 1.1.1.12, i/f Se0/0/0, percent 1
*Mar  1 17:11:38.007: %OER_MC-5-NOTICE: Load OOP BR 1.1.1.12, i/f Se0/0/1,  load 64 policy 48
*Mar  1 17:11:38.007: %OER_MC-5-NOTICE: Exit 1.1.1.12 intf Se0/0/1 OOP, Tx BW 64, Rx BW 64, Tx Load 100, Rx Load 100
*Mar  1 17:11:58.111: %OER_MC-5-NOTICE: Range OOP BR 1.1.1.12, i/f Se0/0/1, percent 100. Other BR 1.1.1.12, i/f Se0/0/0, percent 2
*Mar  1 17:11:58.111: %OER_MC-5-NOTICE: Load OOP BR 1.1.1.12, i/f Se0/0/1,  load 64 policy 48
*Mar  1 17:11:58.111: %OER_MC-5-NOTICE: Exit 1.1.1.12 intf Se0/0/1 OOP, Tx BW 64, Rx BW 64, Tx Load 100, Rx Load 100
*Mar  1 17:12:06.311: OER MC PFX 192.168.11.0/24: Current exit: BR 1.1.1.12, intf 5, Se0/0/1
*Mar  1 17:12:18.143: %OER_MC-5-NOTICE: Range OOP BR 1.1.1.12, i/f Se0/0/1, percent 100. Other BR 1.1.1.12, i/f Se0/0/0, percent 2
*Mar  1 17:12:18.143: %OER_MC-5-NOTICE: Load OOP BR 1.1.1.12, i/f Se0/0/1,  load 64 policy 48
*Mar  1 17:12:18.143: %OER_MC-5-NOTICE: Exit 1.1.1.12 intf Se0/0/1 OOP, Tx BW 64, Rx BW 64, Tx Load 100, Rx Load 100
*Mar  1 17:12:36.311: OER MC PFX 192.168.11.0/24: Current exit: BR 1.1.1.12, intf 5, Se0/0/1
*Mar  1 17:12:37.267: OER MC PFX 192.168.11.0/24: Prefix timeout, state HOLDDOWN
*Mar  1 17:12:37.267: OER MC PFX 192.168.11.0/24: Goto in policy
*Mar  1 17:12:37.267: OER MC PFX 192.168.11.0/24: PDP no start timer, prefix state = INPOLICY
*Mar  1 17:12:38.199: %OER_MC-5-NOTICE: Range OOP BR 1.1.1.12, i/f Se0/0/1, percent 100. Other BR 1.1.1.12, i/f Se0/0/0, percent 2
*Mar  1 17:12:38.199: %OER_MC-5-NOTICE: Load OOP BR 1.1.1.12, i/f Se0/0/1,  load 64 policy 48
*Mar  1 17:12:38.199: %OER_MC-5-NOTICE: Exit 1.1.1.12 intf Se0/0/1 OOP, Tx BW 64, Rx BW 64, Tx Load 100, Rx Load 100

So from my point of view, the config is working when Loss threshold is reached

Could anybody tell me why it does not work when delay threshold is reached?

When Loss is not reaching the threshold but delay is, the debug does not show anything useful (to me) because the delay is not reported for inbound prefixes (but why?)

*Mar  1 16:27:07.327: %OER_MC-5-NOTICE: Exit 1.1.1.12 intf Se0/0/1 OOP, Tx BW 391, Rx BW 556, Tx Load 39, Rx Load 55
*Mar  1 16:27:09.067: OER MC PFX 192.168.11.0/24: PDP inside check oop, ext Se0/0/1, log 1
*Mar  1 16:27:09.067: OER MC PFX 192.168.11.0/24: Goto in policy
*Mar  1 16:27:09.067: OER MC PFX 192.168.11.0/24: PDP no start timer, prefix state = INPOLICY*
*Mar  1 16:27:09.067: OER MC PFX 192.168.11.0/24: PDP inside check oop, ext Se0/0/1, log 1
*Mar  1 16:27:09.067: OER MC PFX 192.168.11.0/24: Goto in policy
*Mar  1 16:27:09.067: OER MC PFX 192.168.11.0/24: PDP no start timer, prefix state = INPOLICY*

Thank you very much for any answer or suggestion in advance,

Best regards,

Martin Kriska

1 Reply 1

chrilau
Cisco Employee
Cisco Employee

Hi Martin, 

Did you manage to get it fixed at the end?

Review Cisco Networking for a $25 gift card