Hello everybody,
I am trying to test various use cases of Cisco Performance Routing technology ina lab environment
the devices used are 2811's with either:
15.1(3)T3 Enterprise Services PfR version 3.0
or
12.4(24)T2 Advanced IP Services PfR version 2.2
I tested successfully following PfR scenarios (with minor modifications of course)
static routing --> PfR injected static route to divert some traffic .... OK
static PBR --> PfR performed PBR for the traffic class I specified manually .... OK
PBR with NBAR/CCE --> PfR performed PBR for traffic that NBAR recognized and matched .... OK
BGP --> PfR selected different path to prefix than which was chosen by BGP path selection algorithm .... OK
I watched INE PfR/OER vSeminar which was really the best resource for me so far
I also read the PfR configuration guide which is great for theoretical background but in my opinion lacks some more complex examples.
So here I am, struggling with PfR Inbound BGP optimization
This is what my current topology is
R10 is ISP1
R11 is ISP2
R12 is a company that implemented PfR and wants to do BGP (outbound) optimization for traffic towards 100.100.50.0/24 and 100.100.51.0/24 prefixes based on delay.
Path via R12 -> R11 -> R10 is clocked to 4Mbps end to end and bandwidth command is set on R12 S0/0/0
Path via R12 -> R10 is clocked to 1Mbps and bandwidth command is set on R12 S0/0/1
Because of default BGP behavior, path R12 -> R10 is preferred because of shorter AS Path. Once the delay via that link rises above a threshold, PfR diverts the traffic via R12 -> R11 -> R10 and this is what I was expecting
Of course the traffic in backward direction is not aware of PfR and because of shortest AS Path, all the traffic back to LANs behind R12 passes directly via R10 --> R12
For this reason I wanted to implement Inbound Optimization via BGP using downgrade option to send community attached to eBGP advertisement towards R10 and R11 when needed.
R10 config
router bgp 300
no synchronization
bgp log-neighbor-changes
network 1.1.1.10 mask 255.255.255.255
network 100.100.50.0 mask 255.255.255.0
network 100.100.51.0 mask 255.255.255.0
neighbor 198.0.110.12 remote-as 100
neighbor 198.0.110.12 route-map BGP_COMM in
neighbor 198.0.111.11 remote-as 200
no auto-summary
!
ip bgp-community new-format
ip community-list standard BGP_COMM permit 100:90
!
route-map BGP_COMM permit 10
match community BGP_COMM
set local-preference 90
!
route-map BGP_COMM permit 20
R11 config
router bgp 200
no synchronization
bgp log-neighbor-changes
network 1.1.1.11 mask 255.255.255.255
neighbor 198.0.111.10 remote-as 300
neighbor 198.0.112.12 remote-as 100
neighbor 198.0.112.12 route-map BGP_COMM in
no auto-summary
!
ip bgp-community new-format
ip community-list standard BGP_COMM permit 100:90
!
route-map BGP_COMM permit 10
match community BGP_COMM
set local-preference 90
!
route-map BGP_COMM permit 20
R12 config
key chain PFR
key 1
key-string cisco123
!
oer master
policy-rules POLICY
logging
!
border 1.1.1.12 key-chain PFR
interface FastEthernet0/0.10 internal
interface FastEthernet0/0.11 internal
interface Serial0/0/0 external
downgrade bgp community 100:90
interface Serial0/0/1 external
downgrade bgp community 100:90
holddown 90
backoff 90 90
!
oer border
local Loopback0
master 1.1.1.12 key-chain PFR
active-probe address source interface Loopback0
!
router bgp 100
no synchronization
bgp log-neighbor-changes
network 1.1.1.12 mask 255.255.255.255
network 192.168.10.0
network 192.168.11.0
neighbor 198.0.110.10 remote-as 300
neighbor 198.0.110.10 send-community
neighbor 198.0.112.11 remote-as 200
neighbor 198.0.112.11 send-community
no auto-summary
!
ip prefix-list INSIDE_10 seq 5 permit 192.168.10.0/24
!
ip prefix-list INSIDE_11 seq 5 permit 192.168.11.0/24
!
ip prefix-list NET_50 seq 5 permit 100.100.50.0/24
!
oer-map POLICY 10
match traffic-class prefix-list NET_50
set delay threshold 100
set mode route control
set mode monitor fast
set resolve delay priority 1 variance 1
set active-probe echo 100.100.50.2
set probe frequency 2
!
oer-map POLICY 20
match traffic-class prefix-list INSIDE_10 inside
set delay threshold 50
set mode route control
set resolve delay priority 1 variance 1
set resolve range priority 3
no set resolve utilization
!
oer-map POLICY 30
match traffic-class prefix-list INSIDE_11 inside
set delay threshold 50
set mode route control
set resolve delay priority 1 variance 1
set resolve range priority 3
no set resolve utilization
full configuration is attached at the end of this post
I am flooding those serial links with traffic generated by hosts attached to switches communicating over WAN (simulated with Iperf generating TCP traffic) and the delay is ~550ms when R12 -> R10 is fully utilized
According to my policy, this delay should trigger R12 to send eBGP update with community attached that when matched lowers Local Preference and another path is preferred in the inbound direction
here is the output of
R12-PfR#show oer master prefix inside
OER Prefix Statistics:
Pas - Passive, Act - Active, S - Short term, L - Long term, Dly - Delay (ms),
P - Percentage below threshold, Jit - Jitter (ms),
MOS - Mean Opinion Score
Los - Packet Loss (packets-per-million), Un - Unreachable (flows-per-million),
E - Egress, I - Ingress, Bw - Bandwidth (kbps), N - Not applicable
U - unknown, * - uncontrolled, + - control more specific, @ - active probe all
# - Prefix monitor mode is Special, & - Blackholed Prefix
% - Force Next-Hop, ^ - Prefix is denied
Prefix (inside) State Time Curr BR CurrI/F Protocol
PasSDly PasLDly PasSUn PasLUn PasSLos PasLLos
ActSDly ActLDly ActSUn ActLUn EBw IBw
--------------------------------------------------------------------------------
192.168.11.0/24 INPOLICY* 0 1.1.1.12 Se0/0/1 U
U U 0 0 3438 5495
N N N N 6 155
192.168.10.0/24 INPOLICY* 0 1.1.1.12 Se0/0/1 U
U U 0 0 1202 14359
N N N N 4 26
Now I noticed that even though I am generating TCP traffic (Iperf and Character generation on TCP port 19 on R10) I am not seeing any delay being reported, U shows up instead as Unavailable
I guess that this is the reason why my POLICY 20 and 30 entried do not kick in
Inbound optimization happens when certain threshold of Loss is reached and inbound routing changes and I can see community attached to the route on one of ISP routers.
When I check the outside prefixes, Delay (both passive and active as I run in monitor fast) is reported and being acted upon
On the other hand I checked out the Cisco Wiki page http://docwiki.cisco.com/wiki/PfR:Solutions:InternetInboundLoadBalancing
and their PasSDly PasLDly columns have Unavailable value just as mine do, so this might not be it
I tried pasting the Flexible NetFlow sample config that is available on that page but it did not help me resolve the issue
Here are a few lines of debug when Loss triggered inbound optimization
*Mar 1 17:10:17.791: %OER_MC-5-NOTICE: Range OOP BR 1.1.1.12, i/f Se0/0/1, percent 32. Other BR 1.1.1.12, i/f Se0/0/0, percent 0
*Mar 1 17:10:17.791: %OER_MC-5-NOTICE: Exit 1.1.1.12 intf Se0/0/1 OOP, Tx BW 21, Rx BW 64, Tx Load 32, Rx Load 100
*Mar 1 17:10:37.307: OER MC PFX 192.168.11.0/24: PDP inside check oop, ext Se0/0/1, log 1
*Mar 1 17:10:37.307: OER MC PFX 192.168.11.0/24: Goto in policy
*Mar 1 17:10:37.307: OER MC PFX 192.168.11.0/24: PDP no start timer, prefix state = INPOLICY*
*Mar 1 17:10:37.311: OER MC PFX 192.168.11.0/24: Current exit: BR 1.1.1.12, intf 5, Se0/0/1
*Mar 1 17:10:37.311: OER MC PFX 192.168.11.0/24: PDP inside check oop, ext Se0/0/1, log 1
*Mar 1 17:10:37.311: OER MC PFX 192.168.11.0/24: Goto in policy
*Mar 1 17:10:37.311: OER MC PFX 192.168.11.0/24: PDP no start timer, prefix state = INPOLICY*
*Mar 1 17:10:37.831: %OER_MC-5-NOTICE: Range OOP BR 1.1.1.12, i/f Se0/0/1, percent 32. Other BR 1.1.1.12, i/f Se0/0/0, percent 0
*Mar 1 17:10:37.831: %OER_MC-5-NOTICE: Exit 1.1.1.12 intf Se0/0/1 OOP, Tx BW 21, Rx BW 64, Tx Load 32, Rx Load 100
*Mar 1 17:10:58.027: %OER_MC-5-NOTICE: Range OOP BR 1.1.1.12, i/f Se0/0/1, percent 32. Other BR 1.1.1.12, i/f Se0/0/0, percent 0
*Mar 1 17:10:58.027: %OER_MC-5-NOTICE: Exit 1.1.1.12 intf Se0/0/1 OOP, Tx BW 21, Rx BW 64, Tx Load 32, Rx Load 100
*Mar 1 17:11:08.307: OER MC PFX 192.168.11.0/24: PDP inside check oop, ext Se0/0/1, log 1
*Mar 1 17:11:08.307: %OER_MC-5-NOTICE: Passive REL Loss OOP Prefix 192.168.11.0/24 (inside), loss 29, relative change 11
*Mar 1 17:11:08.311: OER MC PFX 192.168.11.0/24: Current exit: BR 1.1.1.12, intf 5, Se0/0/1
*Mar 1 17:11:08.311: OER MC PFX 192.168.11.0/24: PDP choose entrance, prefix state = INPOLICY*, 7
*Mar 1 17:11:08.311: OER MC PFX 192.168.11.0/24: PDP get good entrance pas, prefix state = INPOLICY*, oop reason Loss
*Mar 1 17:11:08.311: OER MC PFX 192.168.11.0/24: PDP enforce best entr
*Mar 1 17:11:08.311: OER MC PFX 192.168.11.0/24: No way to upgrade
*Mar 1 17:11:08.311: OER MC PFX 192.168.11.0/24: PDP inside ctrl save ctx, 5000014 2 274
*Mar 1 17:11:08.311: OER MC PFX 192.168.11.0/24: PDP start timer = 15 secs, prefix state = CHOOSE
*Mar 1 17:11:08.323: OER MC PFX 192.168.11.0/24: PDP inside ctrl status 0, 5000014 2 Se0/0/1
*Mar 1 17:11:08.323: OER MC PFX 192.168.11.0/24: PDP start timer = 90 secs, prefix state = HOLDDOWN
*Mar 1 17:11:08.323: %OER_MC-5-NOTICE: Route changed Prefix 192.168.11.0/24, BR 1.1.1.12, i/f Se0/0/1, Reason None, OOP Reason Loss
*Mar 1 17:11:17.951: %OER_MC-5-NOTICE: Range OOP BR 1.1.1.12, i/f Se0/0/1, percent 100. Other BR 1.1.1.12, i/f Se0/0/0, percent 0
*Mar 1 17:11:17.951: %OER_MC-5-NOTICE: Load OOP BR 1.1.1.12, i/f Se0/0/1, load 64 policy 48
*Mar 1 17:11:17.951: %OER_MC-5-NOTICE: Exit 1.1.1.12 intf Se0/0/1 OOP, Tx BW 64, Rx BW 64, Tx Load 100, Rx Load 100
*Mar 1 17:11:35.311: OER MC PFX 192.168.11.0/24: Current exit: BR 1.1.1.12, intf 5, Se0/0/1
*Mar 1 17:11:38.007: %OER_MC-5-NOTICE: Range OOP BR 1.1.1.12, i/f Se0/0/1, percent 100. Other BR 1.1.1.12, i/f Se0/0/0, percent 1
*Mar 1 17:11:38.007: %OER_MC-5-NOTICE: Load OOP BR 1.1.1.12, i/f Se0/0/1, load 64 policy 48
*Mar 1 17:11:38.007: %OER_MC-5-NOTICE: Exit 1.1.1.12 intf Se0/0/1 OOP, Tx BW 64, Rx BW 64, Tx Load 100, Rx Load 100
*Mar 1 17:11:58.111: %OER_MC-5-NOTICE: Range OOP BR 1.1.1.12, i/f Se0/0/1, percent 100. Other BR 1.1.1.12, i/f Se0/0/0, percent 2
*Mar 1 17:11:58.111: %OER_MC-5-NOTICE: Load OOP BR 1.1.1.12, i/f Se0/0/1, load 64 policy 48
*Mar 1 17:11:58.111: %OER_MC-5-NOTICE: Exit 1.1.1.12 intf Se0/0/1 OOP, Tx BW 64, Rx BW 64, Tx Load 100, Rx Load 100
*Mar 1 17:12:06.311: OER MC PFX 192.168.11.0/24: Current exit: BR 1.1.1.12, intf 5, Se0/0/1
*Mar 1 17:12:18.143: %OER_MC-5-NOTICE: Range OOP BR 1.1.1.12, i/f Se0/0/1, percent 100. Other BR 1.1.1.12, i/f Se0/0/0, percent 2
*Mar 1 17:12:18.143: %OER_MC-5-NOTICE: Load OOP BR 1.1.1.12, i/f Se0/0/1, load 64 policy 48
*Mar 1 17:12:18.143: %OER_MC-5-NOTICE: Exit 1.1.1.12 intf Se0/0/1 OOP, Tx BW 64, Rx BW 64, Tx Load 100, Rx Load 100
*Mar 1 17:12:36.311: OER MC PFX 192.168.11.0/24: Current exit: BR 1.1.1.12, intf 5, Se0/0/1
*Mar 1 17:12:37.267: OER MC PFX 192.168.11.0/24: Prefix timeout, state HOLDDOWN
*Mar 1 17:12:37.267: OER MC PFX 192.168.11.0/24: Goto in policy
*Mar 1 17:12:37.267: OER MC PFX 192.168.11.0/24: PDP no start timer, prefix state = INPOLICY
*Mar 1 17:12:38.199: %OER_MC-5-NOTICE: Range OOP BR 1.1.1.12, i/f Se0/0/1, percent 100. Other BR 1.1.1.12, i/f Se0/0/0, percent 2
*Mar 1 17:12:38.199: %OER_MC-5-NOTICE: Load OOP BR 1.1.1.12, i/f Se0/0/1, load 64 policy 48
*Mar 1 17:12:38.199: %OER_MC-5-NOTICE: Exit 1.1.1.12 intf Se0/0/1 OOP, Tx BW 64, Rx BW 64, Tx Load 100, Rx Load 100
*Mar 1 17:10:17.791: %OER_MC-5-NOTICE: Range OOP BR 1.1.1.12, i/f Se0/0/1, percent 32. Other BR 1.1.1.12, i/f Se0/0/0, percent 0
*Mar 1 17:10:17.791: %OER_MC-5-NOTICE: Exit 1.1.1.12 intf Se0/0/1 OOP, Tx BW 21, Rx BW 64, Tx Load 32, Rx Load 100
*Mar 1 17:10:37.307: OER MC PFX 192.168.11.0/24: PDP inside check oop, ext Se0/0/1, log 1
*Mar 1 17:10:37.307: OER MC PFX 192.168.11.0/24: Goto in policy
*Mar 1 17:10:37.307: OER MC PFX 192.168.11.0/24: PDP no start timer, prefix state = INPOLICY*
*Mar 1 17:10:37.311: OER MC PFX 192.168.11.0/24: Current exit: BR 1.1.1.12, intf 5, Se0/0/1
*Mar 1 17:10:37.311: OER MC PFX 192.168.11.0/24: PDP inside check oop, ext Se0/0/1, log 1
*Mar 1 17:10:37.311: OER MC PFX 192.168.11.0/24: Goto in policy
*Mar 1 17:10:37.311: OER MC PFX 192.168.11.0/24: PDP no start timer, prefix state = INPOLICY*
*Mar 1 17:10:37.831: %OER_MC-5-NOTICE: Range OOP BR 1.1.1.12, i/f Se0/0/1, percent 32. Other BR 1.1.1.12, i/f Se0/0/0, percent 0
*Mar 1 17:10:37.831: %OER_MC-5-NOTICE: Exit 1.1.1.12 intf Se0/0/1 OOP, Tx BW 21, Rx BW 64, Tx Load 32, Rx Load 100
*Mar 1 17:10:58.027: %OER_MC-5-NOTICE: Range OOP BR 1.1.1.12, i/f Se0/0/1, percent 32. Other BR 1.1.1.12, i/f Se0/0/0, percent 0
*Mar 1 17:10:58.027: %OER_MC-5-NOTICE: Exit 1.1.1.12 intf Se0/0/1 OOP, Tx BW 21, Rx BW 64, Tx Load 32, Rx Load 100
*Mar 1 17:11:08.307: OER MC PFX 192.168.11.0/24: PDP inside check oop, ext Se0/0/1, log 1
*Mar 1 17:11:08.307: %OER_MC-5-NOTICE: Passive REL Loss OOP Prefix 192.168.11.0/24 (inside), loss 29, relative change 11
*Mar 1 17:11:08.311: OER MC PFX 192.168.11.0/24: Current exit: BR 1.1.1.12, intf 5, Se0/0/1
*Mar 1 17:11:08.311: OER MC PFX 192.168.11.0/24: PDP choose entrance, prefix state = INPOLICY*, 7
*Mar 1 17:11:08.311: OER MC PFX 192.168.11.0/24: PDP get good entrance pas, prefix state = INPOLICY*, oop reason Loss
*Mar 1 17:11:08.311: OER MC PFX 192.168.11.0/24: PDP enforce best entr
*Mar 1 17:11:08.311: OER MC PFX 192.168.11.0/24: No way to upgrade
*Mar 1 17:11:08.311: OER MC PFX 192.168.11.0/24: PDP inside ctrl save ctx, 5000014 2 274
*Mar 1 17:11:08.311: OER MC PFX 192.168.11.0/24: PDP start timer = 15 secs, prefix state = CHOOSE
*Mar 1 17:11:08.323: OER MC PFX 192.168.11.0/24: PDP inside ctrl status 0, 5000014 2 Se0/0/1
*Mar 1 17:11:08.323: OER MC PFX 192.168.11.0/24: PDP start timer = 90 secs, prefix state = HOLDDOWN
*Mar 1 17:11:08.323: %OER_MC-5-NOTICE: Route changed Prefix 192.168.11.0/24, BR 1.1.1.12, i/f Se0/0/1, Reason None, OOP Reason Loss
*Mar 1 17:11:17.951: %OER_MC-5-NOTICE: Range OOP BR 1.1.1.12, i/f Se0/0/1, percent 100. Other BR 1.1.1.12, i/f Se0/0/0, percent 0
*Mar 1 17:11:17.951: %OER_MC-5-NOTICE: Load OOP BR 1.1.1.12, i/f Se0/0/1, load 64 policy 48
*Mar 1 17:11:17.951: %OER_MC-5-NOTICE: Exit 1.1.1.12 intf Se0/0/1 OOP, Tx BW 64, Rx BW 64, Tx Load 100, Rx Load 100
*Mar 1 17:11:35.311: OER MC PFX 192.168.11.0/24: Current exit: BR 1.1.1.12, intf 5, Se0/0/1
*Mar 1 17:11:38.007: %OER_MC-5-NOTICE: Range OOP BR 1.1.1.12, i/f Se0/0/1, percent 100. Other BR 1.1.1.12, i/f Se0/0/0, percent 1
*Mar 1 17:11:38.007: %OER_MC-5-NOTICE: Load OOP BR 1.1.1.12, i/f Se0/0/1, load 64 policy 48
*Mar 1 17:11:38.007: %OER_MC-5-NOTICE: Exit 1.1.1.12 intf Se0/0/1 OOP, Tx BW 64, Rx BW 64, Tx Load 100, Rx Load 100
*Mar 1 17:11:58.111: %OER_MC-5-NOTICE: Range OOP BR 1.1.1.12, i/f Se0/0/1, percent 100. Other BR 1.1.1.12, i/f Se0/0/0, percent 2
*Mar 1 17:11:58.111: %OER_MC-5-NOTICE: Load OOP BR 1.1.1.12, i/f Se0/0/1, load 64 policy 48
*Mar 1 17:11:58.111: %OER_MC-5-NOTICE: Exit 1.1.1.12 intf Se0/0/1 OOP, Tx BW 64, Rx BW 64, Tx Load 100, Rx Load 100
*Mar 1 17:12:06.311: OER MC PFX 192.168.11.0/24: Current exit: BR 1.1.1.12, intf 5, Se0/0/1
*Mar 1 17:12:18.143: %OER_MC-5-NOTICE: Range OOP BR 1.1.1.12, i/f Se0/0/1, percent 100. Other BR 1.1.1.12, i/f Se0/0/0, percent 2
*Mar 1 17:12:18.143: %OER_MC-5-NOTICE: Load OOP BR 1.1.1.12, i/f Se0/0/1, load 64 policy 48
*Mar 1 17:12:18.143: %OER_MC-5-NOTICE: Exit 1.1.1.12 intf Se0/0/1 OOP, Tx BW 64, Rx BW 64, Tx Load 100, Rx Load 100
*Mar 1 17:12:36.311: OER MC PFX 192.168.11.0/24: Current exit: BR 1.1.1.12, intf 5, Se0/0/1
*Mar 1 17:12:37.267: OER MC PFX 192.168.11.0/24: Prefix timeout, state HOLDDOWN
*Mar 1 17:12:37.267: OER MC PFX 192.168.11.0/24: Goto in policy
*Mar 1 17:12:37.267: OER MC PFX 192.168.11.0/24: PDP no start timer, prefix state = INPOLICY
*Mar 1 17:12:38.199: %OER_MC-5-NOTICE: Range OOP BR 1.1.1.12, i/f Se0/0/1, percent 100. Other BR 1.1.1.12, i/f Se0/0/0, percent 2
*Mar 1 17:12:38.199: %OER_MC-5-NOTICE: Load OOP BR 1.1.1.12, i/f Se0/0/1, load 64 policy 48
*Mar 1 17:12:38.199: %OER_MC-5-NOTICE: Exit 1.1.1.12 intf Se0/0/1 OOP, Tx BW 64, Rx BW 64, Tx Load 100, Rx Load 100
So from my point of view, the config is working when Loss threshold is reached
Could anybody tell me why it does not work when delay threshold is reached?
When Loss is not reaching the threshold but delay is, the debug does not show anything useful (to me) because the delay is not reported for inbound prefixes (but why?)
*Mar 1 16:27:07.327: %OER_MC-5-NOTICE: Exit 1.1.1.12 intf Se0/0/1 OOP, Tx BW 391, Rx BW 556, Tx Load 39, Rx Load 55
*Mar 1 16:27:09.067: OER MC PFX 192.168.11.0/24: PDP inside check oop, ext Se0/0/1, log 1
*Mar 1 16:27:09.067: OER MC PFX 192.168.11.0/24: Goto in policy
*Mar 1 16:27:09.067: OER MC PFX 192.168.11.0/24: PDP no start timer, prefix state = INPOLICY*
*Mar 1 16:27:09.067: OER MC PFX 192.168.11.0/24: PDP inside check oop, ext Se0/0/1, log 1
*Mar 1 16:27:09.067: OER MC PFX 192.168.11.0/24: Goto in policy
*Mar 1 16:27:09.067: OER MC PFX 192.168.11.0/24: PDP no start timer, prefix state = INPOLICY*
Thank you very much for any answer or suggestion in advance,
Best regards,
Martin Kriska
