ASR9000/XR: Implementing QOS policy propagation for BGP (QPPB)

xthuijs · ‎01-16-2014

Introduction

This document describes how to work with and implement QOS Policy Propagation for BGP, or in short QPPB.

QPPB is the concept of marking prefixes received from BGP with a certain QOS group assignment and this QOS-group can be used

in egress policies (or even ingress policies after marking) to perform certain actions.

While this seems so simple, and it is technically, the ability to achieve this functionality in hardware is quite complex.

In this article I hope to unravle some of these gotcha's and operation when it comes to QPPB.

Support

QPPBwith egress qos policy is supported in Trident, Thor and Typhoon. Both IPv4 & IPv6 is supported.

ASR9K is the first XR platform to support IPv6 QPPB.

Based on interface qppb settings, for the source/Destination prefix, the IP Precedence and/or QOS group is set in ingress linecard and policy can be attached in egress linecard to act on the modified IP Precedence and/or QOS group.

Back to top

1. Purpose:

In short the purpose of QPPB is to allow packet classification based on BGP attributes.

IPv4/IPv6 unicast only feature. IPv6 Support only added in 4.2 only in ASR9k platform.
Classification based on source or destination IP address
Classify on BGP IP prefix / AS path / Community
Utilizes Route Policy Language (RPL) for table policies
QoS Group ID values 0 – 31 supported
IOS XR started supporting in 3.6 in CRS & C12K
CRS-1 only supports setting QoS Group ID
C12k supports setting IP Precedence and/or QoS Group ID
ASR9k supports setting IP precedence and/or QoS Group ID and QOS policy can configured to match on modified IP prec/QOS Group only on egress.

Configuration and verification

In the following example, Router A learns routes from AS 200 and AS 100. QoS policy is applied to all packets that match the defined route maps. Any packets from Router A to AS 200 or AS 100 are sent the appropriate QoS policy.

Back to top

2. Sample Scenario

The show commands below follow the application and focus of the QPPB on Router "A" and then on the interface towards and connecting with

Router "B".

Back to top

3. Configuration steps

Configure QPPB to set QoS Group ID

Define Route Policy (routes to match)
Attach Route Policy to BGP instance
Apply QPPB to ingress interface

Configure Egress QoS Service Policy

Example: match on QoS Group ID, set precedence if matched

Route policy configuration

RPL configuration

route-policy qppb

if destination in (60.60.0.0/24) then

set qos-group 5

elseif source in (192.1.1.2) then

set qos-group 10

else

set qos-group 1

endif

pass

end-policy

QPPB ...

•Uses Route Policy Language (RPL)

•IP Destination based on BGP Route

•Source indicates source IP of the BGP Routes

•Also supports AS Path, Community Lists (not shown)

Back to top

4. Application to BGP and Interface

After the route policy configuration you need to apply it to the table policy to do the marking, right when the routes are being received.

This then also needs to be enforced on the interface with the right QOS configuration to leverage this QPPB setting and act upon that.

BGP

BGP configuration

router bgp 300

bgp router-id 19.19.19.19

address-family ipv4 unicast

table-policy qppb

!

neighbor 192.1.1.2

remote-as 200

address-family ipv4 unicast

route-policy pass-all in

route-policy pass-all out

!

Interface

Interface level configuration

interface TenG 0/0/0/0

ipv4 address 192.1.1.1 255.255.255.0

ipv4 bgp policy propagation input qos-group destination

keepalive disable

!

Back to top

5. Show commands and verification

The following commands give an overview how to verify the QPPB configuration

Using show cef

RP/0/RP0/CPU0:pacifica#show cef 60.60.0.0/24 detail

60.60.0.0/24, version 0, internal 0x40000001 (0x78419634) [1], 0x0 (0x0), 0x0 (0x0)

Prefix Len 24, traffic index 0, precedence routine (0)

QoS Group: 5, IP Precedence: 0

gateway array (0x0) reference count 100, flags 0x80200, source 3,

<snip>

Using show uidb

uIDB is a micro interface descriptor block, it is a data structure that represents the features that are applied to an interface.

In this case this command explains what features are applied to the interface specific to QPPB.

RP/0/RP0/CPU0:pacifica#show uidb data TenG 0/0/0/0 location 0/0/cpu0

<snip>

IPV4 QPPB En 0x1

IPV6 QPPB En 0x1

IPV6 QPPB Dst QOS Grp 0x0

IPV6 QPPB Dst Prec 0x1

IPV6 QPPB Src QOS Grp 0x1

IPV6 QPPB Src Prec 0x0

IPV4 QPPB Dst QOS Grp 0x0

IPV4 QPPB Dst Prec 0x1

IPV4 QPPB Src QOS Grp 0x0

IPV4 QPPB Src Prec 0x0

<snip>

Restrictions and limitations

This to the best of my knowledge

In Trident/Typhoon, due to limited space in prefix leaf, all traffic coming in that interface will undergo QPPB processing (IP Precedence and/or QOS Group) based on that interface setting. i.e there is no per prefix level validity checking for IP precedence or QOS group.

TRIDENT

The loopback decision is made in TOP Parse, so every packet coming into the interface, which has the “QOS group” policy configuration with QPPBenabled will be looped back.
Due to the loopback and extra cycles the performance will get impacted.
Also the trigger for loopback is the “QOS group” present in policy, if the user wants to have a policy to act only on the precedence value modified by qppb, user still needs to have a dummy “QOS group” in the policy for the loopback to happen to take effect of the new precedence value.
Only QOS format 1 and format 2 is supported in ingress QOS policy.

TYPHOON

The loopback decision is made in TOP Parse, so every packet coming into the interface, which has the “QOS group” policy configuration with QPPBenabled will be (pipeline)looped back.
Due to the pipeline loopback and extra cycles the performance may get impacted.
Also the trigger for loopback is the “QOS group” present in policy, if the user wants to have a policy to act only on the precedence value modified by qppb, user still needs to have a dummy “QOS group” in the policy for the loopback to happen to take effect of the new precedence value

Debugging QPPB

These debugging examples are not necessarily related to the configuration example above.

Check for QPPB values in route

RP/0/RSP0/CPU0:public-router#show route 120.120.120.120 de
Wed Feb 23 15:20:01.263 UTC
Routing entry for 120.120.120.0/24
Known via "bgp 500", distance 200, metric 0, type internal
Installed Feb 20 23:58:13.096 for 2d15h
Routing Descriptor Blocks
10.1.7.2, from 10.1.7.2
Route metric is 0
Label: None
Tunnel ID: None
Extended communities count: 0
Route version is 0x1 (1)
No local label
IP Precedence: 6
QoS Group ID: 5
Route Priority: RIB_PRIORITY_RECURSIVE (7) SVD Type RIB_SVD_TYPE_LOCAL
No advertising protos.

CEF command output for trident/typhoon

RP/0/RSP0/CPU0:public-router#show cef 120.120.120.120 hardware ingress location 0/0/cPU0 | inc QPPB

QPPBPrec: 6
QPPBQOS Group: 5

CEF Command output in THOR :

Hardware Extended Leaf Data:
fib_leaf_extension_length: 0 interface_receive: 0x0
traffic_index_valid: 0x0 qos_prec_valid: 0x1
qos_group_valid: 0x1 valid_source: 0x0
traffic_index: 0x0 nat_addr: 0x0
reserved: 0x0 qos_precedence: 0x6
qos_group: 0x5 peer_as_number: 0
path_list_ptr: 0x0
connected_intf_id: 0x0 ipsub_session_uidb: 0xffffffff
Path_list:
urpf loose flag: 0x0
List of interfaces:

Chk for QPPBvalue in uIDB

Trident

RP/0/RSP0/CPU0:public-router# show uidb data location 0/0/CPU0 gigabitEthernet 0/0/0/4 ingress | inc QPPB

Wed Feb 23 15:44:33.154 UTC
IPV6 QPPBSrc Lkup 0x0
IPV4 QPPBSrc Prec 0x1
IPV4 QPPBDst Prec 0x0
IPV4 QPPBSrc QOS Grp 0x0
IPV4 QPPBDst QOS Grp 0x0
IPV6 QPPBSrc Prec 0x0
IPV6 QPPBDst Prec 0x1
IPV6 QPPBSrc QOS Grp 0x0
IPV6 QPPBDst QOS Grp 0x0
IPV4 QPPBEn 0x1
IPV6 QPPBEn 0x1
IPV4 QPPBSrc Lkup 0x1

RP/0/RSP0/CPU0:public-router#show run interface gigabitEthernet 0/0/0/4
Wed Feb 23 15:44:52.313 UTC
interface GigabitEthernet0/0/0/4
service-policy output qppp-qos-pol
ipv4 bgp policy propagation input ip-precedence source
ipv4 address 12.12.12.1 255.255.255.0
ipv6 bgp policy propagation input ip-precedence destination
ipv6 address 12:12:12::1/64
!

Typhoon

RP/0/RSP0/CPU0:public-router#show uidb data location 0/2/cpu0 gigabitEthernet 0/2/0/0 ingress | inc QPPB

Wed Feb 23 15:47:32.436 UTC
IPV4 QPPBSrc Lookup En 0x0
IPV6 QPPBSrc Lookup En 0x1
IPV4 QPPBEn 0x1
IPV6 QPPBEn 0x1

RP/0/RSP0/CPU0:public-router#show uidb data location 0/2/cpu0 gigabitEthernet 0/2/0/0 ing-ex | inc QPPB

Wed Feb 23 15:49:19.118 UTC
IPV6 QPPBDst QOS Grp 0x0
IPV6 QPPBDst Prec 0x1
IPV6 QPPBSrc QOS Grp 0x1
IPV6 QPPBSrc Prec 0x0
IPV4 QPPBDst QOS Grp 0x0
IPV4 QPPBDst Prec 0x1
IPV4 QPPBSrc QOS Grp 0x0
IPV4 QPPBSrc Prec 0x0

Thor

(maybe a bit too deep for the general troubleshooting, but since I had this info I thought it might be nice to share in case, for those interested)

Another gotcha to this example below is a Gig interface in a SIP700, while this works, it is officially not supported, it is just for illustrational purposes to show how to feel inside info out to the surface and how to interpret it.

/*
* Flags bit field definitions for BGP_PA and QPPB.
*/
BF_CREATE(bgp_pa_flag_qppb_enabled, 6, 1); // QPPB
BF_CREATE(bgp_pa_flag_src_qos_group, 5, 1); // QPPB
BF_CREATE(bgp_pa_flag_dst_qos_group, 4, 1); // QPPB
BF_CREATE(bgp_pa_flag_src_prec, 3, 1); // QPPB
BF_CREATE(bgp_pa_flag_dst_prec, 2, 1); // QPPB
BF_CREATE(bgp_pa_flag_src, 1, 1); // BPG_PA
BF_CREATE(bgp_pa_flag_dst, 0, 1); // BPG_PA

uidb offset 780 for v4 and 2000 for v6 qppb:

RP/0/RSP0/CPU0:public-router#show controllers pse qfp infrastructure gigabitEthernet 0/1/1/4 input config raw loc 0/1/CPU0 | inc 780

Wed Feb 23 15:52:21.259 UTC
00000780 00 00 00 48 00 00 00 00
00001780 00 00 00 00 00 00 00 00
00002780 00 00 00 00 89 87 54 00

RP/0/RSP0/CPU0:public-router#show controllers pse qfp infrastructure gigabitEthernet 0/1/1/4 input config raw loc 0/1/CPU0 | inc 2000

Wed Feb 23 15:53:49.236 UTC
00002000 00 00 00 48 e7 ff 36 f0

RP/0/RSP0/CPU0:public-router#show run interface gigabitEthernet 0/1/1/4
Wed Feb 23 16:04:34.892 UTC
interface GigabitEthernet0/1/1/4
description TGN PORT 7 3
service-policy output qppp-qos-pol
ipv4 bgp policy propagation input ip-precedence source
ipv4 address 10.1.7.1 255.255.255.0
ipv6 bgp policy propagation input ip-precedence source
ipv6 address 10:1:7::1/64
ipv6 enable
!

Check if packets are loopbacking due to ingress QOS policy

(both trident and typhoon)

RP/0/RSP0/CPU0:ASR-PE3#show uidb data location 0/0/cpu0 gigabitEthernet 0/0/0/4.1 ingress | inc QOS

Mon Feb 20 10:57:22.781 UTC

QOS Enable 0x1

QOS Inherit Parent 0x0

QOS ID 0x2004

AFMON QOS ID 0x0

IPV4 QPPBSrc QOS Grp 0x1

IPV4 QPPBDst QOS Grp 0x0

IPV6 QPPBSrc QOS Grp 0x0

IPV6 QPPBDst QOS Grp 0x0

QOS grp present in policy 0x1

QOS Format 0x1

Verification of the NP counters, we need to see loopbacks happening (or feedback in PXF or Recirculation in 7600 terminology)

RP/0/RSP0/CPU0:ASR-PE3#show controllers np counters np3 location 0/0/cPU0

Mon Feb 20 11:08:49.262 UTC

Node: 0/0/CPU0:

Show global stats counters for NP3, revision v3

Read 33 non-zero NP counters:

Offset Counter FrameValue Rate (pps)

22 PARSE_ENET_RECEIVE_CNT 6861706 45574

23 PARSE_FABRIC_RECEIVE_CNT 6927056 45574

24 PARSE_LOOPBACK_RECEIVE_CNT 6818799 45573

29 MODIFY_FABRIC_TRANSMIT_CNT 6709882 45574

30 MODIFY_ENET_TRANSMIT_CNT 6929272 45574

34 RESOLVE_EGRESS_DROP_CNT 5 0

38 MODIFY_MCAST_FLD_LOOPBACK_CNT 6818940 45574

RP/0/RSP0/CPU0:ASR-PE3#show uidb data location 0/0/cpu0 gigabitEthernet 0/0/0/4.1 ingress | inc QOSQOS Enable 0x1QOS Inherit Parent 0x0QOS ID 0x2004AFMON QOS ID 0x0IPV4 QPPBSrc QOS Grp 0x1IPV4 QPPBDst QOS Grp 0x0IPV6 QPPBSrc QOS Grp 0x0IPV6 QPPBDst QOS Grp 0x0 QOS grp present in policy 0x1 QOS Format 0x1

Extra Q&A

If the user wants to have a policy to act only on the precedence value modified by qppb, user still needs to have a dummy “QOS group” in the policy for the loopback to happen to take effect of the new precedence value. This requires to configure something like this:

router bgp 1
address-family ipv4 unicast
   table-policy QPPB-PREC

route-policy QPPB-PREC
   if (community matches-any GOLD) then
     set ip-precedence 5
     set qos-group 1 <== dummy qos-group
   endif
end-policy
!

And at the interface level I should have both ip-precedence and qos-group:

interface TenGigE0/3/0/5
cdp
mtu 9194
ipv4 bgp policy propagation input ip-precedence destination qos-group destination
ipv4 address 192.1.22.14 255.255.255.254
load-interval 30
!

Although in the ingress QoS policy I only match on ip precedence being set by QPPB.

Answer:

You need to have “match on qos group” in input QOS policy and not on the route policy.

See the example below

router bgp 1

address-family ipv4 unicast
   table-policy QPPB-PREC

route-policy QPPB-PREC
   if (community matches-any GOLD) then
     set ip-precedence 5
   endif
end-policy
!

QOS policy will look like this

class-map match-any qos-grp-dummy
match qos-group 20
end-class-map
!
class-map match-any qos-prec
match precedence 5
end-class-map
!

policy-map qppb-input-pref
class qos-grp-dummy               <--------dummy “qos group”
   police rate percent 20
    exceed-action drop
   !
!
class qos-prec
   police rate percent 10
    exceed-action drop
   !
!
class class-default
!
end-policy-map
!

And at the interface level you don’t need to have qos-group qppb

interface TenGigE0/3/0/5
cdp
mtu 9194
ipv4 bgp policy propagation input ip-precedence destination
service-policy input qppb-input-pref
ipv4 address 192.1.22.14 255.255.255.254
load-interval 30

!

Resolution

This section provides the answer or resolution to the problem provided in the "Core Issue" section.

Back to top

6.

Related Information

Xander Thuijs CCIE #6775

Principal Engineer IOS-XR; ASR9000, CRS and NCS6K

Back to top

elnur.mammadov · ‎08-15-2014

Hello Xander,

I hope you are doing alright!

is it possible to clasiify ip subscriber traffic? I mean:

ipv4 bgp policy propagation input qos-group destination

it is not available under dynamic template, I've applied it to the parent port-bundle but it does not seem to classify incomming traffic

Thank you

Regards

Elnur

p.s. it works otherwise, I've checked it over regular L3 interface.

xthuijs · ‎08-15-2014

qppb is mainly useful on the ingress side for classification and generally (yeah...) BGP doesnt run over the subscriber interface (requires static adds etc), so we didnt make it a priority to enable QPPB on the subscriber interface today.

hence it not being available/visible on the dyn tpl.

cheers

xander

Pok LY · ‎12-29-2014

Hi Alexander,

Can you provide simple diagram showing how the traffic flow ? It is hard to understand the interface direction of it's flow.

Regards, Pok.

xthuijs · ‎12-30-2014

I attached 2 pictures, one with a sample config with the bits and pieces and the other one describing a few packets and what would happen to them.

hopefully this visualizes it for you.

cheers!

xander

Pok LY · ‎01-01-2015

Hi Xander,

Base on my configuration Client download is matched to the policy and policing applied correctly but when Client upload the policy didn't match and it fall into class INTERNET please advised my configuration.

Regards, Pok.

xthuijs · ‎01-02-2015

I think your config is ok Pok, we could do a few verifications:

- check whether the qos-group is correctly set on ingress:

apply a qos policy out on the te0/x interface to see if the egress shaper/policer works properly

- if that test succeeds it is likely an issue with the inbound matching on the set qos group for which we need to verify if the packets are receiving the loopback as required. The tests for that are above described to check the UIDB programming and the packet counters on the NP for the MDF LOOP should increment.

regards

xander

mauricio.andrade81 · ‎03-30-2015

Hi Xander,

It's possible in XR apply QPPB on egress for VPN MPSL? In regular IOS this is a limitant for LFIB lookup en egress (PE->CE) because QPPB rely CEF.

xthuijs · ‎03-31-2015

hi mauricio,

you'd want the packets to be marked with a special qos group on ingress via qppb and use that qos group on egress in a regular policy-map.

The QPPB action(s) are only in effect in the ingress direction.

cheers!

xander

mauricio.andrade81 · ‎03-31-2015

Thanks Alexander, maybe leaking MPLS L3 to VRF lite solve the egress match for qos-group.

regards,

mauricio.andrade81 · ‎04-07-2015

Hi Xander, Can I apply qppb in PWHE interfaces ingress and egress direction?

thanks,

xthuijs · ‎04-07-2015

that is a great question mauricio, I have never tried that to be honest. I am verifying with our test department whether this is part of the original delivery and whether this is meant to be working, somehow it sounds "dangerous" to me...

QPPB requires a feedback loop and PWHE isnt the smallest feature either, hence...

will confirm and get back.

cheers

xander

xthuijs · ‎04-07-2015

oh sorry let me make sure you have it straight in your mind:

qppb means marking based on a (bgp) prefix.

then you obviously want to leverage that marking in a policy. using that remarked value can be done on either an ingress policy (requires feedback loop) or on an egress policy (easier/liter on the fwd path).

But the marking by QPPB is an ingress feature only.

making sense? :)

cheers!

xander

mauricio.andrade81 · ‎04-08-2015

Ok I understund, I need route lookup based in BGP inside the VRF.

thanks,

adrian.taranu · ‎05-02-2015

Hi Xander,

So under the interface, we have "ipv4 bgp policy propagation input". Can you please tell me why we do not have an "ipv4 bgp policy propagation output" ? Thank you!

xthuijs · ‎05-03-2015

hi adrian,

till date there hasnt been a true requirement to implement this. What qppb really is, is marking the traffic based on some bgp parameters. this marking can then be used in a policy for policing or shaping etc.

So technically, if you want to do something in the egress direction, you could have already marked it on ingress, right, so technically there is really no need to tag it on egress. Doing it egress would also impact some forwarding capability: we'd need to tag the traffic, recirculate and apply the qos policy on it, for that reason it is better to tag it on ingress, so we save that recirc/feedback on egress.

cheers

xander