Re: ASR 9000 QoS police on BVI

Amirmahdi.M · ‎05-28-2024

Hi everyone

We have an asr9k-x64 (V7.9.21) with an A9K-48X10GE-1G-TR linecard.

we have bundled 8 TenG Links to BE5 with the following configs for each vlan:

interface Bundle-Ether5.10 l2transport
description VLAN10
encapsulation dot1q 10
rewrite ingress tag pop 1 symmetric

!

l2vpn
bridge group VLAN10
bridge-domain VLAN10
interface Bundle-Ether5.10
routed interface BVI10

!

interface BVI10
description VLAN10
service-policy output 35GB
ipv4 address 10.255.255.253 255.255.255.252

!

policy-map 35GB
class class-default
police rate 35000 mbps
conform-action transmit
violate-action drop

when i apply the service-policy on the interface and enter "show policy-map interface BVI10 output" it doesnt start exceeding the traffic until the output rate reaches about 46Gbps.

I researched a bit and guess the cause of the problem is bundling 10Gb links from different NPs.

this the output of "show controllers np ports all":

Node: 0/0/CPU0:
----------------------------------------------------------------

NP Bridge Fia Ports
-- ------ --- ---------------------------------------------------
0 -- 0 TenGigE0/0/0/0 - TenGigE0/0/0/23
1 -- 1 TenGigE0/0/0/24 - TenGigE0/0/0/47

my Bundle-Ether5 contains 4 interfaces from NP0 and 4 interfaces from NP1

Also This the output of 'show qos interface BV10 output location 0/0/CPU0 ':

Interface: BVI10 
-------------------------------------------------------
Interface: BVI10 NP0 output 
Bandwidth configured: 100000000 kbps Bandwidth programed: 100000000 kbps
ANCP user configured: 0 kbps ANCP programed in HW: 0 kbps
Port Shaper programed in HW: 0 kbps 
Policy: 35GB Total number of classes: 1
----------------------------------------------------------------------
Level: 0 Policy: 35GB Class: class-default
QueueID: 0 (Port Default)
Policer Profile: 75 (Single)
Conform: 35000000 kbps (35000 mbps) Burst: 437500000 bytes (0 Default)
Child Policer Conform: TX
Child Policer Exceed: DROP
Child Policer Violate: DROP
----------------------------------------------------------------------


Interface: BVI10 
-------------------------------------------------------
Interface: BVI10 NP1 output 
Bandwidth configured: 100000000 kbps Bandwidth programed: 100000000 kbps
ANCP user configured: 0 kbps ANCP programed in HW: 0 kbps
Port Shaper programed in HW: 0 kbps 
Policy: 35GB Total number of classes: 1
----------------------------------------------------------------------
Level: 0 Policy: 35GB Class: class-default
QueueID: 0 (Port Default)
Policer Profile: 75 (Single)
Conform: 35000000 kbps (35000 mbps) Burst: 437500000 bytes (0 Default)
Child Policer Conform: TX
Child Policer Exceed: DROP
Child Policer Violate: DROP
----------------------------------------------------------------------

Is my problem of not policing correctly related to the NPs? how can i solve it?

Thanks

Amir

Amirmahdi.M · ‎05-28-2024

I'd be glad if you can help me.

@Harold Ritter

@MHM Cisco World

@Joseph W. Doherty

Joseph W. Doherty · ‎05-28-2024

Sorry, non-existent experience on ASR9Ks, and very, very little with IOS-XR.

smilstea · ‎05-28-2024

A few things. If the policer was on the bundle then it would get replicated to each PHY interface (bundle member) unless you use the hw-module aggregate mode.
Because the policer is applied on the BVI itself the policy gets downloaded to each NP, not PHY, so for example you have 4 ports on NP 0 and 4 ports on NP 1 then you would have 35Gbps on each NP and the 4 ports on the NP share that resource.

"when configuring any QoS on a bundle interface the policy is applied to all the member ports of the bundle. This has the caveat that for policers and shapers and the bandwidth command, the configured rate is not a total aggregate. Each LC NP allows its member interface to run up to the configured rate."

For bundles you can change the mode: hw-module all qos-mode bundle-qos-aggregate-mode

This document doesn't cover it that well unfortunately:
https://www.cisco.com/c/en/us/td/docs/routers/asr9000/software/asr9k-r7-5/qos/b-qos-cg-asr9k-75x/config-mod-qos-congestion-management.html#concept_33941AEA23AF430CA56E78BC4B2CC18C
QoS on BVI
QoS support on BVI will allow the application of the policy map directly on the virtual interface. This will enable aggregate policing and marking on the virtual interface. The policy can be applied on either the ingress or egress side of the BVI to mark and police traffic going to and from the bridge domain.
These are the QoS features supported on BVI QoS policy:

* Classification
* Policing (hierarchical, conform-aware, conditional marking)
* Marking

>From internal documentation "ASR9k BVI QoS shall support aggregate policer rate on a per NP basis"

Thanks,
Sam

Amirmahdi.M · ‎05-28-2024

Hi @smilstea

I deeply appreciate your reply.

So based on our info in my scenario if i have multiple NPs, the BVI interface will be (CIR*NPcount) of the policy i set on it.

Is there any configuration that can aggregate policing on the LC NPs?

if i modify my bundle members so all the traffic of a bridge group (BDI) goes through one LC NP, will the policy work as there is only one NP and police the traffic as i want it?

Thanks

Amir

smilstea · ‎06-10-2024

I have not tried this combination to be able to know for certain. I was doing some research and haven't been able to find much on this, I am going to ask around some more to see if anybody knows for certain.

Sam

smilstea · ‎06-11-2024

Please see attached PPT slides 20-39, especially slide 30.

When packets are routed into a bridge domain, both the ingress and egress L3 features are executed on the ingress NP.

Alternative design could be:

If there are only ACs in the BD: apply the QoS policy on ACs
If there are only PWs in the BD:

use PWHE to determine which interfaces/NPs are eligible to receive/send traffic from/into a specific PW.
Configure the PW-Ether L2 transport (sub-)interface as an AC in the BD.
Configure the QoS policy on the PW-Ether L2 transport (sub-)interface

If there are PWs and ACs in the BD: combine the two above

Sam

Amirmahdi.M · ‎06-02-2024

Hello again

looks like even by changing the traffic flow so the traffic of the BVI goes through the ports of one NP, still doesnt solve my problem. (used ports Te0/0/0/16 - Te0/0/0/19)

no matter what i do, the traffic is divided between the NPs and polices works independently on each NP.

this it the output of "show qos summary police interface BVI10 output location 0/0/CPU0"

Legend:
=======
1. Policer ID is displayed in HEX.
2. A '*' against the counter means the action is drop.
3. Conform displays match counter for non-policer leaf.

Policy:Class                       PoliceID     Conform      Exceed     Violate
BVI10 NP0
35GB                                                    
 :class-default                    2097293     502479573261           0*          0*
BVI10 NP1
35GB                                                    
 :class-default                    2097293     1505657454866  1096814115*          0*

is there any workaround?

Thanks

Amir

@smilstea

@Harold Ritter

@MHM Cisco World

smilstea · ‎06-04-2024

I did some more research and it looks like it could be one of three things.

1. As I mentioned earlier, bundles will multiply the policer by how many members there are, so with 4 members it would be 4x the configured rate.
2. You need to use a PWHE interface and not a BVI. I'm still reading into why that is
3. Its not supported on the LC you have

Can you share the LC model and code you are using as well?

Thanks,
Sam

Amirmahdi.M · ‎06-04-2024

It is an Tomahawk-Based A9K-48X10GE-1G-TR LC

based on my research it doesnt multiply by the bundle members. it multiplies by the number of LC NP number.

I also tried the police on input direction and this the result:

1. if the trafiic is coming from one NP, it policies at the configured rate.

2. if the traffic comes from ports of multiple NPs (bundle) it polices at double the configured rate.

but as for the output direction, it doesnt matter which NP the traffic goes to, the police is applied indepently on both NPs and based on the policy's confromed counter, it is balancing the conform-rate on both NPs. (i posted the output of "show qos summary police" on my last reply)

Thanks,

Amir