Re: ASR9K BNG Bundle-Ether interface policers duplication for pQoS services

peter.ivanov · ‎07-26-2018

Good Day!

We terminate PPPoE sessions on A9K-4T16GE-SE line card. We apply pQoS service for each PPPoE session.

RP/0/RSP0/CPU0:mskl05#show policy-map applied interface TenGigE0/2/0/16.4001.pppoe4228

Wed Jul 25 08:57:13.220 UTC

Input policy-map applied to TenGigE0/2/0/16.4001.pppoe4228:

policy-map __sub_ffffffbf5631ffffffaa

class cmSrvRetail_LocalAccess_In

police rate 100000 kbps

!

class cmSrvRetail_InternetAccess_Night

police rate 45000 kbps

!

class cmSrvRetail_InternetAccess_Day

police rate 15000 kbps

!

class class-default

!

Output policy-map applied to TenGigE0/2/0/16.4001.pppoe4228:

policy-map __sub_21727fffffff94

class cmSrvRetail_LocalAccess_Out

police rate 100000 kbps

!

class cmSrvRetail_InternetAccess_Night

police rate 45000 kbps

!

class cmSrvRetail_InternetAccess_Day

police rate 15000 kbps

!

class class-default

!

This line card support 32K sessions.

RP/0/RSP0/CPU0:mskl05#show subscriber session all summary

Wed Jul 25 08:52:45.741 UTC

Session Summary Information for all nodes

Type PPPoE IPSub IPSub

(DHCP) (PKT)

==== ===== ====== =====

Session Counts by State:

initializing 0 0 0

connecting 0 0 0

connected 141 0 0

activated 32768 0 0

idle 0 0 0

disconnecting 0 0 0

end 0 0 0

Total: 32909 0 0

Session Counts by Address-Family/LAC:

in progress 141 0 0

ipv4-only 32768 0 0

ipv6-only 0 0 0

dual-partial-up 0 0 0

dual-up 0 0 0

lac 0 0 0

Total: 32909 0 0

Within each pQoS service is used 3 ingress and 3 egress policers. That is why the total policers is 196608.

RP/0/RSP0/CPU0:mskl05#show qoshal resource summary location 0/2/CPU0

Wed Jul 25 08:58:22.676 UTC

TY Options argc:5

nphal_show_chk -p 2113 resource summary

Done

-1

Counters: X(Y/Z): X -> Resources Allocated in HW

Y -> Resource Allocated in SW

Z -> Refcount of each resource

Sanity Check: X==Y && Z >= X

:X (Y): X -> Resource Allocated in HW

Y -> Resource Allocated in SW

Client - 0, General - Not any Specific Client

SW and RefCount for Entities are accounted in Client 0(general)

Only HW count for Entities is per client

NP 0

===============================================================

CLIENT : None-General

Policy Instances: Ingress 0 Egress 0 Total: 0

Entities: (L4 level: Queues)

Level Chunk 0 Chunk 1 Chunk 2 Chunk 3

L4 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L3(8Q) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L2 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L1 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

Policers: Internal 0(0) Regular 0(0) Parent 0(0) Child 0(0) Total 0(0)

CLIENT : Pre-IFIB

Policy Instances: Ingress 6 Egress 0 Total: 6

Entities: (L4 level: Queues)

Level Chunk 0 Chunk 1 Chunk 2 Chunk 3

L4 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L3(8Q) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L2 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L1 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

Policers: Internal 0(0) Regular 280(280) Parent 0(0) Child 0(0) Total 280(280)

CLIENT : QoS-EA

Policy Instances: Ingress 32768 Egress 32768 Total: 65536

Entities: (L4 level: Queues)

Level Chunk 0 Chunk 1 Chunk 2 Chunk 3

L4 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L3(8Q) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L3(16Q) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L2 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L1 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

Policers: Internal 2046(2046) Regular 194562(194562) Parent 0(0) Child 0(0) Total 196608(196608)

The problem is noticed when we use Bundle-Ether interface for sessions termination. The whole set of policers are applied for each physical port inside Bundle-Ether interface. It means that the quantity of sessions is reduced several times depending on the number of ports. For example, Bundle-Ether interface which consist of 4 ports the maximum sessions is 8K (4 times less).

RP/0/RSP0/CPU0:mskl05#show subscriber session all summary

Wed Jul 25 14:53:16.414 UTC

Session Summary Information for all nodes

Type PPPoE IPSub IPSub

(DHCP) (PKT)

==== ===== ====== =====

Session Counts by State:

initializing 0 0 0

connecting 0 0 0

connected 200 0 0

activated 8192 0 0

idle 0 0 0

disconnecting 0 0 0

end 0 0 0

Total: 8392 0 0

Session Counts by Address-Family/LAC:

in progress 200 0 0

ipv4-only 8192 0 0

ipv6-only 0 0 0

dual-partial-up 0 0 0

dual-up 0 0 0

lac 0 0 0

Total: 8392 0 0

Applied pQoS service for PPPoE session on Bundle-Ether interface.

RP/0/RSP0/CPU0:mskl05#show policy-map applied interface Bundle-Ether7.4001.pppoe8

Wed Jul 25 14:58:47.160 UTC

Input policy-map applied to Bundle-Ether7.4001.pppoe8:

policy-map __sub_ffffffbf5631ffffffaa

class cmSrvRetail_LocalAccess_In

police rate 100000 kbps

!

class cmSrvRetail_InternetAccess_Night

police rate 45000 kbps

!

class cmSrvRetail_InternetAccess_Day

police rate 15000 kbps

!

class class-default

!

Output policy-map applied to Bundle-Ether7.4001.pppoe8:

policy-map __sub_21727fffffff94

class cmSrvRetail_LocalAccess_Out

police rate 100000 kbps

!

class cmSrvRetail_InternetAccess_Night

police rate 45000 kbps

!

class cmSrvRetail_InternetAccess_Day

police rate 15000 kbps

!

class class-default

!

For 8K sessions total policers is 196608 (4 times more).

RP/0/RSP0/CPU0:mskl05#show qoshal resource summary location 0/2/CPU0

Wed Jul 25 15:00:00.962 UTC

TY Options argc:5

nphal_show_chk -p 2113 resource summary

Done

-1

Counters: X(Y/Z): X -> Resources Allocated in HW

Y -> Resource Allocated in SW

Z -> Refcount of each resource

Sanity Check: X==Y && Z >= X

:X (Y): X -> Resource Allocated in HW

Y -> Resource Allocated in SW

Client - 0, General - Not any Specific Client

SW and RefCount for Entities are accounted in Client 0(general)

Only HW count for Entities is per client

NP 0

===============================================================

CLIENT : None-General

Policy Instances: Ingress 0 Egress 0 Total: 0

Entities: (L4 level: Queues)

Level Chunk 0 Chunk 1 Chunk 2 Chunk 3

L4 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L3(8Q) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L2 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L1 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

Policers: Internal 0(0) Regular 0(0) Parent 0(0) Child 0(0) Total 0(0)

CLIENT : Pre-IFIB

Policy Instances: Ingress 6 Egress 0 Total: 6

Entities: (L4 level: Queues)

Level Chunk 0 Chunk 1 Chunk 2 Chunk 3

L4 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L3(8Q) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L2 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L1 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

Policers: Internal 0(0) Regular 280(280) Parent 0(0) Child 0(0) Total 280(280)

CLIENT : QoS-EA

Policy Instances: Ingress 32768 Egress 32769 Total: 65537

Entities: (L4 level: Queues)

Level Chunk 0 Chunk 1 Chunk 2 Chunk 3

L4 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L3(8Q) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L3(16Q) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L2 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L1 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

Policers: Internal 2043(2043) Regular 194568(194568) Parent 0(0) Child 0(0) Total 196611(196611)

Is it possible to change this behavior for Bundle-Ether interface? We would like to have 32K sessions for Bundle-Ether interface. And it is critical moment because sessions on Bundle-Ether interface is RP sessions without any functional restriction. As we know for LC sessions there are some restrictions. Also Bundle-Ether interface provide load-balance and redundancy in compare with physical ports.

ASR9K Software specification

RP/0/RSP0/CPU0:mskl05#show install

Thu Jul 26 12:26:23.921 UTC

Secure Domain Router: Owner

Node 0/RSP0/CPU0 [RP] [SDR: Owner]

Boot Device: disk0:

Boot Image: /disk0/asr9k-os-mbi-6.1.4/0x100305/mbiasr9k-rsp3.vm

Active Packages:

disk0:asr9k-services-infra-6.1.4

disk0:asr9k-video-px-6.1.4

disk0:asr9k-services-px-6.1.4

disk0:asr9k-optic-px-6.1.4

disk0:asr9k-mpls-px-6.1.4

disk0:asr9k-mini-px-6.1.4

disk0:asr9k-mgbl-px-6.1.4

disk0:asr9k-mcast-px-6.1.4

disk0:asr9k-li-px-6.1.4

disk0:asr9k-k9sec-px-6.1.4

disk0:asr9k-fpd-px-6.1.4

disk0:asr9k-doc-px-6.1.4

disk0:asr9k-bng-px-6.1.4

Node 0/2/CPU0 [LC] [SDR: Owner]

Boot Device: mem:

Boot Image: /disk0/asr9k-os-mbi-6.1.4/lc/mbiasr9k-lc.vm

Active Packages:

disk0:asr9k-services-infra-6.1.4

disk0:asr9k-video-px-6.1.4

disk0:asr9k-services-px-6.1.4

disk0:asr9k-optic-px-6.1.4

disk0:asr9k-mpls-px-6.1.4

disk0:asr9k-mini-px-6.1.4

disk0:asr9k-mcast-px-6.1.4

disk0:asr9k-li-px-6.1.4

disk0:asr9k-k9sec-px-6.1.4

disk0:asr9k-bng-px-6.1.4

Aleksandar Vidakovic · ‎07-26-2018

hi Peter,

this is per design. On distributed platforms all bundle ethernet control structures are replicated over all bundle members. If you are not applying any queueing action on subscriber interfaces (which seems to be the case based on the outputs you have shared), you can go beyond 32k subscribers per NP, all the way until you run out of uidbs (microcode interface descriptors). To see how many uidbs are in use and what's the max available use either "sh controllers np struct EGR-UIDB summary location <location>" or "sh uidb index summary location <location>". This is not something we recommend, mostly because once you approach the max uidb scale, the dynamics of uidb allocation/release (as subscriber sessions are flapping) may prevent you from reaching the full limit. So you can exceed 32k, but don't go all the way. :) The scale per LC remains 64k subscribers, you can't go beyond that.

If you need high scale with a lot of throughput (i.e. more than two bundle members), it would be better to consider a LC with more NPs or go for upgrade to 6.3.3 eXR (requires Tomahawk), where the supported scale is higher.

I'd also suggest upgrading to 6.2.3 + latest Service Pack or SMUs. If not already doing that, use CSM Server to optimise the list for your deployment and manage the actual installation.

hope this helps,

Aleksandar

peter.ivanov · ‎07-26-2018

Aleksandar, thanks a lot for your answer!

But in our case the problem not in uidbs. For the maximum 8K sessions on Bundle-Ether interface (4x10G) we have 8000 uidbs with max available 65536.

RP/0/RSP0/CPU0:mskl05#show controllers np struct EGR-UIDB summary location 0/2/CPU0

Thu Jul 26 16:00:53.858 UTC

Node: 0/2/CPU0:

----------------------------------------------------------------

NP: 0 Struct 10: EGR_UIDB

Struct is a PHYSICAL entity

Reserved Entries: 0, Used Entries: 8056, Max Entries: 65536

There is any way for particular bundle Ethernet interface NOT replicate over all bundle members control structures? In our case it is policers.

Aleksandar Vidakovic · ‎07-26-2018

Can you attach the full output of "show bundle <bundle-int>", "sh controllers np ports all ...", "sh qoshal resource summary ..." and "sh qos interface <subs-interface> output member <member> ..."? If you're stuck at 8k, it means there's some traffic manager resource in use. With queuing actions on subscriber interfaces you are limited to 32k subscribers, with max 8k per TM chunk.

Replicating all structures over all bundle members is at the core of BE implementation. Many of the features wouldn't work as expected should we try to change that. Let's find an alternative way to unleash the full potential of the asr9k BNG solution.

/Aleksandar

peter.ivanov · ‎07-27-2018

Aleksandar, Good day!

Sending output of the commands.

RP/0/RSP0/CPU0:mskl05#show bundle bundle-ether 7

Fri Jul 27 07:24:27.899 UTC

Bundle-Ether7

Status: Up

Local links <active/standby/configured>: 4 / 0 / 4

Local bandwidth <effective/available>: 40000000 (40000000) kbps

MAC address (source): d867.d959.deb7 (Chassis pool)

Inter-chassis link: No

Minimum active links / bandwidth: 1 / 1 kbps

Maximum active links: 64

Wait while timer: 2000 ms

Load balancing:

Link order signaling: Not configured

Hash type: Default

Locality threshold: None

LACP: Operational

Flap suppression timer: Off

Cisco extensions: Disabled

Non-revertive: Disabled

mLACP: Not configured

IPv4 BFD: Not configured

IPv6 BFD: Not configured

Port Device State Port ID B/W, kbps

-------------------- --------------- ----------- -------------- ----------

Te0/2/0/16 Local Active 0x8000, 0x000a 10000000

Link is Active

Te0/2/0/17 Local Active 0x8000, 0x0009 10000000

Link is Active

Te0/2/0/18 Local Active 0x8000, 0x0008 10000000

Link is Active

Te0/2/0/19 Local Active 0x8000, 0x0007 10000000

Link is Active

RP/0/RSP0/CPU0:mskl05#show controllers np ports all location 0/2/CPU0

Fri Jul 27 07:27:47.792 UTC

Node: 0/2/CPU0:

----------------------------------------------------------------

NP Bridge Fia Ports

-- ------ --- ---------------------------------------------------

0 -- 0 GigabitEthernet0/2/0/0 - GigabitEthernet0/2/0/15, TenGigE0/2/0/16 - TenGigE0/2/0/19

RP/0/RSP0/CPU0:mskl05#show qoshal resource summary location 0/2/CPU0

Fri Jul 27 08:09:24.125 UTC

TY Options argc:5

nphal_show_chk -p 2113 resource summary

Done

-1

Counters: X(Y/Z): X -> Resources Allocated in HW

Y -> Resource Allocated in SW

Z -> Refcount of each resource

Sanity Check: X==Y && Z >= X

:X (Y): X -> Resource Allocated in HW

Y -> Resource Allocated in SW

Client - 0, General - Not any Specific Client

SW and RefCount for Entities are accounted in Client 0(general)

Only HW count for Entities is per client

NP 0

===============================================================

CLIENT : None-General

Policy Instances: Ingress 0 Egress 0 Total: 0

Entities: (L4 level: Queues)

Level Chunk 0 Chunk 1 Chunk 2 Chunk 3

L4 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L3(8Q) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L2 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L1 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

Policers: Internal 0(0) Regular 0(0) Parent 0(0) Child 0(0) Total 0(0)

CLIENT : Pre-IFIB

Policy Instances: Ingress 6 Egress 0 Total: 6

Entities: (L4 level: Queues)

Level Chunk 0 Chunk 1 Chunk 2 Chunk 3

L4 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L3(8Q) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L2 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L1 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

Policers: Internal 0(0) Regular 280(280) Parent 0(0) Child 0(0) Total 280(280)

CLIENT : QoS-EA

Policy Instances: Ingress 32768 Egress 32769 Total: 65537

Entities: (L4 level: Queues)

Level Chunk 0 Chunk 1 Chunk 2 Chunk 3

L4 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L3(8Q) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L3(16Q) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L2 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L1 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

Policers: Internal 2043(2043) Regular 194568(194568) Parent 0(0) Child 0(0) Total 196611(196611)

CLIENT : L2 Fib

Policy Instances: Ingress 2 Egress 0 Total: 2

Entities: (L4 level: Queues)

Level Chunk 0 Chunk 1 Chunk 2 Chunk 3

L4 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L3(8Q) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L2 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L1 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

Policers: Internal 4(4) Regular 0(0) Parent 0(0) Child 0(0) Total 4(4)

CLIENT : Interflex

Policy Instances: Ingress 0 Egress 0 Total: 0

Entities: (L4 level: Queues)

Level Chunk 0 Chunk 1 Chunk 2 Chunk 3

L4 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L3(8Q) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L2 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L1 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

Policers: Internal 0(0) Regular 0(0) Parent 0(0) Child 0(0) Total 0(0)

CLIENT : Span

Policy Instances: Ingress 0 Egress 0 Total: 0

Entities: (L4 level: Queues)

Level Chunk 0 Chunk 1 Chunk 2 Chunk 3

L4 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L3(8Q) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L2 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L1 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

Policers: Internal 0(0) Regular 0(0) Parent 0(0) Child 0(0) Total 0(0)

CLIENT : Platform Manager

Policy Instances: Ingress 20 Egress 20 Total: 40

Entities: (L4 level: Queues)

Level Chunk 0 Chunk 1 Chunk 2 Chunk 3

L4 40( 40/ 40) 40( 40/ 40) 40( 40/ 40) 40( 40/ 40)

L3(8Q) 10( 10/ 10) 10( 10/ 10) 10( 10/ 10) 10( 10/ 10)

L2 10( 10/ 10) 10( 10/ 10) 10( 10/ 10) 10( 10/ 10)

L1 40( 40/ 40) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

Policers: Internal 0(0) Regular 0(0) Parent 0(0) Child 0(0) Total 0(0)

CLIENT : IRB

Policy Instances: Ingress 0 Egress 0 Total: 0

Entities: (L4 level: Queues)

Level Chunk 0 Chunk 1 Chunk 2 Chunk 3

L4 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L3(8Q) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L2 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L1 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

Policers: Internal 0(0) Regular 0(0) Parent 0(0) Child 0(0) Total 0(0)

CLIENT : Internal

Policy Instances: Ingress 5 Egress 0 Total: 5

Entities: (L4 level: Queues)

Level Chunk 0 Chunk 1 Chunk 2 Chunk 3

L4 16( 16/ 16) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L3(8Q) 4( 4/ 4) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L2 3( 3/ 3) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L1 3( 3/ 3) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

Policers: Internal 1(1) Regular 0(0) Parent 0(0) Child 0(0) Total 1(1)

CLIENT : SVC

Policy Instances: Ingress 0 Egress 0 Total: 0

Entities: (L4 level: Queues)

Level Chunk 0 Chunk 1 Chunk 2 Chunk 3

L4 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L3(8Q) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L2 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L1 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

Policers: Internal 0(0) Regular 0(0) Parent 0(0) Child 0(0) Total 0(0)

CLIENT : Experimental 2

Policy Instances: Ingress 0 Egress 0 Total: 0

Entities: (L4 level: Queues)

Level Chunk 0 Chunk 1 Chunk 2 Chunk 3

L4 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L3(8Q) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L2 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L1 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

Policers: Internal 0(0) Regular 0(0) Parent 0(0) Child 0(0) Total 0(0)

CLIENT : SCOPP

Policy Instances: Ingress 0 Egress 0 Total: 0

Entities: (L4 level: Queues)

Level Chunk 0 Chunk 1 Chunk 2 Chunk 3

L4 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L3(8Q) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L2 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L1 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

Policers: Internal 0(0) Regular 0(0) Parent 0(0) Child 0(0) Total 0(0)

CLIENT : CLUSTER

Policy Instances: Ingress 0 Egress 0 Total: 0

Entities: (L4 level: Queues)

Level Chunk 0 Chunk 1 Chunk 2 Chunk 3

L4 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L3(8Q) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L2 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L1 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

Policers: Internal 0(0) Regular 0(0) Parent 0(0) Child 0(0) Total 0(0)

CLIENT : QoS-Virtual-IF

Policy Instances: Ingress 0 Egress 0 Total: 0

Entities: (L4 level: Queues)

Level Chunk 0 Chunk 1 Chunk 2 Chunk 3

L4 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L3(8Q) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L2 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L1 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

Policers: Internal 0(0) Regular 0(0) Parent 0(0) Child 0(0) Total 0(0)

CLIENT : SAT-EA

Policy Instances: Ingress 0 Egress 0 Total: 0

Entities: (L4 level: Queues)

Level Chunk 0 Chunk 1 Chunk 2 Chunk 3

L4 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L3(8Q) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L2 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L1 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

Policers: Internal 0(0) Regular 0(0) Parent 0(0) Child 0(0) Total 0(0)

CLIENT : MAX

Policy Instances: Ingress 0 Egress 0 Total: 0

Entities: (L4 level: Queues)

Level Chunk 0 Chunk 1 Chunk 2 Chunk 3

L4 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L3(8Q) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L2 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L1 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

Policers: Internal 0(0) Regular 0(0) Parent 0(0) Child 0(0) Total 0(0)

CLIENT : P

Policy Instances: Ingress 0 Egress 0 Total: 0

Entities: (L4 level: Queues)

Level Chunk 0 Chunk 1 Chunk 2 Chunk 3

L4 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L3(8Q) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L2 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L1 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

Policers: Internal 0(0) Regular 0(0) Parent 0(0) Child 0(0) Total 0(0)

SUMMARY per NP:

=========================

Policy Instances: Ingress 32801 Egress 32789 Total: 65590

Entities: (L4 level: Queues)

Level Chunk 0 Chunk 1 Chunk 2 Chunk 3

L4 56( 56/ 56) 40( 40/ 40) 40( 40/ 40) 40( 40/ 40)

L3(8Q) 14( 14/ 14) 10( 10/ 10) 10( 10/ 10) 10( 10/ 10)

L3(16Q) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L2 13( 13/ 13) 10( 10/ 10) 10( 10/ 10) 10( 10/ 10)

L1 43( 43/ 43) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

Groups:

Level Chunk 0 Chunk 1 Chunk 2 Chunk 3

L4 14( 14/ 14) 10( 10/ 10) 10( 10/ 10) 10( 10/ 10)

L3(8Q) 13( 13/ 13) 10( 10/ 10) 10( 10/ 10) 10( 10/ 10)

L3(16Q) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L2 13( 13/ 13) 10( 10/ 10) 10( 10/ 10) 10( 10/ 10)

L1 43( 43/ 43) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

Policers: Internal 2048(2048) Regular 194848(194848) Parent 0(0) Child 0(0) Total 196896(196896)

PROFILES:

WFQ:

Level Chunk 0 Chunk 1 Chunk 2 Chunk 3

L4 254( 254/ 56) 254( 254/ 40) 254( 254/ 40) 254( 254/ 40)

L3 256( 256/ 14) 256( 256/ 10) 256( 256/ 10) 256( 256/ 10)

L2 256( 256/ 13) 256( 256/ 10) 256( 256/ 10) 256( 256/ 10)

L1 64( 64/ 40) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

SHAPE:

Level Chunk 0 Chunk 1 Chunk 2 Chunk 3

L4 1( 1/ 1) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L3 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L2 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L1 2( 2/ 20) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

WRED:

Level Chunk 0 Chunk 1 Chunk 2 Chunk 3

L4 16( 16/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L3 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L2 0( 0/ 0) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

L1 3( 3/ 23) 0( 0/ 0) 0( 0/ 0) 0( 0/ 0)

POLICE:

In HW (In Sw/ Ref count) : 66 ( 66 / 196896)

RP/0/RSP0/CPU0:mskl05#show qos interface Bundle-Ether7.4003.pppoe7812 output member tenGigE 0/2/0/16

Fri Jul 27 08:12:02.206 UTC

Interface: TenGigE0_2_0_16 output

Bandwidth configured: 10000000 kbps Bandwidth programed: 10000000 kbps

ANCP user configured: 0 kbps ANCP programed in HW: 0 kbps

Port Shaper programed in HW: 0 kbps

Policy: __sub_21727fffffff94 Total number of classes: 4

----------------------------------------------------------------------

Level: 0 Policy: __sub_21727fffffff94 Class: cmSrvRetail_LocalAccess_Out

QueueID: 386 (Port Default)

Policer Profile: 63 (Single)

Conform: 100000 kbps (100000 kbps) Burst: 1250000 bytes (0 Default)

Child Policer Conform: TX

Child Policer Exceed: DROP

Child Policer Violate: DROP

----------------------------------------------------------------------

Level: 0 Policy: __sub_21727fffffff94 Class: cmSrvRetail_InternetAccess_Night

QueueID: 386 (Port Default)

Policer Profile: 64 (Single)

Conform: 45000 kbps (45000 kbps) Burst: 562500 bytes (0 Default)

Child Policer Conform: TX

Child Policer Exceed: DROP

Child Policer Violate: DROP

----------------------------------------------------------------------

Level: 0 Policy: __sub_21727fffffff94 Class: cmSrvRetail_InternetAccess_Day

QueueID: 386 (Port Default)

Policer Profile: 65 (Single)

Conform: 15000 kbps (15000 kbps) Burst: 187500 bytes (0 Default)

Child Policer Conform: TX

Child Policer Exceed: DROP

Child Policer Violate: DROP

----------------------------------------------------------------------

Level: 0 Policy: __sub_21727fffffff94 Class: class-default

QueueID: 386 (Port Default)

----------------------------------------------------------------------

RP/0/RSP0/CPU0:mskl05#

RP/0/RSP0/CPU0:mskl05#show qos interface Bundle-Ether7.4003.pppoe7812 output member tenGigE 0/2/0/17

Fri Jul 27 08:13:34.039 UTC

Interface: TenGigE0_2_0_17 output

Bandwidth configured: 10000000 kbps Bandwidth programed: 10000000 kbps

ANCP user configured: 0 kbps ANCP programed in HW: 0 kbps

Port Shaper programed in HW: 0 kbps

Policy: __sub_21727fffffff94 Total number of classes: 4

----------------------------------------------------------------------

Level: 0 Policy: __sub_21727fffffff94 Class: cmSrvRetail_LocalAccess_Out

QueueID: 65826 (Port Default)

Policer Profile: 63 (Single)

Conform: 100000 kbps (100000 kbps) Burst: 1250000 bytes (0 Default)

Child Policer Conform: TX

Child Policer Exceed: DROP

Child Policer Violate: DROP

----------------------------------------------------------------------

Level: 0 Policy: __sub_21727fffffff94 Class: cmSrvRetail_InternetAccess_Night

QueueID: 65826 (Port Default)

Policer Profile: 64 (Single)

Conform: 45000 kbps (45000 kbps) Burst: 562500 bytes (0 Default)

Child Policer Conform: TX

Child Policer Exceed: DROP

Child Policer Violate: DROP

----------------------------------------------------------------------

Level: 0 Policy: __sub_21727fffffff94 Class: cmSrvRetail_InternetAccess_Day

QueueID: 65826 (Port Default)

Policer Profile: 65 (Single)

Conform: 15000 kbps (15000 kbps) Burst: 187500 bytes (0 Default)

Child Policer Conform: TX

Child Policer Exceed: DROP

Child Policer Violate: DROP

----------------------------------------------------------------------

Level: 0 Policy: __sub_21727fffffff94 Class: class-default

QueueID: 65826 (Port Default)

----------------------------------------------------------------------

RP/0/RSP0/CPU0:mskl05#

RP/0/RSP0/CPU0:mskl05#show qos interface Bundle-Ether7.4003.pppoe7812 output member tenGigE 0/2/0/18

Fri Jul 27 08:13:48.248 UTC

Interface: TenGigE0_2_0_18 output

Bandwidth configured: 10000000 kbps Bandwidth programed: 10000000 kbps

ANCP user configured: 0 kbps ANCP programed in HW: 0 kbps

Port Shaper programed in HW: 0 kbps

Policy: __sub_21727fffffff94 Total number of classes: 4

----------------------------------------------------------------------

Level: 0 Policy: __sub_21727fffffff94 Class: cmSrvRetail_LocalAccess_Out

QueueID: 131362 (Port Default)

Policer Profile: 63 (Single)

Conform: 100000 kbps (100000 kbps) Burst: 1250000 bytes (0 Default)

Child Policer Conform: TX

Child Policer Exceed: DROP

Child Policer Violate: DROP

----------------------------------------------------------------------

Level: 0 Policy: __sub_21727fffffff94 Class: cmSrvRetail_InternetAccess_Night

QueueID: 131362 (Port Default)

Policer Profile: 64 (Single)

Conform: 45000 kbps (45000 kbps) Burst: 562500 bytes (0 Default)

Child Policer Conform: TX

Child Policer Exceed: DROP

Child Policer Violate: DROP

----------------------------------------------------------------------

Level: 0 Policy: __sub_21727fffffff94 Class: cmSrvRetail_InternetAccess_Day

QueueID: 131362 (Port Default)

Policer Profile: 65 (Single)

Conform: 15000 kbps (15000 kbps) Burst: 187500 bytes (0 Default)

Child Policer Conform: TX

Child Policer Exceed: DROP

Child Policer Violate: DROP

----------------------------------------------------------------------

Level: 0 Policy: __sub_21727fffffff94 Class: class-default

QueueID: 131362 (Port Default)

----------------------------------------------------------------------

RP/0/RSP0/CPU0:mskl05#show qos interface Bundle-Ether7.4003.pppoe7812 output member tenGigE 0/2/0/19

Fri Jul 27 08:13:58.279 UTC

Interface: TenGigE0_2_0_19 output

Bandwidth configured: 10000000 kbps Bandwidth programed: 10000000 kbps

ANCP user configured: 0 kbps ANCP programed in HW: 0 kbps

Port Shaper programed in HW: 0 kbps

Policy: __sub_21727fffffff94 Total number of classes: 4

----------------------------------------------------------------------

Level: 0 Policy: __sub_21727fffffff94 Class: cmSrvRetail_LocalAccess_Out

QueueID: 196898 (Port Default)

Policer Profile: 63 (Single)

Conform: 100000 kbps (100000 kbps) Burst: 1250000 bytes (0 Default)

Child Policer Conform: TX

Child Policer Exceed: DROP

Child Policer Violate: DROP

----------------------------------------------------------------------

Level: 0 Policy: __sub_21727fffffff94 Class: cmSrvRetail_InternetAccess_Night

QueueID: 196898 (Port Default)

Policer Profile: 64 (Single)

Conform: 45000 kbps (45000 kbps) Burst: 562500 bytes (0 Default)

Child Policer Conform: TX

Child Policer Exceed: DROP

Child Policer Violate: DROP

----------------------------------------------------------------------

Level: 0 Policy: __sub_21727fffffff94 Class: cmSrvRetail_InternetAccess_Day

QueueID: 196898 (Port Default)

Policer Profile: 65 (Single)

Conform: 15000 kbps (15000 kbps) Burst: 187500 bytes (0 Default)

Child Policer Conform: TX

Child Policer Exceed: DROP

Child Policer Violate: DROP

----------------------------------------------------------------------

Level: 0 Policy: __sub_21727fffffff94 Class: class-default

QueueID: 196898 (Port Default)

----------------------------------------------------------------------

Aleksandar Vidakovic · ‎07-28-2018

hi Peter,

the queueing resources are fine. All the queues in use are the port default queues.

The scale limit that you are approaching (but not reached yet!) is 256k policers per NP:

Policers: Internal 2048(2048) Regular 194848(194848) Parent 0(0) Child 0(0) Total 196896(196896)

What kind of error is reported when you try to bring up more subscribers?

/Aleksandar

peter.ivanov · ‎07-30-2018

Hi, Aleksandar

I can't track errors that would clearly identify the cause of the problem. I have got only this output.

RP/0/RSP0/CPU0:Jul 30 08:22:24.736 : pppoe_ma[394]: Bundle-Ether7.4010: I dst d867.d959.deb7 src 5254.0066.0faa: len 38 0x11190000000a010100000120000205dc00000000000000000000000000000000000000000000

RP/0/RSP0/CPU0:Jul 30 08:22:24.736 : pppoe_ma[394]: [PADR-Recv]: Bundle-Ether7.4010 peer-mac 5254.0066.0faa

RP/0/RSP0/CPU0:Jul 30 08:22:24.736 : pppoe_ma[394]: [PADR-Recv]: vlan-id-outer 4010 vlan-id-inner 102

RP/0/RSP0/CPU0:Jul 30 08:22:24.736 : pppoe_ma[394]: [PADR-Recv]: Service-name:

RP/0/RSP0/CPU0:Jul 30 08:22:24.736 : pppoe_ma[394]: [PADR-Recv]: Max-payload: 1500

RP/0/RSP0/CPU0:Jul 30 08:22:24.736 : pppoe_ma[394]: [PADR-Recv]: DROPPED - session throttle or flow control

Aleksandar Vidakovic · ‎08-03-2018

hi Peter,

apologies for the delay. BNG implementation on XR uses the concept of "pppoe in-flight-window". You can see the related stats using the "sh pppoe summary total". It may simply be that the CPS rate was too high for all the work that needs to be done to activate the required features on the subscriber interface.

/Aleksandar