BNG deployment scale guidelines on ASR9000

Aleksandar Vidakovic · ‎06-10-2015

Update (March 2018)

This document was written at the time when 5.3.4 was about to be released. Hence the SW recommendation for BNG deployment has changed. Please refer to https://supportforums.cisco.com/t5/service-providers-documents/ios-xr-release-strategy-and-deployment-recommendation/ta-p/3165422.

The supported BNG scale specified in this document stands for all 32-bit (aka Classic) IOS XR releases, regardless of the line card generation (Typhoon, Tomahawk) or RSP generation (RSP440-SE, RSP880-SE) because the limit is imposed by the max memory that a process can address. Higher BNG scale requires the 64-bit (aka Enhanced) IOS XR, which also implies Tomahawk and RSP880-SE.

Introduction

The purpose of this document is to share the information about the BNG scale tested within the System Integration Testing (SIT) cycle of IOS XR releases 5.2.4 to 5.3.4 on Cisco ASR9000 platform. This is not the only profile in which BNG was tested at Cisco.

The scalability maximums for individual features may be higher than what is specified in this profile. The BNG SIT profile is provides for a multi-dimensional view of how features perform and scale together. The results in this document closely tie with the setup utilised for this profile. The results may vary depending upon the deployment and optimizations tuned for the convergence.

IOS XR release 5.2.4 is the suggested release for deploying BNG on the ASR9000 platform (until the XR release 5.3.3 is available).

IOS XR release 5.3.4 was posted on October 7th 2016. The XR release 5.3.4 is the Cisco suggested release for all BNG deployments. If your deployment requires features not supported yet in 5.3.4, please reach out to your account team for recommendation. The scale numbers for 5.3.4 are identical to those from 5.2.4. The difference between these releases is in the supported features and level of BNG hardening cycles that we at Cisco have completed.

Note: starting with XR release 5.3.4 you can use RSP880 in BNG deployments on asr9k. The scale remains the same because the scale is dictated by line card capabilities. The supported BNG scale will increase with the introduction of BNG support on Tomahawk generation line cards.

Feature And Scale Under Test

BNG With MSE (RP based subscribers)

Scale parameter	Scale
Access ports	bundle GigE/TenGigE
Number of members in a bundle	2
uplink ports	TenGige
Scale - PPP session PTA	64k (Without any Other BNG Sessions)
Scale - PPP Sessions L2TP	64k (Without any Other BNG Sessions)
Scale - IPv4 Sessions	64k (Without any Other BNG Sessions)
Scale - PPP v6 session	64k (Without any Other BNG Sessions)
Scale - IP v6 Session	64k (Without any Other BNG Sessions)
Scale - Dual-Stack Session	32k (Without any Other BNG Sessions)
Sessions Per Port	8k/32k
Sessions Per NPU	32k
Sessions Per Line Card	64k
Number of classes per QoS policy	8
Total number of unique QoS policies	50
Number of sub-interfaces	2k
Number LI taps	100
Number CPLs	50
Session QoS	Ingress: Policing Egress: Hierarchical QoS; Model F with 3 queues per session
Number IPv4 ACLs	50
Number ACEs per IPv4 ACL	100
Number IPv6 ACLs	50
Number ACEs per IPv6 ACL	100
Number IGP routes	10k
Number VRFs for subscribers	100
Number BGP routes	30k
Number VPLS	3000
Number VPWS	2000
Number MAC addresses	2000
Number of sessions with multicast receivers	8000
AAA Accounting	Start-Stop every 10min Interim
Service Accounting	64k + 2 services
BNG over Satellite	32k
CPS rate	80 calls per second
CoA	40 requests per second
DHCP (proxy or server)	60 requests per second

BNG Only (RP based subscribers)

Scale parameter	Scale
Access ports	bundle GigE/TenGigE
Number of members in a bundle	2
Scale - PPP session PTA	128k (Without any Other BNG Sessions)
Scale - PPP Sessions L2TP	64k (Without any Other BNG Sessions)
Scale - IPv4 Sessions	128k (Without any Other BNG Sessions)
Scale - PPP v6 session	64k (Without any Other BNG Sessions)
Scale - IP v6 Session	64k (Without any Other BNG Sessions)
DS PPPoE	64k (Without any Other BNG Sessions)
Static Session	8k (Without any Other BNG Sessions)
Routed Subscribers	32k (Without any Other BNG Sessions)
Service Accounting	64K with 2 Services , 24 with 3 Services
Routed Pkt Triggered subscriber IPv4/IPv6	64k

CPU Utilisation (RP based subscribers; 128k IPoEv4)

HW Type	A9K-RSP-440-SE	MOD80-SE
Role	RSP	Subscriber facing line card
Major features	Total of 128k RP based IPoEv4 or PPPoE subscribers	Data plane
CPU steady state	14%	10-14%
CPU peak	60-70%	50-60%
Top processes	lpts_pa	prm_server_ty
	iedged	qos_ma_ea
	ppp_ma/dhcpd	prm_ssmh
	ifmgr	netio

CPU Utilisation (RP based subscribers; 64k Dual-Stack)

HW Type	A9K-RSP-440-SE	MOD80-SE
Role	RSP	Subscriber facing line card
Major features	Total of 64k RP based IPoE or PPPoE subscribers	Data plane
CPU steady state	14%	10-14%
CPU peak	40-50%	30-40%
Top processes	lpts_pa	prm_server_ty
	iedged	qos_ma_ea
	ppp_ma/dhcpd	prm_ssmh/ipv6_nd
	ifmgr/ipv6_nd	netio

BNG Only (LC based subscribers)

Scale parameter	Scale
Access ports	GigE/TenGigE
Uplink ports	GigE/TenGigE
Scale - PPP session PTA	256k (Without any Other BNG Sessions)
Scale - IPv4 Sessions	256k (Without any Other BNG Sessions)
Scale - PPP v6 session	128k (Without any Other BNG Sessions)
Scale - IP v6 Session	128k (Without any Other BNG Sessions)
DS PPPoE	128k (Without any Other BNG Sessions)
Static Session	8k (Without any Other BNG Sessions)

CPU Utilisation (LC based subscribers; 256k IPv4 Sessions)

HW Type	A9K-RSP-440-SE	MOD80-SE
Role	RSP	Subscriber facing line card
Major features	LC based IPoE or PPPoE Subscribers (256k)	LC based Subscribers, Control and Data plane (64K per LC)
CPU steady state	10%	10-14%
CPU peak	10%	80-90%
Top processes	dhcpv6/dhcp	prm_server_ty
	iedged/ipv4_rib	dhcpv6/dhcp
	ppp_ma/dhcpd	iedged
		ppp_ma
		qos_ma_ea

PWHE With BNG Only (RP based subscribers)

Scale parameter	Scale
Generic interafce list members	Bundle, GigE/TenGigE
Number of members in a bundle	2
Scale - PPP session PTA	128k (Without any Other BNG Sessions)
Scale - IPv4 Sessions	128k (Without any Other BNG Sessions)
PW Scale	1k

References

ASR9000 BNG Links.

Fredrik Lönnman · ‎01-14-2016

Thanks you for this summary Aleksandar, really appreciated.

What numbers should we look at for the ASR9001? You're hinting that 5.3.3 will be the next recommended BNG release, is that so? Are there any BNG scale improvements committed for 5.3.3 that would change these numbers?

Aleksandar Vidakovic · ‎01-15-2016

Hi Fredrik,

I'm glad you found the document useful.

5.3.3 offers more BNG features and stability, but it doesn't offer significant scale or performance improvements. That will come with BNG support on the latest generation of Ethernet linecards (Tomahawk), planned for later this year.

Regards,

Aleksandar

Umpri · ‎04-04-2016

Hi Aleksandar,

I was wondering if you coud tell me the BNG deployment guidelines on ASR9000 with release 6.0.1.

We are thinking of evolving BNG to migrate from bundle RP based subscribers to PWHE RP based subscriber. Remaining PPP session PTA model.

Best regards

Javier

Aleksandar Vidakovic · ‎04-04-2016

Hi Javier,

scale numbers have remained the same. We have been focusing on introducing many great new features. Higher scale per LC and per system will come with BNG support on Tomahawk line cards and RSP880.

regards,

Aleksandar

samiwehbi · ‎06-06-2016

Dears,

thank you for he helpful guide,

one question if using A9K-36X10G-AIP-SE Line card which contains 6 NPU's(not sure).

would it mean that it can support 256k sessions ? with or without QOS ?

Regards

Sami

Aleksandar Vidakovic · ‎06-06-2016

hi Sami,

there is a limit 64k of sessions per LC, irrespective of the number of NPs. This scale number will be higher on Tomahawk line cards. On Typhoon it will remain as is.

regards,

Aleksandar

samiwehbi · ‎06-06-2016

thank you Aleksandar.

Umpri · ‎06-09-2016

Hi Alex,.. I wonder if you could explain a bit more this parameter you showed in the table:

Number of sub-interfaces = 2K

I meant in a bundle with two members only 2 thousand L3 subs can be configured?

I look forward to hearing from you asap

Best regards

Javier

Aleksandar Vidakovic · ‎06-10-2016

hi Javier,

that table is not showing the maximum uni-dimensional scale. The table shows what have we tested in this multi-dimensional scale scenario. If your deployment is within the scale limits shown in this table, you should be good. I your deployment exceeds some scale parameters, you need to pay more attention to negative testing in your scenario (link flaps, etc.).

I hope this clarifies.

/Aleksandar

Umpri · ‎06-10-2016

Alex thanks for you reply..

I will try to commit to this limits.

Two things are for me a bit weird, you know, that is 2,000 subinterfaces L3 and only 50 unique QoS policies. For what I understand under this scenario you recommend not to have on a bundle more than those figures.

Best regards,

Javier

Aleksandar Vidakovic · ‎06-11-2016

hi Javier,

these are 2000 BNG access sub-interfaces. On each of those you can have multiple BNG subscriber sub-interfaces, up to the max scale per NP (which is 32k/NP).

What is your deployment scenario? I would like to understand why 2000 BNG access sub-interfaces is not sufficient. Even in that case the 2000 is not a hard limit.

The same stands for the QoS policy. These are configred 50 unique policies, but each had multiple instances applied.

regards,

Aleksandar

Syed Zaidi · ‎06-11-2016

Thanks Aleks

Umpri · ‎06-15-2016

Thanks Ale, is clear for us the BNG scale up to the max scale per NP (which is 32k/NP).

Let me try to explain in some words how our network is working according issus of design.

There are mpls rings in the access connecting two or three uPE, the rings end on the MSE ASR9010 that has the BNG and PE function, for as multiservice bng. To separate those service we use differents pseudowires. The termination for HSI's pw is at the level 2 side of the hairpnning, at the other side of hairpinning we have the bba group in bundle interface as our design is to protect pppoe session on the RPS. Now we are testing 6.0.1 to deploy pw-he and remove all the hairpinnings. A matter of design on this will be the amount of session the int pw-ether can support. So far we have 25.000 sessions for hairpinning to squeeze it at 80% more or less.

As well, we have some frontiers with a L2 Lanswitch which has many conections to PTN transport network where you can see some uPE at the other end. So at the ASR9K we have a 20G eth bundle (10G active+10G stand by) as you can see it on the draw, with many subinterfaces, some of them are level 2 and others are level 3 with IP address. So my cuestion was addressed to know how many L3 subinterfaces can support because we could not test it yet.

That's all. Thanks for your answers.

Regards,

Javier

PD: we need xr6.0.1 as it supports LAC

Aleksandar Vidakovic · ‎06-15-2016

The bundle EFP scale (l2transport bundle sub-interfaces) is 64k/LC and 128k/system.

The bundle L3 subinterface hard limit is 3800 per NP.

All of the above are uni-dimensional scale numbers.

If in MSE you are planning to go above the scale documented in the table at the top of the page, please make sure you test the desired scale in the lab.

braintrust-nward · ‎09-19-2016

Hi Xander,

This is a great doc, very helpful.

We have ASR9001 BNGs with RP based subs. Are the RP based subs here similar numbers to what we'd expect on the ASR9001, or does the RP tested here perform differently?

We don't have queues on all our subs - maybe 50%, so QoS chunks are only consumed for some customers. I presume the 32k limit that is often stated for the ASR9001 is because of QoS chunks, though I don't have anything to back that up.

We currently have a redundant BNG that has no subscribers on it, but one of our active BNGs is at 31k subs - none of these subs have shapers, so no QoS resources are used. If we can go past 32k unshaped subs on this BNG it would mean we don't have to alter our resiliency model in the interim before adding more BNG capacity.

About 50% of the subs on this BNG that's about at capacity don't even have RADIUS accounting, if that makes much difference.