cancel
Showing results for 
Search instead for 
Did you mean: 
cancel

ASR9000/XR Using and understanding BNG GEO-Redundancy

10777
Views
10
Helpful
74
Comments

Introduction

Geo redundancy is a powerful new technology for XR BNG that allows for session synchronization between 2 nodes. This means that a session active on one node has a shadow and fully programmed session on a standby node, so that when the active chassis fails, the standby BNG can take over and continue to forward the session info WITHOUT service interruption to the user.

Geo redundancy overcomes some of the restrictions that other redundancy models have which makes it a solution that is very compelling.

Existing redundancy models

Some of the existing models include the use of PPPoE smart server selection, ASR9K nv Cluster, ISSU, MC-LAG/MSTAG. This section outlines their operation and pros/cons.

Smart Server selection

Smart server selection relies on the operation whereby a host sends a PADI (discovery), which is broadcast to multiple devices/BNG's. Normally all hosts send a PADO (offer) back to the client who then connects with one of the offered BNG's for a single connection. By controlling the response time of the PADO's from all BNG's we can make one node more primary for a particular vlan, and the other(s!) standby.

The solution is stateless, meaning that if the active node dies, the client needs to rediscover and will find one or more stadnby BNG's for connection with.

Pro is that this is simple, useful, it provides N+1 redundancy (multiple BNG nodes can be used on the segment for more sharing of the load).

Con is that this is stateless, clients have to reconnect, per vlan bases and for PPPoE only (not usable for dhcp). Though a similar concept can be leveraged for IP sessions by delaying the offer timers of the dhcp server.

nV Cluster

Clustering two devices by linking their brains together via what we call the EOBC (Ether out of band connection), makes two chassis become a perfect mirror from each other. This automatically means that you have stateful redundancy.

It relies on the fact that you are dual homed with a connection in both both racks of the cluster. If the cluster device or rack we call it, fails the other chassis will take over as sole primary and the forwarding over the bundle all happens without any disruption

Pro: powerful, stateful, high scale

Con: sw upgrades, hw restrictions for cluster, requires bundle intefaces and dual homing into both nodes of the cluster, costly/license

 

MCLAG/MSTAG

Using standard redundancy technologies like mclag or mstag provides for a lot of simplicity. These technologies allow for dual homing and relying on ICCP (mclag) or STP (MSTAG) protocols to detect loops and only have one active link forwarding.

This means that a session is only available and active on one node at the time.

 

Pro: very simple low cost

Con: long convergence times and stateless

Summary

How nice would it be to have the best of all these solutions and not having so much cons? That is where GeoRed comes into play :)

 

How to use GeoRed

Geo redundancy provides for a very powerful M:N or N+1 redundancy model depending on how you like to implement it.

Flexible redundancy models via pairing across routers on Access Link basis

- 1:1 (both active/active with load sharing or active/standby) (like nv Cluster)

- M:N (active/standby roles and load is split across multiple routers)

- N:1 (1 backup for N active)

 

Full circle standby (M:N)

Designated backup (N:1)

There is no special connection required between the BNG's, just an ip connectivity for the redundancy protocol (to be discussed later).

One big advantage is also that the different BNG nodes may be placed in different geo-locations without any limitations!

Complements existing BNG high-availability, redundancy and geo-redundancy mechanisms.

Geo redundancy in a nutshell:

 

A typical design could look like this:

CPEs are agnostic to redundancy and  they see “one BNG / Gateway”. Any switchover is transparent to them. With the redundancy model used, the CPE peers with the same mac address and node ID hence if a failover is required the CPe doesn't even know that it is talking to a different physical device.

 

Access Nodes are dual/multi-homed for redundancy using a variety of technologies such as MCLAG, Dual Homed (MST-AG), Ring (MST-AG or G.8032), xSTP, Seamless MPLS (PWs), etc. Using heartbeat mechanisms like E-OAM, BFD, etc. for faster fault detection/isolation.

BNG is not just a gateway router, it has subscriber state, policies and accounting/authorization details and subscriber features. Redundancy and synchronization also require sharing of protocol state like DHCP and PPP.

A good redundancy solution also should employ seamless integration with external servers like DHCP/Radius and backend policy/billing systems.

 

Implementation details

The concept for geo-redundancy is built on top of a sync protocol that is used in MCLAG also: ICCP (inter chassis communication protocol). It is a reliable protocol that allows for state and info sync between 2 chassis.

One of the basic pieces to that is the definition of what we in GEORED will call the Subscriber Redundancy Group (SRG).

Taking the picture from above, that shows the M:N or N:1 redundancy topologies, an SRG is the equivalent of the "X" or "Y" arrows:

Synchronization

Synchronization from “master” to “slave” is done over TCP on per SRG basis between routers using proprietary mechanism – BNG Sync

This mechanism serves the following purposes:

  • Signaling failures and role changes
  • Synchronization of subscriber sessions’ control plane states
  • Communication of other events and commands

When BNG SRG peers connect, first the master slave determination is done, after which sync of state happens from master to slave followed by regular mirroring that happens without delay with without holding up the session provisioning on master.

Session mirroring takes care of complete state once the session is up; and when there is any change or when it is deleted

Roles

Master/Slave roles are defined by the SRG and not defined as a BNG router by itself. This simply means that SRG1 can be active on router ONE and SRG2 will be active on router TWO, and SRG1 will be standby on router TWO also.

active/active – (eg the M:N) BNG  could be master for one SRG and slave for another

active/standby – (eg the N:1) dedicated  backup BNG could be slave for multiple SRGs from different active BNGs which are masters for those respective SRGs

Role negotiated via BNG sync between routers on per SRG level

Where possible, role can be determined by the underlying access technology

In master role  BNG will handle and process all control traffic it receives

In slave role  BNG will ignore all BNG and related protocols traffic. It will receive state notifications of the session via the ICCP communication from the active node serving that SRG.

 

Modes of operation

GEORED can operate in two distinct redundancy operations. That is hot and warm standby.

Hot-Standby Mode (default)

Sessions provisioned on slave in sync with setup on master

Since the sessions are actively programmed on the standby, this will consume hardware resources on slave. Proper planning is necessary here, since if we have BNG node X and Y both serving 50k sessions each, the slave node needs to be able to support 100k sessions when they are actively programmed!

Minimal action on switchover; data plane is already setup for sub-second traffic impact, this is the highest level of redundancy you can achieve.

And especially useful in deployments requiring high and tight SLA

Warm-Standby Mode (for over-subscription)

Sessions data kept in “shadow” database on slave in sync with setup on master

Only consumes some additional memory in control plane for the shadow copy – no provisioning in hardware

Upon failover trigger, sessions are setup at rapid pace from shadow copy

This allows for over provisioning on backup for subscribers. While it still provides for a high level of redundancy, and the "outage" or forwarding loss is determined by the time it takes to hw program the sessions served by the SRG, the failover will result in some session loss (if the SRG serves high number of sessions that take longer to program then the keepalive/timeout of the session).

 

Session distribution

Example scenario with an active/standby, N:1 model:

  • Sessions are associated with partitions (vlan 1,2,3,4) on BNG1 with each VLAN mapped to different SRG configured with master role
  • BNG2 IS acting as backup for all VLANs
  • Each VLAN has 8k sessions terminated

 

Example scenario with an active/active, N:1 model:

  • Sessions are associated with partitions (vlan 1,2) on BNG1 with each VLAN mapped to different SRG configured to Master role
  • Sessions are associated  with partitions (VLAN 3,4) on BNG2 with each VLAN mapped to different SRG configured to Master role
  • Each VLAN has 8 sessions terminated
  • Each BNG has 16k session terminated

 

Radius Interaction

some important notes regarding radius accounting and authorization information

 

  • Authentication/Authorization done only from the Master and all profile information is syncd to Slave. Slave does not reach out to radius and relies on the session information received from the active node.
  • On Failover, Accounting Stop message is sent from old Master and Accounting Start from New Master.
  • NOTE: Accounting Stop from old master sent on best effort basis and ordering is not guaranteed between it and the Accounting Start sent by the new master. A failover session should be handled as two separate sessions by Radius
  • Radius (Accounting) messages from BNG are paced with jitter (especially around switchover) to avoid load on the server

 

Managing upgrades

One of the big advantages of GEORED that overcomes a painpoint of nV cluster is the sw upgrades.

In cluster, an orchestration is necessary to separate the cluster nodes, upgrade one and make a quick switch over to upgrade the other one.

In Geo Redundancy, the BNG nodes can run different sw versions even! and that is no problem. Although we wouldn't recommend too much version disparity between the devices and for the ease of deployment have all BNG nodes in the network, regardless of being part of the GEORED to be on the same sw version as much as possible with the same smu set.

The SW upgrade procedure would be opaque to the redundancy model chosen (N:1, M:N, active/active or active/standby).

Basically the steps include:

  1. Failover SRG's one by one running active on the BNG node to its standby
  2. If hot standby, step "1" will be quick. If warm standby allow for some time for the sessions to be programmed
  3. upgrade the BNG to the desired sw level
  4. pull back all sessions for the SRG's that need to be running active on this BNG

And do this for all the BNG nodes part of the SRG interaction.

NOTE: you can even setup GEO red just for the upgrade procedure. A node that is synchronizing its sessions during this setup is not affected whatsoever.

 

Session set up and call flow details

The following section graphs out the call flow and messaging between BNG SRG devices and the session.

Initial Session Setup

 

  • Only Master does Radius/Policy server interactions for the subscriber using its NAS IP, NAS Port and accounting session ID
  • In addition to protocol state, subscriber profile (including any further changes as result of CoA) are sync-ed across from master to slave
  • Slave sets up the same subscriber with a different accounting session ID – it has different NAS IP and likely different NAS Port
  • Redundancy design and Slave is invisible to Radius/policy server before the switchover. That is the radius/PCRF have no awareness of the fact that a session is synchronized.

 

Failure scenario

 

 

  • Subscriber already provisioned on slave and ready to forward traffic even before switchover; Loss on fail-over depends on Access network failover or convergence
  • Core network design – fast reroute, BGP PIC, core convergence
  • BNG Sync channel used to signal failures and trigger switchovers between BNG routers; this is control plane sync.
  • Accounting updates – start/stop/interims
  • DHCP state machine on slave takes over without any client/server interactions
  • Lease will continue on slave from when master started it
  • PPPOE/PPP state machine on slave takes over from where master left without any client impact
  • PPP keep-alive will start flowing from new master on takeover

 

Use cases

 

MSTAG

 

The MSTP protocol is used here to block standby path so we have only one active

In this case each BNG have their own MAC which is used for MST and other Ethernet protocols. In this scenario we need to setup SRG vMAC for BNG sessions. Which will act like an HSRP/VRRP virtual mac in the same facinity. The BNG's use their own mac for the STP communication, we'll use the vmac towards the sessions as their peering/communication point.

For dual homing two MST instances required with VLANs split across them to enable active/active load balancing to each of the 2 BNGs

MST provides “preempt delay” knobs to throttle switchovers and allow stabilization of subscribers on top of it after failure recovery.

Failure detection, or the improved detection for it is done via CFM sessions (at least one per MST instance in any of its VLAN). The CFM session is used to monitor connectivity and to detect which BNG has the forwarding path and which one has the standby/drop path (i.e. CFM session will be UP on active & DOWN on standby)

Coupling the CFM session via EFD with each of the BNG  L3 access sub-interfaces on that interface will result in that sub-interface status tracking UP on active side and DOWN on standby side.

Access tracking object monitoring this sub-interface status (which is in turn controlled via EFD based on CFM session) is used for determining SRG role as well as controlling the subscriber subnet route advertisement

In event of failures, as MST re-converges and switches paths, the CFM session status changes and the L3 BNG sub-interfaces get notified of status via EFD such that the SRG role can be switched

MST and CFM timers can be as aggressive as supported by the access devices with stable operations even with full subscriber load

 

MCLAG

 

 

MC-LAG provides consistency of MAC and IP address across the two PoA (i.e. BNG routers). In this scenario there is no need for SRG vMAC since it is managed by MCLAG natively already.

The failure is induced by an object that directly tracks MC-LAG bundle interface status and signals to both SRG (for role determination) & the routing entity (to control the subnet/pool advertisement).

MC-LAG provides knobs to throttle switchovers and allow stabilization of subscribers on top of it in event of link flaps and after failure recovery

Parameters to consider when using MCLAG:

mlacp switchover recovery-delay – to ensure bundle remains slave after recovery from failure and allows subscribers to get sync and stabilized on it in slave mode

mlacp switchover type revertive – means that when the primary comes back, it will assume the primary role also and basically pull everything from the standby back. Like HSRP preempt.

lacp switchover suppress-flaps – to avoid switchover for transient link-flaps

BFD or CFM with EFD can be used for faster detection of failures in addition to LACP protocol mechanisms

 

LAC

 

Configuration and setup

Now that you know everything about GEORED you want to go set it up right?! Here is a config piece and explanation what it is for.

Enable BNG GEo Redundancy

group 1

peer 1.1.1.1

Set up SRG and define which group holds which 

interface. Multiple groups can be defined.

subscriber redundancy

group 1

interface-list

interface bundle-ether1 id 1

Setup Access Object Tracking for SRG and Summary Subscriber route.

In this example we are tracking the interface bundle state that MCLAG is providing to us.

If we see that the state is going down, that will result in a static route withdraw from the table.

If we have redistribute static configured, the pool summary

will be removed so that the previous standby,

now active can start advertising the summary to start

pulling the traffic.

track access-mclag

type line-protocol state

interface bundle-ether1

subscriber redundancy

group 1

access-tracking access-mclag

router static

address-family ipv4 unicast

10.0.0.0/24 null0 track access-mclag desc sub-pool-summ

Optional SRG configuration to determine more deterministically what the preferred role is and what redundancy mode should be run.

subscriber redundancy

preferred-role master

slave-mode warm

hold-timer 15

 

A little more detail of the subscriber redundancy configlet

 

 

Restrictions and limitations

As with everything in technology, there is always some trade off. This table below is what exists currently as know restrictions for the GEORED solution as of XR 5.3.3

Note that XR6 has quite a significant amount of improvements, that will be documented separately. Since XR 5.3.3 is the going release today for ASR9000 I thought it is important to know what you get and where you need to think about.

 

Limitation

Recommendation

With just Core tracking, if core interface goes down, SRG switchover is triggered causing traffic black hole on access

EEM script can be used to shut access when core goes down

RA will send with both SRG vMAC as well as interface MAC towards access

use  RA preference CLI   under dynamic template or access-interface 

Accounting records may get lost if we do back-to-back switchover before they sync on master and slave 

we should wait for 15 mins before doing

Switchover (128k sessions)

Admin clear of sessions from the slave is prohibited

1.  If slave is out of sync from master,  subscriber redundancy synchronize  command can be issued from slave to replay

 

 

2.  SRG clear command can be issued either from slave or Master  to get slave back to normal state

Master reload is not recommended  on the access with non-revertive protocol  support

Enable revertive configuration on the access-protocol

On flight vmac modification for IPv6 sessions is not supported

 

 

 

Features not supported:

  • Static subscriber
  • DHCP Routed subscriber 
  • Packet trigger Sessions
  • Multicast on subscriber and Qos Correlation
  • SLAAC for subscriber
  • BNG as DHCP server
  • IPv6 ND as SRG client
  • Diameter & Geo-redundancy interworking (6.2.x)

 

 

 

XR6 enhancements details

 

 

 

 

Miscellaneous

With great thanks to the GEORED dev team for some of the visualizations used in this paper.

 

PS. it is highly important not to use pppoe bba-group Global. This is a reserved keyword that is known to break certain SRG cases. name your bba-group to anything but global/Global.

 

 

 

Comments
Community Member

Hi Xander,

great guides you created,

we are testing geo redundancy and faced a problem, after sessions are shifted to the slave BNG they get disconnected after a while and after debugging pppoe protocol we get the below output,

could it be that we are testing with free radius, since its a requirement that the AAA is redundancy aware ? (dynamic template, pool, aaa and control policies are identical).

RP/0/RSP0/CPU0:Apr 18 02:38:30.910 : pppoe_ma[384]: [PADI-Recv]: Bundle-Ether10.10 peer-mac d4ca.6dd7.4bf0
RP/0/RSP0/CPU0:Apr 18 02:38:30.910 : pppoe_ma[384]: [PADI-Recv]:    vlan-id-outer 10
RP/0/RSP0/CPU0:Apr 18 02:38:30.910 : pppoe_ma[384]: [PADI-Recv]:   DROPPED - interface unknown to PPPoE
RP/0/RSP0/CPU0:Apr 18 02:38:30.910 : pppoe_ma[384]: Bundle-Ether10.10: I dst ffff.ffff.ffff src d4ca.6dd7.4bf0: len 46 0x11090000000c01030004000000a40101000000000000000000000000000000000000000000000000000000000000
RP/0/RSP0/CPU0:Apr 18 02:38:30.910 : pppoe_ma[384]: [PADI-Recv]: Bundle-Ether10.10 peer-mac d4ca.6dd7.4bf0
RP/0/RSP0/CPU0:Apr 18 02:38:30.910 : pppoe_ma[384]: [PADI-Recv]:    vlan-id-outer 10
RP/0/RSP0/CPU0:Apr 18 02:38:30.910 : pppoe_ma[384]: [PADI-Recv]:   DROPPED - interface unknown to PPPoE
RP/0/RSP0/CPU0:Apr 18 02:38:31.727 : pppoe_ma[384]: Bundle-Ether10.10: I dst ffff.ffff.ffff src d4ca.6dd7.4bf0: len 46 0x11090000000c01030004000000a50101000000000000000000000000000000000000000000000000000000000000
RP/0/RSP0/CPU0:Apr 18 02:38:31.727 : pppoe_ma[384]: [PADI-Recv]: Bundle-Ether10.10 peer-mac d4ca.6dd7.4bf0
RP/0/RSP0/CPU0:Apr 18 02:38:31.727 : pppoe_ma[384]: [PADI-Recv]:    vlan-id-outer 10
RP/0/RSP0/CPU0:Apr 18 02:38:31.727 : pppoe_ma[384]: [PADI-Recv]:   DROPPED - interface unknown to PPPoE
RP/0/RSP0/CPU0:Apr 18 02:38:31.727 : pppoe_ma[384]: Bundle-Ether10.10: I dst ffff.ffff.ffff src d4ca.6dd7.4bf0: len 46 0x11090000000c01030004000000a60101000000000000000000000000000000000000000000000000000000000000

please advise,

Thanks

Rami

Cisco Employee

Ah thanks Rami :) yeah it wasnt meant to get posted yet, as I am still working on it :)

Say, it smells like the access interface has no bba-group attached to it, could that be Rami?

either case the interface manager doesnt know that this bundle-e10.10 is enabled for pppoe access causing the PADI to be dropped and this message to be printed.

config and version would help here also.

cheers

xander

Community Member

Hi Xander,

found out the issue, it was because we were configuring a mac-address under the bundle-Ethernet on both BNG's, seems that MC-LAG already takes care of that, as you also mention in your guide no need for vmac as well in SRG.

it was causing the problem.

thanks

Rami

Community Member

Hi Xander,

when creating the MC-LAG it adds a unique Mac-address to the bundle on both routers, so is it normal to keep it that way without configuring it under the bundle, since its working now without the mac-address command under the bundle-ethernet interface.

thanks for your support

Rami

Community Member

Dear Xander,

Concerning the unsupported features, we are trying to enable Multicast for PPPOE subscribers with Geo redundancy ,when applying "Multicast ipv4 passive" under the ppp dynamic-template we get the below errors in the logs,

LC/0/3/CPU0:Apr 26 01:03:48.936 : subdb_svr[356]: %SUBSCRIBER-TMPL-5-UNSUPPORTED : dynamic-template 'CITY-TPL' has unsupported configuration 'ipv4-igmp/ord_d/default/multicast' for this node

LC/0/0/CPU0:Apr 26 01:03:48.937 : subdb_svr[363]: %SUBSCRIBER-TMPL-5-UNSUPPORTED : dynamic-template 'CITY-TPL' has unsupported configuration 'ipv4-igmp/ord_d/default/multicast' for this node

LC/0/1/CPU0:Apr 26 01:03:48.937 : subdb_svr[363]: %SUBSCRIBER-TMPL-5-UNSUPPORTED : dynamic-template 'CITY-TPL' has unsupported configuration 'ipv4-igmp/ord_d/default/multicast' for this node

In the Cisco configuration guide for version 5.3.3 its mentioned that Multicast for PPPOE subscribers is not supported for BNG over Pseudowire Headend, in our Case we are using MC-LAG.

In this guide you mention that Multicast is not supported at all with Geo redundancy.

please advise and thank you

Cisco Employee

Multicast is not supported yet in BNG Geo-Redudancy.

Community Member

Dear Xander,

thanks for the confirmation, the thing here is that this dynamic-template is not applied to any policy-map yet it gives these errors, is that normal.

thanks again,

Rami

Hello

This is a feature that is very interesting since it addresses an important aspect of the BNG operation. We are interested in testing the Georedundancy feature and we have some questions about its expected behavior.

- In the example config, the BNG has a dedicted pool/static route for the SRG group. With multiple interfaces (and correspoding SRG groups) we will need to have multiple pools. Is it possible to have a single pool (much easier to manage) serving all interfaces / SRG groups, that will be common for both active / standby BNGs?

 ** This means that we will have to take care of the routing ourselves between active and standby BNG.
 ** Furthermore in the case of this common pool, if a a particular SRG group switches over to the stanby BNG, this standby BNG must make sure the the addresses it allocates, do not overlap with addresses already assigned by the active BNG. Is this the expected behavior with georedundancy? (do the active and standby BNGs keep track of the IPs assigned by each one for every SRG group)?


- The case that we need to cover is losing the entire BNG. Is it possible to create one SRG group covering the entire BNG (not just an individual interface)?
 In this case what will happen if a particular interface switches to the standby BNG (due to mlacp)? Will the PPPoE sessions be reestablished to standby-BNG?

Thanks for the support

Cisco Employee

Hi Alejandro, the ip pool must be unique per SRG node. because the allocation/reservations is/are not synchronized.

so you have x/24 on SRG-1 and y/24 on SRG-2. SRG-1 advertises a summary for X/24, when SRG-1 dies, SRG-2 will take over, and if configured/set up correctly it will now advertise the X/24 as the sessions from SRG-1 will now live on SRG-2.

So there is a unique pool per SRG and the conditional advertisement tells if the summary is injected or not for that pool on BOTH srg's.

cheers!

xander

Hello Xander

If we move towards the other direction (having one SRG group covering the whole box) do you know what the expected behavior of standby BNG will be?
For example if an individual interface, that participates in this SRG, switches to the standby BNG (using e.g. mlacp) will the standby BNG be
able to establish new PPP sessions? (or all SRG interfaces in the standby BNG stop accepting new PPPoE sessions)?
The reason we ask is that in a BNG topology with many intefaces / SRG groups it will be difficult to manage multiple pools

Thanks for your support

Cisco Employee

correct, it will fail over on a per srg bases, if one srg covers everything then it is all or nothing.

it would make sense to split your pool in half and divide it over the 2 devices so you get some nice sharing of the load also.

xander

Cisco Employee

Hello Xander,

we are in-progress with PoC lab at our customer for BNG GeoRedundancy.

Will you please share more details regarding non supported feature of static subscriber?

Do you mean subscribers that get assigned static IP address from RADIUS, and not from local-pool configured on the BNG?

BR

Harris

Cisco Employee

a static subscriber in this regard is a hard configured one, not one with a sticky IP addr.

that one, sticky addr, as provided by a pool designation or framed ip is fully supported and functional on GEORED no problemo (may need some routing tricks to have a standby route available on advertise etc, but RPL is your friend here)

a static subscriber is effectively a configured (sub)interface with a (qi)q-combo that doesnt need a trigger like dhcp or padi or unclassified source for the session to be created. Such sessions are not synced with geored.

cheers!

xander

Cisco Employee

Thank you so very much Xander!!!!

Beginner

Hi Xander,

hope you are doing well,

want to ask you about the below link,

"http://www.cisco.com/c/en/us/td/docs/routers/asr9000/software/asr9k_r5-3/bng/configuration/guide/b-bng-cg53xasr9k/b-bng-cg53xasr9k_chapter_01110.html#d123551e1143a1635"

States the below

"Although geo redundancy with vMAC and ambiguous VLAN are supported in BNG, both these features are not supported simultaneously."

So basically if we have a scenario with QinQ (ambiguos VLAN) we can only do it using MC-LAG, since MSTAG will need the vMac feature, is there any workaround for this or support in later IOS-XR.

please advise & thank you for your usual support.

 

CreatePlease to create content
Content for Community-Ad
August's Community Spotlight Awards