BNG cluster - redundancy

[elnur.mammadov]

Hello everyone,

What would be the best way to achieve link/cluster redundancy into the access plant given the second diagram? 

What is the best practice for redundant BNG configuration?

Cluster is a couple of ASR9001 with a 2 port 10gbps line card in each, and access nodes can be considered third party L2 switches chain linked to each other over 10 gpbs ( this is the OSP for FTTx ).

BNG (experimental, both PPPoE and IPoE work) on a bundle-ether interface works, members of the bundle-ether are east and south interfaces from distinct cluster members, haven't tested cluster redundancy yet. 

EAPS Ring redundancy with the setup #1 works but it has a single point of failure, the access node itself with the up links.

In the second setup with the TLS (Basically an L2 bridge without any protection protocol, none) bundle-ether link redundancy only works when links directly connected to the cluster go down.  As per Cisco papers, bundle-ether interface are for point-to-point links, so it is not supposed to work if an intermediate link goes does down, I understand that, there's no signalization.

I guess my question is how would you suggest to implement link redundancy in setup 2 with TLS, thus eliminate the single point of failure of EAPS setup, I've already contacted the vendor of the access nodes on the subject, awaiting their response, it has been a little over an year.

Thank you.

http://en.wikipedia.org/wiki/Ethernet_Automatic_Protection_Switching

Regards

Elnur

[xthuijs]

Hi Elnur,

Because you have an L2 loop between these access nodes, and it is a ring, I dont see any other options then:

MST(AG) or MCLAG.

This to break the loop and loadbalance some vlans on the southern route and some vlans on the northern route with each other as back up.

Because the access interfaces on the cluster are different, this is not stateful redundancy. Which means that if the souther route dies, MSTAG converges, clients will re-establish on the norhtern path and access interface on the BNG.

MCLAG somewhat solves that problem because now I have a single bundle interface, but with an active/standby member , and this can alternative between vlans, to maintan stateful redundancy, but it requires your access nodes to be the POA's. Ugh, I dont like that at all writing that down even, but wanted you to know of the option.

Another potential is using the ASR9000v satellite. It can ring, cascade, dual home and all that.

It solves your spanning tree configuration and relies on the "satellite" discovery protocol to figure out the ring. It is very robust and very simple.

More over, all your interfaces of those satellites are managed out of the asr9000! So it simplifies the design and everything big time.

If that is no option, yeah then you're bound to some sort of ring protection whether it be 8032, REP or MST(AG).

ps. I hope in this case the vendor of hte access node is not cisco if you have to wait a year for a response :)

regards

xander

~~

Xander Thuijs CCIE #6775

Principal Engineer ASR9000/XR

[elnur.mammadov]

Hi Xander

thank you for the reply.

Seamless Redundancy and Load balancing are a concern, in that particular order, I mean I could leave without the former, the final goal is to terminate all sessions in IPoE, PPPoE will be required only for a couple of month for the duration of migration.

Can you please point me to some documents with BNG and 8032/REP/MST configuration snippets, or just 8032/REP/MST, i.e. if there're any, so I could try figure out how to get up BNG on top of them. Most of the documents I found on BNG are provided for bundle-ether interface.

As I understand so far there're a few ways to achieve our requirements, please confirm:

1.    Satellites in the OSP access - a bit expensive for us at the moment, given 1Gbps Access ports, Port Count per rack U/Cost per Port, nevertheless I'm seriously considering it for the second phase of the rollout.

2. Satellites in the distribution for the OSP – BNG cluster in the diagram are replaced with a couple of Satellites and EAPS/MST/8032 does its magic.

3.    MCLAG - did not understand what you meant by POA above, the access plant is L2 and logical POA will be the BNG cluster, please elaborate. MC-LAG is vendor-specific, so how is this going to work? Shall I factor in a couple of Cisco switches that understand MC-LAG and let the EAPS deal with the protection of the ring, or can ASK9001 (with or without EAPS) handle it given the topology I’ve provided? i.e. will I have to change the physical topology from the second diagram provided in my original post.

4.    Somehow, which I’m researching now, to implement one of the 8032/REP/MST on the BNG cluster.

All in all some configuration samples of BNG on top of 8032/REP/MST would really help me get this up and running.

Thank you.

Regards

Elnur

p.s. the vendor is definitely not Cisco:) and I believe Cisco will help overcome shortcomings of EAPS:) 

p.s.s apologize for the lengthy post, and hope it did not bore you too much.

[xthuijs]

hi elnur.

here some detail on MSTAG: https://supportforums.cisco.com/document/61401/asr9000xr-using-mst-ag-mst-access-gateway-mst-and-vpls

BNG doesnt change the story a hell of a lot, because the BNG is just running on a subinterface on the bundle-e (or gige for that matter) which is also part of the STP topology. So the MSTP/MSTAG config we can tie loose from the bng, which makes it easier to setip (in phases for testing) and troubleshooting.

Forget about MCLAG, but in case you want more info go here: https://supportforums.cisco.com/document/9868751/asr9000xr-multichassis-lag-or-mc-lag-mclag-guide

So first setup MSTAG on say a bundle-e100.1 untagged EFP under the spanning-tree mstag YOURSTP section.

Then add the bng configs on say bundle-e100.20 l3 subinterfaces.

cheers!

xander

ps not a problem on the lengty post!

[elnur.mammadov]

hi Xander

do you mean I configure MST/MSTAG on phy interface and later put it into a bundle? 

Thanks

Elnour

[xthuijs]

The decision of using bundle out of the 9k to your switch(es) depends on a few things:

1) bundles on the a9k will pull subscriber control to the RSP and you will cap session scale to 128k TOPs.

2) linecard based subscribers using a phy 10/1G (sub) interface will leave control of the sub on the LC with increasing scale release over release. First support for LC subs is in XR511.

3) the ability for your switches to support bundle (or lacp for that matter).

The MSTAG concept works the same for a gig/tengig or bundle

So you need to first determine whether you want RP or LC based subs.

And if bundle access is desired whether your switch supports LACp or link bundling.

If not and due to eg XR/A9K release restriction you have (For whatever reason), you can always consider a single member bundle and disable lacp on the a9k side so the switch doesnt even know we are making it a bundle.

The MSTAG function will break your ring, but make sure your switches do support MSTP (or PVST which is another alternative if necessary).

regards

xander

[elnur.mammadov]

thank you very much for clarifications.

LACP is out, along with the bundles, my access plant does not support MCLAG.

Now, how do we get Seamless Redundancy and BNG cluster node protection, seamless as in the BNG session id/state is preserved between a cluster node/link failures.

With bundle I kind of get it, a physical interface from each chassis is joined into a bundle, and BNG is running over it, the state of each session is replicated between RSP's of the cluster because of the bundle.

With bundle if I shutdown a any given member streaming video would not even stutter, haven't tested node protection yet, I believe it should work.

As I understand the sessions will have to be reestablished in all cases expect for bundle. Please confirm.

Regards

Elnur

[xthuijs]

Because the ring terminates on different interfaces, and you can't use bundle statefull is out...

XR52 will provide what we call "geo redundancy" this is 2 separate BNG's that sync their subs via a control channel (like iccp-ish) and then you can make that design with the ring you have work.

Another option is to do hub and spoke from the bng to the ring switches, so you dont have a ring anymore, but yeah hub and spoke, this requires more fiber between the switches and the bng's obviously.

redundancy doesn't come for free eh :)

cheers

xander

[elnur.mammadov]

Hi Xander

Thank you for the confirmation. I'll deal with seamless redundancy later:)

Will g.8032 work or MSTAG is a better option? I saw your post where you are saying that g.8032 could work as well. My systems in the OSP do support MSTP to the standard (that's what they claim)

I'm very new to XR (a couple of weeks), the link that you've provided above for MSTAG guide is probably for another version of XR (not sure) as some of the instructions are missing on my system (XR511)

I tried to do g.8032 on my own, following the guide below

http://www.cisco.com/c/en/us/td/docs/routers/asr9000/software/asr9k_r5-1/lxvpn/configuration/guide/lesc51x/lesc51p2mps.html#pgfId-1332653

yet vlan-ids instruction is not available in the context of the ring instance configuration, even if it was, I could not figure out the interface where to configure IP address (e.g. for management access into the OSP nor BNG)

Please advise. Really appreciate your effort!

Rgds,

Elnur

RP/0/RSP0/CPU0:ironman0#show ver
Fri Jun 13 09:02:24.538 UTC

Cisco IOS XR Software, Version 5.1.1[Default]
Copyright (c) 2014 by Cisco Systems, Inc.

ROM: System Bootstrap, Version 2.03(20131022:110718) [ASR9K ROMMON],

ironman0 uptime is 9 weeks, 2 days, 4 hours, 54 minutes
System image file is "bootflash:disk0/asr9k-os-mbi-5.1.1/0x100000/mbiasr9k-rp.vm"

cisco ASR9K Series (P4040) processor with 8388608K bytes of memory.
P4040 processor at 1500MHz, Revision 2.0
ASR9001S Chassis

[xthuijs]

Hi Elnur,

for those couple of days only, you're doing pretty great!!

Say, the concept is here to break the ring, and you generally want some sort of loadbalancing. That means, that there are several options. 8032 is one, MSTP is one.

In order to reduce the load on the root's (the a9k's in this case) the MSTAG is a great concept. it runs MSTP, but only the bear bones: it sends precanned bpdu's out, showing to the ring it is the root. the switches in the ring based on that knowledge will forward some vlans (part of an instance) out on one link and other vlans (part of another instance) out another uplink interface.

Both options are perfectly doable 8032/mstag, they both achieve the same thing, I just happen to feel that MSTAG is probably a bit easier to approach and troubleshoot.

While 8032 is theoretically faster; if you're not concerned about a few msec convergence delay, hey, we're stateless already anyway, when we fail over the session needs to re-establish which by the nature of PPPoE takes seconds already right there.

then my train of thought is, go with the solution that is easiest for you to manage.

cheers

xander

[elnur.mammadov]

Hi Xander

In the last couple of days I've finally got some time to deal with MSTAG and got it up on running with my nodes, let's just say everything seems to be alright on ASR cluster part MSTAG-wise, expect I could not find a way to attach BNG control policy to the routed BVI, I that where am I suppose to configure BNG control, or I shall try without l2vpn configure it individually on the bridge member interfaces(makes sense me, not sure how yet), or else?

please advise where is the right track:)

Thank you, your help is much appreciated!

e.g. VLAN 47 below is suppose to handle IPoE

Regards,

Elnur

spanning-tree mstag ring0-10gbps
 interface TenGigE0/0/0/2.1
  name ring0
  revision 1
  bridge-id 0000.0000.0001
  instance 0
   root-id 0000.0000.0001
   priority 4096
   root-priority 4096
  !
  instance 1
   vlan-ids 20,35,47
   root-id 0000.0000.0001
   priority 4096
   root-priority 4096
  !
  instance 2
   vlan-ids 49
   root-id 0000.0000.0001
   priority 4096
   root-priority 4096
  !
 !
 interface TenGigE1/0/0/2.1
  name ring0
  revision 1
  bridge-id 0000.0000.0002
  instance 0
   root-id 0000.0000.0001
   priority 8192
   root-priority 4096
  !
  instance 1
   vlan-ids 20,35,47
   root-id 0000.0000.0001
   priority 8192
   root-priority 4096
  !
  instance 2
   vlan-ids 49
   root-id 0000.0000.0001
   priority 8192
   root-priority 4096
  !
 !
!

l2vpn
 bridge group ring0
  bridge-domain ring0-ipoe
   transport-mode vlan passthrough
   interface TenGigE0/0/0/2.47
   !
   interface TenGigE1/0/0/2.47
   !
   routed interface BVI47
  !
  bridge-domain ring0-mngmnt
   interface TenGigE0/0/0/2.20
   !
   interface TenGigE1/0/0/2.20
   !
   routed interface BVI20
  !
  bridge-domain ring0-cpe-mnmngt
   interface TenGigE0/0/0/2.49
   !
   interface TenGigE1/0/0/2.49
   !
   routed interface BVI49
  !
 !
!

[xthuijs · Accepted solution]

Hey Elnur,

your finding is correct that BVI doesnt support subscribers. You can either terminate subscribers on the phy (sub) interface or bundle (sub)interface.

In this model with phy interfaces, since you can't run bundle due to your access environment, and you're stuck with those phy's then you have to apply the control policy to the te 0/0/0/2.49 interface.

Right now it is in l2transport mode, because you pulled it in a bridge domain. But if you put it back in l3 mode, then you can apply the control policy to it for the ip session activation. The "gotcha" with this design is, that this is active/standby for the both access interfcaes and the subscriber, when created is tied to that interface. Which means upon switch over it needs to be recreated. (stateless failover)

So what you can do here is potentially set a packet trigger on the standby interface for instance and recreate the subscriber on the unknown source ip to provide more seamless subscriber recreation in case there is ring convergence.

cheers!

xander

[elnur.mammadov]

Thank you for your response! I kind of figured this out myself, and got stuck again, to be honest have not had a chance to dig deeper, and using this opportunity would like to ask.

All the great guides here have:

interface Bundle-Ether100.2 proxy profile AutoSelectGiaddr

here is my problem, bundle-ether nor Phy interfaces on my system do not have 'proxy' option. I hope I'm missing something obvious.

Cisco IOS XR Software, Version 5.1.1[Default] 

Thank you

Elnur

[elnur.mammadov]

Found it

[xthuijs]

hi elnur, ok cool yeah it is under the dhcp ipv4 context, or should be :)

you're all set?

cheers!

xander