topic Thank you Xander, I was in XR OS and Platforms

BNG cluster - redundancy

elnur.mammadov — Tue, 10 Jun 2014 00:38:49 GMT

Hello everyone,

What would be the best way to achieve link/cluster redundancy into the access plant given the second diagram?

What is the best practice for redundant BNG configuration?

Cluster is a couple of ASR9001 with a 2 port 10gbps line card in each, and access nodes can be considered third party L2 switches chain linked to each other over 10 gpbs ( this is the OSP for FTTx ).

BNG (experimental, both PPPoE and IPoE work) on a bundle-ether interface works, members of the bundle-ether are east and south interfaces from distinct cluster members, haven't tested cluster redundancy yet.

EAPS Ring redundancy with the setup #1 works but it has a single point of failure, the access node itself with the up links.

In the second setup with the TLS (Basically an L2 bridge without any protection protocol, none) bundle-ether link redundancy only works when links directly connected to the cluster go down. As per Cisco papers, bundle-ether interface are for point-to-point links, so it is not supposed to work if an intermediate link goes does down, I understand that, there's no signalization.

I guess my question is how would you suggest to implement link redundancy in setup 2 with TLS, thus eliminate the single point of failure of EAPS setup, I've already contacted the vendor of the access nodes on the subject, awaiting their response, it has been a little over an year.

Thank you.

http://en.wikipedia.org/wiki/Ethernet_Automatic_Protection_Switching

Regards

Elnur

Hi Elnur,Because you have an

xthuijs — Wed, 11 Jun 2014 13:33:04 GMT

Hi Elnur,

Because you have an L2 loop between these access nodes, and it is a ring, I dont see any other options then:

MST(AG) or MCLAG.

This to break the loop and loadbalance some vlans on the southern route and some vlans on the northern route with each other as back up.

Because the access interfaces on the cluster are different, this is not stateful redundancy. Which means that if the souther route dies, MSTAG converges, clients will re-establish on the norhtern path and access interface on the BNG.

MCLAG somewhat solves that problem because now I have a single bundle interface, but with an active/standby member , and this can alternative between vlans, to maintan stateful redundancy, but it requires your access nodes to be the POA's. Ugh, I dont like that at all writing that down even, but wanted you to know of the option.

Another potential is using the ASR9000v satellite. It can ring, cascade, dual home and all that.

It solves your spanning tree configuration and relies on the "satellite" discovery protocol to figure out the ring. It is very robust and very simple.

More over, all your interfaces of those satellites are managed out of the asr9000! So it simplifies the design and everything big time.

If that is no option, yeah then you're bound to some sort of ring protection whether it be 8032, REP or MST(AG).

ps. I hope in this case the vendor of hte access node is not cisco if you have to wait a year for a response 🙂

regards

xander

Xander Thuijs CCIE #6775

Principal Engineer ASR9000/XR

Hi Xander thank you for the

elnur.mammadov — Thu, 12 Jun 2014 16:34:41 GMT

Hi Xander

thank you for the reply.

Seamless Redundancy and Load balancing are a concern, in that particular order, I mean I could leave without the former, the final goal is to terminate all sessions in IPoE, PPPoE will be required only for a couple of month for the duration of migration.

Can you please point me to some documents with BNG and 8032/REP/MST configuration snippets, or just 8032/REP/MST, i.e. if there're any, so I could try figure out how to get up BNG on top of them. Most of the documents I found on BNG are provided for bundle-ether interface.

As I understand so far there're a few ways to achieve our requirements, please confirm:

1. Satellites in the OSP access - a bit expensive for us at the moment, given 1Gbps Access ports, Port Count per rack U/Cost per Port, nevertheless I'm seriously considering it for the second phase of the rollout.

Satellites in the distribution for the OSP – BNG cluster in the diagram are replaced with a couple of Satellites and EAPS/MST/8032 does its magic.

3. MCLAG - did not understand what you meant by POA above, the access plant is L2 and logical POA will be the BNG cluster, please elaborate. MC-LAG is vendor-specific, so how is this going to work? Shall I factor in a couple of Cisco switches that understand MC-LAG and let the EAPS deal with the protection of the ring, or can ASK9001 (with or without EAPS) handle it given the topology I’ve provided? i.e. will I have to change the physical topology from the second diagram provided in my original post.

4. Somehow, which I’m researching now, to implement one of the 8032/REP/MST on the BNG cluster.

All in all some configuration samples of BNG on top of 8032/REP/MST would really help me get this up and running.

Thank you.

Regards

Elnur

p.s. the vendor is definitely not Cisco:) and I believe Cisco will help overcome shortcomings of EAPS:)

p.s.s apologize for the lengthy post, and hope it did not bore you too much.

Xander, Google found this

elnur.mammadov — Thu, 12 Jun 2014 16:35:36 GMT

Xander,

Google found this http://www.ciscoknowledgenetwork.com/files/96_03-15-11_BNG_evolution_seminar_v0.9.pdf

on the page 19 I was very happy to see the attached snipped:)

Regards

Elnur

hi elnur.here some detail on

xthuijs — Thu, 12 Jun 2014 16:39:14 GMT

hi elnur.

here some detail on MSTAG: https://supportforums.cisco.com/document/61401/asr9000xr-using-mst-ag-mst-access-gateway-mst-and-vpls

BNG doesnt change the story a hell of a lot, because the BNG is just running on a subinterface on the bundle-e (or gige for that matter) which is also part of the STP topology. So the MSTP/MSTAG config we can tie loose from the bng, which makes it easier to setip (in phases for testing) and troubleshooting.

Forget about MCLAG, but in case you want more info go here: https://supportforums.cisco.com/document/9868751/asr9000xr-multichassis-lag-or-mc-lag-mclag-guide

So first setup MSTAG on say a bundle-e100.1 untagged EFP under the spanning-tree mstag YOURSTP section.

Then add the bng configs on say bundle-e100.20 l3 subinterfaces.

cheers!

xander

ps not a problem on the lengty post!

hi Xanderdo you mean I

elnur.mammadov — Thu, 12 Jun 2014 19:17:49 GMT

hi Xander

do you mean I configure MST/MSTAG on phy interface and later put it into a bundle?

Thanks

Elnour

The decision of using bundle

xthuijs — Thu, 12 Jun 2014 19:41:22 GMT

The decision of using bundle out of the 9k to your switch(es) depends on a few things:

1) bundles on the a9k will pull subscriber control to the RSP and you will cap session scale to 128k TOPs.

2) linecard based subscribers using a phy 10/1G (sub) interface will leave control of the sub on the LC with increasing scale release over release. First support for LC subs is in XR511.

3) the ability for your switches to support bundle (or lacp for that matter).

The MSTAG concept works the same for a gig/tengig or bundle

So you need to first determine whether you want RP or LC based subs.

And if bundle access is desired whether your switch supports LACp or link bundling.

If not and due to eg XR/A9K release restriction you have (For whatever reason), you can always consider a single member bundle and disable lacp on the a9k side so the switch doesnt even know we are making it a bundle.

The MSTAG function will break your ring, but make sure your switches do support MSTP (or PVST which is another alternative if necessary).

regards

xander

thank you very much for

elnur.mammadov — Thu, 12 Jun 2014 21:38:49 GMT

thank you very much for clarifications.

LACP is out, along with the bundles, my access plant does not support MCLAG.

Now, how do we get Seamless Redundancy and BNG cluster node protection, seamless as in the BNG session id/state is preserved between a cluster node/link failures.

With bundle I kind of get it, a physical interface from each chassis is joined into a bundle, and BNG is running over it, the state of each session is replicated between RSP's of the cluster because of the bundle.

With bundle if I shutdown a any given member streaming video would not even stutter, haven't tested node protection yet, I believe it should work.

As I understand the sessions will have to be reestablished in all cases expect for bundle. Please confirm.

Regards

Elnur

Because the ring terminates

xthuijs — Fri, 13 Jun 2014 11:41:32 GMT

Because the ring terminates on different interfaces, and you can't use bundle statefull is out...

XR52 will provide what we call "geo redundancy" this is 2 separate BNG's that sync their subs via a control channel (like iccp-ish) and then you can make that design with the ring you have work.

Another option is to do hub and spoke from the bng to the ring switches, so you dont have a ring anymore, but yeah hub and spoke, this requires more fiber between the switches and the bng's obviously.

redundancy doesn't come for free eh 🙂

cheers

xander

Hi XanderThank you for the

elnur.mammadov — Fri, 13 Jun 2014 17:06:40 GMT

Hi Xander

Thank you for the confirmation. I'll deal with seamless redundancy later:)

Will g.8032 work or MSTAG is a better option? I saw your post where you are saying that g.8032 could work as well. My systems in the OSP do support MSTP to the standard (that's what they claim)

I'm very new to XR (a couple of weeks), the link that you've provided above for MSTAG guide is probably for another version of XR (not sure) as some of the instructions are missing on my system (XR511)

I tried to do g.8032 on my own, following the guide below

http://www.cisco.com/c/en/us/td/docs/routers/asr9000/software/asr9k_r5-1/lxvpn/configuration/guide/lesc51x/lesc51p2mps.html#pgfId-1332653

yet vlan-ids instruction is not available in the context of the ring instance configuration, even if it was, I could not figure out the interface where to configure IP address (e.g. for management access into the OSP nor BNG)

Please advise. Really appreciate your effort!

Rgds,

Elnur

RP/0/RSP0/CPU0:ironman0#show ver
Fri Jun 13 09:02:24.538 UTC

ROM: System Bootstrap, Version 2.03(20131022:110718) [ASR9K ROMMON],

ironman0 uptime is 9 weeks, 2 days, 4 hours, 54 minutes
System image file is "bootflash:disk0/asr9k-os-mbi-5.1.1/0x100000/mbiasr9k-rp.vm"

cisco ASR9K Series (P4040) processor with 8388608K bytes of memory.
P4040 processor at 1500MHz, Revision 2.0
ASR9001S Chassis

Hi Elnur,for those couple of

xthuijs — Sat, 14 Jun 2014 15:34:12 GMT

Hi Elnur,

for those couple of days only, you're doing pretty great!!

Say, the concept is here to break the ring, and you generally want some sort of loadbalancing. That means, that there are several options. 8032 is one, MSTP is one.

In order to reduce the load on the root's (the a9k's in this case) the MSTAG is a great concept. it runs MSTP, but only the bear bones: it sends precanned bpdu's out, showing to the ring it is the root. the switches in the ring based on that knowledge will forward some vlans (part of an instance) out on one link and other vlans (part of another instance) out another uplink interface.

Both options are perfectly doable 8032/mstag, they both achieve the same thing, I just happen to feel that MSTAG is probably a bit easier to approach and troubleshoot.

While 8032 is theoretically faster; if you're not concerned about a few msec convergence delay, hey, we're stateless already anyway, when we fail over the session needs to re-establish which by the nature of PPPoE takes seconds already right there.

then my train of thought is, go with the solution that is easiest for you to manage.

cheers

xander

Hi Xander In the last couple

elnur.mammadov — Mon, 21 Jul 2014 06:57:01 GMT

Hi Xander

In the last couple of days I've finally got some time to deal with MSTAG and got it up on running with my nodes, let's just say everything seems to be alright on ASR cluster part MSTAG-wise, expect I could not find a way to attach BNG control policy to the routed BVI, I that where am I suppose to configure BNG control, or I shall try without l2vpn configure it individually on the bridge member interfaces(makes sense me, not sure how yet), or else?

please advise where is the right track:)

Thank you, your help is much appreciated!

e.g. VLAN 47 below is suppose to handle IPoE

Regards,

Elnur

spanning-tree mstag ring0-10gbps
interface TenGigE0/0/0/2.1
name ring0
revision 1
bridge-id 0000.0000.0001
instance 0
root-id 0000.0000.0001
priority 4096
root-priority 4096
!
instance 1
vlan-ids 20,35,47
root-id 0000.0000.0001
priority 4096
root-priority 4096
!
instance 2
vlan-ids 49
root-id 0000.0000.0001
priority 4096
root-priority 4096
!
!
interface TenGigE1/0/0/2.1
name ring0
revision 1
bridge-id 0000.0000.0002
instance 0
root-id 0000.0000.0001
priority 8192
root-priority 4096
!
instance 1
vlan-ids 20,35,47
root-id 0000.0000.0001
priority 8192
root-priority 4096
!
instance 2
vlan-ids 49
root-id 0000.0000.0001
priority 8192
root-priority 4096
!
!
!

l2vpn
bridge group ring0
bridge-domain ring0-ipoe
transport-mode vlan passthrough
interface TenGigE0/0/0/2.47
!
interface TenGigE1/0/0/2.47
!
routed interface BVI47
!
bridge-domain ring0-mngmnt
interface TenGigE0/0/0/2.20
!
interface TenGigE1/0/0/2.20
!
routed interface BVI20
!
bridge-domain ring0-cpe-mnmngt
interface TenGigE0/0/0/2.49
!
interface TenGigE1/0/0/2.49
!
routed interface BVI49
!
!
!

Hey Elnur,your finding is

xthuijs — Mon, 21 Jul 2014 11:43:04 GMT

Hey Elnur,

your finding is correct that BVI doesnt support subscribers. You can either terminate subscribers on the phy (sub) interface or bundle (sub)interface.

In this model with phy interfaces, since you can't run bundle due to your access environment, and you're stuck with those phy's then you have to apply the control policy to the te 0/0/0/2.49 interface.

Right now it is in l2transport mode, because you pulled it in a bridge domain. But if you put it back in l3 mode, then you can apply the control policy to it for the ip session activation. The "gotcha" with this design is, that this is active/standby for the both access interfcaes and the subscriber, when created is tied to that interface. Which means upon switch over it needs to be recreated. (stateless failover)

So what you can do here is potentially set a packet trigger on the standby interface for instance and recreate the subscriber on the unknown source ip to provide more seamless subscriber recreation in case there is ring convergence.

cheers!

xander

Thank you for your response!

elnur.mammadov — Mon, 21 Jul 2014 12:49:40 GMT

Thank you for your response! I kind of figured this out myself, and got stuck again, to be honest have not had a chance to dig deeper, and using this opportunity would like to ask.

All the great guides here have:

interface Bundle-Ether100.2 proxy profile AutoSelectGiaddr

here is my problem, bundle-ether nor Phy interfaces on my system do not have 'proxy' option. I hope I'm missing something obvious.

Cisco IOS XR Software, Version 5.1.1[Default]

Thank you

Elnur

Found it

elnur.mammadov — Tue, 22 Jul 2014 06:40:36 GMT

Found it

hi elnur, ok cool yeah it is

xthuijs — Tue, 22 Jul 2014 15:36:16 GMT

hi elnur, ok cool yeah it is under the dhcp ipv4 context, or should be 🙂

you're all set?

cheers!

xander

Hi Xander,Not quite set yet,

elnur.mammadov — Wed, 23 Jul 2014 01:07:57 GMT

Hi Xander,

Not quite set yet, how did you know?:) if you do not mind, I have more problems and a few questions:

I'm moving forward, overall XR to me seems more intuitive, ones you get used to it:)

problem: if the the ring is open in the middle, both nodes must play, everything works as expected on one interface, does not on the other, please take a look below, the only difference is that one of them is on LC of one chassis the other one is on LC of the other (this is ASR9001S nV cluster), local dhcp server shows the session stuck in INIT_DPM_WAIT state. I thought it is because of the shared pool (some sort of protection against split brain) and downed the working interface, did not help. Is there some particular reason that you know of that might be causing this?

How do I debug IPOE sessions? I googled all I could, no luck.

interface TenGigE1/0/0/2.47

description rign0-ipoe-west
ipv4 point-to-point
ipv4 unnumbered Loopback20
service-policy type control subscriber ES_IPOE_PM0
shutdown
encapsulation dot1q 47
ipsubscriber ipv4 l2-connected
initiator dhcp
!
interface TenGigE0/0/0/2.47

description rign0-ipoe-east
ipv4 point-to-point
ipv4 unnumbered Loopback20
service-policy type control subscriber ES_IPOE_PM0
encapsulation dot1q 47
ipsubscriber ipv4 l2-connected
initiator dhcp
!
!

Thank you.

Elnur

Dear Xander,please advice,

elnur.mammadov — Wed, 23 Jul 2014 01:07:58 GMT

Dear Xander,

please advice, what is INIT_DPM_WAIT state, the problem I have looks like L2 issue of the ring equipment, ASR is doing great! anyhow please let me know if you suspect(know) anything is wrong with ASR in this regards - ignore otherwise:) (I'm already planning procurement of CPT 200 based ring)

About debugging IPOE sessions, still stands, please advise.

IOS ISG based portal we developed is using portbundle to identify redirected (captured) sessions, as I understand there is no portbundle support in XR (increased scalability?). I need option82 for automation of the service provisioning in the billing system, given client IP extracting it from accounting records is trivial, the only problem that I see is with VRF's (client IP collision), a portal per VRF could be used to avoid this collision (additional interfaces), have not thought it through yet. Do you see any problems with this? please advise.

Is there another document besides this with iEdge (CoA) commands?

Thank your very much!

Elnur

This is what DPM debug saysLC

elnur.mammadov — Thu, 24 Jul 2014 03:18:48 GMT

This is what DPM debug says

LC/0/0/CPU0:Jul 24 08:14:37.759 : dhcpd[153]: DPM INTERNAL: TP13: DPM Session create called for client DHCPV4
LC/0/0/CPU0:Jul 24 08:14:37.759 : dhcpd[153]: DPM INTERNAL: TP14: DPM Session create params client DHCPV4: chaddr 3a02.7138.f050chaddr_len 6, client_id , client_id_len 0, parentIfHandle 0x4000740 (67110720)
LC/0/0/CPU0:Jul 24 08:14:37.759 : dhcpd[153]: DPM INTERNAL: TP15: Session create params client DHCPV4 circuit_id 31302E3139322E36342E32303A312D392D332D302D6574682D34373A2D3437, circuit_id_len 31
LC/0/0/CPU0:Jul 24 08:14:37.759 : dhcpd[153]: DPM INTERNAL: TP38: Session create params client DHCPV4 vendor_id , vendor_id_len 0
LC/0/0/CPU0:Jul 24 08:14:37.759 : dhcpd[153]: DPM INTERNAL: TP57: Session create params client DHCPV4 port 2 chan 0subif 47 rack 0 slot 0 instance 8
LC/0/0/CPU0:Jul 24 08:14:37.759 : dhcpd[153]: DPM INTERNAL: TP66: Session create params client DHCPV4 nas port type 22tag_count 1 outer tag 47 inner tag 0
LC/0/0/CPU0:Jul 24 08:14:58.041 : dhcpd[153]: DPM INTERNAL: TP50: Session start response callback received with sub_label = 0x4000014 (67108884), client = DHCPV4 ctx = 0x3001351 (50336593), result = 'AAA_BASE' detected the 'fatal' condition 'Invalid state (aaa base lib error)', trans_id = 0x1379 (49
85)
LC/0/0/CPU0:Jul 24 08:14:58.041 : dhcpd[153]: DPM ERROR: TP7: Session start response for sub_label 0x4000014 (67108884) client DHCPV4result is not ok: 'AAA_BASE' detected the 'fatal' condition 'Invalid state (aaa base lib error)'
LC/0/0/CPU0:Jul 24 08:14:58.042 : dhcpd[153]: DPM INTERNAL: TP119: Session disconnect called for client DHCPV4 reason Session start failure
LC/0/0/CPU0:Jul 24 08:14:58.042 : dhcpd[153]: DPM ERROR: TP36: Invalid sub_label passed to disconnect sessionfor client DHCPV4

I think this is a known issue

xthuijs — Thu, 24 Jul 2014 18:24:18 GMT

I think this is a known issue. Can you recover from this scenario by doing a proc restart on the dhcp process on LC0 and the RP?

If so, you may be hitting CSCuo78296 , it doesnt have any release note yet, but I'll check that out. Try the proc restart first.

Besides the link you have found for the VSA/attributes, there is not much else today, however our doc group is putting some effort in documenting this properly, so you'll likely see it appearing on CCO at some point soon.

regards

xander