on 03-31-2011 08:57 AM
In this document the concept of MST (multiple spanning tree) access gateway and the difference to regular MST will be highlighted. Also it shows some design scenarios of when to chose for which particular implementation.
It is generally well understood that in a layer 2 network loops are disastrous. Spanning-Tree Protocol (STP) aids in the detection of loops and breaking that loop to prevent broadcast and unknown unicast packets for circuling around for forever and bringing down the network eventually.
STP operates by selecting a root bridge in the network, who will have all its ports in forwarding and the other switches determining their path cost towards the root switch.
The way the root switch is elected is by means of a priority and when all are equal the lowest switch mac address is selected to be the root. While in default settings a loop is prevented, for good network design it is important that the priorities are set correctly to define where the root switch (and its potential backup) are going to be located.
Root switch election
The states that a port can be in are determined as follows:
Just as a summary overview the follow topology:
In this example we consider this picture after all BPDU's have been exchanged. Based on priority and mac address S1 is elected as root bridge.
S1 will move both his ports from blocking to forwarding.
Switch S2 will move his root facing port in forwarding mode, which will become the RP (root port).The Port towards S3 will also be in forwarding
because S3, although having a direct link to the root switch S1, the path cost is higher (100) then going via S2->S1 (38). For this reason
the port on S3 to S1 will be blocking.
S4's path to the root bridge is either via S2 or S3. The path to S2 is a higher cost then via S3, ehcne S4->S2 is blocking and the path
to S3 is chosen and that port on S3 will become the RP.
This picture assumes that there are no vlans and just plain ethernet, also this diagram shows a local area network.
What if we want to use multiple vlans or interconnecting this network via a carrier ethernet network to a remote LAN?
In the scenario where you have multiple vlans, regular STP will block the link for all vlans. While this prevents the loop, it is maybe not that
efficient as one node/path is completely in standby mode. It might be nice to forward a few vlans over to switch 1 and the others to switch 2.
Effectively that means that Sw1 is then root for a vlan set and Sw2 for another vlan set.
Regular STP cannot do this, and the logical evolution of htat is MSTP (multiple spanning tree) which is hte more standardized version, and PVST(+) which is a cisco proprietary solution.
They effectively achieve the same PVST and MSTP, one of the key differences is that MSTP sends the bpdu's out untagged on the port, where
PVST sends the bpdu's inline with the vlan, hence are vlan tagged.
The most common way to connect 2 separate l2 segments or networks together is via VPLS.
With VPLS the edge nodes from the provider are aggregating the customers L2 traffic, and participate in the L2 spanning tree loop prevention as well as Pseudo Wires over an MPLS core to remote PE's (Provider Edge) to bring the traffic from one segment to the remote site.
The way VPLS works and the interaction with MST or MSTAG and what the differences are will be discussed below.
MST allows us then to run an STP instance per set of vlans that we can configure.
A sample configuration on the ASR9000 PE node looks as follows. Each set of vlans is defined under an instance.
We can adjust the priority per instance if needed, multiple vlans per instance are allowed also.
Sample MST configuration on the ASR9000
spanning-tree mst MYSTP_DOMAIN
name testme
! The name of the MST region is very important, it must be the same for all switches in this region.
! also the definitions of your MST instances need to be the same on all nodes.
revision 1
instance 0
priority 4096
!
instance 1
vlan-ids 100
priority 4096
!
instance 2
vlan-ids 101
priority 4096
!
interface TenGigE0/3/0/6
! Interfaces that are enabled for MST. Note that these are the main interfaces
interface TenGigE0/3/0/7
!
interface Bundle-Ether100
!
interface GigabitEthernet0/0/0/27
!
!
To provide a more graphical example of how MST can be used with 9k's is shown here:
In this case we have a clear STP loop between the 2 9K PE devices interconnceted via a bundle ethernet.
Associated configuration for this example would be as follows:
spanning-tree mst Example
name testme
revision 1
instance 0
priority 4096
!
instance 1
vlan-ids 100
priority 4096
!
interface Bundle-Ether100
!
interface GigabitEthernet0/0/0/27
!
With this config we are providing the ability to the 9k PE to send bpdu's out to the other 9k and the access switch.
Also with this config, one link will be blocked and if we elect either 9k to be the root switch (say the one on the left), the link marked RED will be blocked.
This config however doesn't provide for any data configuration forwarding. For this we need to establish a separate bridge-domain whereby we pull in the right EFP's for forwarding the data traffic:
interface bundle-e100.100 l2transport
encapsulation dot1q 100
rewrite ingress tag pop 1 symmetric
!rewrite is optional depending on whether all EFP's are in the same vlan.
interface gigabitethernet 0/0/0/27.100 l2transport
encapsulation dot1q 100
rewrite ingress tag pop 1 symmetric
!rewrite is optional depending on whether all EFP's are in the same vlan.
l2vpn
bridge group EXAMPLE
bridge-domain FWD_1
interface g0/0/0/27.100
interface bundle-e100.100
You need to repeat this configuration for every vlan you want to forward. There is another article detailing more about vlan rewrites and the EFP concept in case you're interested. See the related documentation section for a reference.
VPLS is the concept of connecting multiple layer 2 domains over an MPLS network for instance. On the ASR9000, a bridge-group is used to pull in the attachment circuits (physical interfaces towards the lan segment) and Pseudo Wires (PW) to the remote PE's.
A sample configuration achieve vpls looks as follows, this provides the configuration for the data plane and assumes there are no loops in your L2 topology either at the customer access site or within your VPLS domain.
To prevent loops in the access network we need to leverage either MSTP, MSTAG or PVSTAG. We'll discuss MSTAG in the next section below.
l2vpn
bridge group VPLS
bridge-domain vpls_1
! the bridge-group vs domain is just a configuration hierarchy, it doesn't serve any special functionality.
interface GigabitEthernet0/0/0/0.100
! Phyiscal interfaces towards a subscriber switch
neighbor 1.1.2.3 pw-id 123
! for H-VPLS we can use PW's also as an attachment circuit
vfi vpls_1_vfi_1
neighbor 5.5.5.5 pw-id 333
! definition of a pseudo wire underneath a Virtual Forwarding Instance
neighbor 6.6.6.6 pw-id 444
!
!
!
!
!
end
The IP address providing in the "neighbor" statement are the MPLS router ID's from the remote PE's. the PW-ID is an arbitrary number, unique, that defines the VC label.
Whether you put the PW's in the VFI or outside the VFI or across VFI's depends on your needs and whether you need SPLIT HORIZON (see below).
The following picture explains what might happen when we don't use any STP in a VPLS scenario.
In this case there is a loop, but the access switch doesn't know about it because both the 9k PE's, nor the switches
form a closed ring.
Even if we'd be running MSTP in this scenario there is no loop detected, since by default BPDU's are not forwarded over the pseudo wires.
A potential solution might be to run an L2 link between the 2 southern PE's so that there is a loop on the SOUTH segment and indeed one UP link will be blocked from the access switch to one of the 2 PE's as per regular (M)STP.
The problem is here however that a (broadcast/unknown unicast) packet arriving on the Left South PE's pseudo wire is now then sent to the access switch south AND over the interchassis link (not drawn in this picture) to the SOUTH PE on the right. There will be a loop again.
A proper solution for this model is the use of MST Access gateway which will be highlighted below.
Normally in a bridge domain, broadcast and unknown unicast from Attachment circuits are replicated to all bridge ports.
Obviously packets are never sent to the AC or PW that the traffic was actually received on.
By default AC's can always forward packets to each other and to (all) Pseudo Wires.
So traffic from the AC "west" will be replicated over all PW's and the South-West AC.
When traffc arrives on a PW then by default packets are never sent out the PW that they are received on and to other Pseudo Wires in the same
VFI. All PW's in the same VFI share the same split horizon group. And traffic is not replicated within the same split horizon group.
When packets arrive on the Pseudo Wire they are NOT forwarded out of PW's in the same VFI.
You can also move Attachment Circuits into a split horizon group to prevent them from speaking with each other by means of the "split horizon" group command underneath the interface which is configured in the l2vpn bridge-domain.
l2vpn
bridge group TEST
bridge-domain SAMPLE
interface g0/1/0/0
split-horizon
interface g0/1/0/10
split horizon
interface te0/2/0/3
vfi VFI_TEST
neighbor 2.2.2.2 pw-id 100
In this case traffic cannot flow between 0/1/0/0 and 0/1/0/10.
For illustrational purposes consider this sample VPLS design.
In order for MST to work, we need an inter PE attachment circuit to exchange the BPDU's between the two PE nodes drawn in blue.
In this given example the PE_left (PE-1) is considered ROOT.
Imagine there is a broadcast coming in on PW-1. Because of split horizon the traffic is not replicated to the PW 2, 3, 4 and 5. But traffic will go down the AC-2 and also sent over to AC-1.
When traffic enters the PE_right (PE-2), it will not go down AC-3 because it is blocking since PE-1 is root, but it will enter the VFI and gets replicated to the PW's in the VFI there so PW 3, 4 and 5. This poses a big problem considering with a replication loop back to PE-3 and also to PE-1.
Omitting PW-5 solves part of the issue so that traffic is not replicated back to PE-1, but it might slow down convergence in case AC-2 is going offline and PE3/4 have not yet updated their mac tables yet.Traffic will still get back to PE-3 and PE-4.
The AC-1 is required for BPDU, and you might want to consider only creating an EFP for hte untagged traffic (BPDU's), but then you might have also forwarding issues in case the PW 1 and 2 are down and you want to send traffic over the AC 1 to PE-right so it can forward the traffic for us to the PE3 and 4.
Wouldn't it be nice if we can live without the AC-1 all together, still run spanning-tree to the CE and have optimum convergence?
Yes. Enter MSTAG.
In MSTAG we define the BPDU's on the PE nodes statically presenting them as 1 virtual bridge to the CE.
One link will be blocked, but there is no need for an inter chassis link anymore in this case.
What’s the main function of the MST access gateway?
Major Advantage – scale and local significant
Light MST implementation, for example, it doesn’t keep STP state machine, it doesn’t need to handle received BPDU (except TCN)
The MST is per port scope
Other Advantages
Doesn’t require inter-PE special PW, no single point of failure, no temp L2 loop.
Much robust than the “MST over special PW” solution
Standard based solution, inter-operable with 3rd vendors, work with any network topology
Self protection, even with user mis-configuration, it won’t cause L2 loops
Disadvantages
MST convergence depends on the number of VLANs in the access ring and the MST implementation of the access switches. In any case, don't expect 50msec convergence time
With Cisco 3400 as access switch, the baseline convergence show sub second for link failure, sub 100msec for link recovery, 2-3 seconds for node failure
Note that in this configuration we use the interface with suffix .1 in the MSTAG configuration.
This means we need to define an EFP (Ethernet Flow Point) to capture the BPDU's and TCN packets. In fact, we're not even using the bpdu's received, as we perceive ourselves to be root on the 9k and send these precanned BPDU's out.
We will consume the TCN (topology change notification) and send these into the VPLS network as mac withdrawl messages.
interface gigabitEthernet 0/0/0.10.1 l2transport
encapsulation untagged
Aside from the MST configuration we still need to configure our bridge domains with the EFP's for the data forwarding and our Pseudo Wires to our remote PE's as described above.
1) MSTP: There is a single protocol instance which can have the
standard 64 MST Instances (MSTIs) within it. These 64 MSTIs create 64
logical spanning tree topologies within one MSTP region/ domain.
2) MSTAG: You can create a separate protocol instance per physical
interface and each each protocol instance can be in a separate MSTP
region by itself and each one can in turn support 64 MST Instances
(MSTIs) within it.
In general MSTAG is more scalable (multiple regions with 64 MSTIs each)
but can only be used if the ASR9K is in the root (or backup root)
position for every MSTI. MSTP is the normal Cat 6K like version but you
can use all 64 MSTIs without any issues. Both of these can interoperate
with any IEEE standard MSTP implementation so should work with the N7K
VDCs.
For more details on regular MST and some IOS interoperability considerations, check this reference:
ASR9000 MST interop with IOS/7600: VLAN pruning
Learn more about vlan rewrites and the concept of EFP's
ASR9000/XR Flexible VLAN matching, EVC, VLAN-Tag rewriting, IRB/BVI and defining L2 services
Xander Thuijs, CCIE #6775
Sr Tech Lead ASR9000
Hi Philippe, your design screams for REP-AG :)
yeah here is a great usecase for the rep access gateway to support that rep ring.
your core is I assume MPLS enabled, so you'll have PW's between teh different nodes to support the vlans in the ring.
although you dont need mpls/pw's to support the AG component, it makes more sense when you do in case of this core/access design.
cheers!
xander
Thanks for your feedback Alex,
yes backbone is mpls ready, and we would like to transport vlans from rep ring to some switchs attach to the core backbone by using PW.
just to confirm, in case of link down on REP side all traffic will pass throught core ? do i need specific configuration for that ?
thanks
Philippe
Hi Xander,
A great discussion on ST-AGs, and still very relevant. I cant help but wonder though, what the solution would be to loop avoidance in a VPLS where the EFP is double-tagged and you need to influence the STP within the inner-tag?
The use case would be to consider the scenario where there is another telco in the middle between your CPE and PE doing the physical connectivity between site and SP, and the telco pushes its S-VLAN on for identification of the B-end. The C-VLAN is where I need to influence blocking (imagine in the diagram multiple copies of this C-VLAN 10 on other PEs linking back to the same site).
MSTAG appears to fail because the canned BPDUs are sent untagged so they wont pass through the telco network. PVSTAG also appears to fail as the canned BPDUs will get through the telco network to site, but pop out at the site end with the S-VLAN info, not C-VLAN, and not tagged with the C-VLAN when it reaches CPE. MCLAG appears to be one way to fill some shortcomings by trying to make it loop-free, but depends on the CPE being just a single (or stacked) device.
All the best!
Sandy
hi sandy thanks:) and you are correct that stp in general is not too keen on vlan manipulations like this.
both mstag and pvst have an issue here. also stp would run on the outer vlan only. Since the vlan is not end to end, and the s vlan is left on the nte to agg side, possibly you could strip the s tag at the nte sides and impose it there.
or request a pw between the nte's.
the fact that the second provider is carrying it with some encap is not a problem per-se, but it needs to be transparent between CPE and AGG.
mc lag is an option, the cpe doesnt even know it is talking to two separate devices so any cpe that supports link bundling will do.
cheers!
xander
Hi Xander!
I´d would like to know the behaviour of MST-AG if you connect, by mistake for example, the two interfaces of the Access Gateways. I mean, bypassing the access switches. MST is able to protect the network in this rare situation, and I´m wondering if MST-AG can, too. Have you tested it?
Thanks in advance!
Max
hi max,
in mstag mode, the AN runs a "lite" version of STP where it only pays attention to TCN and converts that into a mac flush.
so if you make that cabling error and the AN gets mgmt BPDU's in, it would just toss em.
moreover, the bpdu's that the AN would sent are "pre-crafted", and they are the same for both AN's. so it looks to the switch if it were to pay attention to it that is connected to itself.
cheers!
xander
OK, Xander! Thanks for your soon response. So, in this situation you will have a layer 2 loop in the network, right ? Do yo think that MST-AG will improve this in the future ?
Thanks again!
Max
hi max! mstag will not improve here; it is the concept of that it perceives itself to be root only working and using TCN this so the implementation is "lite" meaning higher scale and better/faster convergence.
If you like to protect yourself against such a cabling error, probably better to run full MSTP and possibly use root/bpdu guard as additional protection, but this was not meant to be part of the mstag implementation.
cheers!
xander
hi tmlplicensing! ah thanks :) so hey the MSTP implementation is compatible with the RTSP (dot1-D-2004) so you should be fine there.
If the LC is rebooting, that is not good, loop or not, we need to look into that further. I think, if there are crashes, it would be good to have a tac case so we can verify the sw, config and all that a bit easier then through the forums here possibly and give you some dedicated assistance.
Both RSTP and 8032 should work and are supported.
I think I need to post some documentation on 8032, I'll get on it.
cheers!
xander
Hi Xander,
Great article, very insightful! I'd like to ask you a question regarding the above, I currently have MSTP running in an IOS like enviroment with 2 ME3600X's(QinQ) as the root bridge and backup root bridge which I'm looking to move away from and implement an ASR9006 and provide a layer 2 service to the client only however would this work? Basically create a QinQ L2 tunnel to the edge device.
interface Gig0/0/1/0.2155 l2transport
descrip Client1
encapsulation dot1q 100
rewrite ingress tag pop 1 symmetric
!
interface Gig0/0/1/0.2156 l2transport
descrip Client2
encapsulation dot1q 101
rewrite ingress tag pop 1 symmetric
!
l2vpn
!
bridge group SP
bridge-domain 2155
interface Gig0/0/1/0.2155
!
bridge-domain 2156
interface Gig0/0/1/0.2156
!
hi calebmo! in this design from this configlet, what will happen is that any outer vlan of 100 or 101 maps to the EFP (gig subif) and it could have any number of inner vlans. we pop the outer vlan in this case. But since the bridge domain has only one EFP and split horizon would prevent a packet incoming on an interface to get out of it, nothing will happen when packets are received.
So you need something else in the bridge domain to move packets somewhere. eg a pseudo wire, or another EFP.
if you have a vlan combo like 100/200 and 100/300 you probably need to define 2 EFP's for those vlan sets, pop both 2 tags and put these 2 EFP's in a bridge domain.
Basically for 2 endpoints you can use a xconnect, for more then 2 you will need a bridge domain.
xcon is faster, because it omits all the mac learning.
xander
Thanks again for your reply! So if I were to disable split horizon only on the sub interface ? that would allow it act like ME3600X a service instance under the sub-interface ?
technically yes, but end to end it won't work:
if you receive a vlan 100/200 packet, we strip 100, and we loop that back out, it gets again 100/200. a device in vlan 300 wont interpret that packet.
if you want to link 100/200 and 100/300 together, you need to do this:
int g 0/0/0/0.100200
encap dot1q 100 sec 200
rewr ing tag pop 2
int g 0/0/0/0.100300
encap dot1q 100 sec 300
rewr ing tag pop 2
l2vpn xcon p2p
int g 0/0/0/0.100200
int g 0/0/0/0.100300
xander
Thanks for your response, much appreciated. What if I were to only match the outer tag ? because the inner tag would be the same for every client so I'd only match the outer tag and pop it however client 1 and 2 should never see each other while maintaining a L2 service because effectively a QinQ tunnel is built between the CE and NNI(facing toward SP).
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: