Extending VLAN across remote WAN sites

[spfister336]

We have several remote sites in a hub-and-spoke arrangement (central site: 9500 pair, remote sites: 4500X pair). Remote sites are connected back to the central site via the AT&T ASE on demand service.

For each remote site, there is exactly one VLAN we need to extend back to the central site for the next several months (i.e., a different VLAN per remote site; each VLAN is not being used at more than one remote site). This involves AT&T learning MAC addresses from us from one end to pass to the other end. This is working as is, but we've been told there may be a limited capacity (and very high fee) to increase the number of MAC addresses they learn (from the default 250). We would like to look at alternatives, maybe a GRE tunnel. Just something where AT&T doesn't have to learn individual MACs. What is the best and simplest solution? We may need to implement something in the very near future.

[Joseph W. Doherty]

You may be, somewhat, on the right track when you mention GRE, although GRE wouldn't be a solution.

There are technologies that, much like GRE does for L3, can encapsulate L2 and run it across a L3 infrastructure, and example of one of these technologies (although "old") would be L2TPv3.  A much more current technology would be VXLAN or Cisco's OTV.

Again, there are multiple technologies that might be used but your choices will be platform limited and/or possibly limited by the WAN infrastructure you'll be running across.

I'm not current on your hardware, whose features might also vary based on IOS version you're running and/or feature licenses.

I'm also (very much) unfamiliar with AT&T's ASE capabilities and/or restrictions.

Hopefully, other folk will have more specific information.

[MHM Cisco World]

There is no other solution' your SP run vpls which connect between sites'

That why your SP learn mac address.

Any other solution like mGRE can not achieve connect l2 multi sites.

[Joseph W. Doherty]

Reading @MHM Cisco World reply, with mention of vpls, and "no other solutions", caused me to try to find some information on this AT&T service (which I already noted I'm [very much] unfamiliar with).  I found it's a L2 service (my bad, I was assuming it was L3).  Your OP information, and @MHM Cisco World reply, make much more sense, now.

Is @MHM Cisco World correct there's no other solution (short term)?  Very possibly without cooperation of AT&T.

Technically, whatever technology AT&T is using to provide your service is very likely able to do many things, which AT&T may, or may not, offer, and if they do offer something else, it may also be expensive (possibly except for Internet bandwidth, as that's often very competitively priced).

To be clear, what I had in mind was technologies that carry L2 over L3, but you already have L2.  The only technology that immediately comes to mind, carrying L2 over L2, is Q in Q, which should need AT&T support.

Doing anything "in the very near future" is likely to be difficult unless it's something AT&T can provide using your/their infrastructure.  Given more time you might have other options.

BTW, these sites don't have local Internet access, correct?  Because if they did, you might have L2 over L3 options as mentioned in my first reply.

[spfister336]

No, no local internet access at the remote sites.

I suspect we might be OK until August. Depending on how much work gets done over the summer, more and more sites should be completely on Meraki and the VLANs for those sites shouldn't need to be extended. But I'm not sure how many will get done. I'm just trying to work out the best plan B for our situation.

Nothing suggested yet by AT&T. 

[MHM Cisco World]

hope you get good deal with AT&T 

[spfister336]

I brought up the Q-in-Q subject with ATT and this is what they sent me. They seem to be saying that I can do this just with changes to our equipment. There were some images in the email text, but those didn't 

Q-in-Q is possible over ASEoD, and is also called “Overlay”.

Below is a description of how this is accomplished.  The key take away is that you are breaking up the broadcast domains into different vlans before sending them into the ASEoD network.  You would be doing that with your L2 or L3 edge equipment.

This breaks up the MAC addresses into separate vlans and then with Q-in-Q you are encapsulating the vlans (inner tags) into a single vlan (outer tag).

Overlay Connections

The basic reference connections provide connectivity among a set of routers within a single community of interest. The members of each EVC share a single connectivity domain. If additional communities of interest are needed, they are accommodated through the provisioning of additional EVCs.

An alternative architecture is to decouple the provisioned service architecture from the customer connectivity architecture. With this approach, a single Port-Based EVC is provisioned among all customer sites. CPE is then configured with one or more VLANs which are carried transparently across the EVC. These tags are not interpreted by AT&T Switched Ethernet ServiceSM; they are transparently carried from end-to-end. This end[1]to-end tagging may be used by end equipment to segregate traffic into multiple communities of interest (e.g. VLANs/VRFs) over a single EVC27. Further, since the end-to[1]end tagging is completely controlled by the CPE, additions and changes to communities of interest may be dynamically configured by the customer without needing to engage AT&T. The ordering and provisioning of AT&T Switched Ethernet ServiceSM is identical to the ordering and provisioning for ‘basic’ reference connections. The overlay function is accomplished purely through additional CPE configuration

overlay reference connections, there is a CPE configuration for Port-Based UNIs (CE[1]A) and for VLAN-Based UNIs (CE-B). This configuration is the same whether communication is among like UNI types (Port-based to Port-Based, VLAN-Based to VLAN[1]Based) or among a mix of UNI types.

Port-Based Service, 802.1q (CE-C)

Example Interface Configuration for Port-Based UNI (CE-C in Figure 11 - Overlay Reference Connections above):

[image002.png]

In this example, VLAN 101 and VLAN 102 are transported transparently across the ASE network. The EVC CIR in this example is 50 Mbps. A service policy is attached to the main interface output to enforce the CIR traffic contract and to perform any desired Class of Service functions (See Traffic Shaping & CoS). The shaping and policy are applied at the ‘port’ level and apply to all VLAN traffic traversing the EVC. Port level treatment allows statistical sharing of the available bandwidth among the VLANs i.e. any VLAN has the ability to burst to the full CIR with priority/bandwidth allocation determined by the CoS policy. For each VLAN to be carried, a sub-interface is configured with the appropriate 802.1q VLAN tag and an IP address/subnet. Any number of additional tagged sub-interfaces could be added. Note that though these represent separate IP sub-interfaces with independent VLAN tags, they are all part of the same IP routing domain in the CE router. If there is a security requirement to maintain separation among VLANs, then a VRF Lite configuration can be used. This would involve configuring an independent VRF for each VLAN, and then associating the WAN sub interfaces and corresponding LAN side (sub)interfaces with each VRF. This establishes a separate routing instance for each security domain. Note that this separation should NOT be used for VLANs that are allowed to communicate. If kept separated by VRFs at the CE, the VLAN to VLAN traffic within a single site could potentially be forced to traverse the WAN to reach a L3 device allowing them to communicate. Please refer to your Cisco documentation for more information on configuring VRFs.

VLAN-Based Service, QinQ (CE-D)

Example Interface Configuration for Port-Based UNI (CE-D).

[image003.png]

In this example, VLAN 101 and VLAN 102 are transported transparently across the ASE network. The EVC CIR in this example is 50 Mbps. A service policy is attached to the main interface output to enforce the CIR traffic contract and to perform any desired Class of Service functions (See Traffic Shaping & CoS). The shaping and policy are applied at the ‘port’ level and apply to all VLAN traffic traversing the EVC. Port level treatment allows statistical sharing of the available bandwidth among the VLANs i.e. any VLAN has the ability to burst to the full CIR with priority/bandwidth allocation determined by the CoS policy. For each VLAN to be carried, a sub-interface is configured with two VLAN tags. The first/outer tag identifies the EVC to be used across AT&T Switched Ethernet ServiceSM and is removed by the ingress Provider Edge (PE) device. The second/inner tag is carried transparently across the network. An appropriate IP address/subnet is also configured for each sub-interface. Any number of additional tagged sub-interfaces could be added. Note that though these represent separate IP sub-interfaces with independent VLAN tags, they are all part of the same IP routing domain in the CE router. If there is a security requirement to maintain separation among VLANs, then a VRF Lite configuration can be used. This would involve configuring an independent VRF for each VLAN, and then associating the WAN sub interfaces and corresponding LAN side (sub)interfaces with each VRF. This establishes a separate routing instance for each security domain. Note that this separation should NOT be used for VLANs that are allowed to communicate. If kept separated by VRFs at the CE, the VLAN to VLAN traffic within a single site could potentially be forced to traverse the WAN to reach a L3 device allowing them to communicate. Please refer to your Cisco documentation for more information on configuring VRFs.

[MHM Cisco World]

Below is a description of how this is accomplished.  The key take away is that you are breaking up the broadcast domains into different vlans before sending them into the ASEoD network.  You would be doing that with your L2 or L3 edge equipment.

this correct point, all site emulate as connect to one SW when you use Q-in-Q I think the broadcast is connect and you loss L2 connectivity between the Sites

[Giuseppe Larosa]

Hello @spfister336 ,

802.1qQ tunneling Q inQ does not provide what you need that is scalability in the MAC address count. With Q in Q an additional 802.1Q is added before the customer VLAN but the original MAC addresses are left unchanged and exposed to MAC address learning on Service provider side.

There are solutions in the metro ethernet field that go beyond 802.1Q in Q and they allow for encapsulation the user ethernet frame inside a provider ethernet frame in order to provide MAC address scalability, like Provider Backbone Bridge but I don't think this can be done from the customer side only.

As the MAC address count limitation is enforced on AT&T devices for their own scalability, you should check with them how much it costs to increase the max MAC address limit on your service.

Hope to help

Giuseppe

[Joseph W. Doherty]

@Giuseppe Larosa is 100% correct about q in q doesn't change MACs, so the MAC count limitation is very likely to still be a problem.  However, possibly, as a long shot, I was thinking maybe the count is per VLAN, and if AT&T counts per VLAN, and you don't exceed that count, it might work.  Also, q in q adds to the frame, unlike a tunnel solution which reduces payload capacity.

If I understand OP correctly, however AT&T counts MACs, they can increase the limit, just expensively, correct?  Also, how much they can actually increase the count limit, might also be a problem, correct?  If the latter is true, I would be surprised (sigh, I'm often surprised) if they cannot deal with a thousand or so MACs. What's your overall count likely to be?

Lastly, OP mentions a possible very new future need.  Well, in my experience, often getting new MAN/WAN physical implementations takes time.  Logical changes, though, might be done same day. So, an option might be, short term, you have AT&T increase MAC count limit, long term, plan to replace the current AT&T implementation with something else less expensive (one client I worked, at WAN provider's suggestion, moved from frame-relay to ATM, to reduce cost while getting something better - client forgot "buyer beware"!).  Just insure such a less expensive implementation is really less expensive and all your network needs are still met.

[Joseph W. Doherty]

Oh, another possible approach might be to treat the WAN as a transit VLAN.  This would require routers bordering the AT&T L2.  I.e., treat the AT&T service as a switch connecting your various site routers.

If your sites don't really need to be with a shared L2 domain, you only need to route between your routers.  If you do need to share a L2 domain, then a l2tp might be used.

[spfister336]

so this isn't going to fix our issue?

[MHM Cisco World]

You want to reduce mac address but that not meaning disable l2 between sites.

I Will make double check about qinq with vpls' I will update you tomorrow.

[spfister336]

They've increased our MAC limit to 500 but I'm not sure they're willing to go above that. At first, I thought the cost was $5 per MAC, but it's $5 per site, so not as expensive as I thought.

If it makes a difference, we don't really need remote sites directly connecting to other remote sites at all. In fact, I've been asked to block that to help prevent the spread of malware.

Here is the situation on how many MAC addresses we'll need. The current WiFi implement (Cisco WLC with Cisco APs) involves all user VLANs being defined at the central site. The new WiFi implementation (cloud-managed Meraki APs) involves the user VLANs being defined at the remote sites. So, during the transition period from old to new, I have extended the VLAN across the WAN. Once the transition at a site is done, the VLAN extension can be removed. We have one team doing most AP installs, and our maintenance department doing the ones that require a lift (gym ceilings, etc). Since, the lift installs are being done by one person, we could be looking at a transition period of at least a year.

I think we may be ok for the summer, but the concern is that our students come back in mid-August and our normal wireless user count is normally 4-5k users. Some sites may be able to have their user VLANs unextended but I really don't know how many that will be.

[Joseph W. Doherty]

Unless it has changed, a WLC communicates with LWAPs using its own tunnels.  I.e. wireless client MACs shouldn't be visible across a L2 transit WAN unless some how you've L2 interconnected your wireless VLANs with the transit LAN.

Unfortunately, I'm unfamiliar with Meraki environments, and perhaps, for that transition, you do need to somehow bridge L2 between the WLC and Meraki (hopefully not).  (Are you trying to use the same SSIDs and VLANs between the two wireless systems?)