Assistance with design best practices

[Bob Landis]

Over the years, our company has had several contracting firms implement various portions of our corporate network. Each time there has been no consistency in regards to following best practices, and now several problems are surfacing as a result.  Our core resides at the main site and is a 6509E switch. This main site has 3750X/3850 access layer switch stacks deployed between five IDF closets. Each IDF switch stack is cabled directly back to the 6509E core via multimode fiber configured in an L2 Etherchannel.  The remote sites connected via various methods. On the right hand side of the attached diagram, six sites are connected via a Charter Spectrum EP-LAN Charter Spectrum provides a single copper customer handoff that is connected directly to the 6509E via port gi2/42 configured as a trunk. The six remote sites running 2960X/3750X access layer switches are cabled to the Charter Spectrum ME3400 at each location via Gi1/0/48 in a trunk configuration.  A seventh site is planned to be added to the EP-LAN in the near future that will consist of two Nexus 93180's as distribution layer and three buildings with individual IDF's running 3850 stacks as access layer switches.  On the left side of the attached diagram a 4500X is shown that is cabled directly to the 6509E. The 4500X has three microwave radios attached that provide services to three additional remote sites - Site 8, 9, and 10. Sites 8,9, and 10 each have 3580 access layer switches and the traffic is routed to the 4500X.  Each switch at sites 8, 9, and 10 has four /24 block local vlans  and one /30 vlan configured in transparent mode:

Vlan 800- Management

Vlan 810 - Video

Vlan 820 - Voice

Vlan 820 - Data

Vlan 500 - WAN Routing

Several problems have arisen due to the mix of L2 and L3 connectivity in the topology.  CPU usage is extremely high on both the 6509E as well as the access layer switches at the main site and EP-LAN locations due to large amounts of broadcast traffic.  All five IDF's at the main site and all remote EP-LAN sites were configured as L2 in VLAN1 (Shown in the attached diagram as the red links to the 6509E).  All servers, access layer switches, printers, etc. on each of the floors and remote office locations reside in the same /21 subnet.

Our goal is to clean up this messy topology, minimize/eliminate any issues with broadcast traffic and spanning tree, and have an implementation standard that is consistent across all sites and easy to manage. Everyone we have spoken to seems to have a different theory on how the links should be configured - we've heard suggestions such as:

- create several vlans and svi's on the 6509E and filter vlans allowed across the etherchannel trunks

- create L3 etherchannels, route each main site IDF to the 6509E, create local vlans on each IDF stack

- mix of L3 etherchannels and routing for each main site IDF to the 6509E, create vlans and SVI on 6509E for EP-LAN Sites and trunk allowed vlans to each site for each function

- nothing can be done due to mix of connectivity options

- create L2 management VLAN for switch management, trunk management vlan across trunks to each site

- use loopback addresses to manage switches, no management vlan, etc.

We are at the point we are extremely confused as to how to approach cleaning up these issues due to the various things in which we have been told. It appears that the model on the left side of the topology diagram with the microwave links to the remote sites seems to be very clean as each site has the same five vlan numbers for the same functions versus creating a large amount of individual vlans on the 6509E and trunk them, but we are not certain as to whether or not this is the best solution nor quite sure how to go about configuring it due to the mix of connectivity?

Based on the diagram and the information provided above, what is the best validated design and configuration to implement in order to achieve our goals, stop excessive broadcast/spanning tree issues, and keep things clean?

[Philip D'Ath]

I'm surprised that you can generate enough broadcast traffic with the likely size of the pipes to stress a 6509.

Have you done a packet capture to see where all this broadcast traffic is coming from?

[Bob Landis]

Philip,

We did do a packet capture.  After summarizing the conversations and sorting by byte, the highest broadcasts were coming from virtual machines on our EXSi hosts that reside in VLAN1 and client PC's also receiving their DHCP address leases in VLAN1.

Bob

[Jon Marshall]

Can you clarify the EP-LAN handoff ie. you say it is a trunk but then you say all red links are in vlan 1 ?

That aside the main problem as far as I can see is the use of vlan 1 with a /21 and the amount of sites that are in that subnet.

So the key question is how easy is it going to be for you to break this down into multiple vlans ie. it would mean updating subnet masks on many devices. Obviously if the majority of clients are using DHCP then that will help but printers, servers etc.will all need to changed as well.

If you could do just that then I think you would see an improvement because broadcasts would at least be contained within vlans ie. a broadcast in a remote site does not have to go to all the main site access switches (although it does still need to go to the 6500).

In terms of L2 vs L3 the answer is it depends. The main limitation of L3 from the access layer is that you cannot have the same vlan/IP subnet on multiple access switches which can be limiting especially in terms of servers etc. For your specific setup within the main site your switch stacks could certainly route if that is what you wanted to do. As for the EP-LAN not sure whether this would support L3 uplink, do you know ?

As for the management vlan that  again depends on L2 vs L3 ie.for the left hand side loopbacks. For the main site and the EP-LAN sites it depends on how you do it but if you do stick with L2 I would have management vlans per site or at least the main site should have a separate one to the remote sites.

It's difficult to be more precise than that without understanding more about your traffic patterns because this can have an effect on what you do. A routed access layer used to be more popular before VSS etc. but it can still be a good solution providing you accept it's limitations.

Either way simply creating more vlans would make the whole thing a lot easier to manage.

Happy to discuss further.

Jon

[Bob Landis]

Jon,

 The handoff port for the EP-LAN is configured as a trunk port, but no restrictions have been placed on the allowed vlans across the trunk. The native trunk vlan for the handoff is still vlan1.  Actual port configuration for the EP-LAN handoff is as follows:

interface GigabitEthernet2/41
description Charter EPL Fiber Uplink Trunk
switchport
switchport trunk encapsulation dot1q
switchport mode trunk

Although it may be time consuming to break down into multiple vlans, I don't think it should be that difficult. VLAN 1 was created as a 172.16.24.0/21 scope in DHCP, and static ranges were set as exclusions from the DHCP pool. Most of the servers and static devices are in the 172.16.24.x and 172.16.30.x ranges. If the 172.16.24.0/21 subnet is broken down, perhaps one of the two ranges with static devices could be reduced in size via vlsm and reserved for servers? This would eliminate several changes on the server side and preserve existing firewall rules. The remaining devices in the 172.16.30.x scope could then be migrated over to the 172.16.24.x scope.  As for the printers, would it be easier to include them in the same data vlan as the client devices, or would you create a separate vlan for just printers?  I'm certainly not opposed to revamping the IP structure to fit a clean validated design if that is what it takes.

Currently the traffic pattern is as follows - All servers reside at the main location, and are directly connected to the 6509E. All inbound/outbound traffic originates at the main site as this is where our firewall clusters and border routers reside. All remote links would only have access layer devices with the exception of two - one site (which we will refer to as site 11 as it is not on the diagram) is directly connected to the 6509 in a port channel just like the IDF's and houses a server that the local clients at the site utilize. The second exception is remote site 6 on the diagram (EP-LAN) that also has a server that the local clients utilize. On the left side of the diagram, the remote sites 8, 9, and 10 have IoT devices in the field - all access layer traffic is routed back to the 4500X. EIGRP is running on the 4500X and advertises these networks to the 6509E which is also running EIGRP. There is a static route on the 6509E to send all traffic to the firewall.

I'm not certain if the EP-LAN supports a L3 uplink.  My initial thought is that it may not.

Let me know if you need any additional information that may help clarify things.

Thanks,

Bob

[Jon Marshall]

Bob

Good news with the servers being mainly in just one subnet, that will help a lot. As for the printers it is really up to you but if possible I would use a separate vlan for them as well because you may in future want to implement security between clients vlans ie. if the printers and servers are in different vlans then there is no need (usually) for clients from one vlan to talk to clients in another vlan which makes your acls quite straightforward.

Still I have seen printers on client vlans as well so it is really up to you.

In terms of L2 or L3 I think you are going to end up with a mix because you already have L3 from the 4500 but I also doubt whether the EP-LAN supports a L3 uplink so you are limited to a trunk link there.

The main site access switches can route but if the servers are all connected to the 6500s then there is little routing needed in the access layer so it would really be about broadcast containment.

So whether you use L3 or not you need to have the main site use it's own vlans and each remote site connecting to the EP-LAN should use it's own vlan(s) as well ie. each site has a different vlan(s)/IP subnets. Note I am assuming this is possible for the EP-LAN sites ie. they do not all have to be in the same vlan/IP subnet because of the trunk link but you would need to confirm.

Even if they did all need to be in the same vlan it should still be a different one than any used in the main site.

If you did all of this then in terms of the main site access switches you don't need to make a decision about L3 or not just yet. All the new vlan L3 interfaces (SVIs) would be on the 6500 and if you then decided at a later date you wanted L3 to the access layer you could easily migrate as long as you did not use the same vlan/IP subnet across multiple different switches.

Those are my initial thoughts but others may add to them and I'll have a think about over the weekend as well.

As before happy to discuss any questions or concerns you may have.

Jon

[Jon Marshall]

Bob

Just one last point after rereading your initial post.

Using more vlans should reduce the CPU on the main site access switches and the EP-LAN connected site's switches because currently a broadcast goes everywhere but with vlan segregation it should be limited between the access layer switch and the 6500.

Note that if you use the same vlan/IP subnet  for all EP-LAN sites then a broadcast would be seen by all remote sites but not by the main site access layer which is why I suggested a vlan/IP subnet per site if possible.

But the 6500 will still see all broadcasts so I'm not sure it is going to help there.

Of course if you did route from the access layer in the main site you could reduce broadcasts that way but you still have the EP-LAN remote sites to deal with.

Just wanted to point that out in case you decide it is not worth the effort.

Jon

[Joseph W. Doherty]

(Overlapping with what Jon already posted . . .)

From a design perspective, having a large VLAN and distributing it all over, although convenient, does not scale well for performance.

Ideally, you would want to have VLANs not span more those two logical devices (e.g. your core 6509 and the logical edge device down one logical link, such as an IDF stack or remote switch).  For your edge devices that support L3, enabling L3 on the edge doesn't do much for you unless you have multiple networks there, with traffic that flows between them, or you have multiple uplinks you can take advantage of.

Moving to new VLAN and subnets (as Jon also noted) isn't too difficult for DHCP hosts.  Static IP hosts take some work, although you may find some of those can work just fine as DHCP and some others can work just fine if IPs are assigned statically via DHCP.

While converting host addressing, secondary addressing can be very useful.