Routing at the distribution layer AND the core?

[James Montigny]

I use Layer 3 switches to aggregate my user, datacenter and other local traffic within buildings.
These devices provides routing and security between those networks.

Rather than interconnect each distribution switch to every other distribution switch, I use VSS pairs of core switches to aggregate traffic at each campus.
I'm blessed with access to dark fiber and 10Gb DWDM circuits between all 6 campuses so building a mesh between core switches was easy.

Now that we've established improved architecture at each campus, I am presented with the opportunity to re-architect the core for improved performance, which has sparked a lot of discussion in the office about what works best and why. I'd like to gather outside opinions as well since you are likely to think of things that we had not considered.
What are the advantages and disadvantages to these core connectivity options? Is there a better solution?

A) Dedicated P2P links between each core switch, allowing the routing protocol to make path decisions between distribution and core, core and core and core to distribution.
In this case, distribution switches would be at least 2 hops away from each other.
I prefer EIGRP, but we could use any interior protocol here if there is an obvious advantage.

B) Logical P2P links (VLANs or tunnels) over a Layer 2 core; allowing the core's RPVST to make path decisions based on the distribution switch's routing decision.
In this case, pairs of distribution switches would be layer 3 neighbors over each logical connection.

C) All distribution layer devices on the same transport VLAN, allowing the core's RPVST to make path decisions based on the distribution switch's routing decision.
In this case, all distribution switches would be layer 3 neighbors on a shared broadcast domain.

I don't feel that bandwidth is an issue here, so load-balancing across multiple paths is no a concern.
Convergence time would be important to me if the benefit outweighed the performance hit of extract processing on day to day traffic.
I don't span VLANs between campuses; even if I did it would be via a tunnel.
The fiber is privately owned and cuts are unlikely but still possible as the network spans across a large metro area.

 

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Are all your links between your VSS campus cores, port-channel (configured at p2p) or single p2p links?

Is your mesh between VSS campus cores full or partial?

There are issues using approaches B or C.  If you do any multicast, unless your run PIM (not IGMP) snooping, you flood transit L2 VLANs with multicast traffic.

When you do transit L2 between routers, loss of far side often requires "soft" loss of neighbor detection (BFD can help here) rather than "hard" (link down) loss on physical p2p links.

Normally, A would be the better approach although VSS makes for some additional L3 considerations.

[James Montigny]

Port channels are available since we own the media and have active DWDM shelves.
I typically diversify channel members such that half use the West route and half use the East.

The mesh can be full without issue; today, it is partial and many DWDM channels are unused.
From a L2/L3 perspective, we could add paths on different channels, but they would still travel on the same East and West paths and a fiber cut would disrupt 1/2 of my links.

Great points on Multicast and loss of neighbor detection. The loss if neighbor detection in particular is what was driving the convergence time discussion.

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

When you write of West and East routes, it sounds like your design might "expect" VSS traffic to pass between the VSS pair.  Ideally, that should only happen where there's some port or card failure.  VSS tries to keep ingress to egress traffic on the same chassis, so VSS designs should support that.

I would prioritize getting your VSS links "dualed" across the pair, before worrying about creating a full mesh topology.

[James Montigny]

East and West as in the physical path around the metro area.
The fiber segments form a ring, so I can reach sites via either path simply by having the DWDM shelves pass the appropriate wavelength.
I don't actually care which route a signal takes as long as an alternate path is available when the path in use fails and I know that DWDM failover/rerouting is not automatic so we look to layers 2 or 3 for that functionality.

The team seemed split across a vision of dedicated L3 point to point links, a virtual circuit (almost Frame-Relay-like) topology or a shared IP transport segment. I appreciate your feedback Joseph, it sounds like P2P L3 is your preference and that certainly matches Cisco literature.