High Availability Campus Design: Layer 3 Access to the Distribution

[Steph1963]

Hi Cisco Expert,

I have seen on many Cisco Campus design document the Layer 3 Access to the Distribution Model as shown in figure 6 in the following link

http://www.cisco.com/en/US/docs/solutions/Enterprise/Campus/HA_recovery_DG/campusRecovery.html

This model indicated that a layer 3 link is used between the 2 distribution switches as opposed as a layer 2 trunk for the Distributed Vlans on Access switch.

I am familiar in creating HSRP under vlan interface but I would like to know how we can configure HSRP or VRRP under this layer 3 link. Any configuration example would be greatly appreciated.

Secondly, I would like to know if it is a good practice for the core routers to only send the default route to the distribution switches and for the distribution switches to send a summary route of the access subnet.

Finally, any explanation on when to used a Distributed VLANs on Access Switches vs Layer 3 Access to the distribution would really be helpful.

Thanks for your help

Stephane

[Jon Marshall]

Stephane

The diagram doesn't make any sense. It is showing HSRP with L3 links to the access swtiches.

There is no point in running HSRP on one access switch.

I notice the links to the access layer are blue as opposed to red. Perhaps they are meant to be L2 rather than L3 links.

If they are not then like I say it doesn't make sense.

Where did you find this diagram ?

Jon

[Steph1963]

Hi John,

Thanks for your quick reply,

Sorry if I was not clear enough, I was not saying that only one instance of HSRP should run on the distribution switches but did not understand how HSRP was configured on Layer 2 Access with Layer 3 Distribution Convergence architecture. Does HSRP packets are exchanged via the access layer switch, I am more familiar with HSRP packet exchange via L2 Trunk as in Layer 2 Distribution Switch Interconnection architecture (first attachment).

This drawing (second attachment) comes from the Designing Cisco Network Service Architecture (ARCH) book. Could you please confirm me that this is the same architecture as the one defined in High Availability Campus Recovery Analysis Design Guide. One refers to L2 between distribution and access, second one refers as L3. Configuration from the guide seems to indicate that the access switch used L2 configuration for the link to the distribution.

Is there any best practices for route exchanges between distribution and core network. Does the core routers only send default route to the distribution switches and distribution switches only send a summary route for the access layer subnet.

Finally, is there any rules for choosing a Layer 2 Distribution Switch Interconnection Layer 2 Access with Layer 3 Distribution Convergence architecture? Does one solution provides a better reliability than the other one?

Regards

Stephane

[Jon Marshall · Accepted solution]

I'll answer the other questions I promise but the point about that first diagram you posted is that each access layer switch has vlans that are only that switch. If the links between the access layer and the distribution layer are L3 then there is no point to running HSRP because it could only be run per access layer switch and what is the point.

I suspect the diagram is meant to show L2 links but if it isn't then it makes no sense. Hopefully you can see why otherwise the rest of what I write isn't going to make a lot sense :-)

So assuming you have a L3 link between the distribution switches and an access switches connected via L2 uplinks to both distribution pair, yes HSRP messages flow via the access layer.

If you can make sure a vlan is never on more than one access switch then both uplinks can be forwarding from the access switch because there is no STP loop. If you have this setup then GLBP is a better choice than HSRP because you can utilise both distribution switches and because both uplinks are forwarding you can go direct to either switch.

If you have a vlan that is on multiple access switches and a L3 interconnect between the distribution switches then one access switch will be forwarding on both links for that vlan but every other access switch with that vlan on will have to block one of the uplinks.

If you draw that out then you can see why ie. because STP needs to block otherwise you get loops. Here HSRP might be a better choice.

Notice that with a L3 interconnect between the distribution pair and L2 to the access layer switches even though it is better if a vlan only exists on one switch it is still possible to have that vlan on multiple switches but then you need to block.

With a L3 access layer you cannot have the same vlan on multiple access switches because you route locally on the access switch for the vlans on the switch. To be specific it is not the same vlan as such, it is the same IP subnet because technically with a L3 access layer you could reuse vlan ID on different access switches but not the same IP subnet

You probably wouldn't reuse vlan IDs though, I just wanted to make it clear it was more to do with the IP subnet.

So you get less flexibility with L3 in terms of vlan placement.

In terms of routes between the distribution and core the general rule is to summarise wherever possible. It depends on the redundancy and alternate paths you may have as to how well you can summarise and how much summarisation you want to do.

No rules really, it depends on what you are trying to achieve.

Bear in mind that a lot of the designs you are looking at are not necessarily what is happening with a lot of new installs.

With stack switches, 4500/6500/6800 running VSS and Nexus switches running vPC you can now create etherchannels from access layer switches that can terminate on different chassis and still be forwarding on all links which has relegated STP to the background although it should still be run.

Hope some of that has helped.

Any more questions please feel free to ask.

Jon

[Cisco Freak]

Jon,

Is this design related to the 'routed access layer' design.

http://www.cisco.com/application/pdf/en/us/guest/netsol/ns432/c649/ccmigration_09186a00805fccbf.pdf

CF

[Jon Marshall]

CF

Not sure which design you are referring to ?

If you mean the original diagram Stephane attached to the post then no, not really, because it is showing HSRP being run with the access switches connected at L3 so there is no point in running it.

Can you clarify what you are referring to ?

Jon

[Steph1963]

Hi Jon,

Many thanks for your help.

I think that I have finally understand what you were trying to explain me. Running a Layer 3 between the distribution and the access prevents us of runnning HSRP or GLBP simply because you cannot have the same IP subnet on two different side by sided distribution switches. Having layer 3 link means that one distribution switch act as the gateway for the PC attached to the access switch.

Does this means that the L3 link between the distribution switches will only provided an extra routing link for the core routers access. In other words, this link will never carry and HSRP or GLBP packets but only OSPF or EIGRP protocol that is found between the distribution and the cores routers.

If I correctly understand you, GLBP will have a direct link between the two access switch and it shares the load, both distribution switches will have a direct access to the access switch as opposed to HSRP where only one is active at a time.

Finally, if we had same VLAN on these access switch, where should we put the extra L2 link. Should we put it between the access switch or the distribution switch. Would this means we would have a L3 + a L2 links.

Again many thanks for your very helpful explanation

Stephane

[Jon Marshall · Accepted solution]

Running a Layer 3 between the distribution and the access prevents us of runnning HSRP or GLBP simply because you cannot have the same IP subnet on two different side by sided distribution switches. Having layer 3 link means that one distribution switch act as the gateway for the PC attached to the access switch.

That's not really what I meant.

It can be difficult to visualise but it's nothing to do with the distribution switches. If you run a L3 connection to the access switches then that means between the vlans/IP subnets on the access switch and either distribution switch there is a different IP subnet on the routed link.

So the vlans/IP subnets on the access switch can't use either distribution switch as their default gateway because there is another L3 subnet in between.

If you connect your access switches with L3 links then any vlans on the access switches have to be routed on the access switches. If you wanted their default gateways on the distribution switches you have to connect them with L2 trunks not L3 routed links.

That is why the original diagram you attached doesn't make sense if the links to the access switches are L3.

If this still isn't clear then please just say and i'll try and explain it a different way.

The rest of your questions seem to be about GLBP so i'll assume the access switches are connected at L2 not L3.

You are correct when you say HSRP or GLBP cannot be sent across the L3 interconnect between the distribution switches. Both HSRP and GLBP need L2 adjacency so they cannot send traffic across that L3 link but they can send it via the L2 links connected to the access switches.

With GLBP both uplinks will be used simply because both distribution switches can be active forwarders for traffic per vlan whereas HSRP can't (unless you use MHSRP but that requires extra configuration with client default gateways).

Not sure what you mean by your last question about L2 and L3 links and where to put them.

I suspect it is related to the first part about the L3 access layer.

Like I say it may be the way I am explaining it but please feel free to come back if it is still not clear and we'll go through it in more detail.

Jon

[Steph1963]

Hi Jon,

Your explanation makes a lof of sense, now I underdand why it is not possible to have L3 between the access and distribution switches.

Notice that with a L3 interconnect between the distribution pair and L2 to the access layer switches even though it is better if a vlan only exists on one switch it is still possible to have that vlan on multiple switches but then you need to block.

I have interpreted this as this would be possible if we add another L2 so I was just asking where we should put this extra L2 link, between the distribution switches or between the access switch.

Regards

Stephane

[Jon Marshall]

Hi Stephane

now I underdand why it is not possible to have L3 between the access and distribution switches.

Just to clarify. It is possible but you can't then run HSRP on the distribution switches.

I have interpreted this as this would be possible if we add another L2 so I was just asking where we should put this extra L2 link, between the distribution switches or between the access switch.

You don't need to add another link.

What I meant was if you have an access switch connected to a distribution pair with L2 uplinks and the distribution pair are connected by a L3 then there is no loop for STP to block so both uplinks can be forwarding.

If you have multiple access switches and each vlan is only on one access switch then all access switches can forward on both uplinks.

But imagine a situation where you have two access switches as1 and as2 connected to ds1 and ds2 with L2 links. And ds1 and ds2 are connected with a L3 link.

And the same vlan is on both access switches.

So a client on as1 sends a broadcast, it goes to ds1 which then sends it to as2 because the vlan is also on as2. Because as2 also has a connection to ds2 it forwards the broadcast to ds2 and ds2 then sends it back to as1 which then forwards it back to ds1 and around you go again.

Like I say if you draw it out it becomes clearer.

With the vlan on only one access switch you can't get a loop because notice in the above at no time did ds1 forward the broadcast direct to ds2 because they are interconnected with a L3 link so it can't.

So that was what I meant when I said you certainly can have the same vlan on multiple access switches but only one access switch can forward on both uplinks. The other access switches have to block one of their uplinks to avoid a loop.

Hope that makes sense.

Jon

[Steph1963]

Hi Jon,

This makes perfect sense, HSRP or GLBP would required a layer 2 link between the access switch and the distribution swithc. A layer 3 link between the access switch and the distribution switch would mean that the access route would route so HSRP or GLBP does not make sense here.

Samething for same VLAN on multiple access switch with L3 link between distribution switch. Only one access switch can have one uplink on each distribution switch. Otherwise, this would create a loop so other access switch would only have one uplink to the distribution switch or would need RSTP to break the loop.

Thanks for all your patience and help

Stephane

[Jon Marshall]

Stephane

Exactly (to both points :-))

Glad to have helped.

Jon