Why interconnect distribution layer switches?

[oswaldo81]

Hi,

I am trying to find the reason where in many Cisco docs, I see the Distribution layer switches in the same switch block interconnected.

e.g. you have a switch block containing two Access layer switches and two Distribution layer switches. Each Access switch has link to both Distribution switches. Also, is the interconnecting link L2 or L3?

Any explanation or links to docs which explain the reason for this would be much appreciated.

Thanks,

Alan

[Jon Marshall]

Alan

Probably the most common scenario you will come across is the distribution switches being connected via a L2 trunk, more specifically it's usually an etherchannel trunk for bandwidth.

The access-layer switches are then dual connect as you say with L2 uplinks to the distribution switches.

The distribution switches are responsible for the inter-vlan routing for all the vlans on the access-layer switches. Generally redundancy on the distro switches is achieved by running HSRP or GLBP which run across the L2 trunk interconnecting your distro switches so you will see this sort of setup

distro1

int vlan 10

ip address 192.168.5.2 255.255.255.0

standby 10 ip 192.168.5.1

standby 10 pri 110

standby 10 preempt

distro2

int vlan 10

ip address 192.168.5.3 255.255.255.0

standby 10 ip 192.168.5.1

standby 10 pri 100

Note:- standby commands are the HSRP config

the client on the access-layer switch is in vlan 10 so it's default-gateway is set to the HSRP IP 192.168.5.1.

Because of the higher HSRP priority distro1 is active with HSRP. Now if distro1 fails then the HSRP active gateway switches to distro2 and the client machine can still get to it's default-gateway.

Hope this makes sense.

Edit - the design/layout i have described is not the only way to do it. You may want to have a look at www.cisco.com/go/srnd where you will find a number of Cisco design docs that cover this sort of thing - look at the data centre and campus docs.

Jon

[andrew.butterworth]

If you follow the Cisco blueprint you shouldn't need a Layer-2 link between the distribution pair as this will create an STP loop which you really don't want. Typically the link between the distribution pair is Layer-3 and is used to prevent blackholes due to summarisation. You would normally summarise the address space used for the access-layers at each disctribution pair into the core. This way you would only see the summaries in the core instead of all the individual subnets used at the access layer. Unfortunately this creates a problem if an access layer uplink to the distribution breaks as the core will not see this due to the summarisation. Putting a layer-3 link between the distribtution pair solves this issue and still lets you summarise.

Have a read of the 'Campus Network for High Availability Design Guide' at the SRND site:

http://www.cisco.com/go/srnd

HTH

Andy

(Sorry to disagree with you there Jon but I sat through a Networkers presentation on this and remember it well.... Hows it going anyway? You taken another contract yet?)

[Jon Marshall · Accepted solution]

Andy

No problem with disagreeing. I still think that L2 between the distro switches is a more common setup in the wild regardless of what networkers are saying but happy to admit i could be wrong / out of date :)

I remember the exact same conversation we had when we did MSQ1 and you were surprised when we suggested using a L2 link between our distro switches.

Where i do agree is if all servers/clients etc. are not connected into the distro switches but are on switches connected to the distro switches then yes a L3 link is a good way to go. But often due to cost distro switches double up as server switches because a domain-controller, print server and file-server can't justify an entirely separate pair of switches.

If you had a L2 access-layer you could still use L3 link between distro switches and allow HSRP to run across access-layer links but this have never felt "right" to me.

Like i say i don't think either is right or wrong and removing STP has it's advantages altho with RSTP etc. it's not the villian it used to be but i'm happy to concede that with the right amount of kit L3 is a prefectly good way to go.

Would be very interested to hear from any other engineers on this subject.

Nope, still taking time off and not looking just yet. Hope your'e good and work is still keeping you occupied.

Jon

[Jon Marshall]

Andy

Was your networkers presentation talking about best pratice or what is actually seen in customer networks ?.

Thinking i may need to be getting a job fairly soon if i'm getting that out of touch :)

Jon

[oswaldo81]

Jon, Andy,

thankyou very much for such detailed replies.

So there are a few options for this setup - I will have a read through the links you suggested.

Thanks,

Alan

[andrew.butterworth]

Hi Jon

It was a best practise thing and to be honest we actually put this in for a customer, however it was a BIG network - the user access layer was about 80 6509's and I think the server access layer was a similar size. You can imagine though, all the address space was perfectly carved up and was easily summarised at the distribution block. It was pretty much a copy of the SRND guide, it did exactly what it was supposed to though and all the testing we did proved the design was solid.

This was a about 3-4 years ago as well and new stuff has appeared since (VSS and the whole virtualisation approach).

Cheers

Andy

[Jon Marshall]

Andy

Hope you don't mind if i pick your brains a bit on this.

Based on the design doc i've made the following assumptions

You had a L3 connection only between your distro switches.

Your access-layer switches were dual linked to the distro switches using L2 uplinks and you did not span vlans across access-layer switches.

Because you had a L3 link between your distro switches both your L2 uplinks from each access-layer switch were in forwarding state (altho not forwarding both ways i assume - see below).

You were running HSRP for default-gateway on the distro switches.

If the above assumptions are correct can you just verify a couple of things for me as i have never built a design like this.

Presumably the HSRP multicast packets for each vlan have to travel across the uplinks from the access-layer switch because there is only a L3 interconnect between the distro switches ?

When the design talks about both uplinks forwarding this is presumably for return traffic only as at any one time only one of the distro switches would be the HSRP active for the oubound traffic ?

Jon

[andrew.butterworth]

Jon

Yes, the HSRP hellos are sent from each distribution switch down the Layer-2 link to the access switch, since they are layer-2 multicast they obviously reach the other distribution switch so they see each other. If one of the uplinks fails then the Layer-3 SVI on the distribution switch goes down as there are no active Layer-2 interfaces up (obviously HSRP fails over).

Both uplinks are forwarding from an STP point of view (i.e. there is no layer-2 loop), however from an IP hosts point of view there is only one default-gateway so traffic from the host will always go up one uplink, but the return traffic could come down either link(equal cost paths). You could of course use GLBP instead of HSRP and utilise both uplinks more efficiently.

Does this help?

Andy

[Jon Marshall]

Andy

Yes does help, much appreciated. For some reason i have never felt very happy about using access-layer switches as a transit switch for distro to distro traffic. Guess i'll have to get used to the idea :)

Edit - think it helps to just think of it as vlan traffic rather than distro to distro.

Thanks

Jon

[shastymacnasty]

Hey Jon/Andy

Apologies for raising this post again! Ive been trying to replicate the scenario Andy is suggesting (and which is written in the Cisco Campus Network for High Availability Design Guide) and Im having some issues!I have attached a diagram of what Im trying to achieve! and also the configurations for the respective switches. (let me know if u need some more info). Basically what Im doing is pinging the management interface on EDGE A (172.17.1.20) from the management interface from the SERVER FARM Switch (192.168.0.251), which if all the links are up works great both ways.

The problem comes in when i shutdown fa0/2 on EDGE A then i dont get any pings through in either direction!

I have configured a summary address on Fa0/1&4 on DIST A&B of 172.17.0.0 255.255.0.0

I have also configured a L3 link between the distribution switches and ensured the subnet is within the summary address range being advertised.

Hopefully Ive given enough info for you to have a look at! If I change the L3 link for a L2 link then it all works fine! What am I doing wrong?? Been looking at it so long Im cross eyed!

regards

M

[Jon Marshall]

M

On the trunk links from Dist A to Edge A and Edge B - what vlans are going across those links.

You must ensure that on the link from Dist A to Edge A ONLY vlans 10 & 30 are allowed on that link and just as importantly you must ensure that on the link from Dist A to Edge B you are NOT allowing vlans 10 & 30.

Can you confirm how the trunk links are configured.

Jon

[Jon Marshall]

M

The other thing to check.

You are summarising from Dist switches to core switches. So Dist A summarises to core A and core b and ditto for Dist B.

The link between Dist A and Dist B - you are not summarising here are you ? - if you are you can't.

Jon

[Jon Marshall]

M

Sorry, my answers were a bit brief before, i was a bit pushed for time.

One of 2 things could be happening but it would help if we went through how it should work.

Dist A should be advertising a summary route for 172.17.0.0/16 to core A & B. Same for Dist B.

Also there is a L3 link between Dist A and Dist B. So Dist A and Dist B should be EIGRP neigbors but you should not be summarising between these 2 switches.

Lets assume the path at the moment (before shutting down fa0/2 on Edge A) from 192.168.0.251 to 172.17.1.20 is Server Farm A switch -> Core A -> Dist A -> Edge A.

You then shut down fa0/2 and the following sequence should happen

1) The vlan interfaces for vlan 10 and vlan 30 go down on Dist A - this is the key point.

2) Because vlan 10 & vlan 30 interfaces have gone down the directly connected routes for vlan 10 & 30 subnets on Dist A are removed.

3) But these subnets also exist on Dist B and on Dist B the L3 SVI interfaces for vlan 10 & 30 are still up. So Dist A now receive routes from Dist B for these subnets.

4) Dist A forwards traffic for 172.17.1.20 to Dist B.

5) Dist B sends traffic down trunk link to Edge A.

6) Because Fa0/2 was shutdown the active HSRP gateway was moved to Dist B so 172.17.1.20 sends the return packet back to Dist B.

So what could stop this happening.

i) If you are summarising between Dist A & Dist B. Note that when you use a summary address it puts a summary route to Null0 with an AD of 5 in the local routing table.

So going back to step 3 in previous example, if instead of Dist A receiving specific subnet routes you were relying on a summary route from Dist B, this summary route would have a higher AD than the local summary route to Null0 on Dist A. So the traffic would be blackholed because Dist A would use it's summary route to Null0.

Having said that i think the more likely problem is

ii) Your vlan interfaces on Dist A for vlan 10 & 30 MUST go down for this design to work - see step 1 above.

A L3 vlan interface will be in the up/up state if

a) there is a port in that vlan on the switch in the up/up state

OR

b) there is a trunk link with that vlan allowed on it that is still up

So you must make sure that the Dist A -> Edge A trunk link only allows vlans 10 & 30 on it. And you must make sure that the Dist A -> Edge B trunk link only allows vlan 20 & 40. If the Dist A -> Edge B trunk link also allowed vlans 10 & 30 then when you shutdown fa0/2 the L3 vlan interfaces on Dist A will stay up because there is still an active trunk link with those vlans on it.

Hope this makes sense.

Jon

[Jon Marshall]

M

Sorry missed out the final bit.

If the layer 3 SVI's stay up for vlan 10 & 30 on Dist A then traffic will be dropped. It can't be sent across the L3 link to Dist B because vlan 10 & 30 are shown as direcly connected on Dist A so you can't route to these subnets across a L3 link because Dist A has them direcly connected.

Only if they go down can they then be routed across to Dist B.

Jon