L2/L3 Etherchannel

[alistair777]

I am planning a virtual network topology in accordance with Cisco's Campus Network High Availability Guide.

The images below outline roughly what I am trying to achieve:

http://www.cisco.com/c/dam/en/us/td/i/100001-200000/110001-120000/119001-120000/119818.ps/_jcr_content/renditions/119818.jpg

http://www.cisco.com/c/dam/en/us/td/i/100001-200000/110001-120000/119001-120000/119819.ps/_jcr_content/renditions/119819.jpg

I cannot use layer 3 at the access layer and looking to use a mixture of local VLANs and end-to-end VLANs.

The majority will be Local VLANs not spanning multiple locations, but there are a few exceptions in our design where the VLANs must span multiple locations and Layer 2 switches. 

My question is whether I should use a Layer 2 or Layer 3 Etherchannel between the two distribution layer 3 switches, keeping in mind that I will be using Rapid-PVST+ and HSRP to stop loops.

I watched have Cisco training videos where a Layer 2 Etherchannel is recommended for Local VLANs and Layer 3 Etherchannel for end-to-end, but I am confused to what would be the best option in a mixed environment but where most are Local VLANs?

I would also like to run iBGP eventually between the two Layer 3 switches as it is a collapsed core design.

[Steve Fuller]

Hi,

The way I’ve done this is to operate the two links between the distribution routers as an EtherChannel supporting IEEE 802.1Q trunking, and then prune the VLANs that are used at the access layer from the trunk.

In the diagram it shows VLAN 20, 40, 120 and 140 at the access layer. These would be the VLANs that must be pruned from the trunk between the distribution routers. In reality it's probably better to prune all VLANs from the trunk, and then specifically allow those that are needed.

As long as any VLAN is only presented to a single access switch, the result is a topology that has no loops for those VLANs, and all links between access and distribution are forwarding. You might also consider using GLBP on the distribution rather than HSRP and you then also have both routers forwarding traffic from the VLANs.

For those VLANs that must span multiple switches, allow those on the distribution to distribution trunk, and then spanning tree will break the loop at each access switch as shown in your second figure.

You will also need a Layer-3 routing adjacency between the distribution routers across the distribution-to-distribution trunk. This is discussed in the section Ensuring Connectivity in Case of Failure of the design guide. From the guide:

Because the distribution nodes send summarized information towards the core, an individual distribution node does not advertise loss of connectivity to a single VLAN or subnet. This means that the core does not know that it cannot send traffic to the distribution member where the link has failed. Adding an L3 link between the distribution switches allows the distribution node that loses connectivity to a given VLAN or subnet to reroute traffic across the distribution-to-distribution link.

For this you would create a VLAN only on the distribution routers and allow this on the trunk between the two distribution routers. You then create an SVI on each router for this VLAN, assign an IP within a /30 subnet and enable routing for the prefix so the distribution routers peer over this VLAN.

As discussed in the design guide the best practice is to make the routing protocol passive for all VLANs, such that no peering is formed across the subnets at the access layer, and then use the no passive-interface command for the VLAN used for the routing adjacency between  the distribution routers.

Regards

[alistair777]

Hey Steve,

Thank you for the detailed response I would appreciate if you could verify a few things:-

1.) So you are saying Local VLANs should be restricted from the L2 etherchannel as this could lead to a broadcast storm, as both interfaces will be up on the access layer switch.

2.) End-to-End VLANs which span multiple "wiring closets" should be allowed on the L2 trunk but STP enabled on the access layer switches to prevent loops.

I am still a bit confused regarding the L3 link.

I would like to run iBGP between the distribution layer switches as each L3 switch is connected via eBGP to seperate ISPs - a typical multihoming scenario. DistA goes to ISPA and DistB goes to ISPB.

However should an ISP link fail I would still need the option of traffic being routing across the L3 distribution switch link to the other ISP which is connected ot the other distribution switch. For example ISPA Link fails so traffic must be routed across the point-to-point to DistB to ISPB. Would HSRP interface tracking take care of this?

In summary, how would I achieve the iBGP session, L3 link and L2 etherchannel all in one link?

Thank you

[Jon Marshall]

-

[Steve Fuller · Accepted solution]

1.) So you are saying Local VLANs should be pruned from the L2 etherchannel as they have no need to traverse the distribution layer to another part of the access layer.

Correct. Pruning the VLAN off of the link between the distribution routers changes the logical topology from a delta (or triangular) topology which is has Layer-2 loops, to a “V” shaped topology that has no loops. If there are no Layer-2 loops, you can never run into an issue with some spanning tree misconfiguration or bug resulting in a broadcast storm.

2.) End-to-End VLANs which span multiple "wiring closets" should be allowed on the L2 trunk but STP enabled on the access layer switches to prevent loops. Prune all Local VLANs as stated above.

Correct. Though for the sake of clarity you should prune a VLAN from the Layer-2 trunk between the distribution routers if that VLAN is present on two or more access switches. I make this distinction as in some environment a “wiring closet” may have more than a single access switch.

I am still a bit confused regarding the L3 link.

The idea of this is that there’s a path to carry traffic to an access switch in the event that a link from the distribution router to the access switch is down.

Let’s assume you have a number of contiguous subnets behind your distribution routers, for example 192.168.16.0/24 to 192.168.31.0/24. Instead of advertising all sixteen /24 prefix towards the core, you might advertise a summary prefix i.e., 192.168.16.0/20 that covers those sixteen prefix. This means that traffic from the core to any of those prefix can be sent to either distribution router 1 (D1) or distribution router 2 (D2).

Now consider a link failure between the D1 and an access switch. The access switch in this case has been provisioned with VLAN 16 and uses the prefix 192.168.16.0/24. On the distribution router there will be only one physical link carrying VLAN 16, and when that fails, the result is the SVI for VLAN 16 becoming operationally down (see Understanding and Troubleshooting the Autostate Feature in Catalyst Switches for details of why.)

For traffic from VLAN 16 towards the core there’s not a problem as connectivity is maintained via the link to D2, and the FHRP operating on that router. If using HSRP or GLBP, D2 will be the only active router for VLAN 16 now as there’s no connection between D1 and D2 in that VLAN.

Traffic from the core towards VLAN 16 is a different story though. Because of the summary there’s still traffic sent to either D1 or D2. When traffic for 192.168.16.0/24 reaches D1 where the link has failed and the SVI is down, there no way to forward the traffic and so it would be dropped.

This is where the Layer-3 link and routing adjacency between the distribution routers comes in. When the SVI on D1 goes down, D1 learns the prefix from D2 via the routing adjacency, and so forwards traffic to D2, which is then forwarded on to the access switch by the operational link.

In summary, how would I achieve the iBGP session, L3 link and L2 etherchannel all in one link?

I think you can simply add an iBGP session between distribution routers and use the same Layer-3 routing adjacency discussed above. In the event that the connection from D1 to the ISP fails, the external routes would be learnt via the iBGP session, and so traffic would be forwarded via the same Layer-3 link and adjacency out via D2.

Regards