Etherchannel Simultaneous Primary and Sub-Interface Config

[mike0000111111]

Hello Cisco Experts:

Question: Can I run layer 2 traffic across EtherChannel and layer 3 traffic simultaneously across the same etherchannel on a subinterface?  If not, and considering the background information below, is there an advisable alternative?  The documentation I've been reading isn't clear on the subject.

Background

I'd like to split my VLans across (2x) L3 3560 switches interconnected by EtherChannel.  I'll use SVI's for the routing - but if Switch #1 SVI must route to another SVI on Switch #2, I'd like this traffic to cross the EtherChannel instead of heading to another L3 Device before continuing its route to the destination switch.  (I.E. I prefer direct switch to switch routing.)

Design Preference:

1. I don't want my etherchannel to become a 100% routed channel.  
2. I don't want to add another connection between the switches - ports are at a premium and budget is tapped.
3. No access level switches are being used at this time.

Physical Topology

Thank you for your time,

Mike

[Jon Marshall]

Mike

As long as both SVIs are on each switch then the routing between SVIs takes place locally so you don't need a L3 connection between your switches.

Do you mean only create the SVI for each vlan on one of the switches ?

If so then you simply need a dedicated vlan for routing across the etherchannel ie. still no need for L3 etherchannel.

But I would have thought you wanted SVIs for all vlans on both switches ?

Jon

[mike0000111111]

Hi Jon:

I don't understand your suggestions.

After our conversation this weekend, you may recall I'm running HSRP across the two switches.  

Switch 2 will be HSRP Primary Gateway (until failover) for VLAN subnets:

• Management
• Guest
• Voice

Switch 1 will be HSRP Primary Gateway (until failover) for VLAN subnets:

• Enterprise Wifi
• Data
• Servers

This is how I make sure no devices are orphaned when one switch goes down and it's gateway disappears.  The HSRP virtual gateways, then, are being advertised by EIGRP as the available route for each VLAN.  Consequently, all advertised routes lead to either one, or the other, switch.

Conditions: To be clear - I have SVI's for all VLAN's on both switches.  

But, once data from a client hits the L3 SVI for the VLAN, the routing table lookup will say that the subnet is either local to the switch, or on the other switch.  EIGRP is only advertising the Virtual Gateway for each VLAN.  

Result: Sometimes client data enters the L3 SVI on Switch #1, sees the destination subnet on the other switch, gets routed to a Router, then gets routed again to Switch #2 where the advertised L3 SVI (which is the HSRP Virtual Gateway) resides. (At least this is what happens on Packet Tracer.)

My solution: I had suggested my etherchannel could have an encapsulated subinterface with IP address for switch to switch routes.  

Now that I think about it, maybe I just need to add the routes to each routing table so all the routing happens locally!  That is the switching L3 dream, right?  Is this more or less what you were suggesting?

Thanks,

Mike

[Jon Marshall]

Mike

You say you have SVIs for all vlans on both switches.

So each switch is directly connected to all subnets so the switch can route locally ie. it is never going to see another L3 device as the next hop if it has a directly connected interface in that subnet.

This has nothing to do with HSRP, that is just for end clients.

So lets say traffic is sent from a client on Switch 1 in the data vlan to a client connected to Switch 2 in the management vlan.

The traffic goes to the SVI for the data vlan on Switch 1 and it does a route lookup for the destination IP. It has an SVI for the management vlan so it is directly connected to that subnet.

So it routes (L3 switches to be precise) the traffic onto the management vlan locally. Then it L2 switches the traffic across the etherchannel to the management client.

The management client sends it's return packet to Switch 2 because that is HSRP active for the management vlan. Same thing happens ie. Switch 2 has an SVI for the data vlan so it routes the traffic locally and then L2 switches it across the etherchannel back to the client.

The actual traffic flow ie. which switch is used from the client is dependant on the HSRP active gateways but the switches will always L3 switch locally as long as they have SVIs for all the vlans ie. switches do not use the HSRP VIP.

If that doesn't make sense or you have any other queries please come back.

Jon

[mike0000111111]

Jon:

I understand your confusion.  The scenario you described above is what I expected also.  I will need to do a little more investigation before I post a follow-up.

However, I can explain my HSRP design decision.  With no access layer switches, devices in the voice VLAN will be placed on both Switch #1 and Switch #2.  If one switch goes down, at least some of the phones will still work.  

Furthermore, placement of wireless AP's will guarantee that if one switch goes down, wireless AP's in the area of the downed phones will continue to work for voice calls.  (This works vice versa if the other switch goes down.) If both switches go down - then all network devices go down.

It isn't ideal - but it fit budget constraints nicely and voice is still possible in all areas if a switch goes down.

Thanks,

Mike

[Jon Marshall]

Mike

No problem.

There was no criticism of your setup intended at all so I hope it didn't come across like that.

I was just trying to work out why HSRP was being used and I understand why now.

There is definitely something not working properly as it should definitely route locally on each switch.

Let me know if you need any help with it.

Jon

[mike0000111111]

Hi Jon:

First, I didn't begin to think you were criticizing my design.  I just wanted to relieve your confusion.

I tested your ideas this morning, and everything checked out and worked fine.  After some more investigation, I remembered why I was asking the question about using EtherChannel with an encapsulated Subinterface & IP Addr. for switch-to-switch routing.

Regrettably it had nothing to do with Intervlan routing, which was working fine.  But it does have something to do with routing between the two switches.  

Link Failure and High Availability

When I began to consider each case of link failure, I discovered 4 cases of link failure that created problematic results.  Two of the cases led to an extra hop, and two of the cases result in a black hole.  These ideas were tested with packet tracer to verify I had a problem.

These instances occur because I'm routing 3 vlans out of each switch.  Each problem could be resolved by a complete HSRP fail-over to the other switch.  But maybe the more elegant decision is a switch-to-switch route with an appropriate administrative distance (preferably using the EtherChannel)? 

Note: Primary is the primary WAN connection and Backup is the backup WAN connection.

Scenario 1: Extra Hop

Scenario 2: Extra Hop

\

Scenario 3: Black Hole

Scenario 4: Black Hole

Let me know what you think the ideal solution is: 1) use HSRP tracking to failover to the other switch, 2) create a direct switch to switch route using EtherChannel Subinterface with IP, or 3) some third option.

Thank you for your time,

Mike

[Jon Marshall · Accepted solution]

Mike

The extra hops could be resolved by allowing both routers to be active in terms of traffic so it did not have find it's way back the primary.

This would mean both routers advertise out the vlan IP subnets to the WAN and both routers advertise the same routes to the 3560s so each 3560 has two equal cost paths to all remote networks.

Is this not possible ?

How are you preferring the primary router at present from the 3560s because the routers are connected via P2P links so there is no HSRP on that side ?

Scenarios 3 and 4 should not be black holes.

What you need is a dedicated vlan for routing between the 3560s. So this vlan would have an SVI on each switch and be allowed on the trunk.

There would be no end devices in this vlan and it is used purely for routing traffic between the 3560s.

Are you running a routing protocol between the 3560s and the routers currently ?

It would be better if you were because then you run the same routing protocol between your switches and peer with the SVIs for that dedicated vlan.

Then each 3560 sees the best paths to remote networks direct via the WAN routers but if the links fail then it can use the routes via the other switch. It won't use these unless the direct link fails because the routes via the other switch are an extra hop so they will have a worse metric.

So as i say you peer using the SVIs on the dedicated vlan and you make all the other vlan interfaces passive under the routing protocol and that way they do not form peerings between each other across the interconnect.

Can you clarify about the first scenarios ie. how are you currently preferring the primary router ?

As always please feel free to ask any queries if what i have said doesn't make sense.

Jon

[mike0000111111]

Hi Jon:

Great idea about peering over a dedicated VLAN.  You suggested this before, but without the additional context - I wasn't sure if it was feasible.

1. The Primary WAN goes over a wireless connection to a local ISP.  The other WAN goes over cellular connection and uses a cellular data plan at $10/gb.  Preferring the ISP over the cellular connection is definitely ..... preferred.

2.  HSRP stays between the two switches and does not cross over between switches and routers.

3. EIGRP is running between switches and routers.

4.  The secondary WAN connection would have a higher administrative distance than the primary WAN.  I intended to use IP SLA to ping the primary ISP.  If the connection went down, a script would demote the Primary Wan route and have it propagated vis a vis EIGRP to the rest of the LAN. If the connection comes back up, I'd script a promotion of the primary WAN connection and again it would be propagated.  To be candid - this was my next project.

Thank you again for the suggestion - I'll try to rig that up this afternoon.

-Mike

[Jon Marshall]

Mike

Okay that makes sense.

I was assuming a WAN where you were receiving EIGRP but it sounds like you are injecting a default route into EIGRP.

If so then yes the dedicated vlan and IP SLA for failover should sort most issues out.

Jon

[Jon Marshall]

Mike

Just as a follow up to my last post. You said -

This is how I make sure no devices are orphaned when one switch goes down and it's gateway disappears.  The HSRP virtual gateways, then, are being advertised by EIGRP as the available route for each VLAN.  Consequently, all advertised routes lead to either one, or the other, switch.

I don't understand what you mean. HSRP is used for end clients. Why would your routers being advertising the HSRP VIPs and where are they advertising them to ?

I assumed you had access switches and that is why you were using HSRP. If there are no access switches and clients are directly connected to the 3560s then HSRP gives you nothing ie. the switch goes down so do all the clients on that switch.

Jon