Connecting 2 switches with unequal speed links

[mike.gusway]

Hello,

We have installed a 9407 in a new building down the street from our primary datacenter, which has a VPC pair of Nexus 3Ks). We have 2 separate ISPs each providing us a real-time L2 link to that building. The intention is to provide redundant connections to support business-critical devices over there.

The handoffs from both ISPs are 1Gbps. The primary circuit is capped at 1Gbps and the secondary connection is capped at 100Mbps but the bandwidth can be manually increased if needed (for additional cost, of course). Because the ISP handoffs are both 1Gbps, I was able to create an LACP etherchannel between our datacenter and the new building, and it seems to be working during our testing. Both ISPs are showing traffic across their links. Can default source-dest-mac port-channel load balancing saturate the 100Mbps link and cause issues for any hosts/clients doing bandwidth-intensive things like video streaming? Is there a way to prioritize traffic so that it uses the higher-bandwidth primary link unless it is down? Should we not use an etherchannel and instead let spanning tree handle which link is active?

Thank you

[xAdventix]

How much traffic are you actually utilizing? I would think Etherchannel can be a bit messy with different link speeds, if you never max out the 1Gbps link then just run either STP and block the 100Mbps or run HSRP. 

[balaji.bandi]

If the links are not equal, i do not suggest to put them in Port-channel, rather you can use as trunk. or make Active / Standby kind of link.

Once you observer you primary link never go down or how many times down, based on the input you can make business case to increase 100MB to next level. 

once caveat if 1GB fails all the traffic move to 100MB, but you still have connectity ( as you said can be dynamically increased, if the primary link can not be restored certain time).

[mike.gusway]

What about using lacp port-priority to prioritize the 1Gbps link, and set lacp max-bundle on the port-channel to 1? This way, both links would be bundled in the port-channel, but the 1Gbps would be prioritized and active while the 100Mbps would be in standby until the 1Gbps isn't available. Is that feasible? The concern is that by simply using spanning tree, there would be a period of time that the other building would have no connectivity if the 1Gbps link went down while spanning tree recalculates. With both links in a port-channel together, we simulated loss of one circuit or the other and never dropped pings across the port-channel to devices in the other building.

Thanks

[Reza Sharifi]

What about using lacp port-priority to prioritize the 1Gbps link, and set lacp max-bundle on the port-channel to 1? This way, both links would be bundled in the port-channel, but the 1Gbps would be prioritized and active while the 100Mbps would be in standby until the 1Gbps isn't available. Is that feasible? 

Yes, since the physical links from both providers are Gig, I would keep the PO and change the priority for the 100Mb link to something less than the default, which is 32768. This way you are not using STP to block a link.

HTH

[Joseph W. Doherty]

Yes, it should work, and as you've discovered appears to work. (Nice thinking outside the box.)

BTW, if you're using a fast variant of STP, your outage time might be in the same ball park.

Also BTW, what Reza notes about SP bursting limits may be worth checking. I.e., again. some SPs set burst limits relative to some bandwidth utilization over a longer time period, meaning you might be able to use the back-up gig, at gig speed, for a day or two without additional charge. If that's not the default, you might be able to contract that for a very small additional charge. (Why bother, when you can increase the caps when needed? Well, increasing the caps takes some time [usually a couple of hours] and during the time, your traffic, if routinely using more than 100 Mbps, may be subject to a hard bandwidth cap, which depending on the nature of your traffic, might have much negative impact.)

[mike.gusway · Accepted solution]

This worked!

I set "lacp rate fast" on all interfaces so that control packets are sent every second instead of every 30 seconds by default.

On the 9470, I set a low value (higher priority) for system-priority, and set a low value for port-priority on the 1Gbps link, leaving the 100Mbps link at the default port-priority. When I set "lacp max-bundle 1", it put the 100Mbps link into hot standby.

On the Datacenter/Nexus side, it also showed the 100Mbps link in hot standby. I simulated a circuit outage by turning off the port connected to the 1Gbps link, and within a couple of seconds, traffic was flowing across the 100Mbps link. When I turned the 1Gbps link back on, it switched over to being the active link almost instantly. I only dropped 2 pings between buildings. This will suit our needs, as we may exceed 100Mbps of traffic between the 2 buildings quite often, but probably won't come close to hitting 1Gbps and didn't want to pay for a second 1Gpbs link just for redundancy. In the event the 1Gpbs circuit goes down, the building can continue to work at a slightly degraded max bandwidth with hardly any interruption to service and without any manual intervention. If the 1Gbps circuit will be down for an extended period of time, we can increase the bandwidth on the secondary link through the ISP to keep up with demand.

Thank you to everyone who read and commented on this.

[Reza Sharifi]

Nice test and thanks for the feedback!

If the bandwidth is critical to your business, the other option would be to talk to the service provider that hands you the 100Mb circuit and see if they let you burst to a Gig when the primary is down and when you need the speed. This way you only pay for what you use. Depending on the provider and how many circuits you use from them, some let you burst for free for so many hours a month, which would be very handy when the primary provider has a maintenance window for a few hours.  So, it is matter of a conversation with your sales guy.

HTH

[balaji.bandi]

Oh good test, (i was in impression that 2 different Cirtcuits 1Gig and 1 100MB,) - after reading again, you have 2 Gig, but bandwidth only restricted for 1 circuit for 100MB,

so make sense and good report.