Load-balancing over two single homed DS3 links

[Russell Gibbons]

I have a situation where on each side of a link I have one switch and two routers.

The routers at the main site are connected via DS3 to the routers on the other

side, thus: HORouter1 <==> RORouter1, HORouter2 <==> RORouter2. The two HO routers are connected to a HO switch - but not directly to each other, and the same goes for the RO routers and switch. I only control the switches, the ISP controls the routers. Each of the switches are 3750 stacks. Right now, the bandwidth utilizations stays on one circuit till it his about 90%, at which point the second curcuit picks up. Can I configure the switches to do per-packet - or at least better than overflow - load-balancing in the current setup? The ISP routers are 3800s, and have unused GB ports. I would think that adding an additional link between then may enable the ISP to do loadbalancing on their side - is that right? Finally, what if at the HO site only, instead of both routers being attached to a single stack of 3750s, each router was plugged a 6509, and the 2 6509s were connected via 10GB?

Thanks all, and this is my first post here, so be a bit gentle please

[Jon Marshall]

How is the 3750 stack connected to the routers at each site ie. P2P routed links to each router, a shared vlan ?.

Are you routing the vlans off the 3750 stack ?

Are you exchanging routes between the routers and the 3750 stack ? If so when you do a "sh ip route" on the 3750 stack does it see 2 equal cost paths to remote destinations ?

Jon

[Russell Gibbons]

1. shared vlan

2. no

3. no

the only route on the 3750 is the default gateway, which is back to the core. A trace from the 3750 stack at the HO to a WAP at the RO goes into the HO core first. We do use eigrp internally, so I do not believe it would be a big deal to put the stack into eigrp.

From the core 6509, a sh ip route shows two equal path costs through the vlan with the two HO routers as the next hops.

Edit: I added a crudely drawn network diagram.

[Jon Marshall]

regibbons73 wrote:
1. shared vlan
2. no
3. no
the only route on the 3750 is the default gateway, which is back to the core. A trace from the 3750 stack at the HO to a WAP at the RO goes into the HO core first. We do use eigrp internally, so I do not believe it would be a big deal to put the stack into eigrp.
From the core 6509, a sh ip route shows two equal path costs through the vlan with the two HO routers as the next hops.
Edit: I added a crudely drawn network diagram.

So the 3750 switch is acting as L2 only and the 6500 and the 3800 routers are EIGRP neigbors ?

In that case the links should be load-balancing correctly. I say correctly because per-destination load-balancing is never exact ie. one connection could be sending a lot of traffic down one link and one can be sending little down the other.

The 6500 does not do per packet load-balancing, only per destination.

Jon

[Russell Gibbons]

Soooo...either I need a router instead of the 3750 stack between the 6509 and the 3800s, or the 3800s need to be directly connected, or both DS3s need to home in one 3800, the latter two scenarios being an ISP-based load balancing solution? Grrr....

Thanks for all of the help!!

[Jon Marshall]

regibbons73 wrote:
Soooo...either I need a router instead of the 3750 stack between the 6509 and the 3800s, or the 3800s need to be directly connected, or both DS3s need to home in one 3800, the latter two scenarios being an ISP-based load balancing solution? Grrr....
Thanks for all of the help!!

Russell

Another option if you know the traffic patterns between the 2 sites is you could look to use PBR to force one lot of traffic down one of the links and another lot down the other link eg. if you had for example 2 vlans in HO comunicating with the RO and the traffic was relatively equal you could use PBR to send one vlan down one of the links and one vlan down the other.

Just a thought.

I'm still not sure everything is working as expected though. You should not consistently see 90% on one before utilization of the other link. You may see this at certain periods but it should generally speaking even itself out.

The 3750 is definitely acting as a L2 switch ?

Jon

[Russell Gibbons]

that option isn't, because the RO is on one vlan...or at least one data vlan. And while the one circuit isn't always at 90%, the second is always ~5% or less. For weeks or months, you can see that the second circuit only begins to pass traffic once the first circuit is close to full. Would replacing the HO 3750 stack with our own 3800 give us a platform to launch per-packet load-balancing?

I am very sure the 3750 is only being used as a pass-thru L2 device.

[Jon Marshall]

regibbons73 wrote:
that option isn't, because the RO is on one vlan...or at least one data vlan. And while the one circuit isn't always at 90%, the second is always ~5% or less. For weeks or months, you can see that the second circuit only begins to pass traffic once the first circuit is close to full. Would replacing the HO 3750 stack with our own 3800 give us a platform to launch per-packet load-balancing?
I am very sure the 3750 is only being used as a pass-thru L2 device.

PBR is still an option ie. just because it's one vlan doesn't mean you can't split the vlan up ie.

192.168.5.0/24 - simply split it up as 192.168.5.0/25 and 192.168.5.128/25

It might be worth trying this before installing a router. There can be performance issues with per packet load-balancing in that packets can arrive out of order and cause retransmissions etc.

Another thing you may want to consider is collecting Netflow data from your 6500 switch to see exactly what is going between the 2 sites.

As i said before if the 6500 sees 2 equal cost routes to the same RO destination and the equal cost routes are pointing to the LAN interfaces of the 3800 routers then you should be seeing a more even distribution of traffic on the links.

Just to be sure when you say you see equal cost paths on the 6500, have you checked both the 6500 at the HO and the RO sites ?

Jon

[Russell Gibbons]

there are only 6509s at the HO site

CoreSec#sh ip route 10.70.20.0
Routing entry for 10.70.20.0/24
  Known via "eigrp 100", distance 90, metric 152832, type internal
  Redistributing via eigrp 100
  Last update from 10.70.9.12 on Vlan70, 3w3d ago
  Routing Descriptor Blocks:
  * 10.70.9.11, from 10.70.9.11, 3w3d ago, via Vlan70
      Route metric is 152832, traffic share count is 1
      Total delay is 420 microseconds, minimum bandwidth is 18018 Kbit
      Reliability 255/255, minimum MTU 1500 bytes
      Loading 2/255, Hops 3
    10.70.9.12, from 10.70.9.12, 3w3d ago, via Vlan70
      Route metric is 152832, traffic share count is 1
      Total delay is 420 microseconds, minimum bandwidth is 18018 Kbit
      Reliability 255/255, minimum MTU 1500 bytes
      Loading 45/255, Hops 3

CorePrim#sh ip route 10.70.20.0
Routing entry for 10.70.20.0/24
  Known via "eigrp 100", distance 90, metric 152832, type internal
  Redistributing via eigrp 100
  Last update from 10.70.9.12 on Vlan70, 5w5d ago
  Routing Descriptor Blocks:
  * 10.70.9.11, from 10.70.9.11, 5w5d ago, via Vlan70
      Route metric is 152832, traffic share count is 1
      Total delay is 420 microseconds, minimum bandwidth is 18018 Kbit
      Reliability 255/255, minimum MTU 1500 bytes
      Loading 2/255, Hops 3
    10.70.9.12, from 10.70.9.12, 5w5d ago, via Vlan70
      Route metric is 152832, traffic share count is 1
      Total delay is 420 microseconds, minimum bandwidth is 18018 Kbit
      Reliability 255/255, minimum MTU 1500 bytes
      Loading 45/255, Hops 3

[Jon Marshall · Accepted solution]

Russell

Sorry, should have checked your diagram more carefully.

So at the RO is the 3750 L2 as well or is it the default-gateway for the vlan ?

If it is L2 are you using HSRP/GLBP on the 3800 routers ?

Jon

[Russell Gibbons]

the RO 3750 is also L2. The RO is a flat network - 1 vlan for data, 1 for phones, no interaction.

I cannot see the config of the 3800s, as those are the ISP routers, but they do appear to be running either HSRP or GLBP, most likely HSRP since it always takes the first route.

[Jon Marshall]

regibbons73 wrote:
the RO 3750 is also L2. The RO is a flat network - 1 vlan for data, 1 for phones, no interaction.
I cannot see the config of the 3800s, as those are the ISP routers, but they do appear to be running either HSRP or GLBP, most likely HSRP since it always takes the first route.

Then that could well be the problem. You either need to -

1) ask your ISP to run GLBP so you could get a spread across the circuits or -

2) run your 3750 stack as L3, route the vlans off there and have 2 static routes pointing to both 3800 routers. Even if you did this though you would need to talk to your ISP as the internal LAN interfaces of the 3800 switches would need modifying.

Jon