Distributed Load Sharing in OSPF

[arun kumar]

Hi,

I have a scenario where 7604 is connected (upstream) to 7200 (downstream) over 2 Ethernet links. OSPF is configured over 2 ethernet links. Ethernet links are connected to the switch and terminated as sub-interfaces in single physical interface of 7604 and 7200.

Suppose say both the ethernet links are taken for 10 Mb each and the peak traffic to the downstream location is 16 Mb.

Now when the traffic flows from upstream to downstream it is not getting load shared evenly over 2 ethernet links. One ethernet link carries 10 Mb and other one carries 6 Mb. Thus traffic flowing on first ethernet faces heavy packet drops.

But OSPF cost for 2 ethernet links are same and routing entry to the downstream's loopback shows 2 ethernet links in upstream (traffic always flow to the downstreams loopback).

Since 7604 is not supporting per packet load sharing configuration in interface, i m not able to configure it.

Is there any way that I can configure to have a distributed load sharing over 2 ethernet links.

thanks

Arun

[Joseph W. Doherty]

Addendum: "CEF will round-robin traffic flows", alhough it it appears to function as such, technically not truely round-robin for flows. There's a hash involved, and traffic between same IP SRC/DEST will take same path.

Additional details can be found here:

http://www.cisco.com/en/US/products/hw/modules/ps2033/prod_technical_reference09186a00800afeb7.html

and here:

http://www.cisco.com/en/US/tech/tk827/tk831/technologies_tech_note09186a0080094806.shtml

From the 1st reference:

"There are a few scenarios where per-packet load balancing is more advisable, e.g. the majority of traffic is between two hosts. "

From the 2nd reference:

"Packets for a given source-destination host pair might take different paths, which could introduce reordering of packets. This is not recommended for Voice over IP (VoIP) and other flows that require in-sequence delivery."

Since TCP doesn't require in-sequence delivery, it might not be clear how easy it is to degrade your end-to-end performance using per-packet load sharing even though your link balance utilization is fantastic.

Alhough if your equipment supported it, I still believe per-packet shouldn't cause an issue in the situation you've described, it's very easy for network to change causing a issue later. So, unless you really, really had the need to use this technique, I recommend against using it. If you do use it, some how you need to document it being used with the possible issues it might cause.

[arun kumar]

hi all,

thanks for your responses.... since the customer is not ready to upgrade the bandwidth at this point of time.. the only option left is to use etherchannel in the switch and bundle the two ethenet links.

thanks

Arun

[paolo bevilacqua]

Note that etherchannel, just like CEF, doesn't guarantee equal link utilization.

It may work better, or it may not. Or it may work fine one day, but poorly another.

These are all statistics (or stochastic if you want) methods without feedback from actual circuit utilization.

[Joseph W. Doherty]

I had thought about suggesting Etherchannel, but wasn't sure whether it would be an option on the 7200. Paolo's note about its effectiveness compared to CEF is quite correct.

However, unlike CEF, which except for just(?) the 4500 doesn't offer variations to the CEF hashing algorithm, many switches provide different attribute variations for the hashing algorithm which can impact how well it supports your traffic. Even so, to quote Cisco:

"Hash based mechanisms, for example, EtherChannel or Cisco Express Forwarding (CEF), were devised so that flows would be statistically distributed, based on mathematical functions, among different paths. However, the traffic rate for a flow is not considered, and it is possible for multiple high bandwidth flows to be sent on an already congested link while other links remain underutilized. Finally, link bonding technologies, such as Inverse Multiplexing over Asynchronous Transfer Mode (ATM-IMA) and Multilink Point to Point Protocol (MLPPP) try to slice packets so that they each packet is simultaneously sent on multiple links.

Link bonding technologies generally place a substantial load on the fragmenting device and the reassembly device, and are sensitive to intramember-link delay variation. Finally, the remote peer must be the same type of device, limiting the use of multiple WAN providers. "

Link bonding will also allow a single flow to obtain more than one link's bandwidth. Link bonding also preserves flow packet sequence, unlike packet-by-packet. (Not 100% certain, but there might be 3rd party Ethernet imuxs.)

OER/PfR technology supports dynamic link loading, although slow to react. It works well if, for example, there's one flow consuming all the bandwidth on its link over several minutes. It can push other traffic to a different link.

[edit]

"the only option left is to use etherchannel in the switch and bundle the two ethenet links. "

Have you discounted packet-by-packet as an option?

[Joseph W. Doherty]

Addendum 2:

From another post, just discovered the command:

"ip cef load-sharing algorithm {original | tunnel [id] | universal [id] | | include-ports {source [id] | [destination] [id] | source [id] destination [id]}} "

Seems to be supported on later IOSs.

"Load-Balancing Algorithms for Cisco Express Forwarding Traffic

The following load-balancing algorithms are provided for use with Cisco Express Forwarding traffic. You select a load-balancing algorithm with the ip cef load-sharing algorithm command.

•Original algorithm-The original Cisco Express Forwarding load-balancing algorithm produces distortions in load sharing across multiple routers because the same algorithm was used on every router. Depending on your network environment, you should select either the universal algorithm (default) or the tunnel algorithm instead.

•Universal algorithm-The universal load-balancing algorithm allows each router on the network to make a different load sharing decision for each source-destination address pair, which resolves load-sharing imbalances. The router is set to perform universal load sharing by default.

•Tunnel algorithm-The tunnel algorithm is designed to balance the per-packet load when only a few source and destination pairs are involved.

•Include-ports algorithm-The include-ports algorithm allows you to use the Layer 4 source and destination ports as part of the load-balancing decision. This method benefits traffic streams running over equal cost paths that are not load shared because the majority of the traffic is between peer addresses that use different port numbers, such as Real-Time Protocol (RTP) streams. The include-ports algorithm is available in Cisco IOS Release 12.4(11)T and later releases. "

If you platform/IOS supports it, changing CEF's hash might improve your overall per-destination load balancing.