Cisco MDS Port channel load balancing

[andrechamberslv]

A customer recently asked an interesting question about exchange based load balancing on an FC port channel. The platform is UCS with an 8 and 16 port channel per fabric interconnect on two separate UCS domains. The application is Oracles data warehousing which has been known to saturate 4 x 8gb fc links. Since the balancing method is exchange based what constitutes the start and end of an exchange? We are trying to avoid a condition where and intense read write conversation locks to a single link in the port channel and not spread across 8/16 links. Where can I find more information about exchange based routing protocol or how should I go about managing extreme io in a converged infrastructure.

[Paresh Gupta]

FC header contains Originator Exchange ID (generated by initiator) and Responder Exchange ID (generated by target). These two fields are unique per exchange. Frames with different OX_ID or RX_ID belong to different exchanges. For MDS, frames belonging to the same exchange always take the same physical link under a port channel. This is needed to keep the frame in sequence.

If all of the links in the port-channel are touching the max capacity, you should recommend your to increase links in the bundle (max 16) or upgrade to 16G links. If few of the links are heavily utilized while other links in the same port-channel are under utilized, you may want to check the application or HBA for capability of breaking down the large reads/writes under smaller exchanges.

[andrechamberslv]

Thanks Paresh...

Since the UCS doesn't have an option for a load balancing method for the connected SAN port channel what method is used? The MDS will support exchange and flow based but how does UCS know which one to use? If the port channel on the MDS side is set to exchange based does the UCS side of the port channel auto-negotiate and use whatever method is selected on the MDS side of the port channel? I guess I am trying to figure out using a 9700/9500 series director what load balancing options are available and how UCS( 6296 fabric interconnect) figures out which load balancing method to use.

Thanks in Advance....
 

[Paresh Gupta]

The default loadbalance method on FI and MDS is src-dst-ox-id based. Note that loadbalancing is done by a device on *outgoing traffic*. FI and MDS do not have to negotiate anything here. Technically, one device can do src-dst-id based while the other can use src-dst-ox-id. However, in your case there is no reason for such a change.

As I wrote before, If all of the links in the port-channel are touching the max capacity, you should recommend your customer to increase links in the bundle (max 16) or upgrade to 16G links. If few of the links are heavily utilized while other links in the same port-channel are under utilized, you may want to check the application or HBA for capability of breaking down the large reads/writes under smaller exchanges. If no traffic is going on few of the links at all, then I would suspect UCS to FI pinning as well.