During a link failure, frames that were in flight on a failed link will be lost, and successive frames could potentially be re-routed by FSPF to another path. If it is a port-channel, then the port-channel manager (not FSPF) will redirect frames to a working port-channel member link. Regardless, out of order frame delivery is possible when there are link changes. IF the receiving FC device is sensitive to out of order frame delivery (e.g. might lock up) then IOD guarantee can be enabled.
It is also important to make the distinction between out of order frames versus out of order exchanges. As mentioned, out of order frames is possible during link change. However, with any load-balancing scheme, all frames associated with an exchange will traverse the same link. So if that link fails, the exchange will be invalidated by the receiver due to some frames on the wire being lost. The key point is that some frames in flight will be dropped. They won't all arrive at the receiver so the exchange will be incomplete. Rather than pass an incomplete exchange up to the ULP and risk data corruption, the receiver must invalidate the exchange and throw it away (if it complies to the FC standards). When an exchange is invalidated by the receiver, the ULP in the sender should detect a timeout and resend the data.
So, it only makes sense to enable IOD if the receiving device is sensitive to out of order FRAMES.
Here is how IOD works. When IOD is enabled (per switch), the switch will hold incoming traffic for 2 seconds (the default value). We only apply the hold-down timer on the VSANs where IOD is enabled (since 1.3.4) and only to the routes affected by the link change - not all ISLs on the switch. When 2 seconds have lapsed we will forward the traffic we have been buffering, assuming an alternate route exists. This hold down time can be configured and this is called 'network fcdroplatency'. This is a configurable parameter through the CLI and can be increased or decreased depending on number of hops in the fabric.
If the link failure is within a port-channel bundle, the above still applies except that the hold down time is fixed at 500ms before we can forward on another channel member. We call this hold down time the 'switch fcdroplatency'. The default is 500ms and is not configurable. today, even though the command exists in CLI today.
We recommend to leave In Order Guarantee feature disabled (the default). By disabling this feature, this does NOT mean that frames will be freely transmitted out of order. This simply means that the hold down time is bypassed on link change and forwarding will resume much quicker (assuming alternate routes or port-channel members are available). Most devices these days can tolerate out of order frame delivery.
The other scenario is when there is no link changes, ie a stable fabric. If there are equal cost FSPF paths or multiple links available in a port channel, then it is still possible that when doing src-dst-ox based load balancing that some exchanges will arrive at the receiver before other exchanges. If it is a device that cannot handle out of order exchanges then leave IOD disabled and simply configure the MDS load balancing scheme to be based on src-dst. This can be done on a per VSAN basis.
