Hello!
The purpose of the rotate option used in the port-channel load-balance global configuration command is documented under the "Configuring ECMP Load Balancing" section of the "Configuring Port Channels" chapter of the Cisco Nexus 9000 Series NX-OS Interfaces Configuration Guide. Note that in the documentation, this option is defined for the ip load-sharing global configuration command and not the port-channel load-balance global configuration command - however, the rotate option is valid for the port-channel load-balance global configuration command too, and its meaning/definition is identical.
"The rotate option causes the hash algorithm to rotate the link picking selection so that it does not continually choose the same link across all nodes in the network. It does so by influencing the bit pattern for the hash algorithm. This option shifts the flow from one link to another and load balances the already load-balanced (polarized) traffic from the first ECMP level across multiple links.
If you specify a rotate value, the 64-bit stream is interpreted starting from that bit position in a cyclic rotation. The rotate range is from 1 to 63, and the default is 32."
This is an esoteric definition, so explaining the problem that the rotate option attempts to solve is probably the better route.
The rotate option is NX-OS's solution to the classic CEF polarization problem (Another good article that covers this scenario from a Cisco Nexus perspective is the Troubleshoot polarization in port-channel load balancing document.) This is where a multi-tier network topology with redundant active/active links (achieved either through Equal Cost Multipathing [ECMP] or port-channel load balancing via Multi-Chassis EtherChannel [MCEC] or Multi-Chassis Link Aggregation Groups [MC-LAG], either method will encounter the same problem) that has multiple network devices leveraging the same load balancing/hashing algorithm (which is likely if most network devices are the same model or the same series) that are subject to the same traffic flows.
Even though Nexus switches don't run CEF (Cisco Express Forwarding), they are subject to this "same hashing algorithm" issue. In fact, any networking device that leverages hashing algorithms from any vendor would encounter this same issue - this is not a Cisco-specific issue, this is an industry-wide issue!
Before tinkering with the rotate option, my advise would be to investigate the two following points:
- We can see that the traffic on this port-channel/vPC is polarized, with the primary Nexus switch transmitting much more traffic than the second Nexus switch towards the relevant host. What I would investigate next is, where are the Nexus switches receiving this traffic from? Is the traffic originating from other hosts that are directly connected to these two Nexus switches, or is the traffic originating from hosts in remote parts of the network?
- Next, I would investigate the profile of the traffic being sent to this relevant host. Specifically, I would try to identify whether there are one or more elephant flows that are polarized to the primary Nexus switch. If there's multiple elephant flows, then it may be worth tinkering with load balancing algorithms to try to distribute those elephant flows between the primary Nexus switch and the secondary Nexus switch. However, if there's a single elephant flow, tinkering with load balancing algorithms won't help you since hashing algorithms work on flows, not on individual packets.
You brought up per-packet load balancing/hashing in your post, so I want to stress the last sentence of my second bullet point. The overwhelming majority of network devices - including Cisco Nexus switches - only support flow-based load balancing/hashing; per-packet load balancing is not possible with these switches. I have a post on my personal blog that discusses how load balancing (again, either ECMP or port-channel load balancing, both are typically identical in behavior) works on most modern network devices that demonstrates this point in a bit more detail.
I hope this is helpful - thank you!
-Christopher
