With the introduction of rfc2328, or OSPFv2, the path selection criteria for external or summary routes changed.
Specifically, the new changes call for preferring a non-backbone path over a backbone(area 0) path for the prefix.
Previously, we relied on cost to determine path selection, without regard to whether it was a backbone or non-backbone path.
IOS has always treated rfc1583 as the default behavior for path selection, and as support for rfc2328 was added, IOS added a toggle to prefer this new method of path selection, but continued to keep rfc1583 behavior set by default.
NX-OS, however, was launched with rfc2328 only behavior by default, and then to conform with RFC specs, added a command to make it optionally rf1583 compliant.
This difference in default behavior can lead to ospf loops, where you have some devices pointing to one path as the shortest path, and other devices pointing to a different path as the shortest path.
rtrB#show ip route 192.168.0.0 Routing entry for 192.168.0.0/16, supernet Known via "ospf 1", distance 110, metric 25, type extern 1 Last update from 10.0.0.1 on GigabitEthernet0/0, 00:20:40 ago Routing Descriptor Blocks: * 10.0.0.1, from 126.96.36.199, 00:20:40 ago, via GigabitEthernet0/0 Route metric is 25, traffic share count is 1
As you can see, the IOS router points out g0/0 to the n7k, and the n7k(ABR) points out e2/10 to rtrB. This is a loop!
Now, we will turn on the rfc1583compatibility command on the n7k(ABR) and it will allow us to use the metric for our decision, like the IOS router is doing.
ABR(config)# router ospf loop ABR(config-router)# rfc1583compatibility ABR# show ip route 192.168.0.0
rtrB#show ip route 192.168.0.0 Routing entry for 192.168.0.0/16, supernet Known via "ospf 1", distance 110, metric 25, type extern 1 Last update from 10.0.0.1 on GigabitEthernet0/0, 00:30:03 ago Routing Descriptor Blocks: * 10.0.0.1, from 188.8.131.52, 00:30:03 ago, via GigabitEthernet0/0 Route metric is 25, traffic share count is 1
As you can see, if all the ospf routers in this domain are configured to run according to the same rfc mode, then we do not experience these same loops, as the complete domain is consistent in what it considers to be the shortest path.
On NX-OS, you can use the "rfc1583compatibility" command to make the device align with default IOS behavior.
On IOS, you can use the "no compatible rfc1583" command to make the device align with default Nexus behavior.
Which ever of the two options you use is your choice, but I will not go into the major design differences here in this post.