Hello,
Can you let us know the reason you are replacing both the IOMs?
As for the procedure:
1) you need not do any failover for replace,
2) you can replace any IOM first, data path wise the FIs are active-active
3) there is no need to shut the server ports on the FI.
what you need to do,
1) make sure failover or multipath for ethernet ports is working fine. SAN connectivity is configured for multipathing
2) remove on IOM, install the new one.
i) From the CLI, connect to the NXOS for the side for which you have replaced the IOM. runn the command 'show fex' to check the IOM has come online.
ii) In the UCSM - check the IOM to be operable as well as fabric and backplane ports to be UP
iii) check the UCSM cluster state and make sure it is fully up and not downgraded.
./Abhinav