FI Evacuation doesn't release network traffic

[Tiannanman1989]

We're trying to upgrade our 6248UP FIs to 6454s. There are no errors on any FIs/IOMs/servers. However, when we try to evacuate the subordinate FI, we lose all network traffic to the servers and the VMs. We do not lose access to UCSM and cluster lead can be moved between the FIs without issue. The other FI can be evacuated and rebooted normally. 

When we check the statistics for the FI network uplinks the problem FI is handling 300Mbs of data while the healthy FI is handling less than 1Mb. We've tried rebuilding the port-channel on the problem FI and TOR switch (9k) and disable/enable the network uplinks. We've also moved the network to different ports in the problem FI.

We're tempted to do a hard reboot on the problem FI, which may cause a significant outage if traffic doesn't move over to the other FI. We have a TAC case open, but the engineers "have never seen this before" and don't have any suggestions to resolve the issue yet. 

Curious if anyone else has seen something similar. FIs are on firmware 4.2(3e). 

[Kirk J]

I am assuming you likely have ESXi/virtualization in your environment?  Have you confirmed the VMNICs in your individual vswitches have dual path (A|B) connections?  Before attempting evacuation, do some random checks on the MACs of some of your guestVMs on the FIs to confirm where the MACs are learned/active ( connect nxos, show mac address-table |in xxxx)

I have had some cases over the years, where researching outages for FI reboots, and some esxi hosts had issues vmnic reordering/renumbering that resulting in the vswitches having both uplinks going to the same FI side.

Kirk...