Interesting. We started having an issue with this too in the past few weeks. All b200m3 esxi blades management, which sits on an isolated VLAN from production servers, can't talk to each other if their uplink port is on the opposite FI. vcenter VM sits on same VLAN. When hosts are on opposite fabric from vcenter vm, vcenter isn't able to talk to them (unable to ping. Verified arp was correct though). However can ping all of them from the outside of the VLAN without issue.
We used evac mode to test if everything went to A if everything worked and it does. But as soon as we go splitting the fabric we lose connection to whatever sits on the B fabric (vcenter is on A for now).
Our northbound switches are N5Ks that are only layer 2. There's a VPC setup and we have verified the config is right. I've been up and down the ucs config and can't find anything. My last chance hope is a modification we made months ago (problem didn't surface after change though) to the nic policy which was to publish the Mac address to all VLANs (in hopes of making the ESXi dvswitch health check work).
We run our management over a pair of vnics that are using a standard vswitch. No modifications have been made to the vswitch.
Its really strange. I'm in the process of ordering a network tap so I can see if the ucs is even sending traffic northbound or not. If it's not then I'm really stumped. If it is then I think it's a 5k problem
But it sounds like your issue is very similar. Stuff on A can't talk to B. In my case though it appears to be only certain nics or certain VLANs. I tried placing a VM on another VLAN on fabric A and a physical UCS b200m3 windows server on the B side, same VLAN, and they never lost connectivity. In my case it appears to be a specific VLAN. I really don t get it though. Its the same VLAN. All later 2 stays within the FI or within the 5K. It never hits a router or firewall.
I'm gojng to plug away at it more today but if I can't find a solution I'm opening a tac case.
