C-240 cannot communicate with the B200 Blade servers which are connecting to the sameFabric Interconnect after one side's uplinks is down
Recently when we are doing Fabric Interconnect fail-over test, we met a very strange problem.
We have one UCS 5108 Mini chassis with four B200M3 Blade servers and two 6324 Fabric Interconnects. Another C-240 M3 is connecting to this 5108 chassis with two VIC 1228 cards and two Cisco Direct-Attach Breakout Cables, and it is managed by the 5108's UCS manager.
The C-240 has windows 2012 R2 installed, and the B200M3 has VMware ESXi 5.5 installed. and the C-240 and the B200M3's management vmkernels are on the same VLAN.
When both sides' uplinks are up and running, everything works fine. But when we disabled uplinks of one side, the C-240 fail to communicate with the B200M3's management vmkernels, cann't ping with each other. Although all of the C-240 and B200M3's mgmt vmkernels can communicate with outside of the UCS system or even VMs running on the B200M3s.
We tried to disable each side's uplinks separately, and got the same result. The communicate between C-240 and B200M3 resumes as soon as we recover the disabled uplinks.
We also installed windows 2012 R2 on one of the B200M3 servers, when one side's uplinks are disabled, the Windows system on C-240 can communicate with windows system on B200M3 server, and the ESXi mgmt vmkernels can communicate with each other. but Windows systems can't communicate with ESXi hosts.
I'm having a similar issue where I'm using the same service profile for all my blades. However, I have 2- M4 blades that are unable to communicate with the other blades (M3, all are ESXi). The service profile is utilizing 2 nics--one for Fabric A, and one for Fabric B. If I were to remove one of the uplinks from ESX (for example, fabric A), i would get communication. I have ruled out that Fabric A is the issue, since i was able to get communication on Fabric B.
I did recently upgrade the FW for the infrastructure to 2.2(5b) as well as for the 2-new blades. The M3s are still on 2.2(1c)...so i'm thinking i may have to upgrade the FW on those servers to see if that fixes my problem.
My problem resolved by upgrading the enic driver at the ESXi hosts from 188.8.131.52 to 184.108.40.206 to match the VIC's firmware verion 4.0(3a).
However, I encountered another issue: ESXi hosts cannot communicate with each other across Fabrics. For example, if ESXi host1's vmk0 is active at Fabric A, and ESXi host2's vmk0 is active at Fabric B, they cannot talk to each other even they are in the same chassis, but they can talk to other servers outside this chassis.
reboot the hosts may get communication but it appeared three times and I am not sure when this will come out again or how to reproduce this scenario...
Hi, in those cases where you are running ESXi, can you check if the vmk Management PG has picked up a MAC address from one of the physical nics? If it has then you might see random traffic issues like the ones you describe.
If the vmk has a Cisco MAC (00:25:B5...) instead of a VMware MAC (00:50:56...) then have a look at these articles:
Upload the Script to Datastore
1. Browse to any Datastore of ESXi Host2. Upload the folder "lsigetvmware_022817" ex. Suppose the datastore name is "Datastore3"
Run the Commands
Enable SSH of ESXi Host
SSH to ESX...
SymptomsCisco Prime is unresponsive. Web GUI will not load. Device doesn't respond to ping.DiagnosisPrime chassis is off.SolutionSince the CIMC uses Flash you can no longer use your bowser. SSH to CIMC address instead. show chassis detailChassis:Powe...
Very first data cloud — Evolution of data lake to hybrid cloud
Today, in this digitized world, data is being generated at a very high rate and from many different sources, and it is growing exponentially in many use ca...
PXE (Preboot eXecution Environment) Server allows unattended or automated OS installation over the Network. The main benefit of pxe is that we don’t need any bootable drive to boot OS(Operating system) and we do not need to to burn any ISO file into DVD o...
New episode on the Cisco UCS X-Series Modular SystemListen: https://smarturl.it/CCRS8E23 Follow us: https://twitter.com/CiscoChampion The Cisco UCS X-Series Modular System simplifies your data center, adapting to the unpredictable needs of...