I'm installing a proof of concept UCS blade system for a customer and notice something a little strange. After racking and cabling I configured ports for management access to the FIC's and configured VPC's on the Nexus 5010's for uplinks from the FIC's. I was in a bit of a hurry and didn't verify both management ports were active (stupid mistake, I know). The system shipped with firmware 1.3 installed so I have been upgrading to 1.4 to demonstrate the latest and greatest features. During the upgrade process I kept having issues and have had to recover from some bad BIOS upgrades. I also started having problems connecting to the KVM console. I assumed this was also related to the firmware upgrades (not always smooth on UCS yet) and have been trying to diagnose the problem. After checking CIMC status and event logs, resetting, reloading firmware, reseeding servers, doing just about everything I could think of, I thought maybe it was a problem upstream. Maybe an IP range that was already in use elsewhere in the network. None of that was the problem.
Instead the issue was that the management 0 interface on FIC-B was shutdown on the uplink switch. I enable the port and everything is cool! I can connect to all the KVM consoles for all the servers. My question is, with the L1/L2 connections available, why didn't CIMC access traverse this link? At the very least why didn't UCSM notice that the management interface on FIC-B was down and either force all CIMC connections through FIC-A or give some kind of warrning that FIC-B management interface 0 is down?
This was a very frustrating issue. I would hate for someone else to have a management link fail and have to scour all the possibilities for a day like I did.
