vNICs failing on uplinks that are up

[Micheline Murphy]

Hello Cisco Community--I have four blades in a UCS chassis.  All of them are B420-M3s and they are running the following firmware.  BIOS B420M3.2.2.6e 062220160138.  Board controller 12.0. CIMC Controller 3.1(21d).  Each blade has a VIC 1240.

I created an updating service profile template that I used to create four service profiles.  Three of these service profiles were sucessfully associated to blades in my chassis, and these three blades have full connectivity both LAN and SAN. 

The fourth blade has taken its service profile, but all of the vNICs have failed, each one throwing the following errors:

Each blade has twelve vNICs, two each to six different pods.  Each of the different vNICs is pinned to a different port-channel, and each pod has its own VLAN and subnet space.  This configuration is working just fine in the other three servers--full LAN and SAN connectivity.  All of the port-channels are up and all of the member ports within each port-channel is up.

I've tried re-acknowledging the malfunctioning server (twice), disassociating and then reassociating the service profile.  I've tried resetting connectivity and also disable/enable on the vNIC.  I'm at a bit of a loss as to what to try next.  Any thoughts would be greatly appreciated.

MM

[Kirk J]

Greetings.

Please confirm you have decomissioned the blade, before re-acking.

Also, from the UCSM CLI try:

#reset slot x/y (where x is chassis, and Y is problem blade number)

If it's still not bringing the links up after you've installed an OS, then you may need to open a TAC case, where they can check tech support logs for issue with the VIC adapter, etc.

Kirk...

[Brandon Rumer (Cisco)]

Hi Micheline,

You went through all the troubleshooting steps that make sense. Great job!

When reading your post I found myself making a few assumptions, which I don't want to do. Has this ever worked, and is it a new install or did you only perform the firmware upgrade?

If its a new install: Check to make sure the hardware inventory is the exact same on the blades. If the problem blade has a mezzanine adapter and the others don't, then you may have a problem. Check your vnic/vhba placement profiles, do they line up with what they should be? Here's a good link for explaining the pinning which have the possibility of getting you in trouble.
http://www.tbijlsma.com/2012/03/how-ucs-achieves-80gbe-of-bandwidth-per-blade/

All your links from the FIs to the chassis' IO Modules are good, right?

I hope this helps,
Brandon

[padramas]

Other things to consider

1) since you have dis-join L2 ( uplink port-channel with different set of VLANs ) , make sure the vNICs are configured with right set of VLANs allowed in the port-channel

2) OS installed in the blade ? does it have enic driver ?

[Wes Austin]

As others suggested, I would ensure that the corresponding link is actually up and operational on the OS side and the correct driver is installed.