Introduction
The UCS chassis hosts not only the blade servers but also the IOM and fan modules. The fan modules keep the system running cool but efficiently circulating the hot air outside and cool air inside. The fan modules are hot swappable without causing an electrical hazard or damage to the system. However, you can only remove one fan module while the system is operating. Removing more than one fan module could cause overheating.
Problem
Most ot all of the fans in the UCS chassis are flagged as inoperable, although the fans are working fine on physical inspection in the datacenter. Getting the following error:
ack="yes"
cause="equipment-inoperable"
changeSet=""
code="F0373"
created="2014-03-06T21:46:44"
descr="Fan 1 in Fan Module 1-1 under chassis 1 operability: inoperable"
dn="sys/chassis-1/fan-module-1-1/fan-1/fault-F0373"
highestSeverity="major"
id="520430"
lastTransition="2014-03-06T21:46:44"
lc=""
occur="1"
origSeverity="major"
prevSeverity="major"
rule="equipment-fan-inoperable"
severity="major"
status="created"
tags="network,server"
type="equipment">
Troubleshoot
First check if the messages are transient, which means if they get cleared within some time after the issue. If yes, then its a cosmetic issue and there is no problem. If the messages do not get cleared and persist then gather the output of command "show platform soft cmc showi2c \no-more" to identify the issue.
Connect local a
connect iom X <<< X is the chassis number with the issue
show platform soft cmc showi2c \no-more
exit
Connect local b
connect iom X <<< X is the chassis number with the issue
show platform soft cmc showi2c |no-more
Resolution
In the output check for the following messages.
DC-UCS-FI-A# connect iom 1
fex-1# show platform soft cmc showi2c
segment 1 chassis
norxack 5514 <<<<< High numbers indicates a problem
unfinished 1
lostarbitration 1
fixup 2
segment 2 blade
=======snip==================
bus_lost_counter: 991
error_pca9541_per_device:
c.ms 77 <<<< This is a problem with the chassis management selector which is located on both IOMs for mgmt
c.gpio0 1
# I2C Device Statistics
c.seeprom={SUCCESS=262839}
f.fm0.fru={SUCCESS=1} <<<< All fans are talking fine with the IOM that is supposed to manage their behavior
f.fm1.fru={SUCCESS=1}
f.fm2.fru={SUCCESS=1}
f.fm3.fru={SUCCESS=1}
f.fm4.fru={SUCCESS=1}
f.fm5.fru={SUCCESS=1}
f.fm6.fru={SUCCESS=1}
f.fm7.fru={SUCCESS=1}
--------------------------------------------------------------------------------------------------
DC-UCS-FI-B# connect iom 1
fex-1# show platform soft cmc showi2c
segment 1 chassis
norxack 2795 <<<<< High numbers indicates a problem
wait_gt_deadline 207743
segment 2 blade
segment 3 fan
norxack 108550 <<<<<< Is evident the problem when talking to the other IOM
timeout 27815 <<<<<<
unfinished 14
========snip====================
# I2C Device Statistics
c.seeprom={SUCCESS=263768}
f.fm0.fc={SUCCESS=4410402,EIO=2,EBUSY=3477} <<<< this means an efforts to communicate but finds the channel busy
f.fm1.fc={SUCCESS=4407270,ENXIO=2,EBUSY=3478}
f.fm2.fc={SUCCESS=4406286,EBUSY=3478}
f.fm3.fc={SUCCESS=4406492,ENXIO=2,EBUSY=3476}
f.fm4.fc={SUCCESS=4406621,EBUSY=3476}
f.fm5.fc={SUCCESS=4407430,EIO=1,ENXIO=1,EBUSY=3476}
f.fm6.fc={SUCCESS=4407772,EBUSY=3476}
f.fm7.fc={SUCCESS=4407739,ENXIO=1,EBUSY=3480}
This mostly reflects a communication error (not a hardware failure,at least at first glance) with IOM attached to FI-B. Try to reseat the IOM connected to FI-B during a maintenance (take out for about 5 minutes to drain all the power), same as the fans (take the fans out for about 3 minutes, same reason as the IOM) and be sure the system is running the follwoing versions or above, according to the firmware you are running: 2.0.5b or later OR 2.1.1f or later OR any version of 2.2. If the system is already running one of those versions, then just do the reseat. This document is based on following discussion
https://supportforums.cisco.com/discussion/12133126/ucs-chassis-fans#4185971
Related Information
What are the supported expansion modules and license requirement in UCS
Frequently Asked Questions for UCS
