Catalyst 9800-40 chassis fans are LOUD

[hbell]

First "the fans in the power supply modules will continue to run even if the chassis power switch is in the Standby position" is some strange engineering. The power fan, usually loud in itself when compared with my current 5520 power fans, keeps running even when in standby mode.  But the power fans are not my REAL problem. The serious problem is with the chassis fans, which run at 100% even after the device has completed the boot sequence and I am able to enter exec mode. I have seen reports that this problem was resolved through a ROMMON fix with (I believe) pkg 16.12(3r).  Both in my brand new, out of the box, WLC pair were shipped with ROMMON ver 17.7(3r).  There is a ROMMON upgrade that was released on 2023-12-21, but that is behind the "additional entitlement required," meaning I would have to purchase a service contract for our little school district to get what we paid for in these brand new devices that should work properly out of the box. (This feels like extortion, by the way).

It is odd that BOTH brand new controllers are experiencing this chassis fan problem. Is there a fix for this, or am I going to have to RMA these devices?

[RxTx]

I seen similar problem with fan running at full speed in other WLC model, and it can be a hw fault because of temp sensor inside chassis it is faulty or report incorrect temperature.

Try to check in WLC sw interface or from console if it display CPU/chassis temp info, in my case the faulty unit did not display temp and the fan run at full speed and the other one exactly the same model and running the same sw version display temp and the fan run silent.

[hbell]

Thank you for the reply.  The temp sensors seem to be within suitable operating ranges. This room is set at 68 degrees farenheit, so that's not the problem either.  When I sh env the fan speed is 50%, and if this is 50, then I would hate to hear it at 100%.  At just 50% the sound can be heard more than halfway down a 40 foot long hallway!  This is unacceptable for this administrative suite of offices.  (My 5520s were so sweet.) Not having gotten a reply from Cisco TAC. I guess I am looking at RMAs for these two brand new boxes.

WLC#sh env

Number of Critical alarms: 0
Number of Major alarms: 0
Number of Minor alarms: 0

Slot Sensor Current State Reading Threshold(Minor,Major,Critical,Shutdown)
---------- -------------- --------------- ------------ ---------------------------------------
P0 Vin Normal 111 V AC na
P0 Iin Normal 1 A na
P0 Vout Normal 12 V DC na
P0 Iout Normal 8 A na
P0 Temp1 Normal 25 Celsius (na ,na ,na ,na )(Celsius)
P0 Temp2 Normal 22 Celsius (na ,na ,na ,na )(Celsius)
P0 Temp3 Normal 29 Celsius (na ,na ,na ,na )(Celsius)
P1 Vin Normal 111 V AC na
P1 Iin Normal 1 A na
P1 Vout Normal 12 V DC na
P1 Iout Normal 10 A na
P1 Temp1 Normal 25 Celsius (na ,na ,na ,na )(Celsius)
P1 Temp2 Normal 22 Celsius (na ,na ,na ,na )(Celsius)
P1 Temp3 Normal 27 Celsius (na ,na ,na ,na )(Celsius)
R0 VRRX1: VX1 Normal 751 mV na
R0 VRRX1: VX2 Normal 6909 mV na
R0 VRRX1: VX3 Normal 1216 mV na
R0 VRRX1: VX5 Normal 1216 mV na
R0 VRRX1: VP1 Normal 1713 mV na
R0 VRRX1: VP2 Normal 2496 mV na
R0 VRRX1: VP3 Normal 1311 mV na
R0 VRRX1: VP4 Normal 5057 mV na
R0 VRRX1: VH Normal 11924mV na
R0 VRRX2: VX1 Normal 854 mV na
R0 VRRX2: VX4 Normal 1011 mV na
R0 VRRX2: VX5 Normal 1015 mV na
R0 VRRX2: VP1 Normal 3325 mV na
R0 VRRX2: VP3 Normal 1820 mV na
R0 VRRX2: VP4 Normal 1051 mV na
R0 VRRX2: VH Normal 11940mV na
R0 VRRX3: VX1 Normal 986 mV na
R0 VRRX3: VX2 Normal 1005 mV na
R0 VRRX3: VX4 Normal 750 mV na
R0 VRRX3: VX5 Normal 749 mV na
R0 VRRX3: VP1 Normal 2492 mV na
R0 VRRX3: VP2 Normal 1196 mV na
R0 VRRX3: VP3 Normal 1511 mV na
R0 VRRX3: VP4 Normal 1512 mV na
R0 VRRX3: VH Normal 11914mV na
R0 Temp: RCRX IN Normal 19 Celsius (52 ,57 ,62 ,73 )(Celsius)
R0 Temp: RCRX OUT Normal 32 Celsius (62 ,67 ,72 ,80 )(Celsius)
R0 Temp: Yoda Normal 39 Celsius (71 ,76 ,81 ,90 )(Celsius)
R0 Temp: XEPhy Normal 40 Celsius (75 ,80 ,85 ,90 )(Celsius)
R0 Temp: CPU Die Normal 41 Celsius (98 ,103,108,110)(Celsius)
R0 Temp: FC FANS Fan Speed 50% 19 Celsius (26 ,44 ,0 )(Celsius)

[RxTx]

in my case faulty unit display:

show sysinfo
Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 8.5.182.7
Bootloader Version............................... 1.0.20
Field Recovery Image Version..................... 7.6.101.1
Firmware Version................................. PIC 16.0
OUI File Last Update Time........................ Sun Sep 07 10:44:07 IST 2014
Build Type....................................... DATA + WPS
System Name...................................... WLC-1
System Location..................................
System Contact...................................
System ObjectID.................................. 1.3.6.1.4.1.9.1.1279
IP Address....................................... 192.168.122.201
IPv6 Address..................................... ::
Last Reset....................................... Power on reset
System Up Time................................... 0 days 0 hrs 4 mins 29 secs
...
Operating Environment............................ Commercial (0 to 40 C)
Internal Temp Alarm Limits....................... 0 to 65 C
Internal Temperature............................. +0 C
External Temperature............................. +0 C
Fan Status....................................... 0 rpm
State of 802.11b Network......................... Enabled
State of 802.11a Network......................... Enabled
...

[hbell]

Yours
 

Fan Status....................................... 0 rpm

is what I would like to see on the units in my HA pair.  Everything looks "normal," but the chassis fans are running at 50%, and even at persistent 50% the noise level is unacceptable.  I currently run two 5520 WLCs, four 9300 switches, and three servers running in the racks, and the noise level of just one of the 9800 wlcs far exceeds all of that.  If the chassis fans persistently ran at 100% I think it would absolutely, unbearably deafening.

[Leo Laohoo]

---

@hbell wrote:
I would have to purchase a service contract for our little school district to get what we paid for in these brand new devices that should work properly out of the box.

---

Service Contract for what, download firmware?  Not necessarily. 

1.  Go to the Cisco Download portal and get the exact filename of the firmware you wish to download.  
2.  Note the exact web address of where the file(s) is/are located. 
3.  Read this:  Multiple Vulnerabilities in Cisco IOS XE Software Web UI Feature
4.  Scroll down to the Customers Without Service Contracts section.  Read it carefully.  
5.  E-Mail TAC & provide the following information: 

• Serial Number of the WLC
• Filename(s) to be published
• Location of the files(s) 
• Security Vulnerability Bulletin

Hope this helps.

[marce1000]

     - FYI : https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvq88840

 M.

[hbell]

Thanks for the pointer, but I had found that one earlier this week.  Would be nice if that solved my problem, but this pair shipped with ROMMON level 17.3, and of course there's no going backwards on this.  I strongly suspect now that it's a bad sensor in each, which doesn't seem unlikely since the serial numbers are close in sequence. I am waiting for a reply from TAC.

[marce1000]

  - You are probably correct , you could have a look in the commands mentioned below too , for the time being considered informational because they would correlate the problem to heavy usage of the devices :

          show platform resources
          show processes cpu platform sorted | ex 0%      0%      0%
          show platform hardware chassis active qfp datapath utilization | i load
          show processes memory platform sorted
          show processes memory platform accounting

 M,

[hbell]

These are fresh out the box, and I haven't even tried to migrate my 5520 configuration yet. So there are no WAPs or clients drawing resources. The fans never stop spin down and noise level from the highest they hit during self-diag.  TAC is stonewalling me on the RMA, even though the devices are brand new just out of box.  I finally got a tier-one engineer to hear by phone what I am experiencing on the ground, and he recognized that the decibel level is definitely not normal.

How bad is this situation? Bad enough to have me looking to other hardware solutions when we refresh our distribution switches later this year.

[marce1000]

                                   >...Bad enough to have me looking to other hardware solutions 
 - I understand your predicament but for these dollar-type of devices customers will usually and or by default include maintenance contracts (with hardware included). Hardware could always fail later on too.

  - I could see that the rommon version you mention is not listed in https://software.cisco.com/download/home/286316412/type/282046486/release/17.12(1r)  
     but going backwards is usually not advisable , due to possible higher hardware revisions needing it

  - What IOS-XE version is currently installed on them ?

 - You may also attach a console and follow the console messages from a cold startup (watch diagnostics) ; look for errors if any , 

 M.

[marce1000]

  - Added reply ;  checkout these commands too for further troubleshooting
  (not all of these commands are probably applicable and or related to your problem)

show  facility-alarm status
show platform hardware slot R0  alarms visual
show platform hardware slot 0 fan status
show platform hardware slot P1 fan status

show platform software system all
show platform resources

show environment chassis active r0  
show environment
show environment summary
show environment chassis active r0 
show platform hardware slot R0  dram statistics 
show logging onboard dram
show logging onboard slot 0 dram
show logging onboard slot 0 uptime
show logging onboard slot 0 voltage
show logging profile hardware-diagnostics
show logging onboard slot 0 temperature
show logging onboard uptime
show platform hardware slot R0  voltage  margin device 
show platform hardware slot R0  led status 
show platform hardware slot R0  rommon status
show platform hardware slot R0  soft-error statistics


show platform hardware slot R0 ezman config
show platform hardware slot R0 ezman core-temp
show platform hardware slot R0 ezman  info
show platform hardware slot F0 io-port
further mode use show platform hardware slot R0 ? , for additional diagnostic commands info’s
show platform hardware slot R0  voltage margin device (an example of the above)
show platform hardware slot R0  sensor producer all

 M.

[hbell]

Thanks for the suggestions. Some of these I have tried. Some are deprecated on the 9800. Some don't apply and generate an error message to that effect.

show platform hardware slot P1 fan status returns "Slot P1 is empty"

show platform hardware slot P2 fan status returns "Fan group 1 speed: 100%"

Bear in mind that there is no WAP/client/traffic load on this device.  Also, what I believe to be a jr. engineer at TAC stated that the fans should always run at least 50%, which doesn't make sense to me.  Certainly not at 100% when there is nothing drawing resources from the box.

[marce1000]

- Could you also execute the (d)ram related tests from my list , because
sometimes this can be related to faulty memory.

Also pay attention too :
show facility-alarm status

M.

[hbell]

The winter storm shut us down and I am just getting back to this.  Are the fan readings from these commands inconsistent?

WLC#show platform hardware slot P0 fan status
Fan group 1 speed: 0%

WLC#show platform hardware slot P1 fan status
Fan group 1 speed: 0%

WLC#sh env

Number of Critical alarms: 0
Number of Major alarms: 0
Number of Minor alarms: 0

Slot Sensor Current State Reading Threshold(Minor,Major,Critical,Shutdown)
---------- -------------- --------------- ------------ ---------------------------------------
P0 Vin Normal 106 V AC na
P0 Iin Normal 1 A na
P0 Vout Normal 12 V DC na
P0 Iout Normal 9 A na
P0 Temp1 Normal 27 Celsius (na ,na ,na ,na )(Celsius)
P0 Temp2 Normal 23 Celsius (na ,na ,na ,na )(Celsius)
P0 Temp3 Normal 31 Celsius (na ,na ,na ,na )(Celsius)
P1 Vin Normal 106 V AC na
P1 Iin Normal 1 A na
P1 Vout Normal 12 V DC na
P1 Iout Normal 10 A na
P1 Temp1 Normal 27 Celsius (na ,na ,na ,na )(Celsius)
P1 Temp2 Normal 23 Celsius (na ,na ,na ,na )(Celsius)
P1 Temp3 Normal 28 Celsius (na ,na ,na ,na )(Celsius)
R0 VRRX1: VX1 Normal 751 mV na
R0 VRRX1: VX2 Normal 6909 mV na
R0 VRRX1: VX3 Normal 1216 mV na
R0 VRRX1: VX5 Normal 1216 mV na
R0 VRRX1: VP1 Normal 1713 mV na
R0 VRRX1: VP2 Normal 2497 mV na
R0 VRRX1: VP3 Normal 1311 mV na
R0 VRRX1: VP4 Normal 5059 mV na
R0 VRRX1: VH Normal 11919mV na
R0 VRRX2: VX1 Normal 853 mV na
R0 VRRX2: VX4 Normal 1012 mV na
R0 VRRX2: VX5 Normal 1015 mV na
R0 VRRX2: VP1 Normal 3325 mV na
R0 VRRX2: VP3 Normal 1820 mV na
R0 VRRX2: VP4 Normal 1052 mV na
R0 VRRX2: VH Normal 11935mV na
R0 VRRX3: VX1 Normal 986 mV na
R0 VRRX3: VX2 Normal 1004 mV na
R0 VRRX3: VX4 Normal 750 mV na
R0 VRRX3: VX5 Normal 749 mV na
R0 VRRX3: VP1 Normal 2493 mV na
R0 VRRX3: VP2 Normal 1196 mV na
R0 VRRX3: VP3 Normal 1512 mV na
R0 VRRX3: VP4 Normal 1512 mV na
R0 VRRX3: VH Normal 11914mV na
R0 Temp: RCRX IN Normal 22 Celsius (52 ,57 ,62 ,73 )(Celsius)
R0 Temp: RCRX OUT Normal 36 Celsius (62 ,67 ,72 ,80 )(Celsius)
R0 Temp: Yoda Normal 41 Celsius (71 ,76 ,81 ,90 )(Celsius)
R0 Temp: XEPhy Normal 43 Celsius (75 ,80 ,85 ,90 )(Celsius)
R0 Temp: CPU Die Normal 43 Celsius (98 ,103,108,110)(Celsius)
R0 Temp: FC FANS Fan Speed 50% 22 Celsius (26 ,44 ,0 )(Celsius)

WLC#