C200 M1 PSU Fan speed varies

John Early · ‎12-18-2012

I have a C200 M1 running 1.4(3p), I am noticing some performance issues with this server, it just seems slow. I went to it physically and you can hear the PSU fans spin up and slow down every few seconds. Within an SSH to the CIMC I am seeing both fans vary from 2K to 7K RPMs every 3-5 seconds. Here are the PSU details:

ucs-c2xx-m1 /chassis # show psu detail

Name PSU1:

In. Power (Watts): 87

Out. Power (Watts): 650

Firmware : R0E

Status : Present

Product ID : SUNNY651AM

Name PSU2:

In. Power (Watts): 98

Out. Power (Watts): 650

Firmware : R0E

Status : Present

Product ID : SUNNY651AM

Here you can see the speed differences:

ucs-c2xx-m1 /sensor # show fan

Name Sensor Status Reading Units Min. Warning Max. Warning Min. Failure Max. Failure

-------------------- -------------------- ---------- ---------- --------------- --------------- --------------- ---------------

PSU1_FAN_1 Normal 3072 RPM N/A N/A N/A N/A

PSU2_FAN_1 Normal 2048 RPM N/A N/A N/A N/A

W793_FAN1_TACH1 Normal 5700 RPM N/A N/A 800 N/A

W793_FAN1_TACH2 Normal 5500 RPM N/A N/A 800 N/A

W793_FAN2_TACH1 Normal 5800 RPM N/A N/A 800 N/A

W793_FAN2_TACH2 Normal 5500 RPM N/A N/A 800 N/A

W793_FAN3_TACH1 Normal 6000 RPM N/A N/A 800 N/A

W793_FAN3_TACH2 Normal 5600 RPM N/A N/A 800 N/A

W793_FAN4_TACH1 Normal 5800 RPM N/A N/A 800 N/A

W793_FAN4_TACH2 Normal 5500 RPM N/A N/A 800 N/A

W793_FAN5_TACH1 Normal 5900 RPM N/A N/A 800 N/A

W793_FAN5_TACH2 Normal 5300 RPM N/A N/A 800 N/A

ucs-c2xx-m1 /sensor # show fan

Name Sensor Status Reading Units Min. Warning Max. Warning Min. Failure Max. Failure

-------------------- -------------------- ---------- ---------- --------------- --------------- --------------- ---------------

PSU1_FAN_1 Normal 6784 RPM N/A N/A N/A N/A

PSU2_FAN_1 Normal 7360 RPM N/A N/A N/A N/A

W793_FAN1_TACH1 Normal 5700 RPM N/A N/A 800 N/A

W793_FAN1_TACH2 Normal 5500 RPM N/A N/A 800 N/A

W793_FAN2_TACH1 Normal 5800 RPM N/A N/A 800 N/A

W793_FAN2_TACH2 Normal 5500 RPM N/A N/A 800 N/A

W793_FAN3_TACH1 Normal 5900 RPM N/A N/A 800 N/A

W793_FAN3_TACH2 Normal 5600 RPM N/A N/A 800 N/A

W793_FAN4_TACH1 Normal 5800 RPM N/A N/A 800 N/A

W793_FAN4_TACH2 Normal 5600 RPM N/A N/A 800 N/A

W793_FAN5_TACH1 Normal 5900 RPM N/A N/A 800 N/A

W793_FAN5_TACH2 Normal 5300 RPM N/A N/A 800 N/A

This server was running firmware 1.0(2) when I first found this issue, I have since upgraded it to the lastest code. I have done a full shutdown and removed the PSUs and let it sit for a few minutes.

When the fan speeds are low, the CPU performance is very low, VMs running on it are slow. Any thoughts??

Thanks.

Robert Burns · ‎12-18-2012

What is the version of HyperVisor?

What's the memory operation mode (Performance or Energy Saving)?

You've ran the latest Host Update Utility (HUU) for C200 M1 against this host?

Also include the following outputs:

scope fault

show discrete-alarm

top

scope sensor

show temp

top

scope sel

show entries | grep Critical

If easier, log above to a file and attach to the post.

Regards,

Robert

John Early · ‎12-18-2012

Running ESXi 4.1 348481

Memory is running in Performance Mode

I have run the latest HUU with option 8 to upgrade all

Here are the outputs requested, this server has been running for a while with a single power supply connected, hence the multiple errors reflecting that.

ucs-c2xx-m1# scope fault
ucs-c2xx-m1 /fault # show discrete-alarm
ucs-c2xx-m1 /fault # top
ucs-c2xx-m1# scope sensor
ucs-c2xx-m1 /sensor # show temperature
Name                      Sensor Status Reading    Units      Min. Warning Max. Warning Min. Failure Max. Failure
------------------------- -------------- ---------- ---------- ------------ ------------ ------------ ------------
IOH_TEMP_SENS             Normal         45.0       C          N/A          80.0         N/A          85.0
P2_TEMP_SENS              Normal         55.0       C          N/A          87.0         N/A          92.0
P1_TEMP_SENS              Normal         49.5       C          N/A          87.0         N/A          92.0
DDR3_P2_D1_TMP            Normal         29.0       C          N/A          68.0         N/A          70.0
DDR3_P2_E1_TMP            Normal         30.0       C          N/A          68.0         N/A          70.0
DDR3_P2_F1_TMP            Normal         29.0       C          N/A          68.0         N/A          70.0
DDR3_P1_A1_TMP            Normal         35.0       C          N/A          68.0         N/A          70.0
DDR3_P1_B1_TMP            Normal         34.0       C          N/A          68.0         N/A          70.0
DDR3_P1_C1_TMP            Normal         32.0       C          N/A          68.0         N/A          70.0
PSU1_TEMP_1               Normal         40.0       C          N/A          60.0         N/A          65.0
PSU2_TEMP_1               Normal         40.0       C          N/A          60.0         N/A          65.0
FP_AMBIENT_TEMP           Normal         24.0       C          N/A          40.0         N/A          45.0
ucs-c2xx-m1 /sensor # top
ucs-c2xx-m1# scope sel
ucs-c2xx-m1 /sel # show entries | grep Critical
2012-12-17 22:09:14 Critical      "FRU_RAM SEL_FULLNESS: System Event sensor for FRU_RAM, failure event, Upper Critical going high                          (85 > 80 ) was asserted"
2012-12-17 22:05:25 Critical      "FRU_RAM SEL_FULLNESS: System Event sensor for FRU_RAM, failure event, Upper Critical going high                          (84 > 80 ) was asserted"
[System Boot]       Critical      "FRU_RAM SEL_FULLNESS: System Event sensor for FRU_RAM, failure event, Upper Critical going high                          (83 > 80 ) was asserted"
[System Boot]       Critical      "PSU_REDUNDANCY: PS Redundancy sensor, Redundancy Lost was asserted"
[System Boot]       Critical      "FRU_RAM SEL_FULLNESS: System Event sensor for FRU_RAM, failure event, Upper Critical going high                          (80 > 80 ) was asserted"
2012-12-17 20:16:31 Critical      "PSU_REDUNDANCY: PS Redundancy sensor, Redundancy Lost was asserted"
2012-12-17 19:34:47 Critical      "PSU_REDUNDANCY: PS Redundancy sensor, Redundancy Lost was asserted"
2012-12-17 19:34:47 Critical      "PSU1 PSU1_STATUS: Power Supply sensor for PSU1, Power Supply input lost (AC/DC) was asserted"
2012-12-17 19:34:05 Critical      "FRU_RAM SEL_FULLNESS: System Event sensor for FRU_RAM, failure event, Upper Critical going high                          (80 > 80 ) was asserted"
2012-12-17 19:33:28 Critical      "PSU_REDUNDANCY: PS Redundancy sensor, Redundancy Lost was asserted"
2012-12-17 19:33:28 Critical      "PSU2 PSU2_STATUS: Power Supply sensor for PSU2, Power Supply input lost (AC/DC) was asserted"
2012-12-05 20:22:41 Critical      "PSU2 PSU2_STATUS: Power Supply sensor for PSU2, Power Supply input lost (AC/DC) was asserted"
[System Boot]       Critical      "PSU1 PSU1_STATUS: Power Supply sensor for PSU1, Power Supply input lost (AC/DC) was asserted"
[System Boot]       Critical      "PSU_REDUNDANCY: PS Redundancy sensor, Redundancy Lost was asserted"
2012-11-05 21:43:11 Critical      "PSU_REDUNDANCY: PS Redundancy sensor, Redundancy Lost was asserted"
2012-11-05 21:43:11 Critical      "PSU2 PSU2_STATUS: Power Supply sensor for PSU2, Power Supply input lost (AC/DC) was asserted"
2012-06-11 14:51:35 Critical      "PSU_REDUNDANCY: PS Redundancy sensor, Redundancy Lost was asserted"
2012-05-03 10:59:49 Critical      "PSU_REDUNDANCY: PS Redundancy sensor, Redundancy Lost was asserted"
2012-05-02 16:16:45 Critical      "PSU_REDUNDANCY: PS Redundancy sensor, Redundancy Lost was asserted"
2012-05-02 16:16:44 Critical      "PSU1 PSU1_STATUS: Power Supply sensor for PSU1, Power Supply input lost (AC/DC) was asserted"
2012-02-02 14:03:57 Critical      "PSU1 PSU1_STATUS: Power Supply sensor for PSU1, Power Supply input lost (AC/DC) was asserted"
2012-02-02 14:03:52 Critical      "PSU_REDUNDANCY: PS Redundancy sensor, Redundancy Lost was asserted"
[System Boot]       Critical      "PSU1 PSU1_STATUS: Power Supply sensor for PSU1, Power Supply input lost (AC/DC) was asserted"
[System Boot]       Critical      "PSU_REDUNDANCY: PS Redundancy sensor, Redundancy Lost was asserted"
2011-12-21 13:55:17 Critical      "PSU_REDUNDANCY: PS Redundancy sensor, Redundancy Lost was asserted"
2011-12-21 13:55:16 Critical      "PSU1 PSU1_STATUS: Power Supply sensor for PSU1, Power Supply input lost (AC/DC) was asserted"
[System Boot]       Critical      "PSU2 PSU2_STATUS: Power Supply sensor for PSU2, Power Supply input lost (AC/DC) was asserted"
[System Boot]       Critical      "PSU_REDUNDANCY: PS Redundancy sensor, Redundancy Lost was asserted"
2011-09-30 10:02:39 Critical      "PSU_REDUNDANCY: PS Redundancy sensor, Redundancy Lost was asserted"
2011-09-30 10:02:39 Critical      "PSU2 PSU2_STATUS: Power Supply sensor for PSU2, Power Supply input lost (AC/DC) was asserted"
2011-09-25 13:31:33 Critical      "PSU_REDUNDANCY: PS Redundancy sensor, Redundancy Lost was asserted"
2011-09-25 13:31:33 Critical      "PSU2 PSU2_STATUS: Power Supply sensor for PSU2, Power Supply input lost (AC/DC) was asserted"
2011-09-24 13:11:31 Critical      "PSU_REDUNDANCY: PS Redundancy sensor, Redundancy Lost was asserted"
2011-09-24 13:11:31 Critical      "PSU2 PSU2_STATUS: Power Supply sensor for PSU2, Power Supply input lost (AC/DC) was asserted"
2011-09-20 14:28:02 Critical      "PSU_REDUNDANCY: PS Redundancy sensor, Redundancy Lost was asserted"
[System Boot]       Critical      "PSU_REDUNDANCY: PS Redundancy sensor, Redundancy Lost was asserted"

Keny Perez · ‎12-23-2012

John,

From "ucs-c2xx-m1 /chassis # show psu detail", I see that you are using PSUs with the "Product ID : SUNNY651AM" and "Firmware : R0E"... I have had customers using those PSUs, which are an early version of the PSUs that we currently use for this C200s. I would suggest trying with a PSU that has firmware 00.01.05 or later (This is not something that you can upgrade, and this can also be done with a replacement)... Do you happen to have another C200 in which you can confirm the firmware is 00.01.05 or later and install it here?

Note: I am not saying this is part of the Field Notice 63425, I am just saying (IMHO), I have seen how PSUs with firmware older that 00.01.05 (like the one you have) have issues like this one.

If you do not have other C200 you can try with, I would suggest to open a TAC case where we can reproduce the issue and make a decision about what to do next.

-Kenny