Enclosure PD 07(c Port 0 - 3/p1) not responding problem

Lukasz Referda · ‎05-22-2017

Hello

Since about one week I have following problem at 2 servers - they crash and need to reboot them by CIMC, it seems to happend during some disk operations. One of them crashes after 0,5..1h of running (VM with CCTV records), the second one after few minutes/hours after starting copying files to another disk (ok when there is no disk operations). At both of them I have got similar error in CIMC/Storage/Logs: "Enclosure PD 07(c Port 0 - 3/p1) not responding". There is no any other errors and signals that something is wrong.

The problem starts to appear without any reason, everything worked fine before. About 2 months ago drivers were updated, also CIMC firmware and BIOS. I have tried to downgrade and upgrade storage drivers - still the same. Problem of second server could be not seen before because we downloaded files from it for the first time.

Could you tell me what is wrong or what I need to check more please?

Servers details:

UCS C240 M3L

RAID controller: Cisco UCSC RAID SAS 2008M-8i

4 disks in RAID5: TOSHIBA MG03SCA100

Best regards

Lukasz

Kirk J · ‎05-23-2017

Greetings.

I am kind of surprised to see a 2008M-8i used for a video related application, especially combined with a hyper-visor.

The 2008M has no cache or BBU. This means you have no read or writeback caching, and will have poor performance/high latency, and bundled with 7200 RPM disks.

Could be an issue with the controller or back-plane, but would have to review the LSI logs for more info.

You'll probably need to open a TAC case for more in-depth log gathering/analysis.

Thanks,

Kirk..

Lukasz Referda · ‎05-24-2017

Hello Kirk

Thanks for your feedback. Yes, the first server btw is used for CCTV, but there is only 8 cameras with 1fps, so it is not big deal for this machine. Especially everything worked fine for 2 years. If I turn off virtual machine responsible for CCTV then problem with crashes doesn't occur.

The second server is used as storage for temporary data. My friend wanted to download lot of files from it and everytime after some time server crashed. It seems that disc operations cause problems, if server is free then nothing wrong happends.

I put logs from MegaRAID Storage manager below, also I have pointed points which are shown during crash. Do you think you can help me based on those information or better to open TAC case?

ID = 10000
TIME = 24-05-2017 10:13:56
LOCALIZED MESSAGE = Successful log on to the server User: Administrator, Client: 10.1.0.13, Access Mode: Full, Client Time: 2017-05-24,10:13:57

ID = 38
SEQUENCE NUMBER = 53113
TIME = 24-05-2017 10:01:22
LOCALIZED MESSAGE = Controller ID: 0 Patrol Read resumed

ID = 44
SEQUENCE NUMBER = 53112
TIME = 24-05-2017 09:59:28
LOCALIZED MESSAGE = Controller ID: 0 Time established since power on: Time 2017-05-24, 09:59:28 30 Seconds

ID = 44
SEQUENCE NUMBER = 53111
TIME = 24-05-2017 09:59:23
LOCALIZED MESSAGE = Controller ID: 0 Time established since power on: Time 2017-05-24, 09:59:23 24 Seconds

ID = 247
SEQUENCE NUMBER = 53110
TIME = Not Available
LOCALIZED MESSAGE = Controller ID: 0 Device inserted Device Type: Disk Device Id: 8

ID = 91
SEQUENCE NUMBER = 53109
TIME = Not Available
LOCALIZED MESSAGE = Controller ID: 0 PD inserted: Port 0 - 3:1:4

ID = 247
SEQUENCE NUMBER = 53108
TIME = Not Available
LOCALIZED MESSAGE = Controller ID: 0 Device inserted Device Type: Disk Device Id: 6

ID = 91
SEQUENCE NUMBER = 53107
TIME = Not Available
LOCALIZED MESSAGE = Controller ID: 0 PD inserted: Port 0 - 3:1:2

ID = 247
SEQUENCE NUMBER = 53106
TIME = Not Available
LOCALIZED MESSAGE = Controller ID: 0 Device inserted Device Type: Disk Device Id: 5

ID = 91
SEQUENCE NUMBER = 53105
TIME = Not Available
LOCALIZED MESSAGE = Controller ID: 0 PD inserted: Port 0 - 3:1:1

ID = 247
SEQUENCE NUMBER = 53104
TIME = Not Available
LOCALIZED MESSAGE = Controller ID: 0 Device inserted Device Type: Disk Device Id: 3

ID = 91
SEQUENCE NUMBER = 53103
TIME = Not Available
LOCALIZED MESSAGE = Controller ID: 0 PD inserted: Port 0 - 3:1:3

ID = 247
SEQUENCE NUMBER = 53102
TIME = Not Available
LOCALIZED MESSAGE = Controller ID: 0 Device inserted Device Type: Enclosure Device Id: 7

ID = 91
SEQUENCE NUMBER = 53101
TIME = Not Available
LOCALIZED MESSAGE = Controller ID: 0 PD inserted: 7

ID = 167
SEQUENCE NUMBER = 53100
TIME = Not Available
LOCALIZED MESSAGE = Controller ID: 0 Communication restored on enclosure: 1

ID = 164
SEQUENCE NUMBER = 53099
TIME = Not Available
LOCALIZED MESSAGE = Controller ID: 0 SES enclosure discovered: 1

ID = 266
SEQUENCE NUMBER = 53098
TIME = Not Available
LOCALIZED MESSAGE = Controller ID: 0 Board Revision:

ID = 261
SEQUENCE NUMBER = 53097
TIME = Not Available
LOCALIZED MESSAGE = Controller ID: 0 Package version 20.11.1-0159

ID = 1
SEQUENCE NUMBER = 53096
TIME = Not Available
LOCALIZED MESSAGE = Controller ID: 0 Image version: 2.130.384-2748

ID = 0
SEQUENCE NUMBER = 53095
TIME = Not Available
LOCALIZED MESSAGE = Controller ID: 0 Firmware initialization started: ( PCI ID 0x73/ 0x1000/ 0xb1 / 0x1137)

ID = 188
SEQUENCE NUMBER = 53094
TIME = 23-05-2017 23:58:51
LOCALIZED MESSAGE = Controller ID: 0 No response from enclosure: 1

ID = 38
SEQUENCE NUMBER = 53091
TIME = 23-05-2017 23:16:36
LOCALIZED MESSAGE = Controller ID: 0 Patrol Read resumed

ID = 44
SEQUENCE NUMBER = 53090
TIME = 23-05-2017 23:14:43
LOCALIZED MESSAGE = Controller ID: 0 Time established since power on: Time 2017-05-23, 23:14:43 30 Seconds

ID = 44
SEQUENCE NUMBER = 53089
TIME = 23-05-2017 23:14:37
LOCALIZED MESSAGE = Controller ID: 0 Time established since power on: Time 2017-05-23, 23:14:37 24 Seconds

ID = 247
SEQUENCE NUMBER = 53088
TIME = Not Available
LOCALIZED MESSAGE = Controller ID: 0 Device inserted Device Type: Disk Device Id: 8

ID = 91
SEQUENCE NUMBER = 53087
TIME = Not Available
LOCALIZED MESSAGE = Controller ID: 0 PD inserted: Port 0 - 3:1:4

ID = 247
SEQUENCE NUMBER = 53086
TIME = Not Available
LOCALIZED MESSAGE = Controller ID: 0 Device inserted Device Type: Disk Device Id: 6

ID = 91
SEQUENCE NUMBER = 53085
TIME = Not Available
LOCALIZED MESSAGE = Controller ID: 0 PD inserted: Port 0 - 3:1:2

ID = 247
SEQUENCE NUMBER = 53084
TIME = Not Available
LOCALIZED MESSAGE = Controller ID: 0 Device inserted Device Type: Disk Device Id: 5

ID = 91
SEQUENCE NUMBER = 53083
TIME = Not Available
LOCALIZED MESSAGE = Controller ID: 0 PD inserted: Port 0 - 3:1:1

ID = 247
SEQUENCE NUMBER = 53082
TIME = Not Available
LOCALIZED MESSAGE = Controller ID: 0 Device inserted Device Type: Disk Device Id: 3

ID = 91
SEQUENCE NUMBER = 53081
TIME = Not Available
LOCALIZED MESSAGE = Controller ID: 0 PD inserted: Port 0 - 3:1:3

ID = 247
SEQUENCE NUMBER = 53080
TIME = Not Available
LOCALIZED MESSAGE = Controller ID: 0 Device inserted Device Type: Enclosure Device Id: 7

ID = 91
SEQUENCE NUMBER = 53079
TIME = Not Available
LOCALIZED MESSAGE = Controller ID: 0 PD inserted: 7

ID = 167
SEQUENCE NUMBER = 53078
TIME = Not Available
LOCALIZED MESSAGE = Controller ID: 0 Communication restored on enclosure: 1

ID = 164
SEQUENCE NUMBER = 53077
TIME = Not Available
LOCALIZED MESSAGE = Controller ID: 0 SES enclosure discovered: 1

ID = 266
SEQUENCE NUMBER = 53076
TIME = Not Available
LOCALIZED MESSAGE = Controller ID: 0 Board Revision:

ID = 261
SEQUENCE NUMBER = 53075
TIME = Not Available
LOCALIZED MESSAGE = Controller ID: 0 Package version 20.11.1-0159

ID = 1
SEQUENCE NUMBER = 53074
TIME = Not Available
LOCALIZED MESSAGE = Controller ID: 0 Image version: 2.130.384-2748

ID = 0
SEQUENCE NUMBER = 53073
TIME = Not Available
LOCALIZED MESSAGE = Controller ID: 0 Firmware initialization started: ( PCI ID 0x73/ 0x1000/ 0xb1 / 0x1137)

ID = 188
SEQUENCE NUMBER = 53072
TIME = 23-05-2017 15:52:53
LOCALIZED MESSAGE = Controller ID: 0 No response from enclosure: 1

ID = 10000
TIME = 23-05-2017 14:49:28
LOCALIZED MESSAGE = Successful log on to the server User: Administrator, Client: 10.1.0.13, Access Mode: Full, Client Time: 2017-05-23,14:49:28

ID = 38
SEQUENCE NUMBER = 53067
TIME = 23-05-2017 13:12:12
LOCALIZED MESSAGE = Controller ID: 0 Patrol Read resumed

ID = 44
SEQUENCE NUMBER = 53066
TIME = 23-05-2017 13:10:19
LOCALIZED MESSAGE = Controller ID: 0 Time established since power on: Time 2017-05-23, 13:10:19 30 Seconds

ID = 44
SEQUENCE NUMBER = 53065
TIME = 23-05-2017 13:10:13
LOCALIZED MESSAGE = Controller ID: 0 Time established since power on: Time 2017-05-23, 13:10:13 24 Seconds

ID = 247
SEQUENCE NUMBER = 53064
TIME = Not Available
LOCALIZED MESSAGE = Controller ID: 0 Device inserted Device Type: Disk Device Id: 8

ID = 91
SEQUENCE NUMBER = 53063
TIME = Not Available
LOCALIZED MESSAGE = Controller ID: 0 PD inserted: Port 0 - 3:1:4

ID = 247
SEQUENCE NUMBER = 53062
TIME = Not Available
LOCALIZED MESSAGE = Controller ID: 0 Device inserted Device Type: Disk Device Id: 6

ID = 91
SEQUENCE NUMBER = 53061
TIME = Not Available
LOCALIZED MESSAGE = Controller ID: 0 PD inserted: Port 0 - 3:1:2

ID = 247
SEQUENCE NUMBER = 53060
TIME = Not Available
LOCALIZED MESSAGE = Controller ID: 0 Device inserted Device Type: Disk Device Id: 5

ID = 91
SEQUENCE NUMBER = 53059
TIME = Not Available
LOCALIZED MESSAGE = Controller ID: 0 PD inserted: Port 0 - 3:1:1

ID = 247
SEQUENCE NUMBER = 53058
TIME = Not Available
LOCALIZED MESSAGE = Controller ID: 0 Device inserted Device Type: Disk Device Id: 3

ID = 91
SEQUENCE NUMBER = 53057
TIME = Not Available
LOCALIZED MESSAGE = Controller ID: 0 PD inserted: Port 0 - 3:1:3

ID = 247
SEQUENCE NUMBER = 53056
TIME = Not Available
LOCALIZED MESSAGE = Controller ID: 0 Device inserted Device Type: Enclosure Device Id: 7

ID = 91
SEQUENCE NUMBER = 53055
TIME = Not Available
LOCALIZED MESSAGE = Controller ID: 0 PD inserted: 7

ID = 167
SEQUENCE NUMBER = 53054
TIME = Not Available
LOCALIZED MESSAGE = Controller ID: 0 Communication restored on enclosure: 1

ID = 164
SEQUENCE NUMBER = 53053
TIME = Not Available
LOCALIZED MESSAGE = Controller ID: 0 SES enclosure discovered: 1

ID = 266
SEQUENCE NUMBER = 53052
TIME = Not Available
LOCALIZED MESSAGE = Controller ID: 0 Board Revision:

ID = 261
SEQUENCE NUMBER = 53051
TIME = Not Available
LOCALIZED MESSAGE = Controller ID: 0 Package version 20.11.1-0159

ID = 1
SEQUENCE NUMBER = 53050
TIME = Not Available
LOCALIZED MESSAGE = Controller ID: 0 Image version: 2.130.384-2748

ID = 0
SEQUENCE NUMBER = 53049
TIME = Not Available
LOCALIZED MESSAGE = Controller ID: 0 Firmware initialization started: ( PCI ID 0x73/ 0x1000/ 0xb1 / 0x1137)

ID = 42
SEQUENCE NUMBER = 53048
TIME = 23-05-2017 13:09:39
LOCALIZED MESSAGE = Controller ID: 0 Shutdown command received from host