Solved: C220 M4 - 12g SAS Modular RAID consistency check

jonansoz · ‎02-07-2017

Hey all,

So I noticed that the disks were going crazy on this M4 unit with 4x1.2 TB 10k SAS disks.

It shouldn't be as they have nothing on them right now.

Looking in CIMC revealed a consistency check going on for 16 hours and is at 70% so far.

Questions:

I see a few entries like "Consistency Check inconsistency logging disabled on VD 01/0 (too many inconsistencies)" and "Consistency Check found inconsistent parity on VD 01/0 at strip d6".

Is this "bad"?

Might seem like a stupid question, but there are NO errors in CIMC or on the front lights of the unit.

I'd expect some kind of warning if inconsistency's are found?

Should it also be taking this long? 17 hours at 70% done seems like a VERY long time for a 4 disk RAID 6 array with 10k SAS disks.

Kirk J · ‎02-07-2017

Greetings.

Low level checks such as consistency checks or patrol reads are logged in CIMC logs, but will not show up as alerts unless there is a change in the VD status, or a drive has a failure/predictive failure.

I don't believe I've ever seen a published guide on expected consistency checks or patrol read times.

What seems to be 'normal' times range are around 2-4 hrs. I have observed some systems that had consistency checks running 24+ hrs that had issues such as yours (inconsistent parity, punctured blocks, etc)

Do any of your individual drives show 'media error' counters?

Issues with power, or the raid controller experiencing a reset (https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvb20238/?reffering_site=dumpcr) can contribute to consistency issues.

You may need to consider migrating data off the VD, and potentially re-creating it. Prior to resorting to that, you may want to open a TAC case and have them review the raid controller fwterm logs (LSI-GET script) and see if this related to any physical drive related issues (i.e. firmware timeouts), raid controller problems, sas drive backplane issues, etc.

Thanks,

Kirk

View solution in original post

Kirk J · ‎02-07-2017

Greetings.

Low level checks such as consistency checks or patrol reads are logged in CIMC logs, but will not show up as alerts unless there is a change in the VD status, or a drive has a failure/predictive failure.

I don't believe I've ever seen a published guide on expected consistency checks or patrol read times.

What seems to be 'normal' times range are around 2-4 hrs. I have observed some systems that had consistency checks running 24+ hrs that had issues such as yours (inconsistent parity, punctured blocks, etc)

Do any of your individual drives show 'media error' counters?

Issues with power, or the raid controller experiencing a reset (https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvb20238/?reffering_site=dumpcr) can contribute to consistency issues.

You may need to consider migrating data off the VD, and potentially re-creating it. Prior to resorting to that, you may want to open a TAC case and have them review the raid controller fwterm logs (LSI-GET script) and see if this related to any physical drive related issues (i.e. firmware timeouts), raid controller problems, sas drive backplane issues, etc.

Thanks,

Kirk

jonansoz · ‎02-07-2017

Hmm I think I may open a TAC case as the result from the consistency check is:

Consistency Check done with corrections on VD 01/0, (corrections=65535)

That number 65535 seems a bit too "perfect" if you know what I mean.

https://msdn.microsoft.com/en-us/library/ee621251.aspx