Sound Data Management Training » History » Version 85

Steve Welburn, 2012-09-25 12:54 PM

1 5 Steve Welburn
h1. WP1.2 Online Training Material
2 1 Steve Welburn
3 9 Steve Welburn
{{>toc}}
4 9 Steve Welburn
5 68 Steve Welburn
We consider three stages of a research project, and the appropriate research data management considerations for each of those stages. The stages are:
6 75 Steve Welburn
* [[Before The Research|before the research]];
7 75 Steve Welburn
* [[During The Research|during the research]];
8 75 Steve Welburn
* [[At The End Of The Research|at the end of the research]].
9 1 Steve Welburn
10 75 Steve Welburn
In addition, we consider the [[Research Management|responsibilities of a Principal Investigator]] regarding data management.
11 75 Steve Welburn
12 77 Steve Welburn
{{include(Data Management By Research Stage)}}
13 44 Steve Welburn
14 82 Steve Welburn
h2. Why manage research data ?
15 82 Steve Welburn
16 82 Steve Welburn
Funder requirements: http://researchonline.lshtm.ac.uk/208596/
17 82 Steve Welburn
18 83 Steve Welburn
Ponemon reports for Intel on the "Lost Laptop problem" ~10% of Education and Research laptops are lost during their lifetime.
19 83 Steve Welburn
20 83 Steve Welburn
PC World study on laptop failure rates: 20-30% of laptops with a significant failure
21 83 Steve Welburn
22 83 Steve Welburn
h3. Failure Trends In A Large Disk Drive Population
23 83 Steve Welburn
24 83 Steve Welburn
FAST '07 paper on "Failure Trends In A Large Disk Drive Population":https://www.usenix.org/conference/fast-07/failure-trends-large-disk-drive-population
25 83 Steve Welburn
26 83 Steve Welburn
Google report on over 100,000 consumer-grade disk drives from 80-400 GB produced in or after 2001 and used within Google. Data collected December 2005 - August 2006. Disk drives had a burn-in process and only those that were commissioned for use were included in the study - certain basic defects may well be excluded from this report.
27 83 Steve Welburn
28 84 Steve Welburn
bq. the most accurate definition we can present of a failure event for our study is: a drive is considered to have failed if it was replaced as part of a repairs procedure. Note that this definition implicitly excludes drives that were replaced due to an upgrade.
29 84 Steve Welburn
30 84 Steve Welburn
~3% in first 3 months, ~2% up to 1 year, ~8% @ 2 years, ~9% @ 3 years, ~6% @ 4 years, ~7% @ 5 years
31 84 Steve Welburn
32 84 Steve Welburn
NB: Variation with model and manufacturer!
33 84 Steve Welburn
34 84 Steve Welburn
In the first 6 months, the risk of failure is highest for low & high utilisation!
35 84 Steve Welburn
* ~10% for high utilisation in the first 3 months
36 84 Steve Welburn
* for 3-year old drives ~4-5% chance of failure whatever the utilisation
37 84 Steve Welburn
* failures are most likely at low drive temperatures (on start-up ?) i.e. < 25 deg. C
38 84 Steve Welburn
* drives over 2 years old are most likely to fail at high temperatures (could be mode of failure ?)
39 84 Steve Welburn
40 84 Steve Welburn
Disks with SMART scan errors are 10 times more likely to fail - almost 30% of drives with a SMART scan error failed within 8 months of the error.
41 84 Steve Welburn
* If a drive up to 8 months old gets a scan error, there's a 90% chance of it surviving at least 8 months
42 84 Steve Welburn
* If a drive over 2 years old gets a scan error, there's a 60% chance of it surviving at least 8 months
43 84 Steve Welburn
* If you have more than 1 scan error on a drive, it's significantly less likely to survive
44 84 Steve Welburn
* Similar for SMART reallocation counts AFR almost 20% if reallocation occurs in first 3 months
45 85 Steve Welburn
* ...but over 36% of failed drives had zero counts on all variables
46 85 Steve Welburn
47 85 Steve Welburn
bq. Talagala and Patterson [20] perform a detailed error analysis of 368 SCSI disk drives over an eighteen month period, reporting a failure rate of 1.9%. Results on a larger number of desktop-class ATA drives under deployment at the Internet Archive are presented by Schwarz et al [17]. They report on a 2% failure rate for a population of 2489 disks during 2005, while mentioning that replacement rates have been as high as 6% in the past. Gray and van Ingen [9] cite observed failure rates ranging from 3.3-6% in two large web properties with 22,400 and 15,805 disks respectively. A recent study by Schroeder and Gibson [16] helps shed light into the statistical properties of disk drive failures. The study uses failure data from several large scale deployments, including a large number of SATA drives. They report a significant overestimation of mean time to failure by manufacturers and a lack of infant mortality effects. None of these user studies have attempted to correlate failures with SMART parameters or other environmental factors.
48 85 Steve Welburn
49 84 Steve Welburn
50 83 Steve Welburn
Hard drive manufacturers often quote yearly failure rates below 2% [2]
51 83 Steve Welburn
User studies have seen rates as high as 6% [9]
52 83 Steve Welburn
53 83 Steve Welburn
Between 15-60% of drives returned to manufacturers having been considered to have failed by users have no defect as far as the manufacturers are concerned [7]
54 1 Steve Welburn
Between 20-30% “no problem found” cases were observed after analyzing failed drives from a study of 3477 disks [11]
55 83 Steve Welburn
56 84 Steve Welburn
Failure rates are known to be highly correlated with drive models, manufacturers and vintages [18].
57 83 Steve Welburn
58 8 Steve Welburn
h2. Overarching concerns
59 8 Steve Welburn
60 8 Steve Welburn
Human participation - ethics, data protection
61 10 Steve Welburn
62 10 Steve Welburn
Audio data - copyright
63 20 Steve Welburn
64 21 Steve Welburn
Storage - where ? how ? SLA ?
65 20 Steve Welburn
66 21 Steve Welburn
Short-term resilient storage for work-in-progress
67 1 Steve Welburn
68 1 Steve Welburn
Long-term archival storage for research data outputs
69 21 Steve Welburn
70 21 Steve Welburn
Curation of archived data - refreshing media and formats
71 1 Steve Welburn
72 1 Steve Welburn
Drivers - FoI, RCUK