Sound Data Management Training » History » Version 68

Steve Welburn, 2012-08-22 03:39 PM

1 5 Steve Welburn
h1. WP1.2 Online Training Material
2 1 Steve Welburn
3 9 Steve Welburn
{{>toc}}
4 9 Steve Welburn
5 68 Steve Welburn
We consider three stages of a research project, and the appropriate research data management considerations for each of those stages. The stages are:
6 41 Steve Welburn
* before the research;
7 41 Steve Welburn
* during the research;
8 41 Steve Welburn
* at the end of the research.
9 1 Steve Welburn
10 67 Steve Welburn
{{include(Before The Research)}}
11 67 Steve Welburn
12 67 Steve Welburn
h2. [[Before The Research]] - Planning Research Data Management
13 1 Steve Welburn
14 53 Steve Welburn
It is likely that some form of data management plan will be required as part of a grant proposal. The data management plan is an opportunity to think about the resources that will be required during the lifetime of the research project and to make sure that any necessary resources will be available for the project.
15 32 Steve Welburn
16 32 Steve Welburn
The main three questions the plan will cover are:
17 1 Steve Welburn
* What type of storage do you require ?
18 34 Steve Welburn
Do you need a lot of local disk space to store copies of standard datasets ? Will you be creating data which should be deposited in a long-term archive, or published online ? How will you back up your data ?
19 1 Steve Welburn
* How much storage do you require ?
20 34 Steve Welburn
Does it fit within the standard allocation for backed-up storage ?
21 1 Steve Welburn
* How long will you require the storage for ?
22 35 Steve Welburn
Is data being archived or published ? Does your funder require data publication ?
23 32 Steve Welburn
24 58 Steve Welburn
Appropriate answers will relate to:
25 59 Steve Welburn
* the [[types of data]] you will be using and creating;
26 59 Steve Welburn
* available existing [[data management resources]];
27 59 Steve Welburn
* [[funder requirements]];
28 59 Steve Welburn
* and relevant [[research data policies|policies]] (e.g. research group, institutional).
29 35 Steve Welburn
30 35 Steve Welburn
Additional questions may include:
31 57 Steve Welburn
* What is the appropriate [[license]] under which to publish data ?
32 60 Steve Welburn
* Are there any [[ethical concerns]] relating to data management e.g. identifiable participants ?
33 57 Steve Welburn
* Does your research data management plan comply with relevant [[legislation]] ?
34 38 Steve Welburn
e.g. Data Protection, Intellectual Property and Freedom of Information
35 34 Steve Welburn
36 34 Steve Welburn
It is likely that actual requirements will differ from initial estimates. Reviewing the data management plan against actual data use will allow you to assess whether additional resources are required.
37 28 Steve Welburn
38 36 Steve Welburn
In order to create an appropriate data management plan, it is necessary to consider data management requirements during and after the project.
39 36 Steve Welburn
40 42 Steve Welburn
h2. During The Research
41 28 Steve Welburn
42 40 Steve Welburn
During the course of a piece of research, data management is largely risk mitigation - it makes your research more robust and allows you to continue if something goes wrong.
43 31 Steve Welburn
44 31 Steve Welburn
The two main areas to consider are:
45 57 Steve Welburn
* [[backing up]] research data - in case you lose, or corrupt, the main copy of your data;
46 57 Steve Welburn
* [[documenting data]] - in case you need to to return to it later.
47 28 Steve Welburn
48 48 Steve Welburn
In addition to the immediate benefits during research, applying good research data management practices makes it easier to manage your research data at the end of your research project.
49 39 Steve Welburn
50 49 Steve Welburn
We have identified three basic types of research projects, two quantitative (one based on new data, one based on a new algorithm) and one qualitative, and consider the data management techniques appropriate to those workflows. More complex research projects may required a combination of the techniques from these.
51 1 Steve Welburn
52 49 Steve Welburn
h3. Quantitative research - New Data
53 49 Steve Welburn
54 50 Steve Welburn
For this use case, the research workflow involves:
55 50 Steve Welburn
* creating a new dataset
56 50 Steve Welburn
* testing outputs of existing algorithms on the dataset
57 50 Steve Welburn
* publication of results
58 1 Steve Welburn
59 1 Steve Welburn
60 50 Steve Welburn
The new dataset may include:
61 50 Steve Welburn
* Selection or creation of underlying (audio) data (the actual audio may be in the dataset or the dataset may reference material - e.g. for [[Copyright|copyright]] reasons)
62 50 Steve Welburn
* creation of ground-truth annotations for the audio and the type of algorithm (e.g. chord sequences for chord estimation, onset times for onset detection)
63 49 Steve Welburn
64 62 Steve Welburn
The content of the dataset will need to be [[documenting data|documented]].
65 49 Steve Welburn
66 50 Steve Welburn
Data involved includes:
67 64 Steve Welburn
* [[Managing Software As Data|software]] for the algorithms
68 50 Steve Welburn
* the new dataset
69 65 Steve Welburn
* identification of existing datasets against which results will be compared
70 50 Steve Welburn
* results of applying the algorithms to the dataset
71 50 Steve Welburn
* documentation of the testing methodology - including algorithm parameters
72 49 Steve Welburn
73 50 Steve Welburn
Note that *if* existing algorithms have published results using the same existing datasets and methodology, then results should be directly comparable between the published results and the results for the new dataset. In this case, most of the methodology is already documented and only details specific to the new dataset need separately recording.
74 50 Steve Welburn
75 50 Steve Welburn
If the testing is scripted, then the code used would be sufficient documentation during the research - readable documentation only being required at publication.
76 1 Steve Welburn
77 1 Steve Welburn
h3. Quantitative research - New Algorithm
78 1 Steve Welburn
79 29 Steve Welburn
bq. A common use-case in C4DM research is to run a newly-developed analysis algorithm on a set of audio examples and evaluate the algorithm by comparing its output with that of a human annotator. Results are then compared with published results using the same input data to determine whether the newly proposed approach makes any improvement on the state of the art.
80 29 Steve Welburn
81 1 Steve Welburn
Data involved includes:
82 64 Steve Welburn
* [[Managing Software As Data|software]] for the algorithm
83 1 Steve Welburn
* An annotated dataset against which the algorithm can be tested
84 1 Steve Welburn
* Results of applying the new algorithm and competing algorithms to the dataset
85 22 Steve Welburn
* Documentation of the testing methodology
86 1 Steve Welburn
87 23 Steve Welburn
Note that *if* other algorithms have published results using the same dataset and methodology, then results should be directly comparable between the published results and the results for the new algorithm. In this case, most of the methodology is already documented and only details specific to the new algorithm (e.g. parameters) need separately recording.
88 1 Steve Welburn
89 1 Steve Welburn
Also, if the testing is scripted, then the code used would be sufficient documentation during the research - readable documentation only being required at publication.
90 12 Steve Welburn
91 27 Steve Welburn
h3. Qualitative research
92 1 Steve Welburn
93 46 Steve Welburn
An example would be using interviews with performers to evaluate a new instrument design.
94 46 Steve Welburn
95 46 Steve Welburn
The workflow is:
96 46 Steve Welburn
* Gather data for the experiment (e.g. though interviews)
97 46 Steve Welburn
* Analyse data
98 24 Steve Welburn
* Publish data
99 24 Steve Welburn
100 24 Steve Welburn
Data involved may include:
101 24 Steve Welburn
* the interface design
102 24 Steve Welburn
* Captured audio from performances
103 1 Steve Welburn
* Recorded interviews with performers (possibly audio or video)
104 24 Steve Welburn
* Interview transcripts
105 1 Steve Welburn
106 1 Steve Welburn
The research may also involve:
107 1 Steve Welburn
* Demographic details of participants
108 1 Steve Welburn
* Identifiable participants (Data Protection])
109 8 Steve Welburn
* Release forms for people taking part
110 17 Steve Welburn
111 46 Steve Welburn
and *will* involve:
112 44 Steve Welburn
* ethics-board approval
113 1 Steve Welburn
114 44 Steve Welburn
h2. At The End Of The Research
115 44 Steve Welburn
116 66 Steve Welburn
Either the complete end of a research project or on completion of an identifiable unit of research (e.g. on publication of a paper based on your research).
117 44 Steve Welburn
118 63 Steve Welburn
* [[Archiving research data]]
119 63 Steve Welburn
* [[Publishing research data]]
120 56 Steve Welburn
* Reviewing the data management plan (possibly for the project final report)
121 44 Steve Welburn
122 44 Steve Welburn
Publication of the results of your research will require:
123 44 Steve Welburn
* Summarising the results
124 44 Steve Welburn
* Publishing the paper
125 44 Steve Welburn
126 44 Steve Welburn
Note that the EPSRC data management principles require sources of data to be referenced.
127 44 Steve Welburn
128 42 Steve Welburn
h2. Primary Investigator (PI)
129 42 Steve Welburn
130 55 Steve Welburn
The data management concerns of a PI will largely revolve around planning and appraisal of data management for research projects - both to make sure that they conform with institutional and funder requirements and to ensure that the data managment needs of the research project are met.
131 42 Steve Welburn
132 42 Steve Welburn
Areas of interest may involve:
133 57 Steve Welburn
* [[legislation|legalities]] (Freedom of Information, Copyright and Data Protection)
134 42 Steve Welburn
* data management plan
135 42 Steve Welburn
** covering the research council requirements
136 42 Steve Welburn
** during the project
137 42 Steve Welburn
** data archiving
138 42 Steve Welburn
** data publication
139 42 Steve Welburn
* After the project is completed, an appraisal of how the data was managed should be carried out as part of the project's "lessons learned"
140 42 Steve Welburn
141 42 Steve Welburn
Data management training should provide an overview of all the above, and keep PIs informed of any changes in the above that affect data management requirements.
142 18 Steve Welburn
143 8 Steve Welburn
The DCC DMP Online tool provides a series of questions which allow the user to build a data management plan which will match research council requirements.
144 8 Steve Welburn
145 8 Steve Welburn
h2. Overarching concerns
146 8 Steve Welburn
147 8 Steve Welburn
Human participation - ethics, data protection
148 10 Steve Welburn
149 10 Steve Welburn
Audio data - copyright
150 20 Steve Welburn
151 21 Steve Welburn
Storage - where ? how ? SLA ?
152 20 Steve Welburn
153 21 Steve Welburn
Short-term resilient storage for work-in-progress
154 1 Steve Welburn
155 1 Steve Welburn
Long-term archival storage for research data outputs
156 21 Steve Welburn
157 21 Steve Welburn
Curation of archived data - refreshing media and formats
158 1 Steve Welburn
159 1 Steve Welburn
Drivers - FoI, RCUK