Sound Data Management Training

h1. WP1.2 Online Training Material

(back to [[Wiki]])

{{>toc}}

We consider three stages of a reserach project, and the appropriate research data management considerations for each of those stages. The stages are:

* before the research;

* during the research;

* at the end of the research.

h2. Before The Research - Planning Research Data Management

It is likely that some form of data management plan will be required as part of a grant proposal. The data management plan is an opportunity to think about the resources that will be required during the lifetime of the research project and to make sure that any necessary resources can be funded through the project.

The main three questions the plan will cover are:

* What type of storage do you require ?

Do you need a lot of local disk space to store copies of standard datasets ? Will you be creating data which should be deposited in a long-term archive, or published online ? How will you back up your data ?

* How much storage do you require ?

Does it fit within the standard allocation for backed-up storage ?

* How long will you require the storage for ?

Is data being archived or published ? Does your funder require data publication ?

Appropriate answers will relate to: the types of data you will be using and creating; available existing resources; funder requirements; and relevant policies (e.g. research group, institutional).

Additional questions may include:

* What is the appropriate license under which to publish data ?

* Does your research data management plan comply with relevant legislation ?

e.g. Data Protection, Intellectual Property and Freedom of Information

It is likely that actual requirements will differ from initial estimates. Reviewing the data management plan against actual data use will allow you to assess whether additional resources are required.

In order to create an appropriate data management plan, it is necessary to consider data management requirements during and after the project.

h2. During The Research

During the course of a piece of research, data management is largely risk mitigation - it makes your research more robust and allows you to continue if something goes wrong.

The two main areas to consider are:

* backing up research data - in case you lose, or corrupt, the main copy of your data;

* documenting data - in case you need to to return to it later.

In addition to the immediate benefits during research, applying good research data management practices makes it easier to manage your research data at the end of your research project.

We have identified three basic types of research projects, two quantitative (one based on new data, one based on a new algorithm) and one qualitative, and consider the data management techniques appropriate to those workflows. More complex research projects may required a combination of the techniques from these.

h3. Quantitative research - New Data

bq. A common use-case in C4DM research is to run a newly-developed analysis algorithm on a set of audio examples and evaluate the algorithm by comparing its output with that of a human annotator. Results are then compared with published results using the same input data to determine whether the newly proposed approach makes any improvement on the state of the art.

There are two main types of quantitative research which we consider:

* Testing new data using existing algorithms

* Using existing data, and algoritghms, to test a new algorithm.

Data involved includes:

* Software for the algorithm (which can be hosted on "Sound Software":http://www.soundsoftware.ac.uk)

* An annotated dataset against which the algorithm can be tested

* Results of applying the new algorithm and competing algorithms to the dataset

* Documentation of the testing methodology

Note that *if* other algorithms have published results using the same dataset and methodology, then results should be directly comparable between the published results and the results for the new algorithm. In this case, most of the methodology is already documented and only details specific to the new algorithm (e.g. parameters) need separately recording.

Also, if the testing is scripted, then the code used would be sufficient documentation during the research - readable documentation only being required at publication.

If no suitable annotated dataset already exists, a new dataset may be created including:

* Selection of underlying (audio) data (the actual audio may be in the dataset or the dataset may reference material - e.g. for [[Copyright|copyright]] reasons)

* Ground-truth annotations for the audio and the type of algorithm (e.g. chord sequences for chord estimation, onset times for onset detection)

h3. Quantitative research - New Algorithm

bq. A common use-case in C4DM research is to run a newly-developed analysis algorithm on a set of audio examples and evaluate the algorithm by comparing its output with that of a human annotator. Results are then compared with published results using the same input data to determine whether the newly proposed approach makes any improvement on the state of the art.

There are two main types of quantitative research which we consider:

* Testing new data using existing algorithms

* Using existing data, and algoritghms, to test a new algorithm.

Data involved includes:

* Software for the algorithm (which can be hosted on "Sound Software":http://www.soundsoftware.ac.uk)

* An annotated dataset against which the algorithm can be tested

* Results of applying the new algorithm and competing algorithms to the dataset

* Documentation of the testing methodology

Note that *if* other algorithms have published results using the same dataset and methodology, then results should be directly comparable between the published results and the results for the new algorithm. In this case, most of the methodology is already documented and only details specific to the new algorithm (e.g. parameters) need separately recording.

Also, if the testing is scripted, then the code used would be sufficient documentation during the research - readable documentation only being required at publication.

If no suitable annotated dataset already exists, a new dataset may be created including:

* Selection of underlying (audio) data (the actual audio may be in the dataset or the dataset may reference material - e.g. for [[Copyright|copyright]] reasons)

* Ground-truth annotations for the audio and the type of algorithm (e.g. chord sequences for chord estimation, onset times for onset detection)

h3. Qualitative research

An example would be using interviews with performers to evaluate a new instrument design.

The workflow is:

* Gather data for the experiment (e.g. though interviews)

* Analyse data

* Publish data

Data involved may include:

* the interface design

* Captured audio from performances

* Recorded interviews with performers (possibly audio or video)

* Interview transcripts

The research may also involve:

* Demographic details of participants

* Identifiable participants (Data Protection])

* Release forms for people taking part

and *will* involve:

* ethics-board approval

h2. At The End Of The Research

(Includes on publication of a paper based on your research)

* Archiving research data

* Publishing research data

* Reviewing the data management plan

Publication of the results of your research will require:

* Summarising the results

* Publishing the paper

Note that the EPSRC data management principles require sources of data to be referenced.

h2. Primary Investigator (PI)

The data management concerns of a PI will largely revolve around planning and appraisal of data management for research projects.

Areas of interest may involve:

* legalities (Freedom of Information, Copyright and Data Protection)

* data management plan

** covering the research council requirements

** during the project

** data archiving

** data publication

* After the project is completed, an appraisal of how the data was managed should be carried out as part of the project's "lessons learned"

Data management training should provide an overview of all the above, and keep PIs informed of any changes in the above that affect data management requirements.

The DCC DMP Online tool provides a series of questions which allow the user to build a data management plan which will match research council requirements.

h2. Overarching concerns

Human participation - ethics, data protection

Audio data - copyright

Storage - where ? how ? SLA ?

Short-term resilient storage for work-in-progress

Long-term archival storage for research data outputs

Curation of archived data - refreshing media and formats

Drivers - FoI, RCUK