Sound Data Management Training » History » Version 8
Version 7 (Steve Welburn, 2012-07-04 05:54 PM) → Version 8/110 (Steve Welburn, 2012-07-04 05:56 PM)
h1. WP1.2 Online Training Material
(back to [[Wiki]])
h2. C4DM Researcher use cases
h3. Quantitative testing - machine testing
bq. A common use-case in C4DM research is to run a newly-developed analysis algorithm on a set of audio examples and evaluate the algorithm by comparing its output with that of a human annotator. Results are then compared with published results using the same input data to determine whether the newly proposed approach makes any improvement on the state of the art.
Data involved includes:
* Software for the algorithm (which can be hosted on "Sound Software":http://www.soundsoftware.ac.uk)
* An annotated dataset against which the algorithm can be tested
* Results of applying the new algorithm and competing algorithms to the dataset
* Documentation of the testing methodology
Note that *if* other algorithms have published results using the same dataset and methodology, then results should be directly comparable between the published results and the results for the new algorithm. In this case, the methodology is already documented.
Also, if the testing is scripted, then the code used would be sufficient documentation during the research - readable documentation only being at publication.
If no suitable annotated dataset already exists, a new dataset may be created including:
* Selection of underlying (audio) data (which may be [[Copyright|copyright]] material)
* Creation of ground-truth annotations for the audio and the type of algorithm (e.g. chord sequences for chord estimation, onset times for onset detection)
h3. Qualitative testing - Listening tests
An example would be testing audio at various levels of compression using both standard techniques and a newly derived algorithm.
e.g. [[MUSHRA]] type tests.
Data involved includes:
* Software for the algorithm (which can be hosted on "Sound Software":http://www.soundsoftware.ac.uk)
* Original uncompressed audio
* Audio output of the new algorithm on the audio
* Audio output of existing algorithms on the same audio
* Documentation of the testing methodology
We note that for listening tests, the research may involve:
* Demographic details of participants
* Identifiable participants (Data Protection])
* Release forms by people taking part
and *will* involve:
* ethics-board approval
h3. Publication
Additionally, publication of results will require:
* Summarising the results
* Publishing the paper
h2. Overarching concerns
Human participation - ethics, data protection
Audio data - copyright
(back to [[Wiki]])
h2. C4DM Researcher use cases
h3. Quantitative testing - machine testing
bq. A common use-case in C4DM research is to run a newly-developed analysis algorithm on a set of audio examples and evaluate the algorithm by comparing its output with that of a human annotator. Results are then compared with published results using the same input data to determine whether the newly proposed approach makes any improvement on the state of the art.
Data involved includes:
* Software for the algorithm (which can be hosted on "Sound Software":http://www.soundsoftware.ac.uk)
* An annotated dataset against which the algorithm can be tested
* Results of applying the new algorithm and competing algorithms to the dataset
* Documentation of the testing methodology
Note that *if* other algorithms have published results using the same dataset and methodology, then results should be directly comparable between the published results and the results for the new algorithm. In this case, the methodology is already documented.
Also, if the testing is scripted, then the code used would be sufficient documentation during the research - readable documentation only being at publication.
If no suitable annotated dataset already exists, a new dataset may be created including:
* Selection of underlying (audio) data (which may be [[Copyright|copyright]] material)
* Creation of ground-truth annotations for the audio and the type of algorithm (e.g. chord sequences for chord estimation, onset times for onset detection)
h3. Qualitative testing - Listening tests
An example would be testing audio at various levels of compression using both standard techniques and a newly derived algorithm.
e.g. [[MUSHRA]] type tests.
Data involved includes:
* Software for the algorithm (which can be hosted on "Sound Software":http://www.soundsoftware.ac.uk)
* Original uncompressed audio
* Audio output of the new algorithm on the audio
* Audio output of existing algorithms on the same audio
* Documentation of the testing methodology
We note that for listening tests, the research may involve:
* Demographic details of participants
* Identifiable participants (Data Protection])
* Release forms by people taking part
and *will* involve:
* ethics-board approval
h3. Publication
Additionally, publication of results will require:
* Summarising the results
* Publishing the paper
h2. Overarching concerns
Human participation - ethics, data protection
Audio data - copyright