Sound Data Management Training » History » Version 6
Steve Welburn, 2012-07-04 05:52 PM
| 1 | 5 | Steve Welburn | h1. WP1.2 Online Training Material |
|---|---|---|---|
| 2 | 1 | Steve Welburn | |
| 3 | 1 | Steve Welburn | h2. C4DM Researcher use cases |
| 4 | 1 | Steve Welburn | |
| 5 | 1 | Steve Welburn | h3. Quantitative testing - machine testing |
| 6 | 1 | Steve Welburn | |
| 7 | 1 | Steve Welburn | bq. A common use-case in C4DM research is to run a newly-developed analysis algorithm on a set of audio examples and evaluate the algorithm by comparing its output with that of a human annotator. Results are then compared with published results using the same input data to determine whether the newly proposed approach makes any improvement on the state of the art. |
| 8 | 1 | Steve Welburn | |
| 9 | 1 | Steve Welburn | Data involved includes: |
| 10 | 1 | Steve Welburn | * Software for the algorithm (which can be hosted on "Sound Software":http://www.soundsoftware.ac.uk) |
| 11 | 1 | Steve Welburn | * An annotated dataset against which the algorithm can be tested |
| 12 | 1 | Steve Welburn | * Results of applying the new algorithm and competing algorithms to the dataset |
| 13 | 1 | Steve Welburn | * Documentation of the testing methodology |
| 14 | 1 | Steve Welburn | |
| 15 | 1 | Steve Welburn | Note that *if* other algorithms have published results using the same dataset and methodology, then results should be directly comparable between the published results and the results for the new algorithm. In this case, the methodology is already documented. |
| 16 | 1 | Steve Welburn | |
| 17 | 1 | Steve Welburn | Also, if the testing is scripted, then the code used would be sufficient documentation during the research - readable documentation only being at publication. |
| 18 | 1 | Steve Welburn | |
| 19 | 1 | Steve Welburn | If no suitable annotated dataset already exists, a new dataset may be created including: |
| 20 | 3 | Steve Welburn | * Selection of underlying (audio) data (which may be [[Copyright|copyright]] material) |
| 21 | 1 | Steve Welburn | * Creation of ground-truth annotations for the audio and the type of algorithm (e.g. chord sequences for chord estimation, onset times for onset detection) |
| 22 | 1 | Steve Welburn | |
| 23 | 1 | Steve Welburn | h3. Qualitative testing - Listening tests |
| 24 | 1 | Steve Welburn | |
| 25 | 2 | Steve Welburn | An example would be testing audio at various levels of compression using both standard techniques and a newly derived algorithm. |
| 26 | 1 | Steve Welburn | |
| 27 | 6 | Steve Welburn | e.g. [[MUSHRA]] type tests. |
| 28 | 1 | Steve Welburn | |
| 29 | 1 | Steve Welburn | Data involved includes: |
| 30 | 1 | Steve Welburn | * Software for the algorithm (which can be hosted on "Sound Software":http://www.soundsoftware.ac.uk) |
| 31 | 1 | Steve Welburn | * Original uncompressed audio |
| 32 | 1 | Steve Welburn | * Audio output of the new algorithm on the audio |
| 33 | 1 | Steve Welburn | * Audio output of existing algorithms on the same audio |
| 34 | 1 | Steve Welburn | * Documentation of the testing methodology |
| 35 | 1 | Steve Welburn | |
| 36 | 1 | Steve Welburn | We note that for listening tests, the research may involve: |
| 37 | 1 | Steve Welburn | * Demographic details of participants |
| 38 | 3 | Steve Welburn | * Identifiable participants (Data Protection]) |
| 39 | 3 | Steve Welburn | * Release forms by people taking part |
| 40 | 1 | Steve Welburn | |
| 41 | 1 | Steve Welburn | and *will* involve: |
| 42 | 1 | Steve Welburn | * ethics-board approval |
| 43 | 1 | Steve Welburn | |
| 44 | 1 | Steve Welburn | h3. Publication |
| 45 | 1 | Steve Welburn | |
| 46 | 1 | Steve Welburn | Additionally, publication of results will require: |
| 47 | 1 | Steve Welburn | * Summarising the results |
| 48 | 1 | Steve Welburn | * Publishing the paper |