Mercurial > hg > webaudioevaluationtool
changeset 413:1c5894cdcb9c Dev_main
Instructions update (WIP); removed basic instructions from README.
| author | Brecht De Man <b.deman@qmul.ac.uk> |
|---|---|
| date | Fri, 18 Dec 2015 01:45:54 +0000 |
| parents | 930354145f6c |
| children | c418c34cd55e |
| files | README.txt docs/Instructions/Instructions.pdf docs/Instructions/Instructions.tex scripts/generate_report.py scripts/score_parser.py |
| diffstat | 5 files changed, 373 insertions(+), 161 deletions(-) [+] |
line wrap: on
line diff
--- a/README.txt Thu Dec 17 11:11:57 2015 +0000 +++ b/README.txt Fri Dec 18 01:45:54 2015 +0000 @@ -1,127 +1,11 @@ WEB AUDIO EVALUATION TOOL -This is not (yet) a fully fledged manual. - - AUTHORS Nicholas Jillings <n.g.r.jillings@se14.qmul.ac.uk> Brecht De Man <b.deman@qmul.ac.uk> David Moffat <d.j.moffat@qmul.ac.uk> Joshua D. Reiss (supervisor) <j.d.reiss@qmul.ac.uk> +INSTRUCTIONS FOR USE -PACKAGE CONTENTS - -- main folder (/) - - ape.css, core.css, graphics.css, structure.css: style files (edit to change appearance) - - ape.js: JavaScript file for APE-style interface [1] - - core.js: JavaScript file with core functionality - - index.html: webpage where interface should appear - - jquery-2.1.4.js: jQuery JavaScript Library - - pythonServer.py: webserver for running tests locally - - pythonServer-legacy.py: webserver with limited functionality (no storing of output XML files) -- Documentation (/docs/) - - Project Specification Document (LaTeX/PDF) - - Results Specification Document (LaTeX/PDF) - - SMC15: PDF and LaTeX source of corresponding SMC2015 publication -- Example project (/example_eval/) - An example of what the set up XML should look like, with example audio files 0.wav-10.wav which are short recordings at 44.1kHz, 16bit of a woman saying the corresponding number (useful for testing randomisation and general familiarisation with the interface). -- Output files (/saves/) - The output XML files of tests will be stored here by default by the pythonServer.py script. -- Auxiliary scripts (/scripts/) - Helpful Python scripts for extraction and visualisation of data. -- Test creation tool (/test_create/) - Webpage for easily setting up a test without having to delve into the XML. - - -QUICK START - -Using the example project: -1. Make sure your system sample rate corresponds with the sample rate of the audio files, if the input XML file enforces the given sample rate. -2. Run pythonServer.py (make sure you have Python installed). -3. Open a browser (anything but Internet Explorer). -4. Go to ‘localhost:8000’. -5. The test should open; complete it and look at the output XML file in /saves/. - - -LEGACY - -The APE interface and most of the functionality of the interface is inspired by the APE toolbox for MATLAB [1]. See https://code.soundsoftware.ac.uk/projects/ape for the source code and corresponding paper. - - -CITING - -We request that you acknowledge the authors and cite our work [2], see CITING.txt. - - -LICENSE - -See LICENSE.txt. This code is shared under the GNU General Public License v3.0 (http://choosealicense.com/licenses/gpl-3.0/). Generally speaking, this is a copyleft license that requires anyone who distributes our code or a derivative work to make the source available under the same terms. - - -FEATURE REQUESTS AND BUG REPORTS - -We continually develop this tool to fix issues and implement features useful to us or our user base. See https://code.soundsoftware.ac.uk/projects/webaudioevaluationtool/issues for a list of feature requests and bug reports, and their status. - -Please contact the authors if you experience any bugs, if you would like additional functionality, if you have questions about using the interface or if you would like to give any feedback (even positive!) about the interface. We look forward to learning how the tool has (not) been useful to you. - - -TROUBLESHOOTING - -Thanks to feedback from using the interface in experiments by the authors and others, many bugs have been caught and fatal crashes due to the interface (provided it is set up properly by the user) seem to be a thing of the past. -However, if things do go wrong or the test needs to be interrupted for whatever reason, all data is not lost. In a normal scenario, the test needs to be completed until the end (the final ‘Submit’), at which point the output XML is stored in ‘saves/‘. If this stage is not reached, open the JavaScript Console (see below for how to find it) and type ‘createProjectSave()’ (without the quotes) and hit enter. This will open a pop-up window with a hyperlink that reads ‘Save File’; click it and an XML file with results until that point should be stored in your download folder. -Alternatively, a lot of data can be read from the same console, in which the tool prints a lot of debug information. Specifically: - - the randomisation of pages and fragments are logged; - - any time a slider is played, its ID and the time stamp (in seconds since the start of the test) are displayed; - - any time a slider is dragged and dropped, the location where it is dropped including the time stamp are shown; - - any comments and pre- or post-test questions and their answers are logged as well. - -You can select all this and save into a text file, so that none of this data is lost. You may to choose to do this even when a test was successful as an extra precaution. - -In Google Chrome, the JavaScript Console can be found in View>Developer>JavaScript Console, or via the keyboard shortcut Cmd + Alt + J (Mac OS X). -In Safari, the JavaScript Console can be found in Develop>Show Error Console, or via the keyboard shortcut Cmd + Alt + C (Mac OS X). Note that for the Developer menu to be visible, you have to go to Preferences (Cmd + ,) and enable ‘Show Develop menu in menu bar’ in the ‘Advanced’ tab. -In Firefox, go to Tools>Web Developer>Web Console, or hit Cmd + Alt + K. - - -REMOTE TESTS - -As the test is browser-based, it can be run remotely from a web server without modification. To allow for remote storage of the output XML files (as opposed to saving them locally on the subject’s machine, which is the default if no ‘save’ path is specified or found), a PHP script on the server needs to accept the output XML files. An example of such script will be included in a future version. - - -SCRIPTS - -The tool comes with a few handy Python (2.7) scripts for easy extraction of ratings or comments, and visualisation of ratings and timelines. See below for a quick guide on how to use them. All scripts written for Python 2.7. Visualisation requires the free matplotlib toolbox (http://matplotlib.org), numpy and scipy. -By default, the scripts can be run from the ‘scripts’ folder, with the result files in the ‘saves’ folder (the default location where result XMLs are stored). Each script takes the XML file folder as an argument, along with other arguments in some cases. -Note: to avoid all kinds of problems, please avoid using spaces in file and folder names (this may work on some systems, but others don’t like it). - - comment_parser.py - Extracts comments from the output XML files corresponding with the different subjects found in ‘saves/’. It creates a folder per ‘audioholder’/page it finds, and stores a CSV file with comments for every ‘audioelement’/fragment within these respective ‘audioholders’/pages. In this CSV file, every line corresponds with a subject/output XML file. Depending on the settings, the first column containing the name of the corresponding XML file can be omitted (for anonymisation). - Beware of Excel: sometimes the UTF-8 is not properly imported, leading to problems with special characters in the comments (particularly cumbersome for foreign languages). - - evaluation_stats.py - Shows a few statistics of tests in the ‘saves/‘ folder so far, mainly for checking for errors. Shows the number of files that are there, the audioholder IDs that were tested (and how many of each separate ID), the duration of each page, the duration of each complete test, the average duration per page, and the average duration in function of the page number. - - generate_report.py - Similar to ‘evaluation_stats.py’, but generates a PDF report based on the output files in the ‘saves/‘ folder - or any folder specified as command line argument. Uses pdflatex to write a LaTeX document, then convert to a PDF. - - score_parser.py - Extracts rating values from the XML to CSV - necessary for running visualisation of ratings. Creates the folder ‘saves/ratings/‘ if not yet created, to which it writes a separate file for every ‘audioholder’/page in any of the output XMLs it finds in ‘saves/‘. Within each file, rows represent different subjects (output XML file names) and columns represent different ‘audioelements’/fragments. - - score_plot.py - Plots the ratings as stored in the CSVs created by score_parser.py - Depending on the settings, it displays and/or saves (in ‘saves/ratings/’) a boxplot, confidence interval plot, scatter plot, or a combination of the aforementioned. - Requires the free matplotlib library. - At this point, more than one subjects are needed for this script to work. - - timeline_view_movement.py - Creates a timeline for every subject, for every ‘audioholder’/page, corresponding with any of the output XML files found in ‘/saves’. It shows the marker movements of the different fragments, along with when each fragment was played (red regions). Automatically takes fragment names, rating axis title, rating axis labels, and audioholder name from the XML file (if available). - - timeline_view.py - Creates a timeline for every subject, for every ‘audioholder’/page, corresponding with any of the output XML files found in ‘/saves’. It shows when and for how long the subject listened to each of the fragments. - - - -REFERENCES -[1] B. De Man and Joshua D. Reiss, “APE: Audio Perceptual Evaluation toolbox for MATLAB,” 136th Convention of the Audio Engineering Society, 2014. - -[2] Nicholas Jillings, Brecht De Man, David Moffat and Joshua D. Reiss, "Web Audio Evaluation Tool: A Browser-Based Listening Test Environment," 12th Sound and Music Computing Conference, July 2015. +Please refer to ‘docs/Instructions/Instructions.pdf’ \ No newline at end of file
--- a/docs/Instructions/Instructions.tex Thu Dec 17 11:11:57 2015 +0000 +++ b/docs/Instructions/Instructions.tex Fri Dec 18 01:45:54 2015 +0000 @@ -7,6 +7,11 @@ % TeX will automatically convert eps --> pdf in pdflatex \usepackage{listings} % Source code +\usepackage{xcolor} % colour (source code for instance) +\definecolor{grey}{rgb}{0.1,0.1,0.1} +\definecolor{darkblue}{rgb}{0.0,0.0,0.6} +\definecolor{cyan}{rgb}{0.0,0.6,0.6} + \usepackage{amssymb} \usepackage{cite} \usepackage{hyperref} % Hyperlinks @@ -21,7 +26,12 @@ \begin{document} \maketitle -These instructions are about use of the Web Audio Evaluation Tool \cite{waet} on Windows and Mac OS X platforms. +These instructions are about use of the Web Audio Evaluation Tool on Windows and Mac OS X platforms. + +We request that you acknowledge the authors and cite our work when using it \cite{waet}, see also CITING.txt. + +The tool is available in its entirety including source code on \url{https://code.soundsoftware.ac.uk/projects/webaudioevaluationtool/}, under the GNU General Public License v3.0 (\url{http://choosealicense.com/licenses/gpl-3.0/}), see also LICENSE.txt. + % TO DO: Linux \tableofcontents @@ -29,7 +39,7 @@ \clearpage \section{Installation} - Download the folder (\url{https://code.soundsoftware.ac.uk/hg/webaudioevaluationtool/archive/tip.zip}) and unzip in a location of your choice. + Download the folder (\url{https://code.soundsoftware.ac.uk/hg/webaudioevaluationtool/archive/tip.zip}) and unzip in a location of your choice, or pull the source code from \url{https://code.soundsoftware.ac.uk/hg/webaudioevaluationtool} (Mercurial). \subsection{Contents} The folder should contain the following elements: \\ @@ -39,22 +49,24 @@ \item \texttt{analyse.html}: analysis and diagnostics of a set of result XML files \item \texttt{ape.css, core.css, graphics.css, mushra.css, structure.css}: style files (edit to change appearance) \item \texttt{ape.js}: JavaScript file for APE-style interface \cite{ape} - \item \texttt{mushra.js}: JavaScript file for MUSHRA-style interface \cite{mushra} - \item \texttt{CITING.txt, LICENSE.txt, README.txt}: text files with, respectively, the citation which we ask to include in any work where this tool or any portion thereof is used, modified or otherwise; the license under which the software is shared; and a general readme file. + \item \texttt{CITING.txt, LICENSE.txt, README.txt}: text files with, respectively, the citation which we ask to include in any work where this tool or any portion thereof is used, modified or otherwise; the license under which the software is shared; and a general readme file referring to these instructions. \item \texttt{core.js}: JavaScript file with core functionality \item \texttt{index.html}: webpage where interface should appear (includes link to test configuration XML) \item \texttt{jquery-2.1.4.js}: jQuery JavaScript Library + \item \texttt{loudness.js}: Allows for automatic calculation of loudness of Web Audio API Buffer objects, return gain values to correct for a target loudness or match loudness between multiple objects + \item \texttt{mushra.js}: JavaScript file for MUSHRA-style interface \cite{mushra} \item \texttt{pythonServer.py}: webserver for running tests locally \item \texttt{pythonServer-legacy.py}: webserver with limited functionality (no automatic storing of output XML files) \item \texttt{save.php}: PHP script to store result XML files to web server\\ \end{itemize} \textbf{Documentation (./docs/)} \begin{itemize} + \item \href{http://c4dm.eecs.qmul.ac.uk/dmrn/events/dmrnp10/#posters}{DMRN+10}: PDF and \LaTeX source of poster for 10\textsuperscript{th} Digital Music Research Network One-Day workshop (``soft launch'') \item Instructions: PDF and \LaTeX source of these instructions \item Project Specification Document (\LaTeX/PDF) \item Results Specification Document (\LaTeX/PDF) - \item SMC15: PDF and \LaTeX source of corresponding SMC2015 publication \cite{waet} - \item WAC2016: PDF and \LaTeX source of corresponding WAC2016 publication\\ + \item SMC15: PDF and \LaTeX source of 12th Sound and Music Computing Conference paper \cite{waet} + \item WAC2016: PDF and \LaTeX source of 2nd Web Audio Conference paper\\ \end{itemize} \textbf{Example project (./example\_eval/)} \begin{itemize} @@ -73,13 +85,15 @@ \item Webpage for easily setting up your own test without having to delve into the XML.\\ \end{itemize} - \subsection{Browser} + \subsection{Compatibility} As Microsoft Internet Explorer doesn't support the Web Audio API\footnote{\url{http://caniuse.com/\#feat=audio-api}}, you will need another browser like Google Chrome, Safari or Firefox (all three are tested and confirmed to work). + + Firefox does not currently support other bit depths than 8 or 16 bit for PCM wave files. In the future, this will throw a warning message to tell the user that their content is being quantised automatically. %Nick? Right? To be removed if and when actually implemented The tool is platform-independent and works in any browser that supports the Web Audio API. It does not require any specific, proprietary software. However, in case the tool is hosted locally (i.e. you are not hosting it on an actual webserver) you will need Python (2.7), which is a free programming language - see the next paragraph. +\clearpage -\clearpage \section{Test setup} @@ -197,9 +211,263 @@ \subsection{Remote test} Put all files on a web server which supports PHP. This allows the `save.php' script to store the XML result files in the `saves/' folder. If the web server is not able to store the XML file there at the end of the test, it will present the XML file locally to the user, as a `Save file' link. - + + Make sure the \texttt{projectReturn} attribute of the \texttt{setup} node is set to the \texttt{save.php} script. + + Then, just go to the URL of the corresponding HTML file, e.g. \texttt{http://server.com/path/to/WAET/index.html?url=test/my-test.xml}. If storing on the server doesn't work at submission (e.g. if the \texttt{projectReturn} attribute isn't properly set), the result XML file will be presented to the subject on the client side, as a `Save file' link. + + \clearpage + +\section{Interfaces} + + The Web Audio Evaluation Tool comes with a number of interface styles, each of which can be customised extensively, either by configuring them differently using the many optional features, or by modifying the JavaScript files. + + To set the interface style for the whole test, %Nick? change when this is not the case anymore, i.e. when the interface can be set per page + add \texttt{interface="APE"} to the \texttt{setup} node, where \texttt{"APE"} is one of the interface names below. + + \subsection{APE} + The APE interface is based on \cite{ape}, and consists of one or more axes, each corresponding with an attribute to be rated, on which markers are placed. As such, it is a multiple stimulus interface where (for each dimension or attribute) all elements are on one axis so that they can be maximally compared against each other, as opposed to rated individually or with regards to a single reference. + It also contains an optional text box for each element, to allow for clarification by the subject, tagging, and so on. + + \subsection{MUSHRA} + This is a straightforward implementation of \cite{mushra}, especially common for the rating of audio quality, for instance for the evaluation of audio codecs. + +\clearpage + +\section{Features} + + This section goes over the different features implemented in the Web Audio Evaluation Tool, how to use them, and what to know about them. + + Unless otherwise specified, \emph{each} feature described here is optional, i.e. it can be enabled or disabled and adjusted to some extent. + + As the example project showcases (nearly) all of these features, please refer to its configuration XML document for a demonstration of how to enable and adjust them. + + \subsection{Surveys} + \subsubsection{Pre- and post-page surveys} + + \subsubsection{Pre- and post-test surveys} + + \subsubsection{Survey elements} + All survey elements (which `pop up' in the centre of the browser) have an \texttt{id} attribute, for retrieval of the responses in post-processing of the results, and a \texttt{mandatory} attribute, which if set to ``true'' requires the subjects to respond before they can continue. + + \begin{description} + \item[statement] Simply shows text to the subject until `Next' or `Start' is clicked. + \item[question] Expects a text answer (in a text box). Has the \texttt{boxsize} argument: set to ``large'' or ``huge'' for a bigger box size. + \item[number] Expects a numerical value. Attribute \texttt{min="0"} specifies the minimum value - in this case the answer must be stricly positive before the subject can continue. + \item[radio] Radio buttons. + \item[checkbox] Checkboxes.\\ + \end{description} + + \textbf{Example usage:}\\ + + \lstset{ + basicstyle=\ttfamily, + columns=fullflexible, + showstringspaces=false, + commentstyle=\color{grey}\upshape + } + + \lstdefinelanguage{XML} + { + morestring=[b]", + morestring=[s]{>}{<}, + morecomment=[s]{<?}{?>}, + stringstyle=\color{black} \bfseries, + identifierstyle=\color{darkblue} \bfseries, + keywordstyle=\color{cyan} \bfseries, + morekeywords={xmlns,version,type}, + breaklines=true% list your attributes here + } + \scriptsize + \lstset{language=XML} + + \begin{lstlisting} +<PostTest> + <question id="location" mandatory="true" boxsize="large">Please enter your location. (example mandatory text question)</question> + <number id="age" min="0">Please enter your age (example non-mandatory number question)</number> + <radio id="rating"> + <statement>Please rate this interface (example radio button question)</statement> + <option name="bad">Bad</option> + <option name="poor">Poor</option> + <option name="good">Good</option> + <option name="great">Great</option> + </radio> + <statement>Thank you for taking this listening test. Please click 'Submit' and your results will appear in the 'saves/' folder.</statement> +</PostTest> + \end{lstlisting} + + + \subsection{Randomisation} + + \subsubsection{Randomisation of configuration XML files} + % how to + % explain how this is implemented in the pythonServer + %Nick? already implemented in the PHP? + + + \subsubsection{Randomsation of page order} + + + \subsubsection{Randomisation of axis order} + + \subsubsection{Randomisation of fragment order} + + + \subsubsection{Randomisation of initial slider position} + + % /subsubsection{Randomisation of survey question order} + % should be an attribute of the individual 'pretest' and 'posttest' elements + % uncomment once we have it + + \subsection{Looping} + Loops the fragments + % how to enable? + If the fragments are not of equal length initially, they are padded with zeros so that they are equal length, to enable looping without the fragments going out of sync relative to each other. + + Note that fragments cannot be played until all fragments are loaded when in looped mode, as the engine needs to assess whether all %Nick? Is this accurate? + + \subsection{Sample rate} + If you require the test to be conducted at a certain sample rate (i.e. you do not tolerate resampling of the elements to correspond with the system's sample rate), add \texttt{sampleRate="96000"} - where ``96000'' can be any support sample rate - so that a warning message is shown alerting the subject the system's sample rate is different from this enforced sample rate. This of course means that in one test, all sample rates must be equal as it is impossible to change the system's sample rates during the test (even if you were to manually change it, then the browser must be restarted for it to take effect). + + \subsection{Scrubber bar} + The scrubber bar, or transport bar (that is the name of the visualisation of the playhead thing with an indication of time and showing the portion of the file played so far) is at this point just a visual, and not a controller to adjust the playhead position. + + Make visible by adding \texttt{<option name='playhead'/>} to the \texttt{interface} node (see Section \ref{sec:checks}: Checks). + + \subsection{Metrics} + Enable the collection of metrics by adding \texttt{collectMetrics=`true'} in the \texttt{setup} node. + + The \texttt{Metric} node, which contains the metrics to be tracked during the complete test, is a child of the \texttt{setup} node, and it could look as follows. + + \begin{lstlisting} +<Metric> + <metricEnable>testTimer</metricEnable> + <metricEnable>elementTimer</metricEnable> + <metricEnable>elementInitialPosition</metricEnable> + <metricEnable>elementTracker</metricEnable> + <metricEnable>elementFlagListenedTo</metricEnable> + <metricEnable>elementFlagMoved</metricEnable> + <metricEnable>elementListenTracker</metricEnable> +</Metric> + \end{lstlisting} + + When in doubt, err on the inclusive side, as one never knows which information is needed in the future. Most of these metrics are necessary for post-processing scripts such as timeline\_view\_movement.py. + + \subsubsection{Time test duration} + \texttt{testTimer}\\ + + \subsubsection{Time fragment playback} + \texttt{elementTimer}\\ + This keeps track of when each fragment is played back and stopped again, \emph{and} which part of the fragment has been played back at that time. + % example output? + + \subsubsection{Initial positions} + \texttt{elementInitialPosition}\\ + Tracks the initial position of the sliders, especially relevant when these are randomised, so their influence + + \subsubsection{Track movements} + + \subsubsection{Which fragments listened to} + + \subsubsection{Which fragments moved} + Binary check whether or not a the marker corresponding with a particular fragment was moved at all throughout the experiment. + + \subsubsection{elementListenTracker} %Nick? No idea what this does, if it's not what I wrote under 'Time fragment playback' + + \subsection{References and anchors} + \subsubsection{Reference} + %... + \subsubsection{Hidden reference} + %... + \subsubsection{Hidden anchor} + %... + + \subsection{Checks} + \label{sec:checks} + + %blabla + These checks are enabled in the \texttt{interface} node, which is a child of the \texttt{setup} node. + \subsubsection{Playback checks} + % what it does/is + Enforce playing each sample at least once, for at least a little bit (e.g. this test is satisfied even if you only play a tiny portion of the file), by alerting the user to which samples have not been played upon clicking `Submit'. When enabled, one cannot proceed to the next page, answer a survey question, or finish the test, before clicking each sample at least once. + % how to enable/disable + + Alternatively, one can check whether the \emph{entire} fragment was listened to at least once. + % how to enable + + Add \texttt{<check name="fragmentPlayed"/>} to the \texttt{interface} node. + + + \subsubsection{Movement check} + Enforce moving each sample at least once, for at least a little bit (e.g. this test is satisfied even if you only play a tiny portion of the file), by alerting the user to which samples have not been played upon clicking `Submit'. When enabled, one cannot proceed to the next page, answer a survey question, or finish the test, before clicking each sample at least once. + If there are several axes, the warning will specify which samples have to be moved on which axis. + + Add \texttt{<check name="fragmentMoved"/>} to the \texttt{interface} node. + + \subsubsection{Comment check} + % How to enable/disable? + + Enforce commenting, by alerting the user to which samples have not been commented on upon clicking `Submit'. When enabled, one cannot proceed to the next page, answer a survey question, or finish the test, before putting at least one character in each comment box. + + Note that this does not apply to any extra (text, radio button, checkbox) elements, unless these have the `mandatory' option enabled. %Nick? is this extra 'mandatory' option implemented? + + Add \texttt{<check name="fragmentComments"/>} to the \texttt{interface} node. + + %ADD: how to add a custom comment box + + \subsubsection{Scale use check} + It is possible to enforce a certain usage of the scale, meaning that at least one slider needs to be below and/or above a certain percentage of the slider. + + Add \texttt{<check name="scalerange" min="25" max="75"/>} to the \texttt{interface} node. + + \subsubsection{Note on the use of multiple rating axes} + I.e. what if more than one axis? How to specify which axis the checks relate to? %Nick? to add? + + \subsection{Layout options} + \texttt{title}, \texttt{scale}, \texttt{position}, \texttt{commentBoxPrefix} + + \subsection{Multiple sliders} + (APE example) + + \begin{lstlisting} +<interface name="preference"> + <title>Preference</title> + <scale position="0">Min</scale> + <scale position="100">Max</scale> + <scale position="50">Middle</scale> + <commentBoxPrefix>Comment on fragment</commentBoxPrefix> +</interface> +<interface name="depth"> + <title>Depth</title> + <scale position="0">Low</scale> + <scale position="100">High</scale> + <scale position="50">Middle</scale> + <commentBoxPrefix>Comment on fragment</commentBoxPrefix> +</interface> + \end{lstlisting} + where the \texttt{interface} nodes are children of the \texttt{audioholder} node. + + \subsection{Platform information} + % what does it do, what does it look like + % limitations? + + \subsection{Show progress} + Add \texttt{<option name="page-count"/>} to the \texttt{interface} node (see Section \ref{sec:checks}: Checks) to add the current page number and the total number of pages to the interface. + + \subsection{Gain} + It is possible to set the gain (in decibel) applied to the different audioelements, as an attribute of the \texttt{audioelement} nodes in the configuration XML file: + + \texttt{<audioElements url="sample-01.wav" gain="-6" id="sample01quieter" />} + + \subsection{Loudness} + % automatic loudness equalisation + % guide to loudness.js + Set the loudness for a complete test by adding \texttt{loudness="-23"} to the \texttt{setup} node in the configuration XML file, where -23 is an example loudness in LUFS. + +\clearpage + \section{Using the test create tool} We provide a test creation tool, available in the directory test\_create. This tool is a self-contained web page, so doubling clicking will launch the page in your system default browser. @@ -245,54 +513,96 @@ It is also the job the interfaceDOM to call any metric collection functions necessary, however some functions may be better placed outside (for example, the APE interface uses drag and drop, therefore the best way was to call the metric functions from the dragEnd function, which is called when the interface object is dropped). Metrics based upon listening are handled by the audioObject. The interfaceDOM object must manage any movement metrics. For a list of valid metrics and their behaviours, look at the project specification document included in the repository/docs location. The same goes for any checks required when pressing the submit button, or any other method to proceed the test state. +\clearpage +\section{Analysis and diagnostics} + \subsection{In the browser} + See `analysis.html' in the main folder: immediate visualisation of (by default) all results in the `saves/' folder. + + \subsection{Python scripts} + The package includes Python (2.7) scripts (in `scripts/') to extract ratings and comments, generate visualisations of ratings and timelines, and produce a fully fledged report. + + Visualisation requires the free matplotlib toolbox (http://matplotlib.org), numpy and scipy. + By default, the scripts can be run from the `scripts' folder, with the result files in the `saves' folder (the default location where result XMLs are stored). Each script takes the XML file folder as an argument, along with other arguments in some cases. + Note: to avoid all kinds of problems, please avoid using spaces in file and folder names (this may work on some systems, but others don't like it). + + \subsubsection{comment\_parser.py} + Extracts comments from the output XML files corresponding with the different subjects found in `saves/'. It creates a folder per `audioholder'/page it finds, and stores a CSV file with comments for every `audioelement'/fragment within these respective `audioholders'/pages. In this CSV file, every line corresponds with a subject/output XML file. Depending on the settings, the first column containing the name of the corresponding XML file can be omitted (for anonymisation). + Beware of Excel: sometimes the UTF-8 is not properly imported, leading to problems with special characters in the comments (particularly cumbersome for foreign languages). + + \subsubsection{evaluation\_stats.py} + Shows a few statistics of tests in the `saves/' folder so far, mainly for checking for errors. Shows the number of files that are there, the audioholder IDs that were tested (and how many of each separate ID), the duration of each page, the duration of each complete test, the average duration per page, and the average duration in function of the page number. + + \subsubsection{generate\_report.py} + Similar to `evaluation\_stats.py', but generates a PDF report based on the output files in the `saves/' folder - or any folder specified as command line argument. Uses pdflatex to write a LaTeX document, then convert to a PDF. + + \subsubsection{score\_parser.py} + Extracts rating values from the XML to CSV - necessary for running visualisation of ratings. Creates the folder `saves/ratings/' if not yet created, to which it writes a separate file for every `audioholder'/page in any of the output XMLs it finds in `saves/'. Within each file, rows represent different subjects (output XML file names) and columns represent different `audioelements'/fragments. + + \subsubsection{score\_plot.py} + Plots the ratings as stored in the CSVs created by score\_parser.py + Depending on the settings, it displays and/or saves (in `saves/ratings/') a boxplot, confidence interval plot, scatter plot, or a combination of the aforementioned. + Requires the free matplotlib library. + At this point, more than one subjects are needed for this script to work. + + \subsubsection{timeline\_view\_movement.py} + Creates a timeline for every subject, for every `audioholder'/page, corresponding with any of the output XML files found in `saves/'. It shows the marker movements of the different fragments, along with when each fragment was played (red regions). Automatically takes fragment names, rating axis title, rating axis labels, and audioholder name from the XML file (if available). + + \subsubsection{timeline\_view.py} % should be omitted or absorbed by the above soon + Creates a timeline for every subject, for every `audioholder'/page, corresponding with any of the output XML files found in `saves/'. It shows when and for how long the subject listened to each of the fragments. + \clearpage \section{Troubleshooting} \label{sec:troubleshooting} - Thanks to feedback from using the interface in experiments by the authors and others, many bugs have been caught and fatal crashes due to the interface (provided it is set up properly by the user) seem to be a thing of the past. + \subsection{Reporting bugs and requesting features} + Thanks to feedback from using the interface in experiments by the authors and others, many bugs have been caught and fatal crashes due to the interface seem to be a thing of the past entirely. - However, if things do go wrong or the test needs to be interrupted for whatever reason, all data is not lost. In a normal scenario, the test needs to be completed until the end (the final `Submit'), at which point the output XML is stored in the \texttt{saves/}. If this stage is not reached, open the JavaScript Console (see below for how to find it) and type + We continually develop this tool to fix issues and implement features useful to us or our user base. See \url{https://code.soundsoftware.ac.uk/projects/webaudioevaluationtool/issues} for a list of feature requests and bug reports, and their status. - \texttt{createProjectSave()} + Please contact the authors if you experience any bugs, if you would like additional functionality, if you spot any errors or gaps in the documentation, if you have questions about using the interface, or if you would like to give any feedback (even positive!) about the interface. We look forward to learning how the tool has (not) been useful to you. - to present the result XML file on the client side, or - \texttt{createProjectSave(specification.projectReturn)} + \subsection{First aid} + Meanwhile, if things do go wrong or the test needs to be interrupted for whatever reason, all data is not lost. In a normal scenario, the test needs to be completed until the end (the final `Submit'), at which point the output XML is stored in the \texttt{saves/}. If this stage is not reached, open the JavaScript Console (see below for how to find it) and type - to try to store it to the specified location, e.g. the `saves/' folder on the web server or the local machine (on failure the result XML should be presented directly in the web browser instead) + \texttt{createProjectSave()} - and hit enter. This will open a pop-up window with a hyperlink that reads `Save File'; click it and an XML file with results until that point should be stored in your download folder. - - Alternatively, a lot of data can be read from the same console, in which the tool prints a lot of debug information. Specifically: - \begin{itemize} - \item the randomisation of pages and fragments are logged; - \item any time a slider is played, its ID and the time stamp (in seconds since the start of the test) are displayed; - \item any time a slider is dragged and dropped, the location where it is dropped including the time stamp are shown; - \item any comments and pre- or post-test questions and their answers are logged as well. - \end{itemize} + to present the result XML file on the client side, or - You can select all this and save into a text file, so that none of this data is lost. You may to choose to do this even when a test was successful as an extra precaution. + \texttt{createProjectSave(specification.projectReturn)} - If you encounter any issue which you believe to be caused by any aspect of the , or which the documentation does not mention, please do let us know! + to try to store it to the specified location, e.g. the `saves/' folder on the web server or the local machine (on failure the result XML should be presented directly in the web browser instead) - \subsection{Opening the JavaScript Console} - \begin{itemize} - \item In Google Chrome, the JavaScript Console can be found in \textbf{View$>$Developer$>$JavaScript Console}, or via the keyboard shortcut Cmd + Alt + J (Mac OS X). - \item In Safari, the JavaScript Console can be found in \textbf{Develop$>$Show Error Console}, or via the keyboard shortcut Cmd + Alt + C (Mac OS X). Note that for the Developer menu to be visible, you have to go to Preferences (Cmd + ,) and enable `Show Develop menu in menu bar' in the `Advanced' tab. \textbf{Note that as long as the Developer menu is not visible, nothing is logged to the console, i.e. you will only be able to see diagnostic information from when you switched on the Developer tools onwards.} - \item In Firefox, go to \textbf{Tools$>$Web Developer$>$Web Console}, or hit Cmd + Alt + K. - \end{itemize} + and hit enter. This will open a pop-up window with a hyperlink that reads `Save File'; click it and an XML file with results until that point should be stored in your download folder. + + Alternatively, a lot of data can be read from the same console, in which the tool prints a lot of debug information. Specifically: + \begin{itemize} + \item the randomisation of pages and fragments are logged; + \item any time a slider is played, its ID and the time stamp (in seconds since the start of the test) are displayed; + \item any time a slider is dragged and dropped, the location where it is dropped including the time stamp are shown; + \item any comments and pre- or post-test questions and their answers are logged as well. + \end{itemize} -\clearpage + You can select all this and save into a text file, so that none of this data is lost. You may to choose to do this even when a test was successful as an extra precaution. -\section{Known issues and limitations} -\label{sec:issues} + If you encounter any issue which you believe to be caused by any aspect of the tool, and/or which the documentation does not mention, please do let us know! - The following is a non-exhaustive list of problems and limitations you may experience using this tool, due to not being supported yet by us, or by the Web Audio API and/or (some) browsers. + \subsubsection*{Opening the JavaScript Console} + \begin{itemize} + \item In Google Chrome, the JavaScript Console can be found in \textbf{View$>$Developer$>$JavaScript Console}, or via the keyboard shortcut Cmd + Alt + J (Mac OS X). + \item In Safari, the JavaScript Console can be found in \textbf{Develop$>$Show Error Console}, or via the keyboard shortcut Cmd + Alt + C (Mac OS X). Note that for the Developer menu to be visible, you have to go to Preferences (Cmd + ,) and enable `Show Develop menu in menu bar' in the `Advanced' tab. \textbf{Note that as long as the Developer menu is not visible, nothing is logged to the console, i.e. you will only be able to see diagnostic information from when you switched on the Developer tools onwards.} + \item In Firefox, go to \textbf{Tools$>$Web Developer$>$Web Console}, or hit Cmd + Alt + K. + \end{itemize} - \begin{itemize} - \item Issue \href{https://code.soundsoftware.ac.uk/issues/1463}{\textbf{\#1463}}: \textbf{Firefox} only supports 8 bit and 16 bit WAV files. Pending automatic requantisation (which deteriorates the audio signal's dynamic range to some extent), WAV format stimuli need to adhere to these limitations in order for the test to be compatible with Firefox. - \item Issues \href{https://code.soundsoftware.ac.uk/issues/1474}{\textbf{\#1474}} and \href{https://code.soundsoftware.ac.uk/issues/1462}{\textbf{\#1462}}: On occasions, audio is not working - or only a continuous `beep' can be heard - notably in \textbf{Safari}. Refreshing, quitting the browser and even enabling Developer tools in Safari's Preferences pane (`Advanced' tab: ``Show `Develop' menu in menu bar'') has helped resolve this. If no (high quality) audio can be heard, make sure your entire playback system's settings are all correct. - \end{itemize} + \subsection{Known issues and limitations} + \label{sec:knownissues} + + The following is a non-exhaustive list of problems and limitations you may experience using this tool, due to not being supported yet by us, or by the Web Audio API and/or (some) browsers. + + \begin{itemize} + \item Issue \href{https://code.soundsoftware.ac.uk/issues/1463}{\textbf{\#1463}}: \textbf{Firefox} only supports 8 bit and 16 bit WAV files. Pending automatic requantisation (which deteriorates the audio signal's dynamic range to some extent), WAV format stimuli need to adhere to these limitations in order for the test to be compatible with Firefox. + \item Issues \href{https://code.soundsoftware.ac.uk/issues/1474}{\textbf{\#1474}} and \href{https://code.soundsoftware.ac.uk/issues/1462}{\textbf{\#1462}}: On occasions, audio is not working - or only a continuous `beep' can be heard - notably in \textbf{Safari}. Refreshing, quitting the browser and even enabling Developer tools in Safari's Preferences pane (`Advanced' tab: ``Show `Develop' menu in menu bar'') has helped resolve this. If no (high quality) audio can be heard, make sure your entire playback system's settings are all correct. + \end{itemize} \clearpage \bibliographystyle{ieeetr} @@ -302,6 +612,11 @@ \clearpage \appendix +\section{Legacy} + The APE interface and most of the functionality of the first WAET editions are inspired by the APE toolbox for MATLAB \cite{ape}. See \url{https://code.soundsoftware.ac.uk/projects/ape} for the source code and \url{http://brechtdeman.com/publications/aes136.pdf} for the corresponding paper. + +\clearpage + \section{Listening test instructions example} Before each test, show the instructions below or similar and make sure it is available to the subject throughout the test. Make sure to ask whether the participant has any questions upon seeing and/or reading the instructions. @@ -326,7 +641,20 @@ \clearpage -\section*{Contact details} +\section{Vocabulary} % just to keep track of what exactly we call things. Don't use terms that are too different, to avoid confusion. + \begin{description} + \item[Subject] The word we use for a participant, user, ... of the test, i.e. not the experimenter who designs the test but the person who evaluates the audio under test as part of an experiment (or the preparation of one). + \item[Page] A screen in a test; corresponds with an \texttt{audioholder} + \item[Fragment] An element or sample in a test; corresponds with an \texttt{audioelement} + \item[Test] A complete test which can consist of several pages; corresponds with an entire configuration XML file + \item[Configuration XML file] The XML file containing the necessary information on interface, samples, survey questions, configurations, ... which the JavaScript modules read to produce the desired test. + \item[Results XML file] The output of a successful test, including ratings, comments, survey responses, timing information, and the complete configuration XML file with which the test was generated in the first place. + \end{description} + +\clearpage + +\setcounter{secnumdepth}{0} % don't number this last bit +\section{Contact details} % maybe add web pages, Twitter accounts, whatever you like \label{sec:contact} \begin{itemize}
--- a/scripts/generate_report.py Thu Dec 17 11:11:57 2015 +0000 +++ b/scripts/generate_report.py Fri Dec 18 01:45:54 2015 +0000 @@ -11,7 +11,7 @@ import numpy as np # numbers # Command line arguments -assert len(sys.argv)<4, "evaluation_stats takes at most 2 command line argument\n"+\ +assert len(sys.argv)<4, "generate_report takes at most 2 command line arguments\n"+\ "Use: python generate_report.py [results_folder] [no_render | -nr]" render_figures = True
--- a/scripts/score_parser.py Thu Dec 17 11:11:57 2015 +0000 +++ b/scripts/score_parser.py Fri Dec 18 01:45:54 2015 +0000 @@ -97,7 +97,7 @@ filewriter.writerow(row + ['']*len(newfragments)) os.rename('temp.csv', file_name) # replace old file with temp file headerrow = headerrow + newfragments - + # if not, create file and make header else: headerrow = sorted(fragmentnamelist) # sort alphabetically