Mercurial > hg > webaudioevaluationtool
changeset 727:3b637867eafe
Minor edits to paper, updated installation instructions, added to be ignored files (e.g. LaTeX auxiliary files)
| author | Brecht De Man <BrechtDeMan@users.noreply.github.com> |
|---|---|
| date | Wed, 30 Sep 2015 18:34:40 +0200 |
| parents | 949185ac04a9 |
| children | 28b385057ded |
| files | .hgignore docs/Instructions/ListeningTestInstructions.bib docs/Instructions/ListeningTestInstructions.pdf docs/Instructions/ListeningTestInstructions.tex docs/Instructions/img/cmd.png docs/Instructions/img/python.png docs/Instructions/img/pythonServer.png docs/Instructions/img/test.png docs/Instructions/img/warning.png docs/WAC2016/WAC2016.pdf docs/WAC2016/WAC2016.tex |
| diffstat | 11 files changed, 162 insertions(+), 101 deletions(-) [+] |
line wrap: on
line diff
--- a/.hgignore Tue Sep 29 23:55:10 2015 +0200 +++ b/.hgignore Wed Sep 30 18:34:40 2015 +0200 @@ -37,3 +37,9 @@ re:^docs/WAC2016/\._WAC2016\.bib$ re:^docs/WAC2016/\._WAC2016\.pdf$ re:^docs/WAC2016/\._WAC2016\.tex$ +*.out +*.fdb_latexmk +*.toc +subject +resample +test-data \ No newline at end of file
--- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/docs/Instructions/ListeningTestInstructions.bib Wed Sep 30 18:34:40 2015 +0200 @@ -0,0 +1,31 @@ +%% This BibTeX bibliography file was created using BibDesk. +%% http://bibdesk.sourceforge.net/ + +%% Created for Brecht De Man at 2015-09-30 17:44:12 +0200 + + +%% Saved with string encoding Unicode (UTF-8) + + + +@conference{ape, + Author = {De Man, Brecht and Joshua D. Reiss}, + Booktitle = {136th Convention of the Audio Engineering Society}, + Date-Added = {2015-09-29 17:07:16 +0000}, + Date-Modified = {2015-09-29 17:07:20 +0000}, + Keywords = {perceptual evaluation}, + Month = {April}, + Read = {1}, + Title = {{APE}: {A}udio {P}erceptual {E}valuation toolbox for {MATLAB}}, + Year = {2014}, + Bdsk-File-1 = {YnBsaXN0MDDUAQIDBAUGJCVYJHZlcnNpb25YJG9iamVjdHNZJGFyY2hpdmVyVCR0b3ASAAGGoKgHCBMUFRYaIVUkbnVsbNMJCgsMDxJXTlMua2V5c1pOUy5vYmplY3RzViRjbGFzc6INDoACgAOiEBGABIAFgAdccmVsYXRpdmVQYXRoWWFsaWFzRGF0YV8QOi4uLy4uLy4uLy4uL0dvb2dsZSBEcml2ZS9Xcml0aW5ncy9fcHVibGljYXRpb25zL2FlczEzNi5wZGbSFwsYGVdOUy5kYXRhTxEBsgAAAAABsgACAAAMTWFjaW50b3NoIEhEAAAAAAAAAAAAAAAAAAAA0Fxdh0grAAAACl8UCmFlczEzNi5wZGYAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAKaS7PXHsUAAAAAAAAAAAABAAEAAAJIAAAAAAAAAAAAAAAAAAAAA1fcHVibGljYXRpb25zAAAQAAgAANBcQWcAAAARAAgAAM9cbQQAAAABABQACl8UAApeugAKXQIACUReAAKT1QACAE1NYWNpbnRvc2ggSEQ6VXNlcnM6AEJyZWNodDoAR29vZ2xlIERyaXZlOgBXcml0aW5nczoAX3B1YmxpY2F0aW9uczoAYWVzMTM2LnBkZgAADgAWAAoAYQBlAHMAMQAzADYALgBwAGQAZgAPABoADABNAGEAYwBpAG4AdABvAHMAaAAgAEgARAASADtVc2Vycy9CcmVjaHQvR29vZ2xlIERyaXZlL1dyaXRpbmdzL19wdWJsaWNhdGlvbnMvYWVzMTM2LnBkZgAAEwABLwAAFQACAA3//wAAgAbSGxwdHlokY2xhc3NuYW1lWCRjbGFzc2VzXU5TTXV0YWJsZURhdGGjHR8gVk5TRGF0YVhOU09iamVjdNIbHCIjXE5TRGljdGlvbmFyeaIiIF8QD05TS2V5ZWRBcmNoaXZlctEmJ1Ryb290gAEACAARABoAIwAtADIANwBAAEYATQBVAGAAZwBqAGwAbgBxAHMAdQB3AIQAjgDLANAA2AKOApAClQKgAqkCtwK7AsICywLQAt0C4ALyAvUC+gAAAAAAAAIBAAAAAAAAACgAAAAAAAAAAAAAAAAAAAL8}} + +@conference{waet, + Author = {Nicholas Jillings and David Moffat and De Man, Brecht and Joshua D. Reiss}, + Booktitle = {12th Sound and Music Computing Conference}, + Date-Added = {2015-09-22 16:48:27 +0000}, + Date-Modified = {2015-09-22 16:48:33 +0000}, + Month = {July}, + Read = {1}, + Title = {Web {A}udio {E}valuation {T}ool: {A} browser-based listening test environment}, + Year = {2015}}
--- a/docs/Instructions/ListeningTestInstructions.tex Tue Sep 29 23:55:10 2015 +0200 +++ b/docs/Instructions/ListeningTestInstructions.tex Wed Sep 30 18:34:40 2015 +0200 @@ -10,6 +10,7 @@ \usepackage{amssymb} \usepackage{cite} \usepackage{hyperref} % Hyperlinks +\usepackage[nottoc,numbib]{tocbibind} % 'References' in TOC \graphicspath{{img/}} % Relative path where the images are stored. @@ -20,50 +21,57 @@ \begin{document} \maketitle -These instructions are about use of the Web Audio Evaluation Tool \cite{deman2015c} with the APE interface \cite{deman2014b} on Windows and Mac OS X platforms. +These instructions are about use of the Web Audio Evaluation Tool \cite{waet} with the APE interface \cite{ape} on Windows and Mac OS X platforms. % TO DO: Linux \tableofcontents - +\clearpage \section{Installation and set up} - Download the folder and unzip in a location of your choice. + Download the folder (\url{https://code.soundsoftware.ac.uk/hg/webaudioevaluationtool/archive/tip.zip}) and unzip in a location of your choice. \subsection{Contents} The folder should contain the following elements: \\ \textbf{Main folder:} - \begin{itemize} - \item \texttt{ape.css, core.css, graphics.css}, structure.css: style files (edit to change appearance) - \item \texttt{ape.js}: JavaScript file for APE-style interface \cite{deman2014b} - \item \texttt{core.js}: JavaScript file with core functionality - \item \texttt{index.html}: webpage where interface should appear - \item \texttt{jquery-2.1.4.js}: jQuery JavaScript Library - \item \texttt{pythonServer.py}: webserver for running tests locally - \item \texttt{pythonServer-legacy.py}: webserver with limited functionality (no automatic storing of output XML files)\\ - \end{itemize} - \textbf{Documentation (/docs/)} - \begin{itemize} - \item Project Specification Document (\LaTeX/PDF) - \item Results Specification Document (\LaTeX/PDF) - \item SMC15: PDF and \LaTeX source of corresponding SMC2015 publication \cite{deman2015c}\\ - \end{itemize} - \textbf{Example project (/example\_eval/)} - \begin{itemize} - \item An example of what the set up XML should look like, with example audio files 0.wav-10.wav which are short recordings at 44.1kHz, 16bit of a woman saying the corresponding number (useful for testing randomisation and general familiarisation with the interface).\\ \end{itemize} - \textbf{Output files (/saves/)} - \begin{itemize} - \item The output XML files of tests will be stored here by default by the \texttt{pythonServer.py} script.\\ \end{itemize} - \textbf{Auxiliary scripts (/scripts/)} - \begin{itemize} - \item Helpful Python scripts for extraction and visualisation of data.\\ \end{itemize} - \textbf{Test creation tool (/test\_create/)} - \begin{itemize} - \item Webpage for easily setting up your own test without having to delve into the XML.\\ \end{itemize} + \begin{itemize} + \item \texttt{ape.css, core.css, graphics.css, structure.css}: style files (edit to change appearance) + \item \texttt{ape.js}: JavaScript file for APE-style interface \cite{ape} + \item \texttt{CITING.txt, LICENSE.txt, README.txt}: text files with, respectively, the citation which we ask to include in any work where this tool or any portion thereof is used, modified or otherwise; the license under which the software is shared; and a general readme file. + \item \texttt{core.js}: JavaScript file with core functionality + \item \texttt{index.html}: webpage where interface should appear + \item \texttt{jquery-2.1.4.js}: jQuery JavaScript Library + \item \texttt{pythonServer.py}: webserver for running tests locally + \item \texttt{pythonServer-legacy.py}: webserver with limited functionality (no automatic storing of output XML files)\\ + \end{itemize} + \textbf{Documentation (./docs/)} + \begin{itemize} + \item Instructions: PDF and \LaTeX source of these instructions + \item Project Specification Document (\LaTeX/PDF) + \item Results Specification Document (\LaTeX/PDF) + \item SMC15: PDF and \LaTeX source of corresponding SMC2015 publication \cite{waet} + \item WAC2016: PDF and \LaTeX source of corresponding WAC2016 publication\\ + \end{itemize} + \textbf{Example project (./example\_eval/)} + \begin{itemize} + \item An example of what the set up XML should look like, with example audio files 0.wav-10.wav which are short recordings at 44.1kHz, 16bit of a woman saying the corresponding number (useful for testing randomisation and general familiarisation with the interface).\\ + \end{itemize} + \textbf{Output files (./saves/)} + \begin{itemize} + \item The output XML files of tests will be stored here by default by the \texttt{pythonServer.py} script.\\ + \end{itemize} + \textbf{Auxiliary scripts (./scripts/)} + \begin{itemize} + \item Helpful Python scripts for extraction and visualisation of data.\\ + \end{itemize} + \textbf{Test creation tool (./test\_create/)} + \begin{itemize} + \item Webpage for easily setting up your own test without having to delve into the XML.\\ + \end{itemize} \subsection{Browser} - As Microsoft Internet Explorer doesn't support the Web Audio API \footnote{\url{http://caniuse.com/\#feat=audio-api}}, you will need another browser like Google Chrome, Safari or Firefox (all three are tested and confirmed to work). + As Microsoft Internet Explorer doesn't support the Web Audio API\footnote{\url{http://caniuse.com/\#feat=audio-api}}, you will need another browser like Google Chrome, Safari or Firefox (all three are tested and confirmed to work). The tool is platform-independent and works in any browser that supports the Web Audio API. It does not require any specific, proprietary software. However, in case the tool is hosted locally (i.e. you are not hosting it on an actual webserver) you will need Python, which is a free programming language - see the next paragraph. @@ -72,71 +80,76 @@ On Mac OS X, Python comes preinstalled. +\clearpage -\section{Listening test} +\section{Listening test: Local} \subsection{Start local webserver} If the test is hosted locally, you will need to run the local webserver provided with this tool. + \subsubsection{Mac OS X} + Open the Terminal (find it in \textbf{Applications/Terminal} or via Spotlight), and go to the folder you downloaded. To do this, type \texttt{cd [folder]}, where \texttt{[folder]} is the folder where to find the \texttt{pythonServer.py} script you downloaded. For instance, if the location is \texttt{/Users/John/Documents/test/}, then type + + \texttt{cd /Users/John/Documents/test/} + + Then hit enter and run the Python script by typing + + \texttt{python pythonServer.py} + + and hit enter again. See also Figure \ref{fig:terminal}. + + \begin{figure}[htbp] + \begin{center} + \includegraphics[width=.75\textwidth]{pythonServer.png} + \caption{Mac OS X: The Terminal window after going to the right folder (\texttt{cd [folder\_path]}) and running \texttt{pythonServer.py}.} + \label{fig:terminal} + \end{center} + \end{figure} + + Alternatively, you can simply type \texttt{python} (follwed by a space) and drag the file into the Terminal window from Finder. % DOESN'T WORK YET + + You can leave this running throughout the different experiments (i.e. leave the Terminal open). + \subsubsection{Windows} - Simply double click the Python script \texttt{pythonServer.py} in the folder you downloaded. + Simply double click the Python script \texttt{pythonServer.py} in the folder you downloaded. + + You may see a warning like the one in Figure \ref{fig:warning}. Click `Allow access'. + + \begin{figure}[htbp] + \begin{center} + \includegraphics[width=.6\textwidth]{warning.png} + \caption{Windows: Potential warning message when executing \texttt{pythonServer.py}.} + \label{fig:warning} + \end{center} + \end{figure} + + The process should now start, in the Command prompt that opens - see Figure \ref{fig:python}. + + \begin{figure}[htbp] + \begin{center} + \includegraphics[width=.75\textwidth]{python.png} + \caption{Windows: The Command Prompt after running \texttt{pythonServer.py} and opening the corresponding website.} + \label{fig:python} + \end{center} + \end{figure} + + You can leave this running throughout the different experiments (i.e. leave the Command Prompt open). - You may see a warning like the one in Figure \ref{fig:warning}. Click `Allow access'. - \begin{figure}[htbp] - \begin{center} - \includegraphics[width=.6\textwidth]{warning.png} - \caption{Windows: Potential warning message when executing \texttt{pythonServer.py}.} - \label{fig:warning} - \end{center} - \end{figure} - - The process should now start, in the Command prompt that opens - see Figure \ref{fig:python}. - - \begin{figure}[htbp] - \begin{center} - \includegraphics[width=.75\textwidth]{python.png} - \caption{Windows: The Command Prompt after running \texttt{pythonServer.py} and opening the corresponding website.} - \label{fig:python} - \end{center} - \end{figure} - - You can leave this running throughout the different experiments (i.e. leave the Command Prompt open). - +\clearpage + \subsection{Sample rate} + Depending on how the experiment is set up, audio is resampled automatically (the Web Audio default) or the sample rate is enforced. In the latter case, you will need to make sure that the sample rate of the system is equal to the sample rate of these audio files. For this reason, all audio files in the experiment will have to have the same sample rate. + + Always make sure that all other digital equipment in the playback chain (clock, audio interface, digital-to-analog converter, ...) is set to this same sample rate. \subsubsection{Mac OS X} - Open the Terminal (find it in \textbf{Applications/Terminal} or via Spotlight), and go to the folder you downloaded. To do this, type \texttt{cd [folder]}, where \texttt{[folder]} is the folder where to find the \texttt{pythonServer.py} script you downloaded. For instance, if the location is \texttt{/Users/John/Documents/test/}, then type + To change the sample rate in Mac OS X, go to \textbf{Applications/Utilities/Audio MIDI Setup} or find this application with Spotlight. Then select the output of the audio interface you are using and change the `Format' to the appropriate number. Also make sure the bit depth and channel count are as desired. + If you are using an external audio interface, you may have to go to the preference pane of that device to change the sample rate. - \texttt{cd /Users/John/Documents/test/} - - Then hit enter and run the Python script by typing - - \texttt{python pythonServer.py} - - and hit enter again. See also Figure \ref{fig:terminal}. - - \begin{figure}[htbp] - \begin{center} - \includegraphics[width=.75\textwidth]{pythonServer.png} - \caption{Mac OS X: Potential warning message when executing \texttt{pythonServer.py}.} - \label{fig:terminal} - \end{center} - \end{figure} - - Alternatively, you can simply type \texttt{python} (follwed by a space) and drag the file into the Terminal window from Finder. % DOESN'T WORK YET - - You can leave this running throughout the different experiments (i.e. leave the Terminal open). - - - \subsection{Sample rate} - Depending on how the experiment is set up, audio is resampled automatically (the Web Audio default) or the sample rate is enforced. In the latter case, you will need to make sure that the sample rate of the system is equal to the sample rate of these audio files. For this reason, all audio files in the experiment will have to have the same sample rate. - - To change the sample rate in Mac OS X, go to \textbf{Applications/Utilities/Audio MIDI Setup} or find this application with Spotlight. Then select the output of the audio interface you are using and change the `Format' to the appropriate number. Also make sure the bit depth and channel count are as desired. - If you are using an external audio interface, you may have to go to the preference pane of that device to change the sample rate. - - To change the sample rate in Windows, right-click on the speaker icon in the lower-right corner of your desktop and choose `Playback devices'. Right-click the appropriate playback device and click `Properties'. Click the `Advanced' tab and verify or change the sample rate under `Default Format'. % NEEDS CONFIRMATION - - Always make sure that all other digital equipment in the playback chain (clock, audio interface, digital-to-analog converter, ...) is set to this same sample rate. + \subsubsection{Windows} + To change the sample rate in Windows, right-click on the speaker icon in the lower-right corner of your desktop and choose `Playback devices'. Right-click the appropriate playback device and click `Properties'. Click the `Advanced' tab and verify or change the sample rate under `Default Format'. % NEEDS CONFIRMATION + If you are using an external audio interface, you may have to go to the preference pane of that device to change the sample rate. + \subsection{Setting up a participant} @@ -178,7 +191,7 @@ \end{center} \end{figure} - If at any point in the test the participant reports weird behaviour or an error of some kind, or the test needs to be interrupted, please notify the experimenter and/or refer to Section \ref{sec:troubleshooting}. + If at any point in the test the participant reports weird behaviour or an error of some kind, or the test needs to be interrupted, please notify the experimenter and/or refer to Section \ref{sec:troubleshooting}. When the test is over (the subject should see a message to that effect, and click `Submit' one last time), the output XML file containing all collected data should have appeared in `saves/'. The names of these files are `test-0.xml', `test-1.xml', etc., in ascending order. The Terminal or Command prompt running the local web server will display the following file name. If such a file did not appear, please again refer to Section \ref{sec:troubleshooting}. @@ -190,7 +203,7 @@ \subsubsection{Survey} The tool allows for embedded questions before and after each page, and before and after the whole test. If these do \underline{not} include survey questions (about the participant's background, demographic information, and so on) make sure to ask the participant to complete the survey immediately after the test. Above anything else, this decreases the likelihood that the survey goes forgotten and the experimenters do not receive the data in time. - +\clearpage \subsection{Troubleshooting} \label{sec:troubleshooting} Thanks to feedback from using the interface in experiments by the authors and others, many bugs have been caught and fatal crashes due to the interface (provided it is set up properly by the user) seem to be a thing of the past. However, if things do go wrong or the test needs to be interrupted for whatever reason, all data is not lost. In a normal scenario, the test needs to be completed until the end (the final `Submit'), at which point the output XML is stored in the \texttt{saves/}. If this stage is not reached, open the JavaScript Console (see below for how to find it) and type @@ -198,23 +211,31 @@ \texttt{createProjectSave()} and hit enter. This will open a pop-up window with a hyperlink that reads `Save File'; click it and an XML file with results until that point should be stored in your download folder. + Alternatively, a lot of data can be read from the same console, in which the tool prints a lot of debug information. Specifically: - \begin{itemize} + \begin{itemize} \item the randomisation of pages and fragments are logged; \item any time a slider is played, its ID and the time stamp (in seconds since the start of the test) are displayed; \item any time a slider is dragged and dropped, the location where it is dropped including the time stamp are shown; \item any comments and pre- or post-test questions and their answers are logged as well. - \end{itemize} + \end{itemize} You can select all this and save into a text file, so that none of this data is lost. You may to choose to do this even when a test was successful as an extra precaution. \subsubsection{Opening the JavaScript Console} - \begin{itemize} - \item In Google Chrome, the JavaScript Console can be found in \textbf{View$>$Developer$>$JavaScript Console}, or via the keyboard shortcut Cmd + Alt + J (Mac OS X). - \item In Safari, the JavaScript Console can be found in \textbf{Develop$>$Show Error Console}, or via the keyboard shortcut Cmd + Alt + C (Mac OS X). Note that for the Developer menu to be visible, you have to go to Preferences (Cmd + ,) and enable `Show Develop menu in menu bar' in the `Advanced' tab. - \item In Firefox, go to \textbf{Tools$>$Web Developer$>$Web Console}, or hit Cmd + Alt + K. - \end{itemize} + \begin{itemize} + \item In Google Chrome, the JavaScript Console can be found in \textbf{View$>$Developer$>$JavaScript Console}, or via the keyboard shortcut Cmd + Alt + J (Mac OS X). + \item In Safari, the JavaScript Console can be found in \textbf{Develop$>$Show Error Console}, or via the keyboard shortcut Cmd + Alt + C (Mac OS X). Note that for the Developer menu to be visible, you have to go to Preferences (Cmd + ,) and enable `Show Develop menu in menu bar' in the `Advanced' tab. + \item In Firefox, go to \textbf{Tools$>$Web Developer$>$Web Console}, or hit Cmd + Alt + K. + \end{itemize} -% TO DO: add bibliography here +\clearpage +\section{Listening test: remote} + + (TBA) + +\clearpage +\bibliographystyle{ieeetr} +\bibliography{ListeningTestInstructions}{} \end{document} \ No newline at end of file
--- a/docs/WAC2016/WAC2016.tex Tue Sep 29 23:55:10 2015 +0200 +++ b/docs/WAC2016/WAC2016.tex Wed Sep 30 18:34:40 2015 +0200 @@ -1,5 +1,6 @@ \documentclass{sig-alternate} -\usepackage{hyperref} +\usepackage{hyperref} % make links (like references, links to Sections, ...) clickable +\usepackage{enumitem} % tighten itemize etc by appending '[noitemsep,nolistsep]' \begin{document} @@ -132,6 +133,8 @@ Perceptual evaluation of audio, in the form of listening tests, is a powerful way to assess anything from audio codec quality over realism of sound synthesis to the performance of source separation, automated music production and In less technical areas, the framework of a listening test can be used to measure emotional response to music or test cognitive abilities. % maybe some references? If there's space. + % check out http://link.springer.com/article/10.1007/s10055-015-0270-8 - only paper that cited WAC15 paper + % Why difficult? Challenges? What constitutes a good interface? Technical, interfaces, user friendliness, reliability @@ -192,12 +195,12 @@ Furthermore, a fully browser-based test, where the collection of the results is automatic, is more efficient and technically reliable even when the test still takes place under lab conditions. The following features allow easy and effective remote testing: - \begin{itemize} + \begin{itemize}[noitemsep,nolistsep] \item PHP script to collect result XML files \item Randomly pick specified number of audioholders \item Calibration \item Functionality to participate multiple times - \begin{itemize} + \begin{itemize}[noitemsep,nolistsep] \item Possible to log in with unique ID (no password) \item Pick `new user' (need new, unique ID) or `already participated' (need already available ID) \item Store XML on server with IDs plus which audioholders have already been listened to @@ -220,7 +223,7 @@ We could add more interfaces, such as: - \begin{itemize} + \begin{itemize}[noitemsep,nolistsep] \item (APE style) \cite{ape} \item Multi attribute ratings \item MUSHRA (ITU-R BS. 1534)~\cite{recommendation20031534} @@ -266,7 +269,7 @@ \end{itemize} There are also the following interfaces, which would require a slightly different `engine' underneath, e.g. loading a different page for every possible pair. - \begin{itemize} + \begin{itemize}[noitemsep,nolistsep] \item AB Test~\cite{lipshitz1981great} \item ABX Test~\cite{clark1982high} \item JND @@ -280,7 +283,7 @@ The following could be nice: - \begin{itemize} + \begin{itemize}[noitemsep,nolistsep] \item Web page showing all audioholder IDs, file names, subject IDs, audio element IDs, ... in the collected XMLs so far (\texttt{saves/*.xml}) \item Check/uncheck each of the above for analysis (e.g. zoom in on a certain song, or exclude a subset of subjects) \item Click a mix to hear it (follow path in XML setup file, which is also embedded in the XML result file)