b@378: \documentclass[11pt, oneside]{article} % use "amsart" instead of "article" for AMSLaTeX format b@378: \usepackage{geometry} % See geometry.pdf to learn the layout options. There are lots. b@378: \geometry{letterpaper} % ... or a4paper or a5paper or ... b@378: %\geometry{landscape} % Activate for rotated page geometry b@378: \usepackage[parfill]{parskip} % Activate to begin paragraphs with an empty line rather than an indent b@378: \usepackage{graphicx} % Use pdf, png, jpg, or eps§ with pdflatex; use eps in DVI mode b@378: % TeX will automatically convert eps --> pdf in pdflatex b@378: b@378: \usepackage{listings} % Source code b@413: \usepackage{xcolor} % colour (source code for instance) b@413: \definecolor{grey}{rgb}{0.1,0.1,0.1} b@413: \definecolor{darkblue}{rgb}{0.0,0.0,0.6} b@413: \definecolor{cyan}{rgb}{0.0,0.6,0.6} b@413: b@378: \usepackage{amssymb} b@378: \usepackage{cite} b@378: \usepackage{hyperref} % Hyperlinks b@378: \usepackage[nottoc,numbib]{tocbibind} % 'References' in TOC b@378: b@378: \graphicspath{{img/}} % Relative path where the images are stored. b@378: b@378: \title{Instructions for \\ Web Audio Evaluation Tool} b@378: \author{Nicholas Jillings, Brecht De Man and David Moffat} b@378: \date{7 December 2015} % Activate to display a given date or no date b@378: b@378: \begin{document} b@378: \maketitle b@378: b@413: These instructions are about use of the Web Audio Evaluation Tool on Windows and Mac OS X platforms. b@413: b@413: We request that you acknowledge the authors and cite our work when using it \cite{waet}, see also CITING.txt. b@413: b@413: The tool is available in its entirety including source code on \url{https://code.soundsoftware.ac.uk/projects/webaudioevaluationtool/}, under the GNU General Public License v3.0 (\url{http://choosealicense.com/licenses/gpl-3.0/}), see also LICENSE.txt. b@413: b@429: % TO DO: Linux (Android, iOS) b@378: b@378: \tableofcontents b@378: b@378: \clearpage b@378: b@378: \section{Installation} b@413: Download the folder (\url{https://code.soundsoftware.ac.uk/hg/webaudioevaluationtool/archive/tip.zip}) and unzip in a location of your choice, or pull the source code from \url{https://code.soundsoftware.ac.uk/hg/webaudioevaluationtool} (Mercurial). b@378: b@378: \subsection{Contents} b@378: The folder should contain the following elements: \\ b@378: b@378: \textbf{Main folder:} b@378: \begin{itemize} b@378: \item \texttt{analyse.html}: analysis and diagnostics of a set of result XML files b@378: \item \texttt{ape.css, core.css, graphics.css, mushra.css, structure.css}: style files (edit to change appearance) b@378: \item \texttt{ape.js}: JavaScript file for APE-style interface \cite{ape} b@413: \item \texttt{CITING.txt, LICENSE.txt, README.txt}: text files with, respectively, the citation which we ask to include in any work where this tool or any portion thereof is used, modified or otherwise; the license under which the software is shared; and a general readme file referring to these instructions. b@378: \item \texttt{core.js}: JavaScript file with core functionality b@378: \item \texttt{index.html}: webpage where interface should appear (includes link to test configuration XML) b@378: \item \texttt{jquery-2.1.4.js}: jQuery JavaScript Library b@413: \item \texttt{loudness.js}: Allows for automatic calculation of loudness of Web Audio API Buffer objects, return gain values to correct for a target loudness or match loudness between multiple objects b@413: \item \texttt{mushra.js}: JavaScript file for MUSHRA-style interface \cite{mushra} b@378: \item \texttt{pythonServer.py}: webserver for running tests locally b@378: \item \texttt{pythonServer-legacy.py}: webserver with limited functionality (no automatic storing of output XML files) b@378: \item \texttt{save.php}: PHP script to store result XML files to web server\\ b@378: \end{itemize} b@378: \textbf{Documentation (./docs/)} b@378: \begin{itemize} b@413: \item \href{http://c4dm.eecs.qmul.ac.uk/dmrn/events/dmrnp10/#posters}{DMRN+10}: PDF and \LaTeX source of poster for 10\textsuperscript{th} Digital Music Research Network One-Day workshop (``soft launch'') b@378: \item Instructions: PDF and \LaTeX source of these instructions b@378: \item Project Specification Document (\LaTeX/PDF) b@378: \item Results Specification Document (\LaTeX/PDF) b@413: \item SMC15: PDF and \LaTeX source of 12th Sound and Music Computing Conference paper \cite{waet} b@413: \item WAC2016: PDF and \LaTeX source of 2nd Web Audio Conference paper\\ b@378: \end{itemize} b@378: \textbf{Example project (./example\_eval/)} b@378: \begin{itemize} b@378: \item An example of what the set up XML should look like, with example audio files 0.wav-10.wav which are short recordings at 44.1kHz, 16bit of a woman saying the corresponding number (useful for testing randomisation and general familiarisation with the interface).\\ b@378: \end{itemize} b@378: \textbf{Output files (./saves/)} b@378: \begin{itemize} b@378: \item The output XML files of tests will be stored here by default by the \texttt{pythonServer.py} script.\\ b@378: \end{itemize} b@378: \textbf{Auxiliary scripts (./scripts/)} b@378: \begin{itemize} b@378: \item Helpful Python scripts for extraction and visualisation of data.\\ b@378: \end{itemize} b@378: \textbf{Test creation tool (./test\_create/)} b@378: \begin{itemize} b@378: \item Webpage for easily setting up your own test without having to delve into the XML.\\ b@378: \end{itemize} b@378: b@413: \subsection{Compatibility} b@378: As Microsoft Internet Explorer doesn't support the Web Audio API\footnote{\url{http://caniuse.com/\#feat=audio-api}}, you will need another browser like Google Chrome, Safari or Firefox (all three are tested and confirmed to work). b@413: b@413: Firefox does not currently support other bit depths than 8 or 16 bit for PCM wave files. In the future, this will throw a warning message to tell the user that their content is being quantised automatically. %Nick? Right? To be removed if and when actually implemented b@378: b@378: The tool is platform-independent and works in any browser that supports the Web Audio API. It does not require any specific, proprietary software. However, in case the tool is hosted locally (i.e. you are not hosting it on an actual webserver) you will need Python (2.7), which is a free programming language - see the next paragraph. b@378: b@413: \clearpage b@378: b@378: b@378: \section{Test setup} b@378: b@378: \subsection{Sample rate} b@378: Depending on how the experiment is set up, audio is resampled automatically (the Web Audio default) or the sample rate is enforced. In the latter case, you will need to make sure that the sample rate of the system is equal to the sample rate of these audio files. For this reason, all audio files in the experiment will have to have the same sample rate. b@378: b@378: Always make sure that all other digital equipment in the playback chain (clock, audio interface, digital-to-analog converter, ...) is set to this same sample rate. b@378: b@378: Note that upon changing the sampling rate, the browser will have to be restarted for the change to take effect. b@378: b@378: \subsubsection{Mac OS X} b@378: To change the sample rate in Mac OS X, go to \textbf{Applications/Utilities/Audio MIDI Setup} or find this application with Spotlight (see Figure \ref{fig:audiomidisetup}). Then select the output of the audio interface you are using and change the `Format' to the appropriate number. Also make sure the bit depth and channel count are as desired. b@378: If you are using an external audio interface, you may have to go to the preference pane of that device to change the sample rate. b@378: b@378: Also make sure left and right channel gains are equal, as some applications alter this without changing it back, leading to a predominantly louder left or right channel. See Figure \ref{fig:audiomidisetup} for an example where the channel gains are different. b@378: b@378: \begin{figure}[tb] b@378: \centering b@378: \includegraphics[width=.65\textwidth]{img/audiomidisetup.png} b@378: \caption{The Audio MIDI Setup window in Mac OS X} b@378: \label{fig:audiomidisetup} b@378: \end{figure} b@378: b@378: \subsubsection{Windows} b@378: To change the sample rate in Windows, right-click on the speaker icon in the lower-right corner of your desktop and choose `Playback devices'. Right-click the appropriate playback device and click `Properties'. Click the `Advanced' tab and verify or change the sample rate under `Default Format'. % NEEDS CONFIRMATION b@378: If you are using an external audio interface, you may have to go to the preference pane of that device to change the sample rate. b@378: b@378: \subsection{Local test} b@378: If the test is hosted locally, you will need to run the local webserver provided with this tool. b@378: nicholas@434: \subsubsection{Mac OS X \& Linux} b@378: nicholas@434: On Mac OS X, Python comes preinstalled, as with most Unix/Linux distributions. b@378: b@378: Open the Terminal (find it in \textbf{Applications/Terminal} or via Spotlight), and go to the folder you downloaded. To do this, type \texttt{cd [folder]}, where \texttt{[folder]} is the folder where to find the \texttt{pythonServer.py} script you downloaded. For instance, if the location is \texttt{/Users/John/Documents/test/}, then type b@378: b@378: \texttt{cd /Users/John/Documents/test/} b@378: b@378: Then hit enter and run the Python script by typing b@378: b@378: \texttt{python pythonServer.py} b@378: b@378: and hit enter again. See also Figure \ref{fig:terminal}. b@378: b@378: \begin{figure}[htbp] b@378: \begin{center} b@378: \includegraphics[width=.75\textwidth]{pythonServer.png} b@378: \caption{Mac OS X: The Terminal window after going to the right folder (\texttt{cd [folder\_path]}) and running \texttt{pythonServer.py}.} b@378: \label{fig:terminal} b@378: \end{center} b@378: \end{figure} b@378: b@378: Alternatively, you can simply type \texttt{python} (follwed by a space) and drag the file into the Terminal window from Finder. % DOESN'T WORK YET b@378: nicholas@434: You can leave this running throughout the different experiments (i.e. leave the Terminal open). Once running the terminal will report the current URL to type into your browser to initiate the test, usually this is http://localhost:8000/. b@378: b@378: To start the test, open the browser and type b@378: b@378: \texttt{localhost:8000} b@378: b@378: and hit enter. The test should start (see Figure \ref{fig:test}). b@378: b@378: To quit the server, either close the terminal window or press Ctrl+C on your keyboard to forcibly shut the server. b@378: b@378: \subsubsection{Windows} b@378: b@378: On Windows, Python 2.7 is not generally preinstalled and therefore has to be downloaded\footnote{\url{https://www.python.org/downloads/windows/}} and installed to be able to run scripts such as the local webserver, necessary if the tool is hosted locally. b@378: b@378: Simply double click the Python script \texttt{pythonServer.py} in the folder you downloaded. b@378: b@378: You may see a warning like the one in Figure \ref{fig:warning}. Click `Allow access'. b@378: b@378: \begin{figure}[htbp] b@378: \begin{center} b@378: \includegraphics[width=.6\textwidth]{warning.png} b@378: \caption{Windows: Potential warning message when executing \texttt{pythonServer.py}.} b@378: \label{fig:warning} b@378: \end{center} b@378: \end{figure} b@378: b@378: The process should now start, in the Command prompt that opens - see Figure \ref{fig:python}. b@378: b@378: \begin{figure}[htbp] b@378: \begin{center} b@378: \includegraphics[width=.75\textwidth]{python.png} b@378: \caption{Windows: The Command Prompt after running \texttt{pythonServer.py} and opening the corresponding website.} b@378: \label{fig:python} b@378: \end{center} b@378: \end{figure} b@378: b@378: You can leave this running throughout the different experiments (i.e. leave the Command Prompt open). b@378: b@378: To start the test, open the browser and type b@378: b@378: \texttt{localhost:8000} b@378: b@378: and hit enter. The test should start (see Figure \ref{fig:test}). b@378: b@378: \begin{figure}[htb] b@378: \begin{center} b@378: \includegraphics[width=.8\textwidth]{test.png} b@378: \caption{The start of the test in Google Chrome on Windows 7.} b@378: \label{fig:test} b@378: \end{center} b@378: \end{figure} b@378: b@378: If at any point in the test the participant reports weird behaviour or an error of some kind, or the test needs to be interrupted, please notify the experimenter and/or refer to Section \ref{sec:troubleshooting}. b@378: b@378: When the test is over (the subject should see a message to that effect, and click `Submit' one last time), the output XML file containing all collected data should have appeared in `saves/'. The names of these files are `test-0.xml', `test-1.xml', etc., in ascending order. The Terminal or Command prompt running the local web server will display the following file name. If such a file did not appear, please again refer to Section \ref{sec:troubleshooting}. b@378: b@378: It is advised that you back up these results as often as possible, as a loss of this data means that the time and effort spent by the subject(s) has been in vain. Save the results to an external or network drive, and/or send them to the experimenter regularly. b@378: b@378: To start the test again for a new participant, you do not need to close the browser or shut down the Terminal or Command Prompt. Simply refresh the page or go to \texttt{localhost:8000} again. b@378: b@378: b@378: \subsection{Remote test} b@378: Put all files on a web server which supports PHP. This allows the `save.php' script to store the XML result files in the `saves/' folder. If the web server is not able to store the XML file there at the end of the test, it will present the XML file locally to the user, as a `Save file' link. b@413: b@413: Make sure the \texttt{projectReturn} attribute of the \texttt{setup} node is set to the \texttt{save.php} script. b@413: b@413: Then, just go to the URL of the corresponding HTML file, e.g. \texttt{http://server.com/path/to/WAET/index.html?url=test/my-test.xml}. If storing on the server doesn't work at submission (e.g. if the \texttt{projectReturn} attribute isn't properly set), the result XML file will be presented to the subject on the client side, as a `Save file' link. b@413: nicholas@434: \subsection{Multiple test documents} nicholas@434: By default the index page will load a demo page of tests. To automatically load a test document, you need to append the location in the URL. If your URL is normally http://localhost:8000/index.html you would append the following: \texttt{?url=/path/to/your/test.xml}. Replace the fields with your actual path, the path is local to the running directory, so if you have your test in the directory \texttt{example\_eval} called \texttt{project.xml} you would append \texttt{?url=/example\_eval/project.xml}. b@413: b@378: \clearpage b@413: b@413: \section{Interfaces} b@413: b@413: The Web Audio Evaluation Tool comes with a number of interface styles, each of which can be customised extensively, either by configuring them differently using the many optional features, or by modifying the JavaScript files. b@413: b@413: To set the interface style for the whole test, %Nick? change when this is not the case anymore, i.e. when the interface can be set per page b@413: add \texttt{interface="APE"} to the \texttt{setup} node, where \texttt{"APE"} is one of the interface names below. b@413: b@413: \subsection{APE} b@413: The APE interface is based on \cite{ape}, and consists of one or more axes, each corresponding with an attribute to be rated, on which markers are placed. As such, it is a multiple stimulus interface where (for each dimension or attribute) all elements are on one axis so that they can be maximally compared against each other, as opposed to rated individually or with regards to a single reference. b@413: It also contains an optional text box for each element, to allow for clarification by the subject, tagging, and so on. b@413: b@413: \subsection{MUSHRA} b@413: This is a straightforward implementation of \cite{mushra}, especially common for the rating of audio quality, for instance for the evaluation of audio codecs. b@413: b@378: b@413: \clearpage b@413: b@413: \section{Features} b@413: b@413: This section goes over the different features implemented in the Web Audio Evaluation Tool, how to use them, and what to know about them. b@413: b@413: Unless otherwise specified, \emph{each} feature described here is optional, i.e. it can be enabled or disabled and adjusted to some extent. b@413: b@413: As the example project showcases (nearly) all of these features, please refer to its configuration XML document for a demonstration of how to enable and adjust them. b@413: b@413: \subsection{Surveys} nicholas@434: Surveys are conducted through an in-page popup window which can collect data using various HTML functions, see Survey elements below for a list. Survey questions are placed into the \texttt{} or \texttt{} nodes. Appending these nodes to the \texttt{} node will have the survey options appear before any test pages (if in the \texttt{} node) or after all test pages. Placing the survey options in the \texttt{} node will have them appear before or after the test page they are a child of. b@413: \subsubsection{Survey elements} b@413: All survey elements (which `pop up' in the centre of the browser) have an \texttt{id} attribute, for retrieval of the responses in post-processing of the results, and a \texttt{mandatory} attribute, which if set to ``true'' requires the subjects to respond before they can continue. b@413: b@413: \begin{description} b@413: \item[statement] Simply shows text to the subject until `Next' or `Start' is clicked. nicholas@434: \item[question] Expects a text answer (in a text box). Has the \texttt{boxsize} argument: set to ``large'' or ``huge'' for a bigger box size, or ``small'' for small. nicholas@434: \item[number] Only accepts a numerical value. Attribute \texttt{min="0"} specifies the minimum value - in this case the answer must be stricly positive before the subject can continue. nicholas@434: \item[radio] Radio buttons. Presents a list of options to the user using radio buttons, where only one option from the list can be selected. b@429: \item[checkbox] Checkboxes. Note that when making a checkbox question ``mandatory'', the subject is forced to select at least one option (which could be e.g. `Other' or `None').\\ b@413: \end{description} b@413: b@413: \textbf{Example usage:}\\ b@413: b@413: \lstset{ b@413: basicstyle=\ttfamily, b@413: columns=fullflexible, b@413: showstringspaces=false, b@413: commentstyle=\color{grey}\upshape b@413: } b@413: b@413: \lstdefinelanguage{XML} b@413: { b@413: morestring=[b]", b@413: morestring=[s]{>}{<}, b@413: morecomment=[s]{}, b@413: stringstyle=\color{black} \bfseries, b@413: identifierstyle=\color{darkblue} \bfseries, b@413: keywordstyle=\color{cyan} \bfseries, b@413: morekeywords={xmlns,version,type}, b@413: breaklines=true% list your attributes here b@413: } b@413: \scriptsize b@413: \lstset{language=XML} b@413: b@413: \begin{lstlisting} b@413: b@413: Please enter your location. (example mandatory text question) b@413: Please enter your age (example non-mandatory number question) b@413: b@413: Please rate this interface (example radio button question) b@413: b@413: b@413: b@413: b@413: b@429: b@429: Please select with which activities you have any experience (example checkbox question) b@429: b@429: b@429: b@413: Thank you for taking this listening test. Please click 'Submit' and your results will appear in the 'saves/' folder. b@413: b@413: \end{lstlisting} b@413: b@413: b@413: \subsection{Randomisation} b@413: b@413: \subsubsection{Randomisation of configuration XML files} nicholas@434: The python server has a special function to automatically cycle through a list of test pages. Instead of directly requesting an XML, simply setting the url item in the browser URL to \texttt{pseudo.xml} will cycle through a list of XMLs. These XMLs must be in the local directory called \texttt{pseudo}. b@413: % how to b@413: % explain how this is implemented in the pythonServer nicholas@434: %Nick? already implemented in the PHP? nicholas@434: % Needs to be implemented in PHP and automated better, will complete soon b@413: b@413: b@413: \subsubsection{Randomsation of page order} nicholas@434: The page order randomisation is set by the \texttt{} node attribute \texttt{randomise-order}, for example \texttt{...} will randomise the test page order. When not set, the default is to \textbf{not} randomise the test page order. b@413: b@413: \subsubsection{Randomisation of axis order} b@413: b@413: \subsubsection{Randomisation of fragment order} nicholas@434: The audio fragment randomisation is set by the \texttt{} node attribute \texttt{randomise-order}, for example \texttt{...} will randomise the test page order. When not set, the default is to \textbf{not} randomise the test page order. b@413: b@413: \subsubsection{Randomisation of initial slider position} nicholas@434: By default slider values are randomised on start. The MUSHRA interface supports setting the initial values of all sliders throught the \texttt{} attribute \texttt{initial-position}. This takes an integer between 0 and 100 to signify the slider position. b@413: % /subsubsection{Randomisation of survey question order} b@413: % should be an attribute of the individual 'pretest' and 'posttest' elements b@413: % uncomment once we have it b@413: b@413: \subsection{Looping} nicholas@434: Looping enables the fragments to loop until stopped by the user. Looping is synchronous between samples so all samples start at the same time. nicholas@434: Individual test pages can have their playback looped by the \texttt{} attribute \texttt{loop} with a value of "true" or "false". b@413: If the fragments are not of equal length initially, they are padded with zeros so that they are equal length, to enable looping without the fragments going out of sync relative to each other. b@413: nicholas@434: Note that fragments cannot be played until all page fragments are loaded when in looped mode, as the engine needs to know the amount to pad the fragments. b@413: b@413: \subsection{Sample rate} b@413: If you require the test to be conducted at a certain sample rate (i.e. you do not tolerate resampling of the elements to correspond with the system's sample rate), add \texttt{sampleRate="96000"} - where ``96000'' can be any support sample rate - so that a warning message is shown alerting the subject the system's sample rate is different from this enforced sample rate. This of course means that in one test, all sample rates must be equal as it is impossible to change the system's sample rates during the test (even if you were to manually change it, then the browser must be restarted for it to take effect). b@413: b@413: \subsection{Scrubber bar} b@413: The scrubber bar, or transport bar (that is the name of the visualisation of the playhead thing with an indication of time and showing the portion of the file played so far) is at this point just a visual, and not a controller to adjust the playhead position. b@413: b@413: Make visible by adding \texttt{