annotate docs/SMC15/smc2015template.tex @ 1946:1d08f8aee098

Merge WAC paper
author Dave Moffat <me@davemoffat.com>
date Thu, 24 Sep 2015 09:52:08 +0100
parents 4988c805ff9e
children 8ab5f8969856
rev   line source
me@1942 1 % -----------------------------------------------
me@1942 2 % Template for SMC 2012
me@1942 3 % adapted from the template for SMC 2011, which was adapted from that of SMC 2010
me@1942 4 % -----------------------------------------------
me@1942 5
me@1942 6 \documentclass{article}
me@1942 7 \usepackage{smc2015}
me@1942 8 \usepackage{times}
me@1942 9 \usepackage{ifpdf}
me@1942 10 \usepackage[english]{babel}
me@1942 11 \usepackage{cite}
me@1942 12 \usepackage{enumitem}
me@1942 13 \usepackage{listings}
me@1942 14 \setitemize{noitemsep,topsep=0pt,parsep=0pt,partopsep=0pt}
me@1942 15
me@1942 16
me@1942 17
me@1942 18 \usepackage{color}
me@1942 19 \definecolor{grey}{rgb}{0.1,0.1,0.1}
me@1942 20 \definecolor{darkblue}{rgb}{0.0,0.0,0.6}
me@1942 21 \definecolor{cyan}{rgb}{0.0,0.6,0.6}
me@1942 22
me@1942 23
me@1942 24 \hyphenation{Java-script}
me@1942 25 \hyphenation{OPA-QUE}
me@1942 26
me@1942 27 %%%%%%%%%%%%%%%%%%%%%%%% Some useful packages %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
me@1942 28 %%%%%%%%%%%%%%%%%%%%%%%% See related documentation %%%%%%%%%%%%%%%%%%%%%%%%%%
me@1942 29 %\usepackage{amsmath} % popular packages from Am. Math. Soc. Please use the
me@1942 30 %\usepackage{amssymb} % related math environments (split, subequation, cases,
me@1942 31 %\usepackage{amsfonts}% multline, etc.)
me@1942 32 %\usepackage{bm} % Bold Math package, defines the command \bf{}
me@1942 33 %\usepackage{paralist}% extended list environments
me@1942 34 %%subfig.sty is the modern replacement for subfigure.sty. However, subfig.sty
me@1942 35 %%requires and automatically loads caption.sty which overrides class handling
me@1942 36 %%of captions. To prevent this problem, preload caption.sty with caption=false
me@1942 37 %\usepackage[caption=false]{caption}
me@1942 38 %\usepackage[font=footnotesize]{subfig}
me@1942 39
me@1942 40
me@1942 41 %user defined variables
me@1942 42 \def\papertitle{WEB AUDIO EVALUATION TOOL: A BROWSER-BASED LISTENING TEST ENVIRONMENT} %?
me@1942 43 \def\firstauthor{Nicholas Jillings}
me@1942 44 \def\secondauthor{Brecht De Man}
me@1942 45 \def\thirdauthor{David Moffat}
me@1942 46 \def\fourthauthor{Joshua D. Reiss}
me@1942 47
me@1942 48 % adds the automatic
me@1942 49 % Saves a lot of ouptut space in PDF... after conversion with the distiller
me@1942 50 % Delete if you cannot get PS fonts working on your system.
me@1942 51
me@1942 52 % pdf-tex settings: detect automatically if run by latex or pdflatex
me@1942 53 \newif\ifpdf
me@1942 54 \ifx\pdfoutput\relax
me@1942 55 \else
me@1942 56 \ifcase\pdfoutput
me@1942 57 \pdffalse
me@1942 58 \else
me@1942 59 \pdftrue
me@1942 60 \fi
me@1942 61
me@1942 62 \ifpdf % compiling with pdflatex
me@1942 63 \usepackage[pdftex,
me@1942 64 pdftitle={\papertitle},
me@1942 65 pdfauthor={\firstauthor, \secondauthor, \thirdauthor},
me@1942 66 bookmarksnumbered, % use section numbers with bookmarks
me@1942 67 pdfstartview=XYZ % start with zoom=100% instead of full screen;
me@1942 68 % especially useful if working with a big screen :-)
me@1942 69 ]{hyperref}
me@1942 70 %\pdfcompresslevel=9
me@1942 71
me@1942 72 \usepackage[pdftex]{graphicx}
me@1942 73 % declare the path(s) where your graphic files are and their extensions so
me@1942 74 %you won't have to specify these with every instance of \includegraphics
me@1942 75 \graphicspath{{./figures/}}
me@1942 76 \DeclareGraphicsExtensions{.pdf,.jpeg,.png}
me@1942 77
me@1942 78 \usepackage[figure,table]{hypcap}
me@1942 79
me@1942 80 \else % compiling with latex
me@1942 81 \usepackage[dvips,
me@1942 82 bookmarksnumbered, % use section numbers with bookmarks
me@1942 83 pdfstartview=XYZ % start with zoom=100% instead of full screen
me@1942 84 ]{hyperref} % hyperrefs are active in the pdf file after conversion
me@1942 85
me@1942 86 \usepackage[dvips]{epsfig,graphicx}
me@1942 87 % declare the path(s) where your graphic files are and their extensions so
me@1942 88 %you won't have to specify these with every instance of \includegraphics
me@1942 89 \graphicspath{{./figures/}}
me@1942 90 \DeclareGraphicsExtensions{.eps}
me@1942 91
me@1942 92 \usepackage[figure,table]{hypcap}
me@1942 93 \fi
me@1942 94
me@1942 95 %set up the hyperref package - make the links black without a surrounding frame
me@1942 96 \hypersetup{
me@1942 97 colorlinks,%
me@1942 98 citecolor=black,%
me@1942 99 filecolor=black,%
me@1942 100 linkcolor=black,%
me@1942 101 urlcolor=black
me@1942 102 }
me@1942 103
me@1942 104
me@1942 105 % Title.
me@1942 106 % ------
me@1942 107 \title{\papertitle}
me@1942 108
me@1942 109 % Authors
me@1942 110 % Please note that submissions are NOT anonymous, therefore
me@1942 111 % authors' names have to be VISIBLE in your manuscript.
me@1942 112 %
me@1942 113 % Single address
me@1942 114 % To use with only one author or several with the same address
me@1942 115 % ---------------
me@1942 116 %\oneauthor
me@1942 117 % {\firstauthor} {Affiliation1 \\ %
me@1942 118 % {\tt \href{mailto:author1@smcnetwork.org}{author1@smcnetwork.org}}}
me@1942 119
me@1942 120 %Two addresses
me@1942 121 %--------------
me@1942 122 % \twoauthors
me@1942 123 % {\firstauthor} {Affiliation1 \\ %
me@1942 124 % {\tt \href{mailto:author1@smcnetwork.org}{author1@smcnetwork.org}}}
me@1942 125 % {\secondauthor} {Affiliation2 \\ %
me@1942 126 % {\tt \href{mailto:author2@smcnetwork.org}{author2@smcnetwork.org}}}
me@1942 127
me@1942 128
me@1942 129
me@1942 130 % FIX!!!
me@1942 131 \fourauthors
me@1942 132 {\firstauthor} {%Affiliation1 \\
me@1942 133 {\tt \href{mailto:b.deman@qmul.ac.uk}{n.g.r.jillings@se14.qmul.ac.uk, }}}
me@1942 134 {\secondauthor} {%Affiliation2\\ %
me@1942 135 {\tt \href{mailto:n.g.r.jillings@se14.qmul.ac.uk}{\{b.deman,}}}
me@1942 136 {\thirdauthor} {%Affiliation3\\ %
me@1942 137 {\tt \href{mailto:d.j.moffat@qmul.ac.uk}{d.j.moffat, }}}
me@1942 138 {\fourthauthor} {%Affiliation4\\ %
me@1942 139 {\tt \href{mailto:joshua.reiss@qmul.ac.uk}{joshua.reiss\}@qmul.ac.uk}}}
me@1942 140
me@1942 141 % ***************************************** the document starts here ***************
me@1942 142 \begin{document}
me@1942 143 %
me@1942 144 \capstartfalse
me@1942 145 \maketitle
me@1942 146 \capstarttrue
me@1942 147 %
me@1942 148 \begin{abstract}
me@1942 149 Perceptual evaluation tests where subjects assess certain qualities of different audio fragments are an integral part of audio and music research. These require specialised software, usually custom-made, to collect large amounts of data using meticulously designed interfaces with carefully formulated questions, and play back audio with rapid switching between different samples.
me@1942 150 New functionality in HTML5 included in the Web Audio API allows for increasingly powerful media applications in a platform independent environment. The advantage of a web application is easy deployment on any platform, without requiring any other application, enabling multiple tests to be easily conducted across locations. In this paper we propose a tool supporting a wide variety of easily configurable, multi-stimulus perceptual audio evaluation tests over the web with multiple test interfaces, pre- and post-test surveys, custom configuration, collection of test metrics and other features. Test design and setup doesn't require programming background, and results are gathered automatically using web friendly formats for easy storing of results on a server.
me@1942 151 % Currently at 150, don't think anything more needs to be done here??
me@1942 152 %Place your abstract at the top left column on the first page.
me@1942 153 %Please write about 150-200 words that specifically highlight the purpose of your work,
me@1942 154 %its context, and provide a brief synopsis of your results.
me@1942 155 %Avoid equations in this part.\\
me@1942 156
me@1942 157 \end{abstract}
me@1942 158
me@1942 159 % TOTAL PAPER: Minimum 4 pages, 6 preferred, max. 8 (6 for demos/posters)\\
me@1942 160
me@1942 161 \section{Introduction}\label{sec:introduction}
me@1942 162
me@1942 163 %NICK: examples of what kind of audio applications HTML5 has made possible, with references to publications (or website)\\
me@1942 164
me@1942 165 Perceptual evaluation of audio plays an important role in a wide range of research on audio quality \cite{schoeffler2013impact,repp}, sound synthesis \cite{de2013real,durr2015implementation}, audio effect design \cite{deman2014a}, source separation \cite{mushram,uhlereiss}, music and emotion analysis \cite{song2013a,eerola2009prediction}, and many others \cite{friberg2011comparison}. % codec design?
me@1942 166
me@1942 167 %This work is based in part on the APE audio perceptual evaluation interface for MATLAB \cite{deman2014b}. An important drawback of this toolbox is the need to have MATLAB to create a test and even to run (barring the use of an executable generated by MATLAB), and limited compatibility with both earlier and newer versions of MATLAB, which makes it hard to maintain. On the other hand, a web application generally has the advantage of running in most browsers on most applications.
me@1942 168
me@1942 169 % IMPORTANT
me@1942 170 %[TO ADD: other interfaces for perceptual evaluation of audio, browser-based or not!] \\
me@1942 171 %BROWSER-BASED: \cite{song2013b,song2013a,beaqlejs} \\
me@1942 172 %MATLAB: \cite{whisper,mushram,scale}
me@1942 173 % to add: OPAQUE, Rumsey's repertory grid technique
me@1942 174
me@1942 175
me@1942 176 \begin{table}[htdp]
me@1942 177 \caption{Available audio perceptual evaluation tools}
me@1942 178 \begin{center}
me@1942 179 \begin{tabular}{|*{3}{l|}}
me@1942 180 % order?
me@1942 181 \hline
me@1942 182 \textbf{Name} & \textbf{Language} & \textbf{Ref.}\\
me@1942 183 \hline
me@1942 184 APE & MATLAB & \cite{deman2014b} \\
me@1942 185 BeaqleJS & HTML5/JS & \cite{beaqlejs}\\ % ABX, mushra
me@1942 186 %C4DM\footnote{http://isophonics.org/test - collection of listening tests developed by Gy\"{o}rgy Fazekas and Thomas Wilmering at Centre for Digital Music.} & JS & \cite{song2013a,song2013b}\\
me@1942 187 HULTI-GEN & Max & \cite{hulti-gen}\\
me@1942 188 MUSHRAM & MATLAB & \cite{mushram}\\ % type: mushra
me@1942 189 Scale & MATLAB & \cite{scale} \\
me@1942 190 WhisPER & MATLAB & \cite{whisper}\\
me@1942 191 \hline
me@1942 192 \end{tabular}
me@1942 193 \end{center}
me@1942 194 \label{tab:interfaces}
me@1942 195 \end{table}%
me@1942 196
me@1942 197 Various listening test design tools are already available, see Table \ref{tab:interfaces}. A few other listening test tools, such as OPAQUE \cite{opaque} and GuineaPig \cite{guineapig}, are described but not available to the public at the time of writing.
me@1942 198
me@1942 199 Many are MATLAB-based, useful for easily processing and visualising the data produced by the listening tests, but requiring MATLAB to be installed to run or - in the case of an executable created with MATLAB - at least create the test.
me@1942 200 Furthermore, compatibility is usually limited across different versions of MATLAB.
me@1942 201 Similarly, Max requires little or no programming background but it is proprietary software as well, which is especially undesirable when tests need to be deployed at different sites.
me@1942 202 More recently, BeaqleJS \cite{beaqlejs} makes use of the HTML5 audio capabilities and comes with a number of predefined, established test interfaces such as ABX and MUSHRA \cite{mushra}. BeaqleJS provides a number of similar features including saving of test data to a web server. The main difference is that with BeaqleJS, the configuration is done through writting a JavaScript file holding a JavaScript Object of the notation. Instead our presented system uses the XML document standard, which allows configuration outside of a web-centric editor. The results are also presented in XML again allowing 3\textsuperscript{rd} party editors and programs to easily access. Finally, the presented system does not require web access to run, instead being deployed with a Python server script. This is particularly useful in studios where machines may not, by design, be web connected, or use in locations where web access is limited.
me@1942 203
me@1942 204 A browser-based perceptual evaluation tool for audio has a number of advantages. First of all, it doesn't need any other software than a browser, meaning deployment is very easy and cheap. As such, it can also run on a variety of devices and platforms. The test can be hosted on a central server with subjects all over the world, who can simply go to a webpage. This means that multiple participants can take the test simultaneously, potentially in their usual listening environment if this is beneficial for the test. Naturally, the constraints on the listening environment and other variables still need to be controlled if they are important to the experiment. Depending on the requirements a survey or a variety of tests preceding the experiment could establish whether remote participants and their environments are adequate for the experiment at hand.
me@1942 205
me@1942 206 The Web Audio API is a high-level JavaScript Application Programming Interface (API) designed for real-time processing of audio inside the browser through various processing nodes\footnote{http://webaudio.github.io/web-audio-api/}. Various web sites have used the Web Audio API for creative purposes, such as drum machines and score creation tools\footnote{http://webaudio.github.io/demo-list/},
me@1942 207 others from the list show real-time captured audio processing such as room reverberation tools and a phase vocoder from the system microphone. The BBC Radiophonic Workshop shows effects used on famous TV shows such as Doctor Who, being simulated inside the browser\footnote{http://webaudio.prototyping.bbc.co.uk/}.
me@1942 208 Another example is the BBC R\&D personalised compressor which applies a dynamic range compressor on a radio station that dynamically adjusts the compressor settings to match the listener's environment \cite{mason2015compression}.
me@1942 209
me@1942 210
me@1942 211
me@1942 212 % [How is this one different from all these?] improve
me@1942 213
me@1942 214 % FLEXIBLE (reference (not) appropriate)
me@1942 215 In contrast with the tools listed above, we aim to provide an environment in which a variety of multi-stimulus tests can be designed, with a wide range of configurability, while keeping setup and collecting results as straightforward as possible. For instance, the option to provide free-text comment fields allows for tests with individual vocabulary methods, as opposed to only allowing quantitative scales associated to a fixed set of descriptors.
me@1942 216 % EASE OF USE: no need to go in the code
me@1942 217 To make the tool accessible to a wide range of researchers, we aim to offer maximum functionality even to those with little or no programming background. The tool we present can set up a listening test without reading or adjusting any code, provided no new types of interfaces need to be created.
me@1942 218
me@1942 219 % ENVIRONMENT %In this paper, we provide a listening test back end that allows for easy set up of a wide variety of listening tests, highly flexible yet very simple and not requiring any programming skills.
me@1942 220 Specifically, we present a browser-based perceptual evaluation tool from which any kind of multiple stimulus audio evaluation tool where subjects need to rank, rate, select, or comment on different audio samples can be built.
me@1942 221 We also include an example of the multiple stimulus user interface included with the APE tool \cite{deman2014b}, which presents the subject with a number of axes on which a number of markers, corresponding to audio samples, can be moved to reflect any subjective quality, as well as corresponding comment boxes.
me@1942 222 However, other graphical user interfaces can be put on top of the engine that we provide with minimal or no modifications. Examples of this are the MUSHRA test \cite{mushra}, single or multiple stimulus evaluation with a two-dimensional interface (such as valence and arousal dimensions), or simple annotation (using free-form text, check boxes, radio buttons or drop-down menus) of one or more audio samples at a time.
me@1942 223 In some cases, such as method of adjustment, where the audio is processed by the user, or AB test, where the interface does not show all audio samples to be evaluated at once \cite{bech}, the back end of the tool needs to be modified as well.
me@1942 224
me@1942 225 In the following sections, we describe the included interface in more detail, discuss the implementation, and cover considerations that were made in the design process of this tool.
me@1942 226
me@1942 227 %\section{Requirements}\label{sec:requirements}
me@1942 228 %???
me@1942 229 %
me@1942 230 %\begin{itemize}
me@1942 231 %\item
me@1942 232 %\end{itemize}
me@1942 233 \section{Interface}\label{sec:interface}
me@1942 234
me@1942 235 At this point, we have implemented the interface of the MATLAB-based APE (Audio Perceptual Evaluation) toolbox \cite{deman2014b}. This shows one marker for each simultaneously evaluated audio fragment on one or more horizontal axes, that can be moved to rate or rank the respective fragments in terms of any subjective property, as well as a comment box for every marker, and any extra text boxes for extra comments.
me@1942 236 The reason for such an interface, where all stimuli are presented on a single rating axis (or multiple axes if multiple subjective qualities need to be evaluated), is that it urges the subject to consider the rating and/or ranking of the stimuli relative to one another, as opposed to comparing each individual stimulus to a given reference, as is the case with e.g. a MUSHRA test \cite{mushra}. As such, it is ideal for any type of test where the goal is to carefully compare samples against each other, like perceptual evaluation of different mixes of music recordings \cite{deman2015a} or sound synthesis models \cite{durr2015implementation}, as opposed to comparing results of source separation algorithms \cite{mushram} or audio with lower data rate \cite{mushra} to a high quality reference signal.
me@1942 237
me@1942 238 The markers on the slider at the top of the page are positioned randomly, to minimise the bias that may be introduced when the initial positions are near the beginning, end or middle of the slider. Another approach is to place the markers outside of the slider bar at first and have the subject drag them in, but the authors believe this doesn't encourage careful consideration and comparison of the different fragments as the implicit goal of the test becomes to audition and drag each fragment in just once, rather than to compare all fragments rigorously.
me@1942 239
me@1942 240 See Figure \ref{fig:interface} for an example of the interface. %? change if a new interface is shown
me@1942 241
me@1942 242 %Most of these functions are specific to the APE interface design, for instance the AB test will need a different structure for the audio engine and loading of files, since multiple instances of the same file are required. % more generally these pertain to any typeof multi-stimulus test - not quite useful for AB tests, method of adjustment, ABX, and so on.
me@1942 243 %There are some areas of the design where certain design choices had to be made such as with the markers.
me@1942 244
me@1942 245 %For instance, the option to provide free-text comment fields allows for tests with individual vocabulary methods, as opposed to only allowing quantitative scales associated to a fixed set of descriptors.
me@1942 246
me@1942 247 \begin{figure*}[ht]
me@1942 248 \centering
me@1942 249 \includegraphics[width=.95\textwidth]{interface.png}
me@1942 250 \caption{Example interface, with one axis, seven fragments, and text, radio button and check box style comments.}
me@1942 251 \label{fig:interface}
me@1942 252 \end{figure*}
me@1942 253
me@1942 254
me@1942 255 \section{Architecture}\label{sec:architecture} % or implementation?
me@1942 256
me@1942 257 The tool uses entirely client side processing utilising the new HTML5 Web Audio API, supported by most major web browsers. The API allows for constructing audio processing elements and connecting them together to produce a high quality, real time signal process to manipulate audio streams. The API supports multichannel processing and has an accurate playback timer for precise, scheduled playback control. The API is controlled through the browser JavaScript engine and is therefore highly configurable. Processing is all performed in a low latency thread separate from the main JavaScript thread, so there is no blocking due to real time processing.
me@1942 258
me@1942 259 The web tool itself is split into several files to operate:
me@1942 260 \begin{itemize}
me@1942 261 \item \texttt{index.html}: The main index file to load the scripts, this is the file the browser must request to load.
me@1942 262 \item \texttt{core.js}: Contains global functions and object prototypes to define the audio playback engine, audio objects and loading media files
me@1942 263 \item \texttt{ape.js}: Parses setup files to create the interface as instructed, following the same style chain as the MATLAB APE Tool \cite{deman2014b}.
me@1942 264 \end{itemize}
me@1942 265
me@1942 266 The HTML file loads the \texttt{core.js} file along with a few other ancillary files (such as the jQuery JavaScript extensions\footnote{http://jquery.com/}), at which point the browser JavaScript begins to execute the on-page instructions, which gives the URL of the test setup XML document (outlined in Section \ref{sec:setupresultsformats}). \texttt{core.js} parses this document and executes the functions in \texttt{ape.js} to build the web page. The reason for separating these two files is to allow for further interface designs (such as MUSHRA \cite{mushra} or 2D rating \cite{bech}) to be used, which would still require the same underlying core functions outlined in \texttt{core.js}.
me@1942 267
me@1942 268 The \texttt{ape.js} file has several main functions but the most important are documented here. \textit{loadInterface(xmlDoc)} is called to decode the supplied project document in respect for the interface specified and define any global structures (such as the slider interface). It also identifies the number of pages in the test and randomises the order, if specified to do so. This is the only mandatory function in any of the interface files as this is called by \texttt{core.js} when the document is ready. \texttt{core.js} cannot 'see' any interface specific functions and therefore cannot assume any are available. Therefore \textit{loadInterface(xmlDoc)} is essential to set up the entire test environment. Because the interface files are loaded by \texttt{core.js} and because the functions in \texttt{core.js} are global, the interface files can `see' the \texttt{core.js} file and can therefore not only interact with it, but also modify it.
me@1942 269
me@1942 270 Each test page is loaded using \textit{loadTest(id)} which performs two major tasks: to populate the interface with the slider elements and comment boxes; and secondly to instruct the \textit{audioEngine} to load the audio fragments and construct the backend audio graph. \textit{loadTest(id)} also instructs the audio engine in \texttt{core.js} to create the \textit{audioObject}.
me@1942 271 These are custom audio nodes, one representing each audio element specified in each page.
me@1942 272 They consist of a \textit{bufferSourceNode} (a node which holds a buffer of audio samples for playback) and a \textit{gainNode}, both of which are Web Audio API Nodes. Various functions are applied, depending on which metrics are enabled, to record the interaction with the audio element. These nodes are then connected to the \textit{audioEngine} (itself a custom web audio node) containing a \textit{gainNode} (where the various \textit{audioObject}s connect to) for summation before passing the output to the \textit{destinationNode}, a permanent node of the Web Audio API created as the master output. Here, the browser then passes the audio information to the system. % Does this now make sense?
me@1942 273 % audio object/audioObject/Audio Object: -- should always be audioObject if talking about the JavaScript object, otherwise should say audio element or audio fragment.
me@1942 274
me@1942 275 When an \textit{audioObject} is created, it is given the URL of the audio sample to load. This is downloaded into the browser asynchronously using the \textit{XMLHttpRequest} object, which downloads any file into the JavaScript environment for further processing. This is particularly useful for the Web Audio API because it supports downloading of files in their binary form for decoding. Once downloaded the file is decoded using the Web Audio API offline decoder. This uses the browser available decoding schemes to decode the audio files into raw float32 arrays, which are in turn passed to the relevant \textit{audioObject} for playback.
me@1942 276
me@1942 277 Once each page of the test is completed, identified by pressing the Submit button, the \textit{pageXMLSave(testId)} is called to store all of the collected data until all pages of the test are completed. After the final test and any post-test questions are completed, the \textit{interfaceXMLSave()} function is called. This function generates the final XML file for submission as outlined in Section \ref{sec:setupresultsformats}.
me@1942 278
me@1942 279 \vspace{-1em}
me@1942 280
me@1942 281 \section{Support and limitations}\label{sec:support}
me@1942 282
me@1942 283 Different browsers support a different set of audio file formats and are not consistent in any format. Currently the Web Audio API is best supported in Chrome, Firefox, Opera and Safari. All of these support the use of the uncompressed WAV format. Although not a compact, web friendly format, most transport systems are of a high enough bandwidth this should not be a problem. Ogg Vorbis is another well supported format across the four supported major desktop browsers, as well as MP3 (although Firefox may not support all MP3 types\footnote{https://developer.mozilla.org/en-US/docs/Web/HTML/\\Supported\_media\_formats}). %https://developer.mozilla.org/en-US/docs/Web/HTML/Supported_media_formats
me@1942 284 One issue of the Web Audio API is that the sample rate is assigned by the system sound device, rather than requested and does not have the ability to request a different one. % Does this make sense? The problem is across all audio files.
me@1942 285 As the sampling rate and the effect of resampling may be critical for some listening tests, the default operation when an audio file is loaded with a different sample rate to that of the system is to convert the sample rate. To provide a check for this, the desired sample rate can be supplied with the setup XML and checked against. If the sample rates do not match, a browser alert window is shown asking for the sample rate to be correctly adjusted.
me@1942 286 This happens before any loading or decoding of audio files so the browser will only be instructed to fetch files if the system sample rate meets the requirements, avoiding multiple requests for large files until they are actually needed.
me@1942 287
me@1942 288 %During playback, the playback nodes loop indefinitely until playback is stopped. The gain nodes in the \textit{audioObject}s enable dynamic muting of nodes. When a bar in the sliding ranking is clicked, the audio engine mutes all \textit{audioObject}s and un-mutes the clicked one. Therefore, if the audio samples are perfectly aligned up and of the same sample length, they will remain perfectly aligned with each other.
me@1942 289 % Don't think this is relevant anymore
me@1942 290
me@1942 291
me@1942 292 \section{Input and result files}\label{sec:setupresultsformats}
me@1942 293
me@1942 294 The setup and result files both use the common XML document format to outline the various parameters. The setup file determines the interface to use, the location of audio files, the number of pages and other parameters to define the testing environment. Having one document to modify allows for quick manipulation in a `human readable' form to create new tests, or adjust current ones, without needing to edit multiple web files. Furthermore, we also provide a simple web page to enter all these settings without needing to manipulate the raw XML. An example of such an XML document is presented below. % I mean the .js and .html files, though not sure if any better.
me@1942 295
me@1942 296
me@1942 297
me@1942 298
me@1942 299 \lstset{
me@1942 300 basicstyle=\ttfamily,
me@1942 301 columns=fullflexible,
me@1942 302 showstringspaces=false,
me@1942 303 commentstyle=\color{grey}\upshape
me@1942 304 }
me@1942 305
me@1942 306 \lstdefinelanguage{XML}
me@1942 307 {
me@1942 308 morestring=[b]",
me@1942 309 morestring=[s]{>}{<},
me@1942 310 morecomment=[s]{<?}{?>},
me@1942 311 stringstyle=\color{black} \bfseries,
me@1942 312 identifierstyle=\color{darkblue} \bfseries,
me@1942 313 keywordstyle=\color{cyan} \bfseries,
me@1942 314 morekeywords={xmlns,version,type},
me@1942 315 breaklines=true% list your attributes here
me@1942 316 }
me@1942 317 \scriptsize
me@1942 318 \lstset{language=XML}
me@1942 319
me@1942 320 \begin{lstlisting}
me@1942 321 <?xml version="1.0" encoding="utf-8"?>
me@1942 322 <BrowserEvalProjectDocument>
me@1942 323 <setup interface="APE" projectReturn="/save" randomiseOrder='false' collectMetrics='true'>
me@1942 324 <PreTest>
me@1942 325 <question id="location" mandatory="true">Please enter your location.</question>
me@1942 326 <number id="age" min="0">Please enter your age</number>
me@1942 327 </PreTest>
me@1942 328 <PostTest>
me@1942 329 <statement>Thank you for taking this listening test!</statement>
me@1942 330 </PostTest>
me@1942 331 <Metric>
me@1942 332 <metricEnable>testTimer</metricEnable>
me@1942 333 <metricEnable>elementTimer</metricEnable>
me@1942 334 <metricEnable>elementInitialPosition</metricEnable>
me@1942 335 <metricEnable>elementTracker</metricEnable>
me@1942 336 <metricEnable>elementFlagListenedTo</metricEnable>
me@1942 337 <metricEnable>elementFlagMoved</metricEnable>
me@1942 338 </Metric>
me@1942 339 <interface>
me@1942 340 <anchor>20</anchor>
me@1942 341 <reference>80</reference>
me@1942 342 </interface>
me@1942 343 </setup>
me@1942 344 <audioHolder id="test-0" hostURL="example_eval/" randomiseOrder='true'>
me@1942 345 <interface>
me@1942 346 <title>Example Test Question</title>
me@1942 347 <scale position="0">Min</scale>
me@1942 348 <scale position="100">Max</scale>
me@1942 349 <commentBoxPrefix>Comment on fragment</commentBoxPrefix>
me@1942 350 </interface>
me@1942 351 <audioElements url="1.wav" id="elem1"/>
me@1942 352 <audioElements url="2.wav" id="elem2"/>
me@1942 353 <audioElements url="3.wav" id="elem3"/>
me@1942 354 <CommentQuestion id="generalExperience" type="text">General Comments</CommentQuestion>
me@1942 355 <PreTest/>
me@1942 356 <PostTest>
me@1942 357 <question id="songGenre" mandatory="true">Please enter the genre of the song.</question>
me@1942 358 </PostTest>
me@1942 359 </audioHolder>
me@1942 360 </BrowserEvalProjectDocument>
me@1942 361
me@1942 362 \end{lstlisting}
me@1942 363
me@1942 364 \normalsize
me@1942 365 \vspace{-1em}
me@1942 366
me@1942 367 \subsection{Setup and configurability}
me@1942 368
me@1942 369 The setup document has several defined nodes and structure which are documented with the source code. For example, there is a section for general setup options where any pre-test and post-test questions and statements can be defined. Pre- and post-test dialogue boxes allow for comments or questions to be presented before or after the test, to convey listening test instructions, and gather information about the subject, listening environment, and overall experience of the test. In the example set up document above, a question box with the id `location' is added, which is set to be mandatory to answer. The question is in the PreTest node meaning it will appear before any testing will begin. When the result for the entire test is shown, the response will appear in the PreTest node with the id `location' allowing it to be found easily, provided the id values are meaningful.
me@1942 370
me@1942 371 We try to cater to a diverse audience with this toolbox, while ensuring it is simple, elegant and straightforward. To that end, we currently include the following options that can be easily switched on and off, by setting the value in the input XML file.
me@1942 372
me@1942 373 \begin{itemize}[leftmargin=*]%Should have used a description list for this.
me@1942 374 \item \textbf{Snap to corresponding position}: When enabled and a fragment is playing, the playhead skips to the same position in the next fragment that is clicked. Otherwise, each fragment is played from the start.
me@1942 375 \item \textbf{Loop fragments}: Repeat current fragment when end is reached, until the `Stop' or `Submit' button is clicked.
me@1942 376 \item \textbf{Comments}: Displays a separate comment box for each fragment in the page.
me@1942 377 \item \textbf{General comment}: Create additional comment boxes to the fragment comment boxes, with a custom question and various input formats such as checkbox or radio.
me@1942 378 \item \textbf{Resampling}: When this is enabled, fragments are resampled to match the subject's system's sample rate (a default feature of the Web Audio API). When it is not, an error is shown when the system does not match the requested sample rate.
me@1942 379 \item \textbf{Randomise page order}: Randomises the order in which different `pages' are presented. % are we calling this 'pages'?
me@1942 380 \item \textbf{Randomise fragment order}: Randomises the order and numbering of the markers and comment boxes corresponding to the fragments. Fragments are referenced to their given ID so referencing is possible (such as `this is much brighter than fragment 4').
me@1942 381 \item \textbf{Require (full) playback}: Require that each fragment has been played at least once, partly or fully.
me@1942 382 \item \textbf{Require moving}: Require that each marker is moved (dragged) at least once.
me@1942 383 \item \textbf{Require comments}: Require the subject to write a comment for each fragment.
me@1942 384 \item \textbf{Repeat test}: Number of times each page in the test should be repeated (none by default), to allow familiarisation with the content and experiment, and to investigate consistency of user and variability due to familiarity. These are all gathered before shuffling the order so repeated tests are not back-to-back if possible.
me@1942 385 \item \textbf{Returning to previous pages}: Indicates whether it is possible to go back to a previous `page' in the test.
me@1942 386 \item \textbf{Lowest rating below [value]}: To enforce a certain use of the rating scale, it can be required to rate at least one sample below a specified value.
me@1942 387 \item \textbf{Highest rating above [value]}: To enforce a certain use of the rating scale, it can be required to rate at least one sample above a specified value.
me@1942 388 \item \textbf{Reference}: Allows for a separate sample (outside of the axis) to be the `reference', which the subject can play back during the test to help with the task at hand \cite{mushra}.
me@1942 389 \item \textbf{Hidden reference/anchor}: Whether or not an explicit `reference' is provided, the `hidden reference' should be rated above a certain value \cite{mushra} - this can be enforced.
me@1942 390 Similarly, a `hidden anchor' should be rated lower than a certain value \cite{mushra}.
me@1942 391 \item \textbf{Show scrub bar}: Display a playhead on a scrub bar to show the position in the current fragment.
me@1942 392 %\item \textbf{Drag playhead}: If scrub bar is visible, allow dragging to move back or forward in a fragment.
me@1942 393 \end{itemize}
me@1942 394
me@1942 395 When one of these options is not included in the setup file, they assume a default value. As a result, the input file can be kept very compact if default values suffice for the test.
me@1942 396
me@1942 397 % loop, snap to corresponding position, comments, 'general' comment, require same sampling rate, different types of randomisation
me@1942 398
me@1942 399 \subsection{Results}
me@1942 400
me@1942 401 The results file is dynamically generated by the interface upon clicking the `Submit' button. This also executes checks, depending on the setup file, to ensure that all fragments have been played back, rated and commented on. The XML output returned contains a node per fragment and contains both the corresponding marker's position and any comments written in the associated comment box. The rating returned is normalised to be a value between 0 and 1, normalising the pixel representation of different browser windows. The results also contain information collected by any defined pre/post questions. An excerpt of an output file is presented below detailing the data collected for a single audioElement.
me@1942 402
me@1942 403 \scriptsize
me@1942 404 \lstset{language=XML}
me@1942 405
me@1942 406 \begin{lstlisting}
me@1942 407 <browserevaluationresult>
me@1942 408 <datetime>
me@1942 409 <date year="2015" month="5" day="28">2015/5/28</date>
me@1942 410 <time hour="13" minute="19" secs="17">13:19:17</time>
me@1942 411 </datetime>
me@1942 412 <pretest>
me@1942 413 <comment id="location">Control Room</comment>
me@1942 414 </pretest>
me@1942 415 <audioholder>
me@1942 416 <pretest></pretest>
me@1942 417 <posttest>
me@1942 418 <comment id="songGenre">Pop</comment>
me@1942 419 </posttest>
me@1942 420 <metric>
me@1942 421 <metricresult id="testTime">813.32</metricresult>
me@1942 422 </metric>
me@1942 423 <audioelement id="elem1">
me@1942 424 <comment>
me@1942 425 <question>Comment on fragment 1</question>
me@1942 426 <response>Good, but vocals too quiet.</response>
me@1942 427 </comment>
me@1942 428 <value>0.639010989010989</value>
me@1942 429 <metric>
me@1942 430 <metricresult id="elementTimer">111.05</metricresult>
me@1942 431 <metricresult id="elementTrackerFull">
me@1942 432 <timepos id="0">
me@1942 433 <time>61.60</time>
me@1942 434 <position>0.6390</position>
me@1942 435 </timepos>
me@1942 436 </metricresult>
me@1942 437 <metricresult id="elementInitialPosition">0.6571</metricresult>
me@1942 438 <metricresult id="elementFlagListenedTo">true</metricresult>
me@1942 439 </metric>
me@1942 440 </audioelement>
me@1942 441 </audioHolder>
me@1942 442 </browserevaluationresult>
me@1942 443
me@1942 444 \end{lstlisting}
me@1942 445
me@1942 446 \normalsize
me@1942 447 \vspace{-.5em}
me@1942 448 Each page of testing is returned with the results of the entire page included in the structure. One \texttt{audioelement} node is created per audio fragment per page, along with its ID. This includes several child nodes including the rating between 0 and 1, the comment, and any other collected metrics including how long the element was listened for, the initial position, and boolean flags showing if the element was listened to, moved and commented on. Furthermore, each user action (manipulation of any interface element, such as playback or moving a marker) can be logged along with a the corresponding time code.
me@1942 449 We also store session data such as the time the test took place and the duration of the test.
me@1942 450 We provide the option to store the results locally, and/or to have them sent to a server.
me@1942 451
me@1942 452 %Here is an example of the set up XML and the results XML: % perhaps best to refer to each XML after each section (set up <> results)
me@1942 453 % Should we include an Example of the input and output XML structure?? --> Sure.
me@1942 454
me@1942 455 %An example of the returned \textit{audioElement} node in the results XML file is as follows.
me@1942 456 %
me@1942 457 %\texttt{<audioelement id="8"> \\
me@1942 458 %<comment> \\
me@1942 459 %<question>Comment on track 0</question> \\
me@1942 460 %<response> The drums were punchy </response> \\
me@1942 461 %</comment> \\
me@1942 462 %<value> 0.25169491525423726 </value> \\
me@1942 463 %<metric> \\
me@1942 464 %<metricresult id="elementTimer"> \\ 2.3278004535147385< /metricresult> \\
me@1942 465 %<metricresult id="elementTrackerFull"> \\
me@1942 466 %<timepos id="0"> \\
me@1942 467 %<time>1.7937414965986385</time> \\
me@1942 468 %<position>0.41694915254237286</position> \\
me@1942 469 %</timepos> \\
me@1942 470 %<timepos id="1"> \\
me@1942 471 %<time>2.6993197278911563</time> \\
me@1942 472 %<position>0.45847457627118643</position> \\
me@1942 473 %</timepos> \\</metricresult> \\
me@1942 474 %<metricresult id="elementInitialPosition"> 0.47796610169491527 </metricresult> \\
me@1942 475 %<metricresult id="elementFlagListenedTo"> true< /metricresult> \\
me@1942 476 %<metricresult id="elementFlagMoved"> true </metricresult> \\
me@1942 477 %</metric> \\
me@1942 478 %</audioelement>}
me@1942 479
me@1942 480 % BRECHT: scripts
me@1942 481
me@1942 482 \begin{figure}[htpb]
me@1942 483 \centering
me@1942 484 \includegraphics[width=.45\textwidth]{boxplot.png}
me@1942 485 \caption{An example boxplot showing ratings by different subjects on fragments labeled `A' through `G'. }
me@1942 486 \label{fig:boxplot}
me@1942 487 \end{figure}
me@1942 488
me@1942 489 Python scripts are included to easily store ratings and comments in a CSV file, and to display graphs of numerical ratings (see Figure \ref{fig:boxplot}) or visualise the test's timeline.
me@1942 490 Visualisation of plots requires the free matplotlib library\footnote{http://matplotlib.org}.
me@1942 491
me@1942 492
me@1942 493 \section{Conclusions and future work}\label{sec:conclusions}
me@1942 494
me@1942 495 In this paper we have presented an approach to creating a browser-based listening test environment that can be used for a variety of types of perceptual evaluation of audio.
me@1942 496 Specifically, we discussed the use of the toolbox in the context of assessment of preference for different production practices, with identical source material.
me@1942 497 The purpose of this paper is to outline the design of this tool, to describe our implementation using basic HTML5 functionality, and to discuss design challenges and limitations of our approach. This tool differentiates itself from other perceptual audio tools by enabling web technologies for multiple participants to perform the test without the need for proprietary software such as MATLAB. The tool also allows for any interface to be built using HTML5 elements to create a variety of dynamic, multiple-stimulus listening test interfaces. It enables quick setup of simple tests with the ability to manage complex tests through a single file. Finally it uses the XML document format to store the results allowing for processing and analysis of results in various third party software such as MATLAB or Python.
me@1942 498
me@1942 499 % future work
me@1942 500 Further work may include the development of other common test designs, such as MUSHRA \cite{mushra}, 2D valence and arousal/activity \cite{eerola2009prediction}, and others. We will add functionality to assist with setting up large-scale tests with remote subjects, so this becomes straightforward and intuitive.
me@1942 501 In addition, we will keep on improving and expanding the tool, and highly welcome feedback and contributions from the community.
me@1942 502
me@1942 503 The source code of this tool can be found on \\ \texttt{code.soundsoftware.ac.uk/projects/}\\ \texttt{webaudioevaluationtool}.
me@1942 504
me@1942 505
me@1942 506 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
me@1942 507 %bibliography here
me@1942 508 \bibliography{smc2015template}
me@1942 509
me@1942 510 \end{document}