comparison docs/SMC15/smc2015template.tex @ 988:cd426747743b

Paper: merge, table of listening test tools, other changes

 
 \section{Introduction}\label{sec:introduction}
 
 %NICK: examples of what kind of audio applications HTML5 has made possible, with references to publications (or website)\\
 
-Perceptual evaluation of audio plays an important role in a wide range of research including audio quality \cite{schoeffler2013impact,repp}, sound synthesis \cite{de2013real,durr2015implementation}, audio effect design \cite{deman2014a}, source separation \cite{uhlereiss}, music and emotion analysis \cite{song2013b,song2013a}, and many others \cite{friberg2011comparison}.  % codec design? 
+Perceptual evaluation of audio plays an important role in a wide range of research including audio quality \cite{schoeffler2013impact,repp}, sound synthesis \cite{de2013real,durr2015implementation}, audio effect design \cite{deman2014a}, source separation \cite{mushram,uhlereiss}, music and emotion analysis \cite{song2013b,song2013a}, and many others \cite{friberg2011comparison}.  % codec design? 
 
 This work is based in part on the APE audio perceptual evaluation interface for MATLAB \cite{deman2014b}. An important drawback of this toolbox is the need to have MATLAB to create a test and even to run (barring the use of an executable generated by MATLAB), and limited compatibility with both earlier and newer versions of MATLAB, which makes it hard to maintain. On the other hand, a web application generally has the advantage of running in most browsers on most applications.
 
 % IMPORTANT
 %[TO ADD: other interfaces for perceptual evaluation of audio, browser-based or not!] \\
 %BROWSER-BASED: \cite{song2013b,song2013a,beaqlejs} \\
 %MATLAB: \cite{whisper,mushram,scale}  
 % to add: OPAQUE, Rumsey's repertory grid technique
 
-Many perceptual audio interfaces are already available, such as the MATLAB-based tools MUSHRAM \cite{mushram}, WhisPER \cite{whisper}, and Scale \cite{scale}, and the browser-based tool used in \cite{song2013a,song2013b} and BeaqleJS \cite{beaqlejs}. The latter makes use of the HTML5 audio capabilities and comes with a number of predefined, established test interfaces such as ABX and MUSHRA \cite{mushra}. 
-In this paper, we provide a listening test back end that allows for easy set up of a wide variety of listening tests, highly flexible yet very simple and not requiring any programming skills. 
+
+
+\begin{table}[htdp]
+\caption{Available audio perceptual evaluation interfaces}
+\begin{center}
+\begin{tabular}{|*{3}{l|}}
+% order? 
+\hline
+\textbf{Name} & \textbf{Language} & \textbf{Ref.}\\
+\hline
+APE & MATLAB & \cite{deman2014b} \\
+BeaqleJS & HTML5/JS & \cite{beaqlejs}\\ % ABX, mushra
+%C4DM\footnote{http://isophonics.org/test - collection of listening tests developed by Gy\"{o}rgy Fazekas and Thomas Wilmering at Centre for Digital Music.} & JS & \cite{song2013a,song2013b}\\
+HULTI-GEN & Max & \cite{hulti-gen}\\
+MUSHRAM & MATLAB & \cite{mushram}\\ % type: mushra
+Scale & MATLAB & \cite{scale} \\ 
+WhisPER & MATLAB & \cite{whisper}\\
+\hline
+\end{tabular}
+\end{center}
+\label{tab:interfaces}
+\end{table}%
+
+Various perceptual audio interfaces are already available, see Table \ref{tab:interfaces}. 
+Many are MATLAB-based, useful for easily processing and visualising the data produced by the listening tests, but requiring the application to be installed to run or - in the case of an executable created with MATLAB - at least create the test. 
+Furthermore, compatibility is limited across different versions of MATLAB. 
+Similarly, Max requires little or no programming background but it is proprietary software as well, which is especially undesirable when tests need to be deployed at different sites. 
+More recently, BeaqleJS \cite{beaqlejs} makes use of the HTML5 audio capabilities and comes with a number of predefined, established test interfaces such as ABX and MUSHRA \cite{mushra}. % 
+Another listening test tool, GuineaPig \cite{guineapig}, is not available to the public at the time of writing. 
+
 % [How is this one different from all these?] improve
 
+% FLEXIBLE (reference (not) appropriate)
+
+Furthermore, the option to provide free-text comment fields allows for tests with individual vocabulary methods, as opposed to only allowing quantitative scales associated to a fixed set of descriptors.
+
+
+% ENVIRONMENT
+There are a number of advantages to building a web audio based listening test environment. The ability to easily deploy a flexible and scalable testing environment that requires no proprietary software to run makes the web audio evaluation tool a very flexible testing tool. The ability to host a single test server and create multiple clients not only allows multiple participants to be involved in a trial simultaneously, but also permits participants to be located anywhere in the world. There are also less user experience issues, since all users should have some experience with using existing web technologies. 
+
+% EASE OF USE: no need to go in the code
+To make the tool accessible to a wide range of researchers, we aim to offer maximum functionality even to those with little or no programming background. The tool we present can set up a listening test without reading or adjusting any code, provided no new types of interfaces need to be created. 
+
+We present a browser-based perceptual evaluation tool from which any kind of multiple stimulus audio evaluation tool where subjects need to rank, rate, select, or comment on different audio samples can be built. %In this paper, we provide a listening test back end that allows for easy set up of a wide variety of listening tests, highly flexible yet very simple and not requiring any programming skills. 
 The Web Audio API is a high-level JavaScript Application Programming Interface (API) designed for real-time processing of audio inside the browser through various processing nodes\footnote{http://webaudio.github.io/web-audio-api/}. Various web sites have used the Web Audio API for either creative purposes, such as drum machines and score creation tools\footnote{http://webaudio.github.io/demo-list/}, 
 others from the list show real-time captured audio processing such as room reverberation tools and a phase vocoder from the system microphone. The BBC Radiophonic Workshop shows effects used on famous TV shows such as Doctor Who, being simulated inside the browser\footnote{http://webaudio.prototyping.bbc.co.uk/}. 
 Another example is the BBC R\&D personalised compressor which applies a dynamic range compressor on a radio station that dynamically adjusts the compressor settings to match the listener's environment \cite{mason2015compression}. 
 
 We present a browser-based perceptual evaluation tool from which any kind of multiple stimulus audio evaluation tool where subjects need to rank, rate, select, or comment on different audio samples can be built. 

 However, other graphical user interfaces can be put on top of the engine that we provide with minimal or no modifications. Examples of this are the MUSHRA test \cite{mushra}, single or multiple stimulus evaluation with a two-dimensional interface (such as valence and arousal dimensions), or simple annotation (using free-form text, check boxes, radio buttons or drop-down menus) of one or more audio samples at a time. 
 In some cases, such as method of adjustment, where the audio is processed by the user \cite{bech}, or AB test \cite{bech}, where the interface does not show all audio samples to be evaluated at once, the back end of the tool needs to be modified as well. 
 
 There are a number of advantages to building a web audio based listening test environment. The ability to easily deploy a flexible and scalable testing environment that requires no proprietary software to run makes the web audio evaluation tool a very flexible testing tool. The ability to host a single test server not only allows multiple participants to be involved in a trial simultaneously, but also permits participants to be located anywhere in the world. There are also less user experience issues, since all users should have some experience with using existing web technologies.
 
+
 In the following sections, we describe the included interface in more detail, discuss the implementation, and cover considerations that were made in the design process of this tool. 
+
+%\section{Requirements}\label{sec:requirements}
+%???
+%
+%\begin{itemize}
+%\item 
+%\end{itemize}
+
 
 \section{Interface}\label{sec:interface}
 
 At this point, we have implemented the interface of the MATLAB-based APE (Audio Perceptual Evaluation) toolbox \cite{deman2014b}. This shows one marker for each simultaneously evaluated audio fragment on one or more horizontal axes, that can be moved to rate or rank the respective fragments in terms of any subjective quality, as well as a comment box for every marker, and any extra text boxes for extra comments. 
 The reason for such an interface, where all stimuli are presented on a single rating axis (or multiple axes if multiple subjective qualities need to be evaluated), is that it urges the subject to consider the rating and/or ranking of the stimuli relative to one another, as opposed to comparing each individual stimulus to a given reference, as is the case with e.g. a MUSHRA test \cite{mushra}. See Figure \ref{fig:interface} for an example of the interface, with eleven fragments and one axis. %? change if a new interface is shown

 \item \textbf{Lowest rating below [value]}: To enforce a certain use of the rating scale, it can be required to rate at least one sample below a specified value. 
 \item \textbf{Highest rating above [value]}: To enforce a certain use of the rating scale, it can be required to rate at least one sample above a specified value. 
 \item \textbf{Reference}: Allows for a separate sample (outside of the axis) to be the `reference', which the subject can play back during the test to help with the task at hand \cite{mushra}. 
 \item \textbf{Hidden reference}: Whether or not an explicit `reference' is provided, the `hidden reference' should be rated above a certain value \cite{mushra} - this can be enforced. 
 \item \textbf{Hidden anchor}: The `hidden anchor' should be rated lower than a certain value \cite{mushra} - this can be enforced. 
+\item \textbf{Show scrub bar}: Display a playhead on a scrub bar to show the position in the current fragment. 
+\item \textbf{Drag playhead}: If scrub bar is visible, allow dragging to move back or forward in a fragment. 
 \end{itemize}
 
 When one of these options is not included in the set up file, they assume a default value. As a result, the input file can be kept very compact if default values suffice for the test.  
 
 % loop, snap to corresponding position, comments, 'general' comment, require same sampling rate, different types of randomisation