Mercurial > hg > webaudioevaluationtool
changeset 94:550048cf9495
Paper: Setup, output, conclusion, bibliography
author | Brecht De Man <b.deman@qmul.ac.uk> |
---|---|
date | Mon, 27 Apr 2015 22:00:22 +0100 |
parents | caf1ac4a80b5 |
children | 6b35f18fcae6 |
files | docs/SMC15/smc2015template.bbl docs/SMC15/smc2015template.bib docs/SMC15/smc2015template.tex |
diffstat | 3 files changed, 150 insertions(+), 151 deletions(-) [+] |
line wrap: on
line diff
--- a/docs/SMC15/smc2015template.bbl Mon Apr 27 20:43:08 2015 +0100 +++ b/docs/SMC15/smc2015template.bbl Mon Apr 27 22:00:22 2015 +0100 @@ -1,129 +1,129 @@ -% Generated by IEEEtran.bst, version: 1.12 (2007/01/11) -\begin{thebibliography}{10} -\providecommand{\url}[1]{#1} -\csname url@samestyle\endcsname -\providecommand{\newblock}{\relax} -\providecommand{\bibinfo}[2]{#2} -\providecommand{\BIBentrySTDinterwordspacing}{\spaceskip=0pt\relax} -\providecommand{\BIBentryALTinterwordstretchfactor}{4} -\providecommand{\BIBentryALTinterwordspacing}{\spaceskip=\fontdimen2\font plus -\BIBentryALTinterwordstretchfactor\fontdimen3\font minus - \fontdimen4\font\relax} -\providecommand{\BIBforeignlanguage}[2]{{% -\expandafter\ifx\csname l@#1\endcsname\relax -\typeout{** WARNING: IEEEtran.bst: No hyphenation pattern has been}% -\typeout{** loaded for the language `#1'. Using the pattern for}% -\typeout{** the default language instead.}% -\else -\language=\csname l@#1\endcsname -\fi -#2}} -\providecommand{\BIBdecl}{\relax} -\BIBdecl - -\bibitem{schoeffler2013impact} -M.~Schoeffler and J.~Herre, ``About the impact of audio quality on overall - listening experience,'' in \emph{Proceedings of Sound and Music Computing - Conference}, 2013, pp. 48--53. - -\bibitem{repp} -R.~Repp, ``Recording quality ratings by music professionals,'' in - \emph{Proceedings of the 2006 International Computer Music Conference}, 2006, - pp. 468--474. - -\bibitem{de2013real} -A.~de~G{\"o}tzen, E.~Sikstr{\"o}m, F.~Grani, and S.~Serafin, ``Real, foley or - synthetic? {A}n evaluation of everyday walking sounds,'' in \emph{Proceedings - of SMC 2013 : 10th Sound and Music Computing Conference}, 2013. - -\bibitem{durr2015implementation} -G.~Durr, L.~Peixoto, M.~Souza, R.~Tanoue, and J.~D. Reiss, ``Implementation and - evaluation of dynamic level of audio detail,'' in \emph{Audio Engineering - Society Conference: 56th International Conference: Audio for Games}, 2015. - -\bibitem{deman2014a} -B.~De~Man and J.~D. Reiss, ``Adaptive control of amplitude distortion - effects,'' in \emph{53rd Conference of the Audio Engineering Society}, 2014. - -\bibitem{mushram} -E.~Vincent, M.~G. Jafari, and M.~D. Plumbley, ``Preliminary guidelines for - subjective evalutation of audio source separation algorithms,'' in \emph{UK - ICA Research Network Workshop}, 2006. - -\bibitem{uhlereiss} -J.~D. Reiss and C.~Uhle, ``Determined source separation for microphone - recordings using {IIR} filters,'' in \emph{129th Convention of the Audio - Engineering Society}.\hskip 1em plus 0.5em minus 0.4em\relax Audio - Engineering Society, 2010. - -\bibitem{song2013b} -Y.~Song, S.~Dixon, and A.~R. Halpern, ``Do online social tags predict perceived - or induced emotional responses to music?'' in \emph{Proceedings of the 14th - International Society for Music Information Retrieval (ISMIR2013)}, 2013. - -\bibitem{song2013a} -Y.~Song, S.~Dixon, M.~T. Pearce, and G.~Fazekas, ``Using tags to select stimuli - in the study of music and emotion,'' \emph{Proceedings of the 3rd - International Conference on Music \& Emotion (ICME)}, 2013. - -\bibitem{friberg2011comparison} -A.~Friberg and A.~Hedblad, ``A comparison of perceptual ratings and computed - audio features,'' in \emph{Proceedings of the SMC2011, 8th Sound and Music - Computing Conference}, 2011, pp. 122--127. - -\bibitem{deman2014b} -B.~De~Man and J.~D. Reiss, ``{APE}: {A}udio {P}erceptual {E}valuation toolbox - for {MATLAB},'' in \emph{136th Convention of the Audio Engineering Society}, - 2014. - -\bibitem{beaqlejs} -S.~Kraft and U.~Z{\"o}lzer, ``{BeaqleJS}: {HTML5} and {JavaScript} based - framework for the subjective evaluation of audio quality,'' in \emph{Linux - Audio Conference, Karlsruhe, DE}, 2014. - -\bibitem{hulti-gen} -C.~Gribben and H.~Lee, ``Toward the development of a universal listening test - interface generator in max,'' in \emph{138th Convention of the Audio - Engineering Society}, 2015. - -\bibitem{scale} -A.~V. Giner, ``Scale - a software tool for listening experiments,'' in - \emph{AIA/DAGA Conference on Acoustics, Merano (Italy)}, 2013. - -\bibitem{whisper} -S.~Ciba, A.~Wlodarski, and H.-J. Maempel, ``Whis{PER} -- {A} new tool for - performing listening tests,'' in \emph{126th Convention of the Audio - Engineering Society}, 2009. - -\bibitem{opaque} -J.~Berg, ``{OPAQUE} -- {A} tool for the elicitation and grading of audio - quality attributes,'' in \emph{118th Convention of the Audio Engineering - Society}, 2005. - -\bibitem{guineapig} -J.~Hynninen and N.~Zacharov, ``{GuineaPig} - {A} generic subjective test system - for multichannel audio,'' in \emph{106th Convention of the Audio Engineering - Society Convention}, 1999. - -\bibitem{mushra} -\emph{Method for the subjective assessment of intermediate quality level of - coding systems}.\hskip 1em plus 0.5em minus 0.4em\relax Recommendation {ITU-R - BS.1534-1}, 2003. - -\bibitem{mason2015compression} -A.~Mason, N.~Jillings, Z.~Ma, J.~D. Reiss, and F.~Melchior, ``Adaptive audio - reproduction using personalized compression,'' in \emph{Audio Engineering - Society Conference: 57th International Conference: The Future of Audio - Entertainment Technology -- Cinema, Television and the Internet}, 2015. - -\bibitem{bech} -S.~Bech and N.~Zacharov, \emph{Perceptual Audio Evaluation - Theory, Method and - Application}.\hskip 1em plus 0.5em minus 0.4em\relax John Wiley \& Sons, - 2007. - -\bibitem{deman2015a} -B.~De~Man, M.~Boerum, B.~Leonard, G.~Massenburg, R.~King, and J.~D. Reiss, - ``Perceptual evaluation of music mixing practices,'' in \emph{138th - Convention of the Audio Engineering Society}, 2015. - -\end{thebibliography} +% Generated by IEEEtran.bst, version: 1.12 (2007/01/11) +\begin{thebibliography}{10} +\providecommand{\url}[1]{#1} +\csname url@samestyle\endcsname +\providecommand{\newblock}{\relax} +\providecommand{\bibinfo}[2]{#2} +\providecommand{\BIBentrySTDinterwordspacing}{\spaceskip=0pt\relax} +\providecommand{\BIBentryALTinterwordstretchfactor}{4} +\providecommand{\BIBentryALTinterwordspacing}{\spaceskip=\fontdimen2\font plus +\BIBentryALTinterwordstretchfactor\fontdimen3\font minus + \fontdimen4\font\relax} +\providecommand{\BIBforeignlanguage}[2]{{% +\expandafter\ifx\csname l@#1\endcsname\relax +\typeout{** WARNING: IEEEtran.bst: No hyphenation pattern has been}% +\typeout{** loaded for the language `#1'. Using the pattern for}% +\typeout{** the default language instead.}% +\else +\language=\csname l@#1\endcsname +\fi +#2}} +\providecommand{\BIBdecl}{\relax} +\BIBdecl + +\bibitem{schoeffler2013impact} +M.~Schoeffler and J.~Herre, ``About the impact of audio quality on overall + listening experience,'' in \emph{Proceedings of the 10th Sound and Music + Computing Conference}, 2013, pp. 48--53. + +\bibitem{repp} +R.~Repp, ``Recording quality ratings by music professionals,'' in + \emph{Proceedings of the 2006 International Computer Music Conference}, 2006, + pp. 468--474. + +\bibitem{de2013real} +A.~de~G{\"o}tzen, E.~Sikstr{\"o}m, F.~Grani, and S.~Serafin, ``Real, foley or + synthetic? {A}n evaluation of everyday walking sounds,'' in \emph{Proceedings + of SMC 2013 : 10th Sound and Music Computing Conference}, 2013. + +\bibitem{durr2015implementation} +G.~Durr, L.~Peixoto, M.~Souza, R.~Tanoue, and J.~D. Reiss, ``Implementation and + evaluation of dynamic level of audio detail,'' in \emph{Audio Engineering + Society Conference: 56th International Conference: Audio for Games}, 2015. + +\bibitem{deman2014a} +B.~De~Man and J.~D. Reiss, ``Adaptive control of amplitude distortion + effects,'' in \emph{Audio Engineering Society Conference: 53rd International + Conference: Semantic Audio}, 2014. + +\bibitem{mushram} +E.~Vincent, M.~G. Jafari, and M.~D. Plumbley, ``Preliminary guidelines for + subjective evalutation of audio source separation algorithms,'' in \emph{UK + ICA Research Network Workshop}, 2006. + +\bibitem{uhlereiss} +J.~D. Reiss and C.~Uhle, ``Determined source separation for microphone + recordings using {IIR} filters,'' in \emph{129th Convention of the Audio + Engineering Society}, 2010. + +\bibitem{song2013b} +Y.~Song, S.~Dixon, and A.~R. Halpern, ``Do online social tags predict perceived + or induced emotional responses to music?'' in \emph{Proceedings of the 14th + International Society for Music Information Retrieval (ISMIR2013)}, 2013. + +\bibitem{song2013a} +Y.~Song, S.~Dixon, M.~T. Pearce, and G.~Fazekas, ``Using tags to select stimuli + in the study of music and emotion,'' \emph{Proceedings of the 3rd + International Conference on Music \& Emotion (ICME)}, 2013. + +\bibitem{friberg2011comparison} +A.~Friberg and A.~Hedblad, ``A comparison of perceptual ratings and computed + audio features,'' in \emph{Proceedings of the 8th Sound and Music Computing + Conference}, 2011, pp. 122--127. + +\bibitem{deman2014b} +B.~De~Man and J.~D. Reiss, ``{APE}: {A}udio {P}erceptual {E}valuation toolbox + for {MATLAB},'' in \emph{136th Convention of the Audio Engineering Society}, + 2014. + +\bibitem{beaqlejs} +S.~Kraft and U.~Z{\"o}lzer, ``{BeaqleJS}: {HTML5} and {JavaScript} based + framework for the subjective evaluation of audio quality,'' in \emph{Linux + Audio Conference, Karlsruhe, DE}, 2014. + +\bibitem{hulti-gen} +C.~Gribben and H.~Lee, ``Toward the development of a universal listening test + interface generator in {Max},'' in \emph{138th Convention of the Audio + Engineering Society}, 2015. + +\bibitem{scale} +A.~V. Giner, ``Scale - a software tool for listening experiments,'' in + \emph{AIA/DAGA Conference on Acoustics, Merano (Italy)}, 2013. + +\bibitem{whisper} +S.~Ciba, A.~Wlodarski, and H.-J. Maempel, ``Whis{PER} -- {A} new tool for + performing listening tests,'' in \emph{126th Convention of the Audio + Engineering Society}, 2009. + +\bibitem{opaque} +J.~Berg, ``{OPAQUE} -- {A} tool for the elicitation and grading of audio + quality attributes,'' in \emph{118th Convention of the Audio Engineering + Society}, 2005. + +\bibitem{guineapig} +J.~Hynninen and N.~Zacharov, ``{GuineaPig} - {A} generic subjective test system + for multichannel audio,'' in \emph{106th Convention of the Audio Engineering + Society}, 1999. + +\bibitem{mushra} +\emph{Method for the subjective assessment of intermediate quality level of + coding systems}.\hskip 1em plus 0.5em minus 0.4em\relax Recommendation {ITU-R + BS.1534-1}, 2003. + +\bibitem{mason2015compression} +A.~Mason, N.~Jillings, Z.~Ma, J.~D. Reiss, and F.~Melchior, ``Adaptive audio + reproduction using personalized compression,'' in \emph{Audio Engineering + Society Conference: 57th International Conference: The Future of Audio + Entertainment Technology -- Cinema, Television and the Internet}, 2015. + +\bibitem{bech} +S.~Bech and N.~Zacharov, \emph{Perceptual Audio Evaluation - Theory, Method and + Application}.\hskip 1em plus 0.5em minus 0.4em\relax John Wiley \& Sons, + 2007. + +\bibitem{deman2015a} +B.~De~Man, M.~Boerum, B.~Leonard, G.~Massenburg, R.~King, and J.~D. Reiss, + ``Perceptual evaluation of music mixing practices,'' in \emph{138th + Convention of the Audio Engineering Society}, 2015. + +\end{thebibliography}
--- a/docs/SMC15/smc2015template.bib Mon Apr 27 20:43:08 2015 +0100 +++ b/docs/SMC15/smc2015template.bib Mon Apr 27 22:00:22 2015 +0100 @@ -37,7 +37,7 @@ @inproceedings{friberg2011comparison, title={A comparison of perceptual ratings and computed audio features}, author={Friberg, Anders and Hedblad, Anton}, - booktitle={Proceedings of the SMC2011, 8th Sound and Music Computing Conference}, + booktitle={Proceedings of the 8th Sound and Music Computing Conference}, pages={122--127}, year={2011} } @@ -58,14 +58,13 @@ @inproceedings{uhlereiss, Author = {Reiss, Joshua D. and Uhle, Christian}, Booktitle = {129th Convention of the Audio Engineering Society}, - Organization = {Audio Engineering Society}, Title = {Determined Source Separation for Microphone Recordings Using {IIR} Filters}, Year = {2010}} @conference{deman2014a, Author = {De Man, Brecht and Joshua D. Reiss}, - Booktitle = {53rd Conference of the Audio Engineering Society}, + Booktitle = {Audio Engineering Society Conference: 53rd International Conference: Semantic Audio}, Title = {Adaptive Control of Amplitude Distortion Effects}, Year = {2014}} @@ -101,7 +100,7 @@ @inproceedings{schoeffler2013impact, Author = {Schoeffler, Michael and Herre, J{\"u}rgen}, - Booktitle = {Proceedings of Sound and Music Computing Conference}, + Booktitle = {Proceedings of the 10th Sound and Music Computing Conference}, Pages = {48--53}, Title = {About the Impact of Audio Quality on Overall Listening Experience}, Year = {2013}} @@ -128,12 +127,12 @@ @conference{hulti-gen, Author = {Christopher Gribben and Hyunkook Lee}, Booktitle = {138th Convention of the Audio Engineering Society}, - Title = {Toward the Development of a Universal Listening Test Interface Generator in Max}, + Title = {Toward the Development of a Universal Listening Test Interface Generator in {Max}}, Year = {2015}} @conference{guineapig, Author = {Hynninen, Jussi and Zacharov, Nick}, - Booktitle = {106th Convention of the Audio Engineering Society Convention}, + Booktitle = {106th Convention of the Audio Engineering Society}, Title = {{GuineaPig} - {A} generic subjective test system for multichannel audio}, Year = {1999}}
--- a/docs/SMC15/smc2015template.tex Mon Apr 27 20:43:08 2015 +0100 +++ b/docs/SMC15/smc2015template.tex Mon Apr 27 22:00:22 2015 +0100 @@ -256,7 +256,7 @@ \item \texttt{ape.js}: Parses setup files to create the interface as instructed, following the same style chain as the MATLAB APE Tool \cite{deman2014b}. \end{itemize} -The HTML file loads the \texttt{core.js} file along with a few other ancillary files (such as the jQuery JavaScript extensions\footnote{http://jquery.com/}), at which point the browser JavaScript begins to execute the on-page instructions, which gives the URL of the test set up XML document (outlined in Section \ref{sec:setupresultsformats}). \texttt{core.js} parses this document and executes the functions in \texttt{ape.js} to build the web page. The reason for separating these two files is to allow for further interface designs (such as MUSHRA \cite{mushra} or AB tests \cite{bech}) to be used, which would still require the same underlying core functions outlined in \texttt{core.js}. +The HTML file loads the \texttt{core.js} file along with a few other ancillary files (such as the jQuery JavaScript extensions\footnote{http://jquery.com/}), at which point the browser JavaScript begins to execute the on-page instructions, which gives the URL of the test setup XML document (outlined in Section \ref{sec:setupresultsformats}). \texttt{core.js} parses this document and executes the functions in \texttt{ape.js} to build the web page. The reason for separating these two files is to allow for further interface designs (such as MUSHRA \cite{mushra} or AB tests \cite{bech}) to be used, which would still require the same underlying core functions outlined in \texttt{core.js}. The \texttt{ape.js} file has several main functions but the most important are documented here. \textit{loadInterface(xmlDoc)} is called to decode the supplied project document in respect for the interface specified and define any global structures (such as the slider interface). It also identifies the number of pages in the test and randomises the order, if specified to do so. This is the only mandatory function in any of the interface files as this is called by \texttt{core.js} when the document is ready. \texttt{core.js} cannot 'see' any interface specific functions and therefore cannot assume any are available. Therefore \textit{loadInterface(xmlDoc)} is essential to set up the entire test environment. Because the interface files are loaded by \texttt{core.js} and because the functions in \texttt{core.js} are global, the interface files can `see' the \texttt{core.js} file and can therefore not only interact with it, but also modify it. @@ -273,7 +273,7 @@ Browsers support various audio file formats and are not consistent in any format. Currently the Web Audio API is best supported in Chrome, Firefox, Opera and Safari. All of these support the use of the uncompressed WAV format. Although not a compact, web friendly format, most transport systems are of a high enough bandwidth this should not be a problem. Ogg Vorbis is another well supported format across the four supported major desktop browsers, as well as MP3 (although Firefox may not support all MP3 types) \footnote{https://developer.mozilla.org/en-US/docs/Web/HTML/\\Supported\_media\_formats}. %https://developer.mozilla.org/en-US/docs/Web/HTML/Supported_media_formats One issue of the Web Audio API is that the sample rate is assigned by the system sound device, rather than requested and does not have the ability to request a different one. % Does this make sense? The problem is across all audio files. -As the sampling rate and the effect of resampling may be critical for some listening tests, the default operation when an audio file is loaded with a different sample rate to that of the system is to convert the sample rate. To provide a check for this, the desired sample rate can be supplied with the set up XML and checked against. If the sample rates do not match, a browser alert window is shown asking for the sample rate to be correctly adjusted. +As the sampling rate and the effect of resampling may be critical for some listening tests, the default operation when an audio file is loaded with a different sample rate to that of the system is to convert the sample rate. To provide a check for this, the desired sample rate can be supplied with the setup XML and checked against. If the sample rates do not match, a browser alert window is shown asking for the sample rate to be correctly adjusted. This happens before any loading or decoding of audio files so the browser will only be instructed to fetch files if the system sample rate meets the requirements, avoiding multiple requests for large files until they are actually needed. %During playback, the playback nodes loop indefinitely until playback is stopped. The gain nodes in the \textit{audioObject}s enable dynamic muting of nodes. When a bar in the sliding ranking is clicked, the audio engine mutes all \textit{audioObject}s and un-mutes the clicked one. Therefore, if the audio samples are perfectly aligned up and of the same sample length, they will remain perfectly aligned with each other. @@ -282,21 +282,21 @@ \section{Input and result files}\label{sec:setupresultsformats} -The set up and result files both use the common XML document format to outline the various parameters. The set up file determines which interface to use, the location of audio files, how many pages and other parameters to define the testing environment. Having one document to modify allows for quick manipulation in a `human readable' form to create new tests, or adjust current ones, without needing to edit multiple web files. An example of this XML document is presented in Figure~\ref{fig:xmlIn}% I mean the .js and .html files, though not sure if any better. +The setup and result files both use the common XML document format to outline the various parameters. The setup file determines the interface to use, the location of audio files, the number of pages and other parameters to define the testing environment. Having one document to modify allows for quick manipulation in a `human readable' form to create new tests, or adjust current ones, without needing to edit multiple web files. Furthermore, we also provide a simple web page to enter all these settings without needing to manipulate the raw XML. An example of this XML document is presented in Figure~\ref{fig:xmlIn}. % I mean the .js and .html files, though not sure if any better. -\subsection{Set up and configurability} +\subsection{Setup and configurability} \begin{figure}[ht] \begin{center} \includegraphics[width=0.5\textwidth]{XMLInput2.png} -\caption{An Example Input XML File} +\caption{An example input XML file} \label{fig:xmlIn} \end{center} \end{figure} -The set up document has several defined nodes and structure which are documented with the source code. For example there is a section for general set up options where any pre-test and post-test questions and statements can be defined. Pre- and post-test dialogue boxes allow for comments or questions to be presented before or after the test, to convey listening test instructions, gather information about the subject, listening environment, and overall experience of the test. From the example in Figure~\ref{fig:xmlIn}, it can be seen that a question box should be generated, with the id 'location' and it is mandatory to answer. The question is in the PreTest node meaning it will appear before any testing will begin. When the result for the entire test is shown, the response will appear in the PreTest node with the id 'location' allowing it to be found easily. This outlines the importance of having clear and meaningful ID values. +The setup document has several defined nodes and structure which are documented with the source code. For example, there is a section for general setup options where any pre-test and post-test questions and statements can be defined. Pre- and post-test dialogue boxes allow for comments or questions to be presented before or after the test, to convey listening test instructions, and gather information about the subject, listening environment, and overall experience of the test. In the example in Figure~\ref{fig:xmlIn}, a question box with the id `location' is added, which is set to be mandatory to answer. The question is in the PreTest node meaning it will appear before any testing will begin. When the result for the entire test is shown, the response will appear in the PreTest node with the id `location' allowing it to be found easily, provided the id values are meaningful. -We try to cater to a diverse audience with this toolbox, while ensuring the toolbox is simple, elegant and straightforward. To that end, we include the following options that can be easily switched on and off, by setting the value in the input XML file. +We try to cater to a diverse audience with this toolbox, while ensuring it is simple, elegant and straightforward. To that end, we currently include the following options that can be easily switched on and off, by setting the value in the input XML file. \begin{itemize} %Should have used a description list for this. \item \textbf{Snap to corresponding position}: When this is enabled, and a fragment is playing, the playhead skips to the same position in the next fragment that is clicked. If it is not enabled, every fragment is played from the start. @@ -310,7 +310,7 @@ \item \textbf{Require full playback}: If `Require playback' is active, require that each fragment has been played in full. \item \textbf{Require moving}: Require that each marker is moved (dragged) at least once. \item \textbf{Require comments}: This option allows requiring the subject to require a comment for each track. -\item \textbf{Repeat test}: Number of times test should be repeated (none by default), to allow familiarisation with the content and experiment, and to investigate consistency of user and variability due to familiarity. In the set up, each 'page' can be given a repeat count. These are all gathered before shuffling the order so repeated tests are not back-to-back if possible. +\item \textbf{Repeat test}: Number of times each page in the test should be repeated (none by default), to allow familiarisation with the content and experiment, and to investigate consistency of user and variability due to familiarity. In the setup, each 'page' can be given a repeat count. These are all gathered before shuffling the order so repeated tests are not back-to-back if possible. \item \textbf{Returning to previous pages}: Indicates whether it is possible to go back to a previous `page' in the test. \item \textbf{Lowest rating below [value]}: To enforce a certain use of the rating scale, it can be required to rate at least one sample below a specified value. \item \textbf{Highest rating above [value]}: To enforce a certain use of the rating scale, it can be required to rate at least one sample above a specified value. @@ -321,26 +321,26 @@ \item \textbf{Drag playhead}: If scrub bar is visible, allow dragging to move back or forward in a fragment. \end{itemize} -When one of these options is not included in the set up file, they assume a default value. As a result, the input file can be kept very compact if default values suffice for the test. +When one of these options is not included in the setup file, they assume a default value. As a result, the input file can be kept very compact if default values suffice for the test. % loop, snap to corresponding position, comments, 'general' comment, require same sampling rate, different types of randomisation \subsection{Results} -The results file is dynamically generated by the interface upon clicking the `Submit' button. This also executes checks, depending on the set up file, to ensure that all tracks have been played back, rated and commented on. The XML output returned contains a node per audioObject and contains both the corresponding marker's position and any comments written in the associated comment box. The rating returned is normalised to be a value between 0 and 1, normalising the pixel representation of different browser windows. An example output file is presented in Figure~\ref{fig:xmlOut} +The results file is dynamically generated by the interface upon clicking the `Submit' button. This also executes checks, depending on the setup file, to ensure that all tracks have been played back, rated and commented on. The XML output returned contains a node per audioObject and contains both the corresponding marker's position and any comments written in the associated comment box. The rating returned is normalised to be a value between 0 and 1, normalising the pixel representation of different browser windows. An example output file is presented in Figure~\ref{fig:xmlOut}. \begin{figure}[ht] \begin{center} \includegraphics[width=0.5\textwidth]{XMLOutput2.png} -\caption{An Example Output XML File} +\caption{An example output XML file} \label{fig:xmlOut} \end{center} \end{figure} -The results also contain information collected by any defined pre/post questions. These are referenced against the set up XML by using the same ID so readable responses can be obtained. Taking from the earlier example of setting up a pre-test question, an example response can be seen in Figure \ref{fig:xmlOut}. +The results also contain information collected by any defined pre/post questions. These are referenced against the setup XML by using the same ID so readable responses can be obtained. Taking from the earlier example of setting up a pre-test question, an example response can be seen in Figure \ref{fig:xmlOut}. Each page of testing is returned with the results of the entire page included in the structure. One `audioElement' node is created per audio fragment per page, along with its ID. This includes several child nodes including the rating between 0 and 1, the comment, and any other collected metrics including how long the element was listened for, the initial position, boolean flags if the element was listened to, if the element was moved and if the element comment box had any comment. Furthermore, each user action (manipulation of any interface element, such as playback or moving a marker) can be logged along with a the corresponding time code. -Furthermore, we also store session data such as the browser the tool was used in. +We also store session data such as the browser the tool was used in. We provide the option to store the results locally, and/or to have them sent to a server. @@ -372,15 +372,15 @@ %</metric> \\ %</audioelement>} -The parent tag \texttt{audioelement} holds the ID of the element passed in from the set up document. The first child element is \texttt{comment} and holds both the question shown and the response from the comment box inside. -The child element \texttt{value} holds the normalised ranking value. Next comes the metric node structure, there is one metric result node per metric event collected. The id of the node identifies the type of data it contains. For example, the first holds the id \textit{elementTimer} and the data contained represents how long, in seconds, the audio element was listened to. There is one \texttt{audioelement} tag per audio element on each test page. +The parent tag \texttt{audioelement} holds the ID of the element passed in from the setup document. The first child element is \texttt{comment} and holds both the question shown and the response from the comment box inside. +The child element \texttt{value} holds the normalised ranking value. Next comes the metric node structure, with one metric result node per metric event collected. The id of the node identifies the type of data it contains. For example, the first holds the id \textit{elementTimer} and the data contained represents how long, in seconds, the audio element was listened to. There is one \texttt{audioelement} tag per audio element on each test page. \section{Conclusions and future work}\label{sec:conclusions} In this paper we have presented an approach to creating a browser-based listening test environment that can be used for a variety of types of perceptual evaluation of audio. Specifically, we discussed the use of the toolbox in the context of assessment of preference for different production practices, with identical source material. -The purpose of this paper is to outline the design of this tool, to describe our implementation using basic HTML5 functionality, and to discuss design challenges and limitations of our approach. This tool differentiates itself from other perceptual audio tools by enabling web technologies for multiple participants to perform the test without the need for proprietary software such as MATLAB. The tool also allows for any interface to be built using HTML5 elements to create dynamic interfaces built either on current evaluation techniques or newer techniques. It enables quick set up of simple tests with the ability to manage complex tests through a single file. And finally it uses the XML document format to store the results allowing for processing and analysis of results in various third party software such as MATLAB or Python. +The purpose of this paper is to outline the design of this tool, to describe our implementation using basic HTML5 functionality, and to discuss design challenges and limitations of our approach. This tool differentiates itself from other perceptual audio tools by enabling web technologies for multiple participants to perform the test without the need for proprietary software such as MATLAB. The tool also allows for any interface to be built using HTML5 elements to create a variety of dynamic, multiple-stimulus listening test interfaces. It enables quick setup of simple tests with the ability to manage complex tests through a single file. Finally it uses the XML document format to store the results allowing for processing and analysis of results in various third party software such as MATLAB or Python. % future work Further work may include the development of other common test designs, such as MUSHRA \cite{mushra}, 2D valence and arousal rating, and others. We will add functionality to assist with setting up large-scale tests with remote subjects, so this becomes straightforward and intuitive.