Mercurial > hg > webaudioevaluationtool

--- a/docs/SMC15/smc2015template.bbl	Mon Apr 27 16:31:46 2015 +0100
+++ b/docs/SMC15/smc2015template.bbl	Mon Apr 27 16:43:50 2015 +0100
@@ -1,111 +1,121 @@
-% Generated by IEEEtran.bst, version: 1.12 (2007/01/11)
-\begin{thebibliography}{10}
-\providecommand{\url}[1]{#1}
-\csname url@samestyle\endcsname
-\providecommand{\newblock}{\relax}
-\providecommand{\bibinfo}[2]{#2}
-\providecommand{\BIBentrySTDinterwordspacing}{\spaceskip=0pt\relax}
-\providecommand{\BIBentryALTinterwordstretchfactor}{4}
-\providecommand{\BIBentryALTinterwordspacing}{\spaceskip=\fontdimen2\font plus
-\BIBentryALTinterwordstretchfactor\fontdimen3\font minus
-  \fontdimen4\font\relax}
-\providecommand{\BIBforeignlanguage}[2]{{%
-\expandafter\ifx\csname l@#1\endcsname\relax
-\typeout{** WARNING: IEEEtran.bst: No hyphenation pattern has been}%
-\typeout{** loaded for the language `#1'. Using the pattern for}%
-\typeout{** the default language instead.}%
-\else
-\language=\csname l@#1\endcsname
-\fi
-#2}}
-\providecommand{\BIBdecl}{\relax}
-\BIBdecl
-
-\bibitem{schoeffler2013impact}
-M.~Schoeffler and J.~Herre, ``About the impact of audio quality on overall
-  listening experience,'' in \emph{Proceedings of Sound and Music Computing
-  Conference}, 2013, pp. 48--53.
-
-\bibitem{repp}
-R.~Repp, ``Recording quality ratings by music professionals,'' in
-  \emph{Proceedings of the 2006 International Computer Music Conference}, 2006,
-  pp. 468--474.
-
-\bibitem{de2013real}
-A.~de~G{\"o}tzen, E.~Sikstr{\"o}m, F.~Grani, and S.~Serafin, ``Real, foley or
-  synthetic? {A}n evaluation of everyday walking sounds,'' in \emph{Proceedings
-  of SMC 2013 : 10th Sound and Music Computing Conference}, 2013.
-
-\bibitem{durr2015implementation}
-G.~Durr, L.~Peixoto, M.~Souza, R.~Tanoue, and J.~D. Reiss, ``Implementation and
-  evaluation of dynamic level of audio detail,'' in \emph{Audio Engineering
-  Society Conference: 56th International Conference: Audio for Games}, February
-  2015.
-
-\bibitem{deman2014a}
-B.~De~Man and J.~D. Reiss, ``Adaptive control of amplitude distortion
-  effects,'' in \emph{53rd Conference of the Audio Engineering Society},
-  January 2014.
-
-\bibitem{uhlereiss}
-J.~D. Reiss and C.~Uhle, ``Determined source separation for microphone
-  recordings using {IIR} filters,'' in \emph{129th Convention of the Audio
-  Engineering Society}.\hskip 1em plus 0.5em minus 0.4em\relax Audio
-  Engineering Society, November 2010.
-
-\bibitem{song2013b}
-Y.~Song, S.~Dixon, and A.~R. Halpern, ``Do online social tags predict perceived
-  or induced emotional responses to music?'' in \emph{Proceedings of the 14th
-  International Society for Music Information Retrieval (ISMIR2013)}, 2013.
-
-\bibitem{song2013a}
-Y.~Song, S.~Dixon, M.~T. Pearce, and G.~Fazekas, ``Using tags to select stimuli
-  in the study of music and emotion,'' \emph{Proceedings of the 3rd
-  International Conference on Music \& Emotion (ICME)}, 2013.
-
-\bibitem{friberg2011comparison}
-A.~Friberg and A.~Hedblad, ``A comparison of perceptual ratings and computed
-  audio features,'' in \emph{Proceedings of the SMC2011, 8th Sound and Music
-  Computing Conference}, 2011, pp. 122--127.
-
-\bibitem{mason2015compression}
-A.~Mason, N.~Jillings, Z.~Ma, J.~D. Reiss, and F.~Melchior, ``Adaptive audio
-  reproduction using personalized compression,'' in \emph{Audio Engineering
-  Society Conference: 57th International Conference: The Future of Audio
-  Entertainment Technology -- Cinema, Television and the Internet}, March 2015.
-
-\bibitem{deman2014b}
-B.~De~Man and J.~D. Reiss, ``{APE}: {A}udio {P}erceptual {E}valuation toolbox
-  for {MATLAB},'' in \emph{136th Convention of the Audio Engineering Society},
-  April 2014.
-
-\bibitem{mushram}
-E.~Vincent, M.~G. Jafari, and M.~D. Plumbley, ``Preliminary guidelines for
-  subjective evalutation of audio source separation algorithms,'' in \emph{UK
-  ICA Research Network Workshop}, 2006.
-
-\bibitem{whisper}
-S.~Ciba, A.~Wlodarski, and H.-J. Maempel, ``Whis{PER} -- a new tool for
-  performing listening tests,'' in \emph{126th Convention of the Audio
-  Engineering Society}, May 7-10 2009.
-
-\bibitem{scale}
-A.~V. Giner, ``Scale - a software tool for listening experiments,'' in
-  \emph{AIA/DAGA Conference on Acoustics, Merano (Italy)}, 2013.
-
-\bibitem{beaqlejs}
-S.~Kraft and U.~Z{\"o}lzer, ``{BeaqleJS}: {HTML5} and {JavaScript} based
-  framework for the subjective evaluation of audio quality,'' in \emph{Linux
-  Audio Conference, Karlsruhe, DE}, 2014.
-
-\bibitem{mushra}
-\emph{Method for the subjective assessment of intermediate quality level of
-  coding systems}.\hskip 1em plus 0.5em minus 0.4em\relax Recommendation {ITU-R
-  BS.1534-1}, 2003.
-
-\bibitem{bech}
-S.~Bech and N.~Zacharov, \emph{Perceptual Audio Evaluation - Theory, Method and
-  Application}.\hskip 1em plus 0.5em minus 0.4em\relax John Wiley \& Sons,
-  2007.
-
-\end{thebibliography}
+% Generated by IEEEtran.bst, version: 1.12 (2007/01/11)
+\begin{thebibliography}{10}
+\providecommand{\url}[1]{#1}
+\csname url@samestyle\endcsname
+\providecommand{\newblock}{\relax}
+\providecommand{\bibinfo}[2]{#2}
+\providecommand{\BIBentrySTDinterwordspacing}{\spaceskip=0pt\relax}
+\providecommand{\BIBentryALTinterwordstretchfactor}{4}
+\providecommand{\BIBentryALTinterwordspacing}{\spaceskip=\fontdimen2\font plus
+\BIBentryALTinterwordstretchfactor\fontdimen3\font minus
+  \fontdimen4\font\relax}
+\providecommand{\BIBforeignlanguage}[2]{{%
+\expandafter\ifx\csname l@#1\endcsname\relax
+\typeout{** WARNING: IEEEtran.bst: No hyphenation pattern has been}%
+\typeout{** loaded for the language `#1'. Using the pattern for}%
+\typeout{** the default language instead.}%
+\else
+\language=\csname l@#1\endcsname
+\fi
+#2}}
+\providecommand{\BIBdecl}{\relax}
+\BIBdecl
+
+\bibitem{schoeffler2013impact}
+M.~Schoeffler and J.~Herre, ``About the impact of audio quality on overall
+  listening experience,'' in \emph{Proceedings of Sound and Music Computing
+  Conference}, 2013, pp. 48--53.
+
+\bibitem{repp}
+R.~Repp, ``Recording quality ratings by music professionals,'' in
+  \emph{Proceedings of the 2006 International Computer Music Conference}, 2006,
+  pp. 468--474.
+
+\bibitem{de2013real}
+A.~de~G{\"o}tzen, E.~Sikstr{\"o}m, F.~Grani, and S.~Serafin, ``Real, foley or
+  synthetic? {A}n evaluation of everyday walking sounds,'' in \emph{Proceedings
+  of SMC 2013 : 10th Sound and Music Computing Conference}, 2013.
+
+\bibitem{durr2015implementation}
+G.~Durr, L.~Peixoto, M.~Souza, R.~Tanoue, and J.~D. Reiss, ``Implementation and
+  evaluation of dynamic level of audio detail,'' in \emph{Audio Engineering
+  Society Conference: 56th International Conference: Audio for Games}, February
+  2015.
+
+\bibitem{deman2014a}
+B.~De~Man and J.~D. Reiss, ``Adaptive control of amplitude distortion
+  effects,'' in \emph{53rd Conference of the Audio Engineering Society},
+  January 2014.
+
+\bibitem{mushram}
+E.~Vincent, M.~G. Jafari, and M.~D. Plumbley, ``Preliminary guidelines for
+  subjective evalutation of audio source separation algorithms,'' in \emph{UK
+  ICA Research Network Workshop}, 2006.
+
+\bibitem{uhlereiss}
+J.~D. Reiss and C.~Uhle, ``Determined source separation for microphone
+  recordings using {IIR} filters,'' in \emph{129th Convention of the Audio
+  Engineering Society}.\hskip 1em plus 0.5em minus 0.4em\relax Audio
+  Engineering Society, November 2010.
+
+\bibitem{song2013b}
+Y.~Song, S.~Dixon, and A.~R. Halpern, ``Do online social tags predict perceived
+  or induced emotional responses to music?'' in \emph{Proceedings of the 14th
+  International Society for Music Information Retrieval (ISMIR2013)}, 2013.
+
+\bibitem{song2013a}
+Y.~Song, S.~Dixon, M.~T. Pearce, and G.~Fazekas, ``Using tags to select stimuli
+  in the study of music and emotion,'' \emph{Proceedings of the 3rd
+  International Conference on Music \& Emotion (ICME)}, 2013.
+
+\bibitem{friberg2011comparison}
+A.~Friberg and A.~Hedblad, ``A comparison of perceptual ratings and computed
+  audio features,'' in \emph{Proceedings of the SMC2011, 8th Sound and Music
+  Computing Conference}, 2011, pp. 122--127.
+
+\bibitem{deman2014b}
+B.~De~Man and J.~D. Reiss, ``{APE}: {A}udio {P}erceptual {E}valuation toolbox
+  for {MATLAB},'' in \emph{136th Convention of the Audio Engineering Society},
+  April 2014.
+
+\bibitem{beaqlejs}
+S.~Kraft and U.~Z{\"o}lzer, ``{BeaqleJS}: {HTML5} and {JavaScript} based
+  framework for the subjective evaluation of audio quality,'' in \emph{Linux
+  Audio Conference, Karlsruhe, DE}, 2014.
+
+\bibitem{hulti-gen}
+C.~Gribben and H.~Lee, ``Toward the development of a universal listening test
+  interface generator in max,'' in \emph{138th Convention of the Audio
+  Engineering Society}, 2015.
+
+\bibitem{scale}
+A.~V. Giner, ``Scale - a software tool for listening experiments,'' in
+  \emph{AIA/DAGA Conference on Acoustics, Merano (Italy)}, 2013.
+
+\bibitem{whisper}
+S.~Ciba, A.~Wlodarski, and H.-J. Maempel, ``Whis{PER} -- a new tool for
+  performing listening tests,'' in \emph{126th Convention of the Audio
+  Engineering Society}, May 7-10 2009.
+
+\bibitem{mushra}
+\emph{Method for the subjective assessment of intermediate quality level of
+  coding systems}.\hskip 1em plus 0.5em minus 0.4em\relax Recommendation {ITU-R
+  BS.1534-1}, 2003.
+
+\bibitem{guineapig}
+J.~Hynninen and N.~Zacharov, ``{GuineaPig} - {A} generic subjective test system
+  for multichannel audio,'' in \emph{106th Convention of the Audio Engineering
+  Society Convention}, 1999.
+
+\bibitem{mason2015compression}
+A.~Mason, N.~Jillings, Z.~Ma, J.~D. Reiss, and F.~Melchior, ``Adaptive audio
+  reproduction using personalized compression,'' in \emph{Audio Engineering
+  Society Conference: 57th International Conference: The Future of Audio
+  Entertainment Technology -- Cinema, Television and the Internet}, March 2015.
+
+\bibitem{bech}
+S.~Bech and N.~Zacharov, \emph{Perceptual Audio Evaluation - Theory, Method and
+  Application}.\hskip 1em plus 0.5em minus 0.4em\relax John Wiley \& Sons,
+  2007.
+
+\end{thebibliography}
--- a/docs/SMC15/smc2015template.bib	Mon Apr 27 16:31:46 2015 +0100
+++ b/docs/SMC15/smc2015template.bib	Mon Apr 27 16:43:50 2015 +0100
@@ -134,3 +134,21 @@
 	Title = {Implementation and Evaluation of Dynamic Level of Audio Detail},
 	Year = {2015}}

+@techreport{itu-peaq,
+	Author = {ITU},
+	Institution = {International Telecommunication Union},
+	Title = {Recommendation {ITU-R BS.1387-1}: Method for objective measurements of perceived audio quality},
+	Year = {2001}}
+
+@conference{hulti-gen,
+	Author = {Christopher Gribben and Hyunkook Lee},
+	Booktitle = {138th Convention of the Audio Engineering Society},
+	Title = {Toward the Development of a Universal Listening Test Interface Generator in Max},
+	Year = {2015}}
+
+@conference{guineapig,
+	Author = {Hynninen, Jussi and Zacharov, Nick},
+	Booktitle = {106th Convention of the Audio Engineering Society Convention},
+	Title = {{GuineaPig} - {A} generic subjective test system for multichannel audio},
+	Year = {1999}}
+
--- a/docs/SMC15/smc2015template.tex	Mon Apr 27 16:31:46 2015 +0100
+++ b/docs/SMC15/smc2015template.tex	Mon Apr 27 16:43:50 2015 +0100
@@ -151,7 +151,7 @@

 %NICK: examples of what kind of audio applications HTML5 has made possible, with references to publications (or website)\\

-Perceptual evaluation of audio plays an important role in a wide range of research including audio quality \cite{schoeffler2013impact,repp}, sound synthesis \cite{de2013real,durr2015implementation}, audio effect design \cite{deman2014a}, source separation \cite{uhlereiss}, music and emotion analysis \cite{song2013b,song2013a}, and many others \cite{friberg2011comparison}.  % codec design?
+Perceptual evaluation of audio plays an important role in a wide range of research including audio quality \cite{schoeffler2013impact,repp}, sound synthesis \cite{de2013real,durr2015implementation}, audio effect design \cite{deman2014a}, source separation \cite{mushram,uhlereiss}, music and emotion analysis \cite{song2013b,song2013a}, and many others \cite{friberg2011comparison}.  % codec design?

 This work is based in part on the APE audio perceptual evaluation interface for MATLAB \cite{deman2014b}. An important drawback of this toolbox is the need to have MATLAB to create a test and even to run (barring the use of an executable generated by MATLAB), and limited compatibility with both earlier and newer versions of MATLAB, which makes it hard to maintain. On the other hand, a web application generally has the advantage of running in most browsers on most applications.

@@ -161,10 +161,50 @@
 %MATLAB: \cite{whisper,mushram,scale}
 % to add: OPAQUE, Rumsey's repertory grid technique

-Many perceptual audio interfaces are already available, such as the MATLAB-based tools MUSHRAM \cite{mushram}, WhisPER \cite{whisper}, and Scale \cite{scale}, and the browser-based tool used in \cite{song2013a,song2013b} and BeaqleJS \cite{beaqlejs}. The latter makes use of the HTML5 audio capabilities and comes with a number of predefined, established test interfaces such as ABX and MUSHRA \cite{mushra}.
-In this paper, we provide a listening test back end that allows for easy set up of a wide variety of listening tests, highly flexible yet very simple and not requiring any programming skills.
+
+
+\begin{table}[htdp]
+\caption{Available audio perceptual evaluation interfaces}
+\begin{center}
+\begin{tabular}{|*{3}{l|}}
+% order?
+\hline
+\textbf{Name} & \textbf{Language} & \textbf{Ref.}\\
+\hline
+APE & MATLAB & \cite{deman2014b} \\
+BeaqleJS & HTML5/JS & \cite{beaqlejs}\\ % ABX, mushra
+%C4DM\footnote{http://isophonics.org/test - collection of listening tests developed by Gy\"{o}rgy Fazekas and Thomas Wilmering at Centre for Digital Music.} & JS & \cite{song2013a,song2013b}\\
+HULTI-GEN & Max & \cite{hulti-gen}\\
+MUSHRAM & MATLAB & \cite{mushram}\\ % type: mushra
+Scale & MATLAB & \cite{scale} \\
+WhisPER & MATLAB & \cite{whisper}\\
+\hline
+\end{tabular}
+\end{center}
+\label{tab:interfaces}
+\end{table}%
+
+Various perceptual audio interfaces are already available, see Table \ref{tab:interfaces}.
+Many are MATLAB-based, useful for easily processing and visualising the data produced by the listening tests, but requiring the application to be installed to run or - in the case of an executable created with MATLAB - at least create the test.
+Furthermore, compatibility is limited across different versions of MATLAB.
+Similarly, Max requires little or no programming background but it is proprietary software as well, which is especially undesirable when tests need to be deployed at different sites.
+More recently, BeaqleJS \cite{beaqlejs} makes use of the HTML5 audio capabilities and comes with a number of predefined, established test interfaces such as ABX and MUSHRA \cite{mushra}. %
+Another listening test tool, GuineaPig \cite{guineapig}, is not available to the public at the time of writing.
+
 % [How is this one different from all these?] improve

+% FLEXIBLE (reference (not) appropriate)
+
+Furthermore, the option to provide free-text comment fields allows for tests with individual vocabulary methods, as opposed to only allowing quantitative scales associated to a fixed set of descriptors.
+
+
+% ENVIRONMENT
+There are a number of advantages to building a web audio based listening test environment. The ability to easily deploy a flexible and scalable testing environment that requires no proprietary software to run makes the web audio evaluation tool a very flexible testing tool. The ability to host a single test server and create multiple clients not only allows multiple participants to be involved in a trial simultaneously, but also permits participants to be located anywhere in the world. There are also less user experience issues, since all users should have some experience with using existing web technologies.
+
+% EASE OF USE: no need to go in the code
+To make the tool accessible to a wide range of researchers, we aim to offer maximum functionality even to those with little or no programming background. The tool we present can set up a listening test without reading or adjusting any code, provided no new types of interfaces need to be created.
+
+We present a browser-based perceptual evaluation tool from which any kind of multiple stimulus audio evaluation tool where subjects need to rank, rate, select, or comment on different audio samples can be built. %In this paper, we provide a listening test back end that allows for easy set up of a wide variety of listening tests, highly flexible yet very simple and not requiring any programming skills.
 The Web Audio API is a high-level JavaScript Application Programming Interface (API) designed for real-time processing of audio inside the browser through various processing nodes\footnote{http://webaudio.github.io/web-audio-api/}. Various web sites have used the Web Audio API for either creative purposes, such as drum machines and score creation tools\footnote{http://webaudio.github.io/demo-list/},
 others from the list show real-time captured audio processing such as room reverberation tools and a phase vocoder from the system microphone. The BBC Radiophonic Workshop shows effects used on famous TV shows such as Doctor Who, being simulated inside the browser\footnote{http://webaudio.prototyping.bbc.co.uk/}.
 Another example is the BBC R\&D personalised compressor which applies a dynamic range compressor on a radio station that dynamically adjusts the compressor settings to match the listener's environment \cite{mason2015compression}.
@@ -176,8 +216,17 @@

 There are a number of advantages to building a web audio based listening test environment. The ability to easily deploy a flexible and scalable testing environment that requires no proprietary software to run makes the web audio evaluation tool a very flexible testing tool. The ability to host a single test server not only allows multiple participants to be involved in a trial simultaneously, but also permits participants to be located anywhere in the world. There are also less user experience issues, since all users should have some experience with using existing web technologies.

+
 In the following sections, we describe the included interface in more detail, discuss the implementation, and cover considerations that were made in the design process of this tool.

+%\section{Requirements}\label{sec:requirements}
+%???
+%
+%\begin{itemize}
+%\item
+%\end{itemize}
+
+
 \section{Interface}\label{sec:interface}

 At this point, we have implemented the interface of the MATLAB-based APE (Audio Perceptual Evaluation) toolbox \cite{deman2014b}. This shows one marker for each simultaneously evaluated audio fragment on one or more horizontal axes, that can be moved to rate or rank the respective fragments in terms of any subjective quality, as well as a comment box for every marker, and any extra text boxes for extra comments.
@@ -265,6 +314,8 @@
 \item \textbf{Reference}: Allows for a separate sample (outside of the axis) to be the `reference', which the subject can play back during the test to help with the task at hand \cite{mushra}.
 \item \textbf{Hidden reference}: Whether or not an explicit `reference' is provided, the `hidden reference' should be rated above a certain value \cite{mushra} - this can be enforced.
 \item \textbf{Hidden anchor}: The `hidden anchor' should be rated lower than a certain value \cite{mushra} - this can be enforced.
+\item \textbf{Show scrub bar}: Display a playhead on a scrub bar to show the position in the current fragment.
+\item \textbf{Drag playhead}: If scrub bar is visible, allow dragging to move back or forward in a fragment.
 \end{itemize}

 When one of these options is not included in the set up file, they assume a default value. As a result, the input file can be kept very compact if default values suffice for the test.