comparison docs/WAC2016/WAC2016.tex @ 1950:6eec4abc85e7

Minor edits to paper, updated installation instructions, added to be ignored files (e.g. LaTeX auxiliary files)

 \documentclass{sig-alternate}
-\usepackage{hyperref}
+\usepackage{hyperref}	% make links (like references, links to Sections, ...) clickable
+\usepackage{enumitem}	% tighten itemize etc by appending '[noitemsep,nolistsep]'
 
 \begin{document}
 
 % Copyright
 \setcopyright{waclicense}

 	% Listening tests/perceptual audio evaluation: what are they, why are they important
 	% As opposed to limited scope of WAC15 paper: also musical features, realism of sound effects / sound synthesis, performance of source separation and other algorithms... 
 	Perceptual evaluation of audio, in the form of listening tests, is a powerful way to assess anything from audio codec quality over realism of sound synthesis to the performance of source separation, automated music production and 
 	In less technical areas, the framework of a listening test can be used to measure emotional response to music or test cognitive abilities. % maybe some references? If there's space.
 
+	% check out http://link.springer.com/article/10.1007/s10055-015-0270-8 - only paper that cited WAC15 paper
+
 	% Why difficult? Challenges? What constitutes a good interface? 
 	Technical, interfaces, user friendliness, reliability 
 
 	Note that the design of an effective listening test further poses many challenges unrelated to interface design, which are beyond the scope of this paper \cite{bech}. 
 

 
 	If the experimenter is willing to trade some degree of control for a higher number of participants, the test can be hosted on a web server so that subjects can take part remotely. This way, a link can be shared widely in the hope of attracting a large amount of subjects, while listening conditions and subject reliability may be less ideal. However, a sound system calibration page and a wide range of metrics logged during the test mitigate these problems. Note also that in some experiments, it may be preferred that the subject has a `real life', familiar listening set-up, for instance when perceived quality differences on everyday sound systems are investigated. 
 	Furthermore, a fully browser-based test, where the collection of the results is automatic, is more efficient and technically reliable even when the test still takes place under lab conditions.
 
 	The following features allow easy and effective remote testing: 
-	\begin{itemize}
+	\begin{itemize}[noitemsep,nolistsep]
 		\item PHP script to collect result XML files
 		\item Randomly pick specified number of audioholders
 		\item Calibration
 		\item Functionality to participate multiple times
-			\begin{itemize}
+			\begin{itemize}[noitemsep,nolistsep]
 				\item Possible to log in with unique ID (no password)
 				\item Pick `new user' (need new, unique ID) or `already participated' (need already available ID)
 				\item Store XML on server with IDs plus which audioholders have already been listened to
 				\item Don't show `post-test' survey after first time
 				\item Pick `new' audioholders if available

 
 	Established tests (see below) included as `presets' in the build-your-own-test page. 
 
 
 	We could add more interfaces, such as: 
-	\begin{itemize}
+	\begin{itemize}[noitemsep,nolistsep]
 		\item (APE style) \cite{ape}
 		\item Multi attribute ratings
 		\item MUSHRA (ITU-R BS. 1534)~\cite{recommendation20031534}
 		\item Interval Scale~\cite{zacharov1999round}
 		\item Rank Scale~\cite{pascoe1983evaluation}

 			\item 2 point Scale - Better or Worse - (not sure how to default this - they default everything to better, which is an interesting choice)
 		\end{itemize}
 	\end{itemize}
 	
 	There are also the following interfaces, which would require a slightly different `engine' underneath, e.g. loading a different page for every possible pair. 
-	\begin{itemize}
+	\begin{itemize}[noitemsep,nolistsep]
 		\item AB Test~\cite{lipshitz1981great}
 		\item ABX Test~\cite{clark1982high}
 		\item JND
 	\end{itemize}
 	

 	% don't mention Python scripts
 	It would be great to have easy-to-use analysis tools to visualise the collected data and even do science with it. Even better would be to have all this in the browser. Complete perfection would be achieved if and when only limited setup, installation time, and expertise are required for the average non-CS researcher to use this. 
 	
 	The following could be nice: 
 	
-	\begin{itemize}
+	\begin{itemize}[noitemsep,nolistsep]
 		\item Web page showing all audioholder IDs, file names, subject IDs, audio element IDs, ... in the collected XMLs so far (\texttt{saves/*.xml})
 		\item Check/uncheck each of the above for analysis (e.g. zoom in on a certain song, or exclude a subset of subjects)
 		\item Click a mix to hear it (follow path in XML setup file, which is also embedded in the XML result file)
 		\item Box plot, confidence plot, scatter plot of values (for a given audioholder)
 		\item Timeline for a specific subject (see Python scripts), perhaps re-playing the experiment in X times realtime. (If actual realtime, you could replay the audio...)