BrechtDeMan@719: \documentclass{sig-alternate} BrechtDeMan@727: \usepackage{hyperref} % make links (like references, links to Sections, ...) clickable BrechtDeMan@727: \usepackage{enumitem} % tighten itemize etc by appending '[noitemsep,nolistsep]' djmoffat@729: \usepackage{cleveref} BrechtDeMan@719: BrechtDeMan@719: \begin{document} BrechtDeMan@719: BrechtDeMan@719: % Copyright BrechtDeMan@719: \setcopyright{waclicense} BrechtDeMan@719: BrechtDeMan@719: BrechtDeMan@719: %% DOI BrechtDeMan@719: %\doi{10.475/123_4} BrechtDeMan@719: % BrechtDeMan@719: %% ISBN BrechtDeMan@719: %\isbn{123-4567-24-567/08/06} BrechtDeMan@719: % BrechtDeMan@719: %%Conference BrechtDeMan@719: %\conferenceinfo{PLDI '13}{June 16--19, 2013, Seattle, WA, USA} BrechtDeMan@719: % BrechtDeMan@719: %\acmPrice{\$15.00} BrechtDeMan@719: BrechtDeMan@719: % BrechtDeMan@719: % --- Author Metadata here --- BrechtDeMan@719: \conferenceinfo{Web Audio Conference WAC-2016,}{April 4--6, 2016, Atlanta, USA} BrechtDeMan@719: \CopyrightYear{2016} % Allows default copyright year (20XX) to be over-ridden - IF NEED BE. BrechtDeMan@719: %\crdata{0-12345-67-8/90/01} % Allows default copyright data (0-89791-88-6/97/05) to be over-ridden - IF NEED BE. BrechtDeMan@719: % --- End of Author Metadata --- BrechtDeMan@719: BrechtDeMan@728: \title{Web Audio Evaluation Tool: A framework for subjective assessment of audio} BrechtDeMan@719: %\subtitle{[Extended Abstract] BrechtDeMan@719: %\titlenote{A full version of this paper is available as BrechtDeMan@719: %\textit{Author's Guide to Preparing ACM SIG Proceedings Using BrechtDeMan@719: %\LaTeX$2_\epsilon$\ and BibTeX} at BrechtDeMan@719: %\texttt{www.acm.org/eaddress.htm}}} BrechtDeMan@719: % BrechtDeMan@719: % You need the command \numberofauthors to handle the 'placement BrechtDeMan@719: % and alignment' of the authors beneath the title. BrechtDeMan@719: % BrechtDeMan@719: % For aesthetic reasons, we recommend 'three authors at a time' BrechtDeMan@719: % i.e. three 'name/affiliation blocks' be placed beneath the title. BrechtDeMan@719: % BrechtDeMan@719: % NOTE: You are NOT restricted in how many 'rows' of BrechtDeMan@719: % "name/affiliations" may appear. We just ask that you restrict BrechtDeMan@719: % the number of 'columns' to three. BrechtDeMan@719: % BrechtDeMan@719: % Because of the available 'opening page real-estate' BrechtDeMan@719: % we ask you to refrain from putting more than six authors BrechtDeMan@719: % (two rows with three columns) beneath the article title. BrechtDeMan@719: % More than six makes the first-page appear very cluttered indeed. BrechtDeMan@719: % BrechtDeMan@719: % Use the \alignauthor commands to handle the names BrechtDeMan@719: % and affiliations for an 'aesthetic maximum' of six authors. BrechtDeMan@719: % Add names, affiliations, addresses for BrechtDeMan@719: % the seventh etc. author(s) as the argument for the BrechtDeMan@719: % \additionalauthors command. BrechtDeMan@719: % These 'additional authors' will be output/set for you BrechtDeMan@719: % without further effort on your part as the last section in BrechtDeMan@719: % the body of your article BEFORE References or any Appendices. BrechtDeMan@719: BrechtDeMan@725: % FIVE authors instead of four, to leave space between first two authors. djmoffat@721: \numberofauthors{5} % in this sample file, there are a *total* BrechtDeMan@719: % of EIGHT authors. SIX appear on the 'first-page' (for formatting BrechtDeMan@719: % reasons) and the remaining two appear in the \additionalauthors section. BrechtDeMan@719: % BrechtDeMan@719: \author{ BrechtDeMan@719: % You can go ahead and credit any number of authors here, BrechtDeMan@719: % e.g. one 'row of three' or two rows (consisting of one row of three BrechtDeMan@719: % and a second row of one, two or three). BrechtDeMan@719: % BrechtDeMan@719: % The command \alignauthor (no curly braces needed) should BrechtDeMan@719: % precede each author name, affiliation/snail-mail address and BrechtDeMan@719: % e-mail address. Additionally, tag each line of BrechtDeMan@719: % affiliation/address with \affaddr, and tag the BrechtDeMan@719: % e-mail address with \email. BrechtDeMan@719: % BrechtDeMan@719: % 1st. author BrechtDeMan@719: \alignauthor Nicholas Jillings\\ BrechtDeMan@719: \email{n.g.r.jillings@se14.qmul.ac.uk} BrechtDeMan@725: % dummy author for nicer spacing BrechtDeMan@725: \alignauthor BrechtDeMan@719: % 2nd. author BrechtDeMan@719: \alignauthor Brecht De Man\\ BrechtDeMan@719: \email{b.deman@qmul.ac.uk} BrechtDeMan@719: \and % use '\and' if you need 'another row' of author names BrechtDeMan@719: % 3rd. author BrechtDeMan@719: \alignauthor David Moffat\\ BrechtDeMan@719: \email{d.j.moffat@qmul.ac.uk} BrechtDeMan@719: % 4th. author BrechtDeMan@719: \alignauthor Joshua D. Reiss\\ BrechtDeMan@719: \email{joshua.reiss@qmul.ac.uk} BrechtDeMan@725: \and % new line for address BrechtDeMan@719: \affaddr{Centre for Digital Music}\\ BrechtDeMan@719: \affaddr{School of Electronic Engineering and Computer Science}\\ BrechtDeMan@719: \affaddr{Queen Mary University of London}\\ BrechtDeMan@719: \affaddr{Mile End Road,} BrechtDeMan@719: \affaddr{London E1 4NS}\\ BrechtDeMan@719: \affaddr{United Kingdom}\\ BrechtDeMan@719: } BrechtDeMan@719: %Centre for Digital Music, School of Electronic Engineering and Computer Science, Queen Mary University of London BrechtDeMan@719: %% 5th. author BrechtDeMan@719: %\alignauthor Sean Fogarty\\ BrechtDeMan@719: % \affaddr{NASA Ames Research Center}\\ BrechtDeMan@719: % \affaddr{Moffett Field}\\ BrechtDeMan@719: % \email{fogartys@amesres.org} BrechtDeMan@719: %% 6th. author BrechtDeMan@719: %\alignauthor Charles Palmer\\ BrechtDeMan@719: % \affaddr{Palmer Research Laboratories}\\ BrechtDeMan@719: % \affaddr{8600 Datapoint Drive}\\ BrechtDeMan@719: % \email{cpalmer@prl.com} BrechtDeMan@719: %} BrechtDeMan@719: % There's nothing stopping you putting the seventh, eighth, etc. BrechtDeMan@719: % author on the opening page (as the 'third row') but we ask, BrechtDeMan@719: % for aesthetic reasons that you place these 'additional authors' BrechtDeMan@719: % in the \additional authors block, viz. BrechtDeMan@719: %\additionalauthors{Additional authors: John Smith (The Th{\o}rv{\"a}ld Group, BrechtDeMan@719: %email: {\texttt{jsmith@affiliation.org}}) and Julius P.~Kumquat BrechtDeMan@719: %(The Kumquat Consortium, email: {\texttt{jpkumquat@consortium.net}}).} BrechtDeMan@719: \date{1 October 2015} BrechtDeMan@719: % Just remember to make sure that the TOTAL number of authors BrechtDeMan@719: % is the number that will appear on the first page PLUS the BrechtDeMan@719: % number that will appear in the \additionalauthors section. BrechtDeMan@719: BrechtDeMan@719: \maketitle BrechtDeMan@719: \begin{abstract} BrechtDeMan@719: Here comes the abstract. BrechtDeMan@719: \end{abstract} BrechtDeMan@719: BrechtDeMan@719: BrechtDeMan@719: \section{Introduction} BrechtDeMan@726: BrechtDeMan@726: % Listening tests/perceptual audio evaluation: what are they, why are they important BrechtDeMan@726: % As opposed to limited scope of WAC15 paper: also musical features, realism of sound effects / sound synthesis, performance of source separation and other algorithms... BrechtDeMan@726: Perceptual evaluation of audio, in the form of listening tests, is a powerful way to assess anything from audio codec quality over realism of sound synthesis to the performance of source separation, automated music production and BrechtDeMan@726: In less technical areas, the framework of a listening test can be used to measure emotional response to music or test cognitive abilities. % maybe some references? If there's space. BrechtDeMan@726: BrechtDeMan@727: % check out http://link.springer.com/article/10.1007/s10055-015-0270-8 - only paper that cited WAC15 paper BrechtDeMan@727: BrechtDeMan@726: % Why difficult? Challenges? What constitutes a good interface? BrechtDeMan@726: Technical, interfaces, user friendliness, reliability BrechtDeMan@726: BrechtDeMan@726: Note that the design of an effective listening test further poses many challenges unrelated to interface design, which are beyond the scope of this paper \cite{bech}. BrechtDeMan@726: BrechtDeMan@726: % Why in the browser? BrechtDeMan@726: Web Audio API has made some essential features like sample manipulation of audio streams possible \cite{schoeffler2015mushra}. BrechtDeMan@719: BrechtDeMan@725: Situating the Web Audio Evaluation Tool between other currently available evaluation tools, ... BrechtDeMan@725: BrechtDeMan@725: % only browser-based? djmoffat@729: \begin{table*}[ht] BrechtDeMan@725: \caption{Table with existing listening test platforms and their features} BrechtDeMan@725: \begin{center} BrechtDeMan@725: \begin{tabular}{|*{6}{l|}} BrechtDeMan@725: \hline BrechtDeMan@726: \textbf{Name} & \textbf{Ref.} & \textbf{Language} & \textbf{Interfaces} & \textbf{Remote} & \textbf{All UI} \\ BrechtDeMan@725: \hline BrechtDeMan@726: APE & \cite{ape} & MATLAB & multiple stimulus one axis & & \\ BrechtDeMan@725: BeaqleJS & \cite{beaqlejs} & JavaScript & & not natively supported & \\ BrechtDeMan@726: HULTI-GEN & \cite{hultigen} & MAX & & & \checkmark \\ BrechtDeMan@726: \textbf{WAET} & \cite{waet} & JavaScript & \textbf{all of the above} & \checkmark & \checkmark \\ BrechtDeMan@725: \hline BrechtDeMan@725: \end{tabular} BrechtDeMan@725: \end{center} BrechtDeMan@725: \label{tab:toolboxes} BrechtDeMan@725: \end{table*}% BrechtDeMan@725: BrechtDeMan@725: % about BeaqleJS BrechtDeMan@725: ... However, BeaqleJS \cite{beaqlejs} does not make use of the Web Audio API, %requires programming knowledge... BrechtDeMan@725: BrechtDeMan@725: % BrechtDeMan@726: Selling points: remote tests, visualisaton, create your own test in the browser, many interfaces, few/no dependencies, flexibility BrechtDeMan@726: BrechtDeMan@726: As recruiting participants can be very time-consuming, and as for some tests a large number of participants is needed, browser-based tests \cite{schoeffler2015mushra}. However, to our knowledge, no tool currently exists that allows the creation of a remotely accessible listening test. % I wonder what you can do with Amazon Mechanical Turk and the likes. BrechtDeMan@726: BrechtDeMan@726: [Talking about what we do in the various sections of this paper. Referring to \cite{waet}. ] BrechtDeMan@725: BrechtDeMan@728: % MEETING 8 OCTOBER BrechtDeMan@728: \subsection{Meeting 8 October} BrechtDeMan@728: \begin{itemize} BrechtDeMan@728: \item Do we manipulate audio?\\ BrechtDeMan@728: \begin{itemize} BrechtDeMan@728: \item Add loudness equalisation? (test\_create.html) Tag with gains. BrechtDeMan@728: \item Add volume slider? BrechtDeMan@728: \item Cross-fade (in interface node): default 0, number of seconds BrechtDeMan@728: \item Also: we use the playback buffer to present metrics of which portion is listened to BrechtDeMan@728: \end{itemize} BrechtDeMan@728: \item Logging system information: whichever are possible (justify others) BrechtDeMan@728: \item Input streams as audioelements BrechtDeMan@728: \item Capture microphone to estimate loudness (especially Macbook) BrechtDeMan@728: \item Test page (in-built oscillators): left-right calibration, ramp up test tone until you hear it; optional compensating EQ (future work implementing own filters) --> Highlight issues! BrechtDeMan@728: \item Record IP address (PHP function, grab and append to XML file) BrechtDeMan@728: \item Expand anchor/reference options BrechtDeMan@728: \item AB / ABX BrechtDeMan@728: \end{itemize} BrechtDeMan@728: BrechtDeMan@728: \subsubsection{Issues} BrechtDeMan@728: \begin{itemize} BrechtDeMan@728: \item Filters not consistent (Nick to test across browsers) BrechtDeMan@728: \item Playback audiobuffers need to be destroyed and rebuilt each time BrechtDeMan@728: \item Can't get channel data, hardware input/output... BrechtDeMan@728: \end{itemize} BrechtDeMan@728: BrechtDeMan@725: BrechtDeMan@726: \section{Architecture} % title? 'back end'? % NICK BrechtDeMan@719: A slightly technical overview of the system. Talk about XML, JavaScript, Web Audio API, HTML5. BrechtDeMan@725: Describe and/or visualise audioholder-audioelement-... structure. BrechtDeMan@725: BrechtDeMan@726: % see also SMC12 - less detail here BrechtDeMan@726: BrechtDeMan@726: Which type of files? % WAV, anything else? Perhaps not exhaustive list, but say something along the lines of 'whatever browser supports' BrechtDeMan@724: BrechtDeMan@725: Streaming audio? % probably not, unless it's easy BrechtDeMan@725: BrechtDeMan@726: Compatibility? % not IE, everything else fine? BrechtDeMan@726: BrechtDeMan@726: BrechtDeMan@724: BrechtDeMan@724: BrechtDeMan@725: \section{Remote tests} % with previous? BrechtDeMan@726: BrechtDeMan@726: If the experimenter is willing to trade some degree of control for a higher number of participants, the test can be hosted on a web server so that subjects can take part remotely. This way, a link can be shared widely in the hope of attracting a large amount of subjects, while listening conditions and subject reliability may be less ideal. However, a sound system calibration page and a wide range of metrics logged during the test mitigate these problems. Note also that in some experiments, it may be preferred that the subject has a `real life', familiar listening set-up, for instance when perceived quality differences on everyday sound systems are investigated. BrechtDeMan@726: Furthermore, a fully browser-based test, where the collection of the results is automatic, is more efficient and technically reliable even when the test still takes place under lab conditions. BrechtDeMan@726: BrechtDeMan@724: The following features allow easy and effective remote testing: BrechtDeMan@727: \begin{itemize}[noitemsep,nolistsep] BrechtDeMan@724: \item PHP script to collect result XML files BrechtDeMan@724: \item Randomly pick specified number of audioholders BrechtDeMan@726: \item Calibration BrechtDeMan@724: \item Functionality to participate multiple times BrechtDeMan@727: \begin{itemize}[noitemsep,nolistsep] BrechtDeMan@724: \item Possible to log in with unique ID (no password) BrechtDeMan@724: \item Pick `new user' (need new, unique ID) or `already participated' (need already available ID) BrechtDeMan@724: \item Store XML on server with IDs plus which audioholders have already been listened to BrechtDeMan@724: \item Don't show `post-test' survey after first time BrechtDeMan@724: \item Pick `new' audioholders if available BrechtDeMan@724: \item Copy survey information first time to new XMLs BrechtDeMan@724: \end{itemize} BrechtDeMan@724: \item Intermediate saves BrechtDeMan@724: \item Collect IP address information (privacy issues?) --> geo-related API? BrechtDeMan@726: \item Time measurement - see before or BrechtDeMan@724: \end{itemize} BrechtDeMan@724: BrechtDeMan@719: BrechtDeMan@725: \section{Interfaces} % title? 'Front end'? % Dave djmoffat@729: djmoffat@729: The purpose of this listening test framework is to allow any user the maximum flexibility to design a listening test for their exact application with minimum effort. To this end, a large range of standard listening test interfaces have been implemented. A review of existing listening test frameworks was undertaken and presented in~\Cref{tab:toolboxes}. HULTI-GEN~\cite{hultigen} is a single toolbox that presents the user with a large number of different test interfaces and allows for customisation of each test interface. djmoffat@729: djmoffat@729: To provide users with a flexible system, a large range of `standard' listening test interfaces have been implemented, including: djmoffat@729: \begin{itemize}[noitemsep,nolistsep] djmoffat@729: \item MUSHRA (ITU-R BS. 1534)~\cite{recommendation20031534} djmoffat@729: \begin{itemize} djmoffat@729: \item Multiple stimuli are presented and rated on a continuous scale, which includes a reference, hidden reference and hidden anchors. djmoffat@729: \end{itemize} djmoffat@729: \item Rank Scale~\cite{pascoe1983evaluation} djmoffat@729: \begin{itemize} djmoffat@729: \item Stimuli ranked on single horizontal scale, where they are ordered in preference order. djmoffat@729: \end{itemize} djmoffat@729: \item Likert scale~\cite{likert1932technique} djmoffat@729: \begin{itemize} djmoffat@729: \item Each stimuli has a five point scale with values: Strongly Agree, Agree, Neutral, Disagree and Strongly Disagree. djmoffat@729: \end{itemize} djmoffat@729: \item ABC/HR (ITU-R BS. 1116)~\cite{recommendation19971116} (Mean Opinion Score: MOS) djmoffat@729: \begin{itemize} djmoffat@729: \item Each stimulus has a continuous scale (5-1), labeled as Imperceptible, Perceptible but not annoying, slightly annoying, annoying, very annoying. djmoffat@729: \end{itemize} djmoffat@729: \item -50 to 50 Bipolar with Ref djmoffat@729: \begin{itemize} djmoffat@729: \item Each stimulus has a continuous scale -50 to 50 with default values as 0 in middle and a comparison. There is also a provided reference \end{itemize} djmoffat@729: \item Absolute Category Rating (ACR) Scale~\cite{rec1996p} djmoffat@729: \begin{itemize} djmoffat@729: \item Each stimuli has a five point scale with values: Bad, Poor, Fair, Good, Excellent djmoffat@729: \end{itemize} djmoffat@729: \item Degredation Category Rating (DCR) Scale~\cite{rec1996p} djmoffat@729: \begin{itemize} djmoffat@729: \item Each stimuli has a five point scale with values: (5) Inaudible, (4) Audible but not annoying, (3) slightly annoying, (2) annoying, (1) very annoying. djmoffat@729: \end{itemize} djmoffat@729: \item Comparison Category Rating (CCR) Scale~\cite{rec1996p} djmoffat@729: \begin{itemize} djmoffat@729: \item Each stimuli has a seven point scale with values: Much Better, Better, Slightly Better, About the same, slightly worse, worse, much worse. There is also a provided reference. djmoffat@729: \end{itemize} djmoffat@729: \item 9 Point Hedonic Category Rating Scale~\cite{peryam1952advanced} djmoffat@729: \begin{itemize} djmoffat@729: \item Each stimuli has a seven point scale with values: Like Extremely, Like Very Much, Like Moderate, Like Slightly, Neither Like nor Dislike, dislike Extremely, dislike Very Much, dislike Moderate, dislike Slightly. There is also a provided reference. djmoffat@729: \end{itemize} djmoffat@729: \item ITU-R 5 Point Continuous Impairment Scale~\cite{rec1997bs} djmoffat@729: \begin{itemize} djmoffat@729: \item Each stimuli has a five point scale with values: (5) Imperceptible, (4) Perceptible but not annoying, (3) slightly annoying, (2) annoying, (1) very annoying. There is also a provided reference. djmoffat@729: \end{itemize} djmoffat@729: \item Pairwise Comparison (Better/Worse)~\cite{david1963method} djmoffat@729: \begin{itemize} djmoffat@729: \item A reference is provided and ever stimulus is rated as being either better or worse than the reference. djmoffat@729: \end{itemize} djmoffat@729: \item APE style \cite{ape} djmoffat@729: \begin{itemize} djmoffat@729: \item Multiple stimuli on a single horizontal slider for inter-sample rating. djmoffat@729: \end{itemize} djmoffat@729: \item Multi attribute ratings djmoffat@729: \begin{itemize} djmoffat@729: \item Multiple stimuli as points on a 2D plane for inter-sample rating (eg. Valence Arousal) djmoffat@729: \end{itemize} djmoffat@729: \item AB Test~\cite{lipshitz1981great} djmoffat@729: \begin{itemize} djmoffat@729: \item Two stimuli are presented at a time and the participant has to select a preferred stimulus. djmoffat@729: \end{itemize} djmoffat@729: \item ABX Test~\cite{clark1982high} djmoffat@729: \begin{itemize} djmoffat@729: \item Two stimuli are presented along with a reference and the participant has to select a preferred stimulus, often the closest to the reference. djmoffat@729: \end{itemize} djmoffat@729: \end{itemize} djmoffat@729: djmoffat@729: While implementing all of these interfaces, it is possible to include any number of references, anchors, hidden references and hidden anchors into all of these listening test formats. djmoffat@729: djmoffat@729: %%%% \begin{itemize}[noitemsep,nolistsep] djmoffat@729: %%%% \item (APE style) \cite{ape} djmoffat@729: %%%% \item Multi attribute ratings djmoffat@729: %%%% \item MUSHRA (ITU-R BS. 1534)~\cite{recommendation20031534} djmoffat@729: %%%% \item Interval Scale~\cite{zacharov1999round} djmoffat@729: %%%% \item Rank Scale~\cite{pascoe1983evaluation} djmoffat@729: %%%% djmoffat@729: %%%% \item 2D Plane rating - e.g. Valence vs. Arousal~\cite{carroll1969individual} djmoffat@729: %%%% \item Likert scale~\cite{likert1932technique} djmoffat@729: %%%% djmoffat@729: %%%% \item {\bf All the following are the interfaces available in HULTI-GEN~\cite{hultigen} } djmoffat@729: %%%% \item ABC/HR (ITU-R BS. 1116)~\cite{recommendation19971116} djmoffat@729: %%%% \begin{itemize} djmoffat@729: %%%% \item Continuous Scale (5-1) Imperceptible, Perceptible but not annoying, slightly annoying, annoying, very annoying. (default Inaudible?) djmoffat@729: %%%% \end{itemize} djmoffat@729: %%%% \item -50 to 50 Bipolar with Ref djmoffat@729: %%%% \begin{itemize} djmoffat@729: %%%% \item Scale -50 to 50 on Mushra with default values as 0 in middle and a comparison ``Reference'' to compare to 0 value djmoffat@729: %%%% \end{itemize} djmoffat@729: %%%% \item Absolute Category Rating (ACR) Scale~\cite{rec1996p} djmoffat@729: %%%% \begin{itemize} djmoffat@729: %%%% \item 5 point Scale - Bad, Poor, Fair, Good, Excellent (Default fair?) djmoffat@729: %%%% \end{itemize} djmoffat@729: %%%% \item Degredation Category Rating (DCR) Scale~\cite{rec1996p} djmoffat@729: %%%% \begin{itemize} djmoffat@729: %%%% \item 5 point Scale - Inaudible, Audible but not annoying, slightly annoying, annoying, very annoying. (default Inaudible?) - {\it Basically just quantised ABC/HR?} djmoffat@729: %%%% \end{itemize} djmoffat@729: %%%% \item Comparison Category Rating (CCR) Scale~\cite{rec1996p} djmoffat@729: %%%% \begin{itemize} djmoffat@729: %%%% \item 7 point scale: Much Better, Better, Slightly Better, About the same, slightly worse, worse, much worse - Default about the same with reference to compare to djmoffat@729: %%%% \end{itemize} djmoffat@729: %%%% \item 9 Point Hedonic Category Rating Scale~\cite{peryam1952advanced} djmoffat@729: %%%% \begin{itemize} djmoffat@729: %%%% \item 9 point scale: Like Extremely, Like Very Much, Like Moderate, Like Slightly, Neither Like nor Dislike, dislike Extremely, dislike Very Much, dislike Moderate, dislike Slightly - Default Neither Like nor Dislike with reference to compare to djmoffat@729: %%%% \end{itemize} djmoffat@729: %%%% \item ITU-R 5 Point Continuous Impairment Scale~\cite{rec1997bs} djmoffat@729: %%%% \begin{itemize} djmoffat@729: %%%% \item 5 point Scale (5-1) Imperceptible, Perceptible but not annoying, slightly annoying, annoying, very annoying. (default Inaudible?)- {\it Basically just quantised ABC/HR, or Different named DCR} djmoffat@729: %%%% \end{itemize} djmoffat@729: %%%% \item Pairwise Comparison (Better/Worse)~\cite{david1963method} djmoffat@729: %%%% \begin{itemize} djmoffat@729: %%%% \item 2 point Scale - Better or Worse - (not sure how to default this - they default everything to better, which is an interesting choice) djmoffat@729: %%%% \end{itemize} djmoffat@729: %%%% \end{itemize} djmoffat@729: djmoffat@729: { \bf A screenshot would be nice. djmoffat@729: BrechtDeMan@725: `Build your own test' BrechtDeMan@725: BrechtDeMan@726: Elements present to build any of the following interfaces, and many more: axes, markers, labels, anchors, references, reference signal button, stop button, comment boxes, radio buttons, checkboxes, transport/scrubber bar BrechtDeMan@726: djmoffat@729: Established tests (see below) included as `presets' in the build-your-own-test page. } BrechtDeMan@719: BrechtDeMan@719: \section{Analysis and diagnostics} BrechtDeMan@726: % don't mention Python scripts BrechtDeMan@719: It would be great to have easy-to-use analysis tools to visualise the collected data and even do science with it. Even better would be to have all this in the browser. Complete perfection would be achieved if and when only limited setup, installation time, and expertise are required for the average non-CS researcher to use this. BrechtDeMan@719: BrechtDeMan@722: The following could be nice: BrechtDeMan@722: BrechtDeMan@727: \begin{itemize}[noitemsep,nolistsep] BrechtDeMan@722: \item Web page showing all audioholder IDs, file names, subject IDs, audio element IDs, ... in the collected XMLs so far (\texttt{saves/*.xml}) BrechtDeMan@722: \item Check/uncheck each of the above for analysis (e.g. zoom in on a certain song, or exclude a subset of subjects) BrechtDeMan@722: \item Click a mix to hear it (follow path in XML setup file, which is also embedded in the XML result file) BrechtDeMan@722: \item Box plot, confidence plot, scatter plot of values (for a given audioholder) BrechtDeMan@722: \item Timeline for a specific subject (see Python scripts), perhaps re-playing the experiment in X times realtime. (If actual realtime, you could replay the audio...) BrechtDeMan@722: \item Distribution plots of any radio button and number questions (drop-down menu with `pretest', `posttest', ...; then drop-down menu with question `IDs' like `gender', `age', ...; make pie chart/histogram of these values over selected range of XMLs) BrechtDeMan@722: \item All `comments' on a specific audioelement BrechtDeMan@722: \item A `download' button for a nice CSV of various things (values, survey responses, comments) people might want to use for analysis, e.g. when XML scares them BrechtDeMan@724: \item Validation of setup XMLs (easily spot `errors', like duplicate IDs or URLs, missing/dangling tags, ...) BrechtDeMan@722: \end{itemize} BrechtDeMan@722: BrechtDeMan@722: A subset of the above would already be nice for this paper. BrechtDeMan@722: BrechtDeMan@719: Some pictures here please. BrechtDeMan@719: BrechtDeMan@725: \section{Concluding remarks and future work} BrechtDeMan@719: BrechtDeMan@726: The code and documentation can be pulled or downloaded from \url{code.soundsoftware.ac.uk/projects/webaudioevaluationtool}. BrechtDeMan@719: BrechtDeMan@726: [Talking a little bit about what else might happen. Unless we really want to wrap this up. ] BrechtDeMan@725: BrechtDeMan@726: Use \cite{schoeffler2015mushra} as a `checklist', even though it only considers subjective evaluation of audio systems (and focuses on the requirements for a MUSHRA test). BrechtDeMan@726: % remote BrechtDeMan@726: % language support (not explicitly stated) BrechtDeMan@726: % crossfades BrechtDeMan@726: % choosing speakers/sound device from within browser? BrechtDeMan@726: % collect information about software and sound system BrechtDeMan@726: % buttons, scales, ... UI elements BrechtDeMan@726: % must be able to load uncompressed PCM BrechtDeMan@726: BrechtDeMan@726: [What can we not do? `Method of adjustment', as in \cite{schoeffler2015mushra} is another can of worms, because, like, you could adjust lots of things (volume is just one of them, that could be done quite easily). Same for using input signals like the participant's voice. Either leave out, or mention this requires modification of the code we provide.] BrechtDeMan@719: BrechtDeMan@719: % BrechtDeMan@719: % The following two commands are all you need in the BrechtDeMan@719: % initial runs of your .tex file to BrechtDeMan@719: % produce the bibliography for the citations in your paper. BrechtDeMan@719: \bibliographystyle{abbrv} BrechtDeMan@719: \bibliography{WAC2016} % sigproc.bib is the name of the Bibliography in this case BrechtDeMan@719: % You must have a proper ".bib" file BrechtDeMan@719: % and remember to run: BrechtDeMan@719: % latex bibtex latex latex BrechtDeMan@719: % to resolve all references BrechtDeMan@719: % BrechtDeMan@719: % ACM needs 'a single self-contained file'! BrechtDeMan@719: % BrechtDeMan@719: \end{document}