Mercurial > hg > webaudioevaluationtool

comparison docs/WAC2016/WAC2016.tex @ 716:cbfc18a7d068

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Paper: edits in various sections; Added preliminary analysis page
author Brecht De Man <BrechtDeMan@users.noreply.github.com>
date Wed, 14 Oct 2015 20:12:52 +0100
parents
children 71956765e768
comparison
equal deleted inserted replaced
-1:000000000000 716:cbfc18a7d068
1 \documentclass{sig-alternate}
2 \usepackage{hyperref} % make links (like references, links to Sections, ...) clickable
3 \usepackage{enumitem} % tighten itemize etc by appending '[noitemsep,nolistsep]'
4 \usepackage{cleveref}
5
6 \graphicspath{{img/}} % put the images in this folder
7
8 \begin{document}
9
10 % Copyright
11 \setcopyright{waclicense}
12
13
14 %% DOI
15 %\doi{10.475/123_4}
16 %
17 %% ISBN
18 %\isbn{123-4567-24-567/08/06}
19 %
20 %%Conference
21 %\conferenceinfo{PLDI '13}{June 16--19, 2013, Seattle, WA, USA}
22 %
23 %\acmPrice{\$15.00}
24
25 %
26 % --- Author Metadata here ---
27 \conferenceinfo{Web Audio Conference WAC-2016,}{April 4--6, 2016, Atlanta, USA}
28 \CopyrightYear{2016} % Allows default copyright year (20XX) to be over-ridden - IF NEED BE.
29 %\crdata{0-12345-67-8/90/01} % Allows default copyright data (0-89791-88-6/97/05) to be over-ridden - IF NEED BE.
30 % --- End of Author Metadata ---
31
32 \title{Web Audio Evaluation Tool: A framework for subjective assessment of audio}
33 %\subtitle{[Extended Abstract]
34 %\titlenote{A full version of this paper is available as
35 %\textit{Author's Guide to Preparing ACM SIG Proceedings Using
36 %\LaTeX$2_\epsilon$\ and BibTeX} at
37 %\texttt{www.acm.org/eaddress.htm}}}
38 %
39 % You need the command \numberofauthors to handle the 'placement
40 % and alignment' of the authors beneath the title.
41 %
42 % For aesthetic reasons, we recommend 'three authors at a time'
43 % i.e. three 'name/affiliation blocks' be placed beneath the title.
44 %
45 % NOTE: You are NOT restricted in how many 'rows' of
46 % "name/affiliations" may appear. We just ask that you restrict
47 % the number of 'columns' to three.
48 %
49 % Because of the available 'opening page real-estate'
50 % we ask you to refrain from putting more than six authors
51 % (two rows with three columns) beneath the article title.
52 % More than six makes the first-page appear very cluttered indeed.
53 %
54 % Use the \alignauthor commands to handle the names
55 % and affiliations for an 'aesthetic maximum' of six authors.
56 % Add names, affiliations, addresses for
57 % the seventh etc. author(s) as the argument for the
58 % \additionalauthors command.
59 % These 'additional authors' will be output/set for you
60 % without further effort on your part as the last section in
61 % the body of your article BEFORE References or any Appendices.
62
63 % FIVE authors instead of four, to leave space between first two authors.
64 \numberofauthors{5} % in this sample file, there are a *total*
65 % of EIGHT authors. SIX appear on the 'first-page' (for formatting
66 % reasons) and the remaining two appear in the \additionalauthors section.
67 %
68 \author{
69 % You can go ahead and credit any number of authors here,
70 % e.g. one 'row of three' or two rows (consisting of one row of three
71 % and a second row of one, two or three).
72 %
73 % The command \alignauthor (no curly braces needed) should
74 % precede each author name, affiliation/snail-mail address and
75 % e-mail address. Additionally, tag each line of
76 % affiliation/address with \affaddr, and tag the
77 % e-mail address with \email.
78 %
79 % 1st. author
80 \alignauthor Nicholas Jillings\\
81 \email{n.g.r.jillings@se14.qmul.ac.uk}
82 % dummy author for nicer spacing
83 \alignauthor
84 % 2nd. author
85 \alignauthor Brecht De Man\\
86 \email{b.deman@qmul.ac.uk}
87 \and % use '\and' if you need 'another row' of author names
88 % 3rd. author
89 \alignauthor David Moffat\\
90 \email{d.j.moffat@qmul.ac.uk}
91 % 4th. author
92 \alignauthor Joshua D. Reiss\\
93 \email{joshua.reiss@qmul.ac.uk}
94 \and % new line for address
95 \affaddr{Centre for Digital Music}\\
96 \affaddr{School of Electronic Engineering and Computer Science}\\
97 \affaddr{Queen Mary University of London}\\
98 \affaddr{Mile End Road,}
99 \affaddr{London E1 4NS}\\
100 \affaddr{United Kingdom}\\
101 }
102 %Centre for Digital Music, School of Electronic Engineering and Computer Science, Queen Mary University of London
103 %% 5th. author
104 %\alignauthor Sean Fogarty\\
105 % \affaddr{NASA Ames Research Center}\\
106 % \affaddr{Moffett Field}\\
107 % \email{fogartys@amesres.org}
108 %% 6th. author
109 %\alignauthor Charles Palmer\\
110 % \affaddr{Palmer Research Laboratories}\\
111 % \affaddr{8600 Datapoint Drive}\\
112 % \email{cpalmer@prl.com}
113 %}
114 % There's nothing stopping you putting the seventh, eighth, etc.
115 % author on the opening page (as the 'third row') but we ask,
116 % for aesthetic reasons that you place these 'additional authors'
117 % in the \additional authors block, viz.
118 %\additionalauthors{Additional authors: John Smith (The Th{\o}rv{\"a}ld Group,
119 %email: {\texttt{jsmith@affiliation.org}}) and Julius P.~Kumquat
120 %(The Kumquat Consortium, email: {\texttt{jpkumquat@consortium.net}}).}
121 \date{1 October 2015}
122 % Just remember to make sure that the TOTAL number of authors
123 % is the number that will appear on the first page PLUS the
124 % number that will appear in the \additionalauthors section.
125
126 \maketitle
127 \begin{abstract}
128 Here comes the abstract.
129 \end{abstract}
130
131
132 \section{Introduction}
133
134 % Listening tests/perceptual audio evaluation: what are they, why are they important
135 % As opposed to limited scope of WAC15 paper: also musical features, realism of sound effects / sound synthesis, performance of source separation and other algorithms...
136 Perceptual evaluation of audio, in the form of listening tests, is a powerful way to assess anything from audio codec quality over realism of sound synthesis to the performance of source separation, automated music production and other auditory evaluations.
137 In less technical areas, the framework of a listening test can be used to measure emotional response to music or test cognitive abilities.
138 % maybe some references? If there's space.
139
140 % check out http://link.springer.com/article/10.1007/s10055-015-0270-8 - only paper that cited WAC15 paper
141
142 % Why difficult? Challenges? What constitutes a good interface?
143 % Technical, interfaces, user friendliness, reliability
144 Several applications for performing perceptual listening tests currently exist, as can be seen in Table \ref{tab:toolboxes}. The Web Audio Evaluation Toolbox stands out as it does not require proprietary software or a specific platform and provides a wide range of interface and test types in one user friendly environment. Furthermore, it does not require any progamming experience as any test based on the default test types can be configured in the browser as well. Note that the design of an effective listening test further poses many challenges unrelated to interface design, which are beyond the scope of this paper \cite{bech}.
145
146 % Why in the browser?
147 Web Audio API has important features for performing perceptual tests including sample level manipulation of audio streams \cite{schoeffler2015mushra}, synchronous playback and flexible playback. Being in the browser also allows leveraging the flexible object oriented JavaScript format and native support for web documents, such as the extensible markup language (XML) which is used for configuration and test result files. Using the web also reduces deployment requirements to a basic web server with advanced functionality such as test collection and automatic processing using PHP. As recruiting participants can be very time-consuming, and as for some tests a large number of participants is needed, browser-based tests \cite{schoeffler2015mushra}. However, to our knowledge, no tool currently exists that allows the creation of a remotely accessible listening test.
148
149 Both BeaqleJS \cite{beaqlejs} and mushraJS\footnote{https://github.com/akaroice/mushraJS} also operate in the browser, however BeaqleJS does not make use of the Web Audio API and therefore lacks arbitrary manipulation of audio stream samples, and neither offer an adequately wide choice of test designs for them to be useful to many researchers. %requires programming knowledge?...
150
151 % only browser-based?
152 \begin{table*}[ht]
153 \caption{Table with existing listening test platforms and their features}
154 \begin{center}
155 \begin{tabular}{|*{6}{l|}}
156 \hline
157 \textbf{Name} & \textbf{Ref.} & \textbf{Language} & \textbf{Interfaces} & \textbf{Remote} & \textbf{All UI} \\
158 \hline
159 APE & \cite{ape} & MATLAB & multi-stimulus, 1 axis per attribute & & \\
160 BeaqleJS & \cite{beaqlejs} & JavaScript & ABX, MUSHRA & (not natively supported) & \\
161 HULTI-GEN & \cite{hultigen} & MAX & See Table \ref{tab:toolbox_interfaces} & & \checkmark \\
162 mushraJS & & JavaScript & MUSHRA & \checkmark & \\
163 MUSHRAM & \cite{mushram} & MATLAB & MUSHRA & & \\
164 Scale & \cite{scale} & MATLAB & See Table \ref{tab:toolbox_interfaces} & & & \\
165 WhisPER & \cite{whisper} & MATLAB & See Table \ref{tab:toolbox_interfaces} & & \checkmark \\
166 \textbf{WAET} & \cite{waet} & JavaScript & \textbf{All of the above} & \checkmark & \checkmark \\
167 \hline
168 \end{tabular}
169 \end{center}
170 \label{tab:toolboxes}
171 \end{table*}%
172
173 \begin{table*}[ht]
174 \caption{Table with interfaces and which toolboxes support them}
175 \begin{center}
176 \begin{tabular}{|*{5}{l|}}
177 \hline
178 \textbf{Interface} & \textbf{HULTI-GEN} & \textbf{Scale} & \textbf{WhisPER} & \textbf{WAET} \\
179 \hline
180 MUSHRA (ITU-R BS. 1534) & \checkmark & & & \checkmark \\
181 Rank scale & \checkmark & & & \checkmark \\
182 Likert scale & \checkmark & & \checkmark & \checkmark \\
183 ABC/HR (ITU-R BS. 1116) & \checkmark & & & \checkmark \\
184 -50 to 50 Bipolar with Ref & \checkmark & & & \checkmark \\
185 Absolute Category Rating (ACR) Scale & \checkmark & & & \checkmark \\
186 Degredation Category Rating (DCR) Scale & \checkmark & & & \checkmark \\
187 Comparison Category Rating (CCR) Scale & \checkmark & & \checkmark & \checkmark \\
188 9 Point Hedonic Category Rating Scale & \checkmark & & \checkmark & \checkmark \\
189 ITU-R 5 Point Continuous Impairment Scale & \checkmark & & & \checkmark \\
190 Pairwise Comparison / AB test & \checkmark & & & \checkmark \\
191 Multi-attribute ratings & \checkmark & & & \checkmark \\
192 ABX Test & \checkmark & & & \checkmark \\
193 Adaptive psychophysical methods & & & \checkmark & \\
194 Repertory Grid Technique (RGT) & & & \checkmark & \\
195 Semantic differential & & \checkmark & \checkmark & \\
196 n-Alternative Forced choice & & \checkmark & & \\
197
198 \hline
199 \end{tabular}
200 \end{center}
201 \label{tab:toolbox_interfaces}
202 \end{table*}%
203
204 %
205 %Selling points: remote tests, visualisaton, create your own test in the browser, many interfaces, few/no dependencies, flexibility
206
207 To meet the need for a cross-platform, versatile and easy-to-use listening test tool, we previously developed the Web Audio Evaluation Tool \cite{waet} which at the time of its inception was capable of running a listening test in the browser from an XML configuration file, and storing an XML file as well, with one particular interface. We have now expanded this into a tool with which a wide range of listening test types can easily be constructed and set up remotely, without any need for manually altering code or configuration files, and which allows visualisation of the collected results in the browser. In this paper, we discuss these different aspects and explore which future improvements would be possible. Specifically, in Section \ref{sec:architecture} we cover the general implementation aspects, with a focus on the Web Audio API, followed by a discussion of the requirements for successful remote tests in Section \ref{sec:remote}. Section \ref{sec:interfaces} describes the various interfaces the tool supports, as well as how to keep this manageable. Finally, in Section \ref{sec:analysis} we provide an overview of the analysis capabilities in the browser, before summarising our findings and listing future research directions in Section \ref{sec:conclusion}.
208
209 \begin{comment}
210 % MEETING 8 OCTOBER
211 \subsection{Meeting 8 October}
212 \begin{itemize}
213 \item Do we manipulate audio?\\
214 \begin{itemize}
215 \item Add loudness equalisation? (test\_create.html) Tag with gains.
216 \item Add volume slider?
217 \item Cross-fade (in interface node): default 0, number of seconds
218 \item Also: we use the playback buffer to present metrics of which portion is listened to
219 \end{itemize}
220 \item Logging system information: whichever are possible (justify others)
221 \item Input streams as audioelements
222 \item Capture microphone to estimate loudness (especially Macbook)
223 \item Test page (in-built oscillators): left-right calibration, ramp up test tone until you hear it; optional compensating EQ (future work implementing own filters) --> Highlight issues!
224 \item Record IP address (PHP function, grab and append to XML file)
225 \item Expand anchor/reference options
226 \item AB / ABX
227 \end{itemize}
228
229 \subsubsection{Issues}
230 \begin{itemize}
231 \item Filters not consistent (Nick to test across browsers)
232 \item Playback audiobuffers need to be destroyed and rebuilt each time
233 \item Can't get channel data, hardware input/output...
234 \end{itemize}
235 \end{comment}
236
237 \section{Architecture} % title? 'back end'? % NICK
238 \label{sec:architecture}
239
240 While WAET uses a sparse subset of the Web Audio API functionality, its performance comes directly from using the Web Audio API for audio playback. Listening tests can convey large amounts of information other than obtaining the perceptual relationship between the audio fragments. Specifically, with WAET one can obtain which parts of the audio fragments were listened to and when, at what point in the audio stream the participant switched to a different fragment, and how a fragment's rating was adjusted over time within a session, to name a few. Not only does this allow to evaluate a wealth of perceptual aspects, but it helps detect poor participants whose results are potentially not representative.
241
242 One of the key initial design parameters for WAET is to make the tool as open as possible to non-programmers and to this end the tool has been designed in such a way that all of the user modifiable options are included in a single XML document. This document is loaded up automatically by the web page and the JavaScript code parses and loads any extra resources required to create the test.
243
244 The specification document also contains the URL of the audio fragments for each test page. These fragments are downloaded asynchronously and decoded offline by the Web Audio offline decoder. The resulting buffers are assigned to a custom Audio Objects node which tracks the fragment buffer, the playback bufferSourceNode, the XML information including its unique test ID, the interface object(s) associated with the fragment and any metric or data collection objects. The Audio Object is controlled by an over-arching custom Audio Context node (not to be confused with the Web Audio Context), this parent JS Node allows for session wide control of the Audio Objects including starting and stopping playback of specific nodes.
245
246 The only issue with this model is the bufferNode in the Web Audio API, which is implemented as a `use once' object which, once the buffer has been played, the buffer must be discarded as it cannot be instructed to play the buffer again. Therefore on each start request the buffer object must be created and then linked with the stored bufferSourceNode. This is an odd behaviour for such a simple object which has no alternative except to use the HTML5 audio element, however they do not have the ability to synchronously start on a given time and therefore not suited.
247
248 The media files supported depend on the browser level support for the initial decoding of information and is the same as the browser support for the HTML5 audio element. Therefore the most widely supported media file is the wave (.WAV) format which can be accpeted by every browser supporting the Web Audio API. The next best supported audio only formats are MP3 and AAC (in MP4) which are supported by all major browsers, Firefox relies on OS decoders and therefore its support is predicated by the OS support.
249
250 All the collected session data is returned in an XML document structured similarly to the configuration document, where test pages contain the audio elements with their trace collection, results, comments and any other interface-specific data points.
251
252 %A slightly technical overview of the system. Talk about XML, JavaScript, Web Audio API, HTML5.
253 %Describe and/or visualise audioholder-audioelement-... structure.
254
255 % see also SMC12 - less detail here
256
257 %Which type of files? % WAV, anything else? Perhaps not exhaustive list, but say something along the lines of 'whatever browser supports'
258
259 %Streaming audio? % probably not, unless it's easy
260
261 %Compatibility? % not IE, everything else fine?
262
263
264
265
266 \section{Remote tests} % with previous?
267 \label{sec:remote}
268
269 If the experimenter is willing to trade some degree of control for a higher number of participants, the test can be hosted on a web server so that subjects can take part remotely. This way, a link can be shared widely in the hope of attracting a large amount of subjects, while listening conditions and subject reliability may be less ideal. However, a sound system calibration page and a wide range of metrics logged during the test mitigate these problems. Note also that in some experiments, it may be preferred that the subject has a `real life', familiar listening set-up, for instance when perceived quality differences on everyday sound systems are investigated.
270 Furthermore, a fully browser-based test, where the collection of the results is automatic, is more efficient and technically reliable even when the test still takes place under lab conditions.
271
272 The following features allow easy and effective remote testing:
273 \begin{description}[noitemsep,nolistsep]
274 \item[PHP script to collect result XML files] and store on central server.
275 \item[Randomly pick a specified number of pages] to ensure an equal and randomised spread of the different pages (`audioHolders') across participants.
276 \item[Calibration of the sound system (and participant)] by a perceptual pre-test to gather information about the frequency response and speaker configuration - this can be supplemented with a survey.
277 % In theory calibration could be applied anywhere??
278 % \item Functionality to participate multiple times
279 % \begin{itemize}[noitemsep,nolistsep]
280 % \item Possible to log in with unique ID (no password)
281 % \item Pick `new user' (generates new, unique ID) or `already participated' (need already available ID)
282 % \item Store XML on server with IDs plus which audioholders have already been listened to
283 % \item Don't show `post-test' survey after first time
284 % \item Pick `new' audioholders if available
285 % \item Copy survey information first time to new XMLs
286 % \end{itemize}
287 \item[Intermediate saves] for tests which were interrupted or unfinished.
288 \item[Collect IP address information] for geographic location, through PHP function which grabs address and appends to XML file.
289 \item[Collect Browser and Display information] to the extent it is available and reliable.
290 \end{description}
291
292
293 \section{Interfaces} % title? 'Front end'? % Dave
294 \label{sec:interfaces}
295
296 The purpose of this listening test framework is to allow any user the maximum flexibility to design a listening test for their exact application with minimum effort. To this end, a large range of standard listening test interfaces have been implemented. A review of existing listening test frameworks was undertaken and presented in~\Cref{tab:toolboxes}. HULTI-GEN~\cite{hultigen} is a single toolbox that presents the user with a large number of different test interfaces and allows for customisation of each test interface.
297
298 To provide users with a flexible system, a large range of `standard' listening test interfaces have been implemented, including: % pretty much the same wording as two sentences earlier
299 \begin{itemize}[noitemsep,nolistsep]
300 \item MUSHRA (ITU-R BS. 1534)~\cite{recommendation20031534}
301 \begin{itemize}
302 \item Multiple stimuli are presented and rated on a continuous scale, which includes a reference, hidden reference and hidden anchors.
303 \end{itemize}
304 \item Rank Scale~\cite{pascoe1983evaluation}
305 \begin{itemize}
306 \item Stimuli ranked on single horizontal scale, where they are ordered in preference order.
307 \end{itemize}
308 \item Likert scale~\cite{likert1932technique}
309 \begin{itemize}
310 \item Each stimuli has a five point scale with values: Strongly Agree, Agree, Neutral, Disagree and Strongly Disagree.
311 \end{itemize}
312 \item ABC/HR (ITU-R BS. 1116)~\cite{recommendation19971116} (Mean Opinion Score: MOS)
313 \begin{itemize}
314 \item Each stimulus has a continuous scale (5-1), labeled as Imperceptible, Perceptible but not annoying, slightly annoying, annoying, very annoying.
315 \end{itemize}
316 \item -50 to 50 Bipolar with Ref
317 \begin{itemize}
318 \item Each stimulus has a continuous scale -50 to 50 with default values as 0 in middle and a comparison. There is also a provided reference \end{itemize}
319 \item Absolute Category Rating (ACR) Scale~\cite{rec1996p}
320 \begin{itemize}
321 \item Each stimuli has a five point scale with values: Bad, Poor, Fair, Good, Excellent
322 \end{itemize}
323 \item Degredation Category Rating (DCR) Scale~\cite{rec1996p}
324 \begin{itemize}
325 \item Each stimuli has a five point scale with values: (5) Inaudible, (4) Audible but not annoying, (3) slightly annoying, (2) annoying, (1) very annoying.
326 \end{itemize}
327 \item Comparison Category Rating (CCR) Scale~\cite{rec1996p}
328 \begin{itemize}
329 \item Each stimuli has a seven point scale with values: Much Better, Better, Slightly Better, About the same, slightly worse, worse, much worse. There is also a provided reference.
330 \end{itemize}
331 \item 9 Point Hedonic Category Rating Scale~\cite{peryam1952advanced}
332 \begin{itemize}
333 \item Each stimuli has a seven point scale with values: Like Extremely, Like Very Much, Like Moderate, Like Slightly, Neither Like nor Dislike, dislike Extremely, dislike Very Much, dislike Moderate, dislike Slightly. There is also a provided reference.
334 \end{itemize}
335 \item ITU-R 5 Point Continuous Impairment Scale~\cite{rec1997bs}
336 \begin{itemize}
337 \item Each stimuli has a five point scale with values: (5) Imperceptible, (4) Perceptible but not annoying, (3) slightly annoying, (2) annoying, (1) very annoying. There is also a provided reference.
338 \end{itemize}
339 \item Pairwise Comparison (Better/Worse)~\cite{david1963method}
340 \begin{itemize}
341 \item A reference is provided and ever stimulus is rated as being either better or worse than the reference.
342 \end{itemize}
343 \item APE style \cite{ape}
344 \begin{itemize}
345 \item Multiple stimuli on a single horizontal slider for inter-sample rating.
346 \end{itemize}
347 \item Multi attribute ratings
348 \begin{itemize}
349 \item Multiple stimuli as points on a 2D plane for inter-sample rating (eg. Valence Arousal)
350 \end{itemize}
351 \item AB Test~\cite{lipshitz1981great}
352 \begin{itemize}
353 \item Two stimuli are presented at a time and the participant has to select a preferred stimulus.
354 \end{itemize}
355 \item ABX Test~\cite{clark1982high}
356 \begin{itemize}
357 \item Two stimuli are presented along with a reference and the participant has to select a preferred stimulus, often the closest to the reference.
358 \end{itemize}
359 \end{itemize}
360
361 While implementing all of these interfaces, it is possible to include any number of references, anchors, hidden references and hidden anchors into all of these listening test formats.
362
363 Because of the design choice to separate the core code and interface modules, it is possible for a 3rd party interface to be built with minimal effort. The repository includes documentation on which functions must be called and the specific functions they expect your interface to perform. To this end, there is an `Interface' object which includes functions for creating the on-page comment boxes (including those with radio or checkbox responses), start and stop buttons with function handles pre-attached and the playhead / transport bars.
364
365 %%%% \begin{itemize}[noitemsep,nolistsep]
366 %%%% \item (APE style) \cite{ape}
367 %%%% \item Multi attribute ratings
368 %%%% \item MUSHRA (ITU-R BS. 1534)~\cite{recommendation20031534}
369 %%%% \item Interval Scale~\cite{zacharov1999round}
370 %%%% \item Rank Scale~\cite{pascoe1983evaluation}
371 %%%%
372 %%%% \item 2D Plane rating - e.g. Valence vs. Arousal~\cite{carroll1969individual}
373 %%%% \item Likert scale~\cite{likert1932technique}
374 %%%%
375 %%%% \item {\bf All the following are the interfaces available in HULTI-GEN~\cite{hultigen} }
376 %%%% \item ABC/HR (ITU-R BS. 1116)~\cite{recommendation19971116}
377 %%%% \begin{itemize}
378 %%%% \item Continuous Scale (5-1) Imperceptible, Perceptible but not annoying, slightly annoying, annoying, very annoying. (default Inaudible?)
379 %%%% \end{itemize}
380 %%%% \item -50 to 50 Bipolar with Ref
381 %%%% \begin{itemize}
382 %%%% \item Scale -50 to 50 on Mushra with default values as 0 in middle and a comparison ``Reference'' to compare to 0 value
383 %%%% \end{itemize}
384 %%%% \item Absolute Category Rating (ACR) Scale~\cite{rec1996p}
385 %%%% \begin{itemize}
386 %%%% \item 5 point Scale - Bad, Poor, Fair, Good, Excellent (Default fair?)
387 %%%% \end{itemize}
388 %%%% \item Degredation Category Rating (DCR) Scale~\cite{rec1996p}
389 %%%% \begin{itemize}
390 %%%% \item 5 point Scale - Inaudible, Audible but not annoying, slightly annoying, annoying, very annoying. (default Inaudible?) - {\it Basically just quantised ABC/HR?}
391 %%%% \end{itemize}
392 %%%% \item Comparison Category Rating (CCR) Scale~\cite{rec1996p}
393 %%%% \begin{itemize}
394 %%%% \item 7 point scale: Much Better, Better, Slightly Better, About the same, slightly worse, worse, much worse - Default about the same with reference to compare to
395 %%%% \end{itemize}
396 %%%% \item 9 Point Hedonic Category Rating Scale~\cite{peryam1952advanced}
397 %%%% \begin{itemize}
398 %%%% \item 9 point scale: Like Extremely, Like Very Much, Like Moderate, Like Slightly, Neither Like nor Dislike, dislike Extremely, dislike Very Much, dislike Moderate, dislike Slightly - Default Neither Like nor Dislike with reference to compare to
399 %%%% \end{itemize}
400 %%%% \item ITU-R 5 Point Continuous Impairment Scale~\cite{rec1997bs}
401 %%%% \begin{itemize}
402 %%%% \item 5 point Scale (5-1) Imperceptible, Perceptible but not annoying, slightly annoying, annoying, very annoying. (default Inaudible?)- {\it Basically just quantised ABC/HR, or Different named DCR}
403 %%%% \end{itemize}
404 %%%% \item Pairwise Comparison (Better/Worse)~\cite{david1963method}
405 %%%% \begin{itemize}
406 %%%% \item 2 point Scale - Better or Worse - (not sure how to default this - they default everything to better, which is an interesting choice)
407 %%%% \end{itemize}
408 %%%% \end{itemize}
409
410 % Build your own test
411 \begin{comment}
412 { \bf A screenshot would be nice.
413
414 Established tests (see below) included as `presets' in the build-your-own-test page. }
415 \end{comment}
416
417 \section{Analysis and diagnostics}
418 \label{sec:analysis}
419 % don't mention Python scripts
420 There are several benefits to providing basic analysis tools in the browser: they allow diagnosing problems, with the interface or with the test subject; they may be sufficient for many researchers' purposes; and test subjects may enjoy seeing an overview of their own results and/or results thus far at the end of their tests.
421 % \begin{figure*}[bhf]
422 % \centering
423 % \includegraphics[width=.7\textwidth]{timeline.pdf}
424 % \caption{This timeline of a single subject's listening test shows playback of fragments (red segments) and marker movements on the rating axis in function of time. }
425 % \label{fig:timeline}
426 % \end{figure*}
427 For this reason, we include a proof-of-concept web page with:
428 \begin{itemize}[noitemsep,nolistsep]
429 \item All audioholder IDs, file names, subject IDs, audio element IDs, ... in the collected XMLs so far (\texttt{saves/*.xml})
430 \item Selection of subjects and/or test samples to zoom in on a subset of the data %Check/uncheck each of the above for analysis (e.g. zoom in on a certain song, or exclude a subset of subjects)
431 \item Embedded audio to hear corresponding test samples % (follow path in XML setup file, which is also embedded in the XML result file)
432 \item Box plot, confidence plot, and scatter plot of rating values
433 \item Timeline for a specific subject (see Figure \ref{fig:timeline})%, perhaps re-playing the experiment in X times realtime. (If actual realtime, you could replay the audio...)
434 \item Distribution plots of any radio button and number questions in pre- and post-test survey %(drop-down menu with `pretest', `posttest', ...; then drop-down menu with question `IDs' like `gender', `age', ...; make pie chart/histogram of these values over selected range of XMLs)
435 \item All `comments' on a specific audioelement
436 \item A `download' function for a CSV of ratings, survey responses and comments% various things (values, survey responses, comments) people might want to use for analysis, e.g. when XML scares them
437 %\item Validation of setup XMLs (easily spot `errors', like duplicate IDs or URLs, missing/dangling tags, ...)
438 \end{itemize}
439
440 \section{Concluding remarks and future work}
441 \label{sec:conclusion}
442
443 The code and documentation can be pulled or downloaded from \url{code.soundsoftware.ac.uk/projects/webaudioevaluationtool}.
444
445 [Talking a little bit about what else might happen. Unless we really want to wrap this up. ]
446
447 \cite{schoeffler2015mushra} gives a `checklist' for subjective evaluation of audio systems. The Web Audio Evaluation Toolbox meets most of its given requirements including remote testing, crossfading between audio streams, collecting browser information, utilising UI elements and working with various audio formats including uncompressed PCM or WAV format.
448 % remote
449 % language support (not explicitly stated)
450 % crossfades
451 % choosing speakers/sound device from within browser? --- NOT POSSIBLE, can only determine channel output counts and its up to the hardware to determine
452 % collect information about software and sound system
453 % buttons, scales, ... UI elements
454 % must be able to load uncompressed PCM
455
456 [What can we not do? `Method of adjustment', as in \cite{schoeffler2015mushra} is another can of worms, because, like, you could adjust lots of things (volume is just one of them, that could be done quite easily). Same for using input signals like the participant's voice. Either leave out, or mention this requires modification of the code we provide.]
457
458 %
459 % The following two commands are all you need in the
460 % initial runs of your .tex file to
461 % produce the bibliography for the citations in your paper.
462 \bibliographystyle{abbrv}
463 \bibliography{WAC2016} % sigproc.bib is the name of the Bibliography in this case
464 % You must have a proper ".bib" file
465 % and remember to run:
466 % latex bibtex latex latex
467 % to resolve all references
468 %
469 % ACM needs 'a single self-contained file'!
470 %
471 \end{document}