Mercurial > hg > webaudioevaluationtool

changeset 745:e010cdf6563d

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Paper: Final edit. Spell checked. Submitted version.
author Nicholas Jillings <nicholas.jillings@eecs.qmul.ac.uk>
date Thu, 15 Oct 2015 22:19:29 +0100
parents 1ac0469ff485
children c64529e5dee4
files docs/WAC2016/WAC2016.pdf docs/WAC2016/WAC2016.tex
diffstat 2 files changed, 34 insertions(+), 82 deletions(-) [+]
line wrap: on
line diff
Binary file docs/WAC2016/WAC2016.pdf has changed
--- a/docs/WAC2016/WAC2016.tex	Thu Oct 15 21:30:19 2015 +0100
+++ b/docs/WAC2016/WAC2016.tex	Thu Oct 15 22:19:29 2015 +0100
@@ -127,7 +127,7 @@
 \maketitle
 \begin{abstract}
 
-Perceptual listening tests are commonplace in audio research and a vital form of evaluation. Many tools exist to run such tests, however many operate one test type and are therefore limited whilst most require proprietary software. Using Web Audio the Web Audio Evaluation Tool (WAET) addresses these concerns by having one toolbox which can be configured to run many differen tests, perform it through a web browser and without needing proprietary software or computer programming knowledge. In this paper the role of the Web Audio API in giving WAET key functionalities are shown. The paper also highlights less common features, available to web based tools, such as easy remote testing environment and in-browser analytics.
+Perceptual listening tests are commonplace in audio research and a vital form of evaluation. Many tools exist to run such tests, however many operate one test type and are therefore limited whilst most require proprietary software. Using Web Audio the Web Audio Evaluation Tool (WAET) addresses these concerns by having one toolbox which can be configured to run many different tests, perform it through a web browser and without needing proprietary software or computer programming knowledge. In this paper the role of the Web Audio API in giving WAET key functionalities are shown. The paper also highlights less common features, available to web based tools, such as easy remote testing environment and in-browser analytics.
 
 \end{abstract}
 
@@ -145,10 +145,11 @@
 	% Why difficult? Challenges? What constitutes a good interface?
 	% Technical, interfaces, user friendliness, reliability 
 	Several applications for performing perceptual listening tests currently exist. A review of existing listening test frameworks was undertaken and presented in~\Cref{tab:toolboxes}. Note that many rely on proprietary, 3rd party software such as MATLAB and MAX, making them less attractive for many. With the exception of the existing JavaScript-based toolboxes, remote deployment (web-based test hosting and result collection) is not possible. 
-	HULTI-GEN~\cite{hultigen} is a single example of a toolbox that presents the user with a large number of different test interfaces and allows for customisation of each test interface, without requiring knowledge of any programming language. The Web Audio Evaluation Toolbox (WAET), presented here, stands out as it does not require proprietary software or a specific platform. It also provides a wide range of interface and test types in one user friendly environment. Furthermore, it does not require any progamming experience as any test based on the default test types can be configured in the browser as well. Note that the design of an effective listening test further poses many challenges unrelated to interface design, which are beyond the scope of this paper \cite{bech}. 
+	
+	HULTI-GEN~\cite{hultigen} is a single example of a toolbox that presents the user with a large number of different test interfaces and allows for customisation of each test interface, without requiring knowledge of any programming language. The Web Audio Evaluation Toolbox (WAET), presented here, stands out as it does not require proprietary software or a specific platform. It also provides a wide range of interface and test types in one user friendly environment. Furthermore any test based on the default test types can be configured in the browser as well. Note that the design of an effective listening test further poses many challenges unrelated to interface design, which are beyond the scope of this paper \cite{bech}. 
 
 	% Why in the browser? 
-	The Web Audio API provides important features for performing perceptual tests including sample level manipulation of audio streams \cite{schoeffler2015mushra} and synchronous and flexible playback. Being in the browser allows leveraging the flexible object oriented JavaScript language and native support for web documents, such as the extensible markup language (XML) which is used for configuration and test result files. Using the web also reduces deployment requirements to a basic web server with some extra functionality such as test collection and automatic processing using PHP. As recruiting participants can be very time-consuming, and as for some tests a large number of participants is needed, browser-based tests can enable participants in multiple locations to perform the test \cite{schoeffler2015mushra}. However, to our knowledge, no tool currently exists that allows the creation of a remotely accessible listening test. 
+	The Web Audio API provides important features including sample level manipulation of audio streams \cite{schoeffler2015mushra} and synchronous and flexible playback. Being in the browser allows leveraging the flexible object oriented JavaScript language and native support for web documents, such as the extensible markup language (XML) which is used for configuration and test result files. Using the web also reduces deployment requirements to a basic web server with extra functionality, such as test collection and automatic processing, using PHP. As recruiting participants can be very time-consuming, and as for some tests a large number of participants is needed, browser-based tests can enable participants in multiple locations to perform the test \cite{schoeffler2015mushra}.
 
 	Both BeaqleJS \cite{beaqlejs} and mushraJS\footnote{https://github.com/akaroice/mushraJS} also operate in the browser. However, BeaqleJS does not make use of the Web Audio API and therefore lacks arbitrary manipulation of audio stream samples, and neither offer an adequately wide choice of test designs for them to be useful to many researchers. %requires programming knowledge?... 
 	
@@ -170,7 +171,7 @@
 	    		 ABC/HR (ITU-R BS. 1116)   & & & \checkmark & & & & & \checkmark \\ \hline
 	    		 -50 to 50 Bipolar with ref. & & & \checkmark & & & & & \checkmark \\ \hline
 	    		 Absolute Category Rating Scale & & & \checkmark & & & & & \checkmark \\ \hline
-	    		 Degredation Category Rating Scale & & & \checkmark & & & & & \checkmark \\ \hline
+	    		 Degradation Category Rating Scale & & & \checkmark & & & & & \checkmark \\ \hline
 	    		 Comparison Category Rating Scale & & & \checkmark & & & & \checkmark & \checkmark \\ \hline
 	    		 9 Point Hedonic Category Rating Scale & & & \checkmark & & & & \checkmark & \checkmark \\ \hline
 	    		 ITU-R 5 Continuous Impairment Scale & & & \checkmark & & & & & \checkmark \\ \hline
@@ -189,9 +190,7 @@
         %Selling points: remote tests, visualisaton, create your own test in the browser, many interfaces, few/no dependencies, flexibility
 
         %[Talking about what we do in the various sections of this paper. Referring to \cite{waet}. ]
-    To meet the need for a cross-platform, versatile and easy-to-use listening test tool, we previously developed the Web Audio Evaluation Tool \cite{waet} which at the time of its inception was capable of running a listening test in the browser from an XML configuration file, and storing an XML file as well, with one particular interface. We have now expanded this into a tool with which a wide range of listening test types can easily be constructed and set up remotely, without any need for manually altering code or configuration files, and which allows visualisation of the collected results in the browser. In this paper, we discuss these different aspects and explore which future improvements would be possible.
-
-    In this paper, we discuss these different aspects and explore which future improvements would be possible. Specifically, in Section \ref{sec:architecture} we cover the general implementation aspects, with a focus on the Web Audio API, followed by a discussion of the requirements for successful remote tests in Section \ref{sec:remote}. Section \ref{sec:interfaces} describes the various interfaces the tool supports, as well as how to keep this manageable. Finally, in Section \ref{sec:analysis} we provide an overview of the analysis capabilities in the browser, before summarising our findings and listing future directions in Section \ref{sec:conclusion}.
+    To meet the need for a cross-platform, versatile and easy-to-use listening test tool, we previously developed the Web Audio Evaluation Tool \cite{waet} which at the time of its inception was capable of running a listening test in the browser from an XML configuration file, and storing an XML file as well, with one particular interface. This has now expanded into a tool with which a wide range of listening test types can easily be constructed and set up remotely, without any need for manually altering code or configuration files, and allows visualisation of the collected results in the browser. In this paper, we discuss these different aspects and explore which future improvements would be possible.
 
     \begin{figure}[tb]
     	\centering
@@ -232,28 +231,29 @@
 \label{sec:architecture}
 %A slightly technical overview of the system. Talk about XML, JavaScript, Web Audio API, HTML5. 
 
-    Although WAET uses a sparse subset of the Web Audio API functionality, its performance comes directly from using it. Listening tests can convey large amounts of information other than obtaining the perceptual relationship between the audio fragments. With WAET it is possible to track which parts of the audio fragments were listened to and when, at what point in the audio stream the participant switched to a different fragment, and how a fragment's rating was adjusted over time within a session, to name a few. Not only does this allow evaluation of a wealth of perceptual aspects, but it also helps detect poor participants whose results are potentially not representative.
+    Although WAET uses a sparse subset of the Web Audio API functionality, its performance comes directly from it. Listening tests can convey large amounts of information other than obtaining the perceptual relationship between the audio fragments. With WAET it is possible to track which parts of the audio fragments were listened to and when, at what point in the audio stream the participant switched to a different fragment, and how a fragment's rating was adjusted over time within a session, to name a few. Not only does this allow evaluation of a wealth of perceptual aspects, but it also helps detect poor participants whose results are potentially not representative.
     
-    One of the key initial design parameters for WAET was to make the tool as open as possible to non-programmers and to this end all of the user modifiable options are included in a single XML document. This document is called the specification document and can be designed either by manually writing the XML (or modifying an existing document or template) or using the included test creator. These are standalone HTML pages which do not require any server or internet connection and help a build the test specification document. The first (test\_create.html) is for simpler tests and operates step-by-step to guide the user. It supports media through drag and drop and a clutter free interface. The advanced version is for more advanced tests where raw XML manipulation is not wanted but the same freedom is required (whilst keeping a safety net). Both models support automatic verification to ensure the XML file is valid and will highlight areas which are either incorrect and would cause an error, or options which should be removed as they are blank.
+    One of the key initial design parameters for WAET was to make the tool as open as possible to non-programmers and to this end all of the user modifiable options are included in a single XML document. This document is the specification document and can be designed either by manually writing the XML (or modifying an existing document or template) or using the included test creator. These standalone HTML pages do not require any server or internet connection and help a build the specification document. The first (test\_create.html) is for simple tests and operates step-by-step to guide the user through a drag and drop, clutter free interface. The advanced version is for more complex tests. Both models support automatic verification to ensure the XML file is valid and will highlight areas which are either incorrect and would cause an error, or options which should be removed as they are blank.
     
-    The basic test creator, Figre \ref{fig:test_create}, utilises the Web Audio API to perform quick playback checks and also allows for loudness normalisation techniques inspired from \cite{ape}. These are calculated offline by accessing the raw audio samples exposed from the buffer before being applied to the audio element as a gain attribute. This is used in the test to perform loudness normalisation without needing to edit any audio files. Equally the gain can be modified in either editor using an HTML5 slider or number box.
-    
+    The basic test creator, Figure \ref{fig:test_create}, utilises the Web Audio API to perform quick playback checks and also allows for loudness normalisation techniques inspired from \cite{ape}. These are calculated offline by accessing the raw audio samples exposed from the buffer before being applied to the audio element as a gain attribute. Therefore the tool performs loudness normalisation without editing any audio files. Equally the gain attribute can be modified in either editor using an HTML5 slider or number box respectively.
+    \begin{comment}
     \begin{figure}[h!]
 	\centering
 	\includegraphics[width=.45\textwidth]{test_create_2.png}
 	\caption{Screen-shot of test creator tool using drag and drop to create specification document}
 	\label{fig:test_create}
 	\end{figure}
+	\end{comment}
     
     %Describe and/or visualise audioholder-audioelement-... structure. 
-    The specification document contains the URL of the audio fragments for each test page. These fragments are downloaded asynchronously in the test and decoded offline by the Web Audio offline decoder. The resulting buffers are assigned to a custom Audio Objects node which tracks the fragment buffer, the playback bufferSourceNode, the XML information including its unique test ID, the interface object(s) associated with the fragment and any metric or data collection objects. The Audio Object is controlled by an over-arching custom Audio Context node (not to be confused with the Web Audio Context). This parent JS Node allows for session wide control of the Audio Objects including starting and stopping playback of specific nodes.
+    The specification document contains the URL of the audio fragments for each test page. These fragments are downloaded asynchronously in the test and decoded offline by the Web Audio offline decoder. The resulting buffers are assigned to a custom Audio Objects node which tracks the fragment buffer, the playback \textit{bufferSourceNode}, other specification attributes including its unique test ID, the interface object(s) associated with the fragment and any metric or data collection objects. The Audio Object is controlled by an over-arching custom Audio Context node (not to be confused with the Web Audio Context). This parent JS Node allows for session wide control of the Audio Objects including starting and stopping playback of specific nodes.
     
-    The only issue with this model is the bufferNode in the Web Audio API, which is implemented in the standard as a `use once' object. Once the bufferNode has been played, the bufferNode must be discarded as it cannot be instructed to play the same bufferSourceNode again. Therefore on each start request the buffer object must be created and then linked with the stored bufferSourceNode. This is an odd behaviour for such a simple object which has no alternative except to use the HTML5 audio element. However, they do not have the ability to synchronously start on a given time and therefore not suited.
+    The only issue with this model is the \textit{bufferNode} in the Web Audio API, implemented in the standard as a `use once' object. Once this has been played, the node must be discarded as it cannot be instructed to play the same \textit{bufferSourceNode} again. Therefore on each play request the buffer object must be created and then linked with the stored \textit{bufferSourceNode}. This is an odd behaviour for such a simple object which has no alternative except to use the HTML5 audio element. However, they do not have the ability to synchronously start on a given time and therefore not suited.
     
-    In the test, each buffer node is connected to a gain node which will operate at the level determined by the specification document. Therefore it is possible to perform a `Method of Adjustment' test where an interface could directly manipulate these gain nodes. There is also an optional `Master Volume' slider which can be shown on the test GUI. This slider modifies a gain node before the destination node. This slider can also be monitored and therefore its data tracked providing extra validation. This slider is not indicative of the final volume exiting the speakers and therefore its use should only be considered in a lab condition environment to ensure proper behaviour. Finally, the gain nodes allow for cross-fading between samples when operating in synchronous playback. Cross-fading can either be fade-out fade-in or a true cross-fade.
+    In the test, each buffer node is connected to a gain node which will operate at the level determined by the specification document. Therefore it is possible to perform a `Method of Adjustment' test where an interface could directly manipulate these gain nodes. These gain nodes are used for cross-fading between samples when operating in synchronous playback. Cross-fading can either be fade-out fade-in or a true cross-fade. There is also an optional `Master Volume' slider which can be shown on the test GUI. This slider modifies a gain node before the destination node. This slider can also be monitored and therefore its data tracked providing extra validation. This is not indicative of the final volume exiting the speakers and therefore its use should only be considered in a lab environment to ensure proper usage.
     
     %Which type of files?  WAV, anything else? Perhaps not exhaustive list, but say something along the lines of 'whatever browser supports'. Compatability?
-    The media files supported depend on the browser level support for the initial decoding of information and is the same as the browser support for the HTML5 audio element. The most widely supported media file is the wave (.WAV) format which is accpeted by every browser supporting the Web Audio API. The toolbox will work in any browser which supports the Web Audio API.
+    The media files supported depend on the browser level support for the initial decoding of information and is the same as the browser support for the HTML5 audio element. The most widely supported media file is the wave (.WAV) format which is accepted by every browser supporting the Web Audio API. The toolbox will work in any browser which supports the Web Audio API.
     
     All the collected session data is returned in an XML document structured similarly to the configuration document, where test pages contain the audio elements with their trace collection, results, comments and any other interface-specific data points.
     
@@ -292,69 +292,29 @@
 To provide users with a flexible system, a large range of `standard' listening test interfaces have been implemented, including: % pretty much the same wording as two sentences earlier
 	\begin{itemize}[noitemsep,nolistsep]
 		\item MUSHRA (ITU-R BS. 1534)~\cite{recommendation20031534}
+		\begin{comment}
 		\begin{itemize}[noitemsep,nolistsep]
 			\item Multiple stimuli are presented and rated on a continuous scale, which includes a reference, hidden reference and hidden anchors.
 		\end{itemize}
-		\item Rank Scale~\cite{pascoe1983evaluation}
-		\begin{itemize}[noitemsep,nolistsep]
-			\item Stimuli ranked on single horizontal scale, where they are ordered in preference order.
-		\end{itemize}
-		\item Likert scale~\cite{likert1932technique}
-		\begin{itemize}[noitemsep,nolistsep]
-			\item Each stimuli has a five point scale with values: Strongly Agree, Agree, Neutral, Disagree and Strongly Disagree.
-		\end{itemize}
-		\item ABC/HR (ITU-R BS. 1116)~\cite{recommendation19971116} (Mean Opinion Score: MOS)
-		\begin{itemize}[noitemsep,nolistsep]
-			\item Each stimulus has a continuous scale (5-1), labeled as Imperceptible, Perceptible but not annoying, slightly annoying, annoying, very annoying.
-		\end{itemize}
-		\item -50 to 50 Bipolar with Ref
-		\begin{itemize}[noitemsep,nolistsep]
-			\item Each stimulus has a continuous scale -50 to 50 with default values as 0 in middle and a comparison. There is also a provided reference		\end{itemize}
-		\item Absolute Category Rating (ACR) Scale~\cite{rec1996p}
-		\begin{itemize}[noitemsep,nolistsep]
-			\item Each stimuli has a five point scale with values: Bad, Poor, Fair, Good, Excellent
-		\end{itemize}
-		\item Degredation Category Rating (DCR) Scale~\cite{rec1996p}
-		\begin{itemize}[noitemsep,nolistsep]
-			\item Each stimuli has a five point scale with values: (5) Inaudible, (4) Audible but not annoying, (3) slightly annoying, (2) annoying, (1) very annoying.
-		\end{itemize}
-		\item Comparison Category Rating (CCR) Scale~\cite{rec1996p}
-		\begin{itemize}[noitemsep,nolistsep]
-			\item Each stimuli has a seven point scale with values: Much Better, Better, Slightly Better, About the same, slightly worse, worse, much worse. There is also a provided reference.
-		\end{itemize}
-		\item 9 Point Hedonic Category Rating Scale~\cite{peryam1952advanced}
-		\begin{itemize}[noitemsep,nolistsep]
-			\item Each stimuli has a seven point scale with values: Like Extremely, Like Very Much, Like Moderate, Like Slightly, Neither Like nor Dislike, dislike Extremely, dislike Very Much, dislike Moderate, dislike Slightly. There is also a provided reference.
-			\end{itemize}
-		\item ITU-R 5 Point Continuous Impairment Scale~\cite{rec1997bs}
-		\begin{itemize}[noitemsep,nolistsep]
-			\item Each stimuli has a five point scale with values: (5) Imperceptible, (4) Perceptible but not annoying, (3) slightly annoying, (2) annoying, (1) very annoying. There is also a provided reference.
-		\end{itemize}
-		\item Pairwise Comparison (Better/Worse)~\cite{david1963method}
-		\begin{itemize}[noitemsep,nolistsep]
-			\item A reference is provided and ever stimulus is rated as being either better or worse than the reference.
-		\end{itemize}
-		\item APE style \cite{ape}
-		\begin{itemize}[noitemsep,nolistsep]
-			\item Multiple stimuli on a single horizontal slider for inter-sample rating.
-		\end{itemize}
-		\item Multi attribute ratings
-		\begin{itemize}[noitemsep,nolistsep]
-			\item Multiple stimuli as points on a 2D plane for inter-sample rating (eg. Valence Arousal)
-		\end{itemize}
-		\item AB Test~\cite{lipshitz1981great}
-		\begin{itemize}[noitemsep,nolistsep]
-			\item Two stimuli are presented at a time and the participant has to select a preferred stimulus.
-		\end{itemize}
-		\item ABX Test~\cite{clark1982high}
-		\begin{itemize}[noitemsep,nolistsep]
-			\item Two stimuli are presented along with a reference and the participant has to select a preferred stimulus, often the closest to the reference.
-		\end{itemize}
+		\end{comment}
+		\item Rank Scale~\cite{pascoe1983evaluation}: stimuli ranked on single horizontal scale, where they are ordered in preference order.
+		\item Likert scale~\cite{likert1932technique}: each stimuli has a five point scale with values: Strongly Agree, Agree, Neutral, Disagree and Strongly Disagree.
+		\item ABC/HR (ITU-R BS. 1116)~\cite{recommendation19971116} (Mean Opinion Score: MOS): each stimulus has a continuous scale (5-1), labeled as Imperceptible, Perceptible but not annoying, slightly annoying, annoying, very annoying.
+		\item -50 to 50 Bipolar with Ref: each stimulus has a continuous scale -50 to 50 with default values as 0 in middle and a reference.
+		\item Absolute Category Rating (ACR) Scale~\cite{rec1996p}: Likert but labels are Bad, Poor, Fair, Good, Excellent
+		\item Degredation Category Rating (DCR) Scale~\cite{rec1996p}: ABC \& Likert but labels are (5) Inaudible, (4) Audible but not annoying, (3) slightly annoying, (2) annoying, (1) very annoying.
+		\item Comparison Category Rating (CCR) Scale~\cite{rec1996p}: ACR \& DCR but 7 point scale: Much Better, Better, Slightly Better, About the same, slightly worse, worse, much worse. There is also a provided reference.
+		\item 9 Point Hedonic Category Rating Scale~\cite{peryam1952advanced}: each stimuli has a seven point scale with values: Like Extremely, Like Very Much, Like Moderate, Like Slightly, Neither Like nor Dislike, dislike Extremely, dislike Very Much, dislike Moderate, dislike Slightly. There is also a provided reference.
+		\item ITU-R 5 Point Continuous Impairment Scale~\cite{rec1997bs}: Same as ABC/HR but with a reference.
+		\item Pairwise Comparison (Better/Worse)~\cite{david1963method}: every stimulus is rated as being either better or worse than the reference.
+		\item APE style \cite{ape}: Multiple stimuli as points on a 2D plane for inter-sample rating (eg. Valence Arousal)
+		\item AB Test~\cite{lipshitz1981great}: Two stimuli presented at a time, participant selects a preferred stimulus.
+		\item ABX Test~\cite{clark1982high}: Two stimuli are presented along with a reference and the participant has to select a preferred stimulus, often the closest to the reference.
 	\end{itemize}
 	
 	It is possible to include any number of references, anchors, hidden references and hidden anchors into all of these listening test formats.
 	
-	Because of the design choice to separate the core code and interface modules, it is possible for a 3rd party interface to be built with minimal effort. The repository includes documentation on which functions must be called and the specific functions they expect your interface to perform. To this end, there is an `Interface' object which includes object prototypes for creating the on-page comment boxes (including those with radio or checkbox responses), start and stop buttons with function handles pre-attached and the playhead / transport bars.
+	Because of the design to separate the core code and interface modules, it is possible for a 3rd party interface to be built with minimal effort. The repository includes documentation on which functions must be called and the specific functions they expect your interface to perform. The core includes an `Interface' object which includes object prototypes for the on-page comment boxes (including those with radio or checkbox responses), start and stop buttons and the playhead / transport bars.
 	
 %%%%	\begin{itemize}[noitemsep,nolistsep]
 %%%%		\item (APE style) \cite{ape}
@@ -438,15 +398,11 @@
 \section{Concluding remarks and future work}
 \label{sec:conclusion}
 
-	We have developed a browser-based tool for the design and deployment of listening tests, essentially requiring no programming experience and third party software. 
+	We have developed a browser-based tool for the design and deployment of listening tests, essentially requiring no programming experience and third party software. Following the predictions or guidelines in \cite{schoeffler2015mushra}, it supports remote testing, cross-fading between audio streams, collecting information about the system, among others. 
 
-	Following the predictions or guidelines in \cite{schoeffler2015mushra}, it supports remote testing, cross-fading between audio streams, collecting information about the system, among others. 
-
-	Whereas many other types of interfaces do exist, we felt that supporting e.g. a range of `method of adjusment' tests would be beyond the scope of a tool that aims to be versatile enough while not claiming to support any custom experiment one might want to set up. Rather, it supports any non-adaptive listening test up to multi-stimulus, multi-attribute evaluation including references, anchors, text boxes, radio buttons and/or checkboxes, with arbitrary placement of the various UI elements. 
+	Whereas many other types of interfaces do exist, we felt that supporting e.g. a range of `method of adjustment' tests would be beyond the scope of a tool that aims to be versatile enough while not claiming to support any custom experiment one might want to set up. Rather, it supports any non-adaptive listening test up to multi-stimulus, multi-attribute evaluation including references, anchors, text boxes, radio buttons and/or checkboxes, with arbitrary placement of the various UI elements. 
 	
 	The code and documentation can be pulled or downloaded from our online repository available at \url{code.soundsoftware.ac.uk/projects/webaudioevaluationtool}.
-	
-	\cite{schoeffler2015mushra} gives a `checklist' for subjective evaluation of audio systems. The Web Audio Evaluation Toolbox meets most of its given requirements including remote testing, crossfading between audio streams, collecting browser information, utilising UI elements and working with various audio formats including uncompressed PCM or WAV format.
 		% remote
 		% language support (not explicitly stated)
 		% crossfades
@@ -455,16 +411,12 @@
 		% buttons, scales, ... UI elements
 		% must be able to load uncompressed PCM
 
-    The use of the Web Audio API is therefore key to WAET to meeting these requirements and others for performing perceptual evaluation tests. Along with the power of the HTML DOM environment giving the ability to interact with all on-page elements creates a powerful and flexible tool capable of performing a multitude of tests out of the box, whilst other tests could easily be built on top of the framework provided.
-    \begin{comment}
-	[What can we not do? `Method of adjustment', as in \cite{schoeffler2015mushra} is another can of worms, because, like, you could adjust lots of things (volume is just one of them, that could be done quite easily). Same for using input signals like the participant's voice. Either leave out, or mention this requires modification of the code we provide.]
-	\end{comment}
-
 %
 % The following two commands are all you need in the
 % initial runs of your .tex file to
 % produce the bibliography for the citations in your paper.
 \bibliographystyle{ieeetr}
+\small
 \bibliography{WAC2016}  % sigproc.bib is the name of the Bibliography in this case
 % You must have a proper ".bib" file
 %  and remember to run: