# HG changeset patch # User Nicholas Jillings # Date 1453831383 0 # Node ID fb062819d9562fe7c91e55979e502256d0cea44d # Parent a679987d13cc454ad5673d2c32f04dd1cafdb075 Updated the Instructions to match recent specification changes. wip. diff -r a679987d13cc -r fb062819d956 docs/Instructions/Instructions.pdf Binary file docs/Instructions/Instructions.pdf has changed diff -r a679987d13cc -r fb062819d956 docs/Instructions/Instructions.tex --- a/docs/Instructions/Instructions.tex Tue Jan 26 14:54:39 2016 +0000 +++ b/docs/Instructions/Instructions.tex Tue Jan 26 18:03:03 2016 +0000 @@ -47,14 +47,12 @@ \textbf{Main folder:} \begin{itemize} \item \texttt{analyse.html}: analysis and diagnostics of a set of result XML files - \item \texttt{ape.css, core.css, graphics.css, mushra.css, structure.css}: style files (edit to change appearance) - \item \texttt{ape.js}: JavaScript file for APE-style interface \cite{ape} + \item \texttt{core.css, graphics.css, structure.css}: core style files (edit to change appearance) \item \texttt{CITING.txt, LICENSE.txt, README.txt}: text files with, respectively, the citation which we ask to include in any work where this tool or any portion thereof is used, modified or otherwise; the license under which the software is shared; and a general readme file referring to these instructions. \item \texttt{core.js}: JavaScript file with core functionality \item \texttt{index.html}: webpage where interface should appear (includes link to test configuration XML) \item \texttt{jquery-2.1.4.js}: jQuery JavaScript Library \item \texttt{loudness.js}: Allows for automatic calculation of loudness of Web Audio API Buffer objects, return gain values to correct for a target loudness or match loudness between multiple objects - \item \texttt{mushra.js}: JavaScript file for MUSHRA-style interface \cite{mushra} \item \texttt{pythonServer.py}: webserver for running tests locally \item \texttt{pythonServer-legacy.py}: webserver with limited functionality (no automatic storing of output XML files) \item \texttt{save.php}: PHP script to store result XML files to web server\\ @@ -72,6 +70,11 @@ \begin{itemize} \item An example of what the set up XML should look like, with example audio files 0.wav-10.wav which are short recordings at 44.1kHz, 16bit of a woman saying the corresponding number (useful for testing randomisation and general familiarisation with the interface).\\ \end{itemize} + \textbf{Interface files (./interfaces/} + \begin{itemize} + \item Each interface class has a JavaScript file and an optional CSS style file. These are loaded as needed. + \end{itemize} + \textbf{Output files (./saves/)} \begin{itemize} \item The output XML files of tests will be stored here by default by the \texttt{pythonServer.py} script.\\ @@ -216,7 +219,7 @@ Then, just go to the URL of the corresponding HTML file, e.g. \texttt{http://server.com/path/to/WAET/index.html?url=test/my-test.xml}. If storing on the server doesn't work at submission (e.g. if the \texttt{projectReturn} attribute isn't properly set), the result XML file will be presented to the subject on the client side, as a `Save file' link. - \subsection{Multiple test documents} + \subsection{Load a test / Multiple test documents} By default the index page will load a demo page of tests. To automatically load a test document, you need to append the location in the URL. If your URL is normally http://localhost:8000/index.html you would append the following: \texttt{?url=/path/to/your/test.xml}. Replace the fields with your actual path, the path is local to the running directory, so if you have your test in the directory \texttt{example\_eval} called \texttt{project.xml} you would append \texttt{?url=/example\_eval/project.xml}. \clearpage @@ -225,53 +228,163 @@ The Web Audio Evaluation Tool comes with a number of interface styles, each of which can be customised extensively, either by configuring them differently using the many optional features, or by modifying the JavaScript files. - To set the interface style for the whole test, %Nick? change when this is not the case anymore, i.e. when the interface can be set per page - add \texttt{interface="APE"} to the \texttt{setup} node, where \texttt{"APE"} is one of the interface names below. + To set the interface style for the whole test, set the attribute of the \texttt{setup} node to \texttt{interface="APE"}, where \texttt{"APE"} is one of the interface names below. \subsection{APE} The APE interface is based on \cite{ape}, and consists of one or more axes, each corresponding with an attribute to be rated, on which markers are placed. As such, it is a multiple stimulus interface where (for each dimension or attribute) all elements are on one axis so that they can be maximally compared against each other, as opposed to rated individually or with regards to a single reference. It also contains an optional text box for each element, to allow for clarification by the subject, tagging, and so on. \subsection{MUSHRA} - This is a straightforward implementation of \cite{mushra}, especially common for the rating of audio quality, for instance for the evaluation of audio codecs. + This is a straightforward implementation of \cite{mushra}, especially common for the rating of audio quality, for instance for the evaluation of audio codecs. This can also operate any vertical slider style test and does not necessarily have to match the MUSHRA specification. + + \subsection{AB} + Performs a pairwise comparison, but supports ABX and n-way comparison (in the example we demonstrate it performing a 7-way comparison). + + \subsection{discrete/Likert} + Each audio element is given a discrete set of values based on the number of slider options specified. For instance, Likert specifies 5 values and therefore each audio element must be one of those 5 values. + + \subsection{ACR/CCR/DCR/horizontal} + Creates the same interfaces as MUSHRA except the sliders are horizontal, not vertical. \clearpage +\section{Project XML} + + Each test is defined by its project XML file, examples of these can be seen in the ./example\_eval/ directory. + + In the XML there are several nodes which must be defined: + \begin{itemize} + \item \texttt{}: The root node. + \item \texttt{}: The first child node, defines whole-test parameters + \item \texttt{}: Specifies a test page, attached \emph{after} the \texttt{}. + \item \texttt{}: Specifies an audio element. + \end{itemize} + + The test uses XML validation, so the ordering of nodes is important to pass this validation. Some nodes also have specific attributes which must be set and may even have a certain format to apply them. This is done so error checking can be performed both quickly and succintly with easy to find errors before loading and running a test session. + + Before identifying any features, this part will walk you through the available nodes, their function and their attributes. + + \subsection{Root} + The root node is \texttt{}, it must have the following attributes: + + \texttt{xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"} + + \texttt{xsi:noNamespaceSchemaLocation="test-schema.xsd"}. + + This will ensure it is checked against the XML schema for validation. + + \subsection{Set up} + The first child node, \texttt{} specifies any one time and global parameters. It takes the following attributes: + \begin{itemize} + \item \texttt{interface}: String, mandatory, specifies the interface to load + \item \texttt{projectReturn}: URL, mandatory, specifies the return point. Can be a 3rd party server or the local server. Set to null to disable automatic saving. Specifying "save.php" will trigger the return if either the PHP or python servers are used. On error, it will always default to presenting the save on page. + \item \texttt{randomiseOrder}: Boolean, optional, if true it will randomise the order of the test pages. Default is false. + \item \texttt{testPages}: non-negative integer, optional. Specifies the number of test pages to actually test with. Combined with randomiseOrder being true will give a random set of test pages per participant from the given pool of \texttt{} nodes. Specifying 0 disables this option, default is 0. + \item \texttt{loudness}: non-positive integer, optional. Set the default LUFS target value. See \ref{sec:loudness} for more. + \item \texttt{sampleRate}: positive integer, optional. If set, the sample rate reported by the Web Audio API must match this number. See \ref{sec:samplerate}. + \end{itemize} + + The \texttt{} node takes the following child nodes, note these must appear in this order: + \begin{itemize} + \item \texttt{}: Min of 0, max of 2 occurences. See \ref{sec:survey} + \item \texttt{}: Must appear only once. + \item \texttt{}: Must appear only once. + \end{itemize} + + \subsection{Page} + The only other first level child nodes, these specify the test pages. It takes the following attributes: + \begin{itemize} + \item \texttt{id}: ID, mandatory. A string which must be unique across the entire XML. It is used to identify the page on test completion as pages are returned in the results in the order they appeared, not specified. + \item \texttt{hostURL}: URL, mandatory. Used in conjuction with the \texttt{} url to specify where the audio files are located. For instance if all your files are in the directory \texttt{./test/} you can set this attribute to "/test/" and the \texttt{} url attribute only needs to file name. Set to "" if no hostURL prefix desired. + \item \texttt{randomiseOrder}: Boolean, optional. If true the audio fragments are presented randomly rather than the order specified. See \ref{sec:randomisation}. Default is false. + \item \texttt{repeatCount}: non-negative integer, optional. Specify the number of times to repeat the test page (re-present). Each presentation will appear as an individual page in the results. Default is 0. + \item \texttt{loop}: Boolean, optional. If true, the audio elements will loop synchronously with each other. See \ref{sec:looping}. Default is false. + \item \texttt{showElementComments}: Boolean, optional. If true then there will be a comment box on the test page for each audio element presented, see \ref{sec:commentboxes}. + \item \texttt{loudness}: non-positive integer, optional. Set the LUFS target value for this page. Supersedes the \texttt{} loudness attribute for this page. See \ref{sec:loudness} for more. + \end{itemize} + + The \texttt{} node takes the following child, nodes note these must appear in this order: + \begin{itemize} + \item \texttt{}: Appear once or not at all. The text content of this node specifies the title of the test page, for instance \texttt{<title>John Doe's Test} + \item \texttt{}: Must appear only once. + \item \texttt{}: Minimum of one. Specifies an audio element, see \ref{sec:audioelement}. + \item \texttt{}: Min of 0, max of 2 occurences. See \ref{sec:survey} + \end{itemize} + + \subsection{Survey} + \label{sec:survey} + These specify any survey items to be presented. The must be a maximum of two of these per \texttt{} and \texttt{} nodes. These have one attribute, location, which must be set to one of the following: before, pre, after or post. In this case before == pre and after == post. This specifies where the survey must appear before or after the node it is associated with. When a child of \texttt{} then pre/before will be shown before the first test page and after/post shown after completing the last test page. When a child of \texttt{} then pre/before is before the test commences and after/post is once the test has been submitted. + + The survey node takes as its only set of childs the \texttt{} node of which there can be any number. + + \subsubsection{Survey Entry} + These nodes have the following attributes, which vary depending on the survey type wanted: + \begin{itemize} + \item \texttt{id}: ID, mandatory. Must be unique across the entire XML, used to identify the response in the results. + \item \texttt{type}: String, mandatory. Must be one of the following: statement, question, checkbox, radio or number. This defines the type to show. + \item \texttt{mandatory}: Boolean, optional. Defines if the survey must have a response or not. Does not apply to statements. Default is false. + \item \texttt{min}: Number, optional. Only applies when \texttt{type="number"}, the minimum valid response. + \item \texttt{max}: Number, optional. Only applies when \texttt{type="number"}, the maximum valid response. + \item \texttt{boxsize}: String, optional. Only applies when \texttt{type="question"} and must be one of the following: normal (default), small, large or huge. + \end{itemize} + + The nodes have the following children, which vary depending on the survey type wanted. + \begin{itemize} + \item \texttt{}: Must appear only once. Its text content specifies the text to appear as the statement or question for the user to respond to. + \item \texttt{