comparison docs/Instructions/Instructions.tex @ 2103:cc2fb78ddd89

Minor changes to instructions, demo page, and test examples
author Brecht De Man <b.deman@qmul.ac.uk>
date Fri, 19 Feb 2016 16:08:54 +0100
parents 7c39c22a417f
children 124e6c702845
comparison
equal deleted inserted replaced
2102:c589f38abc26 2103:cc2fb78ddd89
238 This is a straightforward implementation of \cite{mushra}, especially common for the rating of audio quality, for instance for the evaluation of audio codecs. This can also operate any vertical slider style test and does not necessarily have to match the MUSHRA specification. 238 This is a straightforward implementation of \cite{mushra}, especially common for the rating of audio quality, for instance for the evaluation of audio codecs. This can also operate any vertical slider style test and does not necessarily have to match the MUSHRA specification.
239 239
240 \subsection{AB} 240 \subsection{AB}
241 Performs a pairwise comparison, but supports ABX and n-way comparison (in the example we demonstrate it performing a 7-way comparison). 241 Performs a pairwise comparison, but supports ABX and n-way comparison (in the example we demonstrate it performing a 7-way comparison).
242 242
243 \subsection{discrete/Likert} 243 \subsection{Discrete/Likert}
244 Each audio element is given a discrete set of values based on the number of slider options specified. For instance, Likert specifies 5 values and therefore each audio element must be one of those 5 values. 244 Each audio element is given a discrete set of values based on the number of slider options specified. For instance, Likert specifies 5 values and therefore each audio element must be one of those 5 values.
245 245
246 \subsection{ACR/CCR/DCR/horizontal} 246 \subsection{ACR/CCR/DCR/horizontal}
247 Creates the same interfaces as MUSHRA except the sliders are horizontal, not vertical. 247 Creates the same interfaces as MUSHRA except the sliders are horizontal, not vertical.
248 248
276 276
277 \subsection{Set up} 277 \subsection{Set up}
278 The first child node, \texttt{<setup>} specifies any one time and global parameters. It takes the following attributes: 278 The first child node, \texttt{<setup>} specifies any one time and global parameters. It takes the following attributes:
279 \begin{itemize} 279 \begin{itemize}
280 \item \texttt{interface}: String, mandatory, specifies the interface to load 280 \item \texttt{interface}: String, mandatory, specifies the interface to load
281 \item \texttt{projectReturn}: URL, mandatory, specifies the return point. Can be a 3rd party server or the local server. Set to null to disable automatic saving. Specifying "save.php" will trigger the return if either the PHP or python servers are used. On error, it will always default to presenting the save on page. 281 \item \texttt{projectReturn}: URL, mandatory, specifies the return point. Can be a 3rd party server or the local server. Set to null to disable automatic saving. Specifying ``save.php'' will trigger the return if either the PHP or python servers are used. On error, it will always default to presenting the save on page.
282 \item \texttt{randomiseOrder}: Boolean, optional, if true it will randomise the order of the test pages. Default is false. 282 \item \texttt{randomiseOrder}: Boolean, optional, if true it will randomise the order of the test pages. Default is false.
283 \item \texttt{testPages}: non-negative integer, optional. Specifies the number of test pages to actually test with. Combined with randomiseOrder being true will give a random set of test pages per participant from the given pool of \texttt{<page>} nodes. Specifying 0 disables this option, default is 0. 283 \item \texttt{testPages}: non-negative integer, optional. Specifies the number of test pages to actually test with. Combined with randomiseOrder being true will give a random set of test pages per participant from the given pool of \texttt{<page>} nodes. Specifying 0 disables this option, default is 0.
284 \item \texttt{loudness}: non-positive integer, optional. Set the default LUFS target value. See \ref{sec:loudness} for more. 284 \item \texttt{loudness}: non-positive integer, optional. Set the default LUFS target value. See \ref{sec:loudness} for more.
285 \item \texttt{sampleRate}: positive integer, optional. If set, the sample rate reported by the Web Audio API must match this number. See \ref{sec:samplerate}. 285 \item \texttt{sampleRate}: positive integer, optional. If set, the sample rate reported by the Web Audio API must match this number. See \ref{sec:samplerate}.
286 \end{itemize} 286 \end{itemize}
295 \subsection{Page} 295 \subsection{Page}
296 \label{sec:page} 296 \label{sec:page}
297 The only other first level child nodes, these specify the test pages. It takes the following attributes: 297 The only other first level child nodes, these specify the test pages. It takes the following attributes:
298 \begin{itemize} 298 \begin{itemize}
299 \item \texttt{id}: ID, mandatory. A string which must be unique across the entire XML. It is used to identify the page on test completion as pages are returned in the results in the order they appeared, not specified. 299 \item \texttt{id}: ID, mandatory. A string which must be unique across the entire XML. It is used to identify the page on test completion as pages are returned in the results in the order they appeared, not specified.
300 \item \texttt{hostURL}: URL, mandatory. Used in conjuction with the \texttt{<audioelement>} url to specify where the audio files are located. For instance if all your files are in the directory \texttt{./test/} you can set this attribute to "/test/" and the \texttt{<audioelement>} url attribute only needs to file name. Set to "" if no hostURL prefix desired. 300 \item \texttt{hostURL}: URL, mandatory. Used in conjuction with the \texttt{<audioelement>} url to specify where the audio files are located. For instance if all your files are in the directory \texttt{./test/} you can set this attribute to ``/test/'' and the \texttt{<audioelement>} url attribute only needs to file name. Set to ``'' if no hostURL prefix desired.
301 \item \texttt{randomiseOrder}: Boolean, optional. If true the audio fragments are presented randomly rather than the order specified. See \ref{sec:randomisation}. Default is false. 301 \item \texttt{randomiseOrder}: Boolean, optional. If true the audio fragments are presented randomly rather than the order specified. See \ref{sec:randomisation}. Default is false.
302 \item \texttt{repeatCount}: non-negative integer, optional. Specify the number of times to repeat the test page (re-present). Each presentation will appear as an individual page in the results. Default is 0. 302 \item \texttt{repeatCount}: non-negative integer, optional. Specify the number of times to repeat the test page (re-present). Each presentation will appear as an individual page in the results. Default is 0.
303 \item \texttt{loop}: Boolean, optional. If true, the audio elements will loop synchronously with each other. See \ref{sec:looping}. Default is false. 303 \item \texttt{loop}: Boolean, optional. If true, the audio elements will loop synchronously with each other. See \ref{sec:looping}. Default is false.
304 \item \texttt{showElementComments}: Boolean, optional. If true then there will be a comment box on the test page for each audio element presented, see \ref{sec:commentboxes}. 304 \item \texttt{showElementComments}: Boolean, optional. If true then there will be a comment box on the test page for each audio element presented, see \ref{sec:commentboxes}.
305 \item \texttt{loudness}: non-positive integer, optional. Set the LUFS target value for this page. Supersedes the \texttt{<setup>} loudness attribute for this page. See \ref{sec:loudness} for more. 305 \item \texttt{loudness}: non-positive integer, optional. Set the LUFS target value for this page. Supersedes the \texttt{<setup>} loudness attribute for this page. See \ref{sec:loudness} for more.
369 As the example project showcases (nearly) all of these features, please refer to its configuration XML document for a demonstration of how to enable and adjust them. 369 As the example project showcases (nearly) all of these features, please refer to its configuration XML document for a demonstration of how to enable and adjust them.
370 370
371 \subsection{Interface options} 371 \subsection{Interface options}
372 The interface node has children of interface options which are used to specify modifications to the test environment. These are divided into two catagories: check and show. Check are used to specify conditions which must be met before a page can be completed, these include checking all fragments have been played or checking all fragments have a comment and so on. Show is used to show an optional on page element or control, such as the playhead or master volume. 372 The interface node has children of interface options which are used to specify modifications to the test environment. These are divided into two catagories: check and show. Check are used to specify conditions which must be met before a page can be completed, these include checking all fragments have been played or checking all fragments have a comment and so on. Show is used to show an optional on page element or control, such as the playhead or master volume.
373 373
374 Check items have the attribute "type" set to "check". The following list gives the string to give the "name" attribute along with a description of the check. 374 Check items have the attribute ``type'' set to ``check''. The following list gives the string to give the ``name'' attribute along with a description of the check.
375 \begin{itemize} 375 \begin{itemize}
376 \item \texttt{fragmentPlayed}: Checks that all fragments have been at least partially played 376 \item \texttt{fragmentPlayed}: Checks that all fragments have been at least partially played
377 \item \texttt{fragmentFullPlayback}: Checks that all fragments have been fully played. \emph{NOTE:} This will always clear if the page is looping as it is not possible to know every sample has been played. 377 \item \texttt{fragmentFullPlayback}: Checks that all fragments have been fully played. \emph{NOTE:} This will always clear if the page is looping as it is not possible to know every sample has been played.
378 \item \texttt{fragmentMoved}: Checks that all fragments have been moved. This is interface dependent, for instance on AB this will always clear as there is no movement. 378 \item \texttt{fragmentMoved}: Checks that all fragments have been moved. This is interface dependent, for instance on AB this will always clear as there is no movement.
379 \item \texttt{fragmentComments}: Cheks that all fragments have a comment. Will clear if there are no on page comments but with a console warning. 379 \item \texttt{fragmentComments}: Cheks that all fragments have a comment. Will clear if there are no on page comments but with a console warning.
380 \item \texttt{scalerange}: Has two extra attributes "min" and "max". Checks that at least one element is below the min value and one element is above the max value. 380 \item \texttt{scalerange}: Has two extra attributes ``min'' and ``max''. Checks that at least one element is below the min value and one element is above the max value.
381 \end{itemize} 381 \end{itemize}
382 382 % QUANTISATION OF THE SCALE: to be implemented?
383 Show items have the attribute "type" set to "show". The following list gives the string to give the "name" attribute along with a description. 383
384 Show items have the attribute ``type'' set to ``show''. The following list gives the string to give the ``name'' attribute along with a description.
384 \begin{itemize} 385 \begin{itemize}
385 \item \texttt{playhead}: Shows the playhead to the end user indicating where in the file they are currently listening 386 \item \texttt{playhead}: Shows the playhead to the end user indicating where in the file they are currently listening
386 \item \texttt{page-count}: Shows the current test page number and the total number of test pages. 387 \item \texttt{page-count}: Shows the current test page number and the total number of test pages.
387 \item \texttt{volume}: Shows a master volume control to the user to manipulate the output gain of the page. This is tracked. 388 \item \texttt{volume}: Shows a master volume control to the user to manipulate the output gain of the page. This is tracked.
388 \end{itemize} 389 \end{itemize}
420 % uncomment once we have it 421 % uncomment once we have it
421 422
422 \subsection{Looping} 423 \subsection{Looping}
423 \label{sec:looping} 424 \label{sec:looping}
424 Looping enables the fragments to loop until stopped by the user. Looping is synchronous so all fragments start at the same time on each loop. 425 Looping enables the fragments to loop until stopped by the user. Looping is synchronous so all fragments start at the same time on each loop.
425 Individual test pages can have their playback looped by the \texttt{<page>} attribute \texttt{loop} with a value of "true" or "false". 426 Individual test pages can have their playback looped by the \texttt{<page>} attribute \texttt{loop} with a value of ``true'' or ``false''.
426 If the fragments are not of equal length initially, they are padded with zeros so that they are equal length, to enable looping without the fragments going out of sync relative to each other. 427 If the fragments are not of equal length initially, they are padded with zeros so that they are equal length, to enable looping without the fragments going out of sync relative to each other.
427 428
428 Note that fragments cannot be played until all page fragments are loaded when in looped mode, as the engine needs to know the length of each fragment to calculate the padding. 429 Note that fragments cannot be played until all page fragments are loaded when in looped mode, as the engine needs to know the length of each fragment to calculate the padding.
429 430
430 \subsection{Sample rate} 431 \subsection{Sample rate}
476 477
477 \subsection{References and anchors} 478 \subsection{References and anchors}
478 \label{sec:referencesandanchors} 479 \label{sec:referencesandanchors}
479 The audio elements, \texttt{<audioelement>} have the attribute \texttt{type}, which defaults to normal. Setting this to one of the following will have the following effects. 480 The audio elements, \texttt{<audioelement>} have the attribute \texttt{type}, which defaults to normal. Setting this to one of the following will have the following effects.
480 \subsubsection{Outside Reference} 481 \subsubsection{Outside Reference}
481 Set type to 'outside-reference'. This will place the object in a separate playback element clearly labelled as an outside reference. This is exempt of any movement checks but will still be included in any listening checks. 482 Set type to `outside-reference'. This will place the object in a separate playback element clearly labelled as an outside reference. This is exempt of any movement checks but will still be included in any listening checks.
482 \subsubsection{Hidden reference} 483 \subsubsection{Hidden reference}
483 Set type to 'reference'. The element will still be randomised as normal (if selected) and presented to the user. However the element will have the 'reference' type in the results to quickly find it. The reference can be forced to be below a value before completing the test page by setting the attribute 'marker' to be a value between 0 and 100 representing the integer value position it must be equal to or above. 484 Set type to `reference'. The element will still be randomised as normal (if selected) and presented to the user. However the element will have the `reference' type in the results to quickly find it. The reference can be forced to be below a value before completing the test page by setting the attribute `marker' to be a value between 0 and 100 representing the integer value position it must be equal to or above.
484 \subsubsection{Hidden anchor} 485 \subsubsection{Hidden anchor}
485 Set type to 'anchor'. The element will still be randomised as normal (if selected) and presented to the user. However the element will have the 'anchor' type in the results to quickly find it. The anchor can be forced to be below a value before completing the test page by setting the attribute 'marker' to be a value between 0 and 100 representing the integer value position it must be equal to or below. 486 Set type to `anchor'. The element will still be randomised as normal (if selected) and presented to the user. However the element will have the `anchor' type in the results to quickly find it. The anchor can be forced to be below a value before completing the test page by setting the attribute `marker' to be a value between 0 and 100 representing the integer value position it must be equal to or below.
486 487
487 \subsection{Checks} 488 \subsection{Checks}
488 \label{sec:checks} 489 \label{sec:checks}
489 490
490 %blabla 491 %blabla
552 % guide to loudness.js 553 % guide to loudness.js
553 Each audio fragment on loading has its loudness calculated. The tool uses the EBU R 128 recommendation following the ITU-R BS.1770-4 loduness calculations to return the integreated LUFS loudness. The attribute \texttt{loudness} will set the loudness from the scope it is applied in. Applying it in the \texttt{<setup>} node will set the loudness for all test pages. Applying it in the \texttt{<page>} node will set the loudness for that page. Applying it in the \texttt{<audioelement>} node will set the loudness for that fragment. The scope is set locally, so if there is a loudness on both the \texttt{<page>} and \texttt{<setup>} nodes, that test page will take the value associated with the \texttt{<page>}. The loudness attribute is set in LUFS 554 Each audio fragment on loading has its loudness calculated. The tool uses the EBU R 128 recommendation following the ITU-R BS.1770-4 loduness calculations to return the integreated LUFS loudness. The attribute \texttt{loudness} will set the loudness from the scope it is applied in. Applying it in the \texttt{<setup>} node will set the loudness for all test pages. Applying it in the \texttt{<page>} node will set the loudness for that page. Applying it in the \texttt{<audioelement>} node will set the loudness for that fragment. The scope is set locally, so if there is a loudness on both the \texttt{<page>} and \texttt{<setup>} nodes, that test page will take the value associated with the \texttt{<page>}. The loudness attribute is set in LUFS
554 555
555 \subsection{Comment Boxes} 556 \subsection{Comment Boxes}
556 \label{sec:commentboxes} 557 \label{sec:commentboxes}
557 There are two types of comment boxes which can be presented, those linked to the audio fragments on the page and those which pose a general question. The audio fragment boxes are shown by setting the attribute \texttt{showElementComments} to true of the page in question. This will then show a comment box below the main interface for every fragment on the page. There is some customisation around the text that accompanies the box, by default the text will read "Comment on fragment " followed by the fragment identifier (the number / letter shown by the interface). This 'prefix' can be modified using the page node \texttt{<commentboxprefix>}, see \ref{sec:page} for where to place this node in the document. The comment box prefix node takes no attribute and the text contained by the node represents to the prefix. For instance if we have a node \texttt{<commentboxprefix> Describe fragment </commentboxprefix>}, then the interface will show "Describe fragment " followed by the identifier. 558 There are two types of comment boxes which can be presented, those linked to the audio fragments on the page and those which pose a general question. The audio fragment boxes are shown by setting the attribute \texttt{showElementComments} to true of the page in question. This will then show a comment box below the main interface for every fragment on the page. There is some customisation around the text that accompanies the box, by default the text will read ``Comment on fragment'' followed by the fragment identifier (the number / letter shown by the interface). This `prefix' can be modified using the page node \texttt{<commentboxprefix>}, see \ref{sec:page} for where to place this node in the document. The comment box prefix node takes no attribute and the text contained by the node represents to the prefix. For instance if we have a node \texttt{<commentboxprefix> Describe fragment </commentboxprefix>}, then the interface will show ``Describe fragment'' followed by the identifier.
558 559
559 The second type of comment box is slightly more complex because it can handle different types of response data. These are called comment questions because they are located in the comment section of the test but pose a specific question. 560 The second type of comment box is slightly more complex because it can handle different types of response data. These are called comment questions because they are located in the comment section of the test but pose a specific question.
560 561
561 \clearpage 562 \clearpage
562 563