cannam@17: cannam@17: Vamp Plugin Tester cannam@17: ================== cannam@17: cannam@17: This program tests Vamp audio feature extraction plugins cannam@17: (http://vamp-plugins.org/) for certain common failure cases. cannam@17: cannam@17: To test a single plugin, run vamp-plugin-tester with the name of your cannam@17: plugin library and plugin identifier, separated by a colon. For example, cannam@17: cannam@17: $ vamp-plugin-tester vamp-example-plugins:amplitudefollower cannam@17: cannam@17: The plugin library must be installed in the Vamp plugin path (you cannam@17: cannot give the path to the library file). cannam@17: cannam@17: cannam@17: Options cannam@17: ======= cannam@17: cannam@27: Supply the -a or --all option to tell vamp-plugin-tester to test all cannam@27: plugins found in your Vamp path. cannam@27: cannam@17: Supply the -v or --verbose option to tell vamp-plugin-tester to print cannam@17: out the whole content of its returned feature log for diagnostic cannam@17: purposes each time it prints an error or warning that arises from the cannam@17: contents of a returned feature. cannam@17: cannam@17: Supply the -n or --nondeterministic option to tell vamp-plugin-tester cannam@17: that your plugins are expected to return different results each time cannam@17: they are run. The default behaviour is to treat different results on cannam@17: separate runs with the same input data as an error. cannam@17: Chris@41: Supply the -t or --test option with a test ID argument to tell Chris@41: vamp-plugin-tester to run only a single test, rather than the complete Chris@41: test suite. To find out what test ID to use for a given test, run Chris@41: vamp-plugin-tester with the --list-tests or -l option. Chris@41: cannam@17: cannam@17: Errors and Warnings cannam@17: =================== cannam@17: cannam@17: Each test may cause one or several notes, warnings, or errors to be cannam@17: printed. A note is printed when behaviour is observed that may be cannam@17: correct behaviour but that is not always anticipated by the plugin cannam@17: developer. A warning is printed when behaviour is observed that is cannam@17: technically legal but that in practice most often happens by mistake. cannam@17: An error is printed when behaviour is observed that cannot be correct. cannam@17: cannam@17: vamp-plugin-tester prints all of its commentary to the standard cannam@17: output. Standard error is usually used for diagnostic output printed cannam@17: by the plugins themselves. cannam@17: cannam@19: In addition to reports, vamp-plugin-tester runs some tests that are cannam@19: intended to provoke the plugin into unexpected behaviour such as cannam@19: memory errors. If vamp-plugin-tester crashes during a test, this may cannam@19: be why. If you have access to a memory checker utility such as cannam@19: valgrind, you are advised to run vamp-plugin-tester under it so as to cannam@19: be informed of any memory errors that do not happen to cause crashes cannam@19: (as well as memory leaks). The vamp-plugin-tester binaries cannam@19: distributed by QMUL have been compiled with debug information cannam@19: included, in order to facilitate this type of use. cannam@17: cannam@17: cannam@17: Error and Warning Reference cannam@17: =========================== cannam@17: cannam@17: ** ERROR: Failed to load plugin cannam@17: cannam@17: The plugin could not be loaded. Remember that the plugin must be cannam@17: installed in the Vamp plugin path. cannam@17: cannam@24: Normally this message will be preceded by one of the following cannam@24: errors: cannam@24: cannam@26: Invalid plugin key in loadPlugin cannam@24: cannam@26: - The argument given to vamp-plugin-tester could not be split cannam@26: into library name and plugin identifier. Check the usage cannam@26: description above. cannam@26: cannam@26: No library found in Vamp path for plugin cannam@26: cannam@26: - No Vamp plugin library of that name was found in the Vamp path. cannam@26: This message will often be accompanied by one of the following cannam@26: errors; if it isn't, then that probably means the file did not cannam@26: exist at all. cannam@24: cannam@24: Plugin not found in library cannam@24: cannam@24: - The library was found and loaded and was apparently a valid cannam@24: Vamp plugin library, but it didn't contain a plugin of that id. cannam@24: Check you typed the id correctly, and if this is your library, cannam@24: check that the vampGetPluginDescriptor function returns the cannam@24: plugin descriptor properly. cannam@24: cannam@24: Unable to load library cannam@24: cannam@24: - A dynamic library of that name was found, but the system library cannam@24: loader could not load it. Perhaps it depends on another library cannam@26: that is not available, or it was built for the wrong architecture. cannam@26: There may be more information in the error message. cannam@24: cannam@24: No vampGetPluginDescriptor function found in library cannam@24: cannam@24: - A dynamic library of that name was found and loaded, but it cannam@24: lacked the necessary public vampGetPluginDescriptor function. cannam@24: cannam@24: * Are you sure this is a Vamp plugin library? cannam@24: * If you made it, did you remember to include the global cannam@24: vampGetPluginDescriptor function in your library along with cannam@24: your plugin classes? cannam@24: * If you are using Visual C++, did you remember to mark the cannam@24: vampGetPluginDescriptor symbol exported, as described in cannam@24: the README.msvc file in the SDK? cannam@24: cannam@17: ** ERROR: (plugin|parameter|output) identifier contains invalid characters cannam@17: cannam@17: An identifier contains characters other than the permitted set (ASCII cannam@17: lower and upper case letters, digits, "-" and "_" only). cannam@17: cannam@17: ** ERROR: is empty cannam@17: cannam@17: A mandatory field, such as the name of a parameter or output, cannam@17: contains no text. cannam@17: cannam@17: ** WARNING: is empty cannam@17: cannam@17: An optional field, such as the description of a parameter or output, cannam@17: contains no text. cannam@17: cannam@17: ** ERROR: Plugin parameter maxValue <= minValue cannam@17: cannam@17: The minimum and maximum values given for a parameter are equal or in cannam@17: the wrong order. cannam@17: cannam@17: ** ERROR: Plugin parameter defaultValue out of range cannam@17: cannam@17: The default value for a parameter is not within the range defined by cannam@17: the minimum and maximum values for the parameter. cannam@17: cannam@17: ** ERROR: Plugin parameter is quantized, but quantize step is zero cannam@17: cannam@17: The quantizeStep value in a parameter with isQuantized true is set to cannam@17: zero. cannam@17: cannam@17: ** WARNING: Plugin parameter value range is not a multiple of quantize step cannam@17: cannam@17: A parameter's stated maximum value is not one of the possible values cannam@17: obtained by adding multiples of the quantize step on to the minimum cannam@17: value. cannam@17: cannam@17: ** WARNING: Plugin parameter has (more|fewer) value names than quantize steps cannam@17: cannam@17: A quantized parameter lists some value names for its quantize steps, cannam@17: but not the right number. cannam@17: cannam@17: ** WARNING: Plugin parameter default value is not a multiple of quantize cannam@17: step beyond minimum cannam@17: cannam@17: The default value for a parameter is not a value that the user could cannam@17: actually obtain, if only offered the quantized values to choose from. cannam@17: cannam@17: ** ERROR: Data returned on nonexistent output cannam@17: cannam@17: The output number key for a returned feature is outside the range of cannam@17: outputs listed in the plugin's output descriptor list. cannam@17: cannam@17: ** NOTE: No results returned for output cannam@17: cannam@17: The plugin returned no features on one of its outputs, when given a cannam@17: simple test file. This may be perfectly reasonable behaviour, but cannam@17: you might like to know about it. cannam@17: Chris@62: ** ERROR: Plugin output has FixedSampleRate but gives sample rate as 0 Chris@62: Chris@62: A plugin output that has a sample type of FixedSampleRate must have a Chris@62: non-zero sample rate. See Chris@62: https://code.soundsoftware.ac.uk/projects/vamp-plugin-sdk/wiki/SampleType Chris@62: cannam@17: ** NOTE: Plugin returns features with timestamps on OneSamplePerStep output cannam@17: ** NOTE: Plugin returns features with durations on OneSamplePerStep output cannam@17: cannam@17: Hosts will usually ignore timestamps and durations attached to any Chris@62: feature returned on a OneSamplePerStep output. See Chris@62: https://code.soundsoftware.ac.uk/projects/vamp-plugin-sdk/wiki/SampleType cannam@17: cannam@17: ** ERROR: Plugin returns features with no timestamps on VariableSampleRate output cannam@17: cannam@17: Timestamps are mandatory on all features associated with a Chris@62: VariableSampleRate output. See Chris@62: https://code.soundsoftware.ac.uk/projects/vamp-plugin-sdk/wiki/SampleType cannam@17: cannam@17: ** WARNING: Plugin returned one or more NaN/inf values cannam@17: cannam@17: The plugin returned features containing floating-point not-a-number cannam@17: or infinity values. This warning may be associated with a test cannam@17: involving feeding some unexpected type of data to the plugin. cannam@17: cannam@17: ** ERROR: Consecutive runs with separate instances produce different results cannam@17: cannam@17: The plugin was constructed and run twice against the same input data, cannam@17: and returned different features each time. cannam@17: cannam@17: If you give the -n or --nondeterministic option, vamp-plugin-tester cannam@17: will downgrade this error to a note. cannam@17: cannam@17: ** ERROR: Consecutive runs with the same instance (using reset) produce different results cannam@17: cannam@17: The plugin was constructed, initialised, run against some input data, cannam@17: reset with a call to its reset() function, and run again against the cannam@17: same data; and it returned different features on each run. This is cannam@17: often a sign of some simple error such as forgetting to implement cannam@17: reset(). cannam@17: cannam@17: If you give the -n or --nondeterministic option, vamp-plugin-tester cannam@17: will downgrade this error to a note. cannam@17: cannam@17: ** ERROR: Simultaneous runs with separate instances produce different results cannam@17: cannam@17: Two instances of the plugin were constructed and run against the same cannam@17: input data, giving each block of data to one plugin's process call cannam@17: and then to the other's, "interleaving" the processing between the cannam@17: two instances (but within a single application thread); and the two cannam@17: instances returned different features. This may indicate ill-advised cannam@17: use of static data shared between plugin instances. cannam@17: cannam@17: If you give the -n or --nondeterministic option, vamp-plugin-tester cannam@17: will downgrade this error to a note. cannam@17: cannam@17: ** WARNING: Consecutive runs with different starting timestamps produce the same result cannam@17: cannam@17: The plugin was run twice on the same audio data, but with different cannam@17: input timestamps, and it returned the same results each time. While cannam@17: this is often unproblematic, it can indicate that a plugin failed to cannam@17: take the input timestamp into account when calculating its output cannam@17: timestamps (if any). cannam@17: cannam@17: If you give the -n or --nondeterministic option, vamp-plugin-tester cannam@17: will downgrade this warning to a note. cannam@17: cannam@17: ** ERROR: Explicitly setting current program to its supposed current value changes the results cannam@17: cannam@17: The plugin was constructed and run twice on the same data, once cannam@17: without changing its "program" setting, and again having set the cannam@17: program to the vaule returned by getCurrentProgram() (i.e. the same cannam@17: program that was supposed to be in effect already). It returned cannam@17: different results for the two runs, suggesting that some internal cannam@17: data was changed in selectProgram in a way that differed from its cannam@17: default. cannam@17: cannam@17: If you give the -n or --nondeterministic option, vamp-plugin-tester cannam@17: will downgrade this error to a note. cannam@17: cannam@17: ** ERROR: Explicitly setting parameters to their supposed default values changes the results cannam@17: cannam@17: The plugin was constructed and run twice on the same data, once cannam@17: without changing any of its parameters, and again having set the cannam@17: parameters to their specified default values. It returned different cannam@17: results for the two runs, suggesting that some internal data was cannam@17: changed when a parameter was set to its default, in a way that cannam@17: differed from the plugin's initially constructed state. cannam@17: cannam@17: If you give the -n or --nondeterministic option, vamp-plugin-tester cannam@17: will downgrade this error to a note. cannam@17: Chris@62: ** ERROR: Call to reset after setting parameters, but before processing, changes the results (parameter values not retained through reset?) Chris@62: Chris@62: The plugin was constructed and run twice on the same data. The first Chris@62: time, its parameters were set to some arbitrary values and it was Chris@62: initialised and run. The second time, its parameters were set to the Chris@62: same values and it was initialised, then reset(), then run. The two Chris@62: runs returned different results, suggesting that perhaps some Chris@62: parameter value was being modified within the reset() function. (This Chris@62: function should reset internal state within the plugin, but not Chris@62: parameter configuration.) Chris@62: Chris@62: If you give the -n or --nondeterministic option, vamp-plugin-tester Chris@62: will downgrade this error to a note. Chris@62: cannam@17: ** WARNING: Constructor takes some time to run: work should be deferred to initialise? cannam@17: cannam@17: The plugin took a long time to construct. You should ensure that the cannam@17: constructor for the plugin runs as quickly as possible, because it cannam@17: may be called by a host that is only scanning the properties of all cannam@17: available plugins on startup. Any serious initialisation work should cannam@17: be done in the initialise() function rather than the constructor. cannam@17: cannam@17: cannam@18: Authors cannam@18: ======= cannam@18: cannam@18: This program was written at the Centre for Digital Music at Queen Chris@58: Mary, University of London, by Chris Cannam. Copyright (c) 2009-2015 QMUL. cannam@18: