Mercurial > hg > vamp-plugin-tester

changeset 17:ea8865f488a0

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
* Add README, tweak some messages
author cannam
date Fri, 20 Mar 2009 14:45:30 +0000
parents 419d538874a8
children 809fcc3d7f4e
files Makefile README TestInputExtremes.cpp TestStaticData.cpp Tester.cpp
diffstat 5 files changed, 235 insertions(+), 20 deletions(-) [+]
line wrap: on
line diff
--- a/Makefile	Fri Mar 20 13:42:07 2009 +0000
+++ b/Makefile	Fri Mar 20 14:45:30 2009 +0000
@@ -1,6 +1,6 @@
 
 LDFLAGS 	+= -lvamp-hostsdk -ldl
-CXXFLAGS	+= -Wall -Wextra
+CXXFLAGS	+= -g -Wall -Wextra
 
 OBJECTS		:= vamp-plugin-tester.o Tester.o Test.o TestStaticData.o TestInputExtremes.o TestMultipleRuns.o TestOutputs.o TestDefaults.o TestInitialise.o
 
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/README	Fri Mar 20 14:45:30 2009 +0000
@@ -0,0 +1,215 @@
+
+Vamp Plugin Tester
+==================
+
+This program tests Vamp audio feature extraction plugins
+(http://vamp-plugins.org/) for certain common failure cases.
+
+To test a single plugin, run vamp-plugin-tester with the name of your
+plugin library and plugin identifier, separated by a colon.  For example,
+
+  $ vamp-plugin-tester vamp-example-plugins:amplitudefollower
+
+The plugin library must be installed in the Vamp plugin path (you
+cannot give the path to the library file).
+
+If you run vamp-plugin-tester with no arguments, it will test all
+plugins found in your Vamp plugin path.
+
+
+Options
+=======
+
+Supply the -v or --verbose option to tell vamp-plugin-tester to print
+out the whole content of its returned feature log for diagnostic
+purposes each time it prints an error or warning that arises from the
+contents of a returned feature.
+
+Supply the -n or --nondeterministic option to tell vamp-plugin-tester
+that your plugins are expected to return different results each time
+they are run.  The default behaviour is to treat different results on
+separate runs with the same input data as an error.
+
+
+Errors and Warnings
+===================
+
+Each test may cause one or several notes, warnings, or errors to be
+printed.  A note is printed when behaviour is observed that may be
+correct behaviour but that is not always anticipated by the plugin
+developer.  A warning is printed when behaviour is observed that is
+technically legal but that in practice most often happens by mistake.
+An error is printed when behaviour is observed that cannot be correct.
+
+vamp-plugin-tester prints all of its commentary to the standard
+output.  Standard error is usually used for diagnostic output printed
+by the plugins themselves.
+
+In addition to reporting on tests, vamp-plugin-tester runs some tests
+that are intended to provoke the plugin into unexpected behaviour such
+as memory errors.  If vamp-plugin-tester crashes during a test, this
+may be why.  Also, if you have access to a memory checker utility such
+as valgrind, you are advised to run vamp-plugin-tester under it so as
+to be informed of any memory errors that do not happen to cause
+crashes.
+
+
+Error and Warning Reference
+===========================
+
+ ** ERROR: Failed to load plugin
+
+ The plugin could not be loaded.  Remember that the plugin must be
+ installed in the Vamp plugin path.
+
+ ** ERROR: (plugin|parameter|output) identifier <x> contains invalid characters
+
+ An identifier contains characters other than the permitted set (ASCII
+ lower and upper case letters, digits, "-" and "_" only).
+
+ ** ERROR: <field> is empty
+
+ A mandatory field, such as the name of a parameter or output,
+ contains no text.
+
+ ** WARNING: <field> is empty
+
+ An optional field, such as the description of a parameter or output,
+ contains no text.
+
+ ** ERROR: Plugin parameter <x> maxValue <= minValue
+
+ The minimum and maximum values given for a parameter are equal or in
+ the wrong order.
+
+ ** ERROR: Plugin parameter <x> defaultValue out of range
+
+ The default value for a parameter is not within the range defined by
+ the minimum and maximum values for the parameter.
+
+ ** ERROR: Plugin parameter <x> is quantized, but quantize step is zero
+
+ The quantizeStep value in a parameter with isQuantized true is set to
+ zero.
+
+ ** WARNING: Plugin parameter <x> value range is not a multiple of quantize step
+
+ A parameter's stated maximum value is not one of the possible values
+ obtained by adding multiples of the quantize step on to the minimum
+ value.
+
+ ** WARNING: Plugin parameter <x> has (more|fewer) value names than quantize steps
+
+ A quantized parameter lists some value names for its quantize steps,
+ but not the right number.
+ 
+ ** WARNING: Plugin parameter <x> default value is not a multiple of quantize
+ step beyond minimum
+
+ The default value for a parameter is not a value that the user could
+ actually obtain, if only offered the quantized values to choose from.
+
+ ** ERROR: Data returned on nonexistent output
+
+ The output number key for a returned feature is outside the range of
+ outputs listed in the plugin's output descriptor list.
+
+ ** NOTE: No results returned for output <x>
+
+ The plugin returned no features on one of its outputs, when given a
+ simple test file.  This may be perfectly reasonable behaviour, but
+ you might like to know about it.
+
+ ** NOTE: Plugin returns features with timestamps on OneSamplePerStep output
+ ** NOTE: Plugin returns features with durations on OneSamplePerStep output
+
+ Hosts will usually ignore timestamps and durations attached to any
+ feature returned on a OneSamplePerStep output.
+
+ ** ERROR: Plugin returns features with no timestamps on VariableSampleRate output
+
+ Timestamps are mandatory on all features associated with a
+ VariableSampleRate output.
+
+ ** WARNING: Plugin returned one or more NaN/inf values
+
+ The plugin returned features containing floating-point not-a-number
+ or infinity values.  This warning may be associated with a test
+ involving feeding some unexpected type of data to the plugin.
+ 
+ ** ERROR: Consecutive runs with separate instances produce different results
+
+ The plugin was constructed and run twice against the same input data,
+ and returned different features each time.
+
+ If you give the -n or --nondeterministic option, vamp-plugin-tester
+ will downgrade this error to a note.
+
+ ** ERROR: Consecutive runs with the same instance (using reset) produce different results
+
+ The plugin was constructed, initialised, run against some input data,
+ reset with a call to its reset() function, and run again against the
+ same data; and it returned different features on each run.  This is
+ often a sign of some simple error such as forgetting to implement
+ reset().
+
+ If you give the -n or --nondeterministic option, vamp-plugin-tester
+ will downgrade this error to a note.
+
+ ** ERROR: Simultaneous runs with separate instances produce different results
+
+ Two instances of the plugin were constructed and run against the same
+ input data, giving each block of data to one plugin's process call
+ and then to the other's, "interleaving" the processing between the
+ two instances (but within a single application thread); and the two
+ instances returned different features.  This may indicate ill-advised
+ use of static data shared between plugin instances.
+
+ If you give the -n or --nondeterministic option, vamp-plugin-tester
+ will downgrade this error to a note.
+
+ ** WARNING: Consecutive runs with different starting timestamps produce the same result
+
+ The plugin was run twice on the same audio data, but with different
+ input timestamps, and it returned the same results each time.  While
+ this is often unproblematic, it can indicate that a plugin failed to
+ take the input timestamp into account when calculating its output
+ timestamps (if any).
+
+ If you give the -n or --nondeterministic option, vamp-plugin-tester
+ will downgrade this warning to a note.
+
+ ** ERROR: Explicitly setting current program to its supposed current value changes the results
+
+ The plugin was constructed and run twice on the same data, once
+ without changing its "program" setting, and again having set the
+ program to the vaule returned by getCurrentProgram() (i.e. the same
+ program that was supposed to be in effect already).  It returned
+ different results for the two runs, suggesting that some internal
+ data was changed in selectProgram in a way that differed from its
+ default.
+
+ If you give the -n or --nondeterministic option, vamp-plugin-tester
+ will downgrade this error to a note.
+
+ ** ERROR: Explicitly setting parameters to their supposed default values changes the results
+
+ The plugin was constructed and run twice on the same data, once
+ without changing any of its parameters, and again having set the
+ parameters to their specified default values.  It returned different
+ results for the two runs, suggesting that some internal data was
+ changed when a parameter was set to its default, in a way that
+ differed from the plugin's initially constructed state.
+
+ If you give the -n or --nondeterministic option, vamp-plugin-tester
+ will downgrade this error to a note.
+
+ ** WARNING: Constructor takes some time to run: work should be deferred to initialise?
+
+ The plugin took a long time to construct.  You should ensure that the
+ constructor for the plugin runs as quickly as possible, because it
+ may be called by a host that is only scanning the properties of all
+ available plugins on startup.  Any serious initialisation work should
+ be done in the initialise() function rather than the constructor.
+
+
--- a/TestInputExtremes.cpp	Fri Mar 20 13:42:07 2009 +0000
+++ b/TestInputExtremes.cpp	Fri Mar 20 14:45:30 2009 +0000
@@ -94,7 +94,7 @@
     if (allFeaturesValid(f)) {
         r.push_back(success());
     } else {
-        r.push_back(warning("plugin returned one or more NaN/inf values"));
+        r.push_back(warning("Plugin returned one or more NaN/inf values"));
         if (options & Verbose) dump(f);
     }
     return r;
@@ -111,7 +111,7 @@
     if (allFeaturesValid(fs)) {
         r.push_back(success());
     } else {
-        r.push_back(warning("plugin returned one or more NaN/inf values"));
+        r.push_back(warning("Plugin returned one or more NaN/inf values"));
     }
     return r;
 }
@@ -141,7 +141,7 @@
     if (allFeaturesValid(f)) {
         r.push_back(success());
     } else {
-        r.push_back(warning("plugin returned one or more NaN/inf values"));
+        r.push_back(warning("Plugin returned one or more NaN/inf values"));
         if (options & Verbose) dump(f);
     }
     return r;
@@ -173,7 +173,7 @@
     if (allFeaturesValid(f)) {
         r.push_back(success());
     } else {
-        r.push_back(warning("plugin returned one or more NaN/inf values"));
+        r.push_back(warning("Plugin returned one or more NaN/inf values"));
         if (options & Verbose) dump(f);
     }
     return r;
@@ -207,7 +207,7 @@
     if (allFeaturesValid(f)) {
         r.push_back(success());
     } else {
-        r.push_back(warning("plugin returned one or more NaN/inf values"));
+        r.push_back(warning("Plugin returned one or more NaN/inf values"));
         if (options & Verbose) dump(f);
     }
     return r;
@@ -241,7 +241,7 @@
     if (allFeaturesValid(f)) {
         r.push_back(success());
     } else {
-        r.push_back(warning("plugin returned one or more NaN/inf values"));
+        r.push_back(warning("Plugin returned one or more NaN/inf values"));
         if (options & Verbose) dump(f);
     }
     return r;
--- a/TestStaticData.cpp	Fri Mar 20 13:42:07 2009 +0000
+++ b/TestStaticData.cpp	Fri Mar 20 14:45:30 2009 +0000
@@ -62,16 +62,16 @@
     auto_ptr<Plugin> p(load(key));
     
     Results r;
-    r.push_back(testIdentifier(p->getIdentifier(), "plugin identifier"));
+    r.push_back(testIdentifier(p->getIdentifier(), "Plugin identifier"));
 
     Plugin::ParameterList params = p->getParameterDescriptors();
     for (int i = 0; i < (int)params.size(); ++i) {
-        r.push_back(testIdentifier(params[i].identifier, "parameter identifier"));
+        r.push_back(testIdentifier(params[i].identifier, "Parameter identifier"));
     }
 
     Plugin::OutputList outputs = p->getOutputDescriptors();
     for (int i = 0; i < (int)outputs.size(); ++i) {
-        r.push_back(testIdentifier(outputs[i].identifier, "output identifier"));
+        r.push_back(testIdentifier(outputs[i].identifier, "Output identifier"));
     }
 
     return r;
@@ -100,29 +100,29 @@
 
     Results r;
 
-    r.push_back(testMandatory(p->getName(), "plugin name"));
-    r.push_back(testRecommended(p->getDescription(), "plugin description"));
-    r.push_back(testRecommended(p->getMaker(), "plugin maker"));
-    r.push_back(testRecommended(p->getCopyright(), "plugin copyright"));
+    r.push_back(testMandatory(p->getName(), "Plugin name"));
+    r.push_back(testRecommended(p->getDescription(), "Plugin description"));
+    r.push_back(testRecommended(p->getMaker(), "Plugin maker"));
+    r.push_back(testRecommended(p->getCopyright(), "Plugin copyright"));
     
     Plugin::ParameterList params = p->getParameterDescriptors();
     for (int i = 0; i < (int)params.size(); ++i) {
         r.push_back(testMandatory
                     (params[i].name,
-                     "plugin parameter \"" + params[i].identifier + "\" name"));
+                     "Plugin parameter \"" + params[i].identifier + "\" name"));
         r.push_back(testRecommended
                     (params[i].description,
-                     "plugin parameter \"" + params[i].identifier + "\" description"));
+                     "Plugin parameter \"" + params[i].identifier + "\" description"));
     }
     
     Plugin::OutputList outputs = p->getOutputDescriptors();
     for (int i = 0; i < (int)outputs.size(); ++i) {
         r.push_back(testMandatory
                     (outputs[i].name,
-                     "plugin output \"" + outputs[i].identifier + "\" name"));
+                     "Plugin output \"" + outputs[i].identifier + "\" name"));
         r.push_back(testRecommended
                     (outputs[i].description,
-                     "plugin output \"" + outputs[i].identifier + "\" description"));
+                     "Plugin output \"" + outputs[i].identifier + "\" description"));
     }
 
     return r;
@@ -156,7 +156,7 @@
     Plugin::ParameterList params = p->getParameterDescriptors();
     for (int i = 0; i < (int)params.size(); ++i) {
         Plugin::ParameterDescriptor &pd(params[i]);
-        string pfx("plugin parameter \"" + pd.identifier + "\"");
+        string pfx("Plugin parameter \"" + pd.identifier + "\"");
         float min = pd.minValue;
         float max = pd.maxValue;
         float deft = pd.defaultValue;
--- a/Tester.cpp	Fri Mar 20 13:42:07 2009 +0000
+++ b/Tester.cpp	Fri Mar 20 14:45:30 2009 +0000
@@ -181,7 +181,7 @@
             }
         }
     } catch (Test::FailedToLoadPlugin) {
-        std::cout << "ERROR: Failed to load plugin (key = \"" << m_key
+        std::cout << " ** ERROR: Failed to load plugin (key = \"" << m_key
                   << "\")" << std::endl;
     }