SampleType » History » Version 70

Chris Cannam, 2014-02-10 06:19 PM

1 1 Chris Cannam
h1. Output Sample Type and Sample Rate
2 1 Chris Cannam
3 15 Chris Cannam
{{>toc}}
4 15 Chris Cannam
5 30 Chris Cannam
h2. Who should read this document
6 29 Chris Cannam
7 30 Chris Cannam
This is a detailed document about the "sample type" and "sample rate" properties of a Vamp plugin's output descriptor.
8 29 Chris Cannam
9 67 Chris Cannam
 * If you are new to the Vamp plugin API, look at the "Programmer's Guide":http://vamp-plugins.org/guide.pdf first. The section "Sample Types and Timestamps" starting on page 9 introduces this subject.
10 29 Chris Cannam
11 67 Chris Cannam
 * Many of the requirements for these properties are also described in the inline documentation for the "Plugin":/projects/vamp-plugin-sdk/repository/entry/vamp-sdk/Plugin.h and "PluginBase":/projects/vamp-plugin-sdk/repository/entry/vamp-sdk/PluginBase.h classes.
12 55 Chris Cannam
13 66 Chris Cannam
Unfortunately both of the above sources leave some ambiguities, which this document aims to clear up.
14 65 Chris Cannam
15 66 Chris Cannam
If you are writing a plugin, read the "Rules of Thumb" section (below) after the Programmer's Guide. You probably won't need to read the rest of this document. You should use the "Vamp Plugin Tester":/projects/vamp-plugin-tester to test your plugin.
16 29 Chris Cannam
17 66 Chris Cannam
If you are writing a host, you should probably read the whole of this as well as the Guide. You should also use the "Vamp Test Plugin":/projects/vamp-test-plugin to test your host's interpretation of the feature structures.
18 29 Chris Cannam
19 29 Chris Cannam
h2. Rules of Thumb for Plugin Developers
20 29 Chris Cannam
21 60 Chris Cannam
The quick summary section.
22 29 Chris Cannam
23 60 Chris Cannam
 * If your output returns things that are always regularly-spaced in time, and there is one such thing returned for every @process@ block, and the calculation is causal so that results are available immediately, and there is no latency added beyond the length of the processing block, then you probably want to use @OneSamplePerStep@ sample type and omit the feature timestamps.
24 1 Chris Cannam
25 60 Chris Cannam
 * If your output returns things that are regularly-spaced in time but the other limitations above are not true, use @FixedSampleRate@ sample type, set the output sample rate to the (perhaps fractional) number of returned features per second, and use a timestamp for each feature.
26 60 Chris Cannam
27 60 Chris Cannam
 * If your output returns anything else, use @VariableSampleRate@ sample type, set the output sample rate to zero unless you know better, and use a timestamp for each feature.
28 29 Chris Cannam
29 29 Chris Cannam
h2. Introduction
30 29 Chris Cannam
31 12 Chris Cannam
A Vamp plugin receives audio and produces a series of descriptive feature structures.
32 1 Chris Cannam
33 12 Chris Cannam
The audio input is provided as a series of fixed-length sample blocks, equally spaced in time, provided to successive calls to the plugin's @process@ function. The plugin may return any number of features from each @process@ call, and may also return any number of features from @getRemainingFeatures@ after all the audio has been received.
34 11 Chris Cannam
35 11 Chris Cannam
Features are each associated with a particular output of the plugin. The plugin declares that each output has certain properties, which constrain the sort of feature data the host can expect to see. (See diagram.)
36 10 Chris Cannam
37 7 Chris Cannam
!/attachments/download/980/feature-structures-20pc.png!
38 9 Chris Cannam
39 22 Chris Cannam
A feature may or may not have a timestamp (as well as, optionally, a duration). Whether a timestamp is needed -- and, if it is provided, what it means -- are determined by the @sampleType@ and @sampleRate@ properties of the output on which the feature is returned.
40 7 Chris Cannam
41 50 Chris Cannam
An output's @sampleType@ property may be either @OneSamplePerStep@, @FixedSampleRate@, or @VariableSampleRate@. Here's what they mean, in order of complexity.
42 1 Chris Cannam
43 27 Chris Cannam
h2. OneSamplePerStep
44 1 Chris Cannam
45 23 Chris Cannam
This is the simplest option.
46 1 Chris Cannam
47 23 Chris Cannam
If an output is declared as having a @sampleType@ of @OneSamplePerStep@, then any features returned from a @process@ call are assumed to match up with the audio block provided to that @process@ call.
48 23 Chris Cannam
49 14 Chris Cannam
The @sampleRate@ and @hasDuration@ output properties are ignored for outputs of this type.
50 1 Chris Cannam
51 62 Chris Cannam
h3. Timestamps and durations
52 62 Chris Cannam
53 14 Chris Cannam
For any features returned through an output declared with @OneSamplePerStep@ type,
54 14 Chris Cannam
55 51 Chris Cannam
 * The plugin _should not_ set timestamps on the features and _should_ set their @hasTimestamp@ property to @false@;
56 51 Chris Cannam
 * The plugin _should not_ set durations on the features and _should_ set their @hasDuration@ property to @false@;
57 51 Chris Cannam
 * The host _must_ ignore any timestamps or durations that the plugin may set on the features;
58 51 Chris Cannam
 * The host _must_ treat all features returned from a given @process@ call as if they had the same timestamp as it passed to that @process@ call;
59 51 Chris Cannam
 * The host _must_ treat all features returned from @getRemainingFeatures@ as if they were immediately following the final @process@ block (i.e. with the same time as the next equally-spaced @process@ block would have had if the input had not ended);
60 63 Chris Cannam
 * The host _must_ treat all features as spanning a duration equal to the spacing between process blocks.
61 1 Chris Cannam
62 47 Chris Cannam
h3. Handling non-compliant plugins
63 47 Chris Cannam
64 52 Chris Cannam
There is no required behaviour from plugins for features on @OneSamplePerStep@ outputs. The host always ignores any timing information in these features whether present or not.
65 52 Chris Cannam
66 47 Chris Cannam
67 27 Chris Cannam
h3. Examples
68 14 Chris Cannam
69 14 Chris Cannam
@OneSamplePerStep@ is most often used for simple measurements and visualisations, in which some internal calculation is updated on each process call and a new result returned. For example: envelope trackers; power calculations; spectrograms. These outputs are typically visualised using line graphs or colour matrix plots.
70 14 Chris Cannam
71 1 Chris Cannam
@OneSamplePerStep@ is often used for intermediate results calculated during processing of a more sophisticated feature. For example, a beat tracker might have an auxiliary output with @OneSamplePerStep@ type returning its internal onset detection function value.
72 14 Chris Cannam
73 27 Chris Cannam
h2. VariableSampleRate
74 20 Chris Cannam
75 21 Chris Cannam
If the @OneSamplePerStep@ output type essentially means that the plugin leaves all time calculations up to the host, @VariableSampleRate@ is the opposite.
76 22 Chris Cannam
77 1 Chris Cannam
If an output is declared as having a @SampleType@ of @VariableSampleRate@, the features returned through it will have timestamps set by the plugin, and they won't necessarily have any relationship to the process block timestamps provided by the host.
78 1 Chris Cannam
79 27 Chris Cannam
h3. Timestamps
80 25 Chris Cannam
81 1 Chris Cannam
For any features returned through an output declared with @VariableSampleRate@ type,
82 24 Chris Cannam
83 24 Chris Cannam
 * The plugin _must_ set timestamps on these features and _must_ set their @hasTimestamp@ property to @true@;
84 28 Chris Cannam
 * The host _must_ obtain the features' start times from their timestamps rather than calculating them itself.
85 25 Chris Cannam
86 68 Chris Cannam
Note that features do not have to go forward in time. A feature's timestamp is permitted to be earlier than its predecessor.
87 68 Chris Cannam
88 27 Chris Cannam
h3. Durations
89 25 Chris Cannam
90 1 Chris Cannam
Features returned through @VariableSampleRate@ outputs may optionally have durations.
91 25 Chris Cannam
92 26 Chris Cannam
If the output's @hasDuration@ property is @true@, then
93 26 Chris Cannam
94 45 Chris Cannam
 * The plugin _must_ set the @duration@ property of features returned through this output and _must_ set their the @hasDuration@ property to @true@;
95 43 Chris Cannam
 * The host _must_ use the feature's @duration@ property as the feature duration.
96 1 Chris Cannam
97 28 Chris Cannam
If the output's @hasDuration@ property is @false@, then
98 43 Chris Cannam
99 46 Chris Cannam
 * The plugin _should not_ set the @duration@ property of that output's features;
100 46 Chris Cannam
 * The host _must_ ignore the @hasDuration@ and @duration@ properties of the output's features, and treat them as having "minimal" duration (see below).
101 46 Chris Cannam
102 28 Chris Cannam
h3. Sample rate and "minimal" duration
103 28 Chris Cannam
104 28 Chris Cannam
The plugin may optionally set a @sampleRate@ property for each @VariableSampleRate@ output. A @sampleRate@ of zero indicates no value.
105 28 Chris Cannam
106 28 Chris Cannam
If a @sampleRate@ is set,
107 28 Chris Cannam
108 28 Chris Cannam
 * The host _must_ use 1/@sampleRate@ seconds as the "minimal" duration assigned to features that have no duration supplied.
109 1 Chris Cannam
110 37 Chris Cannam
If no @sampleRate@ is set, i.e. if the @sampleRate@ property is zero,
111 1 Chris Cannam
112 1 Chris Cannam
 * The host _must_ use zero as the "minimal" duration used for features with no duration supplied.
113 1 Chris Cannam
114 47 Chris Cannam
h3. Handling non-compliant plugins
115 47 Chris Cannam
116 47 Chris Cannam
If a feature returned through a @VariableSampleRate@ output has @hasTimestamp@ set to @false@, the host should ignore that feature entirely rather than attempt to read (or guess) its timestamp.
117 47 Chris Cannam
118 47 Chris Cannam
If a feature's @hasDuration@ value does not match the output's @hasDuration@ value, the host should assign the feature "minimal" duration.
119 47 Chris Cannam
120 54 Chris Cannam
h3. Examples
121 54 Chris Cannam
122 54 Chris Cannam
@VariableSampleRate@ is most often used for features that primarily return timing information: beat or note onset times, notes, segmentation extents, event labels, etc.
123 54 Chris Cannam
124 1 Chris Cannam
h2. FixedSampleRate
125 1 Chris Cannam
126 49 Chris Cannam
This is a more complex case than @OneSamplePerStep@ or @VariableSampleRate@.
127 49 Chris Cannam
128 49 Chris Cannam
A @OneSamplePerStep@ output never uses individual feature timestamps; a @VariableSampleRate@ output always uses them; a @FixedSampleRate@ output sometimes does.
129 37 Chris Cannam
130 1 Chris Cannam
h3. Timestamps
131 1 Chris Cannam
132 39 Chris Cannam
An output with @FixedSampleRate@ type _must_ also have a non-zero @sampleRate@ property.
133 1 Chris Cannam
134 43 Chris Cannam
Conceptually the @sampleRate@ property defines a set of discrete sample times, spaced at @sampleRate@ points per second, starting at time zero. Every returned feature must start at one of these times, whether the time is calculated and assigned by the plugin (through the timestamp of the feature) or by the host (for features without timestamps).
135 1 Chris Cannam
136 39 Chris Cannam
For any features returned through an output declared with @FixedSampleRate@ type,
137 39 Chris Cannam
138 40 Chris Cannam
 * The plugin _may_ set timestamps on these features, and it _must_ set the feature's corresponding @hasTimestamp@ property to @true@ or @false@ depending on whether it has set a timestamp;
139 39 Chris Cannam
 * Where a feature has a timestamp, the host _must_ take that timestamp, round it to the nearest 1/@sampleRate@ seconds, and use that as the time of the feature;
140 42 Chris Cannam
 * Where a feature has no timestamp, the host _must_ take the time of the previous feature, add 1/@sampleRate@ seconds to it, and use that as the time of the new feature;
141 1 Chris Cannam
 * Where the very first feature has no timestamp, the host _must_ take its time as zero.
142 43 Chris Cannam
143 43 Chris Cannam
The "Programmer's Guide":http://vamp-plugins.org/guide.pdf leaves this behaviour incompletely defined, saying "The host may round the timestamp according to the sample rate given in the output descriptor's sampleRate field". Here we are declaring that the host _must_ round the timestamp in this way. Note that the "Vamp Test Plugin":/projects/vamp-test-plugin/wiki assumes this behaviour.
144 43 Chris Cannam
145 70 Chris Cannam
As with @VariableSampleRate@ outputs, the spec does not forbid features on @FixedSampleRate@ outputs from going backwards in time. A feature is technically allowed to have a timestamp earlier than its predecessor. As a pragmatic matter, though, plugins might be advised not to do this because it doesn't fit very well with users' expectations of what a fixed sample-rate feature looks like, so hosts might well have bugs in how they handle it. If you want to do this, reconsider or at least use a @VariableSampleRate@ output. (If you're writing a host, though, do consider this case.)
146 69 Chris Cannam
147 43 Chris Cannam
h3. Durations
148 1 Chris Cannam
149 1 Chris Cannam
Features returned through @FixedSampleRate@ outputs may optionally have durations.
150 43 Chris Cannam
151 43 Chris Cannam
If the output's @hasDuration@ property is @true@, then
152 43 Chris Cannam
153 47 Chris Cannam
 * The plugin _must_ set the @duration@ property of features returned through this output and _must_ set their the @hasDuration@ property to @true@;
154 47 Chris Cannam
 * The host _must_ take the value of the feature's @duration@ property, round it to the nearest 1/@sampleRate@ seconds, and use that as the feature duration.
155 1 Chris Cannam
156 43 Chris Cannam
If the output's @hasDuration@ property is @false@, then
157 1 Chris Cannam
158 1 Chris Cannam
 * The plugin _should not_ set the @duration@ property of that output's features;
159 47 Chris Cannam
 * The host _must_ ignore the @hasDuration@ and @duration@ properties of the output's features, giving the features a duration of zero.
160 1 Chris Cannam
161 1 Chris Cannam
h3. Handling non-compliant plugins
162 47 Chris Cannam
163 53 Chris Cannam
If a @FixedSampleRate@ output has a zero @sampleRate@, the host should ignore all features from it.
164 53 Chris Cannam
165 48 Chris Cannam
If a @FixedSampleRate@ feature's @hasDuration@ value does not match its output's @hasDuration@ value, the host should assign the feature a duration of zero.
166 47 Chris Cannam
167 48 Chris Cannam
h3. Examples
168 54 Chris Cannam
169 54 Chris Cannam
@FixedSampleRate@ is most often used in cases where @OneSamplePerStep@ won't work because the output can't return the features straight away. This may be because the plugin has some internal block size that differs from the processing block size, or because more than one process block's worth of history is needed to produce an output, or because the plugin works completely "offline" and produces no output at all until @getRemainingFeatures@ is called at the end of input.
170 54 Chris Cannam
171 54 Chris Cannam
The most likely uses, therefore, are similar to those for @OneSamplePerStep@: block-by-block measurements, columnar visualisations, etc.
172 54 Chris Cannam
173 54 Chris Cannam
However, @FixedSampleRate@ is also sometimes used in cases where feature timestamps are not equally spaced (i.e. there may be more than one feature per sample time, or there may be gaps) and where @VariableSampleRate@ might be a reasonable alternative, except that the plugin author chooses to reflect explicitly the limited precision of some internal block-resolution processing step.
174 54 Chris Cannam
175 64 Chris Cannam
So, for example, a beat tracker that was constrained to processing-block resolution might choose to return beat times on a @FixedSampleRate@ output, although @VariableSampleRate@ plus a @sampleRate@ property is probably more usual in the wild.