Wiki » History » Version 18

Gyorgy Fazekas, 2015-08-12 10:34 PM

1 1 Mi Tian
h1. Wiki
2 1 Mi Tian
3 1 Mi Tian
h2. Vamp environment for feature extraction
4 1 Mi Tian
5 4 Mi Tian
_Vamp_ is an audio processing plugin system for plugins that extract descriptive information from audio data — typically referred to as audio analysis plugins or audio feature extraction plugins. The _vamp plugins_ are distributed in shared library files with extension _.dll_, _.so_, or _.dylib_ depending on the platform.  
6 1 Mi Tian
7 4 Mi Tian
Several common audio file formats are supported, including MP3, Ogg, and a number of PCM formats such as WAV and AIFF. AAC is supported on OS/X only, and only if not DRM protected. WMA is not supported.
8 1 Mi Tian
9 10 Gyorgy Fazekas
See http://vamp-plugins.org/ for more introduction about Vamp environment and the plugins.
10 1 Mi Tian
11 18 Gyorgy Fazekas
12 16 Gyorgy Fazekas
h2. Vampy wrapper plugin (for running plugins written in Python)
13 10 Gyorgy Fazekas
14 1 Mi Tian
Vamp plugins are typically written in C++. To use Vamp plugins written in Python you need to install the "Vampy wrapper plugin":http://www.vamp-plugins.org/vampy.html. 
15 18 Gyorgy Fazekas
Then, any python script following some simple conventions will be presented to a vamp host as if it was an ordinary Vamp plugin.
16 10 Gyorgy Fazekas
17 10 Gyorgy Fazekas
Download Vampy: http://www.vamp-plugins.org/vampy.html
18 10 Gyorgy Fazekas
Tutorial: http://isophonics.net/content/getting-started-vampy
19 10 Gyorgy Fazekas
20 10 Gyorgy Fazekas
Put the plugins in your Vamp plugin folder or any other designated directory. You may set an environment variable to tell
21 17 Mi Tian
your system where to look for Vamp plugins. On Linux/Unix or OS/X: $ export VAMPY_PATH="/your/vamp/plugin/directory"
22 10 Gyorgy Fazekas
23 10 Gyorgy Fazekas
Optionally, you may need to tell your system where to find Python for Vampy to work correctly:
24 1 Mi Tian
To do this, set the VAMPY_PYLIB environment variable to the location of the python shared library (.dylib / .so).
25 10 Gyorgy Fazekas
26 11 Gyorgy Fazekas
For more information see "this readme file in the Vampy package":https://code.soundsoftware.ac.uk/projects/vampy/repository/entry/README
27 10 Gyorgy Fazekas
28 6 Mi Tian
h2. Host applications
29 1 Mi Tian
30 1 Mi Tian
A vamp plugin cannot be used on its own, but only with a conforming _host_ application which loads the plugin from its shared library and calls functions within the plugin code to configure it, supply it with data, and run it.
31 1 Mi Tian
32 10 Gyorgy Fazekas
* "Sonic Visualiser":http://www.sonicvisualiser.org/ -- An interactive audio analysis and visualisation tool using Vamp plugins
33 6 Mi Tian
34 10 Gyorgy Fazekas
* "Sonic Annotator":https://code.soundsoftware.ac.uk/projects/sonic-annotator -- A command line tool facilitating facilitating feature extraction, parameter configuration and output manipulation for a large bunch of audio files.
35 6 Mi Tian
36 1 Mi Tian
* "Audacity":http://audacityteam.org/ -- An audio editor which enables recording, labelling and editing the audio.
37 13 Gyorgy Fazekas
38 15 Gyorgy Fazekas
__Note for developers:__ If you are developing new plugins in Python it is customary to use a command line host such as Sonic Annotator or the "Vamp Simple Host":http://vamp-plugins.org/develop.html available in the Vamp Plugin SDK. This will allow you to see error messages printed to the terminal, which would otherwise be invisible. Vampy plugins can be used in any GUI host like Sonic Visualiser, but errors reported by plugin scripts will be more difficult to see.
39 6 Mi Tian
40 6 Mi Tian
41 1 Mi Tian
h2. Plugin installation
42 4 Mi Tian
43 10 Gyorgy Fazekas
The vamp plugins need to be put in a directory where the host can locate, i.e., the _VAMP_PATH_. See http://vamp-plugins.org/download.html#install for installation instructions. 
44 1 Mi Tian
45 1 Mi Tian
Alternatively, you can set your own VAMP_PATH environment variable to list the locations a host should look in for Vamp plugins. For example, the following command will set @/homes/mitian/hg/onset-fusion@ as an alternative VAMP_PATH. Plugins placed in this directory will also be locatable for the hosts besides those under the system specified vamp directories.
46 10 Gyorgy Fazekas
47 4 Mi Tian
<pre>
48 5 Mi Tian
mitian$ export VAMP_PATH=$VAMP_PATH:/homes/mitian/hg/onset-fusion
49 4 Mi Tian
</pre>
50 4 Mi Tian
51 4 Mi Tian
Once the plugins are installed in the Vamp path, they will show themselves under the _Transform_ tab in Sonic Visualiser, or the _Analyze_ tab in Audacity, or in the terminal when you list them using Sonic Annotator (@ sonic-annotator -l@). 
52 4 Mi Tian
53 1 Mi Tian
h2. Parameters and outputs of this plugin set
54 4 Mi Tian
55 4 Mi Tian
h3. Parameters
56 4 Mi Tian
<pre>
57 5 Mi Tian
    **Adjustable parameters**
58 9 Gyorgy Fazekas
	Fusion Type                         Fusion type [0: early fusion, 1: linear fusion, 2: decision fusion]
59 9 Gyorgy Fazekas
	Detection Sensitivity               Sensitivity level for peak detection in the onset likelihood function.
60 9 Gyorgy Fazekas
                                            The higher the sensitivity, the more onsets will (rightly or wrongly) be detected.
61 9 Gyorgy Fazekas
	Delta Threshold                     Constant offset used for adaptive theresholding using the median of the detection function.
62 9 Gyorgy Fazekas
	Backtracking Threshold              Backtracking threshold used determine the stopping condition for backtracking the detected onset location to an earlier 'perceived' location.
63 9 Gyorgy Fazekas
	Tolerance window                    Tolerance window to combine events from different detection functions for decision fusion.
64 9 Gyorgy Fazekas
	Linear combination coefficient      Combination coefficient for linear fusion.
65 9 Gyorgy Fazekas
	Median filter window                Median filter window size.
66 9 Gyorgy Fazekas
	Cutoff frequency                    Cutoff frequency for the low pass filter.
67 8 Mi Tian
	
68 9 Gyorgy Fazekas
	**bool parameters**                 [0: off, 1: on]
69 9 Gyorgy Fazekas
	Polynomial fitting                  Use polynomial fitting to evaluate detection function peaks.
70 9 Gyorgy Fazekas
	Median filtering                    Use median filter based adaptive thresholding.
71 4 Mi Tian
	Low-pass filtering                  Use low pass filtering to smooth onset detection functions.
72 4 Mi Tian
	Adaptive whitening                  Use adaptive whitening. It evens out the temporal and frequency variation in the signal.
73 4 Mi Tian
</pre>
74 4 Mi Tian
75 1 Mi Tian
h3. Outputs
76 6 Mi Tian
77 6 Mi Tian
78 6 Mi Tian
# *Note Onsets* – The detected note onset times, returned as a single feature with timestamp.
79 1 Mi Tian
# *Smoothed Detection Function* – The smoothed onset detection function.
80 1 Mi Tian
# *Note Onsets* – The detected note onset times, returned as a single feature with timestamp.
81 5 Mi Tian
82 1 Mi Tian
83 1 Mi Tian
NOTE: The last two outputs are not available in fusion based onset detectors. Please check individual plugins for specific output types.
84 1 Mi Tian
 
85 1 Mi Tian
h2. Example usages of the onset detectors
86 1 Mi Tian
87 1 Mi Tian
To use these onset detector plugins in Sonic Visualiser and Audacity is quite straightforward. Load your audio file first, then select a feature extractor/plugin from the transform list. You will also find a set of adjustable parameters in the pop-out box associated with each transform. The extracted features, whether it is onset locations or detection functions, will be visualised in the pane on top of the waveform, synchronised with time.
88 5 Mi Tian
 
89 5 Mi Tian
For comprehensive user manuals, see
90 10 Gyorgy Fazekas
http://www.sonicvisualiser.org/doc/reference/2.1/en/index.html
91 10 Gyorgy Fazekas
http://wiki.audacityteam.org/wiki/Category:Tutorial
92 5 Mi Tian
93 5 Mi Tian
Rather than for visualisation purpose, Sonic Annotator lets you to extract and store audio features efficiently. Here we give a quick tutorial of how to detect onset for a given audio file and save the results to local folder using Sonic Annotator.
94 10 Gyorgy Fazekas
95 3 Mi Tian
<pre>
96 3 Mi Tian
mitian$ sonic-annotator -l | grep vampy-onsets
97 3 Mi Tian
vamp:vampy:vampy-onsets-bersf:vampy-onsets-bersf-onsets
98 3 Mi Tian
vamp:vampy:vampy-onsets-cdsf:vampy-onsets-cdsf-onsets
99 3 Mi Tian
vamp:vampy:vampy-onsets-ber:vampy-onsets-ber-df
100 3 Mi Tian
vamp:vampy:vampy-onsets-ber:vampy-onsets-ber-onsets
101 3 Mi Tian
vamp:vampy:vampy-onsets-ber:vampy-onsets-ber-sdf
102 3 Mi Tian
vamp:vampy:vampy-onsets-cd:vampy-onsets-cd-df
103 3 Mi Tian
vamp:vampy:vampy-onsets-cd:vampy-onsets-cd-onsets
104 3 Mi Tian
vamp:vampy:vampy-onsets-cd:vampy-onsets-cd-sdf
105 3 Mi Tian
vamp:vampy:vampy-onsets-hfc:vampy-onsets-hfc-df
106 3 Mi Tian
vamp:vampy:vampy-onsets-hfc:vampy-onsets-hfc-onsets
107 3 Mi Tian
vamp:vampy:vampy-onsets-hfc:vampy-onsets-hfc-sdf
108 3 Mi Tian
vamp:vampy:vampy-onsets-pd:vampy-onsets-pd-df
109 3 Mi Tian
vamp:vampy:vampy-onsets-pd:vampy-onsets-pd-onsets
110 3 Mi Tian
vamp:vampy:vampy-onsets-pd:vampy-onsets-pd-sdf
111 1 Mi Tian
vamp:vampy:vampy-onsets-sd:vampy-onsets-sd-df
112 3 Mi Tian
vamp:vampy:vampy-onsets-sd:vampy-onsets-sd-onsets
113 3 Mi Tian
vamp:vampy:vampy-onsets-sd:vampy-onsets-sd-sdf
114 3 Mi Tian
vamp:vampy:vampy-onsets-sf:vampy-onsets-sf-df
115 3 Mi Tian
vamp:vampy:vampy-onsets-sf:vampy-onsets-sf-onsets
116 3 Mi Tian
vamp:vampy:vampy-onsets-sf:vampy-onsets-sf-sdf
117 1 Mi Tian
mitian$
118 1 Mi Tian
mitian$ sonic-annotator -d vamp:vampy:vampy-onsets-cdsf:vampy-onsets-cdsf-onsets -w csv test.wav --csv-basedir /homes/mitian/hg/workspace/onsetdetection
119 3 Mi Tian
Extracting and writing features... Done
120 4 Mi Tian
121 3 Mi Tian
Vampy::~PyExtensionManager: Extension module cleaned.
122 3 Mi Tian
mitian$
123 1 Mi Tian
mitian$
124 3 Mi Tian
</pre>
125 3 Mi Tian
126 6 Mi Tian
Each line of the output csv file consists of the timestamp (and duration where applicable) for a single feature, the value of that feature and the feature's label (where applicable), separated by comma.
127 1 Mi Tian
128 3 Mi Tian
Sonic Anntator also allows for batch processing in one go. 
129 3 Mi Tian
<pre>
130 11 Gyorgy Fazekas
131 1 Mi Tian
mitian$ sonic-annotator -d vamp:vampy:vampy-onsets-cdsf:vampy-onsets-cdsf-onsets -w csv /homes/mitian/documents/audio/*.wav --csv-basedir /homes/mitian/hg/workspace/onsetdetection
132 1 Mi Tian
Extracting and writing features... Done
133 3 Mi Tian
134 3 Mi Tian
Vampy::~PyExtensionManager: Extension module cleaned.
135 11 Gyorgy Fazekas
136 11 Gyorgy Fazekas
137 3 Mi Tian
</pre>
138 3 Mi Tian
139 4 Mi Tian
h2. Play around with your vamp plugin feature extractors
140 1 Mi Tian
141 4 Mi Tian
We may want use parameter settings other the default. With Sonic Visualiser or Audacity, you can easily access the GUI to modify them. Whiles with Sonic Annotator, you would need to interact with 
142 3 Mi Tian
143 4 Mi Tian
Firstly, we can check the current parameter configurations in the skeleton transform description using Sonic Annotator's @-s@ command.
144 3 Mi Tian
145 3 Mi Tian
<pre>
146 11 Gyorgy Fazekas
147 3 Mi Tian
mitian$ sonic-annotator -s vamp:vampy:vampy-onsets-cdsf:vampy-onsets-cdsf-onsets
148 3 Mi Tian
@prefix xsd:      <http://www.w3.org/2001/XMLSchema> .
149 3 Mi Tian
  @prefix vamp:     <http://purl.org/ontology/vamp/> .
150 3 Mi Tian
  @prefix :         <#> .
151 3 Mi Tian
152 3 Mi Tian
 
153 3 Mi Tian
:transform_plugin a vamp:Plugin ;
154 3 Mi Tian
    vamp:identifier "vampy-onsets-cdsf" .
155 3 Mi Tian
156 3 Mi Tian
:transform_library a vamp:PluginLibrary ;
157 3 Mi Tian
    vamp:identifier "vampy" ;
158 3 Mi Tian
    vamp:available_plugin :transform_plugin .
159 3 Mi Tian
160 3 Mi Tian
:transform a vamp:Transform ;
161 3 Mi Tian
    vamp:plugin :transform_plugin ;
162 3 Mi Tian
    vamp:step_size "512"^^xsd:int ;
163 3 Mi Tian
    vamp:block_size "1024"^^xsd:int ;
164 3 Mi Tian
    vamp:parameter_binding [
165 3 Mi Tian
        vamp:parameter [ vamp:identifier "bt-threshold" ] ;
166 3 Mi Tian
        vamp:value "0.9"^^xsd:float ;
167 3 Mi Tian
    ] ;
168 3 Mi Tian
    vamp:parameter_binding [
169 3 Mi Tian
        vamp:parameter [ vamp:identifier "cut-off" ] ;
170 3 Mi Tian
        vamp:value "0.34"^^xsd:float ;
171 3 Mi Tian
    ] ;
172 3 Mi Tian
    vamp:parameter_binding [
173 3 Mi Tian
        vamp:parameter [ vamp:identifier "dthreshold" ] ;
174 3 Mi Tian
        vamp:value "0"^^xsd:float ;
175 3 Mi Tian
    ] ;
176 3 Mi Tian
    vamp:parameter_binding [
177 3 Mi Tian
        vamp:parameter [ vamp:identifier "filtfilt" ] ;
178 3 Mi Tian
        vamp:value "1"^^xsd:float ;
179 3 Mi Tian
    ] ;
180 3 Mi Tian
    vamp:parameter_binding [
181 3 Mi Tian
        vamp:parameter [ vamp:identifier "fusion-type" ] ;
182 3 Mi Tian
        vamp:value "2"^^xsd:float ;
183 3 Mi Tian
    ] ;
184 3 Mi Tian
    vamp:parameter_binding [
185 3 Mi Tian
        vamp:parameter [ vamp:identifier "lin-threshold" ] ;
186 3 Mi Tian
        vamp:value "0.5"^^xsd:float ;
187 3 Mi Tian
    ] ;
188 3 Mi Tian
    vamp:parameter_binding [
189 3 Mi Tian
        vamp:parameter [ vamp:identifier "med-threshold" ] ;
190 3 Mi Tian
        vamp:value "7"^^xsd:float ;
191 3 Mi Tian
    ] ;
192 3 Mi Tian
    vamp:parameter_binding [
193 3 Mi Tian
        vamp:parameter [ vamp:identifier "medfilt" ] ;
194 3 Mi Tian
        vamp:value "1"^^xsd:float ;
195 3 Mi Tian
    ] ;
196 3 Mi Tian
    vamp:parameter_binding [
197 3 Mi Tian
        vamp:parameter [ vamp:identifier "polyfit" ] ;
198 3 Mi Tian
        vamp:value "1"^^xsd:float ;
199 3 Mi Tian
    ] ;
200 3 Mi Tian
    vamp:parameter_binding [
201 3 Mi Tian
        vamp:parameter [ vamp:identifier "threshold" ] ;
202 3 Mi Tian
        vamp:value "50"^^xsd:float ;
203 3 Mi Tian
    ] ;
204 3 Mi Tian
    vamp:parameter_binding [
205 1 Mi Tian
        vamp:parameter [ vamp:identifier "tol-threshold" ] ;
206 1 Mi Tian
        vamp:value "0.03"^^xsd:float ;
207 1 Mi Tian
    ] ;
208 1 Mi Tian
    vamp:parameter_binding [
209 1 Mi Tian
        vamp:parameter [ vamp:identifier "whitening" ] ;
210 1 Mi Tian
        vamp:value "0"^^xsd:float ;
211 1 Mi Tian
    ] ;
212 1 Mi Tian
    vamp:output [ vamp:identifier "vampy-onsets-cdsf-onsets" ] .
213 1 Mi Tian
mitian$
214 1 Mi Tian
</pre>
215 1 Mi Tian
216 6 Mi Tian
Transforms are usually described in _RDF_, following the transform part of the [["Vamp plugin ontology":http://www.omras2.org/VampOntology]]. Now, how can be change the parameter settings in the transform specification file? 
217 1 Mi Tian
218 1 Mi Tian
The solution is to save this RDF file and modify any content in it as we wish, then to run this transform by specifying the saved file with the @-t@ option. Let's see an example.
219 1 Mi Tian
220 6 Mi Tian
<pre>
221 11 Gyorgy Fazekas
222 4 Mi Tian
mitian$ sonic-annotator -s vamp:vampy:vampy-onsets-cdsf:vampy-onsets-cdsf-onsets >param_config.n3
223 11 Gyorgy Fazekas
224 4 Mi Tian
</pre>
225 11 Gyorgy Fazekas
226 4 Mi Tian
Now the transform file has been saved to the .n3 (RDF) file. Say, we want to use a higher threshold and change it from 50 to 70. Open an text editor, simple modify the corresponding lines in the saved n3 file
227 11 Gyorgy Fazekas
228 4 Mi Tian
<pre>
229 4 Mi Tian
        vamp:parameter [ vamp:identifier "threshold" ] ;
230 4 Mi Tian
        vamp:value "50"^^xsd:float ;
231 4 Mi Tian
</pre>
232 1 Mi Tian
into 
233 4 Mi Tian
<pre>
234 4 Mi Tian
        vamp:parameter [ vamp:identifier "threshold" ] ;
235 1 Mi Tian
        vamp:value "70"^^xsd:float ;
236 1 Mi Tian
</pre>
237 1 Mi Tian
and save it.
238 1 Mi Tian
239 1 Mi Tian
Now we can run the same plugin using the modified parameter settings.
240 1 Mi Tian
<pre>
241 11 Gyorgy Fazekas
242 4 Mi Tian
mitian$ sonic-annotator -t param_config.n3 -w csv test.wav --csv-basedir /homes/mitian/hg/workspace/onsetdetection
243 4 Mi Tian
Extracting and writing features... Done
244 4 Mi Tian
245 4 Mi Tian
Vampy::~PyExtensionManager: Extension module cleaned.
246 11 Gyorgy Fazekas
247 4 Mi Tian
</pre>
248 4 Mi Tian
249 6 Mi Tian
This example only involves a couple of operations for Sonic Annotator. You may also want to type
250 6 Mi Tian
@sonic-annotator -h@
251 7 Mi Tian
for a helpbook. Also your working directory should be where Sonic Annotator is installed. 
252 6 Mi Tian
  
253 6 Mi Tian
h2. That's all guys! Happy hacking with Vamp plugins :)