Audio Features and Vamp Plugins » History » Version 25

Chris Cannam, 2015-07-16 09:41 AM

1 1 Chris Cannam
h1. Audio Features and Vamp Plugins
2 1 Chris Cannam
3 1 Chris Cannam
h3. General outline
4 1 Chris Cannam
5 13 Chris Cannam
To be propagated to / synchronised with https://www.doc.gold.ac.uk/~mas01cr/teaching/dhoxss15/
6 1 Chris Cannam
7 13 Chris Cannam
# Introductory notes and slides on acoustics and audio (CR, 25 min)
8 13 Chris Cannam
# Sonic Visualiser - hands on with waveform and spectrograms (CC, 25 min)
9 13 Chris Cannam
# Introductory notes and slides on audio features (CC, 5 min)
10 13 Chris Cannam
# Sonic Visualiser - hands on with Vamp plugins (CC, 20 min)
11 13 Chris Cannam
# Python/IPython intro (CR, 35 min)
12 13 Chris Cannam
# Break (20 min)
13 13 Chris Cannam
# Feature Extraction using Vamp Plugins in Python (IPython Notebook) (CC, 40 min)
14 13 Chris Cannam
# Audio Indexing and Search in Python (IPython Notebook) (CR, 40 min)
15 3 Chris Cannam
16 3 Chris Cannam
h3. Breakdown of CC sections
17 3 Chris Cannam
18 4 Chris Cannam
h5. Sonic Visualiser - hands on with waveform and spectrograms
19 3 Chris Cannam
20 7 Chris Cannam
* Waveform
21 20 Chris Cannam
## Start Sonic Visualiser and open "A Friendly Warning" (very severe, synthetic 80s pop song by Act)
22 6 Chris Cannam
## Show dragging through the file using Navigate tool, and also using the overview at bottom
23 6 Chris Cannam
## Play from the start, just to get an idea what it sounds like
24 5 Chris Cannam
## Return to the start and zoom in (using the zoom wheel, but noting that the mouse wheel also works)
25 6 Chris Cannam
## Notice the different shapes in waveform resulting from different types of synthetic percussive sound (low-frequency kick drum / higher frequency cymbal-type sounds) - refer back to Christophe's notes about correspondence between e.g. signal voltage and speaker cone deflection
26 1 Chris Cannam
## Continue until the vocal starts, and observe that we can see very little that relates to e.g. sung pitch, although if we zoom in we can quite clearly see sibilance (these frequencies around 10kHz are pretty much the sweet spot for visibility in a 44.1kHz waveform)
27 6 Chris Cannam
28 8 Chris Cannam
* Spectrogram
29 1 Chris Cannam
## New session, open piano-scale.wav and play it
30 9 Chris Cannam
## Some information can sort-of be perceived and measured from the waveform here: we can see when the notes start, and can get simple fundamental frequency estimate - zoom in to the first note, switch to Select mode, drag out one cycle - it's about 170 samples, so 44100/170 = 259 Hz - the note is a middle C so true value should be nearer to 261, but this is a fair approximation. (But this is a very simple example!)
31 9 Chris Cannam
## Now open a plain spectrogram - Pane -> Add Spectrogram (or G key). Observe full range on frequency scale; x axis is time, this is a simple time-frequency breakdown.
32 8 Chris Cannam
## Notice that, for each note, we can see the fundamental frequency most strongly and then the harmonic sequence. The harmonics are spaced more widely for higher notes because they are multiples of the fundamental frequency, which is larger. The noise floor is visible because we're using a dB scale, can switch to Linear to isolate only the strong frequencies. There isn't really enough detail to measure much here. (NB the default colour scheme is unhelpful to colour blind users, so it might be worth changing to Sunset scheme.)
33 9 Chris Cannam
## Close that pane and open a "melodic-range spectrogram" - Pane -> Add Melodic Range Spectrogram (or M key). Observe the much more limited frequency range and the fact that this spectrogram uses both Linear colour and Sunset scheme by default.
34 17 Chris Cannam
## Although the higher harmonics quickly disappear off the top of the scale, we can clearly see that the spacing between harmonics is now the same for each note, but the harmonics for a given note get closer together as they go up; and that the semitones are equally-spaced (note spacing corresponding to the major scale intervals). This is because the vertical scale is now logarithmic in frequency, which makes it (fudging the issue a little) linear in pitch. Correspondingly there is now a little representation of a piano keyboard shown at left, with middle C highlighted.
35 16 Chris Cannam
## The above assumes 12tET with A=440Hz, we can change at least the latter part of that in the Preferences and the scale will move immediately when we do so (demonstrate but be sure to restore the default)
36 10 Chris Cannam
## Select the Measure tool and show that we can get a frequency readout with harmonic markers. Return to the Navigate tool and contrast with the readout that is displayed as you move the pointer over the pane.
37 10 Chris Cannam
## Close that pane and open the "peak-frequency spectrogram" - Pane -> Add Peak Frequency Spectrogram (or K key). Notice that here we can just wave the Navigate tool over a bin to get an estimate of the instantaneous frequency there.
38 21 Chris Cannam
## New session, open A Friendly Warning again and open both the plain spectrogram and the melodic-range one -- observe and contrast the various visible elements, in particular vertical lines in full frequency range for noisy percussion, relative invisibility of such broadband sounds in the melodic-range spectrogram, glides in vocal, difficulty of distinguishing harmonic traces from simultaneous notes etc.
39 24 Chris Cannam
## Go to File -> Replace Main Audio, open King Henry. Note among other things that we need to increase the gain on the melodic-range spectrogram, and that the vibrato is visible and things like vibrato rate could be approximately measured, that the long reverb makes the notes appear to overlap in places.
40 5 Chris Cannam
41 1 Chris Cannam
h5. Introductory notes and slides on audio features
42 1 Chris Cannam
43 1 Chris Cannam
h5. Sonic Visualiser - hands on with Vamp plugins
44 16 Chris Cannam
45 25 Chris Cannam
* Quick survey of nominally high-ish level feature extractors
46 25 Chris Cannam
## Start a new session with "A Friendly Warning".
47 25 Chris Cannam
## Feature extraction plugins are found under the Transform menu, so called because it contains things that turn your audio into something else, including both feature extractors and audio effects. You can run a transform and show the output in the same pane as the audio, or in a new one.
48 25 Chris Cannam
## With the audio pane selected, run Transform -> Analysis by Category -> Time -> Tempo -> Bar and Beat Tracker: Beats. 
49 25 Chris Cannam
50 1 Chris Cannam
* Pitch
51 25 Chris Cannam
## Close session, return to "King Henry". Open a melodic-range spectrogram and make sure its pane is current.
52 1 Chris Cannam
## Now run the transform pYIN - Smoothed Pitch Track. A pitch track should appear in a bright colour. Switch its plot type to Discrete Curves and make sure its scale is set to Auto-Align, which means that if it has Hz units, it will be aligned to the same vertical scale as the spectrogram behind it.
53 18 Chris Cannam
## We can of course check this extracted pitch-track visually, but we can also inspect individual values (by mouseover), inspect in bulk (Layer -> Edit Layer Data) including tracking through the data during playback, and export to a file (File -> Export Annotation Layer). Demonstrate this latter. (Note that the correct layer must be selected for any of these to work!)
54 18 Chris Cannam
## This layer can also be synthesised and played back -- switch on the Play button on layer parameters and try it.
55 18 Chris Cannam
## The same plugin can produce note segmentations (for monophonic audio of this type), so run it again requesting the Notes output. Each note is recorded as having a pitch equal to the median of the underlying pitch track's pitches for the time it spans. This is unlikely to sound so nice when played back, because of both segmentation flaws and difficulties (e.g. glides) and interesting properties of pitch perception (e.g. with vibrato). If this kind of use is of interest to you, consider our other program "Tony":/projects/tony.
56 18 Chris Cannam
57 22 Chris Cannam
* Chroma reduction
58 23 Chris Cannam
## Using the NNLS Chroma plugin - open an empty pane and run this transform with the default parameters. This is a single-octave reduction of frequency content (can explain at arbitrary length).
59 23 Chris Cannam
## Run the same transform again in a second pane, but this time with different parameters: local tuning, L2 norm, spectral shape = 0.9. (Local tuning shouldn't make a difference for this file, but may be relevant for singing in parts etc)
60 3 Chris Cannam
61 3 Chris Cannam
h5. Feature Extraction using Vamp Plugins in Python
62 3 Chris Cannam
63 3 Chris Cannam
h3. Materials
64 3 Chris Cannam
65 3 Chris Cannam
* Audio features slides: "PDF":https://code.soundsoftware.ac.uk/projects/dhoxss15/repository/raw/vamp-sv.pdf, "PowerPoint":https://code.soundsoftware.ac.uk/projects/dhoxss15/repository/raw/vamp-sv.pptx
66 3 Chris Cannam
67 3 Chris Cannam
* "IPython Notebook for Vamp Plugins in Python":https://code.soundsoftware.ac.uk/projects/dhoxss15/repository/raw/Vamp.v3.ipynb