DHOxSS - Using Computers to Analyse Recordings

h1. Audio Features and Vamp Plugins

h3. General outline

To be propagated to / synchronised with https://www.doc.gold.ac.uk/~mas01cr/teaching/dhoxss15/

# Introductory notes and slides on acoustics and audio (CR, 25 min)

# Sonic Visualiser - hands on with waveform and spectrograms (CC, 25 min)

# Introductory notes and slides on audio features (CC, 5 min)

# Sonic Visualiser - hands on with Vamp plugins (CC, 20 min)

# Python/IPython intro (CR, 35 min)

# Break (20 min)

# Feature Extraction using Vamp Plugins in Python (IPython Notebook) (CC, 40 min)

# Audio Indexing and Search in Python (IPython Notebook) (CR, 40 min)

h3. Breakdown of CC sections

h5. Sonic Visualiser - hands on with waveform and spectrograms

* Waveform

## Start Sonic Visualiser and open "A Friendly Warning"

## Show dragging through the file using Navigate tool, and also using the overview at bottom

## Play from the start, just to get an idea what it sounds like

## Return to the start and zoom in (using the zoom wheel, but noting that the mouse wheel also works)

## Notice the different shapes in waveform resulting from different types of synthetic percussive sound (low-frequency kick drum / higher frequency cymbal-type sounds) - refer back to Christophe's notes about correspondence between e.g. signal voltage and speaker cone deflection

## Continue until the vocal starts, and observe that we can see very little that relates to e.g. sung pitch, although if we zoom in we can quite clearly see sibilance (these frequencies around 10kHz are pretty much the sweet spot for visibility in a 44.1kHz waveform)

* Spectrogram

## New session, open piano-scale.wav and play it

## Some information can sort-of be perceived and measured from the waveform here: we can see when the notes start, and can get simple fundamental frequency estimate - zoom in to the first note, switch to Select mode, drag out one cycle - it's about 170 samples, so 44100/170 = 259 Hz - the note is a middle C so true value should be nearer to 261, but this is a fair approximation. (But this is a very simple example!)

## Now open a plain spectrogram - Pane -> Add Spectrogram (or G key). Observe full range on frequency scale; x axis is time, this is a simple time-frequency breakdown.

## Notice that, for each note, we can see the fundamental frequency most strongly and then the harmonic sequence. The harmonics are spaced more widely for higher notes because they are multiples of the fundamental frequency, which is larger. The noise floor is visible because we're using a dB scale, can switch to Linear to isolate only the strong frequencies. There isn't really enough detail to measure much here. (NB the default colour scheme is unhelpful to colour blind users, so it might be worth changing to Sunset scheme.)

## Close that pane and open a "melodic-range spectrogram" - Pane -> Add Melodic Range Spectrogram (or M key). Observe the much more limited frequency range and the fact that this spectrogram uses both Linear colour and Sunset scheme by default.

## Although the higher harmonics quickly disappear off the top of the scale, we can clearly see that the spacing between harmonics is now the same for each note, and that the semitones are equally-spaced (note spacing corresponding to the major scale intervals). This is because the vertical scale is now logarithmic in frequency, which makes it (fudging the issue a little) linear in pitch. Correspondingly there is now a little representation of a piano keyboard shown at left, with middle C highlighted. (This assumes A=440Hz, what if that's not true? We'll come back to that in a mo)

## Select the Measure tool and show that we can get a frequency readout with harmonic markers. Return to the Navigate tool and contrast with the readout that is displayed as you move the pointer over the pane.

## Close that pane and open the "peak-frequency spectrogram" - Pane -> Add Peak Frequency Spectrogram (or K key). Notice that here we can just wave the Navigate tool over a bin to get an estimate of the instantaneous frequency there.

## New session, open A Friendly Warning again and open both the plain spectrogram and the melodic-range one -- observe and contrast the various visible elements, in particular vertical lines in full frequency range for noisy percussion, relative invisibility of such broadband sounds in the melodic-range spectrogram, curved slides in vocal, difficulty of distinguishing harmonic traces from simultaneous notes etc.

## Go to File -> Replace Main Audio, open King Henry. Note among other things that we need to increase the gain on the melodic-range spectrogram, and that the vibrato is visible and things like vibrato rate could be approximately measured, that the long reverb makes the notes appear to overlap in places. Leave this file open.

h5. Introductory notes and slides on audio features

h5. Sonic Visualiser - hands on with Vamp plugins

h5. Feature Extraction using Vamp Plugins in Python

h3. Materials

* Audio features slides: "PDF":https://code.soundsoftware.ac.uk/projects/dhoxss15/repository/raw/vamp-sv.pdf, "PowerPoint":https://code.soundsoftware.ac.uk/projects/dhoxss15/repository/raw/vamp-sv.pptx

* "IPython Notebook for Vamp Plugins in Python":https://code.soundsoftware.ac.uk/projects/dhoxss15/repository/raw/Vamp.v3.ipynb