Using Computers to Analyse Recordings 2017 » History » Version 9
Version 8 (Chris Cannam, 2017-06-26 03:13 PM) → Version 9/21 (Chris Cannam, 2017-06-26 03:14 PM)
h1. Using Computers to Analyse Recordings 2017
h3. General outline
# Introductory notes and slides on acoustics and audio (DW)
# Introductory notes and slides on audio features (CC)
# Sonic Visualiser - hands on with waveform and spectrograms (CC)
# Introductory notes and slides on Vamp plugins (CC)
# Sonic Visualiser - hands on with Vamp plugins (CC)
h3. Audio material
* "piano-scale.wav":https://code.soundsoftware.ac.uk/projects/dhoxss15/repository/raw/data/piano-scale.wav - our simplest example, used for basic waveform, spectrogram, & chroma introductions
* "A Friendly Warning":https://code.soundsoftware.ac.uk/projects/dhoxss15/repository/raw/data/A%20Friendly%20Warning.ogg - severe synthetic pop from the 80s, used to illustrate spectrograms a bit more and demonstrate straightforward timing-related features
* "Frog Galliard":https://code.soundsoftware.ac.uk/projects/dhoxss15/repository/raw/data/lutemusic426.mp3 (Dowland) - illustrating both timing and pitch/harmonic features - file will be used throughout the week, possibly (nb it's in G major)
* "King Henry":https://code.soundsoftware.ac.uk/projects/dhoxss15/repository/raw/data/Music/07%20-%20King%20Henry.flac - to show appearance of vibrato vibrato, talk about confusions between harmonics and fundamentals, and discuss aspects of pitch estimation, pitch perception, and temperament
* a live recording of sung audio? (bring a usb mic?) - when illustrating pitch (e.g. pYin) and chroma features
h3. What we hope to cover
# Audio waveforms, and what we can learn from them on their own;
# Spectrograms: what is a spectrogram; how the terms "spectral", "spectrum", and "spectrogram" are related, as well as "discrete Fourier transform", "short-time Fourier transform", etc; time/frequency tradeoff; fundamental frequency and harmonic series; linear and logarithmic frequency scales;
# Some common higher-level features derived from short-time Fourier analysis: onset detection and tempo estimation; pitch estimation and note segmentation;
# Chromagrams: what is a chromagram; how the terms "pitch chroma", "chromagram", and "chroma features" are related; relationship to Constant-Q spectrograms; what use chroma features are; limitations; tuning/parameter considerations (tuning frequency, number of bins per octave, lack of consideration for temperament, etc); what "chroma features" saved to a file might look like compared to how a chromagram looks on screen
h3. Breakdown
(Direct links to the audio files themselves are included here, but for copyright reasons they might disappear at any time after the workshop!)
h5. Introductory notes and slides on audio features
h5. Sonic Visualiser - hands on with waveform and spectrograms
* *Waveform*. To cover:
** Audio waveforms, and what we can learn from them on their own
## Start Sonic Visualiser and open "A Friendly Warning":https://code.soundsoftware.ac.uk/projects/dhoxss15/repository/raw/data/A%20Friendly%20Warning.ogg. (This is a severe, synthetic 80s pop song by an act called Act, which we're using as a first illustrative piece because so much about it is clear-cut and easily visible in waveform and spectrogram.)
## Click-and-drag through the file using Navigate tool, and also using the overview at the bottom of the window, to see the scope of the waveform view.
## Play from the start, just to get an idea what it sounds like.
## Return to the start and zoom in (using the zoom wheel, but noting that the mouse wheel also works).
## Notice the different shapes in waveform resulting from different types of synthetic percussive sound (low-frequency kick drum / higher frequency cymbal-type sounds). These can be related intuitively to the direct correspondence between signal voltage and speaker cone deflection.
## Continue until the vocal starts, and observe that we can see very little that can be directly related to the sung pitch, although if we zoom in we can quite clearly see sibilance (these frequencies around 10kHz are pretty much the sweet spot for visibility in a 44.1kHz waveform).
## Start a new session, open "piano-scale.wav":https://code.soundsoftware.ac.uk/projects/dhoxss15/repository/raw/data/piano-scale.wav, and play it.
## Some information can sort-of be perceived and measured from the waveform here: we can see when the notes start, and can get simple fundamental frequency estimate - zoom in to the first note, switch to Select mode, drag out one cycle - it's about 170 samples, so 44100/170 = 259 Hz - the note is a middle C so true value should be nearer to 261, but this is a fair approximation. (But this is a very simple example, and in particular it's one where the single note's fundamental frequency dominates the harmonic envelope so this simple waveform zero-crossing measurement can be carried out without octave errors.)
* *Spectrogram*. To cover:
** What is a spectrogram?
** How the terms "spectral", "spectrum", and "spectrogram" are related, as well as "discrete Fourier transform", "short-time Fourier transform", etc
** Time/frequency tradeoff
** Fundamental frequency and harmonic series
** Linear and logarithmic frequency scales
## With the piano-scale.wav file open, call up a plain spectrogram - Pane -> Add Spectrogram (or keyboard shortcut on the G key). This is the standard kind of "audio recorder/editor" spectrogram.
## Observe that the Y axis is frequency, with the full recorded frequency range; the X axis is time, as it is for most layers in Sonic Visualiser. The spectrogram is a simple time-frequency breakdown, the output of a series of short-time Fourier transforms, one for each horizontal step.
## Notice that, for each note, we can see the fundamental frequency most strongly and then the harmonics stacked above it. The harmonics are spaced more widely for higher notes because they are at multiples of the note's fundamental frequency, which is larger.
## The colour scale is a dB scale, which means that it boosts quieter content. For this reason we can see the noise floor (general background noise) even though it may not be audible. We can switch the colour scale to Linear to isolate only the strong frequencies. There isn't really enough detail to measure much here. (NB the default colour scheme is unhelpful to colour-blind users, so it might be worth changing to Sunset or Ice scheme.)
## Close that pane and open a "melodic-range spectrogram" - Pane -> Add Melodic Range Spectrogram (or M key). Observe the much more limited frequency range and the fact that this spectrogram uses both Linear colour and Sunset scheme by default.
## However, the biggest difference is that the default vertical (Y) scale for this view is logarithmic in frequency, i.e. with equal spacing between 1, 2, 4, 8, 16 etc rather than between 1, 2, 3, 4, 5 etc as in a linear scale. This makes it essentially linear in pitch, with "one octave" (a doubling in fundamental frequency) always being the same distance on the vertical scale. Correspondingly there is now a little representation of a piano keyboard shown at left, with middle C highlighted.
## The above assumes 12tET with A=440Hz; we can change at least the latter part of that in the Preferences and the scale will move immediately when we do so (try it but, at this point, be sure to restore the default).
## Select the Measure tool and show that we can get a frequency readout with harmonic markers. Return to the Navigate tool and contrast with the readout that is displayed as you move the pointer over the pane.
## Close that pane and open the "peak-frequency spectrogram" - Pane -> Add Peak Frequency Spectrogram (or K key). Notice that here we can just wave the Navigate tool over a bin to get an estimate of the instantaneous frequency there.
## New session, open "A Friendly Warning":https://code.soundsoftware.ac.uk/projects/dhoxss15/repository/raw/data/A%20Friendly%20Warning.ogg again and open both the plain spectrogram and the melodic-range one -- observe and contrast the various visible elements, in particular vertical lines in full frequency range for noisy percussion, relative invisibility of such broadband sounds in the melodic-range spectrogram, glides in vocal, difficulty of distinguishing harmonic traces from simultaneous notes etc.
## Go to File -> Replace Main Audio, open "King Henry":https://code.soundsoftware.ac.uk/projects/dhoxss15/repository/raw/data/Music/07%20-%20King%20Henry.flac. Note among other things that we need to increase the gain on the melodic-range spectrogram, and that the vibrato is visible and things like vibrato rate could be approximately measured, that the long reverb makes the notes appear to overlap in places.
h5. Introductory notes and slides on Vamp plugins
h5. Sonic Visualiser - hands on with Vamp plugins
* *Quick survey of common higher-level features* derived from short-time Fourier analysis. To cover:
** amplitude
** onset detection and tempo estimation
## Start a new session with "A Friendly Warning":https://code.soundsoftware.ac.uk/projects/dhoxss15/repository/raw/data/A%20Friendly%20Warning.ogg.
## Feature extraction plugins are found under the Transform menu, so called because it contains things that turn your audio into something else, including both feature extractors and audio effects. You can run a transform and show the output in the same pane as the audio, or in a new one.
## With the audio pane selected, run Transform -> Analysis by Category -> Time -> Tempo -> Bar and Beat Tracker: Beats. This produces a series of beat locations, each labelled with metrical beat number, and when you play the audio, the beats are played with a tap.
## Create a new pane (Pane -> Add New Pane or shortcut N), and run Transform -> Analysis by Category -> Low Level Features -> Amplitude Follower. This produces an amplitude curve.
## Open another new pane (Pane -> Add New Pane or shortcut N) and run Transform -> Analysis by Category -> Time -> Onsets -> Note Onset Detector: Onsets. Here we have individual note onset positions: it works OK for this sort of music. We can switch playback on and off for the individual feature tracks with the Play toggle on the parameter box at right.
## Some further transforms we can try: Chordino chord estimate; the notes corresponding to Chordino's chord estimate (so as to audition whether the chords sound any good); QM key estimator and key-strength plot.
* *Pitch*. To cover:
** pitch estimation and note segmentation
## Close session, return to "King Henry":https://code.soundsoftware.ac.uk/projects/dhoxss15/repository/raw/data/Music/07%20-%20King%20Henry.flac. Open a melodic-range spectrogram and make sure its pane is current.
## Now run the transform pYIN - Smoothed Pitch Track. A pitch track should appear in a bright colour. Switch its plot type to Discrete Curves and make sure its scale is set to Auto-Align, which means that if it has Hz units, it will be aligned to the same vertical scale as the spectrogram behind it.
## We can of course check this extracted pitch-track visually, but we can also inspect individual values (by mouseover), inspect in bulk (Layer -> Edit Layer Data) including tracking through the data during playback, and export to a file (File -> Export Annotation Layer). Demonstrate this latter. (Note that the correct layer must be selected for any of these to work!)
## This layer can also be synthesised and played back -- switch on the Play button on layer parameters and try it.
## The same plugin can produce note segmentations (for monophonic audio of this type), so run it again requesting the Notes output. Each note is recorded as having a pitch equal to the median of the underlying pitch track's pitches for the time it spans. This is unlikely to sound so nice when played back, because of both segmentation flaws and difficulties (e.g. glides) and interesting properties of pitch perception (e.g. with vibrato). If this kind of use is of interest to you, consider our other program "Tony":/projects/tony.
* Chroma reduction
## Using the NNLS Chroma plugin - open an empty pane and run this transform with the default parameters. This is a single-octave reduction of frequency content (can explain at arbitrary length).
## Run the same transform again in a second pane, but this time with different parameters: local tuning, L2 norm, spectral shape = 0.9.
h3. General outline
# Introductory notes and slides on acoustics and audio (DW)
# Introductory notes and slides on audio features (CC)
# Sonic Visualiser - hands on with waveform and spectrograms (CC)
# Introductory notes and slides on Vamp plugins (CC)
# Sonic Visualiser - hands on with Vamp plugins (CC)
h3. Audio material
* "piano-scale.wav":https://code.soundsoftware.ac.uk/projects/dhoxss15/repository/raw/data/piano-scale.wav - our simplest example, used for basic waveform, spectrogram, & chroma introductions
* "A Friendly Warning":https://code.soundsoftware.ac.uk/projects/dhoxss15/repository/raw/data/A%20Friendly%20Warning.ogg - severe synthetic pop from the 80s, used to illustrate spectrograms a bit more and demonstrate straightforward timing-related features
* "Frog Galliard":https://code.soundsoftware.ac.uk/projects/dhoxss15/repository/raw/data/lutemusic426.mp3 (Dowland) - illustrating both timing and pitch/harmonic features - file will be used throughout the week, possibly (nb it's in G major)
* "King Henry":https://code.soundsoftware.ac.uk/projects/dhoxss15/repository/raw/data/Music/07%20-%20King%20Henry.flac - to show appearance of vibrato vibrato, talk about confusions between harmonics and fundamentals, and discuss aspects of pitch estimation, pitch perception, and temperament
* a live recording of sung audio? (bring a usb mic?) - when illustrating pitch (e.g. pYin) and chroma features
h3. What we hope to cover
# Audio waveforms, and what we can learn from them on their own;
# Spectrograms: what is a spectrogram; how the terms "spectral", "spectrum", and "spectrogram" are related, as well as "discrete Fourier transform", "short-time Fourier transform", etc; time/frequency tradeoff; fundamental frequency and harmonic series; linear and logarithmic frequency scales;
# Some common higher-level features derived from short-time Fourier analysis: onset detection and tempo estimation; pitch estimation and note segmentation;
# Chromagrams: what is a chromagram; how the terms "pitch chroma", "chromagram", and "chroma features" are related; relationship to Constant-Q spectrograms; what use chroma features are; limitations; tuning/parameter considerations (tuning frequency, number of bins per octave, lack of consideration for temperament, etc); what "chroma features" saved to a file might look like compared to how a chromagram looks on screen
h3. Breakdown
(Direct links to the audio files themselves are included here, but for copyright reasons they might disappear at any time after the workshop!)
h5. Introductory notes and slides on audio features
h5. Sonic Visualiser - hands on with waveform and spectrograms
* *Waveform*. To cover:
** Audio waveforms, and what we can learn from them on their own
## Start Sonic Visualiser and open "A Friendly Warning":https://code.soundsoftware.ac.uk/projects/dhoxss15/repository/raw/data/A%20Friendly%20Warning.ogg. (This is a severe, synthetic 80s pop song by an act called Act, which we're using as a first illustrative piece because so much about it is clear-cut and easily visible in waveform and spectrogram.)
## Click-and-drag through the file using Navigate tool, and also using the overview at the bottom of the window, to see the scope of the waveform view.
## Play from the start, just to get an idea what it sounds like.
## Return to the start and zoom in (using the zoom wheel, but noting that the mouse wheel also works).
## Notice the different shapes in waveform resulting from different types of synthetic percussive sound (low-frequency kick drum / higher frequency cymbal-type sounds). These can be related intuitively to the direct correspondence between signal voltage and speaker cone deflection.
## Continue until the vocal starts, and observe that we can see very little that can be directly related to the sung pitch, although if we zoom in we can quite clearly see sibilance (these frequencies around 10kHz are pretty much the sweet spot for visibility in a 44.1kHz waveform).
## Start a new session, open "piano-scale.wav":https://code.soundsoftware.ac.uk/projects/dhoxss15/repository/raw/data/piano-scale.wav, and play it.
## Some information can sort-of be perceived and measured from the waveform here: we can see when the notes start, and can get simple fundamental frequency estimate - zoom in to the first note, switch to Select mode, drag out one cycle - it's about 170 samples, so 44100/170 = 259 Hz - the note is a middle C so true value should be nearer to 261, but this is a fair approximation. (But this is a very simple example, and in particular it's one where the single note's fundamental frequency dominates the harmonic envelope so this simple waveform zero-crossing measurement can be carried out without octave errors.)
* *Spectrogram*. To cover:
** What is a spectrogram?
** How the terms "spectral", "spectrum", and "spectrogram" are related, as well as "discrete Fourier transform", "short-time Fourier transform", etc
** Time/frequency tradeoff
** Fundamental frequency and harmonic series
** Linear and logarithmic frequency scales
## With the piano-scale.wav file open, call up a plain spectrogram - Pane -> Add Spectrogram (or keyboard shortcut on the G key). This is the standard kind of "audio recorder/editor" spectrogram.
## Observe that the Y axis is frequency, with the full recorded frequency range; the X axis is time, as it is for most layers in Sonic Visualiser. The spectrogram is a simple time-frequency breakdown, the output of a series of short-time Fourier transforms, one for each horizontal step.
## Notice that, for each note, we can see the fundamental frequency most strongly and then the harmonics stacked above it. The harmonics are spaced more widely for higher notes because they are at multiples of the note's fundamental frequency, which is larger.
## The colour scale is a dB scale, which means that it boosts quieter content. For this reason we can see the noise floor (general background noise) even though it may not be audible. We can switch the colour scale to Linear to isolate only the strong frequencies. There isn't really enough detail to measure much here. (NB the default colour scheme is unhelpful to colour-blind users, so it might be worth changing to Sunset or Ice scheme.)
## Close that pane and open a "melodic-range spectrogram" - Pane -> Add Melodic Range Spectrogram (or M key). Observe the much more limited frequency range and the fact that this spectrogram uses both Linear colour and Sunset scheme by default.
## However, the biggest difference is that the default vertical (Y) scale for this view is logarithmic in frequency, i.e. with equal spacing between 1, 2, 4, 8, 16 etc rather than between 1, 2, 3, 4, 5 etc as in a linear scale. This makes it essentially linear in pitch, with "one octave" (a doubling in fundamental frequency) always being the same distance on the vertical scale. Correspondingly there is now a little representation of a piano keyboard shown at left, with middle C highlighted.
## The above assumes 12tET with A=440Hz; we can change at least the latter part of that in the Preferences and the scale will move immediately when we do so (try it but, at this point, be sure to restore the default).
## Select the Measure tool and show that we can get a frequency readout with harmonic markers. Return to the Navigate tool and contrast with the readout that is displayed as you move the pointer over the pane.
## Close that pane and open the "peak-frequency spectrogram" - Pane -> Add Peak Frequency Spectrogram (or K key). Notice that here we can just wave the Navigate tool over a bin to get an estimate of the instantaneous frequency there.
## New session, open "A Friendly Warning":https://code.soundsoftware.ac.uk/projects/dhoxss15/repository/raw/data/A%20Friendly%20Warning.ogg again and open both the plain spectrogram and the melodic-range one -- observe and contrast the various visible elements, in particular vertical lines in full frequency range for noisy percussion, relative invisibility of such broadband sounds in the melodic-range spectrogram, glides in vocal, difficulty of distinguishing harmonic traces from simultaneous notes etc.
## Go to File -> Replace Main Audio, open "King Henry":https://code.soundsoftware.ac.uk/projects/dhoxss15/repository/raw/data/Music/07%20-%20King%20Henry.flac. Note among other things that we need to increase the gain on the melodic-range spectrogram, and that the vibrato is visible and things like vibrato rate could be approximately measured, that the long reverb makes the notes appear to overlap in places.
h5. Introductory notes and slides on Vamp plugins
h5. Sonic Visualiser - hands on with Vamp plugins
* *Quick survey of common higher-level features* derived from short-time Fourier analysis. To cover:
** amplitude
** onset detection and tempo estimation
## Start a new session with "A Friendly Warning":https://code.soundsoftware.ac.uk/projects/dhoxss15/repository/raw/data/A%20Friendly%20Warning.ogg.
## Feature extraction plugins are found under the Transform menu, so called because it contains things that turn your audio into something else, including both feature extractors and audio effects. You can run a transform and show the output in the same pane as the audio, or in a new one.
## With the audio pane selected, run Transform -> Analysis by Category -> Time -> Tempo -> Bar and Beat Tracker: Beats. This produces a series of beat locations, each labelled with metrical beat number, and when you play the audio, the beats are played with a tap.
## Create a new pane (Pane -> Add New Pane or shortcut N), and run Transform -> Analysis by Category -> Low Level Features -> Amplitude Follower. This produces an amplitude curve.
## Open another new pane (Pane -> Add New Pane or shortcut N) and run Transform -> Analysis by Category -> Time -> Onsets -> Note Onset Detector: Onsets. Here we have individual note onset positions: it works OK for this sort of music. We can switch playback on and off for the individual feature tracks with the Play toggle on the parameter box at right.
## Some further transforms we can try: Chordino chord estimate; the notes corresponding to Chordino's chord estimate (so as to audition whether the chords sound any good); QM key estimator and key-strength plot.
* *Pitch*. To cover:
** pitch estimation and note segmentation
## Close session, return to "King Henry":https://code.soundsoftware.ac.uk/projects/dhoxss15/repository/raw/data/Music/07%20-%20King%20Henry.flac. Open a melodic-range spectrogram and make sure its pane is current.
## Now run the transform pYIN - Smoothed Pitch Track. A pitch track should appear in a bright colour. Switch its plot type to Discrete Curves and make sure its scale is set to Auto-Align, which means that if it has Hz units, it will be aligned to the same vertical scale as the spectrogram behind it.
## We can of course check this extracted pitch-track visually, but we can also inspect individual values (by mouseover), inspect in bulk (Layer -> Edit Layer Data) including tracking through the data during playback, and export to a file (File -> Export Annotation Layer). Demonstrate this latter. (Note that the correct layer must be selected for any of these to work!)
## This layer can also be synthesised and played back -- switch on the Play button on layer parameters and try it.
## The same plugin can produce note segmentations (for monophonic audio of this type), so run it again requesting the Notes output. Each note is recorded as having a pitch equal to the median of the underlying pitch track's pitches for the time it spans. This is unlikely to sound so nice when played back, because of both segmentation flaws and difficulties (e.g. glides) and interesting properties of pitch perception (e.g. with vibrato). If this kind of use is of interest to you, consider our other program "Tony":/projects/tony.
* Chroma reduction
## Using the NNLS Chroma plugin - open an empty pane and run this transform with the default parameters. This is a single-octave reduction of frequency content (can explain at arbitrary length).
## Run the same transform again in a second pane, but this time with different parameters: local tuning, L2 norm, spectral shape = 0.9.