Musicologists » History » Version 1

Mathieu Barthet, 2011-06-06 04:57 PM
Sonic Visualiser adaptations to musicologists' needs

1 1 Mathieu Barthet
h1. Musicologists
2 1 Mathieu Barthet
3 1 Mathieu Barthet
h2. Ethnographic observations at the British Library
4 1 Mathieu Barthet
5 1 Mathieu Barthet
This section presents several themes to adapt or improve Sonic Visualiser for musicological research purposes based on ethnographic observations carried out at the British Library from February to May 2011 by Mathieu Barthet.
6 1 Mathieu Barthet
7 1 Mathieu Barthet
*User Interface:*
8 1 Mathieu Barthet
9 1 Mathieu Barthet
+Observations:+
10 1 Mathieu Barthet
Musicologists alternate two listening practices: closed listening  (without visualisation), and  multimodal listening (with visualisation). Cross-modal effects between auditory and visual feedback occur. Due to this effect, it is deemed important to start with closed listening, and then use multimodal listening if necessary.
11 1 Mathieu Barthet
12 1 Mathieu Barthet
+Solutions:+
13 1 Mathieu Barthet
SV could allow a closed listening mode (without visualisation) and a multimodal listening mode (with visualisation). Closed listening could be associated with a basic mode (or skin), in the spirit of VLC, with basic playback functionalities (e.g. play/stop, navigation, volume, equalization). Multimodal listening could be associated with an advanced mode (or skin) offering visualisations (waveform, spectrogram) and more advanced functionalities (e.g. Vamp plugin transform).
14 1 Mathieu Barthet
15 1 Mathieu Barthet
*Automation/personalisation of spectrogram measurements:*
16 1 Mathieu Barthet
17 1 Mathieu Barthet
+Observations:+
18 1 Mathieu Barthet
The measurement tool provided by SV is used to measure time-frequency related variations, like the rate and extent of a vibrato. The process presents several drawbacks:
19 1 Mathieu Barthet
- it can be time-consuming (when performed on many different performances/notes), 
20 1 Mathieu Barthet
- it may not be systematic (the measurement tool is adjusted manually to match the amplitude variations of tones’ partial, the precision of the process depends on the level of magnification, eye sensitivity, and the visualisation settings chosen by the user).
21 1 Mathieu Barthet
22 1 Mathieu Barthet
+Solutions:+
23 1 Mathieu Barthet
- The measurement tool could be associated with a tone partial tracking functionality allowing to detect the amplitude variations of the partial in the selected area automatically. Descriptors about the amplitude variations of the partial could be computed (e.g. mean, variance, rate, extent, regularity). A solution would be to integrate some functionalities of Xue Wang’s Harmonic Visualiser into SV (including audio synthesis of selected partials). One of the difficulty would be to develop the framework allowing the interaction with the spectrogram in SV. Another possibility would be to design a tone partial tracking Vamp plugin that would take the parameters of the measurement rectangle as input.
24 1 Mathieu Barthet
25 1 Mathieu Barthet
- Users should be able to save the settings used in spectrogram visualisation (color, scale, window, bins, and magnification) so that they can be further applied when analysing other audio files. This aspect can be managed by the use of customizable SV templates.
26 1 Mathieu Barthet
27 1 Mathieu Barthet
*Audio feedback and sonification of metadata:*
28 1 Mathieu Barthet
29 1 Mathieu Barthet
+Observations:+
30 1 Mathieu Barthet
Users tried to click on the notes from the piano representation going along with the melodic spectrogram visualisation to listen to them.
31 1 Mathieu Barthet
32 1 Mathieu Barthet
+Solutions:+
33 1 Mathieu Barthet
It would be useful to add a note audio feedback in the melodic spectrogram visualisation. To a wider extent, it would be useful to sonify the metadata extracted by Vamp plugins when relevant (e.g. chords).
34 1 Mathieu Barthet
35 1 Mathieu Barthet
*Scores:*
36 1 Mathieu Barthet
37 1 Mathieu Barthet
+Observations:+
38 1 Mathieu Barthet
Musicologists often use scores while listening. They prefer to read the score on a page, not in chunks. They often use specific score editions which can be obtained as PDF scanned copies on online music sheet database (IMLSP). Visualisation of the performers’ expressive deviations from the score could enhance performance practice analysis.
39 1 Mathieu Barthet
40 1 Mathieu Barthet
+Solutions:+
41 1 Mathieu Barthet
- SV could provide an “Import score” functionality allowing to use several formats (symbolic like MIDI and MusicXML, and images, like PDF) [MIDI can already be imported as an annotation layer]. The UI should make possible to see score in a page mode on the screen (using e.g. a specific score template). Part of the code of Rose Garden from Chris Cannam may be used for that purpose (difficulty: Rose Garden is built on GTK and not Qt).
42 1 Mathieu Barthet
43 1 Mathieu Barthet
- An Optical Music Recognition (OMR) engine could be embedded into SV (e.g. SharpEye) to convert PDF scores into machine-readable notations. The state of the art tools still offer poor performance for hand-produced scores (see the related post on the IMSLP forum: http://imslpforums.org/viewtopic.php?f=12&t=2805).
44 1 Mathieu Barthet
45 1 Mathieu Barthet
- A collaborative project with online music sheet database (e.g. MuseScore, IMSLP) could be set up to design dedicated API / SPARQL end-points to automatically retrieve scores within SV when these are available on the database (using the audio file metadata).
46 1 Mathieu Barthet
47 1 Mathieu Barthet
- Score visualisations could be further associated with audio-to-score and lyrics-to-audio alignment techniques.
48 1 Mathieu Barthet
49 1 Mathieu Barthet
- Assuming a reliable audio-to-score alignment technique, SV could provide various visualisations of performers’ expressive deviations from the score including timing, pitch, dynamics. The user should be given the possibility to integrate implicit rules of interpretation not written on the score in the expressive deviations visualisations (e.g. notes inégales pattern).
50 1 Mathieu Barthet
Acoustical features statistics could also be provided at various music time scales using information from the score: e.g. note-based level, phrase-based (requires additional rules), bar-based. 
51 1 Mathieu Barthet
52 1 Mathieu Barthet
*Text editor functionality:*
53 1 Mathieu Barthet
54 1 Mathieu Barthet
+Observations:+
55 1 Mathieu Barthet
Musicologists often write down notes while listening and proof read/enrich them during further listening. They often work with speech recordings and perform transcriptions. Switching between different devices (e.g. a CD player and the computer), or software (between the text editor and the audio player on a computer) can be time-consuming and irritating. Time localisation (e.g. tape counter) are often manually reported to connect the notes with a position in the recording.
56 1 Mathieu Barthet
57 1 Mathieu Barthet
+Solutions:+
58 1 Mathieu Barthet
SV should integrate a text editing pane allowing to write notes while listening. Users could be able to link the written notes with a localisation in time with the audio signal. The notes should be exportable in standard formats (e.g. RTF) to be shared and further modified in standard text editors. The control of the playback could be made easy using keyboard shortcuts (allowing to stay in the text editing pane while listening, rewinding, etc.). SV could provide the possibility to use a Midi footswitch pedal to control the playback.
59 1 Mathieu Barthet
60 1 Mathieu Barthet
*Enhancing speech and music recordings analysis:*
61 1 Mathieu Barthet
62 1 Mathieu Barthet
+Observations:+
63 1 Mathieu Barthet
Musicologists often use broadcast recordings including both speech and music. They often have to transcribe interviews.
64 1 Mathieu Barthet
65 1 Mathieu Barthet
+Solutions:+
66 1 Mathieu Barthet
Content-based MIR techniques could be used to:
67 1 Mathieu Barthet
- facilitate the navigation between speech and music sections (development of a Vamp plugin for automatic speech/music segmentation);
68 1 Mathieu Barthet
- transcribe the interviews (development of a Vamp plugin for automatic speech recognition).
69 1 Mathieu Barthet
70 1 Mathieu Barthet
*Sound editor functionality:*
71 1 Mathieu Barthet
72 1 Mathieu Barthet
+Observations:+
73 1 Mathieu Barthet
Prior to analysing the music recordings, musicologists often have to digitize them (e.g. LP), or rip a CD. They also often need sound examples to illustrate talks or lectures.
74 1 Mathieu Barthet
75 1 Mathieu Barthet
+Solutions:+
76 1 Mathieu Barthet
SV could integrate some basic sound editor functionalities like those proposed by Audacity (e.g. CD ripping, cut and paste, amplitude envelope modifications, record from the input audio channel).