Tony: a tool for melody transcription

h1. Initial Specification

The Tony tool will be a very simple user interface for the exact annotation of notes, note pitches and performance of human singing and other monophonic instruments.

The reason it can be kept very simple is that it exports beautifully to other applications such as Sonic Visualiser, which have different strengths.

h3. Minimum Use Case

The first prototype is ready if this scenario is possible:

# User loads a .wav file of a monophonic singing recording,

# Tony automatically extracts pitch track and note track,

# User corrects the onsets and offsets of the pitch track, inserting new ones if necessary, or setting notes to rests,

# User exports the note track to a csv file, with columns ONSET, DURATION, MEDIAN_PITCH.

h3. Users

We expect users to be unfamiliar with programming, but to be familiar with music and research in general. In order to illustrate this, these are some possible users:

# Friedrich (56) is a musicologist at a university in northern Germany. Sample publication: "Zur Ästhetik der Stimme bei Wagner - Eine vergleichende Analyse historischer und moderner Aufnahmen". He uses a Windows Vista laptop with, mainly for Email correspondence and keeping lists of audio recordings in an Excel spreadsheet. He listens to Music from CDs, noting down singing characteristics on a paper notepad. He plays the cello and organises outings to the Berlin philharmonics with his Seminar students.

# Sam (31) is an ethnomusicologist at the University of Essex studying the music of Brazilian immigrants in Portugal. She records the immigrants' music on location in Lisbon and compares the rhythmic and melodic variations of the music between Brazil and Portugal. Sam uses a MacBook from 2007. She organises the field recordings and edits them using Audacity. She regularly dances Capoeira in the local Capoeira group in Colchester, Essex.

# Etienne (25) is a graduate student in the psychology department of a Belgian university and part of a project that aims at replicating the "Levithin" effect. He's a Windows user with a bit of a knack for writing Python scripts. He's frustrated with the way his boss asks him to annotate melodies in Praat, but doesn't quite have the experience or support to build a tool that does it better. Etienne is also associated with a computer science guy who's looked into African rhythms.

# David (25) is a super-bright statistics graduate who's also a clarinet player and just started his PhD on music transcription of wind instruments at a Canadian university. He realises that there's not enough training data out there to learn his models, so he would like to annotate lots of clarinet recordings.

h3. Framework

It's not entirely decided what the framework should be, but there are some restrictions:

* it should work on PCs and Macs (and Linux boxes)

** tablets are not a priority because interaction with free audio editing and annotating programs is needed, i.e. this is a work tool, not a play tool.

** PCs are often used as standard by scientists and teachers outside of CS/maths/stats

** Macs are often used by the creative guys, and MIR people

h3. Components

   * automatic pitch and note transcription methods

   * GUI for correction, and additional note-based or phrase-based annotations

   * export to RDF and graphics

h4. Automatic Transcription Methods

   * pitch extraction

   ** monophonic via Yin

   ** interface for other algorithms, potentially polyphonic

   ** output of salience function, or at least N candidate pitches (even if pitch estimation thinks there's no pitch)

   * Note alignment: align notes to MIDI-like representation given a pitch file

   * Non-alignment note detection

   * Methods should be easily re-executable with time-dependent parameter settings

h4. GUI

The most important aspect of the GUI is that it should be simple because the users are unlikely to be familiar with complicated programs. The most important point should hence be to keep the layout simple. Hence first some *forbidden things*:

* polyphonic melody processing (diverts attention from main use, prevents easy interaction with monophonic interface)

* audio editing (this is an analysis and annotation program, leave that to Audacity)

* have more than one central pane (too confusing, this is a one task application -- audio and visual are already two dimensions, let's not add more)

* load more than one audio file (we leave that to Sonic Visualiser)

* more than one "instrument" (pointer, scissors, eraser, move tool should be superfluous in a simple monophonic environment)

* additional structural annotation: beats, bars... that can be done post-hoc in Sonic Visualiser

So here are some requirements:

* One *main pane* that always stays in roughly the same, central place.

** displays a fixed set of features: pitch track, overlaid note track; hidden: note salience, other pitch track candidates.

** note track is editable

** pitch track is not editable separately (why should it?), only via the note track

* 2 permanent *single-purpose strips* across the length of the main pane

** select (notes or note ranges)

** annotate note (double click for simple annotation, drop-down for specific fields)

* Note editing

** edit note start and end times,

** edit continuous note pitch,

** turn note to rest (unpitched)

** impossible to delete notes (notes can only be made into rests -- we assume a contiguous, monophonic melody)

** (lock notes, i.e. prevent from re-estimation)

* Audio Playback

** playback original audio

** sonify notes

** sonify pitch track

* Undo note and pitch track changes

* Export notes, pitchtrack, annotations in the following formats

** as csv files (very important)

** as graphics (quite important): pdf, possibly self-contained R script

** as SV session file

** as RDF

** as MIDI

Some explicit differences to Sonic Visualiser:

* Tony has only one pane (I know I mentioned that before, can't ever mention it too often)

* arrow keys should only change the playback position, and only via that the visualisation position

* zoom sliders should be very large -- if possible along the whole sides of the main pane

* only one instrument (i.e. not have a separate selection tool, or scissors tool)

** necessitates permanent selection strip (at the top of the main pane), in order to select contiguous notes)

** double click splits/joins

Some explicit differences to Praat:

* Tony has only one window

* Tony has a concept of a contiguous melody

h4. Further possible GUI features

* automatic re-estimation after edits (kind of Maximum A Posteriori or something)