On Displaying Musical Scores » History » Version 26

Chris Cannam, 2016-01-08 12:53 PM

1 1 Chris Cannam
h1. On Displaying Musical Scores
2 1 Chris Cannam
3 3 Chris Cannam
h3. Use cases
4 1 Chris Cannam
5 3 Chris Cannam
 # Illustrating recordings by attaching a full publication-style score
6 4 Chris Cannam
 ** ...where the score is not intended to be read closely, but rather used as a sort of key-frame reference for navigating the audio
7 4 Chris Cannam
 ** ...or where user expects to be able to zoom the score far enough to actually read it properly
8 3 Chris Cannam
 # Linear rather than page-based score that a user is expected to read, e.g. to play along with, or to check specific notes in a transcription or spectrogram
9 3 Chris Cannam
 # On-the-fly score display of transcriptions and related annotation layers (score from MIDI)
10 3 Chris Cannam
 # Score editing for correction (making usually small edits to substantial scores)
11 3 Chris Cannam
 # Score editing for annotation (generating small scores from scratch)
12 1 Chris Cannam
13 3 Chris Cannam
h3. Sources of score data
14 1 Chris Cannam
15 3 Chris Cannam
 * Score-encoding formats, e.g. MusicXML, MEI
16 3 Chris Cannam
 * PDFs of published scores
17 3 Chris Cannam
 * "Tidy" MIDI files and other cleaned-up annotation data (e.g. chord charts)
18 3 Chris Cannam
 * Untidy output from transcription methods and the like, in MIDI or MIDI-like formats
19 1 Chris Cannam
20 3 Chris Cannam
h3. Possible implementations
21 3 Chris Cannam
22 7 Chris Cannam
 # Integrate code from an existing application (most likely one that is also in C++ using Qt)
23 1 Chris Cannam
 ** Suitable for all use cases, with limitation of requiring score to be available in a digital format
24 19 Chris Cannam
 ** Has drawback of greatly adding to complexity of SV codebase, which is already quite complex
25 7 Chris Cannam
 ** Only possibility if editing is required (but note that adding editing is likely to be very involved even with this approach, maybe prohibitively so)
26 3 Chris Cannam
 ** MuseScore -- most obvious option
27 3 Chris Cannam
 ** Rosegarden -- has a notation editor component but is primarily a sequencer (but does have the advantage that I wrote much of it and so know a lot of the code!)
28 7 Chris Cannam
 # Invoke external application to render to PDF or sequence of images, then display PDF/image pages
29 20 Chris Cannam
 ** Suitable for use case 1; unsuitable for use cases 2, 4, 5 (no way to do linear notation); uncertain for use case 3
30 1 Chris Cannam
 ** Replaces problem of "integrating score display" with problem of "integrating PDF/image page display"
31 19 Chris Cannam
 ** Could possibly draw from existing image layer display in SV
32 6 Chris Cannam
 ** Could use MuseScore as the external application (it has a batch mode) or Lilypond, or something else like VexFlow, or more than one option depending on platform & input format
33 6 Chris Cannam
 ** Has advantage of also introducing ability to display other PDF material than scores
34 5 Chris Cannam
 ** Adds access to non-digital (scanned) score content
35 3 Chris Cannam
36 8 Chris Cannam
Both have their technical challenges, but there is also a conceptual question about how to align score pages or bars in the time axis (more on this below).
37 2 Chris Cannam
38 8 Chris Cannam
h3. Which code to use for option 1?
39 1 Chris Cannam
40 26 Chris Cannam
I think the only reasonable options are MuseScore and Rosegarden. (Lilypond might be a faint possibility, but it's written in a combination of languages and doesn't natively render to Qt.) 
41 26 Chris Cannam
42 26 Chris Cannam
MuseScore produces much better output than Rosegarden and has fairly well-organised code. Rosegarden has a more limited renderer and currently lacks MusicXML import. The only reason to consider anything other than MuseScore is that I'm quite familiar with the Rosegarden code myself already (and, I suppose, that it might be possible to contribute improvements to Rosegarden as well). At any rate, Rosegarden makes a useful baseline comparison.
43 1 Chris Cannam
44 8 Chris Cannam
Here are some examples, deliberately choosing small sizes in order to see how a zoomed-out scale looks, and considering both linear and paginated layouts.
45 8 Chris Cannam
46 8 Chris Cannam
h4. Linear layouts with "good" source material
47 8 Chris Cannam
48 8 Chris Cannam
Both applications are displaying their own example files here, so we have two different pieces but both are pieces that are expected to work well.
49 8 Chris Cannam
50 8 Chris Cannam
h5. MuseScore
51 8 Chris Cannam
52 2 Chris Cannam
!/attachments/download/1776/musescore-tidy-linear.png!
53 1 Chris Cannam
54 8 Chris Cannam
h5. Rosegarden
55 2 Chris Cannam
56 1 Chris Cannam
!/attachments/download/1779/rosegarden-tidy-linear.png!
57 1 Chris Cannam
58 8 Chris Cannam
h4. Paginated layouts with "good" source material
59 1 Chris Cannam
60 8 Chris Cannam
Same pieces as above.
61 2 Chris Cannam
62 8 Chris Cannam
h5. MuseScore
63 8 Chris Cannam
64 2 Chris Cannam
!/attachments/download/1778/musescore-tidy-paginated.png!
65 1 Chris Cannam
66 8 Chris Cannam
h5. Rosegarden
67 2 Chris Cannam
68 1 Chris Cannam
!/attachments/download/1780/rosegarden-tidy-paginated.png!
69 2 Chris Cannam
70 8 Chris Cannam
h4. Imported from an approximate transcription
71 2 Chris Cannam
72 14 Chris Cannam
This is an input that is not expected to work well -- a transcription obtained from the Silvet Vamp plugin of a MAPS database recording of Chopin mazurka op 7 no 1, exported to MIDI. Shown below is the result of taking that MIDI file as exported by Sonic Visualiser and importing it into each application with the default quantization/tidying settings.
73 12 Chris Cannam
74 18 Chris Cannam
Despite their substantially different appearances, these are both displaying the same MIDI data. Besides quantization differences, they use different numbers of staffs and different clefs, keys, and time signatures. None of those are given in the MIDI file here and both programs contain logic to guess them. MuseScore has guessed the wrong key (according to the original score) but the right time signature. Rosegarden has the right key but the wrong time signature.
75 2 Chris Cannam
76 8 Chris Cannam
h5. MuseScore
77 8 Chris Cannam
78 2 Chris Cannam
!/attachments/download/1775/musescore-messy.png!
79 2 Chris Cannam
80 8 Chris Cannam
h5. Rosegarden
81 1 Chris Cannam
82 1 Chris Cannam
!/attachments/download/1777/rosegarden-messy.png!
83 21 Chris Cannam
84 21 Chris Cannam
h3. How much code is involved?
85 21 Chris Cannam
86 21 Chris Cannam
MuseScore's @libmscore@ appears to contain the rendering code (not sure about file I/O). That is 136 .cpp files plus headers, 128K lines total including headers.
87 21 Chris Cannam
88 22 Chris Cannam
Rosegarden doesn't have a dedicated library structure. Its notation code is divided between @base@ and @gui/editor/notation@ directories, with MIDI file I/O elsewhere (a small fraction of the code in a @sound/@ directory). @base@ is 53 .cpp files, @gui/editors/notation@ is 42, combined they have 76K lines including headers.
89 21 Chris Cannam
90 21 Chris Cannam
h3. Aligning score pages and bars in the time axis
91 24 Chris Cannam
92 24 Chris Cannam
There's a conceptual problem with how to dispose a score along the x-axis (time) when aligning with audio or audio feature material in Sonic Visualiser.
93 24 Chris Cannam
94 24 Chris Cannam
First you need a mapping between score position (bar number etc) and audio time. If the score is a representation of a SV note layer (MIDI output from a note transcription perhaps) then this might be intrinsic to the layer. If the audio is on the same timeline as the score (e.g. is an audio rendering from MIDI) then the mapping could be trivial. But more usually an imported score will have no obvious mapping. It will need to be set up manually, by the user e.g. dragging bits of the score around, or automatically, by performing audio-score alignment (or both, with the user correcting an automatic alignment). Scores displayed page-by-page presumably can cope with a coarser mapping than linear ones.
95 24 Chris Cannam
96 24 Chris Cannam
Given a mapping between score position and audio time, it's necessary to position the score elements in the right places.
97 24 Chris Cannam
98 24 Chris Cannam
With a page-by-page score this should be just a question of dropping in page images, except that they may need to be scaled or moved to avoid overlap (since SV zooms in the time axis usually independently of the vertical axis). The existing SV image layer does some of that scaling itself, not always successfully.
99 24 Chris Cannam
100 24 Chris Cannam
With a linear score, the job presumably is to ensure bar lines appear at the right places and stretch or squash bars to ensure that. Any renderer that can do page rendering should be able to handle stretching and squashing individual bars (it's necessary when justifying a system), though there's likely to be a limit on how much a bar can be squished. There may be some room for adapting how the score zoom responds to time-axis zoom (scaling the score in both axes vs changing only the horizontal spacing).
101 24 Chris Cannam
102 24 Chris Cannam
I haven't had any experience with systems that try to display score and audio in the same timeline, presumably there are some examples that could be learned from.