Piano Evaluation for Level Normalisation » History » Version 41

Chris Cannam, 2014-07-23 02:43 PM

1 1 Chris Cannam
h1. Piano Evaluation for Level Normalisation
2 1 Chris Cannam
3 1 Chris Cannam
Lack of normalisation for Vamp plugin inputs is a problem when analysing quiet recordings (see #1028).
4 1 Chris Cannam
5 1 Chris Cannam
Testing using a small set of piano recordings, quickly evaluating performance across the first 30 seconds under a number of different normalisation / level management regimes.
6 1 Chris Cannam
7 3 Chris Cannam
h3. Input files
8 1 Chris Cannam
9 1 Chris Cannam
|Filename|Signal max approx|
10 4 Chris Cannam
|@31.wav@|0.57|
11 4 Chris Cannam
|@MAPS_MUS-bach_846_AkPnBcht.wav@|0.12|
12 4 Chris Cannam
|@MAPS_MUS-chpn_op7_1_ENSTDkAm.wav@|0.33|
13 4 Chris Cannam
|@MAPS_MUS-scn15_7_SptkBGAm.wav@|0.13|
14 4 Chris Cannam
|@mz_333_1MINp_align.wav@|0.10|
15 2 Chris Cannam
16 2 Chris Cannam
The plugin has one internal threshold parameter, which can be lowered to find quieter notes (at the expense of course of more false positives). We don't really want to expose this (or any continuous controls) as a parameter. But we need to have approximately predictable input levels, for this threshold to be meaningful.
17 2 Chris Cannam
18 3 Chris Cannam
h3. Methods
19 2 Chris Cannam
20 2 Chris Cannam
|Name|Hg revision|Description|
21 25 Chris Cannam
|@norm@|commit:d721a17f3e14|Normalise to 0.50 max before running plugin (can't do this in plugin: it's here as the reference case)|
22 4 Chris Cannam
|@as-is@|commit:d721a17f3e14|No normalisation|
23 4 Chris Cannam
|@to-date@|commit:d9b688700819|Track max signal level _so far_, adjust each sample so that max is at 0.50|
24 28 Chris Cannam
|@r2@,@r3@,@r4@,@r5@,@r6@|commit:b5a8836dd2a4|Preprocess with "Flatten Dynamics":/projects/flattendynamics at 0.02, 0.03, 0.04, 0.05, 0.06 target RMS levels respectively|
25 29 Chris Cannam
|@s8@|commit:4ac067799e0b|With "Flatten Dynamics second attempt":/projects/flattendynamics/wiki/Wiki with max RMS targeted to 0.08|
26 32 Chris Cannam
|@t4@|commit:d67fae2bb29e|With Flatten Dynamics attempt 2a with max RMS targeted to 0.04|
27 37 Chris Cannam
|@u4@|commit:70773820e719|With Flatten Dynamics attempt 2b with max RMS targeted to 0.04|
28 32 Chris Cannam
29 3 Chris Cannam
h3. Results
30 3 Chris Cannam
31 3 Chris Cannam
Reporting only the note onset F-measure for the first 30 seconds of each piece.
32 1 Chris Cannam
33 34 Chris Cannam
|Filename|@norm@|@as-is@|@to-date@|@r2@|@r3@|@r4@|@r5@|@r6@|@s8@|@t4@|@u4@|
34 34 Chris Cannam
|@31.wav@|50|33|40|45|47|48|45|43|42|49|45|
35 34 Chris Cannam
|@MAPS_MUS-bach_846_AkPnBcht.wav@|87|15|62|64|85|87|87|86|81|86|87|
36 34 Chris Cannam
|@MAPS_MUS-chpn_op7_1_ENSTDkAm.wav@|33|31|31|11|25|31|32|31|32|34|35|
37 34 Chris Cannam
|@MAPS_MUS-scn15_7_SptkBGAm.wav@|73|16|61|50|57|67|74|75|70|69|68|
38 34 Chris Cannam
|@mz_333_1MINp_align.wav@|66|3|58|42|60|64|66|63|66|63|65|
39 7 Chris Cannam
40 10 Chris Cannam
The precision (_proportion of correct onsets among detected onsets, or 1 minus the false-positive rate_) and recall (_proportion of correctly-detected onsets among all ground-truth onsets, or true-positive rate_) vary as you would hope:
41 10 Chris Cannam
42 10 Chris Cannam
 * when the resulting audio level is quieter than the @norm@ case, precision is high and recall is low but the F-measure is worse than the @norm@ case
43 10 Chris Cannam
 * when the resulting audio level is louder than the @norm@ case, precision is low and recall is high and the F-measure is still worse than the @norm@ case
44 10 Chris Cannam
45 12 Chris Cannam
This suggests that our threshold (which happens to be 6) is moderately well-suited to the @norm@ case, at least to optimise F-measure (this might not be the most perceptually useful measure though).
46 13 Chris Cannam
47 38 Chris Cannam
The best results (apart from @norm@) above seem to be @r5@ and @u4@. Let's try to refine the parameters for each of those and see if any patterns emerge.
48 38 Chris Cannam
49 38 Chris Cannam
h4. Flatten Dynamics fine-tuning
50 38 Chris Cannam
51 39 Chris Cannam
The adjustable parameters within @r5@, with their defaults, are
52 39 Chris Cannam
53 39 Chris Cannam
|Parameter|Description|Default|
54 39 Chris Cannam
|@historySeconds@|Length of RMS window|4.0 sec|
55 39 Chris Cannam
|@catchUpSeconds@|Length of gain slide window|0.5 sec|
56 39 Chris Cannam
|@targetRMS@|Target RMS value|0.05|
57 39 Chris Cannam
|@maxGain@|Hard limit on gain|20.0|
58 39 Chris Cannam
59 39 Chris Cannam
The @targetRMS@ is the one we have been varying across @r2@, @r3@ etc -- for @r5@ it is fixed at 0.05. We don't need to test @maxGain@ variation.
60 39 Chris Cannam
61 39 Chris Cannam
Here @r5hNcM@ represents the @r5@ method with @historySeconds@ = N and @catchUpSeconds@ = M/10. So @r5@ is the same as @r5h4c05@. The @r5@ test was run again, hence variation from above results.
62 39 Chris Cannam
63 41 Chris Cannam
|Filename|@norm@|@r5@|@r5h2c05@|@r5h5c05@|@r5h6c05@|r5h8c05|@r5h4c01@|@r5h4c10@|
64 41 Chris Cannam
|@31.wav@|50|47|38|47|48|46|46|53|
65 41 Chris Cannam
|@MAPS_MUS-bach_846_AkPnBcht.wav@|87|87|87|87|87|88|86|88|
66 41 Chris Cannam
|@MAPS_MUS-chpn_op7_1_ENSTDkAm.wav@|33|32|33|32|29|31|32|31|
67 41 Chris Cannam
|@MAPS_MUS-scn15_7_SptkBGAm.wav@|73|73|66|72|76|73|73|73|
68 41 Chris Cannam
|@mz_333_1MINp_align.wav@|66|66|64|64|66|63|65|66|
69 41 Chris Cannam
70 39 Chris Cannam
71 14 Chris Cannam
h4. For different piano template sets
72 14 Chris Cannam
73 17 Chris Cannam
The above results are all generated using four piano templates, numbered 1-3 plus @pianorwc@.
74 17 Chris Cannam
75 17 Chris Cannam
Here are results using the @norm@ and @as-is@ methods, but with different sets of piano templates: first with three templates (1-3) and then with each template in turn as the only one.
76 17 Chris Cannam
77 19 Chris Cannam
The template turns out not to make an enormous difference -- perhaps because these recordings contain nothing but piano?
78 13 Chris Cannam
79 13 Chris Cannam
|Filename|@norm@/all|@as-is@/all|@norm@/3of4|@as-is@/3of4|@norm@/1|@as-is@/1|@norm@/2|@as-is@/2|@norm@/3|@as-is@/3|@norm@/rwc|@as-is@/rwc|
80 22 Chris Cannam
|@31.wav@|50|33|51|30|50|34|44|42|50|32|56|36|
81 22 Chris Cannam
|@MAPS_MUS-bach_846_AkPnBcht.wav@|87|15|86|16|86|24|75|20|73|10|71|18|
82 22 Chris Cannam
|@MAPS_MUS-chpn_op7_1_ENSTDkAm.wav@|33|31|32|32|31|22|29|31|35|34|32|28|
83 22 Chris Cannam
|@MAPS_MUS-scn15_7_SptkBGAm.wav@|73|16|71|19|71|12|68|14|72|17|70|15|
84 22 Chris Cannam
|@mz_333_1MINp_align.wav@|66|3|68|1|63|4|67|2|67|1|63|3|
85 20 Chris Cannam
86 20 Chris Cannam
h4. For "generic" template set
87 20 Chris Cannam
88 20 Chris Cannam
The above results all use template sets with only piano templates in them.
89 20 Chris Cannam
90 20 Chris Cannam
Here are results using the @norm@ and @as-is@ methods, but with the full set of instrument templates (four pianos plus all the rest).
91 21 Chris Cannam
92 1 Chris Cannam
|Filename|@norm@|@as-is@|
93 1 Chris Cannam
|@31.wav@|49|37|
94 1 Chris Cannam
|@MAPS_MUS-bach_846_AkPnBcht.wav@|79|34|
95 1 Chris Cannam
|@MAPS_MUS-chpn_op7_1_ENSTDkAm.wav@|31|28|
96 1 Chris Cannam
|@MAPS_MUS-scn15_7_SptkBGAm.wav@|67|16|
97 1 Chris Cannam
|@mz_333_1MINp_align.wav@|63|5|
98 34 Chris Cannam
99 34 Chris Cannam
h4. Cross-checking with non-piano test data
100 34 Chris Cannam
101 34 Chris Cannam
The results need to be roughly comparable with those obtained from pre-normalised data using other datasets as well as the piano one. Here is a subset of the TRIOS dataset. The @norm@ result is that obtained from the plugin prior to doing this work, using pre-normalised data.
102 34 Chris Cannam
103 36 Chris Cannam
The @mirex@ result is that from the MIREX 2012 submission in MATLAB, but note that this always uses all instrument templates while the plugin results are based on selecting the "right" instrument for the piece (which is assumed to be the best, though we aren't actually testing that here).
104 35 Chris Cannam
105 35 Chris Cannam
|File|@mirex@|@norm@|@u4@|
106 35 Chris Cannam
|mozart/piano|60|64|56|
107 35 Chris Cannam
|mozart/viola|33|37|35|
108 35 Chris Cannam
|mozart/mix|51|58|55|
109 35 Chris Cannam
|mozart/clarinet|74|80|86|
110 35 Chris Cannam
|lussier/piano|45|52|63|
111 35 Chris Cannam
|lussier/mix|36|43|40|
112 35 Chris Cannam
|lussier/bassoon|43|75|80|
113 35 Chris Cannam
|lussier/trumpet|43|46|51|
114 35 Chris Cannam
|take_five/piano|61|46|69|
115 35 Chris Cannam
|take_five/mix|62|73|69|
116 35 Chris Cannam
|take_five/saxophone|78|80|84|