Tony: a tool for melody transcription

Some ideas:

• select region
• control-P (to say "I am drawing")
• draw new pitch track by clicking and dragging
• hit return to confirm & delete any other pre-existing pitch track

• create new note using note editor
• press some shortcut key to draw a constant pitch track at the note's center frequency for the duration of the note
• hit return to confirm & delete any other pre-existing pitch track

• select region
• press some shortcut key to add new pitch
• prompt for frequency value (type or click?)
• draws constant pitch for duration of time range
• user manually adjusts at frame level?
• hit return to confirm & delete any other pre-existing pitch track

Thoughts?

Hello all,

My inclination is to avoid as much as possible the scenario where the user draws something completely manually. These annotations are supposed to be used as a ground truth, and letting users draw their own pitch track can lead to some pretty inaccurate results.

My vision for scenario 2 would be where the user can select a bounding box (in time / fqr), and pYin is asked for a pitch sequence in there. Shouldn't it be possible to force the algorithm to return the highest scoring pitch value within the bounding box at each point in time, even if it's a poor score?

Just a quick note that I have found a couple of cases where there is actually quite a need for this. In particular, I've run into lots of cases where there is either no candidate returned by pyin or the candidates given are not just octave shifts of the real pitch.

This is mainly happening (not surprisingly) for audio files where there is some bleed from other instruments in the track.

Hmmm I see your point.

Is the "right" pitch visible in the spectrogram view? We could have a draw option with 'snap to spectral peaks' for instance. Though I think that is probably more work than we can afford to do given that we want to get started on annotation next week.

j

Rachel Bittner wrote:

Just a quick note that I have found a couple of cases where there is actually quite a need for this. In particular, I've run into lots of cases where there is either no candidate returned by pyin or the candidates given are not just octave shifts of the real pitch.

This is mainly happening (not surprisingly) for audio files where there is some bleed from other instruments in the track.

Ok, Incidentally, I think the newest version should be better at that than the version I suspect you're using. I will have to chat with Chris, probably on what the "definitive" pyin version we should use is. (The one I'm currently using is the latest I pushed to the pyin repo, to the branch "tony").

My vision for scenario 2 would be where the user can select a bounding box (in time / fqr), and pYin is asked for a pitch sequence in there. Shouldn't it be possible to force the algorithm to return the highest scoring pitch value within the bounding box at each point in time, even if it's a poor score?

Yes, that's possible. Options are:

• return best value per frame (no smoothing)
  • advantage: no weird hard-to-explain jumps introduced by smoothing over a restricted area.
  • disadvantage: pitch track might not be continuous
• return best smoothed pitch track over values in range
  • advantage: you'll have a mostly smooth pitch tracks
  • disadvantage: you might get "unexplainable" jumps if the actual pitch crosses your selection boundary (this is what we saw earlier with the jagged pyin candidates)

Matthias

• return best value per frame (no smoothing)
  • advantage: no weird hard-to-explain jumps introduced by smoothing over a restricted area.
  • disadvantage: pitch track might not be continuous

I've just made a new YIN-derivative plugin that does just that: http://code.soundsoftware.ac.uk/projects/pyin/repository/entry/YinVampFreqConstrained.cpp
Returns the frequency corresponding to the best value (according to the YIN difference function) in the frequency range provided.
So if we have a way of calling that from the interface, it might just work.

Noted. I'll see if I can think of a way to deal with this on the Tony side.

(Note that the constrained-pitch candidate no longer appears straight away, in the latest commit -- as a side-effect of work on #885, you now have to hit Ctrl+Return after selecting the region just as you do with a normal selection. The UI feedback could still be clearer as it shows only a time selection after you have finished dragging the rectangle, so that's still a work in progress, but the quality of the actual results remains more pressing.)

However, more pressingly, the results you get simply don't appear to be very good. If I highlight a region that contains a visible harmonic in the spectrogram, I'm as likely to see a scattering of points above and below that harmonic as I am to see anything that looks like a clean pitch track. (See attached picture, choosing second harmonic from an arbitrary note in happy_birthday_mm_normal.wav)

I've played around a little, and in this case, I think this is more of a conceptional issue: YIN doesn't necessary have high salience at spectral peaks since it's a time-domain approach. Rather, it usually has peaks at sub-harmonics.

It does in fact produce nice results if you select a sub-harmonic. This is counter-intuitive only because we're displaying the spectrogram, which is not directly used in any of the computations. But I think this is not actually a problem because YIN/PYIN essentially never makes an octave error in the downward direction.

So I'd say this is not a bug, and I'd postpone the problem until someone finds a true pitch that isn't captured by the constrained pitch output.

That's cool, I don't mind that. Doesn't have to be all pyin.

The only concern might be mixing algorithms. That might not matter much given that the user has fiddled with the stuff anyway, only that you will get different results if you apply it to the same pitch track.

I'm a bit unsure about patches, so I just trust it works ok for now. Happy for you to commit that.

Hm, I wasn't really proposing a solution -- as I said, I don't think the simple cepstrum plugin is entirely up to it either.

I do think it's probably better than using pyin for this, though I wouldn't mind a second opinion.

Not sure whether we have a harmonic-spectrum plugin with frequency range limits ready to roll, though it'd be easy enough to make one. Maybe I should try that too.

Anyway I take your comment as saying you're busy with something else and would be OK with seeing this at least provisionally changed if it looked appropriate.

Hello all,

Actually, the other day Rachel and I were looking at a real example of an instrumental melody where the estimated pitch was below the real fundamental, but not at an octave relationship, and none of the candidates matched it. Similar to what Chris was experiencing, selecting a bounding box around the true location of the pitch curve did not yield coherent pitch values either.

We were thinking that having alternative pitch tracking algorithms available, which may be robust to different types of errors, would probably be the best solution, but also one that's a little late to implement. In fact, this is something we've all discussed at at the early stages of our collaboration, but it seemed like pYin wasn't making any unrecoverable errors so we'd abandoned the idea.

If we're reviving this idea, I'd like to add my 2 cents: first, I think we probably want a frequency-domain algorithm (as a time-domain algorithm is likely to make the same mistakes as yin). Given this, I'd consider the SWIPE algorithm by Camacho: at least in his evaluations it outperforms a variety of pitch trackers (including the classic yin algorithm). The thesis includes a matlab implementation that we could port, I have also found a python implementation here, and perhaps more interestingly a C implementation here.

I'm still a little weary about this since we won't have time to evaluate any pitch algorithm we implement thoroughly (or at all), but if we limit its use to the places where pYin fails, it's probably OK?

Hi all -- I've just pushed a change that makes Tony use the newly-written Constrained Harmonic Peak plugin for frequency-constrained analysis.

This plugin implements a simple harmonic product spectrum. It relies entirely on being given a reasonable bound for the frequency range (i.e. it is only suitable for, and is only used in, the case where the user draws a bounding box using shift-click-drag in Tony).

I agree that we shouldn't be trying difficult or untested algorithms at this point -- and I didn't really want to introduce an untested plugin, but I couldn't find one that was quite suitable in terms of the interface available. What I do think is that, if we're going to introduce a new method, it should be one so simple that we can review the code easily and test that it is actually right.

So please do take a look at the CHP code and tell me if you think it looks wrong! Or if there is any other equally simple method you'd prefer. Must be really simple though, one spectral input frame at a time and no state.

One problem with this is that I'm not sure about the whole business of making sure its outputs are aligned with pyin's correctly in time. The analyser is quite selective about its timing in order to work nicely with pyin, perhaps it won't work so well with this? This one just uses frequency-domain input (timestamp aligned with centre of frame).

As far as providing other more sophisticated pitch trackers goes -- I think it'd be nice to do that but we should hold off for now and consider providing them later as alternatives within the whole application.

This sounds like a sensible approach to me! Of course we should try the resulting plugin with a couple of real problematic recordings and see if it addresses the problem, but assuming it does, it makes sense to use this approach.

In fact, what you're suggesting is basically to compute a harmonic-summation based salience function in the range indicated by the user and to return the most salient pitch candidate within the range, which is not too far from what MELODIA would do for a very constrained pitch range :)

I had a look at the code, the core bit is this one:

for (int j = 1; j <= m_harmonics; ++j) {
            if (j * bin > hs) break;
            hps[i] *= mags[j * bin];
            ++contributing;
        }
        if (hps[i] <= 0.0) {
            hps[i] = -120.0;
        } else {
            hps[i] = 20.0 / contributing * log10(hps[i]);
        }

If I understand correctly, you're summing the dB amplitudes of the harmonics, and then taking the average amplitude by dividing by the number of harmonics used in the summation, and that's the value that you compare over the given pitch range to choose the max?

I think that makes sense. In MELODIA I use the linear peak magnitudes rather than dB, and instead of dividing the sum by the number of harmonics, I use a weighting function so that the higher the harmonic the less it contributes: 0.8^(nHarm -1) (in this formula the fundamental is nHarm=1). But my salience function is designed for polyphonic signal... in the quasi-monophonic case (melody + bleed) a simpler strategy should probably work. The only snag might be that you assume the harmonic will be located at an exact multiple of the candidate pitch, which it won't always be. Klapuri for instance handles this by defining a frequency range around the estimated location of each harmonic and taking the max bin value within this range - again, maybe not necessary, but potentially more robust to slightly inharmonic sounds.

To summarize all of my blurb, I think this is a good approach, lets test it on some files to see whether the proposed harmonic summation needs any tweaking or does the trick. Good stuff!

Cool stuff.

It pains me a bit that I've spent quite a bit of time on this YIN thing, only to realise more and more its failings! I guess I have to live and learn.

I've got another one, will make that one a new issue...

I tried it quickly, it seems to be producing sensible results. I'll only be able to give proper feedback when we start annotation work.

One thing though - the timestamps - normally in SV a plugin that takes frequency domain input returns a timestamps that is at the center of the frame (which is what we want). Is this still the case here? In terms of window size, 4096 (assuming fs=44100) might be a little big - can we use 2048 instead? That'll be more consistent with pYin as well I think.