Speed¶

Aims¶

We want to make the plugin as fast as possible, but I think there's a case to be made for providing fast and slow modes (see Possibilities for Plugin Parameters).

In "fast" mode we should have the aim of producing a reasonable transcription in faster than real-time on any computer from the past 5 years or so. "Slow" mode has no particular speed constraint, simply as fast as possible an implementation of the best results we can easily do.

See the timing directory in the repo for timing tests, summarised below. See the end of the results file, and "slower computers" below, for some figures from older hardware. Also included at the bottom are some evaluation metrics, just to verify that the different revisions don't actually produce different results.

Work so far¶

Thinkpad T540p, 2-core+HT 64-bit Intel i5-4330M under 64-bit Linux.

• ce64d11ef336, pre-optimisation (release build) takes 104 seconds to process a 43.5-second file. (For reference, a debug build takes over 850 seconds.)

• Experiments to test where the time is spent:
  • 78a7bf247016 removing the unused Vamp plugin outputs: no more than 1% difference
  • f3bf6503e6c6 removing debug printouts: no more than 1% difference
  • Adjusting the CQ resampler parameters to allow a lower SNR: no more than 1% difference
  • 5314d3361dfb halving the number of EM iterations: reduces runtime by 43% (to 59 sec). If this is linear, then EM must be taking around 86% of the total.

• Optimising EM:
  • 97b77e7cb94c storing the templates as double instead of single-precision floats saves around 4% overall, for 100 sec
  • (Alternatively, 840c0d703bbb storing them as floats and using single-precision arithmetic throughout saves around 14%, but presumably produces different results -- not pursued at this point)
  • 19f6832fdc8a using bqvec library for raw vector allocation and manipulation instead of std::vector saves a further 10%, for 89 sec
  • A couple of experiments to try to get the template arrays better aligned failed
  • 6890dea115c3 factoring out a further loop saves another 11%, for 78 sec

• Multi-threading:
  • df05f855f63b using OpenMP for the loop through columns when calling out to EM halves the runtime again (for 41s total), though now consuming 122s "user" time
  • the same code with OMP_NUM_THREADS=1 now runs in 78 sec

That work was merged to default, for a new baseline time of 41s.

• Optimising EM again:
  • f25b8e7de0ed not processing templates that are out of range for an instrument: saves 58% for 24 sec, or 41s single-threaded
  • d2bc51cc7f57 updated constant-Q implementation with slightly faster resampler settings: also 24 sec...
  • fc06b6f33021 converting EM from double to single-precision floats again: 19 sec

Subsequent commits increased the distinction between draft and HQ modes, using fewer iterations and octaves in draft mode and more for HQ, as well as finer timing in HQ mode.

• 8af9b6cd7451 in HQ mode: 26 sec
• 8af9b6cd7451 in draft mode: 6 sec

Slower computers¶

Thinkpad T40p, single-core 32-bit 1.6GHz Pentium-M. This is almost a decade old and quite a lot slower than any reasonable target for real-time performance.

• ce64d11ef336 (104s on reference computer): 541 sec
• f25b8e7de0ed (24s on reference computer or 41s single-threaded): 415 sec (only 23% faster, or less than 11% of real-time speed)
• f25b8e7de0ed in draft mode (no shifts) (14s on reference computer): 210 sec (21% of real-time speed)
• 8af9b6cd7451 in draft mode (6s on reference computer): 79 sec (55% of real-time speed)

Other possibilities¶

• Compare the quality of results using float arithmetic to those using doubles (update: see below)
• Adaptively select the number of EM iterations -- if the process is converging more quickly, break off sooner (how to measure convergence?)
• Optimise the constant-Q -- it wasn't a very significant part of the runtime to start with, but is presumably becoming more significant now

Evaluation¶

For comparison,

Validating MIREX against ground truth at 50 ms:
precision 56.5, recall 68.3, accuracy 44.8, F 61.8

Validating MIREX against ground truth at 100 ms:
precision 65.5, recall 79.1, accuracy 55.9, F 71.7

Validating MIREX against ground truth at 150 ms:
precision 70.4, recall 85, accuracy 62.7, F 77

ce64d11ef336

Validating against ground truth at 50 ms:
precision 34.9, recall 51.8, accuracy 26.3, F 41.7

Validating against MIREX submission at 50 ms:
precision 27.8, recall 34.1, accuracy 18.1, F 30.6

Validating against ground truth at 100 ms:
precision 49.5, recall 73.6, accuracy 42, F 59.2

Validating against MIREX submission at 100 ms:
precision 64.6, recall 79.4, accuracy 55.3, F 71.2

Validating against ground truth at 150 ms:
precision 52.7, recall 78.2, accuracy 45.9, F 63

Validating against MIREX submission at 150 ms:
precision 69.4, recall 85.3, accuracy 62, F 76.5

f25b8e7de0ed

Validating against ground truth at 50 ms:
precision 33.8, recall 52.4, accuracy 25.9, F 41.1

Validating against MIREX submission at 50 ms:
precision 27, recall 34.7, accuracy 17.9, F 30.4

Validating against ground truth at 100 ms:
precision 47.8, recall 74.2, accuracy 41, F 58.2

Validating against MIREX submission at 100 ms:
precision 64.9, recall 83.2, accuracy 57.4, F 72.9

Validating against ground truth at 150 ms:
precision 50.7, recall 78.5, accuracy 44.5, F 61.6

Validating against MIREX submission at 150 ms:
precision 68.9, recall 88.4, accuracy 63.2, F 77.4

840c0d703bbb (float processing in EM)

Validating against ground truth at 50 ms:
precision 34.1, recall 51.8, accuracy 25.9, F 41.1

Validating against MIREX submission at 50 ms:
precision 27.1, recall 34.1, accuracy 17.8, F 30.2

Validating against ground truth at 100 ms:
precision 48.4, recall 73.6, accuracy 41.2, F 58.4

Validating against MIREX submission at 100 ms:
precision 63.1, recall 79.4, accuracy 54.3, F 70.3

Validating against ground truth at 150 ms:
precision 51.5, recall 78.2, accuracy 45, F 62.1

Validating against MIREX submission at 150 ms:
precision 67.8, recall 85.3, accuracy 60.8, F 75.6