# HG changeset patch # User Chris Cannam # Date 1441126263 -3600 # Node ID f35bbb3e4d41caf3ca1be5576ed8ef6ef7c23e34 # Parent c8b934ed4dddb2473e4e36ac29ee50f6d46f9cf3 Change of tack: rename the JSON writer to JAMS. It's simpler than having a JSON writer with a mandatory flag to tell it what sort of JSON to write. diff -r c8b934ed4ddd -r f35bbb3e4d41 CHANGELOG --- a/CHANGELOG Tue Sep 01 17:29:20 2015 +0100 +++ b/CHANGELOG Tue Sep 01 17:51:03 2015 +0100 @@ -1,3 +1,23 @@ + +Changes in Sonic Annotator 1.2 since the previous release 1.1: + +Front-end changes: + + - Add the --segments-from option, providing the ability to read + segment boundaries from a file + +Back-end (feature writer) changes: + + - Rename the JSON feature writer to JAMS. There may be other JSON + formats supported in future + +Bug fixes: + + - Fix invalid JSON written by JAMS feature writer for dense + features. + - Fix invalid UTF-8 output from RDF feature writer when processing + MP3 files having ID3 tags in non-ASCII, non-UTF8 encodings + Changes in Sonic Annotator 1.1 since the previous release 1.0: diff -r c8b934ed4ddd -r f35bbb3e4d41 runner/FeatureWriterFactory.cpp --- a/runner/FeatureWriterFactory.cpp Tue Sep 01 17:29:20 2015 +0100 +++ b/runner/FeatureWriterFactory.cpp Tue Sep 01 17:51:03 2015 +0100 @@ -36,7 +36,7 @@ tags.insert("csv"); tags.insert("lab"); tags.insert("midi"); - tags.insert("json"); + tags.insert("jams"); return tags; } @@ -55,7 +55,7 @@ return new LabFeatureWriter(); } else if (tag == "midi") { return new MIDIFeatureWriter(); - } else if (tag == "json") { + } else if (tag == "jams") { return new JAMSFeatureWriter(); } diff -r c8b934ed4ddd -r f35bbb3e4d41 runner/JAMSFeatureWriter.cpp --- a/runner/JAMSFeatureWriter.cpp Tue Sep 01 17:29:20 2015 +0100 +++ b/runner/JAMSFeatureWriter.cpp Tue Sep 01 17:51:03 2015 +0100 @@ -32,7 +32,7 @@ SupportOneFilePerTrack | SupportOneFileTotal | SupportStdOut, - "json"), + "json"), // file extension is json even with jams writer m_network(false), m_networkRetrieved(false), m_n(1), @@ -61,12 +61,6 @@ p.hasArg = false; pl.push_back(p); - p.name = "format"; - p.description = "JSON format to use. Currently the only supported value is \"jams\". Other formats may be supported in future."; - p.hasArg = true; - p.mandatory = true; - pl.push_back(p); - return pl; } @@ -79,13 +73,6 @@ i != params.end(); ++i) { if (i->first == "network") { m_network = true; - } else if (i->first == "format") { - if (i->second == "jams") { - m_format = i->second; - } else { - throw std::runtime_error - ("Unknown or invalid --json-format parameter \"" + i->second + "\""); - } } } } @@ -329,7 +316,7 @@ cerr << "NOTE: No RDF description for plugin ID \"" << pluginId << "\"" << endl; if (!m_network) { - cerr << " Consider using the --json-network option to retrieve plugin descriptions" << endl; + cerr << " Consider using the --jams-network option to retrieve plugin descriptions" << endl; cerr << " from the network where possible." << endl; } } @@ -391,7 +378,7 @@ } else { - cerr << "WARNING: Cannot currently write dense or track-level outputs to JSON format (only sparse ones). Will proceed using UnknownTask type, but this probably isn't going to work" << endl; + cerr << "WARNING: Cannot currently write dense or track-level outputs to JAMS format (only sparse ones). Will proceed using UnknownTask type, but this probably isn't going to work" << endl; } } diff -r c8b934ed4ddd -r f35bbb3e4d41 tests/test-as-advertised/test-as-advertised.sh --- a/tests/test-as-advertised/test-as-advertised.sh Tue Sep 01 17:29:20 2015 +0100 +++ b/tests/test-as-advertised/test-as-advertised.sh Tue Sep 01 17:51:03 2015 +0100 @@ -31,10 +31,6 @@ # that have no values (but are only point events). I don't know # how reasonable that is, but it's clearly intentional. It also # writes to a subdirectory $basedir/$catid/$trackid.$output - # - # * The "json" reader has a mandatory --json-format parameter that - # currently only accepts one argument ("jams"). It should fail if - # run with any other value or without this parameter. case $type in audiodb) @@ -46,16 +42,6 @@ $r -t $onsets -w $type $tmpwav > $tmpdir/test.out 2>/dev/null || \ fail "Fails to run with reader type \"$type\" and default options" ;; - json) - $r -t $onsets -w $type $tmpwav 2>/dev/null && \ - fail "Wrongly succeeds in running with reader type \"$type\" and default options" - $r -t $onsets -w $type $tmpwav --json-format blah 2>/dev/null && \ - fail "Wrongly succeeds in running with reader type \"$type\" and unknown json-format option" - $r -t $onsets -w $type $tmpwav --json-format 2>/dev/null && \ - fail "Wrongly succeeds in running with reader type \"$type\" and empty json-format option" - $r -t $onsets -w $type $tmpwav --json-format jams 2>/dev/null || \ - fail "Fails to run with reader type \"$type\" and correct json-format option" - ;; *) $r -t $onsets -w $type $tmpwav 2>/dev/null || \ fail "Fails to run with reader type \"$type\" and default options" diff -r c8b934ed4ddd -r f35bbb3e4d41 tests/test-jams-destinations/test-jams-destinations.sh --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/tests/test-jams-destinations/test-jams-destinations.sh Tue Sep 01 17:51:03 2015 +0100 @@ -0,0 +1,175 @@ +#!/bin/bash + +. ../include.sh + +infile1=$audiopath/3clicks8.wav +infile2=$audiopath/6clicks8.wav + +infile1dot=$audiopath/3.clicks.8.wav + +outfile1=3clicks8.json +outfile2=6clicks8.json + +outfile3=3clicks8_vamp_vamp-example-plugins_percussiononsets_onsets.json +outfile4=3clicks8_vamp_vamp-example-plugins_percussiononsets_detectionfunction.json +outfile5=6clicks8_vamp_vamp-example-plugins_percussiononsets_onsets.json +outfile6=6clicks8_vamp_vamp-example-plugins_percussiononsets_detectionfunction.json + +outfile1dot=3.clicks.8.json + +tmpjson=$mypath/tmp_1_$$.json + +trap "rm -f $tmpjson $outfile1 $outfile2 $outfile3 $outfile4 $outfile5 $outfile6 $infile1dot $outfile1dot $audiopath/$outfile1 $audiopath/$outfile2 $audiopath/$outfile3 $audiopath/$outfile4 $audiopath/$outfile5 $audiopath/$outfile6 $audiopath/$outfile1dot" 0 + +transformdir=$mypath/transforms + +mandatory="-w jams" + + +ctx="onsets transform, one audio file, default JAMS writer destination" + +rm -f $audiopath/$outfile1 + +$r -t $transformdir/onsets.n3 $mandatory $infile1 2>/dev/null || \ + fail "Fails to run with $ctx" + +check_json $audiopath/$outfile1 "$ctx" + + +ctx="onsets transform, one audio file with dots in filename, default JAMS writer destination" + +rm -f $audiopath/$outfile1 + +cp $infile1 $infile1dot + +$r -t $transformdir/onsets.n3 $mandatory $infile1dot 2>/dev/null || \ + fail "Fails to run with $ctx" + +check_json $audiopath/$outfile1dot "$ctx" + +rm -f $infile1dot $audiopath/$outfile1dot + + +ctx="onsets and df transforms, one audio file, default JAMS writer destination" + +rm -f $audiopath/$outfile1 + +$r -t $transformdir/onsets.n3 -t $transformdir/detectionfunction.n3 $mandatory $infile1 2>/dev/null || \ + fail "Fails to run with $ctx" + +check_json $audiopath/$outfile1 "$ctx" + + +ctx="onsets transform, two audio files, default JAMS writer destination" + +rm -f $audiopath/$outfile1 +rm -f $audiopath/$outfile2 + +$r -t $transformdir/onsets.n3 $mandatory $infile1 $infile2 2>/dev/null || \ + fail "Fails to run with $ctx" + +check_json $audiopath/$outfile1 "$ctx" +check_json $audiopath/$outfile2 "$ctx" + + +ctx="onsets transform, two audio files, one-file JAMS writer" + +$r -t $transformdir/onsets.n3 $mandatory --jams-one-file $tmpjson $infile1 $infile2 2>/dev/null || \ + fail "Fails to run with $ctx" + +check_json $tmpjson "$ctx" + + +ctx="onsets transform, two audio files, stdout JAMS writer" + +$r -t $transformdir/onsets.n3 $mandatory --jams-stdout $infile1 $infile2 2>/dev/null >$tmpjson || \ + fail "Fails to run with $ctx" + +check_json $tmpjson "$ctx" + + +ctx="onsets transform, one audio file, many-files JAMS writer" + +rm -f $audiopath/$outfile3 + +$r -t $transformdir/onsets.n3 $mandatory --jams-many-files $infile1 2>/dev/null || \ + fail "Fails to run with $ctx" + +check_json $audiopath/$outfile3 "$ctx" + + +ctx="onsets transform, two audio files, many-files JAMS writer" + +rm -f $audiopath/$outfile3 +rm -f $audiopath/$outfile5 + +$r -t $transformdir/onsets.n3 $mandatory --jams-many-files $infile1 $infile2 2>/dev/null || \ + fail "Fails to run with $ctx" + +check_json $audiopath/$outfile3 "$ctx" +check_json $audiopath/$outfile5 "$ctx" + + +ctx="onsets and df transforms, two audio files, many-files JAMS writer" + +rm -f $audiopath/$outfile3 +rm -f $audiopath/$outfile4 +rm -f $audiopath/$outfile5 +rm -f $audiopath/$outfile6 + +$r -t $transformdir/onsets.n3 -t $transformdir/detectionfunction.n3 $mandatory --jams-many-files $infile1 $infile2 2>/dev/null || \ + fail "Fails to run with $ctx" + +check_json $audiopath/$outfile3 "$ctx" +check_json $audiopath/$outfile4 "$ctx" +check_json $audiopath/$outfile5 "$ctx" +check_json $audiopath/$outfile6 "$ctx" + + +ctx="output base directory" + +rm -f ./$outfile1 + +$r -t $transformdir/onsets.n3 -t $transformdir/detectionfunction.n3 $mandatory --jams-basedir . $infile1 2>/dev/null || \ + fail "Fails to run with $ctx" + +check_json ./$outfile1 "$ctx" + + +ctx="output base directory and many-files" + +rm -f ./$outfile3 +rm -f ./$outfile5 + +$r -t $transformdir/onsets.n3 $mandatory --jams-basedir . --jams-many-files $infile1 $infile2 2>/dev/null || \ + fail "Fails to run with $ctx" + +check_json ./$outfile3 "$ctx" +check_json ./$outfile5 "$ctx" + + +ctx="nonexistent output base directory" + +$r -t $transformdir/onsets.n3 $mandatory --jams-basedir ./DOES_NOT_EXIST $infile1 2>/dev/null && \ + fail "Fails with $ctx by completing successfully (should refuse and bail out)" + + +ctx="existing output file and no --jams-force" + +touch $audiopath/$outfile1 + +$r -t $transformdir/onsets.n3 $mandatory $infile1 2>/dev/null && \ + fail "Fails by completing successfully when output file already exists (should refuse and bail out)" + + +ctx="existing output file and --jams-force" + +touch $audiopath/$outfile1 + +$r -t $transformdir/onsets.n3 $mandatory --jams-force $infile1 2>/dev/null || \ + fail "Fails to run with $ctx" + +check_json $audiopath/$outfile1 "$ctx" + + +exit 0 diff -r c8b934ed4ddd -r f35bbb3e4d41 tests/test-jams-destinations/transforms/detectionfunction.n3 --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/tests/test-jams-destinations/transforms/detectionfunction.n3 Tue Sep 01 17:51:03 2015 +0100 @@ -0,0 +1,11 @@ +@prefix rdf: . +@prefix vamp: . +@prefix examples: . +@prefix : <#>. + +:transform0 a vamp:Transform; + vamp:plugin examples:percussiononsets ; + vamp:output examples:percussiononsets_output_detectionfunction . + + + diff -r c8b934ed4ddd -r f35bbb3e4d41 tests/test-jams-destinations/transforms/onsets.n3 --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/tests/test-jams-destinations/transforms/onsets.n3 Tue Sep 01 17:51:03 2015 +0100 @@ -0,0 +1,10 @@ +@prefix rdf: . +@prefix vamp: . +@prefix examples: . +@prefix : <#>. + +:transform0 a vamp:Transform; + vamp:plugin examples:percussiononsets. + + + diff -r c8b934ed4ddd -r f35bbb3e4d41 tests/test-jams-writer/test-jams-writer.sh --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/tests/test-jams-writer/test-jams-writer.sh Tue Sep 01 17:51:03 2015 +0100 @@ -0,0 +1,24 @@ +#!/bin/bash + +. ../include.sh + +tmpjson=$mypath/tmp_1_$$.json + +silentfile=$audiopath/20sec-silence.wav + +trap "rm -f $tmpjson" 0 + +transformdir=$mypath/transforms + +mandatory="-w jams" + +# This does not yet test for correct values, only for parseable json + +for output in instants curve-oss curve-fsr curve-fsr-timed curve-vsr grid-oss grid-fsr notes-regions; do + + $r -d "$testplug:$output" $mandatory --jams-one-file "$tmpjson" --jams-force "$silentfile" 2>/dev/null || \ + fail "Failed to run for plugin $testplug with output $output" + + check_json "$tmpjson" "test plugin output $output" +done + diff -r c8b934ed4ddd -r f35bbb3e4d41 tests/test-json-destinations/test-json-destinations.sh --- a/tests/test-json-destinations/test-json-destinations.sh Tue Sep 01 17:29:20 2015 +0100 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,175 +0,0 @@ -#!/bin/bash - -. ../include.sh - -infile1=$audiopath/3clicks8.wav -infile2=$audiopath/6clicks8.wav - -infile1dot=$audiopath/3.clicks.8.wav - -outfile1=3clicks8.json -outfile2=6clicks8.json - -outfile3=3clicks8_vamp_vamp-example-plugins_percussiononsets_onsets.json -outfile4=3clicks8_vamp_vamp-example-plugins_percussiononsets_detectionfunction.json -outfile5=6clicks8_vamp_vamp-example-plugins_percussiononsets_onsets.json -outfile6=6clicks8_vamp_vamp-example-plugins_percussiononsets_detectionfunction.json - -outfile1dot=3.clicks.8.json - -tmpjson=$mypath/tmp_1_$$.json - -trap "rm -f $tmpjson $outfile1 $outfile2 $outfile3 $outfile4 $outfile5 $outfile6 $infile1dot $outfile1dot $audiopath/$outfile1 $audiopath/$outfile2 $audiopath/$outfile3 $audiopath/$outfile4 $audiopath/$outfile5 $audiopath/$outfile6 $audiopath/$outfile1dot" 0 - -transformdir=$mypath/transforms - -mandatory="-w json --json-format jams" - - -ctx="onsets transform, one audio file, default JSON writer destination" - -rm -f $audiopath/$outfile1 - -$r -t $transformdir/onsets.n3 $mandatory $infile1 2>/dev/null || \ - fail "Fails to run with $ctx" - -check_json $audiopath/$outfile1 "$ctx" - - -ctx="onsets transform, one audio file with dots in filename, default JSON writer destination" - -rm -f $audiopath/$outfile1 - -cp $infile1 $infile1dot - -$r -t $transformdir/onsets.n3 $mandatory $infile1dot 2>/dev/null || \ - fail "Fails to run with $ctx" - -check_json $audiopath/$outfile1dot "$ctx" - -rm -f $infile1dot $audiopath/$outfile1dot - - -ctx="onsets and df transforms, one audio file, default JSON writer destination" - -rm -f $audiopath/$outfile1 - -$r -t $transformdir/onsets.n3 -t $transformdir/detectionfunction.n3 $mandatory $infile1 2>/dev/null || \ - fail "Fails to run with $ctx" - -check_json $audiopath/$outfile1 "$ctx" - - -ctx="onsets transform, two audio files, default JSON writer destination" - -rm -f $audiopath/$outfile1 -rm -f $audiopath/$outfile2 - -$r -t $transformdir/onsets.n3 $mandatory $infile1 $infile2 2>/dev/null || \ - fail "Fails to run with $ctx" - -check_json $audiopath/$outfile1 "$ctx" -check_json $audiopath/$outfile2 "$ctx" - - -ctx="onsets transform, two audio files, one-file JSON writer" - -$r -t $transformdir/onsets.n3 $mandatory --json-one-file $tmpjson $infile1 $infile2 2>/dev/null || \ - fail "Fails to run with $ctx" - -check_json $tmpjson "$ctx" - - -ctx="onsets transform, two audio files, stdout JSON writer" - -$r -t $transformdir/onsets.n3 $mandatory --json-stdout $infile1 $infile2 2>/dev/null >$tmpjson || \ - fail "Fails to run with $ctx" - -check_json $tmpjson "$ctx" - - -ctx="onsets transform, one audio file, many-files JSON writer" - -rm -f $audiopath/$outfile3 - -$r -t $transformdir/onsets.n3 $mandatory --json-many-files $infile1 2>/dev/null || \ - fail "Fails to run with $ctx" - -check_json $audiopath/$outfile3 "$ctx" - - -ctx="onsets transform, two audio files, many-files JSON writer" - -rm -f $audiopath/$outfile3 -rm -f $audiopath/$outfile5 - -$r -t $transformdir/onsets.n3 $mandatory --json-many-files $infile1 $infile2 2>/dev/null || \ - fail "Fails to run with $ctx" - -check_json $audiopath/$outfile3 "$ctx" -check_json $audiopath/$outfile5 "$ctx" - - -ctx="onsets and df transforms, two audio files, many-files JSON writer" - -rm -f $audiopath/$outfile3 -rm -f $audiopath/$outfile4 -rm -f $audiopath/$outfile5 -rm -f $audiopath/$outfile6 - -$r -t $transformdir/onsets.n3 -t $transformdir/detectionfunction.n3 $mandatory --json-many-files $infile1 $infile2 2>/dev/null || \ - fail "Fails to run with $ctx" - -check_json $audiopath/$outfile3 "$ctx" -check_json $audiopath/$outfile4 "$ctx" -check_json $audiopath/$outfile5 "$ctx" -check_json $audiopath/$outfile6 "$ctx" - - -ctx="output base directory" - -rm -f ./$outfile1 - -$r -t $transformdir/onsets.n3 -t $transformdir/detectionfunction.n3 $mandatory --json-basedir . $infile1 2>/dev/null || \ - fail "Fails to run with $ctx" - -check_json ./$outfile1 "$ctx" - - -ctx="output base directory and many-files" - -rm -f ./$outfile3 -rm -f ./$outfile5 - -$r -t $transformdir/onsets.n3 $mandatory --json-basedir . --json-many-files $infile1 $infile2 2>/dev/null || \ - fail "Fails to run with $ctx" - -check_json ./$outfile3 "$ctx" -check_json ./$outfile5 "$ctx" - - -ctx="nonexistent output base directory" - -$r -t $transformdir/onsets.n3 $mandatory --json-basedir ./DOES_NOT_EXIST $infile1 2>/dev/null && \ - fail "Fails with $ctx by completing successfully (should refuse and bail out)" - - -ctx="existing output file and no --json-force" - -touch $audiopath/$outfile1 - -$r -t $transformdir/onsets.n3 $mandatory $infile1 2>/dev/null && \ - fail "Fails by completing successfully when output file already exists (should refuse and bail out)" - - -ctx="existing output file and --json-force" - -touch $audiopath/$outfile1 - -$r -t $transformdir/onsets.n3 $mandatory --json-force $infile1 2>/dev/null || \ - fail "Fails to run with $ctx" - -check_json $audiopath/$outfile1 "$ctx" - - -exit 0 diff -r c8b934ed4ddd -r f35bbb3e4d41 tests/test-json-destinations/transforms/detectionfunction.n3 --- a/tests/test-json-destinations/transforms/detectionfunction.n3 Tue Sep 01 17:29:20 2015 +0100 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,11 +0,0 @@ -@prefix rdf: . -@prefix vamp: . -@prefix examples: . -@prefix : <#>. - -:transform0 a vamp:Transform; - vamp:plugin examples:percussiononsets ; - vamp:output examples:percussiononsets_output_detectionfunction . - - - diff -r c8b934ed4ddd -r f35bbb3e4d41 tests/test-json-destinations/transforms/onsets.n3 --- a/tests/test-json-destinations/transforms/onsets.n3 Tue Sep 01 17:29:20 2015 +0100 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,10 +0,0 @@ -@prefix rdf: . -@prefix vamp: . -@prefix examples: . -@prefix : <#>. - -:transform0 a vamp:Transform; - vamp:plugin examples:percussiononsets. - - - diff -r c8b934ed4ddd -r f35bbb3e4d41 tests/test-json-writer/test-json-writer.sh --- a/tests/test-json-writer/test-json-writer.sh Tue Sep 01 17:29:20 2015 +0100 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,24 +0,0 @@ -#!/bin/bash - -. ../include.sh - -tmpjson=$mypath/tmp_1_$$.json - -silentfile=$audiopath/20sec-silence.wav - -trap "rm -f $tmpjson" 0 - -transformdir=$mypath/transforms - -mandatory="-w json --json-format jams" - -# This does not yet test for correct values, only for parseable json - -for output in instants curve-oss curve-fsr curve-fsr-timed curve-vsr grid-oss grid-fsr notes-regions; do - - $r -d "$testplug:$output" $mandatory --json-one-file "$tmpjson" --json-force "$silentfile" 2>/dev/null || \ - fail "Failed to run for plugin $testplug with output $output" - - check_json "$tmpjson" "test plugin output $output" -done - diff -r c8b934ed4ddd -r f35bbb3e4d41 tests/test.sh --- a/tests/test.sh Tue Sep 01 17:29:20 2015 +0100 +++ b/tests/test.sh Tue Sep 01 17:51:03 2015 +0100 @@ -17,8 +17,8 @@ lab-destinations \ midi-writer \ midi-destinations \ - json-writer \ - json-destinations \ + jams-writer \ + jams-destinations \ summaries \ multiple-audio \ ; do