gyorgyf@2: These notes are based on our initial meeting on 24 January 2012. The aim was to collect some use cases and have an initial idea on what needs to be done to extend or revise the existing Audio Features Ontology.
gyorgyf@2: 
gyorgyf@2: h1. Topics discussed 
gyorgyf@2: 
gyorgyf@2: A rough list of topics discussed during the first meeting:
gyorgyf@2: 
gyorgyf@2: ** What are the main research use cases for an Audio Features Ontology (AF) ?
gyorgyf@2: ** Are they served well by the existing AF ? 
gyorgyf@2: ** If not, what are the most important extensions we need to do?
gyorgyf@2: ** Does the fundamental structure of the ontology need to be changed?
gyorgyf@2: ** What is the relation of AF to existing software, including:
gyorgyf@2: 
gyorgyf@2:  * software like: Sonic Annotator, Sonic Visualiser, SAWA, AudioDB other tools... 
gyorgyf@2:  * and projects like: OMRAS2, EASAIER, SALAMI, new Semantic Media/Semantic Audio grants...
gyorgyf@2: ** Personal Objectives: what are we going to do with a modified/re-engineered ontology?
gyorgyf@2: 
gyorgyf@2: 
gyorgyf@2: h1. Use cases:
gyorgyf@2: 
gyorgyf@2: Use cases discussed so far:
gyorgyf@2: 
gyorgyf@2: +Thomas:+
gyorgyf@2:   
gyorgyf@2: ** drive audio effects -> adaptive effect (controlling effects)
gyorgyf@2: ** KM like use case: association of audio effects and audio features e.g. pitch shifter won’t change onsets
gyorgyf@2: ** part of the AFX ontology
gyorgyf@2: ** more audio features
gyorgyf@2: ** technical classification of audio effects
gyorgyf@2: 
gyorgyf@2: 
gyorgyf@2: +Steve:+
gyorgyf@2: ** Finding structure, repeated sequences of features
gyorgyf@2: ** Beat related stuff, BPM (tempo, major/minor is it an audio feature)
gyorgyf@2: ** Chords, Chord sequences => Chord ontology
gyorgyf@2: ** Melody and notes
gyorgyf@2: 
gyorgyf@2: 
gyorgyf@2: +George:+
gyorgyf@2: ** Improve SAWA
gyorgyf@2: ** Facilitate the development of intelligent music production systems
gyorgyf@2: ** Release large content based metadata repositories in RDF
gyorgyf@2: ** Re-release the MSD in RDF (??)
gyorgyf@2: ** Deploy a knowledge based environment for content-based audio analysis based on the concept of the Knowledge Machine that can combine multiple modalities
gyorgyf@2: ** Research reproducibility using Ontologies as a model to exchange research data.
gyorgyf@2: 
gyorgyf@2: 
gyorgyf@2: h1. Fundamental structure of the existing AF Ontology:
gyorgyf@2: 
gyorgyf@2: The Audio Features Ontology currently provides a core model which distinguishes between audio features based on two attributes:
gyorgyf@2: 
gyorgyf@2: # Temporal characteristics
gyorgyf@2: # Data density
gyorgyf@2: 
gyorgyf@2: The first dichotomy allows for describing features either instantaneous events (e.g. note onsets, tempo change), or features with a known time duration (notes, structural segments, harmonic segments, the extent of an STFT or Chromagram frame).
gyorgyf@2: 
gyorgyf@2: The second dichotomy addresses a representational issue, and allows for describing how a feature relates to the extent of an audio file: 
gyorgyf@2: ** whether it is scattered and irregularly occurs during the course of a track (i.e. sparse),
gyorgyf@2: ** or occurs regularly and have a fixed duration (i.e. dense).
gyorgyf@2: 
gyorgyf@2: Alternative conceptualisations and some examples are summarised below:
gyorgyf@2: 
gyorgyf@2: !http://isophonics.net/sites/isophonics.net/files/FeatureConceptualisations.png!
gyorgyf@2: *Fig 1.* Conceptualisations of content-based features.
gyorgyf@2: 
gyorgyf@2: The main scope of the existing ontology is to provide a framework for communication and to describe the association of features and audio signals. It does not classify features, describe their interrelationships or their computation. It deals with data density, and temporal characteristics only and differentiates between dense signal-like features of various dimensionality, (chromagrams, detection functions) and sparse features that are scattered across the signal timeline. This core model is shown in the following diagram:
gyorgyf@2: 
gyorgyf@2: !http://isophonics.net/sites/isophonics.net/files/AF_ontology_small.png!
gyorgyf@2: *Fig. 2.* Core model of the existing Audio Features Ontology
gyorgyf@2: 
gyorgyf@2: h2. RDF Example:
gyorgyf@2: 
gyorgyf@2: AF heavily relies on the event and timeline ontologies to refer to event in time and timelines corresponding to the duration of an audio signal or a dense signal-like feature. Here's an RDF example produced by SAWA/Sonic Annotator describing temporal segments using the ontology:
gyorgyf@2: 
gyorgyf@2: <pre>
gyorgyf@2: <http://isophonics.net/sawa/audiofile/temp/AU775621fe> a mo:AudioFile ;
gyorgyf@2:     dc:title """music-test.wav""" ;
gyorgyf@2:     mo:encodes :signal_1.
gyorgyf@2: 
gyorgyf@2: :signal_1 a mo:Signal ;
gyorgyf@2:     mo:time [
gyorgyf@2:         a tl:Interval ;
gyorgyf@2:         tl:onTimeLine :signal_timeline_1
gyorgyf@2:     ] .
gyorgyf@2: 
gyorgyf@2: :signal_timeline_1 a tl:Timeline .
gyorgyf@2: 
gyorgyf@2: :event_2 a <http://purl.org/ontology/af/StructuralSegment> ;
gyorgyf@2:     event:time [ 
gyorgyf@2:         a tl:Interval ;
gyorgyf@2:         tl:onTimeLine :signal_timeline_1 ;
gyorgyf@2:         tl:at "PT19.600000000S"^^xsd:duration ;
gyorgyf@2:         tl:duration "PT10.500000000S"^^xsd:duration ;
gyorgyf@2:     ] ;
gyorgyf@2:     af:feature "9" .
gyorgyf@2: </pre>
gyorgyf@2: 
gyorgyf@2: 
gyorgyf@2: h1. Open issues:
gyorgyf@2: 
gyorgyf@2: Some important questions to be decided on:
gyorgyf@2: 
gyorgyf@2: h2. Domain boundaries and scope:
gyorgyf@2: 
gyorgyf@2: ** What is the ideal domain a revised AF?
gyorgyf@2: 
gyorgyf@2:  * Are Musicological concepts outside the domain of an AF ?
gyorgyf@2:  * How about Physical features: 
gyorgyf@2:  ** Acoustic features, 
gyorgyf@2:  ** Perceptual Features, 
gyorgyf@2:  ** DSP type feature, 
gyorgyf@2:  ** Musical Features (musically meaningful features related to acoustics) 
gyorgyf@2: 
gyorgyf@2: ** The scope of the revised ontology may be:
gyorgyf@2: 
gyorgyf@2:  * Facilitate data-exchange for various purposes: (e.g. Linked Open Data, Research reproducibility, etc...)
gyorgyf@2:  * Facilitate building intelligent/knowledge-based systems:
gyorgyf@2:  ** How expressive the Ontology should be?
gyorgyf@2:  ** What kind of reasoning services should be supported?
gyorgyf@2: 
gyorgyf@2: h2. What are the strength and weaknesses of the existing ontology?
gyorgyf@2: 
gyorgyf@2: * Does it serve us well?
gyorgyf@2: * For example, loudness is defined as a segment in AF, and it does not fit a perceptual attribute well.
gyorgyf@2: * What depth do we want ? (both in terms of scope and the level of detail) 
gyorgyf@2: ** do we want to describe feature extraction workflows using this or another ontology
gyorgyf@2: * How AF relates to the DSP workflows used when extracting them?
gyorgyf@2: 
gyorgyf@2: 
gyorgyf@2: h1. Existing resources :
gyorgyf@2: 
gyorgyf@2: h2. Some work related to Steve's use cases, segmentation and Ontologies:
gyorgyf@2: 
gyorgyf@2: ** SALAMI Project: Kevin Page, DaveDeRoure http://salami.music.mcgill.ca/
gyorgyf@2: ** The Segment Ontology: http://users.ox.ac.uk/~oerc0033/preprints/admire2011.pdf
gyorgyf@2: ** PopStructure Ontology: Kurt Jacobson Unpublished. 
gyorgyf@2: (Example available: http://wiki.musicontology.com/index.php/Structural_annotations_of_%22Can%27t_buy_me_love%22_by_the_Beatles) 
gyorgyf@2: ** Similarity Ontology: Kurt Jacobson http://grasstunes.net/ontology/musim/musim.html
gyorgyf@2: 
gyorgyf@2: 
gyorgyf@2: h1. Ideas/resources for new Ontologies:
gyorgyf@2: 
gyorgyf@2: ** Steve has worked on Acoustics related ontology
gyorgyf@2: 
gyorgyf@2: * Creating a DSP ontology:
gyorgyf@2: * include processing steps down to math operations 
gyorgyf@2: ** this can take advantage to the log and math:namespaces in CWM: 
gyorgyf@2: *** http://www.w3.org/DesignIssues/Notation3.html
gyorgyf@2: *** http://markmail.org/download.xqy?id=6xj4qlauo442ymme&number=2
gyorgyf@2: * describe common DSP parameters
gyorgyf@2: 
gyorgyf@2: * create an Acoustics Ontology
gyorgyf@2: * describe Musicological concepts
gyorgyf@2: * describe concepts related to cognitive and perceptual issues
gyorgyf@2: 
gyorgyf@2: 
gyorgyf@2: h2. Currently missing features
gyorgyf@2: 
gyorgyf@2: ** MFCC-s
gyorgyf@2: ** Rythmogram
gyorgyf@2: ** RMS energy
gyorgyf@2: ** combined features, e.g. weighted combinations or statistical averages over features
gyorgyf@2: 
gyorgyf@2: 
gyorgyf@2: h2. Development issues
gyorgyf@2: 
gyorgyf@2: ** chaining, combination, weighting
gyorgyf@2: ** how you associate features with arbitrary data
gyorgyf@2: ** summary feature types 
gyorgyf@2: ** SM (similarity matrix) are they part of the ontoogy?
gyorgyf@2: ** how to describe salience, can you hear it, can you perceive, is there an agreement
gyorgyf@2: ** how to describe weighting, confidence
gyorgyf@2: ** mood, music psychology, cognition, emotion, (perception ?)
gyorgyf@2: ** provenance => music provenance
gyorgyf@2: ** deprecation and versioning
gyorgyf@2: 
gyorgyf@2: 
gyorgyf@2: h1. Objectives:
gyorgyf@2: 
gyorgyf@2: Long term goals and some concrete tasks that can be done as the outcome of the collaboration:
gyorgyf@2: 
gyorgyf@2: ** A version of Sonic Annotator that produces output adhering the new ontology
gyorgyf@2: ** Are we making people happier by doing so?
gyorgyf@2: ** gradual transition period?
gyorgyf@2: ** extend other software toolkits; e.g. a verison of Marsyas in C++
gyorgyf@2: ** multitrack processing using Sonic Annotator (this feature might come along soon)
gyorgyf@2: 
gyorgyf@2: 
gyorgyf@2: h2. Some immediate tasks before the next meeting:
gyorgyf@2: 
gyorgyf@2: ** collect more resources 
gyorgyf@2: ** Verify the relationship between AF as is, and other feature/segmentation Ontologies
gyorgyf@2: ** what other software uses it?
gyorgyf@2: ** papers and literature review
gyorgyf@2: ** relation to projects e.g. SIEMAC
gyorgyf@2: ** collect features that we need
gyorgyf@2: ** define scope (extend the diagram of the set of ontologies: )
gyorgyf@2: ** collect specific application examples from existing processing chain / workflow
gyorgyf@2: 
gyorgyf@2: Collect software/projects that use/produce audio features:
gyorgyf@2: 
gyorgyf@2: ** plugins (analysis, adaptive effects, adaptive synthesis)
gyorgyf@2: ** LADSPA, 
gyorgyf@2: ** VAMP, 
gyorgyf@2: ** Marsyas, 
gyorgyf@2: ** CLAM, 
gyorgyf@2: ** libextract, 
gyorgyf@2: ** COMirva, 
gyorgyf@2: ** MIRtoolbox, 
gyorgyf@2: ** Supercollider, 
gyorgyf@2: ** other frameworks
gyorgyf@2: 
gyorgyf@2: 
gyorgyf@2: 
gyorgyf@2: !http://www.isophonics.net/sites/isophonics.net/files/combined-frameworks.png!