Notes on first meeting » History » Version 24

Gyorgy Fazekas, 2012-02-23 01:54 PM

1 6 Gyorgy Fazekas
These notes are based on our initial meeting on 24 January 2012. The aim was to collect some use cases and have an initial idea on what needs to be done to extend or revise the existing Audio Features Ontology.
2 1 Gyorgy Fazekas
3 14 Gyorgy Fazekas
h1. Topics discussed 
4 1 Gyorgy Fazekas
5 14 Gyorgy Fazekas
A rough list of topics discussed during the first meeting:
6 14 Gyorgy Fazekas
7 1 Gyorgy Fazekas
** What are the main research use cases for an Audio Features Ontology (AF) ?
8 2 Gyorgy Fazekas
** Are they served well by the existing AF ? 
9 1 Gyorgy Fazekas
** If not, what are the most important extensions we need to do?
10 1 Gyorgy Fazekas
** Does the fundamental structure of the ontology need to be changed?
11 1 Gyorgy Fazekas
** What is the relation of AF to existing software, including:
12 10 Gyorgy Fazekas
13 8 Gyorgy Fazekas
 * software like: Sonic Annotator, Sonic Visualiser, SAWA, AudioDB other tools... 
14 1 Gyorgy Fazekas
 * and projects like: OMRAS2, EASAIER, SALAMI, new Semantic Media/Semantic Audio grants...
15 7 Gyorgy Fazekas
** Personal Objectives: what are we going to do with a modified/re-engineered ontology?
16 1 Gyorgy Fazekas
17 2 Gyorgy Fazekas
18 4 Gyorgy Fazekas
h1. Use cases:
19 1 Gyorgy Fazekas
20 12 Gyorgy Fazekas
Use cases discussed so far:
21 12 Gyorgy Fazekas
22 12 Gyorgy Fazekas
+Thomas:+
23 1 Gyorgy Fazekas
  
24 1 Gyorgy Fazekas
** drive audio effects -> adaptive effect (controlling effects)
25 1 Gyorgy Fazekas
** KM like use case: association of audio effects and audio features e.g. pitch shifter won’t change onsets
26 1 Gyorgy Fazekas
** part of the AFX ontology
27 1 Gyorgy Fazekas
** more audio features
28 1 Gyorgy Fazekas
** technical classification of audio effects
29 1 Gyorgy Fazekas
30 1 Gyorgy Fazekas
31 12 Gyorgy Fazekas
+Steve:+
32 1 Gyorgy Fazekas
** Finding structure, repeated sequences of features
33 1 Gyorgy Fazekas
** Beat related stuff, BPM (tempo, major/minor is it an audio feature, not necessarilty)
34 1 Gyorgy Fazekas
** Chords => Chord ontology
35 1 Gyorgy Fazekas
** Melody and notes
36 1 Gyorgy Fazekas
37 1 Gyorgy Fazekas
38 12 Gyorgy Fazekas
+George:+
39 1 Gyorgy Fazekas
** Improve SAWA
40 1 Gyorgy Fazekas
** Facilitate the development of intelligent music production systems
41 1 Gyorgy Fazekas
** Release large content based metadata repositories in RDF
42 1 Gyorgy Fazekas
** Re-release the MSD in RDF (??)
43 1 Gyorgy Fazekas
** Deploy a knowledge based environment for content-based audio analysis based on the concept of the Knowledge Machine that can combine multiple modalities
44 1 Gyorgy Fazekas
** Research reproducibility using Ontologies as a model to exchange research data.
45 1 Gyorgy Fazekas
46 6 Gyorgy Fazekas
47 18 Gyorgy Fazekas
h1. Fundamental structure of the existing AF Ontology:
48 18 Gyorgy Fazekas
49 18 Gyorgy Fazekas
The Audio Features Ontology currently provides a core model which distinguishes between audio features based on two attributes:
50 18 Gyorgy Fazekas
51 18 Gyorgy Fazekas
# Temporal characteristics
52 18 Gyorgy Fazekas
# Data density
53 18 Gyorgy Fazekas
54 18 Gyorgy Fazekas
The first dichotomy allows for describing features either instantaneous events (e.g. note onsets, tempo change), or features with a known time duration (notes, structural segments, harmonic segments, the extent of an STFT or Chromagram frame).
55 18 Gyorgy Fazekas
56 18 Gyorgy Fazekas
The second dichotomy addresses a representational issue, and allows for describing how a feature relates to the extent of an audio file: 
57 18 Gyorgy Fazekas
** whether it is scattered and irregularly occurs during the course of a track (i.e. sparse),
58 18 Gyorgy Fazekas
** or occurs regularly and have a fixed duration (i.e. dense).
59 19 Gyorgy Fazekas
60 18 Gyorgy Fazekas
Alternative conceptualisations and some examples are summarised below:
61 18 Gyorgy Fazekas
62 18 Gyorgy Fazekas
!http://isophonics.net/sites/isophonics.net/files/FeatureConceptualisations.png!
63 18 Gyorgy Fazekas
*Fig 1.* Conceptualisations of content-based features.
64 18 Gyorgy Fazekas
65 18 Gyorgy Fazekas
The main scope of the existing ontology is to provide a framework for communication and to describe the association of features and audio signals. It does not classify features, describe their interrelationships or their computation. It deals with data density, and temporal characteristics only and differentiates between dense signal-like features of various dimensionality, (chromagrams, detection functions) and sparse features that are scattered across the signal timeline. This core model is shown in the following diagram:
66 18 Gyorgy Fazekas
67 18 Gyorgy Fazekas
!http://isophonics.net/sites/isophonics.net/files/AF_ontology_small.png!
68 18 Gyorgy Fazekas
*Fig. 2.* Core model of the existing Audio Features Ontology
69 18 Gyorgy Fazekas
70 21 Gyorgy Fazekas
h2. Example:
71 22 Gyorgy Fazekas
72 23 Gyorgy Fazekas
<pre>
73 23 Gyorgy Fazekas
<http://isophonics.net/sawa/audiofile/temp/AU775621fe> a mo:AudioFile ;
74 21 Gyorgy Fazekas
    dc:title """music-test.wav""" ;
75 21 Gyorgy Fazekas
    mo:encodes :signal_1.
76 21 Gyorgy Fazekas
77 21 Gyorgy Fazekas
:signal_1 a mo:Signal ;
78 21 Gyorgy Fazekas
    mo:time [
79 21 Gyorgy Fazekas
        a tl:Interval ;
80 21 Gyorgy Fazekas
        tl:onTimeLine :signal_timeline_1
81 21 Gyorgy Fazekas
    ] .
82 21 Gyorgy Fazekas
83 21 Gyorgy Fazekas
:signal_timeline_1 a tl:Timeline .
84 21 Gyorgy Fazekas
85 21 Gyorgy Fazekas
:event_2 a <http://purl.org/ontology/af/StructuralSegment> ;
86 21 Gyorgy Fazekas
    event:time [ 
87 21 Gyorgy Fazekas
        a tl:Interval ;
88 21 Gyorgy Fazekas
        tl:onTimeLine :signal_timeline_1 ;
89 21 Gyorgy Fazekas
        tl:at "PT19.600000000S"^^xsd:duration ;
90 21 Gyorgy Fazekas
        tl:duration "PT10.500000000S"^^xsd:duration ;
91 1 Gyorgy Fazekas
    ] ;
92 21 Gyorgy Fazekas
    af:feature "9" .
93 23 Gyorgy Fazekas
</pre>
94 21 Gyorgy Fazekas
95 21 Gyorgy Fazekas
96 1 Gyorgy Fazekas
h1. Open issues:
97 6 Gyorgy Fazekas
98 1 Gyorgy Fazekas
Some important questions to be decided on:
99 12 Gyorgy Fazekas
100 1 Gyorgy Fazekas
h2. Domain boundaries and scope:
101 1 Gyorgy Fazekas
102 19 Gyorgy Fazekas
** What is the ideal domain of an AF?
103 19 Gyorgy Fazekas
104 19 Gyorgy Fazekas
 * Are Musicological concepts outside the domain of an AF ?
105 19 Gyorgy Fazekas
 * How about Physical features: 
106 19 Gyorgy Fazekas
 ** Acoustic features, 
107 19 Gyorgy Fazekas
 ** Perceptual Features, 
108 19 Gyorgy Fazekas
 ** DSP type feature, 
109 19 Gyorgy Fazekas
 ** Musical Features (musically meaningful features related to acoustics) 
110 1 Gyorgy Fazekas
111 11 Gyorgy Fazekas
** The scope of the revised ontology may be:
112 1 Gyorgy Fazekas
113 1 Gyorgy Fazekas
 * Facilitate data-exchange for various purposes: (e.g. Linked Open Data, Research reproducibility, etc...)
114 17 Gyorgy Fazekas
 * Facilitate building intelligent/knowledge-based systems:
115 17 Gyorgy Fazekas
 ** How expressive the Ontology should be?
116 6 Gyorgy Fazekas
 ** What kind of reasoning services should be supported?
117 17 Gyorgy Fazekas
118 6 Gyorgy Fazekas
h2. What are the strength and weaknesses of the existing ontology?
119 17 Gyorgy Fazekas
120 20 Gyorgy Fazekas
* Does it serve us well?
121 20 Gyorgy Fazekas
* For example, loudness is defined as a segment in AF, and it does not fit a perceptual attribute well.
122 20 Gyorgy Fazekas
* What depth do we want ? (both in terms of scope and the level of detail) 
123 20 Gyorgy Fazekas
** do we want to describe feature extraction workflows using this or another ontology
124 20 Gyorgy Fazekas
* How AF relates to the DSP workflows used when extracting them?
125 1 Gyorgy Fazekas
126 1 Gyorgy Fazekas
127 18 Gyorgy Fazekas
h1. Existing resources :
128 2 Gyorgy Fazekas
129 18 Gyorgy Fazekas
h2. Some work related to Steve's use cases, segmentation and Ontologies:
130 2 Gyorgy Fazekas
131 2 Gyorgy Fazekas
** SALAMI Project: Kevin Page, DaveDeRoure http://salami.music.mcgill.ca/
132 2 Gyorgy Fazekas
** The Segment Ontology: http://users.ox.ac.uk/~oerc0033/preprints/admire2011.pdf
133 2 Gyorgy Fazekas
** PopStructure Ontology: Kurt Jacobson Unpublished. 
134 2 Gyorgy Fazekas
(Example available: http://wiki.musicontology.com/index.php/Structural_annotations_of_%22Can%27t_buy_me_love%22_by_the_Beatles) 
135 1 Gyorgy Fazekas
** Similarity Ontology: Kurt Jacobson http://grasstunes.net/ontology/musim/musim.html
136 1 Gyorgy Fazekas
137 2 Gyorgy Fazekas
138 18 Gyorgy Fazekas
h1. Ideas/resources for new Ontologies:
139 2 Gyorgy Fazekas
140 1 Gyorgy Fazekas
** Steve has worked on Acoustics related ontology
141 2 Gyorgy Fazekas
142 24 Gyorgy Fazekas
* Creating a DSP ontology:
143 24 Gyorgy Fazekas
* include processing steps down to math operations 
144 24 Gyorgy Fazekas
** this can take advantage to the log and math:namespaces in CWM: 
145 24 Gyorgy Fazekas
*** http://www.w3.org/DesignIssues/Notation3.html
146 24 Gyorgy Fazekas
*** http://markmail.org/download.xqy?id=6xj4qlauo442ymme&number=2
147 24 Gyorgy Fazekas
* describe common DSP parameters
148 1 Gyorgy Fazekas
149 24 Gyorgy Fazekas
* create an Acoustics Ontology
150 24 Gyorgy Fazekas
* describe Musicological concepts
151 24 Gyorgy Fazekas
* describe concepts related to cognitive and perceptual issues
152 1 Gyorgy Fazekas
153 1 Gyorgy Fazekas
154 2 Gyorgy Fazekas
h2. Currently missing features
155 1 Gyorgy Fazekas
156 1 Gyorgy Fazekas
** MFCC-s
157 1 Gyorgy Fazekas
** Rythmogram
158 1 Gyorgy Fazekas
** RMS energy
159 1 Gyorgy Fazekas
** combined features, e.g. weighted combinations or statistical averages over features
160 1 Gyorgy Fazekas
161 1 Gyorgy Fazekas
162 1 Gyorgy Fazekas
h2. Development issues
163 1 Gyorgy Fazekas
164 2 Gyorgy Fazekas
** chaining, combination, weighting
165 1 Gyorgy Fazekas
** how you associate features with arbitrary data
166 2 Gyorgy Fazekas
** summary feature types 
167 1 Gyorgy Fazekas
** SM (similarity matrix) are they part of the ontoogy?
168 1 Gyorgy Fazekas
** how to describe salience, can you hear it, can you perceive, is there an agreement
169 1 Gyorgy Fazekas
** how to describe weighting, confidence
170 1 Gyorgy Fazekas
** mood, music psychology, cognition, emotion, (perception ?)
171 1 Gyorgy Fazekas
** provenance => music provenance
172 1 Gyorgy Fazekas
** deprecation and versioning
173 1 Gyorgy Fazekas
174 18 Gyorgy Fazekas
175 1 Gyorgy Fazekas
h1. Objectives:
176 18 Gyorgy Fazekas
177 1 Gyorgy Fazekas
Long term goals and some concrete tasks that can be done as the outcome of the collaboration:
178 1 Gyorgy Fazekas
179 1 Gyorgy Fazekas
** A version of Sonic Annotator that produces output adhering the new ontology
180 2 Gyorgy Fazekas
** Are we making people happier by doing so?
181 1 Gyorgy Fazekas
** gradual transition period?
182 1 Gyorgy Fazekas
** extend other software toolkits; e.g. a verison of Marsyas in C++
183 1 Gyorgy Fazekas
** multitrack processing using Sonic Annotator (this feature might come along soon)
184 1 Gyorgy Fazekas
185 18 Gyorgy Fazekas
186 1 Gyorgy Fazekas
h2. Some immediate tasks before the next meeting:
187 2 Gyorgy Fazekas
188 1 Gyorgy Fazekas
** collect more resources 
189 2 Gyorgy Fazekas
** Verify the relationship between AF as is, and other feature/segmentation Ontologies
190 2 Gyorgy Fazekas
** what other software uses it?
191 2 Gyorgy Fazekas
** papers and literature review
192 1 Gyorgy Fazekas
** relation to projects e.g. SIEMAC
193 1 Gyorgy Fazekas
** collect features that we need
194 2 Gyorgy Fazekas
** define scope (extend the diagram of the set of ontologies: )
195 1 Gyorgy Fazekas
** collect specific application examples from existing processing chain / workflow
196 18 Gyorgy Fazekas
197 2 Gyorgy Fazekas
Collect software/projects that use/produce audio features:
198 18 Gyorgy Fazekas
199 18 Gyorgy Fazekas
** plugins (analysis, adaptive effects, adaptive synthesis)
200 18 Gyorgy Fazekas
** LADSPA, 
201 18 Gyorgy Fazekas
** VAMP, 
202 18 Gyorgy Fazekas
** Marsyas, 
203 18 Gyorgy Fazekas
** CLAM, 
204 18 Gyorgy Fazekas
** libextract, 
205 18 Gyorgy Fazekas
** COMirva, 
206 18 Gyorgy Fazekas
** MIRtoolbox, 
207 18 Gyorgy Fazekas
** Supercollider, 
208 18 Gyorgy Fazekas
** other frameworks
209 3 Gyorgy Fazekas
210 3 Gyorgy Fazekas
211 1 Gyorgy Fazekas
212 1 Gyorgy Fazekas
!http://www.isophonics.net/sites/isophonics.net/files/combined-frameworks.png!