Notes on first meeting » History » Version 6

Gyorgy Fazekas, 2012-02-23 09:55 AM

1 6 Gyorgy Fazekas
These notes are based on our initial meeting on 24 January 2012. The aim was to collect some use cases and have an initial idea on what needs to be done to extend or revise the existing Audio Features Ontology.
2 1 Gyorgy Fazekas
3 4 Gyorgy Fazekas
h1. Topics discussed (roughly)
4 1 Gyorgy Fazekas
5 1 Gyorgy Fazekas
** What are the main research use cases for an Audio Features Ontology (AF) ?
6 2 Gyorgy Fazekas
** Are they served well by the existing AF ? 
7 1 Gyorgy Fazekas
** If not, what are the most important extensions we need to do?
8 1 Gyorgy Fazekas
** Does the fundamental structure of the ontology need to be changed?
9 1 Gyorgy Fazekas
** Personal Objectives: what are we going to do with a modified/re-engineered ontology?
10 1 Gyorgy Fazekas
** What is the relation of AF to existing software, including:
11 1 Gyorgy Fazekas
12 5 Gyorgy Fazekas
 * software like: Sonic Annotator, SV, SAWA, AudioDB other tools... 
13 6 Gyorgy Fazekas
 * and projects like: OMRAS2, EASAIER, SALAMI, new Semantic Media/Semantic Audio grants...
14 1 Gyorgy Fazekas
15 2 Gyorgy Fazekas
16 4 Gyorgy Fazekas
h1. Use cases:
17 1 Gyorgy Fazekas
18 1 Gyorgy Fazekas
++Thomas:
19 1 Gyorgy Fazekas
  
20 1 Gyorgy Fazekas
** drive audio effects -> adaptive effect (controlling effects)
21 1 Gyorgy Fazekas
** KM like use case: association of audio effects and audio features e.g. pitch shifter won’t change onsets
22 1 Gyorgy Fazekas
** part of the AFX ontology
23 1 Gyorgy Fazekas
** more audio features
24 1 Gyorgy Fazekas
** technical classification of audio effects
25 1 Gyorgy Fazekas
26 1 Gyorgy Fazekas
27 1 Gyorgy Fazekas
++Steve:
28 1 Gyorgy Fazekas
** Finding structure, repeated sequences of features
29 1 Gyorgy Fazekas
** Beat related stuff, BPM (tempo, major/minor is it an audio feature, not necessarilty)
30 1 Gyorgy Fazekas
** Chords => Chord ontology
31 1 Gyorgy Fazekas
** Melody and notes
32 1 Gyorgy Fazekas
33 1 Gyorgy Fazekas
34 1 Gyorgy Fazekas
++George:
35 1 Gyorgy Fazekas
** Improve SAWA
36 1 Gyorgy Fazekas
** Facilitate the development of intelligent music production systems
37 1 Gyorgy Fazekas
** Release large content based metadata repositories in RDF
38 1 Gyorgy Fazekas
** Re-release the MSD in RDF (??)
39 1 Gyorgy Fazekas
** Deploy a knowledge based environment for content-based audio analysis based on the concept of the Knowledge Machine that can combine multiple modalities
40 1 Gyorgy Fazekas
** Research reproducibility using Ontologies as a model to exchange research data.
41 1 Gyorgy Fazekas
42 1 Gyorgy Fazekas
43 6 Gyorgy Fazekas
h1. Open issues:
44 1 Gyorgy Fazekas
45 6 Gyorgy Fazekas
Some important questions to be decided on:
46 1 Gyorgy Fazekas
47 6 Gyorgy Fazekas
h2. Domain and scope:
48 6 Gyorgy Fazekas
49 1 Gyorgy Fazekas
** Are Musicological concepts outside the scope of an AF ?
50 1 Gyorgy Fazekas
** Physical features : Acoustic features, Perceptual Features, DSP type, Musical Features
51 1 Gyorgy Fazekas
52 6 Gyorgy Fazekas
h2. Fundamental structure of existing AF:
53 1 Gyorgy Fazekas
54 6 Gyorgy Fazekas
The Audio Features Ontology is currently based on the ...
55 6 Gyorgy Fazekas
56 6 Gyorgy Fazekas
The main scope of the ontology is to provide a framework for communication, feature representation, and describe the association of features and audio signals. Therefore it does not classify features, describe their interrelationships or their computation. With re- gards to the different conceptualisations of feature representations presented in table 4.2 (see §4.2.5.1), the Audio Features Ontology deals with data density, and temporal characteristics. It differentiates between dense, signal-like features of various dimensionality, for instance a chromagrams and detection functions, and sparse features that are scattered across the signal timeline, for instance, notes, or onsets.
57 6 Gyorgy Fazekas
58 6 Gyorgy Fazekas
** Does it serve us well?
59 1 Gyorgy Fazekas
** For example, loudness is defined as a segment in AF, and it does not fit a perceptual attribute well.
60 1 Gyorgy Fazekas
** What depth do we want ? (both in terms of scope and the level of detail in describing a feature extraction workflow)
61 1 Gyorgy Fazekas
** How AF relates to the DSP workflows used in extracting them?
62 1 Gyorgy Fazekas
63 1 Gyorgy Fazekas
64 1 Gyorgy Fazekas
h2. Existing resources :
65 1 Gyorgy Fazekas
66 2 Gyorgy Fazekas
h3. Some work related to Steve's use cases, segmentation and Ontologies:
67 1 Gyorgy Fazekas
68 2 Gyorgy Fazekas
** SALAMI Project: Kevin Page, DaveDeRoure http://salami.music.mcgill.ca/
69 2 Gyorgy Fazekas
** The Segment Ontology: http://users.ox.ac.uk/~oerc0033/preprints/admire2011.pdf
70 2 Gyorgy Fazekas
** PopStructure Ontology: Kurt Jacobson Unpublished. 
71 2 Gyorgy Fazekas
(Example available: http://wiki.musicontology.com/index.php/Structural_annotations_of_%22Can%27t_buy_me_love%22_by_the_Beatles) 
72 2 Gyorgy Fazekas
** Similarity Ontology: Kurt Jacobson http://grasstunes.net/ontology/musim/musim.html
73 1 Gyorgy Fazekas
74 1 Gyorgy Fazekas
75 2 Gyorgy Fazekas
h2. Ideas/resources for new Ontologies:
76 1 Gyorgy Fazekas
77 2 Gyorgy Fazekas
** Steve has worked on Acoustics related ontology
78 1 Gyorgy Fazekas
79 2 Gyorgy Fazekas
** Creating a DSP ontology:
80 2 Gyorgy Fazekas
** include processing steps down to math operations 
81 2 Gyorgy Fazekas
  (this can take advantage to the math:namespace in CWM: http://www.w3.org/DesignIssues/Notation3.html)
82 2 Gyorgy Fazekas
** describe common DSP parameters
83 2 Gyorgy Fazekas
84 2 Gyorgy Fazekas
** create an Acoustics Ontology
85 2 Gyorgy Fazekas
** describe Musicological concepts
86 2 Gyorgy Fazekas
** describe concepts related to cognitive and perceptual issues
87 2 Gyorgy Fazekas
88 2 Gyorgy Fazekas
89 1 Gyorgy Fazekas
h2. Currently missing features
90 1 Gyorgy Fazekas
91 2 Gyorgy Fazekas
** MFCC-s
92 1 Gyorgy Fazekas
** Rythmogram
93 1 Gyorgy Fazekas
** RMS energy
94 2 Gyorgy Fazekas
** combined features, e.g. weighted combinations or statistical averages over features
95 1 Gyorgy Fazekas
96 1 Gyorgy Fazekas
97 1 Gyorgy Fazekas
h2. Development issues
98 1 Gyorgy Fazekas
99 1 Gyorgy Fazekas
** chaining, combination, weighting
100 1 Gyorgy Fazekas
** how you associate features with arbitrary data
101 2 Gyorgy Fazekas
** summary feature types 
102 1 Gyorgy Fazekas
** SM (similarity matrix) are they part of the ontoogy?
103 2 Gyorgy Fazekas
** how to describe salience, can you hear it, can you perceive, is there an agreement
104 2 Gyorgy Fazekas
** how to describe weighting, confidence
105 1 Gyorgy Fazekas
** mood, music psychology, cognition, emotion, (perception ?)
106 1 Gyorgy Fazekas
** provenance => music provenance
107 2 Gyorgy Fazekas
** deprecation and versioning
108 1 Gyorgy Fazekas
109 1 Gyorgy Fazekas
110 1 Gyorgy Fazekas
h2. Long term objectives:
111 1 Gyorgy Fazekas
112 1 Gyorgy Fazekas
Some concrete tasks that can be done as the outcome of the collaboration:
113 1 Gyorgy Fazekas
114 2 Gyorgy Fazekas
** A version of Sonic Annotator that produces output adhering the new ontology
115 1 Gyorgy Fazekas
** Are we making people happier by doing so?
116 2 Gyorgy Fazekas
** gradual transition period?
117 2 Gyorgy Fazekas
** extend other software toolkits; e.g. a verison of Marsyas in C++
118 2 Gyorgy Fazekas
** multitrack processing using Sonic Annotator (this feature might come along soon)
119 1 Gyorgy Fazekas
120 1 Gyorgy Fazekas
121 2 Gyorgy Fazekas
h2. Some immediate tasks (TODO):
122 1 Gyorgy Fazekas
123 2 Gyorgy Fazekas
** collect more resources 
124 2 Gyorgy Fazekas
** Verify the relationship between AF as is, and other feature/segmentation Ontologies
125 2 Gyorgy Fazekas
** what other software uses it?
126 2 Gyorgy Fazekas
** papers and literature review
127 2 Gyorgy Fazekas
** relation to projects e.g. SIEMAC
128 2 Gyorgy Fazekas
** collect features that we need
129 2 Gyorgy Fazekas
** define scope (extend the diagram of the set of ontologies: )
130 2 Gyorgy Fazekas
** collect specific application examples from existing processing chain / workflow
131 1 Gyorgy Fazekas
132 2 Gyorgy Fazekas
collect software/projects that use/produce audio features:
133 1 Gyorgy Fazekas
134 1 Gyorgy Fazekas
** plugins, LADSPA, VAMP, Marsyas, CLAM, libextract, COMirva, MIRtoolbox, Supercollider, other frameworks
135 3 Gyorgy Fazekas
136 3 Gyorgy Fazekas
137 3 Gyorgy Fazekas
!http://www.isophonics.net/sites/isophonics.net/files/combined-frameworks.png!