Idyom » History » Version 47

Marcus Pearce, 2014-06-03 08:09 PM

1 11 Jeremy Gow
h1. Running IDyOM 
2 1 Marcus Pearce
3 11 Jeremy Gow
{{>toc}}
4 1 Marcus Pearce
5 44 Marcus Pearce
h2. <code>*idyom:idyom*</code> 
6 1 Marcus Pearce
7 12 Jeremy Gow
The main workhorse function is <code>idyom:idyom</code>, which has three required arguments and a number of optional keyword arguments.
8 1 Marcus Pearce
9 13 Jeremy Gow
h3. Required parameters
10 1 Marcus Pearce
11 23 Jeremy Gow
* @dataset-id@: a dataset id, e.g. 1.
12 23 Jeremy Gow
* @target-viewpoints@: a list of basic viewpoints to predict, e.g. '(:cpitch :bioi)
13 23 Jeremy Gow
* @source-viewpoints@: a list of viewpoints to use in prediction, e.g. '((:cpintfref :cpint) :bioi)
14 12 Jeremy Gow
** Passing <code>:select</code> will trigger viewpoint selection (see further options below)
15 12 Jeremy Gow
16 17 Jeremy Gow
See the [[List of viewpoints]] for a description of the various viewpoints available in IDyOM.  A simple call to IDyOM would be:
17 12 Jeremy Gow
<pre>
18 36 Marcus Pearce
CL-USER> (idyom:idyom 0 '(cpitch) '(cpitch))
19 15 Jeremy Gow
2.2490792
20 15 Jeremy Gow
(1.9049941 2.427845 2.0234334 1.7971386 1.8213106 1.9313766 2.3758402 1.8310248
21 14 Jeremy Gow
...
22 12 Jeremy Gow
</pre>
23 18 Jeremy Gow
This predicts the pitch values in dataset 1, based on previous pitches (cpitch) and pitch intervals (cpint).  IDyOM computes the information content for each note, and by default returns two values: the first is a mean note IC for the dataset, the second a list of mean note ICs for the individual compositions.  The first value is calculated as the mean of the second.
24 2 Marcus Pearce
25 13 Jeremy Gow
h3. Statistical modelling parameters
26 2 Marcus Pearce
27 19 Jeremy Gow
See "Pearce [2005, chapter 6]":http://webprojects.eecs.qmul.ac.uk/marcusp/papers/Pearce2005.pdf for further description and explanation of these parameters.
28 1 Marcus Pearce
29 19 Jeremy Gow
* @models@: the type of IDyOM model to use.  Options are:
30 25 Jeremy Gow
** @:stm@ - short-term model only, trained on the current composition.
31 25 Jeremy Gow
** @:ltm@ - long-term model only, trained on the pretraining and resampling training data.
32 25 Jeremy Gow
** @:ltm+@ - the long-term model, with additional incremental training on the test set;
33 19 Jeremy Gow
** @:both@ - a combination of :stm and :ltm;
34 19 Jeremy Gow
** @:both+@ -  a combination of :stm and :ltm+ (this is the default).
35 19 Jeremy Gow
36 27 Jeremy Gow
The LTM and STM can be configured using the @ltmo@ and @stmo@ parameters.  These accept a property list with the following properties - the defaults are used if a property is omitted or no parameter list is supplied:
37 1 Marcus Pearce
* @:order-bound@: an integer indicating the bound on the order of the model, i.e. the number of past events used by the model.  The default is @nil@, no bound.
38 1 Marcus Pearce
* @:mixtures@: whether to use mixtures for the model. (Default @t@).
39 1 Marcus Pearce
* @:update-exclusion@: whether to use update exclusion. (LTM default @nil@, STM default @t@.)
40 1 Marcus Pearce
* @:escape@: the model's escape method.  One of @:a :b :c :d :x@.  (LTM default @:c@, STM default @:x@.)
41 1 Marcus Pearce
42 1 Marcus Pearce
For example, the following command would combine the STM and LTM, without incremental training for the latter and an STM order bound of 4:
43 25 Jeremy Gow
<pre>
44 26 Jeremy Gow
CL-USER> (idyom:idyom 1 '(cpitch) '(cpitch) :models :both :stmo '(:order-bound 4))
45 25 Jeremy Gow
</pre>
46 1 Marcus Pearce
47 20 Jeremy Gow
h3. Training parameters
48 20 Jeremy Gow
49 20 Jeremy Gow
When using IDyOM to estimate note IC for a given dataset, the long-term models can be trained on other datasets (pretraining) and/or on the current dataset, i.e. via resampling (cross-validation).  In the latter case, the dataset is partitioned into a training set (used to train the LTMs) and a test set (for which note IC is computed).  This split is called a fold, and the modelling process can be repeated with a number of different folds in order to model the entire dataset.
50 20 Jeremy Gow
51 20 Jeremy Gow
* @pretraining-ids@: a list of dataset ids used to pretrain the long-term models (done before resampling).
52 20 Jeremy Gow
* @k@: the number of resampling (cross-validation) folds to use.  The default value is 10.
53 20 Jeremy Gow
** @1@ = no resampling, but also no training set unless the models are pretrained; 
54 20 Jeremy Gow
** @:full@ = as many folds as there are compositions in the dataset
55 20 Jeremy Gow
* @resampling-indices@: a list of numbers designating which resampling folds to use, i.e. a subset of @[0, 1, ..., k - 1]@.  By default, all folds are used.
56 2 Marcus Pearce
57 35 Marcus Pearce
*Note* that cross-validation only applies to the dataset being analysed (i.e., the one specified by the <code>dataset-id</code> argument). If a value of k=1 is supplied, the long-term models are not trained, unless a pretraining set is used.
58 35 Marcus Pearce
59 13 Jeremy Gow
h3. Viewpoint selection parameters
60 2 Marcus Pearce
61 24 Jeremy Gow
* @basis@: Identifies a set of viewpoints to be used in viewpoint selection, i.e. it will attempt to find the 'best' viewpoint system combining these, including by linking them.  The parameter can be a list or one of the following keywords:
62 46 Marcus Pearce
** @:pitch-full@ - The basis is a list of viewpoints useful for predicting pitch in Western music: cpitch, cpitch-class, tessitura, cpint, cpint-size, cpcint, cpcint-size, contour, newcontour, cpintfip, cpintfref, inscale.
63 46 Marcus Pearce
** @:pitch-short@ - A shorter version of the above: cpitch, cpitch-class, cpint, cpint-size, contour, newcontour.
64 46 Marcus Pearce
** @:bioi@ - For predicting Inter-Onset Interval (IOI): bioi, bioi-ratio, bioi-contour.
65 46 Marcus Pearce
** @:onset@ - For predicting onset: onset, ioi, ioi-ratio, ioi-contour, metaccent
66 24 Jeremy Gow
** @:auto@ - the basis is chosen to be the set of viewpoints that are defined in terms of one or more of the target viewpoints.  This is the default.
67 22 Jeremy Gow
* @dp@: the number of decimal places to use when comparing information contents in viewpoint selection.  Full floating point precision is used if this is @nil@ (the default)
68 22 Jeremy Gow
* @max-links@: the maximum number of links to use when creating linked viewpoints in viewpoint selection.  The default is 2.
69 2 Marcus Pearce
70 13 Jeremy Gow
h3. Output parameters
71 2 Marcus Pearce
72 2 Marcus Pearce
* output-path: a string indicating a directory in which to write the output 
73 3 Marcus Pearce
** output is only written to the console if this is <code>nil</code>
74 2 Marcus Pearce
* detail: an integer which determines how the information content is averaged in the output: 
75 1 Marcus Pearce
** 1: averaged over the entire dataset 
76 1 Marcus Pearce
** 2: and also averaged over each composition 
77 28 Jeremy Gow
** 3: and also with raw IC values for each event in each composition
78 2 Marcus Pearce
79 13 Jeremy Gow
h2. Examples
80 1 Marcus Pearce
81 13 Jeremy Gow
h3. Mean melody IC
82 1 Marcus Pearce
83 13 Jeremy Gow
To get mean information contents for each melody of dataset 0 in a list 
84 13 Jeremy Gow
85 1 Marcus Pearce
<pre>
86 37 Marcus Pearce
CL-USER> (idyom:idyom 0 '(cpitch) '(cpintfref cpint) :detail 2)
87 1 Marcus Pearce
2.493305
88 1 Marcus Pearce
(2.1368716 2.8534691 2.6938546 2.6491673 2.4993074 2.6098127 2.7728052 2.772861
89 1 Marcus Pearce
 2.5921957 2.905856 2.3591626 2.957503 2.4042292 2.7562473 2.3996017 2.8073587
90 1 Marcus Pearce
 2.114944 1.7434102 2.2310295 2.6374347 2.361792 1.9476132 2.501488 2.5472867
91 1 Marcus Pearce
 2.1056154 2.8225484 2.134257 2.9162033 3.0715692 2.9012227 2.7291088 2.866882
92 1 Marcus Pearce
 2.8795822 2.4571223 2.9277062 2.7861307 2.6623116 2.3304622 2.4217033
93 1 Marcus Pearce
 2.0556943 2.4048684 2.914848 2.7182267 3.0894585 2.873869 1.8821808 2.640174
94 1 Marcus Pearce
 2.8165438 2.5423129 2.3011856 3.1477294 2.655349 2.5216308 2.0667994 3.2579045
95 1 Marcus Pearce
 2.573013 2.6035044 2.202191 2.622113 2.2621205 2.3617425 2.7526956 2.3281655
96 1 Marcus Pearce
 2.9357266 2.3372407 3.1848125 2.67367 2.1906006 2.7835917 2.6332111 3.206142
97 1 Marcus Pearce
 2.1426969 2.194259 2.415167 1.9769101 2.0870917 2.7844474 2.2373738 2.772138
98 1 Marcus Pearce
 2.9702199 1.724408 2.473073 2.2464263 2.2452457 2.688889 2.6299863 2.2223835
99 1 Marcus Pearce
 2.8082614 2.673671 2.7693706 2.3369458 2.5016947 2.3837066 2.3682225 2.795649
100 1 Marcus Pearce
 2.9063463 2.5880773 2.0457468 1.8635312 2.4522712 1.5877498 2.8802161
101 1 Marcus Pearce
 2.7988417 2.3125513 1.7245895 2.2404804 2.1694546 2.365556 1.5905867 1.3827317
102 1 Marcus Pearce
 2.2706041 3.023884 2.2864542 2.1259797 2.713626 2.1967313 2.5721254 2.5812547
103 1 Marcus Pearce
 2.8233812 2.3134546 2.6203637 2.945946 2.601433 2.1920888 2.3732007 2.440137
104 1 Marcus Pearce
 2.4291563 2.3676903 2.734724 3.0283954 2.8076048 2.7796154 2.326931 2.1779459
105 1 Marcus Pearce
 2.2570527 2.2688026 1.3976555 2.030298 2.640235 2.568248 2.6338177 2.157162
106 1 Marcus Pearce
 2.3915367 2.7873137 2.3088667 2.2176988 2.4402564 2.8062992 2.784044 2.4296925
107 1 Marcus Pearce
 2.3520193 2.6146257)
108 1 Marcus Pearce
</pre>
109 1 Marcus Pearce
110 13 Jeremy Gow
h3. Write note IC to file
111 1 Marcus Pearce
112 13 Jeremy Gow
To write the information contents for each note of each melody in dataset 0 to a file 
113 13 Jeremy Gow
114 1 Marcus Pearce
<pre>
115 38 Marcus Pearce
CL-USER> (idyom:idyom 0 '(cpitch) '(cpintfref cpint)) :detail 3 :output-path "/tmp/")
116 1 Marcus Pearce
</pre>
117 1 Marcus Pearce
118 43 Marcus Pearce
h3. Viewpoint Selection
119 43 Marcus Pearce
120 43 Marcus Pearce
<pre>
121 47 Marcus Pearce
CL-USER> (idyom:idyom 17 '(cpitch) :select :models :stm :dp 3)
122 43 Marcus Pearce
Selecting viewpoints for the STM model on dataset 17 predicting viewpoints (CPITCH).
123 43 Marcus Pearce
Generating candidate viewpoints from: (CPITCH CPITCH-CLASS CPINT
124 43 Marcus Pearce
                                       CPINT-SIZE CONTOUR NEWCONTOUR)
125 43 Marcus Pearce
Max. links 2, whitelist (ANY), blacklist NIL
126 43 Marcus Pearce
Candidate viewpoints: (CPITCH CPITCH-CLASS CPINT CPINT-SIZE CONTOUR
127 43 Marcus Pearce
                       NEWCONTOUR (CONTOUR NEWCONTOUR)
128 43 Marcus Pearce
                       (CPINT-SIZE NEWCONTOUR) (CPINT-SIZE CONTOUR)
129 43 Marcus Pearce
                       (CPINT NEWCONTOUR) (CPINT CONTOUR)
130 43 Marcus Pearce
                       (CPINT CPINT-SIZE) (CPITCH-CLASS NEWCONTOUR)
131 43 Marcus Pearce
                       (CPITCH-CLASS CONTOUR) (CPITCH-CLASS CPINT-SIZE)
132 43 Marcus Pearce
                       (CPITCH-CLASS CPINT) (CPITCH NEWCONTOUR)
133 43 Marcus Pearce
                       (CPITCH CONTOUR) (CPITCH CPINT-SIZE) (CPITCH CPINT)
134 43 Marcus Pearce
                       (CPITCH CPITCH-CLASS))
135 43 Marcus Pearce
136 43 Marcus Pearce
Selected system NIL, mean IC = NIL
137 43 Marcus Pearce
138 43 Marcus Pearce
Selected system ((CPITCH-CLASS CONTOUR)), mean IC = 3.0302427
139 43 Marcus Pearce
 =======================================================================================
140 43 Marcus Pearce
The selected viewpoint system with a mean IC of 3.0302427 is ((CPITCH-CLASS
141 43 Marcus Pearce
                                                               CONTOUR))
142 43 Marcus Pearce
3.0302427
143 43 Marcus Pearce
(3.169925 3.169925 3.0849624 3.0849624 2.9886398 2.9886398 2.8774438 2.8774438)
144 43 Marcus Pearce
((3.169925 3.169925) (3.169925 3.169925) (3.169925 3.0) (3.169925 3.0)
145 43 Marcus Pearce
 (3.169925 2.807355) (3.169925 2.807355) (3.169925 2.5849626)
146 43 Marcus Pearce
 (3.169925 2.5849626))
147 43 Marcus Pearce
</pre>
148 43 Marcus Pearce
149 43 Marcus Pearce
150 13 Jeremy Gow
h3. Conklin & Witten (1995)
151 13 Jeremy Gow
152 13 Jeremy Gow
To simulate the experiments of Conklin & Witten (1995) 
153 1 Marcus Pearce
154 1 Marcus Pearce
<pre>
155 45 Marcus Pearce
CL-USER> (idyom:conkwit95)
156 1 Marcus Pearce
Simulation of the experiments of Conklin & Witten (1995, Table 4).
157 1 Marcus Pearce
System 1; Mean Information Content: 2.33 
158 1 Marcus Pearce
System 2; Mean Information Content: 2.36 
159 1 Marcus Pearce
System 3; Mean Information Content: 2.09 
160 1 Marcus Pearce
System 4; Mean Information Content: 2.01 
161 1 Marcus Pearce
System 5; Mean Information Content: 2.08 
162 1 Marcus Pearce
System 6; Mean Information Content: 1.90 
163 1 Marcus Pearce
System 7; Mean Information Content: 1.88 
164 1 Marcus Pearce
System 8; Mean Information Content: 1.86 
165 1 Marcus Pearce
NIL
166 1 Marcus Pearce
</pre>
167 1 Marcus Pearce
168 1 Marcus Pearce
Compare with "Conklin & Witten [1995, JNMR, table 4]":http://www.sc.ehu.es/ccwbayes/members/conklin/papers/jnmr95.pdf