IDyOM Project

h1. Running IDyOM 

{{>toc}}

h2. <code>*idyom:idyom*</code> 

The top-level point of entry is <code>idyom:idyom</code>, which has three required arguments and a number of optional keyword arguments.

h3. Required parameters

* @dataset-id@: a dataset id, (an integer, e.g., 0)

* @target-viewpoints@: a list of basic viewpoints to predict, e.g. '(cpitch) or '(cpitch onset)

* @source-viewpoints@: a list of viewpoints to use in prediction, e.g. '((cpintfref cpint) ioi) to use two viewpoints: cpintfref linked with cpint and ioi; or '(cpintfref cpint ioi) to use three unlinked viewpoints: cpintfref, cpint, and ioi

** Note that passing <code>:select</code> will trigger viewpoint selection (see further options below)

See the [[List of viewpoints]] for a description of the various viewpoints available in IDyOM.  

A simple call to IDyOM would be:

<pre>

CL-USER> (idyom:idyom 18 '(cpitch) '(cpitch))

3.4519181

(3.7459166 3.7913148 3.5499783 3.2027783 3.5338237 3.4903128 3.4759412 2.8252819)

((3.586735 3.8160942 6.8622913 3.3496706 3.3494937 2.0084002 2.4969249 6.04035

  2.5135455 3.611938 5.617945 4.6973023 3.1194212 4.153834 2.328312 2.863982

  3.5802953 2.198141 4.9777308)

 (3.7181098 6.350471 4.8110385 4.712717 3.4230165 3.5964172 1.7951053 1.3934463

  5.252618 3.8527348 3.8151293 1.9286131 6.499048 4.1175685 1.8474439

  3.5475667)

 (4.031454 2.9693763 4.0899825 2.9261 1.24961 4.785756 2.1267474 2.8533628

  3.0932114 4.859682 3.4375515 3.6497543 5.485611 5.512378 3.1457896 2.5832813)

 (3.8783064 4.2615805 3.4476812 3.516016 3.12956 3.3235698 2.8970401 4.3073235

  3.7609475 2.2025976 3.6883376 2.3482933 1.623888 1.1030108 3.648819

  4.1074767)

 (3.8407595 4.2985744 4.4947147 4.7583337 4.5309863 3.1522655 3.7750213

  5.0408797 3.7760868 3.78026 2.1053405 4.7487717 3.6750562 3.6836185 1.5774668

  4.2355194 2.109648 2.879462 1.1504707 2.2890074 4.3080635)

 (3.7692716 3.8107364 4.005191 2.9590254 1.5087044 1.722277 4.741388 4.64542

  1.9565314 4.9309998 4.805511 3.6190488 3.2736812 1.834998 3.5858262 5.5998607

  3.7004514 2.1804154 1.924814 4.200107 4.5223045)

 (3.8725564 4.1996803 3.89821 3.8978398 2.9178553 1.5882304 1.413055 2.2741494

  3.9745965 3.4585989 7.8948627 2.3402674 2.172695 6.2782865 3.0781124

  1.6451169 1.7057719 5.9570546)

 (3.9283562 1.877443 2.0237117 4.214842 4.316505 2.8062139 2.4793828 2.6004548

  5.554006 2.514725 1.8137112 1.560705 2.7592404 2.4539397 3.0249727 1.2541932

  1.048938 1.128769 6.32025))

</pre>

This predicts the pitch values in dataset 18 (containing 8 short melodies), based on previous pitches (i.e., the target and source viewpoints are both <code>cpitch</code>). IDyOM computes the information content (IC) for each note, and by default returns three values: the first is a mean note IC for the whole dataset, the second a list of mean ICs for the individual compositions, the third is a list of lists containing the IC values for each note in each composition. The <code>:detail</code> parameter controls the detail of the output, while the <code>:output-path</code> parameter allows the user to generate a spreadsheet containing detailed model output (see below for further details).

h3. Statistical modelling parameters

See "Pearce [2005, chapter 6]":http://webprojects.eecs.qmul.ac.uk/marcusp/papers/Pearce2005.pdf for further description and explanation of these parameters.

* @models@: the type of IDyOM model to use.  Options are:

** @:stm@ - short-term model only, trained on the current composition.

** @:ltm@ - long-term model only, trained on the pretraining and resampling training data.

** @:ltm+@ - the long-term model, with additional incremental training on the test set;

** @:both@ - a combination of :stm and :ltm;

** @:both+@ -  a combination of :stm and :ltm+ (this is the default).

The LTM and STM can be configured using the @ltmo@ and @stmo@ parameters.  These accept a property list with the following properties - the defaults are used if a property is omitted or no parameter list is supplied:

* @:order-bound@: an integer indicating the bound on the order of the model, i.e. the number of past events used by the model.  The default is @nil@, no bound.

* @:mixtures@: whether to use mixtures for the model. (Default @t@).

* @:update-exclusion@: whether to use update exclusion. (LTM default @nil@, STM default @t@.)

* @:escape@: the model's escape method.  One of @:a :b :c :d :x@.  (LTM default @:c@, STM default @:x@.)

For example, the following command would combine the STM and LTM, without incremental training for the latter and an STM order bound of 4:

<pre>

CL-USER> (idyom:idyom 1 '(cpitch) '(cpitch) :models :both :stmo '(:order-bound 4))

</pre>

h3. Training parameters

When using IDyOM to estimate note IC for a given dataset, the long-term models can be trained on other datasets (pretraining) and/or on the current dataset, i.e. via resampling (cross-validation).  In the latter case, the dataset is partitioned into a training set (used to train the LTMs) and a test set (for which note IC is computed).  This split is called a fold, and the modelling process can be repeated with a number of different folds in order to model the entire dataset.

* @pretraining-ids@: a list of dataset ids used to pretrain the long-term models (done before resampling). Note that if pretraining-ids are supplied for an STM (i.e., <code> :models :stm</code>) the pretraining datasets are used to set the viewpoint domains (alphabet) for the models although not for training the models themselves (because they are short-term models).

* @k@: the number of resampling (cross-validation) folds to use.  The default value is 10.

** @1@ = no resampling, but also no training set unless the models are pretrained; 

** @:full@ = as many folds as there are compositions in the dataset

* @resampling-indices@: a list of numbers designating which resampling folds to use, i.e. a subset of @[0, 1, ..., k - 1]@.  By default, all folds are used.

*Note* that cross-validation only applies to the dataset being analysed (i.e., the one specified by the <code>dataset-id</code> argument). If a value of k=1 is supplied, the long-term models are not trained, unless a pretraining set is used.

h3. Viewpoint selection parameters

Instead of specifying the source viewpoints directly, IDyOM will undergo automatic viewpoint selection when the source viewpoint supplied is <code>:select</code>. This triggers a hill-climbing procedure that identifies the combination of source viewpoints which minimizes the mean information content of the specified dataset. The following parameters only have an effect when the source viewpoint supplied is <code>:select</code>. 

* @basis@: Identifies a set of viewpoints to be used in viewpoint selection, i.e. it will attempt to find the 'best' viewpoint system combining these, including by linking them. If :basis is excluded, the default (:auto) basis set is used. The parameter can be a list or one of the following keywords:

** @:pitch-full@ - The basis is a list of viewpoints useful for predicting pitch in Western music: cpitch, cpitch-class, tessitura, cpint, cpint-size, cpcint, cpcint-size, contour, newcontour, cpintfip, cpintfref, inscale.

** @:pitch-short@ - A shorter version of the above: cpitch, cpint, contour, cpintfref.

** @:bioi@ - For predicting Inter-Onset Interval (IOI): bioi, bioi-ratio, bioi-contour.

** @:onset@ - For predicting onset: onset, ioi, ioi-ratio, ioi-contour, metaccent

** @:auto@ - the basis is chosen to be the set of viewpoints that are defined in terms of one or more of the target viewpoints.  This is the default.

* @dp@: the number of decimal places to use when comparing information contents in viewpoint selection.  Full floating point precision is used if this is @nil@ (the default)

* @max-links@: the maximum number of links to use when creating linked viewpoints in viewpoint selection.  The default is 2.

* @min-links@: the minimum number of links to use when creating linked viewpoints in viewpoint selection.  The default is 2.

* @viewpoint-selection-output@: a filepath to write output for every viewpoint system considered during viewpoint selection. The default is nil meaning that no files are written.

h3. Output parameters

* <code>output-path</code>: a string indicating a directory in which to write the output 

** see [[IDyOM output]] for an explanation of the output files

** if a value of <code>nil</code> is given, information content (IC) is written to the console (see example below)

* <code>detail</code>: an integer which determines how the information content is averaged in the output: 

** 1: averaged over the entire dataset 

** 2: and also averaged over each composition 

** 3: and also with raw IC values for each event in each composition

* <code>overwrite</code>: whether to overwrite an existing output file if it exists. Default is not to overwrite (<code>nil</code>), pass 

<code>t</code> to overwrite output files.

* <code>separator</code>: a string defining the character to use for delimiting columns in the output file (default is " ", use "," for CSV)

h3. Caching parameters

* <code>use-resampling-set-cache?</code> a Boolean (t/nil) to specify whether to cache resampling sets

** default: t (so that the random division of the dataset into k-folds is stored and reused)

* <code>use-ltms-cache?</code> a Boolean (t/nil) controlling whether long-term models are stored and reused

** default: t

h2. Examples

h3. Mean melody IC

To get mean information contents (IC) for each composition of dataset 0 in a list. The first value represents the average IC for the whole dataset, the second value is a list of average ICs for each composition in the dataset. If <code>:detail 3</code> is specified, then the output would contain a third list, containing lists of ICs for each event in each composition in the database.

<pre>

CL-USER> (idyom:idyom 0 '(cpitch) '(cpintfref cpint) :detail 2)

2.493305

(2.1368716 2.8534691 2.6938546 2.6491673 2.4993074 2.6098127 2.7728052 2.772861

 2.5921957 2.905856 2.3591626 2.957503 2.4042292 2.7562473 2.3996017 2.8073587

 2.114944 1.7434102 2.2310295 2.6374347 2.361792 1.9476132 2.501488 2.5472867

 2.1056154 2.8225484 2.134257 2.9162033 3.0715692 2.9012227 2.7291088 2.866882

 2.8795822 2.4571223 2.9277062 2.7861307 2.6623116 2.3304622 2.4217033

 2.0556943 2.4048684 2.914848 2.7182267 3.0894585 2.873869 1.8821808 2.640174

 2.8165438 2.5423129 2.3011856 3.1477294 2.655349 2.5216308 2.0667994 3.2579045

 2.573013 2.6035044 2.202191 2.622113 2.2621205 2.3617425 2.7526956 2.3281655

 2.9357266 2.3372407 3.1848125 2.67367 2.1906006 2.7835917 2.6332111 3.206142

 2.1426969 2.194259 2.415167 1.9769101 2.0870917 2.7844474 2.2373738 2.772138

 2.9702199 1.724408 2.473073 2.2464263 2.2452457 2.688889 2.6299863 2.2223835

 2.8082614 2.673671 2.7693706 2.3369458 2.5016947 2.3837066 2.3682225 2.795649

 2.9063463 2.5880773 2.0457468 1.8635312 2.4522712 1.5877498 2.8802161

 2.7988417 2.3125513 1.7245895 2.2404804 2.1694546 2.365556 1.5905867 1.3827317

 2.2706041 3.023884 2.2864542 2.1259797 2.713626 2.1967313 2.5721254 2.5812547

 2.8233812 2.3134546 2.6203637 2.945946 2.601433 2.1920888 2.3732007 2.440137

 2.4291563 2.3676903 2.734724 3.0283954 2.8076048 2.7796154 2.326931 2.1779459

 2.2570527 2.2688026 1.3976555 2.030298 2.640235 2.568248 2.6338177 2.157162

 2.3915367 2.7873137 2.3088667 2.2176988 2.4402564 2.8062992 2.784044 2.4296925

 2.3520193 2.6146257)

</pre>

h3. Write note IC to file

To write the information contents for each note of each melody in dataset 0 to a file: 

<pre>

CL-USER> (idyom:idyom 0 '(cpitch) '((cpintfref cpint)) :detail 3 :output-path "/tmp/")

</pre>

See [[IDyOM Output]] for a description of the output files.

h3. Viewpoint Selection

<pre>

CL-USER> (idyom:idyom 17 '(cpitch) :select :models :stm :dp 3)

Selecting viewpoints for the STM model on dataset 17 predicting viewpoints (CPITCH).

Generating candidate viewpoints from: (CPITCH CPITCH-CLASS CPINT

                                       CPINT-SIZE CONTOUR NEWCONTOUR)

Max. links 2, whitelist (ANY), blacklist NIL

Candidate viewpoints: (CPITCH CPITCH-CLASS CPINT CPINT-SIZE CONTOUR

                       NEWCONTOUR (CONTOUR NEWCONTOUR)

                       (CPINT-SIZE NEWCONTOUR) (CPINT-SIZE CONTOUR)

                       (CPINT NEWCONTOUR) (CPINT CONTOUR)

                       (CPINT CPINT-SIZE) (CPITCH-CLASS NEWCONTOUR)

                       (CPITCH-CLASS CONTOUR) (CPITCH-CLASS CPINT-SIZE)

                       (CPITCH-CLASS CPINT) (CPITCH NEWCONTOUR)

                       (CPITCH CONTOUR) (CPITCH CPINT-SIZE) (CPITCH CPINT)

                       (CPITCH CPITCH-CLASS))

Selected system NIL, mean IC = NIL

Selected system ((CPITCH-CLASS CONTOUR)), mean IC = 3.0302427

 =======================================================================================

The selected viewpoint system with a mean IC of 3.0302427 is ((CPITCH-CLASS

                                                               CONTOUR))

3.0302427

(3.169925 3.169925 3.0849624 3.0849624 2.9886398 2.9886398 2.8774438 2.8774438)

((3.169925 3.169925) (3.169925 3.169925) (3.169925 3.0) (3.169925 3.0)

 (3.169925 2.807355) (3.169925 2.807355) (3.169925 2.5849626)

 (3.169925 2.5849626))

</pre>

This predicts the pitch values in dataset 17 using the short-term model based on the optimal set of defined viewpoints (i.e., the set which minimizes the mean information content of the dataset, in this case with precision of three decimal places). The viewpoint selection procedure settled on a system that predicts pitch in terms of linked cpitch-class and contour.

<pre>

CL-USER> (idyom:idyom 12 '(cpitch) :select :basis :pitch-short :max-links 1)

Selecting viewpoints for the BOTH+ model on dataset 12 predicting viewpoints (CPITCH).

Generating candidate viewpoints from: (CPITCH CPINT CONTOUR CPINTFREF)

Max. links 1, whitelist (ANY), blacklist NIL

Candidate viewpoints: (CPITCH CPINT CONTOUR CPINTFREF)

Selected system NIL, mean IC = NIL

Selected system (CPINT), mean IC = 3.7895417

Selected system (CPITCH CPINT), mean IC = 3.7376742

 =======================================================================================

The selected viewpoint system with a mean IC of 3.7376742 is (CPITCH CPINT)

3.7376742

(3.9260304 4.1582417 3.751911 3.6064732 3.2210944 2.8727105 3.0521867 4.3247104

 3.6203105 3.966651 3.7372475 4.083516 3.8313186 4.1036334 4.187075 3.4364297

 3.7849827 3.2999115 3.6294613 4.1595836)

((4.1709256 4.3708363 3.6702404 4.2592373 3.1589127)

 (4.5585337 4.093669 2.4383135 5.0920763 4.53643 4.230427)

 (5.090137 3.5416226 4.473012 2.9148462 3.8977542 2.5940926)

 (3.9261668 4.202055 2.9928908 4.3431 3.5786173 2.5960097)

 (4.0608945 4.230951 3.3359437 2.083827 3.4474165 2.1675336)

 (4.194988 4.1491704 3.8509068 1.0185144 2.9878855 1.0347978)

 (4.201399 3.6695127 2.7513106 4.227906 2.0919714 3.3589873 1.0642183)

 (4.1551943 4.396271 5.0603714 5.8716707 4.067729 2.3970249)

 (3.9431055 4.563211 3.3521545 4.0508695 3.5459816 2.266541)

 (4.221346 3.6652822 4.02317 5.0140486 3.4237287 3.4523292)

 (4.503613 3.3027549 4.369455 3.3479471 1.732882 5.166831)

 (4.0695643 3.6662188 4.4439473 4.9608116 3.2770386)

 (4.2264395 3.8040807 3.924297 4.070324 3.0400128 3.1923976 4.561678)

 (4.089103 3.692827 3.3639944 4.302426 4.5587425 4.6147075)

 (3.986353 5.397743 2.4407306 4.25579 4.2020874 5.218803 3.8080173)

 (4.067979 4.177483 4.3041263 1.6089504 3.4185483 3.0414906)

 (3.8922036 4.4922967 2.7819204 3.1301258 5.2425566 3.1707919)

 (4.2027206 3.868896 2.9337564 3.205565 2.980592 2.6079388)

 (4.309007 3.3705285 4.138041 4.3944063 1.9353238)

 (4.0367713 3.8443778 4.1985188 4.89163 5.230077 3.170318 3.7453916))

</pre>

This predicts the pitch values in dataset 12 based on the optimal set of viewpoints in the <code>pitch-short</code> basis set, with up to one link at a time. The viewpoint selection procedure settled on a system that predicts pitch in terms of (unlinked) cpitch and cpint.

h3. Conklin & Witten (1995)

To simulate the experiments of Conklin & Witten (1995) 

<pre>

CL-USER> (idyom:conkwit95)

Simulation of the experiments of Conklin & Witten (1995, Table 4).

System 1; Mean Information Content: 2.33 

System 2; Mean Information Content: 2.36 

System 3; Mean Information Content: 2.09 

System 4; Mean Information Content: 2.01 

System 5; Mean Information Content: 2.08 

System 6; Mean Information Content: 1.90 

System 7; Mean Information Content: 1.88 

System 8; Mean Information Content: 1.86 

NIL

</pre>

Compare with "Conklin & Witten [1995, JNMR, table 4]":http://www.sc.ehu.es/ccwbayes/members/conklin/papers/jnmr95.pdf

h3. Identifying melodic grouping boundaries (Pearce et al., 2010, _Perception_, 39, 1367-1391). 

This involves applying a peak-picking algorithm to the output of <code>idyom</code>. For example,

<pre>

CL-USER> (multiple-value-bind (d1 d2 d3) 

             (idyom:idyom 19 '(cpitch) '(cpitch) :k 6 :pretraining-ids '(3) :models :both+)

           (declare (ignore d1 d2))

           (mapcar #'segmentation:peak-picker d3))

((0 0 0 0 0 1 0 0 1 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0)

 (0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0

  0 0 0 0 0 0 1 0 0)

 (0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 1 0)

 (0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0

  0 0 0 0 0 0 0 0 0 0 0 0 0 0)

 (0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0)

 (0 0 0 0 0 0 1 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 1 0 0 0 0 0))

CL-USER> 

</pre>

In the output, a one indicates that the event follows a predicted grouping boundary (e.g., the first event in a new phrase) while a zero indicates that this is not the case.