samer@41: \documentclass[conference]{IEEEtran} samer@4: \usepackage{cite} samer@4: \usepackage[cmex10]{amsmath} samer@4: \usepackage{graphicx} samer@4: \usepackage{amssymb} samer@4: \usepackage{epstopdf} samer@4: \usepackage{url} samer@4: \usepackage{listings} samer@18: %\usepackage[expectangle]{tools} samer@9: \usepackage{tools} samer@18: \usepackage{tikz} samer@18: \usetikzlibrary{calc} samer@18: \usetikzlibrary{matrix} samer@18: \usetikzlibrary{patterns} samer@18: \usetikzlibrary{arrows} samer@9: samer@9: \let\citep=\cite samer@33: \newcommand{\colfig}[2][1]{\includegraphics[width=#1\linewidth]{figs/#2}}% samer@18: \newcommand\preals{\reals_+} samer@18: \newcommand\X{\mathcal{X}} samer@18: \newcommand\Y{\mathcal{Y}} samer@18: \newcommand\domS{\mathcal{S}} samer@18: \newcommand\A{\mathcal{A}} samer@25: \newcommand\Data{\mathcal{D}} samer@18: \newcommand\rvm[1]{\mathrm{#1}} samer@18: \newcommand\sps{\,.\,} samer@18: \newcommand\Ipred{\mathcal{I}_{\mathrm{pred}}} samer@18: \newcommand\Ix{\mathcal{I}} samer@18: \newcommand\IXZ{\overline{\underline{\mathcal{I}}}} samer@18: \newcommand\x{\vec{x}} samer@18: \newcommand\Ham[1]{\mathcal{H}_{#1}} samer@18: \newcommand\subsets[2]{[#1]^{(k)}} samer@18: \def\bet(#1,#2){#1..#2} samer@18: samer@18: samer@18: \def\ev(#1=#2){#1\!\!=\!#2} samer@18: \newcommand\rv[1]{\Omega \to #1} samer@18: \newcommand\ceq{\!\!=\!} samer@18: \newcommand\cmin{\!-\!} samer@18: \newcommand\modulo[2]{#1\!\!\!\!\!\mod#2} samer@18: samer@18: \newcommand\sumitoN{\sum_{i=1}^N} samer@18: \newcommand\sumktoK{\sum_{k=1}^K} samer@18: \newcommand\sumjtoK{\sum_{j=1}^K} samer@18: \newcommand\sumalpha{\sum_{\alpha\in\A}} samer@18: \newcommand\prodktoK{\prod_{k=1}^K} samer@18: \newcommand\prodjtoK{\prod_{j=1}^K} samer@18: samer@18: \newcommand\past[1]{\overset{\rule{0pt}{0.2em}\smash{\leftarrow}}{#1}} samer@18: \newcommand\fut[1]{\overset{\rule{0pt}{0.1em}\smash{\rightarrow}}{#1}} samer@18: \newcommand\parity[2]{P^{#1}_{2,#2}} samer@4: samer@4: %\usepackage[parfill]{parskip} samer@4: samer@4: \begin{document} samer@41: \title{Cognitive Music Modelling: an\\Information Dynamics Approach} samer@4: samer@4: \author{ hekeus@16: \IEEEauthorblockN{Samer A. Abdallah, Henrik Ekeus, Peter Foster} hekeus@16: \IEEEauthorblockN{Andrew Robertson and Mark D. Plumbley} samer@4: \IEEEauthorblockA{Centre for Digital Music\\ samer@4: Queen Mary University of London\\ samer@41: Mile End Road, London E1 4NS}} samer@4: samer@4: \maketitle samer@18: \begin{abstract} samer@18: People take in information when perceiving music. With it they continually samer@18: build predictive models of what is going to happen. There is a relationship samer@18: between information measures and how we perceive music. An information samer@18: theoretic approach to music cognition is thus a fruitful avenue of research. samer@18: In this paper, we review the theoretical foundations of information dynamics samer@18: and discuss a few emerging areas of application. hekeus@16: \end{abstract} samer@4: samer@4: samer@25: \section{Introduction} samer@9: \label{s:Intro} samer@9: samer@25: \subsection{Expectation and surprise in music} samer@18: One of the effects of listening to music is to create samer@18: expectations of what is to come next, which may be fulfilled samer@9: immediately, after some delay, or not at all as the case may be. samer@9: This is the thesis put forward by, amongst others, music theorists samer@18: L. B. Meyer \cite{Meyer67} and Narmour \citep{Narmour77}, but was samer@18: recognised much earlier; for example, samer@9: it was elegantly put by Hanslick \cite{Hanslick1854} in the samer@9: nineteenth century: samer@9: \begin{quote} samer@9: `The most important factor in the mental process which accompanies the samer@9: act of listening to music, and which converts it to a source of pleasure, samer@18: is \ldots the intellectual satisfaction samer@9: which the listener derives from continually following and anticipating samer@9: the composer's intentions---now, to see his expectations fulfilled, and samer@18: now, to find himself agreeably mistaken. samer@18: %It is a matter of course that samer@18: %this intellectual flux and reflux, this perpetual giving and receiving samer@18: %takes place unconsciously, and with the rapidity of lightning-flashes.' samer@9: \end{quote} samer@9: An essential aspect of this is that music is experienced as a phenomenon samer@9: that `unfolds' in time, rather than being apprehended as a static object samer@9: presented in its entirety. Meyer argued that musical experience depends samer@9: on how we change and revise our conceptions \emph{as events happen}, on samer@9: how expectation and prediction interact with occurrence, and that, to a samer@9: large degree, the way to understand the effect of music is to focus on samer@9: this `kinetics' of expectation and surprise. samer@9: samer@25: Prediction and expectation are essentially probabilistic concepts samer@25: and can be treated mathematically using probability theory. samer@25: We suppose that when we listen to music, expectations are created on the basis samer@25: of our familiarity with various styles of music and our ability to samer@25: detect and learn statistical regularities in the music as they emerge, samer@25: There is experimental evidence that human listeners are able to internalise samer@25: statistical knowledge about musical structure, \eg samer@25: \citep{SaffranJohnsonAslin1999,EerolaToiviainenKrumhansl2002}, and also samer@25: that statistical models can form an effective basis for computational samer@25: analysis of music, \eg samer@25: \cite{ConklinWitten95,PonsfordWigginsMellish1999,Pearce2005}. samer@25: samer@25: samer@25: \comment{ samer@9: The business of making predictions and assessing surprise is essentially samer@9: one of reasoning under conditions of uncertainty and manipulating samer@9: degrees of belief about the various proposition which may or may not samer@9: hold, and, as has been argued elsewhere \cite{Cox1946,Jaynes27}, best samer@9: quantified in terms of Bayesian probability theory. samer@9: Thus, we suppose that samer@9: when we listen to music, expectations are created on the basis of our samer@24: familiarity with various stylistic norms that apply to music in general, samer@24: the particular style (or styles) of music that seem best to fit the piece samer@24: we are listening to, and samer@9: the emerging structures peculiar to the current piece. There is samer@9: experimental evidence that human listeners are able to internalise samer@9: statistical knowledge about musical structure, \eg samer@9: \citep{SaffranJohnsonAslin1999,EerolaToiviainenKrumhansl2002}, and also samer@9: that statistical models can form an effective basis for computational samer@9: analysis of music, \eg samer@9: \cite{ConklinWitten95,PonsfordWigginsMellish1999,Pearce2005}. samer@25: } samer@9: samer@9: \subsection{Music and information theory} samer@24: With a probabilistic framework for music modelling and prediction in hand, samer@25: we are in a position to apply Shannon's quantitative information theory samer@25: \cite{Shannon48}. samer@25: \comment{ samer@25: which provides us with a number of measures, such as entropy samer@25: and mutual information, which are suitable for quantifying states of samer@25: uncertainty and surprise, and thus could potentially enable us to build samer@25: quantitative models of the listening process described above. They are samer@25: what Berlyne \cite{Berlyne71} called `collative variables' since they are samer@25: to do with patterns of occurrence rather than medium-specific details. samer@25: Berlyne sought to show that the collative variables are closely related to samer@25: perceptual qualities like complexity, tension, interestingness, samer@25: and even aesthetic value, not just in music, but in other temporal samer@25: or visual media. samer@25: The relevance of information theory to music and art has samer@25: also been addressed by researchers from the 1950s onwards samer@25: \cite{Youngblood58,CoonsKraehenbuehl1958,Cohen1962,HillerBean66,Moles66,Meyer67}. samer@25: } samer@9: The relationship between information theory and music and art in general has been the samer@9: subject of some interest since the 1950s samer@9: \cite{Youngblood58,CoonsKraehenbuehl1958,HillerBean66,Moles66,Meyer67,Cohen1962}. samer@9: The general thesis is that perceptible qualities and subjective samer@9: states like uncertainty, surprise, complexity, tension, and interestingness samer@9: are closely related to samer@9: information-theoretic quantities like entropy, relative entropy, samer@9: and mutual information. samer@9: % and are major determinants of the overall experience. samer@9: Berlyne \cite{Berlyne71} called such quantities `collative variables', since samer@9: they are to do with patterns of occurrence rather than medium-specific details, samer@9: and developed the ideas of `information aesthetics' in an experimental setting. samer@9: % Berlyne's `new experimental aesthetics', the `information-aestheticians'. samer@9: samer@9: % Listeners then experience greater or lesser levels of surprise samer@9: % in response to departures from these norms. samer@9: % By careful manipulation samer@9: % of the material, the composer can thus define, and induce within the samer@9: % listener, a temporal programme of varying samer@9: % levels of uncertainty, ambiguity and surprise. samer@9: samer@9: samer@9: \subsection{Information dynamic approach} samer@9: samer@24: Bringing the various strands together, our working hypothesis is that as a samer@24: listener (to which will refer as `it') listens to a piece of music, it maintains samer@25: a dynamically evolving probabilistic model that enables it to make predictions samer@24: about how the piece will continue, relying on both its previous experience samer@24: of music and the immediate context of the piece. As events unfold, it revises samer@25: its probabilistic belief state, which includes predictive samer@25: distributions over possible future events. These samer@25: % distributions and changes in distributions samer@25: can be characterised in terms of a handful of information samer@25: theoretic-measures such as entropy and relative entropy. By tracing the samer@24: evolution of a these measures, we obtain a representation which captures much samer@25: of the significant structure of the music. samer@25: samer@25: One of the consequences of this approach is that regardless of the details of samer@25: the sensory input or even which sensory modality is being processed, the resulting samer@25: analysis is in terms of the same units: quantities of information (bits) and samer@25: rates of information flow (bits per second). The probabilistic and information samer@25: theoretic concepts in terms of which the analysis is framed are universal to all sorts samer@25: of data. samer@25: In addition, when adaptive probabilistic models are used, expectations are samer@25: created mainly in response to to \emph{patterns} of occurence, samer@25: rather the details of which specific things occur. samer@25: Together, these suggest that an information dynamic analysis captures a samer@25: high level of \emph{abstraction}, and could be used to samer@25: make structural comparisons between different temporal media, samer@25: such as music, film, animation, and dance. samer@25: % analyse and compare information samer@25: % flow in different temporal media regardless of whether they are auditory, samer@25: % visual or otherwise. samer@9: samer@25: Another consequence is that the information dynamic approach gives us a principled way samer@24: to address the notion of \emph{subjectivity}, since the analysis is dependent on the samer@24: probability model the observer starts off with, which may depend on prior experience samer@24: or other factors, and which may change over time. Thus, inter-subject variablity and samer@24: variation in subjects' responses over time are samer@24: fundamental to the theory. samer@9: samer@18: %modelling the creative process, which often alternates between generative samer@18: %and selective or evaluative phases \cite{Boden1990}, and would have samer@18: %applications in tools for computer aided composition. samer@18: samer@18: samer@18: \section{Theoretical review} samer@18: samer@34: \subsection{Entropy and information} samer@41: \label{s:entro-info} samer@41: samer@34: Let $X$ denote some variable whose value is initially unknown to our samer@34: hypothetical observer. We will treat $X$ mathematically as a random variable, samer@36: with a value to be drawn from some set $\X$ and a samer@34: probability distribution representing the observer's beliefs about the samer@34: true value of $X$. samer@34: In this case, the observer's uncertainty about $X$ can be quantified samer@34: as the entropy of the random variable $H(X)$. For a discrete variable samer@36: with probability mass function $p:\X \to [0,1]$, this is samer@34: \begin{equation} samer@41: H(X) = \sum_{x\in\X} -p(x) \log p(x), % = \expect{-\log p(X)}, samer@34: \end{equation} samer@41: % where $\expect{}$ is the expectation operator. samer@41: The negative-log-probability samer@34: $\ell(x) = -\log p(x)$ of a particular value $x$ can usefully be thought of as samer@34: the \emph{surprisingness} of the value $x$ should it be observed, and samer@41: hence the entropy is the expectation of the surprisingness $\expect \ell(X)$. samer@34: samer@34: Now suppose that the observer receives some new data $\Data$ that samer@34: causes a revision of its beliefs about $X$. The \emph{information} samer@34: in this new data \emph{about} $X$ can be quantified as the samer@34: Kullback-Leibler (KL) divergence between the prior and posterior samer@34: distributions $p(x)$ and $p(x|\Data)$ respectively: samer@34: \begin{equation} samer@34: \mathcal{I}_{\Data\to X} = D(p_{X|\Data} || p_{X}) samer@36: = \sum_{x\in\X} p(x|\Data) \log \frac{p(x|\Data)}{p(x)}. samer@41: \label{eq:info} samer@34: \end{equation} samer@34: When there are multiple variables $X_1, X_2$ samer@34: \etc which the observer believes to be dependent, then the observation of samer@34: one may change its beliefs and hence yield information about the samer@34: others. The joint and conditional entropies as described in any samer@34: textbook on information theory (\eg \cite{CoverThomas}) then quantify samer@34: the observer's expected uncertainty about groups of variables given the samer@34: values of others. In particular, the \emph{mutual information} samer@34: $I(X_1;X_2)$ is both the expected information samer@34: in an observation of $X_2$ about $X_1$ and the expected reduction samer@34: in uncertainty about $X_1$ after observing $X_2$: samer@34: \begin{equation} samer@34: I(X_1;X_2) = H(X_1) - H(X_1|X_2), samer@34: \end{equation} samer@34: where $H(X_1|X_2) = H(X_1,X_2) - H(X_2)$ is the conditional entropy samer@34: of $X_2$ given $X_1$. A little algebra shows that $I(X_1;X_2)=I(X_2;X_1)$ samer@34: and so the mutual information is symmetric in its arguments. A conditional samer@34: form of the mutual information can be formulated analogously: samer@34: \begin{equation} samer@34: I(X_1;X_2|X_3) = H(X_1|X_3) - H(X_1|X_2,X_3). samer@34: \end{equation} samer@34: These relationships between the various entropies and mutual samer@34: informations are conveniently visualised in Venn diagram-like \emph{information diagrams} samer@34: or I-diagrams \cite{Yeung1991} such as the one in \figrf{venn-example}. samer@34: samer@18: \begin{fig}{venn-example} samer@18: \newcommand\rad{2.2em}% samer@18: \newcommand\circo{circle (3.4em)}% samer@18: \newcommand\labrad{4.3em} samer@18: \newcommand\bound{(-6em,-5em) rectangle (6em,6em)} samer@18: \newcommand\colsep{\ } samer@18: \newcommand\clipin[1]{\clip (#1) \circo;}% samer@18: \newcommand\clipout[1]{\clip \bound (#1) \circo;}% samer@18: \newcommand\cliptwo[3]{% samer@18: \begin{scope} samer@18: \clipin{#1}; samer@18: \clipin{#2}; samer@18: \clipout{#3}; samer@18: \fill[black!30] \bound; samer@18: \end{scope} samer@18: }% samer@18: \newcommand\clipone[3]{% samer@18: \begin{scope} samer@18: \clipin{#1}; samer@18: \clipout{#2}; samer@18: \clipout{#3}; samer@18: \fill[black!15] \bound; samer@18: \end{scope} samer@18: }% samer@18: \begin{tabular}{c@{\colsep}c} samer@18: \begin{tikzpicture}[baseline=0pt] samer@18: \coordinate (p1) at (90:\rad); samer@18: \coordinate (p2) at (210:\rad); samer@18: \coordinate (p3) at (-30:\rad); samer@18: \clipone{p1}{p2}{p3}; samer@18: \clipone{p2}{p3}{p1}; samer@18: \clipone{p3}{p1}{p2}; samer@18: \cliptwo{p1}{p2}{p3}; samer@18: \cliptwo{p2}{p3}{p1}; samer@18: \cliptwo{p3}{p1}{p2}; samer@18: \begin{scope} samer@18: \clip (p1) \circo; samer@18: \clip (p2) \circo; samer@18: \clip (p3) \circo; samer@18: \fill[black!45] \bound; samer@18: \end{scope} samer@18: \draw (p1) \circo; samer@18: \draw (p2) \circo; samer@18: \draw (p3) \circo; samer@18: \path samer@18: (barycentric cs:p3=1,p1=-0.2,p2=-0.1) +(0ex,0) node {$I_{3|12}$} samer@18: (barycentric cs:p1=1,p2=-0.2,p3=-0.1) +(0ex,0) node {$I_{1|23}$} samer@18: (barycentric cs:p2=1,p3=-0.2,p1=-0.1) +(0ex,0) node {$I_{2|13}$} samer@18: (barycentric cs:p3=1,p2=1,p1=-0.55) +(0ex,0) node {$I_{23|1}$} samer@18: (barycentric cs:p1=1,p3=1,p2=-0.55) +(0ex,0) node {$I_{13|2}$} samer@18: (barycentric cs:p2=1,p1=1,p3=-0.55) +(0ex,0) node {$I_{12|3}$} samer@18: (barycentric cs:p3=1,p2=1,p1=1) node {$I_{123}$} samer@18: ; samer@18: \path samer@18: (p1) +(140:\labrad) node {$X_1$} samer@18: (p2) +(-140:\labrad) node {$X_2$} samer@18: (p3) +(-40:\labrad) node {$X_3$}; samer@18: \end{tikzpicture} samer@18: & samer@18: \parbox{0.5\linewidth}{ samer@18: \small samer@18: \begin{align*} samer@18: I_{1|23} &= H(X_1|X_2,X_3) \\ samer@18: I_{13|2} &= I(X_1;X_3|X_2) \\ samer@18: I_{1|23} + I_{13|2} &= H(X_1|X_2) \\ samer@18: I_{12|3} + I_{123} &= I(X_1;X_2) samer@18: \end{align*} samer@18: } samer@18: \end{tabular} samer@18: \caption{ samer@30: I-diagram visualisation of entropies and mutual informations samer@18: for three random variables $X_1$, $X_2$ and $X_3$. The areas of samer@18: the three circles represent $H(X_1)$, $H(X_2)$ and $H(X_3)$ respectively. samer@18: The total shaded area is the joint entropy $H(X_1,X_2,X_3)$. samer@18: The central area $I_{123}$ is the co-information \cite{McGill1954}. samer@18: Some other information measures are indicated in the legend. samer@18: } samer@18: \end{fig} samer@30: samer@30: samer@36: \subsection{Surprise and information in sequences} samer@36: \label{s:surprise-info-seq} samer@30: samer@36: Suppose that $(\ldots,X_{-1},X_0,X_1,\ldots)$ is a sequence of samer@30: random variables, infinite in both directions, samer@36: and that $\mu$ is the associated probability measure over all samer@36: realisations of the sequence---in the following, $\mu$ will simply serve samer@30: as a label for the process. We can indentify a number of information-theoretic samer@30: measures meaningful in the context of a sequential observation of the sequence, during samer@36: which, at any time $t$, the sequence of variables can be divided into a `present' $X_t$, a `past' samer@30: $\past{X}_t \equiv (\ldots, X_{t-2}, X_{t-1})$, and a `future' samer@30: $\fut{X}_t \equiv (X_{t+1},X_{t+2},\ldots)$. samer@41: We will write the actually observed value of $X_t$ as $x_t$, and samer@36: the sequence of observations up to but not including $x_t$ as samer@36: $\past{x}_t$. samer@36: % Since the sequence is assumed stationary, we can without loss of generality, samer@36: % assume that $t=0$ in the following definitions. samer@36: samer@41: The in-context surprisingness of the observation $X_t=x_t$ depends on samer@41: both $x_t$ and the context $\past{x}_t$: samer@36: \begin{equation} samer@41: \ell_t = - \log p(x_t|\past{x}_t). samer@36: \end{equation} samer@36: However, before $X_t$ is observed to be $x_t$, the observer can compute samer@46: the \emph{expected} surprisingness as a measure of its uncertainty about samer@36: the very next event; this may be written as an entropy samer@36: $H(X_t|\ev(\past{X}_t = \past{x}_t))$, but note that this is samer@36: conditional on the \emph{event} $\ev(\past{X}_t=\past{x}_t)$, not samer@36: \emph{variables} $\past{X}_t$ as in the conventional conditional entropy. samer@36: samer@41: The surprisingness $\ell_t$ and expected surprisingness samer@36: $H(X_t|\ev(\past{X}_t=\past{x}_t))$ samer@41: can be understood as \emph{subjective} information dynamic measures, since they are samer@41: based on the observer's probability model in the context of the actually observed sequence samer@36: $\past{x}_t$---they characterise what it is like to `be in the observer's shoes'. samer@36: If we view the observer as a purely passive or reactive agent, this would samer@36: probably be sufficient, but for active agents such as humans or animals, it is samer@36: often necessary to \emph{aniticipate} future events in order, for example, to plan the samer@36: most effective course of action. It makes sense for such observers to be samer@36: concerned about the predictive probability distribution over future events, samer@36: $p(\fut{x}_t|\past{x}_t)$. When an observation $\ev(X_t=x_t)$ is made in this context, samer@41: the \emph{instantaneous predictive information} (IPI) $\mathcal{I}_t$ at time $t$ samer@41: is the information in the event $\ev(X_t=x_t)$ about the entire future of the sequence $\fut{X}_t$, samer@41: \emph{given} the observed past $\past{X}_t=\past{x}_t$. samer@41: Referring to the definition of information \eqrf{info}, this is the KL divergence samer@41: between prior and posterior distributions over possible futures, which written out in full, is samer@41: \begin{equation} samer@41: \mathcal{I}_t = \sum_{\fut{x}_t \in \X^*} samer@41: p(\fut{x}_t|x_t,\past{x}_t) \log \frac{ p(\fut{x}_t|x_t,\past{x}_t) }{ p(\fut{x}_t|\past{x}_t) }, samer@41: \end{equation} samer@41: where the sum is to be taken over the set of infinite sequences $\X^*$. samer@46: Note that it is quite possible for an event to be surprising but not informative samer@46: in predictive sense. samer@41: As with the surprisingness, the observer can compute its \emph{expected} IPI samer@41: at time $t$, which reduces to a mutual information $I(X_t;\fut{X}_t|\ev(\past{X}_t=\past{x}_t))$ samer@41: conditioned on the observed past. This could be used, for example, as an estimate samer@41: of attentional resources which should be directed at this stream of data, which may samer@41: be in competition with other sensory streams. samer@36: samer@36: \subsection{Information measures for stationary random processes} samer@43: \label{s:process-info} samer@30: samer@18: samer@18: \begin{fig}{predinfo-bg} samer@18: \newcommand\subfig[2]{\shortstack{#2\\[0.75em]#1}} samer@18: \newcommand\rad{1.8em}% samer@18: \newcommand\ovoid[1]{% samer@18: ++(-#1,\rad) samer@18: -- ++(2 * #1,0em) arc (90:-90:\rad) samer@18: -- ++(-2 * #1,0em) arc (270:90:\rad) samer@18: }% samer@18: \newcommand\axis{2.75em}% samer@18: \newcommand\olap{0.85em}% samer@18: \newcommand\offs{3.6em} samer@18: \newcommand\colsep{\hspace{5em}} samer@18: \newcommand\longblob{\ovoid{\axis}} samer@18: \newcommand\shortblob{\ovoid{1.75em}} samer@18: \begin{tabular}{c@{\colsep}c} samer@43: \subfig{(a) multi-information and entropy rates}{% samer@43: \begin{tikzpicture}%[baseline=-1em] samer@43: \newcommand\rc{1.75em} samer@43: \newcommand\throw{2.5em} samer@43: \coordinate (p1) at (180:1.5em); samer@43: \coordinate (p2) at (0:0.3em); samer@43: \newcommand\bound{(-7em,-2.6em) rectangle (7em,3.0em)} samer@43: \newcommand\present{(p2) circle (\rc)} samer@43: \newcommand\thepast{(p1) ++(-\throw,0) \ovoid{\throw}} samer@43: \newcommand\fillclipped[2]{% samer@43: \begin{scope}[even odd rule] samer@43: \foreach \thing in {#2} {\clip \thing;} samer@43: \fill[black!#1] \bound; samer@43: \end{scope}% samer@43: }% samer@43: \fillclipped{30}{\present,\bound \thepast} samer@43: \fillclipped{15}{\present,\bound \thepast} samer@43: \fillclipped{45}{\present,\thepast} samer@43: \draw \thepast; samer@43: \draw \present; samer@43: \node at (barycentric cs:p2=1,p1=-0.3) {$h_\mu$}; samer@43: \node at (barycentric cs:p2=1,p1=1) [shape=rectangle,fill=black!45,inner sep=1pt]{$\rho_\mu$}; samer@43: \path (p2) +(90:3em) node {$X_0$}; samer@43: \path (p1) +(-3em,0em) node {\shortstack{infinite\\past}}; samer@43: \path (p1) +(-4em,\rad) node [anchor=south] {$\ldots,X_{-1}$}; samer@43: \end{tikzpicture}}% samer@43: \\[1.25em] samer@43: \subfig{(b) excess entropy}{% samer@18: \newcommand\blob{\longblob} samer@18: \begin{tikzpicture} samer@18: \coordinate (p1) at (-\offs,0em); samer@18: \coordinate (p2) at (\offs,0em); samer@18: \begin{scope} samer@18: \clip (p1) \blob; samer@18: \clip (p2) \blob; samer@18: \fill[lightgray] (-1,-1) rectangle (1,1); samer@18: \end{scope} samer@18: \draw (p1) +(-0.5em,0em) node{\shortstack{infinite\\past}} \blob; samer@18: \draw (p2) +(0.5em,0em) node{\shortstack{infinite\\future}} \blob; samer@18: \path (0,0) node (future) {$E$}; samer@18: \path (p1) +(-2em,\rad) node [anchor=south] {$\ldots,X_{-1}$}; samer@18: \path (p2) +(2em,\rad) node [anchor=south] {$X_0,\ldots$}; samer@18: \end{tikzpicture}% samer@18: }% samer@18: \\[1.25em] samer@43: \subfig{(c) predictive information rate $b_\mu$}{% samer@18: \begin{tikzpicture}%[baseline=-1em] samer@18: \newcommand\rc{2.1em} samer@18: \newcommand\throw{2.5em} samer@18: \coordinate (p1) at (210:1.5em); samer@18: \coordinate (p2) at (90:0.7em); samer@18: \coordinate (p3) at (-30:1.5em); samer@18: \newcommand\bound{(-7em,-2.6em) rectangle (7em,3.0em)} samer@18: \newcommand\present{(p2) circle (\rc)} samer@18: \newcommand\thepast{(p1) ++(-\throw,0) \ovoid{\throw}} samer@18: \newcommand\future{(p3) ++(\throw,0) \ovoid{\throw}} samer@18: \newcommand\fillclipped[2]{% samer@18: \begin{scope}[even odd rule] samer@18: \foreach \thing in {#2} {\clip \thing;} samer@18: \fill[black!#1] \bound; samer@18: \end{scope}% samer@18: }% samer@43: \fillclipped{80}{\future,\thepast} samer@18: \fillclipped{30}{\present,\future,\bound \thepast} samer@18: \fillclipped{15}{\present,\bound \future,\bound \thepast} samer@18: \draw \future; samer@18: \fillclipped{45}{\present,\thepast} samer@18: \draw \thepast; samer@18: \draw \present; samer@18: \node at (barycentric cs:p2=1,p1=-0.17,p3=-0.17) {$r_\mu$}; samer@18: \node at (barycentric cs:p1=-0.4,p2=1.0,p3=1) {$b_\mu$}; samer@18: \node at (barycentric cs:p3=0,p2=1,p1=1.2) [shape=rectangle,fill=black!45,inner sep=1pt]{$\rho_\mu$}; samer@18: \path (p2) +(140:3em) node {$X_0$}; samer@18: % \node at (barycentric cs:p3=0,p2=1,p1=1) {$\rho_\mu$}; samer@18: \path (p3) +(3em,0em) node {\shortstack{infinite\\future}}; samer@18: \path (p1) +(-3em,0em) node {\shortstack{infinite\\past}}; samer@18: \path (p1) +(-4em,\rad) node [anchor=south] {$\ldots,X_{-1}$}; samer@18: \path (p3) +(4em,\rad) node [anchor=south] {$X_1,\ldots$}; samer@18: \end{tikzpicture}}% samer@18: \\[0.5em] samer@18: \end{tabular} samer@18: \caption{ samer@30: I-diagrams for several information measures in samer@18: stationary random processes. Each circle or oval represents a random samer@18: variable or sequence of random variables relative to time $t=0$. Overlapped areas samer@18: correspond to various mutual information as in \Figrf{venn-example}. samer@33: In (b), the circle represents the `present'. Its total area is samer@33: $H(X_0)=\rho_\mu+r_\mu+b_\mu$, where $\rho_\mu$ is the multi-information samer@18: rate, $r_\mu$ is the residual entropy rate, and $b_\mu$ is the predictive samer@43: information rate. The entropy rate is $h_\mu = r_\mu+b_\mu$. The small dark samer@43: region below $X_0$ in (c) is $\sigma_\mu = E-\rho_\mu$. samer@18: } samer@18: \end{fig} samer@18: samer@41: If we step back, out of the observer's shoes as it were, and consider the samer@41: random process $(\ldots,X_{-1},X_0,X_1,\dots)$ as a statistical ensemble of samer@41: possible realisations, and furthermore assume that it is stationary, samer@41: then it becomes possible to define a number of information-theoretic measures, samer@41: closely related to those described above, but which characterise the samer@41: process as a whole, rather than on a moment-by-moment basis. Some of these, samer@41: such as the entropy rate, are well-known, but others are only recently being samer@41: investigated. (In the following, the assumption of stationarity means that samer@41: the measures defined below are independent of $t$.) samer@41: samer@41: The \emph{entropy rate} of the process is the entropy of the next variable samer@41: $X_t$ given all the previous ones. samer@41: \begin{equation} samer@41: \label{eq:entro-rate} samer@41: h_\mu = H(X_t|\past{X}_t). samer@41: \end{equation} samer@51: The entropy rate is a measure of the overall surprisingness samer@51: or unpredictability of the process, and gives an indication of the average samer@51: level of surprise and uncertainty that would be experienced by an observer samer@51: processing a sequence sampled from the process using the methods of samer@51: \secrf{surprise-info-seq}. samer@41: samer@41: The \emph{multi-information rate} $\rho_\mu$ (following Dubnov's \cite{Dubnov2006} samer@41: notation for what he called the `information rate') is the mutual samer@41: information between the `past' and the `present': samer@41: \begin{equation} samer@41: \label{eq:multi-info} samer@41: \rho_\mu = I(\past{X}_t;X_t) = H(X_t) - h_\mu. samer@41: \end{equation} samer@41: It is a measure of how much the context of an observation (that is, samer@41: the observation of previous elements of the sequence) helps in predicting samer@41: or reducing the suprisingness of the current observation. samer@41: samer@41: The \emph{excess entropy} \cite{CrutchfieldPackard1983} samer@41: is the mutual information between samer@41: the entire `past' and the entire `future': samer@41: \begin{equation} samer@41: E = I(\past{X}_t; X_t,\fut{X}_t). samer@41: \end{equation} samer@43: Both the excess entropy and the multi-information rate can be thought samer@43: of as measures of \emph{redundancy}, quantifying the extent to which samer@43: the same information is to be found in all parts of the sequence. samer@41: samer@41: samer@30: The \emph{predictive information rate} (or PIR) \cite{AbdallahPlumbley2009} samer@46: is the mutual information between the present and the infinite future given the infinite samer@46: past: samer@18: \begin{equation} samer@18: \label{eq:PIR} samer@41: b_\mu = I(X_t;\fut{X}_t|\past{X}_t) = H(\fut{X}_t|\past{X}_t) - H(\fut{X}_t|X_t,\past{X}_t). samer@18: \end{equation} samer@18: Equation \eqrf{PIR} can be read as the average reduction samer@18: in uncertainty about the future on learning $X_t$, given the past. samer@18: Due to the symmetry of the mutual information, it can also be written samer@18: as samer@18: \begin{equation} samer@18: % \IXZ_t samer@43: b_\mu = H(X_t|\past{X}_t) - H(X_t|\past{X}_t,\fut{X}_t) = h_\mu - r_\mu, samer@18: % \label{<++>} samer@18: \end{equation} samer@18: % If $X$ is stationary, then samer@41: where $r_\mu = H(X_t|\fut{X}_t,\past{X}_t)$, samer@34: is the \emph{residual} \cite{AbdallahPlumbley2010}, samer@34: or \emph{erasure} \cite{VerduWeissman2006} entropy rate. samer@18: These relationships are illustrated in \Figrf{predinfo-bg}, along with samer@18: several of the information measures we have discussed so far. samer@51: The PIR gives an indication of the average IPI that would be experienced samer@51: by an observer processing a sequence sampled from this process. samer@18: samer@18: samer@46: James et al \cite{JamesEllisonCrutchfield2011} review several of these samer@46: information measures and introduce some new related ones. samer@46: In particular they identify the $\sigma_\mu = I(\past{X}_t;\fut{X}_t|X_t)$, samer@46: the mutual information between the past and the future given the present, samer@46: as an interesting quantity that measures the predictive benefit of samer@25: model-building (that is, maintaining an internal state summarising past samer@46: observations in order to make better predictions). It is shown as the samer@46: small dark region below the circle in \figrf{predinfo-bg}(c). samer@46: By comparing with \figrf{predinfo-bg}(b), we can see that samer@46: $\sigma_\mu = E - \rho_\mu$. samer@43: % They also identify samer@43: % $w_\mu = \rho_\mu + b_{\mu}$, which they call the \emph{local exogenous samer@43: % information} rate. samer@34: samer@4: samer@36: \subsection{First and higher order Markov chains} samer@53: \label{s:markov} samer@36: First order Markov chains are the simplest non-trivial models to which information samer@36: dynamics methods can be applied. In \cite{AbdallahPlumbley2009} we derived samer@41: expressions for all the information measures described in \secrf{surprise-info-seq} for samer@36: irreducible stationary Markov chains (\ie that have a unique stationary samer@36: distribution). The derivation is greatly simplified by the dependency structure samer@36: of the Markov chain: for the purpose of the analysis, the `past' and `future' samer@41: segments $\past{X}_t$ and $\fut{X}_t$ can be collapsed to just the previous samer@46: and next variables $X_{t-1}$ and $X_{t+1}$ respectively. We also showed that samer@36: the predictive information rate can be expressed simply in terms of entropy rates: samer@36: if we let $a$ denote the $K\times K$ transition matrix of a Markov chain over samer@36: an alphabet of $\{1,\ldots,K\}$, such that samer@36: $a_{ij} = \Pr(\ev(X_t=i|X_{t-1}=j))$, and let $h:\reals^{K\times K}\to \reals$ be samer@36: the entropy rate function such that $h(a)$ is the entropy rate of a Markov chain samer@36: with transition matrix $a$, then the predictive information rate $b(a)$ is samer@36: \begin{equation} samer@36: b(a) = h(a^2) - h(a), samer@36: \end{equation} samer@36: where $a^2$, the transition matrix squared, is the transition matrix samer@36: of the `skip one' Markov chain obtained by jumping two steps at a time samer@36: along the original chain. samer@36: samer@36: Second and higher order Markov chains can be treated in a similar way by transforming samer@36: to a first order representation of the high order Markov chain. If we are dealing samer@36: with an $N$th order model, this is done forming a new alphabet of size $K^N$ samer@41: consisting of all possible $N$-tuples of symbols from the base alphabet. samer@41: An observation $\hat{x}_t$ in this new model encodes a block of $N$ observations samer@36: $(x_{t+1},\ldots,x_{t+N})$ from the base model. The next samer@41: observation $\hat{x}_{t+1}$ encodes the block of $N$ obtained by shifting the previous samer@36: block along by one step. The new Markov of chain is parameterised by a sparse $K^N\times K^N$ samer@41: transition matrix $\hat{a}$. Adopting the label $\mu$ for the order $N$ system, samer@41: we obtain: samer@36: \begin{equation} samer@41: h_\mu = h(\hat{a}), \qquad b_\mu = h({\hat{a}^{N+1}}) - N h({\hat{a}}), samer@36: \end{equation} samer@36: where $\hat{a}^{N+1}$ is the $(N+1)$th power of the first order transition matrix. samer@41: Other information measures can also be computed for the high-order Markov chain, including samer@41: the multi-information rate $\rho_\mu$ and the excess entropy $E$. These are identical samer@41: for first order Markov chains, but for order $N$ chains, $E$ can be up to $N$ times larger samer@41: than $\rho_\mu$. samer@43: samer@43: [Something about what kinds of Markov chain maximise $h_\mu$ (uncorrelated `white' samer@43: sequences, no temporal structure), $\rho_\mu$ and $E$ (periodic) and $b_\mu$. We return samer@43: this in \secrf{composition}.] samer@36: samer@36: hekeus@16: \section{Information Dynamics in Analysis} samer@4: samer@24: \begin{fig}{twopages} samer@33: \colfig[0.96]{matbase/fig9471} % update from mbc paper samer@33: % \colfig[0.97]{matbase/fig72663}\\ % later update from mbc paper (Keith's new picks) samer@24: \vspace*{1em} samer@24: \colfig[0.97]{matbase/fig13377} % rule based analysis samer@24: \caption{Analysis of \emph{Two Pages}. samer@24: The thick vertical lines are the part boundaries as indicated in samer@24: the score by the composer. samer@24: The thin grey lines samer@24: indicate changes in the melodic `figures' of which the piece is samer@24: constructed. In the `model information rate' panel, the black asterisks samer@24: mark the samer@24: six most surprising moments selected by Keith Potter. samer@24: The bottom panel shows a rule-based boundary strength analysis computed samer@24: using Cambouropoulos' LBDM. samer@24: All information measures are in nats and time is in notes. samer@24: } samer@24: \end{fig} samer@24: samer@36: \subsection{Musicological Analysis} samer@36: In \cite{AbdallahPlumbley2009}, methods based on the theory described above samer@36: were used to analysis two pieces of music in the minimalist style samer@36: by Philip Glass: \emph{Two Pages} (1969) and \emph{Gradus} (1968). samer@36: The analysis was done using a first-order Markov chain model, with the samer@36: enhancement that the transition matrix of the model was allowed to samer@36: evolve dynamically as the notes were processed, and was tracked (in samer@36: a Bayesian way) as a \emph{distribution} over possible transition matrices, samer@36: rather than a point estimate. The results are summarised in \figrf{twopages}: samer@36: the upper four plots show the dynamically evolving subjective information samer@36: measures as described in \secrf{surprise-info-seq} computed using a point samer@36: estimate of the current transition matrix, but the fifth plot (the `model information rate') samer@36: measures the information in each observation about the transition matrix. samer@36: In \cite{AbdallahPlumbley2010b}, we showed that this `model information rate' samer@36: is actually a component of the true IPI in samer@36: a time-varying Markov chain, which was neglected when we computed the IPI from samer@36: point estimates of the transition matrix as if the transition probabilities samer@36: were constant. samer@36: samer@36: The peaks of the surprisingness and both components of the predictive information samer@36: show good correspondence with structure of the piece both as marked in the score samer@36: and as analysed by musicologist Keith Potter, who was asked to mark the six samer@36: `most surprising moments' of the piece (shown as asterisks in the fifth plot)% samer@36: \footnote{% samer@36: Note that the boundary marked in the score at around note 5,400 is known to be samer@36: anomalous; on the basis of a listening analysis, some musicologists [ref] have samer@36: placed the boundary a few bars later, in agreement with our analysis.}. samer@36: samer@36: In contrast, the analyses shown in the lower two plots of \figrf{twopages}, samer@36: obtained using two rule-based music segmentation algorithms, while clearly samer@37: \emph{reflecting} the structure of the piece, do not \emph{segment} the piece, samer@37: with no tendency to peaking of the boundary strength function at samer@36: the boundaries in the piece. samer@36: samer@46: The complete analysis of \emph{Gradus} can be found in \cite{AbdallahPlumbley2009}, samer@46: but \figrf{metre} illustrates the result of a metrical analysis: the piece was divided samer@46: into bars of 32, 64 and 128 notes. In each case, the average surprisingness and samer@46: IPI for the first, second, third \etc notes in each bar were computed. The plots samer@46: show that the first note of each bar is, on average, significantly more surprising samer@46: and informative than the others, up to the 64-note level, where as at the 128-note, samer@46: level, the dominant periodicity appears to remain at 64 notes. samer@36: samer@24: \begin{fig}{metre} samer@33: % \scalebox{1}[1]{% samer@24: \begin{tabular}{cc} samer@33: \colfig[0.45]{matbase/fig36859} & \colfig[0.48]{matbase/fig88658} \\ samer@33: \colfig[0.45]{matbase/fig48061} & \colfig[0.48]{matbase/fig46367} \\ samer@33: \colfig[0.45]{matbase/fig99042} & \colfig[0.47]{matbase/fig87490} samer@24: % \colfig[0.46]{matbase/fig56807} & \colfig[0.48]{matbase/fig27144} \\ samer@24: % \colfig[0.46]{matbase/fig87574} & \colfig[0.48]{matbase/fig13651} \\ samer@24: % \colfig[0.44]{matbase/fig19913} & \colfig[0.46]{matbase/fig66144} \\ samer@24: % \colfig[0.48]{matbase/fig73098} & \colfig[0.48]{matbase/fig57141} \\ samer@24: % \colfig[0.48]{matbase/fig25703} & \colfig[0.48]{matbase/fig72080} \\ samer@24: % \colfig[0.48]{matbase/fig9142} & \colfig[0.48]{matbase/fig27751} samer@24: samer@24: \end{tabular}% samer@33: % } samer@24: \caption{Metrical analysis by computing average surprisingness and samer@24: informative of notes at different periodicities (\ie hypothetical samer@24: bar lengths) and phases (\ie positions within a bar). samer@24: } samer@24: \end{fig} samer@24: samer@46: \subsection{Content analysis/Sound Categorisation} samer@42: Using analogous definitions of differential entropy, the methods outlined samer@42: in the previous section are equally applicable to continuous random variables. samer@42: In the case of music, where expressive properties such as dynamics, tempo, samer@42: timing and timbre are readily quantified on a continuous scale, the information samer@42: dynamic framework thus may also be considered. peterf@39: samer@42: In \cite{Dubnov2006}, Dubnov considers the class of stationary Gaussian samer@42: processes. For such processes, the entropy rate may be obtained analytically samer@42: from the power spectral density of the signal, allowing the multi-information samer@42: rate to be subsequently obtained. Local stationarity is assumed, which may samer@42: be achieved by windowing or change point detection \cite{Dubnov2008}. %TODO samer@42: mention non-gaussian processes extension Similarly, the predictive information samer@42: rate may be computed using a Gaussian linear formulation CITE. In this view, samer@42: the PIR is a function of the correlation between random innovations supplied samer@42: to the stochastic process. %Dubnov, MacAdams, Reynolds (2006) %Bailes and samer@42: Dean (2009) peterf@39: samer@51: [ Continuous domain information ] samer@51: [Audio based music expectation modelling] samer@51: [ Gaussian processes] peterf@26: samer@4: samer@4: \subsection{Beat Tracking} samer@4: samer@43: A probabilistic method for drum tracking was presented by Robertson samer@43: \cite{Robertson11c}. The algorithm is used to synchronise a music samer@43: sequencer to a live drummer. The expected beat time of the sequencer is samer@43: represented by a click track, and the algorithm takes as input event samer@43: times for discrete kick and snare drum events relative to this click samer@43: track. These are obtained using dedicated microphones for each drum and samer@43: using a percussive onset detector (Puckette 1998). The drum tracker samer@43: continually updates distributions for tempo and phase on receiving a new samer@43: event time. We can thus quantify the information contributed of an event samer@43: by measuring the difference between the system's prior distribution and samer@43: the posterior distribution using the Kullback-Leiber divergence. samer@43: samer@43: Here, we have calculated the KL divergence and entropy for kick and samer@43: snare events in sixteen files. The analysis of information rates can be samer@43: considered \emph{subjective}, in that it measures how the drum tracker's samer@43: probability distributions change, and these are contingent upon the samer@43: model used as well as external properties in the signal. We expect, samer@43: however, that following periods of increased uncertainty, such as fills samer@43: or expressive timing, the information contained in an individual event samer@43: increases. We also examine whether the information is dependent upon samer@43: metrical position. samer@43: samer@4: samer@24: \section{Information dynamics as compositional aid} samer@43: \label{s:composition} samer@43: samer@53: The use of stochastic processes in music composition has been widespread for samer@53: decades---for instance Iannis Xenakis applied probabilistic mathematical models samer@53: to the creation of musical materials\cite{Xenakis:1992ul}. While such processes samer@53: can drive the \emph{generative} phase of the creative process, information dynamics samer@53: can serve as a novel framework for a \emph{selective} phase, by samer@53: providing a set of criteria to be used in judging which of the samer@53: generated materials samer@53: are of value. This alternation of generative and selective phases as been samer@53: noted by art theorist Margaret Boden \cite{Boden1990}. samer@53: samer@53: Information-dynamic criteria can also be used as \emph{constraints} on the samer@53: generative processes, for example, by specifying a certain temporal profile samer@53: of suprisingness and uncertainty the composer wishes to induce in the listener samer@53: as the piece unfolds. samer@53: %stochastic and algorithmic processes: ; outputs can be filtered to match a set of samer@53: %criteria defined in terms of information-dynamical characteristics, such as samer@53: %predictability vs unpredictability samer@53: %s model, this criteria thus becoming a means of interfacing with the generative processes. samer@53: samer@53: The tools of information dynamics provide a way to constrain and select musical samer@53: materials at the level of patterns of expectation, implication, uncertainty, and predictability. samer@53: In particular, the behaviour of the predictive information rate (PIR) defined in samer@53: \secrf{process-info} make it interesting from a compositional point of view. The definition samer@53: of the PIR is such that it is low both for extremely regular processes, such as constant samer@53: or periodic sequences, \emph{and} low for extremely random processes, where each symbol samer@53: is chosen independently of the others, in a kind of `white noise'. In the former case, samer@53: the pattern, once established, is completely predictable and therefore there is no samer@53: \emph{new} information in subsequent observations. In the latter case, the randomness samer@53: and independence of all elements of the sequence means that, though potentially surprising, samer@53: each observation carries no information about the ones to come. samer@53: samer@53: Processes with high PIR maintain a certain kind of balance between samer@53: predictability and unpredictability in such a way that the observer must continually samer@53: pay attention to each new observation as it occurs in order to make the best samer@53: possible predictions about the evolution of the seqeunce. This balance between predictability samer@53: and unpredictability is reminiscent of the inverted `U' shape of the Wundt curve (see \figrf{wundt}), samer@53: which summarises the observations of Wundt that the greatest aesthetic value in art samer@53: is to be found at intermediate levels of disorder, where there is a balance between samer@53: `order' and `chaos'. samer@53: samer@53: Using the methods of \secrf{markov}, we found \cite{AbdallahPlumbley2009} samer@53: a similar shape when plotting entropy rate againt PIR---this is visible in the samer@53: upper envelope of the scatter plot in \figrf{mtriscat}, which is a 3-D scatter plot of samer@53: three of the information measures discussed in \secrf{process-info} for several thousand samer@53: first-order Markov chain transition matrices generated by a random sampling method. samer@53: The coordinates of the `information space' are entropy rate ($h_\mu$), redundancy ($\rho_\mu$), and samer@53: predictive information rate ($b_\mu$). The points along the 'redundancy' axis correspond samer@53: to periodic Markov chains. Those along the `entropy' produce uncorrelated sequences samer@53: with no temporal structure. Processes with high PIR are to be found at intermediate samer@53: levels of entropy and redundancy. samer@53: These observations led us to construct the `Melody Triangle' as a graphical interface samer@53: for exploring the melodic patterns generated by each of the Markov chains represented samer@53: as points in \figrf{mtriscat}. samer@53: samer@43: \begin{fig}{wundt} samer@43: \raisebox{-4em}{\colfig[0.43]{wundt}} samer@43: % {\ \shortstack{{\Large$\longrightarrow$}\\ {\scriptsize\emph{exposure}}}\ } samer@43: {\ {\large$\longrightarrow$}\ } samer@43: \raisebox{-4em}{\colfig[0.43]{wundt2}} samer@43: \caption{ samer@43: The Wundt curve relating randomness/complexity with samer@43: perceived value. Repeated exposure sometimes results samer@43: in a move to the left along the curve \cite{Berlyne71}. samer@43: } samer@43: \end{fig} hekeus@45: hekeus@13: hekeus@45: %It is possible to apply information dynamics to the generation of content, such as to the composition of musical materials. hekeus@45: hekeus@45: %For instance a stochastic music generating process could be controlled by modifying hekeus@45: %constraints on its output in terms of predictive information rate or entropy hekeus@45: %rate. hekeus@45: hekeus@45: hekeus@13: samer@23: \subsection{The Melody Triangle} samer@23: samer@53: The Melody Triangle is an exploratory interface for the discovery of melodic samer@53: content, where the input---positions within a triangle---directly map to information samer@53: theoretic measures of the output. The measures---entropy rate, redundancy and samer@53: predictive information rate---form a criteria with which to filter the output samer@53: of the stochastic processes used to generate sequences of notes. These measures samer@53: address notions of expectation and surprise in music, and as such the Melody samer@53: Triangle is a means of interfacing with a generative process in terms of the samer@53: predictability of its output. samer@53: samer@53: The triangle is `populated' with first order Markov chain transition samer@53: matrices as illustrated in \figrf{mtriscat}. samer@53: samer@51: \begin{fig}{mtriscat} samer@51: \colfig{mtriscat} samer@34: \caption{The population of transition matrices distributed along three axes of samer@34: redundancy, entropy rate and predictive information rate (all measured in bits). samer@34: The concentrations of points along the redundancy axis correspond samer@34: to Markov chains which are roughly periodic with periods of 2 (redundancy 1 bit), samer@34: 3, 4, \etc all the way to period 8 (redundancy 3 bits). The colour of each point samer@34: represents its PIR---note that the highest values are found at intermediate entropy samer@34: and redundancy, and that the distribution as a whole makes a curved triangle. Although samer@51: not visible in this plot, it is largely hollow in the middle.} samer@51: \end{fig} samer@23: samer@43: The distribution of transition matrices plotted in this space forms an arch shape samer@42: that is fairly thin. It thus becomes a reasonable approximation to pretend that samer@42: it is just a sheet in two dimensions; and so we stretch out this curved arc into samer@42: a flat triangle. It is this triangular sheet that is our `Melody Triangle' and samer@42: forms the interface by which the system is controlled. Using this interface samer@46: thus involves a mapping to information space; a user selects a position within samer@51: the triangle, and a corresponding transition matrix is returned. samer@51: \Figrf{TheTriangle} shows how the triangle maps to different measures of redundancy, samer@42: entropy rate and predictive information rate. samer@41: samer@41: samer@42: Each corner corresponds to three different extremes of predictability and samer@42: unpredictability, which could be loosely characterised as `periodicity', `noise' samer@42: and `repetition'. Melodies from the `noise' corner have no discernible pattern; samer@42: they have high entropy rate, low predictive information rate and low redundancy. samer@42: These melodies are essentially totally random. A melody along the `periodicity' samer@42: to `repetition' edge are all deterministic loops that get shorter as we approach samer@42: the `repetition' corner, until it becomes just one repeating note. It is the samer@42: areas in between the extremes that provide the more `interesting' melodies. samer@42: These melodies have some level of unpredictability, but are not completely random. samer@42: Or, conversely, are predictable, but not entirely so. samer@41: samer@51: \begin{fig}{TheTriangle} samer@51: \colfig[0.9]{TheTriangle.pdf} samer@51: \caption{The Melody Triangle} samer@51: \end{fig} samer@41: hekeus@45: %PERHAPS WE SHOULD FOREGO TALKING ABOUT THE hekeus@45: %INSTALLATION VERSION OF THE TRIANGLE? hekeus@45: %feels a bit like a tangent, and could do with the space.. samer@42: The Melody Triangle exists in two incarnations; a standard screen based interface samer@42: where a user moves tokens in and around a triangle on screen, and a multi-user samer@42: interactive installation where a Kinect camera tracks individuals in a space and hekeus@45: maps their positions in physical space to the triangle. In the latter each visitor hekeus@45: that enters the installation generates a melody and can collaborate with their samer@42: co-visitors to generate musical textures---a playful yet informative way to hekeus@45: explore expectation and surprise in music. Additionally visitors can change the hekeus@45: tempo, register, instrumentation and periodicity of their melody with body gestures. samer@41: hekeus@45: As a screen based interface the Melody Triangle can serve as a composition tool. samer@42: A triangle is drawn on the screen, screen space thus mapped to the statistical hekeus@45: space of the Melody Triangle. A number of tokens, each representing a hekeus@45: melody, can be dragged in and around the triangle. For each token, a sequence of symbols with hekeus@45: statistical properties that correspond to the token's position is generated. These samer@51: symbols are then mapped to notes of a scale% samer@51: \footnote{However they could just as well be mapped to any other property, such samer@51: as intervals, chords, dynamics and timbres. It is even possible to map the samer@51: symbols to non-sonic outputs, such as colours. The possibilities afforded by samer@51: the Melody Triangle in these other domains remains to be investigated.}. hekeus@45: Additionally keyboard commands give control over other musical parameters. samer@23: samer@51: The Melody Triangle can generate intricate musical textures when multiple tokens samer@51: are in the triangle. Unlike other computer aided composition tools or programming samer@51: environments, here the composer engages with music on a high and abstract level; samer@51: the interface relating to subjective expectation and predictability. hekeus@45: hekeus@35: hekeus@35: hekeus@38: hekeus@38: \subsection{Information Dynamics as Evaluative Feedback Mechanism} hekeus@38: %NOT SURE THIS SHOULD BE HERE AT ALL..? hekeus@38: samer@46: \begin{fig}{mtri-results} samer@46: \def\scat#1{\colfig[0.42]{mtri/#1}} samer@46: \def\subj#1{\scat{scat_dwells_subj_#1} & \scat{scat_marks_subj_#1}} samer@46: \begin{tabular}{cc} samer@46: \subj{a} \\ samer@46: \subj{b} \\ samer@46: \subj{c} \\ samer@46: \subj{d} samer@46: \end{tabular} samer@46: \caption{Dwell times and mark positions from user trials with the samer@46: on-screen Melody Triangle interface. The left-hand column shows samer@46: the positions in a 2D information space (entropy rate vs multi-information rate samer@46: in bits) where spent their time; the area of each circle is proportional samer@46: to the time spent there. The right-hand column shows point which subjects samer@46: `liked'.} samer@46: \end{fig} hekeus@38: samer@42: Information measures on a stream of symbols can form a feedback mechanism; a hekeus@45: rudimentary `critic' of sorts. For instance symbol by symbol measure of predictive samer@42: information rate, entropy rate and redundancy could tell us if a stream of symbols samer@42: is currently `boring', either because it is too repetitive, or because it is too hekeus@45: chaotic. Such feedback would be oblivious to long term and large scale hekeus@45: structures and any cultural norms (such as style conventions), but hekeus@45: nonetheless could provide a composer with valuable insight on samer@42: the short term properties of a work. This could not only be used for the samer@42: evaluation of pre-composed streams of symbols, but could also provide real-time samer@42: feedback in an improvisatory setup. hekeus@38: hekeus@13: \section{Musical Preference and Information Dynamics} samer@42: We are carrying out a study to investigate the relationship between musical samer@42: preference and the information dynamics models, the experimental interface a samer@42: simplified version of the screen-based Melody Triangle. Participants are asked samer@42: to use this music pattern generator under various experimental conditions in a samer@42: composition task. The data collected includes usage statistics of the system: samer@42: where in the triangle they place the tokens, how long they leave them there and samer@42: the state of the system when users, by pressing a key, indicate that they like samer@42: what they are hearing. As such the experiments will help us identify any samer@42: correlation between the information theoretic properties of a stream and its samer@42: perceived aesthetic worth. hekeus@16: samer@46: Some initial results for four subjects are shown in \figrf{mtri-results}. Though samer@46: subjects seem to exhibit distinct kinds of exploratory behaviour, we have samer@46: not been able to show any systematic across-subjects preference for any particular samer@46: region of the triangle. samer@46: samer@46: Subjects' comments: several noticed the main organisation of the triangle: samer@46: repetative notes at the top, cyclic patters along the right edge, and unpredictable samer@46: notes towards the bottom left (a,c,f). Some did systematic exploration. samer@46: Felt that the right side was more `controllable' than the left (a,f)---a direct consequence samer@46: of their ability to return to a particular periodic pattern and recognise at samer@46: as one heard previously. Some (a,e) felt the trial was too long and became samer@46: bored towards the end. samer@46: One subject (f) felt there wasn't enough time to get to hear out the patterns properly. samer@46: One subject (b) didn't enjoy the lower region whereas another (d) said the lower samer@46: regions were more `melodic' and `interesting'. samer@4: hekeus@38: %\emph{comparable system} Gordon Pask's Musicolor (1953) applied a similar notion hekeus@38: %of boredom in its design. The Musicolour would react to audio input through a hekeus@38: %microphone by flashing coloured lights. Rather than a direct mapping of sound hekeus@38: %to light, Pask designed the device to be a partner to a performing musician. It hekeus@38: %would adapt its lighting pattern based on the rhythms and frequencies it would hekeus@38: %hear, quickly `learning' to flash in time with the music. However Pask endowed hekeus@38: %the device with the ability to `be bored'; if the rhythmic and frequency content hekeus@38: %of the input remained the same for too long it would listen for other rhythms hekeus@38: %and frequencies, only lighting when it heard these. As the Musicolour would hekeus@38: %`get bored', the musician would have to change and vary their playing, eliciting hekeus@38: %new and unexpected outputs in trying to keep the Musicolour interested. samer@4: hekeus@13: samer@4: \section{Conclusion} samer@51: We outlined our information dynamics approach to the modelling of the perception samer@51: of music. This approach models the subjective assessments of an observer that samer@51: updates its probabilistic model of a process dynamically as events unfold. We samer@51: outlined `time-varying' information measures, including a novel `predictive samer@51: information rate' that characterises the surprisingness and predictability of samer@51: musical patterns. samer@4: hekeus@45: samer@51: We have outlined how information dynamics can serve in three different forms of samer@51: analysis; musicological analysis, sound categorisation and beat tracking. hekeus@50: samer@51: We have described the `Melody Triangle', a novel system that enables a user/composer samer@51: to discover musical content in terms of the information theoretic properties of samer@51: the output, and considered how information dynamics could be used to provide samer@51: evaluative feedback on a composition or improvisation. Finally we outline a samer@51: pilot study that used the Melody Triangle as an experimental interface to help samer@51: determine if there are any correlations between aesthetic preference and information samer@51: dynamics measures. hekeus@50: hekeus@45: hekeus@44: \section{acknowledgments} samer@51: This work is supported by EPSRC Doctoral Training Centre EP/G03723X/1 (HE), hekeus@54: GR/S82213/01 and EP/E045235/1(SA), an EPSRC DTA Studentship (PF), an RAEng/EPSRC Research Fellowship 10216/88 (AR), an EPSRC Leadership Fellowship, EP/G007144/1 samer@51: (MDP) and EPSRC IDyOM2 EP/H013059/1. hekeus@44: samer@9: \bibliographystyle{unsrt} samer@43: {\bibliography{all,c4dm,nime,andrew}} samer@4: \end{document}