samer@4: \documentclass[conference]{IEEEtran} samer@4: \usepackage{cite} samer@4: \usepackage[cmex10]{amsmath} samer@4: \usepackage{graphicx} samer@4: \usepackage{amssymb} samer@4: \usepackage{epstopdf} samer@4: \usepackage{url} samer@4: \usepackage{listings} samer@9: \usepackage{tools} samer@9: samer@9: \let\citep=\cite samer@9: \def\squash{} samer@4: samer@4: %\usepackage[parfill]{parskip} samer@4: samer@4: \begin{document} samer@4: \title{Cognitive Music Modelling: an Information Dynamics Approach} samer@4: samer@4: \author{ samer@4: \IEEEauthorblockN{Samer A. Abdallah, Henrik Ekeus,} samer@4: \IEEEauthorblockN{Peter Foster and Mark D. Plumbley} samer@4: \IEEEauthorblockA{Centre for Digital Music\\ samer@4: Queen Mary University of London\\ samer@4: Mile End Road, London E1 4NS}} samer@4: samer@4: \maketitle samer@4: \abstract{People take in information when perceiving music. With it they continually build predictive models of what is going to happen. There is a relationship between information measures and how we perceive music. An information theoretic approach to music cognition is thus a fruitful avenue of research. samer@4: } samer@4: samer@4: samer@9: \section{Expectation and surprise in music} samer@9: \label{s:Intro} samer@9: samer@9: One of the more salient effects of listening to music is to create samer@9: \emph{expectations} of what is to come next, which may be fulfilled samer@9: immediately, after some delay, or not at all as the case may be. samer@9: This is the thesis put forward by, amongst others, music theorists samer@9: L. B. Meyer \cite{Meyer67} and Narmour \citep{Narmour77}. samer@9: In fact, %the gist of samer@9: this insight predates Meyer quite considerably; for example, samer@9: it was elegantly put by Hanslick \cite{Hanslick1854} in the samer@9: nineteenth century: samer@9: \begin{quote} samer@9: `The most important factor in the mental process which accompanies the samer@9: act of listening to music, and which converts it to a source of pleasure, samer@9: is %\ldots samer@9: frequently overlooked. We here refer to the intellectual satisfaction samer@9: which the listener derives from continually following and anticipating samer@9: the composer's intentions---now, to see his expectations fulfilled, and samer@9: now, to find himself agreeably mistaken. It is a matter of course that samer@9: this intellectual flux and reflux, this perpetual giving and receiving samer@9: takes place unconsciously, and with the rapidity of lightning-flashes.' samer@9: \end{quote} samer@9: samer@9: An essential aspect of this is that music is experienced as a phenomenon samer@9: that `unfolds' in time, rather than being apprehended as a static object samer@9: presented in its entirety. Meyer argued that musical experience depends samer@9: on how we change and revise our conceptions \emph{as events happen}, on samer@9: how expectation and prediction interact with occurrence, and that, to a samer@9: large degree, the way to understand the effect of music is to focus on samer@9: this `kinetics' of expectation and surprise. samer@9: samer@9: The business of making predictions and assessing surprise is essentially samer@9: one of reasoning under conditions of uncertainty and manipulating samer@9: degrees of belief about the various proposition which may or may not samer@9: hold, and, as has been argued elsewhere \cite{Cox1946,Jaynes27}, best samer@9: quantified in terms of Bayesian probability theory. samer@9: % Thus, we assume that musical schemata are encoded as probabilistic % samer@9: %\citep{Meyer56} models, and samer@9: Thus, we suppose that samer@9: when we listen to music, expectations are created on the basis of our samer@9: familiarity with various stylistic norms %, that is, using models that samer@9: encode the statistics of music in general, the particular styles of samer@9: music that seem best to fit the piece we happen to be listening to, and samer@9: the emerging structures peculiar to the current piece. There is samer@9: experimental evidence that human listeners are able to internalise samer@9: statistical knowledge about musical structure, \eg samer@9: \citep{SaffranJohnsonAslin1999,EerolaToiviainenKrumhansl2002}, and also samer@9: that statistical models can form an effective basis for computational samer@9: % analysis of music, \eg \cite{Pearce2005}. samer@9: analysis of music, \eg samer@9: \cite{ConklinWitten95,PonsfordWigginsMellish1999,Pearce2005}. samer@9: % \cite{Ferrand2002}. Dubnov and Assayag PSTs? samer@9: samer@9: \squash samer@9: \subsection{Music and information theory} samer@9: Given a probabilistic framework for music modelling and prediction, samer@9: it is a small step to apply quantitative information theory \cite{Shannon48} to samer@9: the models at hand. samer@9: The relationship between information theory and music and art in general has been the samer@9: subject of some interest since the 1950s samer@9: \cite{Youngblood58,CoonsKraehenbuehl1958,HillerBean66,Moles66,Meyer67,Cohen1962}. samer@9: The general thesis is that perceptible qualities and subjective samer@9: states like uncertainty, surprise, complexity, tension, and interestingness samer@9: are closely related to samer@9: information-theoretic quantities like entropy, relative entropy, samer@9: and mutual information. samer@9: % and are major determinants of the overall experience. samer@9: Berlyne \cite{Berlyne71} called such quantities `collative variables', since samer@9: they are to do with patterns of occurrence rather than medium-specific details, samer@9: and developed the ideas of `information aesthetics' in an experimental setting. samer@9: % Berlyne's `new experimental aesthetics', the `information-aestheticians'. samer@9: samer@9: % Listeners then experience greater or lesser levels of surprise samer@9: % in response to departures from these norms. samer@9: % By careful manipulation samer@9: % of the material, the composer can thus define, and induce within the samer@9: % listener, a temporal programme of varying samer@9: % levels of uncertainty, ambiguity and surprise. samer@9: samer@9: samer@9: Previous work in this area \cite{Berlyne74} treated the various samer@9: information theoretic quantities samer@9: such as entropy as if they were intrinsic properties of the stimulus---subjects samer@9: were presented with a sequence of tones with `high entropy', or a visual pattern samer@9: with `low entropy'. These values were determined from some known `objective' samer@9: probability model of the stimuli,% samer@9: \footnote{% samer@9: The notion of objective probabalities and whether or not they can samer@9: usefully be said to exist is the subject of some debate, with advocates of samer@9: subjective probabilities including de Finetti \cite{deFinetti}. samer@9: Accordingly, we will treat the concept of a `true' or `objective' probability samer@9: models with a grain of salt and not rely on them in our samer@9: theoretical development.}% samer@9: % since probabilities are almost always a function of the state of knowledge of the observer samer@9: or from simple statistical analyses such as samer@9: computing emprical distributions. Our approach is explicitly to consider the role samer@9: of the observer in perception, and more specifically, to consider estimates of samer@9: entropy \etc with respect to \emph{subjective} probabilities. samer@9: % !!REV - DONE - explain use of quoted `objective' samer@9: samer@9: % !!REV - previous work on information theory in music samer@9: More recent work on using information theoretic concepts to analyse music in samer@9: includes Simon's \cite{Simon2005} assessments of the entropy of samer@9: Jazz improvisations and Dubnov's samer@9: \cite{Dubnov2006,DubnovMcAdamsReynolds2006,Dubnov2008} samer@9: investigations of the `information rate' of musical processes, which is related samer@9: to the notion of redundancy in a communications channel. samer@9: Dubnov's work in particular is informed by similar concerns to our own samer@9: and we will discuss the relationship between it and our work at samer@9: several points later in this paper samer@9: (see \secrf{Redundancy}, \secrf{methods} and \secrf{RelatedWork}). samer@9: samer@9: samer@9: % !!REV - DONE - rephrase, check grammar (now there are too many 'one's!) samer@9: \squash samer@9: \subsection{Information dynamic approach} samer@9: samer@9: Bringing the various strands together, our working hypothesis is that samer@9: as a listener (to which will refer gender neutrally as `it') samer@9: listens to a piece of music, it maintains a dynamically evolving statistical samer@9: model that enables it to make predictions about how the piece will samer@9: continue, relying on both its previous experience of music and the immediate samer@9: context of the piece. samer@9: As events unfold, it revises its model and hence its probabilistic belief state, samer@9: which includes predictive distributions over future observations. samer@9: These distributions and changes in distributions can be characterised in terms of a handful of information samer@9: theoretic-measures such as entropy and relative entropy. samer@9: % to measure uncertainty and information. %, that is, changes in predictive distributions maintained by the model. samer@9: By tracing the evolution of a these measures, we obtain a representation samer@9: which captures much of the significant structure of the samer@9: music. samer@9: This approach has a number of features which we list below. samer@9: samer@9: (1) \emph{Abstraction}: samer@9: Because it is sensitive mainly to \emph{patterns} of occurence, samer@9: rather the details of which specific things occur, samer@9: it operates at a level of abstraction removed from the details of the sensory samer@9: experience and the medium through which it was received, suggesting that the samer@9: same approach could, in principle, be used to analyse and compare information samer@9: flow in different temporal media regardless of whether they are auditory, samer@9: visual or otherwise. samer@9: samer@9: (2) \emph{Generality}: samer@9: This approach does not proscribe which probabilistic models should be used---the samer@9: choice can be guided by standard model selection criteria such as Bayes samer@9: factors \cite{KassRaftery1995}, \etc samer@9: samer@9: (3) \emph{Richness}: samer@9: It may be effective to use a model with time-dependent latent samer@9: variables, such as a hidden Markov model. In these cases, we can track changes samer@9: in beliefs about the hidden variables as well as the observed ones, adding samer@9: another layer of richness to the description while maintaining the same samer@9: level of abstraction. samer@9: For example, harmony (\ie, the `current chord') in music is not stated explicitly, but rather samer@9: must be inferred from the musical surface; nonetheless, a sense of harmonic samer@9: progression is an important aspect of many styles of music. samer@9: samer@9: (4) \emph{Subjectivity}: samer@9: Since the analysis is dependent on the probability model the observer brings to the samer@9: problem, which may depend on prior experience or other factors, and which may change samer@9: over time, inter-subject variablity and variation in subjects' responses over time are samer@9: fundamental to the theory. It is essentially a theory of subjective response samer@9: samer@9: % !!REV - clarify aims of paper. samer@9: Having outlined the basic ideas, our aims in pursuing this line of thought samer@9: are threefold: firstly, to propose dynamic information-based measures which samer@9: are coherent from a theoretical point of view and consistent with the general samer@9: principles of probabilistic inference, with possible applications in samer@9: regulating machine learning systems; samer@9: % when heuristics are required to manage intractible models or limited computational resources. samer@9: secondly, to construct computational models of what human brains are doing samer@9: in response to music, on the basis that our brains implement, or at least samer@9: approximate, optimal probabilistic inference under the relevant constraints; samer@9: and thirdly, to construct a computational model of a certain restricted samer@9: field of aesthetic judgements (namely judgements related to formal structure) samer@9: that may shed light on what makes a stimulus interesting or aesthetically samer@9: pleasing. This would be of particular relevance to understanding and samer@9: modelling the creative process, which often alternates between generative samer@9: and selective or evaluative phases \cite{Boden1990}, and would have samer@9: applications in tools for computer aided composition. samer@9: samer@4: \section{Information Dynamics Approach} samer@4: samer@4: \subsection{Re-iterate core hypothesis} samer@4: samer@4: \subsection{models/parameters/observations} samer@4: The grouping of elements into past, present and future..s samer@4: \subsection{Information measures} samer@4: Predictive information rate as a measure of structure samer@4: Cruchfield papers, anatomy of abit samer@4: \subsection{Case of this approach being good at modelling music cognition} samer@4: Inverted U samer@4: \section{Applications} samer@4: \subsection{In Analysis} samer@4: refer to the work with the analysis of minimalist pieces samer@4: samer@4: Content analysis - Sound Categorisation. Using Information Dynamics it is possible to segment music. From there we can then use this to search large data sets. Determine musical structure for the purpose of playlist navigation and search. (Peter) samer@4: samer@4: \subsection{Beat Tracking} samer@4: Bayesian belief can be used to predict when things happen (as oppose to just what happens). Information Dynamics of? samer@4: samer@4: samer@4: samer@4: \subsection{Information Dynamics as Design Tool } samer@4: \subsubsection{The Melody Triangle} samer@4: \emph{What the Melody Triangle is\dots} samer@4: samer@4: samer@4: \emph{The Melody Triangle as Composition Assistant\dots} samer@4: samer@4: \emph{comparable tools} The use of stochastic processes for the generation of musical material has been widespread for decades. Just as Information Theory was coming of age Iannis Xenakis applied probabilistic mathematical models to the creation of musical materials. This included the formulation of a theory of Markovian Stochastic Music. With the Melody Triangle similar processes generate the content, however we are able to explore and interface with these processes at the high and abstract level of expectation, randomness and predictability. samer@4: samer@4: \emph{Using the Melody Triangle for the generation of non-sonic content (maybe)} samer@4: samer@4: \subsection{Information Dynamics as Evaluative Feedback Mechanism} samer@4: samer@4: samer@4: \emph{comparable system} Gordon Pask's Musicolor (1953) applied a similar notion of boredom in its design. samer@4: The Musicolour would react to audio input through a microphone by flashing coloured lights. Rather than a direct mapping of sound to light, Pask designed the device to be a partner to a performing musician. It would adapt its lighting pattern based on the rhythms and frequencies it would hear, quickly `learning' to flash in time with the music. However Pask endowed the device with the ability to `be bored'; if the rhythmic and frequency content of the input remained the same for too long it would listen for other rhythms and frequencies, only lighting when it heard these. As the Musicolour would `get bored', the musician would have to change and vary their playing, eliciting new and unexpected outputs in trying to keep the Musicolour interested. samer@4: samer@4: In a similar vain, our \emph{Information Dynamics Critic}(name?) allows for an evaluative measure of an input stream, however containing a more sophisticated notion of boredom that \dots samer@4: samer@4: \subsection{Musical Preference and Information Dynamics} samer@4: Any results from this study samer@4: \section{Conclusion} samer@4: samer@9: \bibliographystyle{unsrt} samer@9: {\bibliography{all,c4dm}} samer@4: \end{document}