Mercurial > hg > mtridoc

Binary file nime2012/mtriange.pdf has changed
--- a/nime2012/mtriange.tex	Sun Feb 05 19:09:12 2012 +0000
+++ b/nime2012/mtriange.tex	Mon Feb 06 12:10:30 2012 +0000
@@ -60,7 +60,7 @@
 \begin{document}
 \maketitle
 \begin{abstract}
-The Melody Triangle is a Markov-chain based melody generator where the input - positions within a triangle - directly map to information theoretic measures of its output.  The measures are the entropy rate, redundancy and \emph{predictive information rate}\cite{Abdallah} in the melody. Predictive information rate is a \emph{time-varying} information measure developed as part of the Information Dynamics of Music project(IDyOM)\footnote{http://www.idyom.org/}.  It characterises temporal structure and is a way of modelling expectation and surprise in the perception of music.
+The Melody Triangle is a Markov-chain based melody generator where the input - positions within a triangle - directly map to information theoretic measures of its output.  The measures are the entropy rate, redundancy and \emph{predictive information rate}\cite{Abdallah:2009p4089} in the melody. Predictive information rate is a \emph{time-varying} information measure developed as part of the Information Dynamics of Music project(IDyOM)\footnote{http://www.idyom.org/}.  It characterises temporal structure and is a way of modelling expectation and surprise in the perception of music.

 We describe the Information Dynamics model and how it forms the basis of the Melody Triangle.  We outline two interfaces and uses of the system.  The first is a multi-user installation where collaboration in a performative setting provides a playful yet informative way to explore expectation and surprise in music.  The second is a screen based interface where the Melody Triangle becomes a compositional tool for the generation of intricate musical textures using an abstract, high-level description of predictability. Finally we outline a study where the screen-based interface was used under experimental conditions to determine how the three measures of predictive information rate, entropy and redundancy might relate to musical preference. We found that\dots

@@ -71,7 +71,7 @@

  Music involves patterns in time.  When listening to music we continually build and re-evaluate expectations of what is to come next.  Composers commonly, consciously or not, play with this expectation by setting up expectations which may, or may not be fulfilled.  This manipulation of expectation and surprise in the listener has been articulated by [ref]. [little more background on expectation]

-The research into Information Dynamics explores several different kinds of predictability in musical patterns, how human listeners might perceive these, and how they shape or affect the listening experience. [more on IDyOM project]
+The research into Information Dynamics explores several different kinds of predictability in musical patterns, how human listeners might perceive these, and how they shape or affect the listening experience. [more on IDyOM project]


 \section{The Melody Triangle}
@@ -79,7 +79,7 @@
 The Melody Triangle is based on first order Markov chains, represented as transition matrixes, that generate streams of symbols.  By mapping the symbols to individual notes, melodies are generated.  Further by layering these streams of notes can result in intricate musical textures. The choice of notes or scale is not a part of the Melody Triangle's core functionality, in fact the symbols could be mapped to anything, even non sonic outputs.

 Any sequence of symbols can be analysed and information theoretic measures taken from it.  The novelty of the Melody Triangle lies in that we go 'backwards' - given desired values for these measures, as determined from the user interface, we return a stream of symbols that match those measures.  The information measures used are redundancy, entropy rate and predictive information rate.
-
+
 \subsection{Information measures}
 \subsubsection{Redundancy}
 [todo - a more formal description]
@@ -90,7 +90,7 @@

 \subsubsection{Predictive Information Rate}
 [todo - a more formal description]
-Predictive information rate tell us the average reduction in uncertainty upon perceiving a symbol; a system with high predictive information rate means that each symbol tells you more about the next one.  If we imagine a purely periodic sequence, each symbol tells you nothing about the next one that we didn't already know as we already know how the pattern is going.  Similarly with a seemingly uncorrelated sequence,  seeing the next symbol does not tell us anymore because they are completely independent anyway; there is no pattern.   There is a subset of transition matrixes that have high predictive information rate, and it is neither the periodic ones, nor the completely un-corellated ones.  Rather they tend to yield output that have certain characteristic patterns, however a listener can't necessarily know when they occur.  However a certain sequence of symbols might tell us about which one of the characteristics patterns will show up next.  Each symbols tell a us little bit about the future but nothing about the infinite future, we only learn about that as time goes on; there is continual building of prediction.
+Predictive information rate tell us the average reduction in uncertainty upon perceiving a symbol; a system with high predictive information rate means that each symbol tells you more about the next one.  If we imagine a purely periodic sequence, each symbol tells you nothing about the next one that we didn't already know as we already know how the pattern is going.  Similarly with a seemingly uncorrelated sequence,  seeing the next symbol does not tell us anymore because they are completely independent anyway; there is no pattern.   There is a subset of transition matrixes that have high predictive information rate, and it is neither the periodic ones, nor the completely un-corellated ones.  Rather they tend to yield output that have certain characteristic patterns, however a listener can't necessarily know when they occur.  However a certain sequence of symbols might tell us about which one of the characteristics patterns will show up next.  Each symbols tell a us little bit about the future but nothing about the infinite future, we only learn about that as time goes on; there is continual building of prediction.


@@ -116,10 +116,12 @@
 \caption{The Melody Triangle [todo fix shading to be more triangular]  \label{TheTriangle}}
 \end{figure}

-\subsection{Making the triangle}
We generate hundreds of transition matrixes, representing first-order Markov chains, by a random sampling method.  These are then plotted in a 3d statistical space of redundancy, entropy rate and predictive information rate.  In figure \ref{InfoDynEngine} we see a representation of how these matrixes are distributed; each one of these points corresponds to a transition matrix.
+\subsection{Making the triangle}
+We generate hundreds of transition matrixes, representing first-order Markov chains, by a random sampling method.  These are then plotted in a 3d statistical space of redundancy, entropy rate and predictive information rate.  In figure \ref{InfoDynEngine} we see a representation of how these matrixes are distributed; each one of these points corresponds to a transition matrix.


-
When we look at the distribution of randomly generated transition matrixes and plotted in this space, we see that it forms an arch shape that is fairly thin.  It thus becomes a reasonable approximation to pretend that it is just a sheet in two dimensions; and so we stretch out this curved arc into a flat triangle.  It is this triangular sheet that is our 'Melody Triangle' and forms the interface by which the system is controlled.
+
+When we look at the distribution of randomly generated transition matrixes and plotted in this space, we see that it forms an arch shape that is fairly thin.  It thus becomes a reasonable approximation to pretend that it is just a sheet in two dimensions; and so we stretch out this curved arc into a flat triangle.  It is this triangular sheet that is our 'Melody Triangle' and forms the interface by which the system is controlled.


@@ -144,13 +146,14 @@
 As a Kinect camera overlooks a space, its the range naturally forms a triangle.  As visitors/users comes into the range of the camera, they start generating a melody, the statistical properties of this melody determined by the mapping of physical space to statistical space as discussed above.  Thus by exploring the physical space the participant explores the predictability of the generated melodic content.  When multiple people are in the space, they can cooperate to create musical polyphonic textures.

 The streams of symbols are mapped to MIDI and then played with software instruments in Logic.  The tracking system was capable of detecting gestures, and these were mapped to different musical effects such as tempo changes, periodicity changes (going to the off-beat), instrument/register changes and volume (see Figure \ref{gestures}).
-
+
 \subsubsection{Tracking and Control}

 Tracking and control was done using the OpenNI libraries' API and high level middle-ware for tracking with Kinect.  This provided reliable blob tracking of humanoid forms in 2d space.  By triangulating this to the Kinect's depth map it became possible to get reliable coordinate of visitors positions in the space.
-
This system was extended to detect gestures.  By detecting the bounding box of the 2d blobs of individuals in the space, and then normalising these based on the distance of the depth map it became possible to work out if an individual had an arm stretched out or if they were crouching.

-With this it was possible to define a series of gestures for controlling the system without the use of any controllers(see table \ref{gestures}).  Thus for instance by sticking out one's left arm quickly, the melody doubles in tempo.  By pulling one's left arm in at the same time as sticking the right arm out the melody would shift onto the offbeat.   Sending out both arms would change instrument.
+This system was extended to detect gestures.  By detecting the bounding box of the 2d blobs of individuals in the space, and then normalising these based on the distance of the depth map it became possible to work out if an individual had an arm stretched out or if they were crouching.
+
+With this it was possible to define a series of gestures for controlling the system without the use of any controllers(see table \ref{gestures}).  Thus for instance by sticking out one's left arm quickly, the melody doubles in tempo.  By pulling one's left arm in at the same time as sticking the right arm out the melody would shift onto the offbeat.   Sending out both arms would change instrument.

 \begin{table}
 \centering
@@ -173,9 +176,10 @@
 Although visitors would need an initial bit of training they could then quickly be made to collaboratively design musical textures.  For example, one person could lay down a predictable repeating bass line by keeping themselves to the periodicity/repetition side of the room, while a companion can generate a freer melodic line by being nearer the 'noise' part of the space.


-
The collaborative nature of this installation is one area that merits attention.  By not having one user be able to control the whole narrative, the participants would communicate verbally and direct each other in the goals of learning to use the system, and eventually towards finding interesting musical textures.  The collaborative nature added an element of playfulness and enjoyment that was obviously apparent.

-As an artefact this installation is an exploratory prototype, and occupies an ambiguous role in terms of purpose; it is in a nebulous middle ground between instrument and art installation[, and could also form a framework for a kind of dance performance].  One thing is clear is that as a vehicle for communicating ideas related to the expectation, pattern and predictability in music it is very effective.
+The collaborative nature of this installation is one area that merits attention.  By not having one user be able to control the whole narrative, the participants would communicate verbally and direct each other in the goals of learning to use the system, and eventually towards finding interesting musical textures.  The collaborative nature added an element of playfulness and enjoyment that was obviously apparent.
+
+As an artefact this installation is an exploratory prototype, and occupies an ambiguous role in terms of purpose; it is in a nebulous middle ground between instrument and art installation[, and could also form a framework for a kind of dance performance].  One thing is clear is that as a vehicle for communicating ideas related to the expectation, pattern and predictability in music it is very effective.

 \subsection{The Screen Based Interface}

@@ -212,6 +216,6 @@

 \section{acknowledgments}
 This work is supported by EPSRC Doctoral Training Centre EP/G03723X/1 (HE), GR/S82213/01 and EP/E045235/1(SA), an EPSRC Leadership Fellowship, EP/G007144/1 (MDP) and EPSRC IDyOM2 EP/H013059/1.
-\bibliographystyle{plain}
-{\bibliography{thebib}}
+\bibliographystyle{abbrv}
+\bibliography{nime}
 \end{document}
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/nime2012/nime.bib	Mon Feb 06 12:10:30 2012 +0000
@@ -0,0 +1,10 @@
+@article{Abdallah:2009p4089,
+author = {Abdallah, S and Plumbley, M},
+title = {{Information dynamics: patterns of expectation and surprise in the perception of music}},
+journal = {Connection Science},
+year = {2009},
+volume = {21},
+number = {2},
+pages = {89--117}
+}
+