# HG changeset patch # User Henrik Ekeus # Date 1328376454 0 # Node ID 2f88a91ad01fd88d634ea5424a0b70a66b9b2a8c # Parent 5e2c50450acefbb0373aeb229cf548ca5eaf042c First pass over multi-user instrument section diff -r 5e2c50450ace -r 2f88a91ad01f nime2012/mtriange.pdf Binary file nime2012/mtriange.pdf has changed diff -r 5e2c50450ace -r 2f88a91ad01f nime2012/mtriange.tex --- a/nime2012/mtriange.tex Sat Feb 04 15:08:36 2012 +0000 +++ b/nime2012/mtriange.tex Sat Feb 04 17:27:34 2012 +0000 @@ -63,7 +63,7 @@ \centering \includegraphics[width=0.2\textwidth]{PeriodicMatrix.png} \includegraphics[width=0.2\textwidth]{NonDeterministicMatrix_bw.png} -\caption{Two transition matrixes. The shade of white represents the probabilities of transition from one symbol to the next (black=0, white=1). The current symbol is along the bottom, and in this case there are twelve possibilities (mapped to a chromatic scale). The left hand matrix has no uncertainty; it represents a periodic pattern. We can see for example that symbol 4 must follow symbol 3, and that symbol 10 must follow symbol 4, and so on.The right hand matrix contains unpredictability but nonetheless is not completely without perceivable structure, it is of a higher entropy rate. \label{TransitionMatrixes}} +\caption{Two transition matrixes. The shade of white represents the probabilities of transition from one symbol to the next (black=0, white=1). The current symbol is along the bottom, and in this case there are twelve possibilities (mapped to a chromatic scale). The left hand matrix has no uncertainty; it represents a periodic pattern. The right hand matrix contains unpredictability but nonetheless is not completely without perceivable structure, it is of a higher entropy rate. \label{TransitionMatrixes}} \end{figure} @@ -73,7 +73,7 @@ \begin{figure} \centering \includegraphics[width=0.5\textwidth]{MatrixDistribution.png} -\caption{The population of transition matrixes distributed along three axes of redundancy, entropy rate and predictive -information rate. Note how the distribution makes triangle-like plane floating in 3d space. \label{InfoDynEngine}} +\caption{The population of transition matrixes distributed along three axes of redundancy, entropy rate and predictive -information rate. Note how the distribution makes a curved triangle-like plane floating in 3d space. \label{InfoDynEngine}} \end{figure} \begin{figure}[h] \centering @@ -88,55 +88,48 @@ - When the Melody Triangle is used, regardless of whether it is as a screen based system, or as an interactive installation, it involves a mapping to this statistical space. As can be seen in figure \ref{TheTriangle}, a position within the triangle maps to different measures of redundancy, entropy rate and predictive information rate. - - [todo improve, include example melodies?] Each corner corresponds to three different extremes of predictability and unpredictability, which could be loosely characterised as periodicity, noise and repetition. Melodies from the 'noise' corner have no discernible pattern; they have high entropy rate, low predictive information rate and low redundancy. These melodies are essentially totally random. A melody along the 'periodicity' to 'repetition' edge are all deterministic loops that get shorter as we approach the 'repetition' corner, until it becomes just one repeating note. It is the areas in between that provide the more interesting melodies, those that have some level of unpredictability, but are not completely random and conversely that are predictable, but not entirely so. This triangular space allows for an intuitive exploration of expectation and surprise in temporal sequences. - - - + When the Melody Triangle is used, regardless of whether it is as a screen based system, or as an interactive installation, it involves a mapping to this statistical space. Then a transition matrix corresponding to this position in statistical space is returned. As can be seen in figure \ref{TheTriangle}, a position within the triangle maps to different measures of redundancy, entropy rate and predictive information rate. + +%%%paragraph explaining what the different parts of the triangle are like. + [todo improve, include example melodies?] Each corner corresponds to three different extremes of predictability and unpredictability, which could be loosely characterised as periodicity, noise and repetition. Melodies from the 'noise' corner have no discernible pattern; they have high entropy rate, low predictive information rate and low redundancy. These melodies are essentially totally random. A melody along the 'periodicity' to 'repetition' edge are all deterministic loops that get shorter as we approach the 'repetition' corner, until it becomes just one repeating note. It is the areas in between that provide the more interesting melodies, those that have some level of unpredictability, but are not completely random and conversely that are predictable, but not entirely so. This triangular space allows for an intuitive exploration of expectation and surprise in temporal sequences based on a simple model of how one might guess the next event given the previous one. \section{User Interfaces} -The Melody Triangle engine, developed in Prolog and MatLab, can be controlled by OSC messages and thus any number of interfaces could be developed to for it. Currently two different interfaces exist; a standard screen based interface where a user moves tokens with a mouse in and around a triangle on screen, and a multi-user interactive installation where a Kinnect camera tracks individuals in a space and maps their positions in the space to the triangle. +The Melody Triangle engine\footnote{developed in Prolog and MatLab} is controlled over OSC messages and thus any number of interfaces could be developed to for it. Currently two different interfaces exist; a standard screen based interface where a user moves tokens with a mouse in and around a triangle on screen, and a multi-user interactive installation where a Kinect\footnote{http://www.xbox.com/en-GB/Kinect} camera tracks individuals in a space and maps their positions in the space to the triangle. \subsection{The Multi-User Installation} \begin{figure} \centering \includegraphics[width=0.5\textwidth]{kinnect.pdf} -\caption{The depth map as seen by the Kinnect, and the bounding box outlines the blobs detected by OpenNI.\label{Kinnect}} +\caption{The depth map as seen by the Kinect, and the bounding box outlines the blobs detected by OpenNI.\label{Kinect}} \end{figure} -[old text - rewrite!] +As a Kinect camera overlooks a space, its the range naturally forms a triangle. As visitors/users comes into the range of the camera, they start generating a melody, the statistical properties of this melody determined by the mapping of physical space to statistical space as discussed above. Thus by exploring the physical space the participant explores the predictability of the generated melodic content. When multiple people are in the space, they can cooperate to create musical polyphonic textures. - the statistical properties of this melody is based on where in the physical room the participant is standing, as this is mapped to a statistical space (see below). By exploring the physical space the participants thus explore the predictability of the melodic and rhythmical patterns, based on a simple model of how might guess the next musical event given the previous one. -\dots +The streams of symbols are mapped to MIDI and then played with software instruments in Logic. The tracking system was capable of detecting gestures, and these were mapped to different musical effects such as tempo changes, periodicity changes (going to the off-beat), instrument/register changes and volume (see Figure \ref{gestures}). + +\subsubsection{Tracking and Control} - -When multiple people are in the space, they can cooperate to create musical polyphonic textures. For example, one person could lay down a predictable repeating bass line by keeping themselves to the periodicity/repetition side of the room, while a companion can generate a freer melodic line by being nearer the 'noise' part of the space. - -A Kinnect camera was used to tack individuals in the space, an application developed in OpenFrameworks would send the individuals positions and a bounding box values (for gesture recognition), to an application running the Information Dynamics Engine [Matlab/Prolog]. This would map the input to a series of symbols that would go over OSC to Max/MSP, where these would be interpreted as MIDI, mapped to channels and the data passed on to Logic for output using software instruments. -\subsubsection{Tracking and Control} -It uses the OpenNI libraries' API and high level middle-ware for tracking with Kinnect. This provided reliable blob tracking of humanoid forms in 2d space. By then triangulating this to the Kinnect's depth map it became possible to get reliable coordinate of visitors positions in the space. - - - - This system was then further extended to detect gestures. By detecting the bounding box of the 2d blobs of individuals in the space, and then normalising these based on the distance of the depth map it became possible to work out if an individual had a an arm stretched out or if they were crouching. +Tracking and control was done using the OpenNI libraries' API and high level middle-ware for tracking with Kinect. This provided reliable blob tracking of humanoid forms in 2d space. By triangulating this to the Kinect's depth map it became possible to get reliable coordinate of visitors positions in the space. + This system was extended to detect gestures. By detecting the bounding box of the 2d blobs of individuals in the space, and then normalising these based on the distance of the depth map it became possible to work out if an individual had an arm stretched out or if they were crouching. With this it was possible to define a series of gestures for controlling the system without the use of any controllers. Thus for instance by sticking out one's left arm quickly, the melody doubles in tempo. By pulling one's left arm in at the same time as sticking the right arm out the melody would shift onto the offbeat. Sending out both arms would change instrument. \begin{figure} \centering \includegraphics[width=0.5\textwidth]{InstructionsText.pdf} -%\caption{\label{Kinnect}} -\end{figure} -\begin{figure} -\centering -\includegraphics[width=1.0\textwidth]{InstructionsImage.pdf} -\caption{Control Gestures\label{Kinnect}} +\caption{Gestures and their resulting effect. For instance sending one's left arm out while keeping the right static would double the tempo of the melody being generated.\label{gestures}} \end{figure} +\subsubsection{Observations} +Although visitors would need an initial bit of training they could then quickly be made to collaboratively design musical textures. For example, one person could lay down a predictable repeating bass line by keeping themselves to the periodicity/repetition side of the room, while a companion can generate a freer melodic line by being nearer the 'noise' part of the space. + + + The collaborative nature of this installation is one area that merits attention. By not having one user be able to control the whole narrative, the participants would communicate verbally and direct each other in the goals of learning to use the system, and eventually towards finding interesting musical textures. The collaborative nature added an element of playfulness and enjoyment that was obviously apparent. + +As an artefact this installation is an exploratory prototype, and occupies an ambiguous role in terms of purpose; it is in a nebulous middle ground between instrument and art installation[, and could also form a framework for a kind of dance performance]. One thing is clear is that as a vehicle for communicating ideas related to the expectation, pattern and predictability in music it is very effective. \subsection{The Screen Based Interface} [todo]