changeset 7:ba4e684e5a41

A number of updates and reshuffles

--- a/cannam.tex	Thu Sep 22 09:47:41 2011 +0100
+++ b/cannam.tex	Thu Sep 22 10:36:09 2011 +0100
@@ -77,7 +77,7 @@
 \subsection{Reproducible Research}
 \label{subsec:rr}
 
-Some researchers have come to realize that the traditional methods of
+Some researchers have come to realize that traditional methods of
 disseminating research outputs are no longer sufficient for
 computational science research.  Dohono and colleagues at Stanford
 have, since the mid 1990’s, aimed to carry out ``Reproducible
@@ -104,102 +104,92 @@
 \subsection{Real-world limitations on software practice}
 \label{subsec:researchsoft}
 
- ``A recent study found that the majority of scientists use desktop
- computers most of the time for developing and using scientific
- software. Further, they found that there was a great deal of variation
-@@ -90,6 +82,7 @@
- concepts by scientists, and that for developing and using scientific
- software, informal self-study or learning from peers was commonplace''
- \cite{gwilson2009}
+Although the Reproducible Research principle provides a comprehensive
+solution to the problem of code dissemination, in practice take-up in
+this field appears limited.  In our survey of UK audio and music
+researchers (see section \ref{sec:survey}), the majority of
+respondents said either that they took no steps to ensure
+reproducibility, or that they only made code or data available on
+request.  Obstacles cited included lack of time, copyright
+restrictions, and the potential for commercial use of the code
+\cite{ssamrsurvey}.  In addition to these, a broader case study by the
+UK Research Information Network into science research across several
+subject areas also identified lack of evidence of benefits, cultures
+of independence and competition, and concerns about quality as typical
+factors inhibiting open sharing of data and code
+\cite{rin2010}. Intuitively, undertaking reproducible research takes
+effort early in the research cycle, before the benefits are
+necessarily apparent and while the value of the reserch is still
+unclear, and can be perceived as delaying the production of ``real''
+research. Once research results have been produced and a paper
+written, there is little apparent incentive to make the research
+reproducible.
 
+In many of the fields within this community, researchers lack the
+skills or desire to write their own code or to make someone else's
+code work, and where they do write their own code, they work on
+different platforms and use a wide variety of batch and real-time
+environments.  A study in 2009 found that scientists typically
+developed and used software on their personal computers rather than
+dedicated servers, that there was a great deal of variation in the
+level of understanding of standard software engineering concepts by
+scientists, and that for developing and using scientific software,
+informal self-study or learning from peers was commonplace
+\cite{gwilson2009}.  Our survey found MATLAB and numerous MATLAB
+toolboxes, C++, Max/MSP, OpenFrameworks, Juce, HTK and MPTK,
+SuperCollider, Clojure and R amongst environments and toolkits used
+\cite{ssamrsurvey}, and recent publications from our group have also
+used Python \cite{fazekas} and Prolog \cite{raimond}.
 
-In many of the
-fields within this community, researchers do not have the skills or
-desire to write their own code or to piece together other code, but
-benefit greatly from ``out of the box'' software. Even where they do
-write their own code, they work on different platforms and use a wide
-variety of batch and real-time environments.  Our survey found MATLAB,
-C++, Max/MSP, OpenFrameworks, Juce, HTK and MPTK, numerous MATLAB
-toolboxes, SuperCollider, Clojure and R amongst environments and
-toolkits used \cite{ssamrsurvey}.
-
-  For example, in the well-known
-subject of beat tracking, Scheirer et al \cite{scheirer} was developed
-in C++ for a legacy platform and is now only available informally;
-Goto et al \cite{goto} was written for a now-defunct parallel
-architecture and never released; Hainsworth \cite{hainsworth} was
-written in MATLAB with a non-portable DLL component and only runs on a
-single platform; Klapuri et al \cite{klapuri} was written in MATLAB
-with a platform-specific extension and is not widely distributed for
-reasons of commercial confidentiality; and methods from Cemgil,
-Laroche, Alonso, and Peeters have not been published as code.
-
-\subsection{Current practice in the field}
-\label{subsec:current}
-
-Although the Reproducible Research principle suggests a comprehensive
-solution to the reproducibility problem, in practice take-up in this
-field is limited.  Undertaking reproducible research needs additional
-effort to adopt the philosophy at an early stage and can be perceived
-as delaying the production of ``real'' research. Once research results
-have been produced and a paper written, there is little apparent
-incentive to make the research reproducible. In practice, it appears
-that most researchers in the audio and music area do not carry out
-reproducible research and many are unfamiliar with the concept.  In
-our survey of UK audio and music researchers \cite{ssamrsurvey}, the
-majority of respondents said either that they took no steps to ensure
-reproducibility, or that they only made code or data available
-on request.  Obstacles cited included ``lack of time'', ``copyright
-restrictions'', and ``potential commercial use'' of the code.  In
-addition to these, a broader case study by the UK Research Information
-Network into science research across several subject areas also
-identified ``lack of evidence of benefits'', ``cultures of
-independence and competition'' and ``concerns about quality'' as
-typical inhibiting factors \cite{rin2010}.
-
+As a consequence of the lack of publication and variety of platforms
+used, software developed in earlier research is not always readily
+available to later researchers.  For example, in the well-known
+subject of beat tracking, the method of Scheirer et al \cite{scheirer}
+was developed in C++ for a legacy platform and is now only available
+informally; Goto et al \cite{goto} was written for a now-defunct
+parallel architecture and never publicly released; Hainsworth
+\cite{hainsworth} was written in MATLAB with a non-portable DLL
+component and only runs on a single platform; Klapuri et al
+\cite{klapuri} was written in MATLAB with a platform-specific
+extension and is not widely distributed for reasons of commercial
+confidentiality; and methods from Cemgil, Laroche, Alonso, and Peeters
+have not been published as code.
 
 \section{Sustainable software: a bottom-up approach}
 \label{sec:philosophy}
 
-The approach of the present project is to facilitate practical,
-incremental improvements to the way software development is managed
-during research.  This approach is somewhat orthogonal to the goals of
-Reproducible Research: while we aim to encourage open publication of
-code and data, and linking between these and paper publications, we
-believe that researchers will appreciate improvements to software
+Our approach is to facilitate incremental improvements to the way
+software is managed during research, by identifying practical barriers
+to software reuse and providing means to reduce or eliminate them.
+While we support the goal of reproducible research and aim to
+encourage open publication of code and data linked with paper
+publications, we believe that this goal is more easily approached
+indirectly because researchers can appreciate improvements to software
 development practice regardless of their beliefs or intentions with
 regard to reproducible research.  By helping researchers to feel
-comfortable with managing provenance and versioning for software; with
-collaborative development of code; and with the perception of code as
-something that may itself be reused, we aim to lay ground in which
-open and reproducible publication can naturally grow.
+comfortable with managing provenance and versioning for software, with
+collaborative development of code, and with the perception of code as
+something that may readily be reused, we aim to prepare ground in
+which open and reproducible publication can naturally grow.
 
-Our approach has three prongs:
+\subsection{Education and Confidence With Code}
 
-\begin{enumerate}
-\item {\em Training} --- 
-\item {\em Tools and facilities} --- 
-\item {\em Hands-on help with significant pieces of code} --- 
-\end{enumerate}
+\subsubsection{SoundSoftware/Software Carpentry Autumn School}
+\label{sec:autumnschool}
+\subsubsection{Videos and Tutorials}
 
-\section{Our work so far}
-\label{sec:sofar}
+\subsection{Facilities and Tools}
+\subsubsection{SoundSoftware Code Site}
+\label{sec:codesite}
 
-\subsection{SoundSoftware/Software Carpentry Autumn School}
-\label{subsec:autumnschool}
+\subsubsubsection{Public and private projects}
+\subsubsubsection{Linking publications with code}
+\subsubsubsection{Tracking external projects}
 
-\subsection{SoundSoftware Code Site}
-\label{subsec:codesite}
+\subsubsection{Version control and EasyMercurial}
+\label{sec:easyhg}
 
-\subsubsection{Public and private projects}
-\subsubsection{Linking publications with code}
-\subsubsection{Tracking external projects}
-
-\subsection{Version control and EasyMercurial}
-\label{subsec:easyhg}
-
-\subsection{Software developer engagement: case studies}
-\label{subsec:engagement}
+\subsection{Platforms and Reuse}
 
 \subsubsection{Sonic Visualiser and Vamp Plugins}
 \label{subsubsec:sv}
@@ -215,7 +205,15 @@
 methods.  We have been able to take advantage of this system in
 several cases such as the following.
 
-%%% (The above gives context for the following, but is perhaps not ideally placed)
+\subsubsection{Hands-on Help} % porting, etc
+
+% something about MATLAB?
+
+
+\subsection{Case studies}
+\label{sec:casestudies}
+
+\subsubsection{Beat Trackers}
 
 \subsubsection{Chordino and NNLS Chroma}
 \label{subsubsec:chordino}
@@ -235,8 +233,8 @@
 code to develop a C++ implementation of the method and turn it into a
 highly usable Vamp plugin for chord estimation, named Chordino.  This
 code has been made available at
-http://code.soundsoftware.ac.uk/projects/nnls-chroma --- a Web
-resource which also identifies the associated publication.  Although
+http://code.soundsoftware.ac.uk/projects/nnls-chroma --- a Web page
+which also links the code with its associated publication.  Although
 it has been updated since publication, the plugin includes a mode in
 which it uses the same method as that submitted to the MIREX
 evaluation and as a consequence, although this paper still lacks true
@@ -255,6 +253,9 @@
 \section{Future work}
 \label{sec:future}
 
+\section{Appendix: SoundSoftware survey 2010}
+\label{sec:survey}
+
 % Below is an example of how to insert images. Delete the ``\vspace'' line,
 % uncomment the preceding line ``\centerline...'' and replace ``imageX.ps''
 % with a suitable PostScript file name.

--- a/refs.bib	Thu Sep 22 09:47:41 2011 +0100
+++ b/refs.bib	Thu Sep 22 10:36:09 2011 +0100
@@ -27,15 +27,14 @@
 
 @misc{vamp,
   author = {C. Cannam},
-  title = {The Vamp Audio Analysis Plugin API: A Programmer's Guide},
+  title = "{The Vamp Audio Analysis Plugin API: A Programmer's Guide}",
 year = {2007},
 howpublished = {http://vamp-plugins.org/guide.pdf}
 }
 
 @InProceedings{tidhar2009,
   author = {D. Tidhar and G. Fazekas and S. Kolozali and M. Sandler},
-  title = "{Publishing Music Similarity Features on the Semantic
-                  Web}",
+  title = {Publishing Music Similarity Features on the Semantic Web},
   booktitle = {Proc. ISMIR},
   pages = {447--452},
   year = 2009,
@@ -67,14 +66,14 @@
 
 @misc{rin2010,
 author = {Research Information Network and NESTA},
-title = {Open to All? Case studies of openness in research},
+title = "{Open to All? Case studies of openness in research}",
 year = 2010,
 howpublished = {http://www.rin.ac.uk/our-work/data-management-and-curation/open-science-case-studies},
 }
 
 @misc{ssamrsurvey,
 author = {I. Damnjanovic and L. Figueira and C. Cannam and M. Plumbley},
-title = {SoundSoftware.ac.uk Survey Report},
+title = "{SoundSoftware.ac.uk Survey Report}",
 year = 2011,
 howpublished = {http://code.soundsoftware.ac.uk/documents/17},
 }