Outline notes for ICASSP 2012 paper submission¶

General form of a paper:

1. What problem we tried to solve
2. How we tried to solve it
3. How well it worked

What is the problem here?¶

(Can we add references here!?)

1. Research in this field involves developing software
2. That software typically is not published
3. Consequently it's hard to get hold of reference implementations of significant algorithms or to reproduce results from papers

We did a survey -- but can we package its results in a way that provide any sense of scientific rigour?

It found:

1. People use lots of different languages and environments
2. Many people don't share their code
3. A surprising number asserted that they did not intend to publish any code, and that their code never left their own computer

We also observed that our own facilities were not ideal and not being used to best advantage.

We also made some observations in the Autumn School (though this is getting a bit circular since the Autumn School was one of the things we've been trying to do as well):

1. Many attendees had never used version control before
2. When its benefits were shown to them, they were generally very receptive & positive (version control was identified as a good point in the programme)
3. But they found it trickier than I had expected to get going with the Subversion client used in the workshop

What have we been trying to do?¶

1. Promote collaborative development from the outset -- if we can encourage people to work together on code even just as much as they would normally work together on a paper, then we will increase their comfort with disclosing code later -- this faces some tricky cultural obstacles though (e.g. necessity to convince supervisors etc that you did your own work on your own)
2. Provide facilities and services that people can use and educate them to make best use of them (or of any facilities they already have)
3. Get hands-on, taking care of code that people really want to use

What have we done so far?¶

1. Autumn School
2. Code repository site
3. EasyMercurial

How well has it worked?¶

What will we do next?¶

1. More learning materials
2. Follow-ups to Autumn School
3. Visits to other UK research institutions

Reprouducible Research References¶

EPFL Page on RR:

• http://lcav.epfl.ch/reproducible_research

EPFL's Repository:

• http://rr.epfl.ch/

Some papers worth reading (http://reproducibleresearch.net/index.php/RR_links)

• WaveLab and Reproducible Research (J. B. Buckheit and D. L. Donoho, )
  • Dept. of Statistics, Stanford University, Tech. Rep. 474, 1995.
  • http://www-stat.stanford.edu/~donoho/Reports/1995/wavelab.pdf
• Reproducible Research: The bottom line (de Leeuw, Jan)
  • Department of Statistics, UCLA, Department of Statistics Papers, March 2001
  • http://escholarship.org/uc/item/9050x4r4#page-1
• Pushing Science into Signal Processing (M. Barni and F. Perez-Gonzalez)
  • IEEE Signal Processing Magazine, vol. 22, no. 4, pp. 119–120, July 2005.
  • http://www.gts.tsc.uvigo.es/~fperez/docs/pushing_science.pdf
• Sharing Detailed Research Data Is Associated with Increased Citation Rate (H. A. Piwowar, R. S. Day, and D. B. Fridsma)
  • PLoS ONE, March 2007
  • http://www.plosone.org/article/info:doi/10.1371/journal.pone.0000308
• How to encourage and publish reproducible research (J. Kovačević)
  • ICASSP 2007
  • http://lcav.epfl.ch/files/content/sites/lcav/files/reproductible_research/ICASSP07/Kovacevic07_pres.pdf
• Reproducible Research in Signal Processing - What, why, and how (P. Vandewalle, J. Kovacevic and M. Vetterli,)
  • IEEE Signal Processing Magazine, May 2009
  • http://rr.epfl.ch/17/

Maybe some similarities?

• SHARE: a web portal for creating and sharing executable research papers (P. Van Gorp and S. Mazanek)
  • Proc. International Conference on Computational Science, 2011