changeset 53:d41901e1fd33

reviews

--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/reviews.txt	Mon Jun 13 12:45:07 2016 +0100
@@ -0,0 +1,103 @@
+
+----------------------- REVIEW 1 ---------------------
+PAPER: 41
+TITLE: MusicWeb: an open linked semantic platform for music metadata
+AUTHORS: Mariano Mora-Mcginity, György Fazekas, Alo Allik and Mark Sandler
+
+OVERALL EVALUATION: 2 (accept)
+
+----------- Review -----------
+Music Web is a portal that proposes to look for music artists and proposes faceted links to browse "related artists", discovering connections that might be extra musical or not straightforward for a user. For building these connections the authors collect metadata from semantic web data sources such as MusicBrainz, DBPedia, last.fm etc., and use sameas.org in order to reunite co-references of the same entities. They create links by matching socio-cultural similarities using YAGO categories and DBPedia categories. They also parse and scrape data from publications available online (research papers, blogs, newspapers...), looking for occurrences of artist names, only in texts relevant to the music discovery domain. Then the audio content is also taken into account using MIR technologies. The data from the audio analysis are grabbed from the AcousticBrainz web service, which provided data (the musicbrainz id can be used to match songs).
+
+The section about literature based linking lacks details. It is not clear if you just create a link between artists of if you grab more information about this link (i.e. politically engaged artists if co-cited in a paper about politic). This section needs to be developed in order to understand what challenges you had to address, etc.
+
+The provided explanation about the computation of the similarities between 2 artist’s profiles is done using a Gaussian Mixture Model. It could be interesting here to cite a few other works, or to look at how Pandora or Spotify / Echonest do. The Million Song project also computed similarities between MIR data, did they use the same method? Do you take into account all songs by an artist? Some artists changed a lot during their career, albums sound very different from one to another... maybe you should take into account this?
+
+Some information could be also provided: do you build your own dataset, then use your server side code to work on your own data, or do you go online for each request? What is the size of your database if we are in the first case? How do you take into account new artists?
+
+Similarities using categories only use a distance of 1 between artists? I mean, you cannot relate two artists because they have indirect connections? The work done by Nicolas Marie on the discovery Hub application (http://discoveryhub.co/) is certainly relevant here, even if not applied directly to music but to all domains.
+
+Do you have an idea about the recommendation algorithms used by the https://developer.seevl.fm/ service?
+
+What problems did you encounter? Reading the paper, it seems you had none ;-) The web of data is so perfect?
+
+When will your application be online so we can try it?
+
+Ok, lots of remarks but: the paper is well written, proposes a very interesting mixed approach (audio + semantic web + full text indexation), and should be a valuable presentation as an ISWC application paper.
+
+----------------------- REVIEW 2 ---------------------
+PAPER: 41
+TITLE: MusicWeb: an open linked semantic platform for music metadata
+AUTHORS: Mariano Mora-Mcginity, György Fazekas, Alo Allik and Mark Sandler
+
+OVERALL EVALUATION: -1 (weak reject)
+
+----------- Review -----------
+1. Paper summary
+In this work, the authors describe a platform that uses multiple data sources and APIs to discover connections between music artists
+
+
+2. Contributions
+There are two contribitions in this work:
+First, the authors show how and which data sources they used to extract information for displaying and linking music artists. The second contribution is to highlight which methods they used to calculate similar artists according to different metadata (e.g. editorial, cultural and acoustic).
+
+
+3. Discussion of aspects of the Paper
+The authors conform to the allowed number of pages and the given template.
+
+Positive:
+- Very good introduction and background into the topic
+- Topic of extracting and integrating data from various data sources is a challenging task, in which the advantages of Semantic Web Technologies can be seen.
+
+Negative:
+- Please provide better descriptions under the figures (e.g. for Fig. 1  i recommend a description like "User interface with information for the music artist Ella from various sources including links to similar music artists.")
+- Architecture diagram (Figure 2) could be improved by clustering data sources and/or tiers. For me, it seems like the Platform is a classical three-tier Architecture. So Client Requests on top. In the middle the logic (MusicWeb API) and on the bottom (data tier) the data sources.
+- The Name "MusicWeb" is sometimes written in small letters and sometimes in capital. I suggest a unique naming convention of the Platform/API.
+
+
+4. Discussion of Emerging Applications
+The authors use different metadata to calculate similar music artists. However, partly a detailed description why they used the methods or a comparison, or evaluation of how good the proposed connections are, is missing. Why did they use Manhattan distance for similarity by mood instead of any other distance (like e.g. Euclidean distance). Or why they don't use cosine similarity. So there, i would like to have more arguments for the used methods.
+Next steps, and ongoing work is well described and one can clearly see that this application will be used in future. It is also very powerful to have artists with metadata from different data sources available. I like the idea of providing similar artists, however, the descriptions how they are calculated and experiments/evaluation is missing.
+
+
+5. Discussion of further criteria
+Integrating information is a relvant topic for the ISWC, as well as exploiting different metadata for calculating similarities between different entities (in this case music artists). Unfortunately, no evaluation were given and i'm missing some arguments for the used methods. The writting and formatting is good. The structure is logical and understandable by the reader.
+
+
+6. Typos
+- Page 4: endpoints sa well --> endpoints as well
+
+
+7. Conclusion
+This application paper is especially interesting for people who needs data from various sources, provided by one API. However, detailed arguments for using certain methods and detailed quality and performance of the solution are missing .
+
+----------------------- REVIEW 3 ---------------------
+PAPER: 41
+TITLE: MusicWeb: an open linked semantic platform for music metadata
+AUTHORS: Mariano Mora-Mcginity, György Fazekas, Alo Allik and Mark Sandler
+
+OVERALL EVALUATION: -2 (reject)
+
+----------- Review -----------
+The paper discusses a system that integrates data from a multitude of sources about music artists for the purposes of allowing users to search and browse similar artists. Unlike other recommendation services for music, MusicWeb is centred around artists and takes into consideration a broad set of information for performing recommendation, including external factors such as religion of artists, appearances in the same articles, etc. The set of sources integrated include DBpedia (for general knowledge), YAGO (for detailled categories), MusicBrainz for further encyclopaedic information, sameas.org for linking DBpedia and MusicBrainz, Last.fm for listening habits, Mendeley and Elsevier for finding research articles mentioning a given set of artist(s), AcousticBrainz for audio content descriptors of artist's tracks, and IML10K for mood indicators associated with artists. The paper discusses how all of these sources are processed and integrated into the MusicWeb system. The authors discuss some current usage, experiences, and future work.
+
+The paper is generally well written and very interesting to read. The amount of work that has gone into the system is really impressive, taking and applying non-trivial processing over a large number of sources. Also the work is quite technical in various different directions, which includes, for example, content analysis of audio, semantic annotation and similarity computation over research articles, etc. The paper has me very curious to see the system in action!
+
+The paper is submitted as an Emerging Application track:
+
+"""
+Papers presenting emerging applications, as early reports on real-world projects, should expose substantial contributions in terms of semantics requirements, testing of approaches or infrastructure, evaluations of early prototypes. Such papers will present mature technology contributions not yet deployed at large scale. We expect early versions of in-use or industry solutions with details on requirements or early phases of large scale deployment in real world settings.
+"""
+
+Unfortunately, I think the paper fails to meet this call. Although it does in some sense describe an emerging application, it does not provide any testing, evaluation, nor really a list of requirements. Hence I think although the system does sound intriguing and the work impressive, it does not meet the requirements for the track and cannot be accepted.
+
+Aside from this, I am a bit concerned about the sense in which artists are recommended. It seems that one can make quick leaps to unrelated artists and it is unclear to me if it would be clear to the user why, or indeed if this is useful in a music recommendation sense. In fact, very few details are provided about the final system, which makes it hard to know how useful it is (though the screenshot looks quite nice!). I also wonder what information need the service solves ... for example, music recommendation services allow users to find music they will probably like to listen to. What specifically does MusicWeb solve in that context? I understand a lot of information about artists is provided and that similar artists are listed, but I am not sure why I would use the service unless perhaps to learn more about an artist I like.
+
+In any case, again I find myself intrigued about the system and am impressed by the work! Unfortunately without any evaluation it cannot be accepted here. If rejected, I would encourage the authors to submit either to a workshop or, perhaps even better, as a demo to ISWC (or both). Otherwise, if submitting to a similar track in future, the authors will have to find some strategy by which they can evaluate the system. (Another option *perhaps* is to gather up more users and write up a Tool/System report for the Semantic Web Journal, or perhaps to target a multimedia conference.)
+
+MINOR COMMENTS:
+* I audibly gasped when I saw page 2. I'm not sure if the paragraphs disappeared due to some LaTeX hiccough or if the intent was not to have paragraphs, but please, put paragraphs.
+* There's various problems with capitalisation (e.g, linked data, dbpedia, freebase, etc.)
+* Name the Pantera guitarist
+* tf/idf
+* characterisation of