If you like The Beatles ... a music survey
In a few weeks, I'll be presenting (along with Oscar Celma), a music recommendation tutorial at ISMIR 2007. As part of this tutorial, Oscar and I will be presenting various evaluations of commercial and academic music recommenders. There are lots of ways to evaluate recommender systems, but when you get right down to it, probably the best way is to ask people what they think about them. And so... we are conducting a survey and we hope that you will participate. The survey is very simple: you are asked to rate the quality of recommendations that are of the form "If you like The Beatles you might like XXX". The survey will take less than 10 minutes to complete. If you are interested, then
The results of the survey will be used to rank various recommendations generated by commercial, and academic recommender systems, as well as recommendations generated by professional music critics. Surveys for recommendations based on other seed artists will be coming soon.
We are hoping to get 100s of replies so feel free to tell others about the survey. I shall be posting results after the conference at the end of the month.
Update: Well, it turns out using the 'mailto:' on
a form post is notoriously unreliable, so I spent a few hours this
morning fixing it, so now things should work quite a bit better.
Paul, I don't think the Beatles are a good choice to evaluate recommender systems. (1) The main target group for music recommendations listens to more contemporary music. Why not try something more realistic like Deerhoof? (2) The Beatles produced such a broad range of music. The different songs have very different styles. I wouldn't be surprised if any recommendation not from the same time period would be considered a poor recommendation, even if there is a Beatles song that sounds almost identical. (3) Furthermore, everyone likes the Beatles. For example, I really like the Beatles, but I'm not into most of the music that gets played on Pandora similar artist radio for the Beatles (or shows up in the Last.fm list of similar artists).
Posted by elias on September 08, 2007 at 07:44 AM EDT #
Indeed, you are exactly right. I have a collection of seed artists that I intend to include in the survey. I started with the 'Beatles' since I didn't want to overwhelm people .. but perhaps that was a mistake. Anyway, look for more surveys using seed artists: Miles Davis, Emerson Lake and Palmer, Deerhoof and Arcade fire.
Posted by Paul on September 08, 2007 at 07:53 AM EDT #
Paul, I am not sure what you are looking to use this survey data for? I am sure you have a specific purpose but on a surface reading of the post it does not seem to make sense?
As Elias points out matching by Artist is very often irrelevant to music and so makes for a very poor set of data for evaluating "music" recommendations (Elenor Rigby has little in common, musically, with Helter Skelter). A look at last.fm's artist recommendations for the Beatles, for example, lists bands whose similarity is more to do with era and fashion that music per se. I know from communications with Pandora that they feel that music needs to be in the same era to be similar which others would disagree with. Now of course a great many users like these approaches, but it is not the same thing as recommendation based on a piece of music (which is my bias).
Now I know this must all be very basic, common knowledge in the MIR community which is why I am curious as to why you would want to use this type of evaluation? I am sure I am missing something glaringly obvious so I would appreciate it if you could give us some more details about what you are thinking?
Posted by Ian on September 09, 2007 at 09:03 AM EDT #