It is hard to evaluate music recommendation systems. Current evaluation techniques will often use web mining for artist co-occurrence on web pages or playlists as a way to infer artist similarity to compare against a recommender, or will try to predict a a set of ratings (1 star or 4 star) such as we see with the Netflix prize.  However, these types of evaluations generally don't measure several aspects that are associated with a good recommendation.   For instance,  evaluations that measure how well a recommender system can predict how a user will rate songs are tested against songs that the user has already rated.  This penalizes recommenders that generate good but novel recommendations.  Since the user hasn't rated these novel recommendations yet (since they are novel), the recommender doesn't receive any credit for the recommendations.  A good recommender should recommend novel items, but most recommender evaluations don't evaluate this aspect of recommendation at all.  A recommender that tells you that if you like 'The Beatles' you might like 'The Rolling Stones' may be accurate, and may be evaluated highly, but it is not a great recommender if all it tells you about are artists that you all ready know about.

Probably the best way to evaluate recommenders are with user studies.  Simply ask people how they like they recommendations to rate the recommendations.  However this can skewed results as well.  People will tend to rate recommendations that include many familiar relevant items as better than recommendations that contain a number of unfamiliar items.  Since it can take a good deal of time to evaluate a recommended item (such as a song, artist, movie or book), it is hard to get accurate evaluations for recommendations that contain large numbers of unfamiliar items.

My co-tutorist, Oscar Celma has created a personalized survey for evaluating a number of different types of music recommendation.  Unlike previous evaluations, Oscar's survey recognizes the importance of novel recommendations.  The survey will offer you a number of  music recommendations (based upon your last.fm listening behavior) and ask you questions about the recommendations, including whether or not you've heard the artist or the track before, and to what degree do you like the music.   With this evaluation Oscar can learn which recommenders tend to recommend familiar music, which recommend novel music - as well as which recommenders are recommending relevant music (that is, music that the user will like to listen to).

Oscar has done a good job designing the survey - you can evaluate as many or as few recommendations as you'd like.  The more participants in the survey, the better the results, so I encourage all of my readers to take the survey.  As a reward, tt the end of it all, you may get a few novel recommendations from a state-of-the-art music recommender to expand your music horizons.

Take Oscar Celma's Music Recommendation Survey

Comments:

oh, but I have to have a last.fm account to take the survey? I guess it's time to sign up, even though Musicovery is doing a nice job for me right now.

Posted by Tim on March 03, 2008 at 08:23 AM EST #

Very fun. And I'm dying to know what Oscar hopes to learn from the data.

I'm interested to know how my personal Last.FM data factors in.

http://www.last.fm/user/DataWhat/

I listen to mostly indie rock and "alternative" with some classic rock, country and jazz thrown in, but my options in the survey were mostly hip-hop and R&B. Does the survey take my Last.FM preferences and try to match them? Or is it trying to find things that are at the opposite end of my spectrum to try to send new things my way? I didn't hear any jazz and only two country-ish samples (I went through 3 pages/25 songs).

Posted by Zac on March 03, 2008 at 09:19 AM EST #

i still think the automated system at mongomusic/msnmusic was the best in the world.

too bad it is gone for good.

Posted by PastParticipleMachineLearned on March 18, 2008 at 02:57 PM EDT #

Last.fm have a nice recommendation system,it can meet me some taste

Posted by songs lyrics on March 21, 2008 at 11:40 AM EDT #

Post a Comment:
Comments are closed for this entry.

This blog copyright 2010 by plamere