Duke Listens!: Visit my main blog at MusicMachinery.com

It is hard to evaluate music recommendation systems. Current evaluation techniques will often use web mining for artist co-occurrence on web pages or playlists as a way to infer artist similarity to compare against a recommender, or will try to predict a a set of ratings (1 star or 4 star) such as we see with the Netflix prize.  However, these types of evaluations generally don't measure several aspects that are associated with a good recommendation.   For instance,  evaluations that measure how well a recommender system can predict how a user will rate songs are tested against songs that the user has already rated.  This penalizes recommenders that generate good but novel recommendations.  Since the user hasn't rated these novel recommendations yet (since they are novel), the recommender doesn't receive any credit for the recommendations.  A good recommender should recommend novel items, but most recommender evaluations don't evaluate this aspect of recommendation at all.  A recommender that tells you that if you like 'The Beatles' you might like 'The Rolling Stones' may be accurate, and may be evaluated highly, but it is not a great recommender if all it tells you about are artists that you all ready know about.

Probably the best way to evaluate recommenders are with user studies.  Simply ask people how they like they recommendations to rate the recommendations.  However this can skewed results as well.  People will tend to rate recommendations that include many familiar relevant items as better than recommendations that contain a number of unfamiliar items.  Since it can take a good deal of time to evaluate a recommended item (such as a song, artist, movie or book), it is hard to get accurate evaluations for recommendations that contain large numbers of unfamiliar items.

My co-tutorist, Oscar Celma has created a personalized survey for evaluating a number of different types of music recommendation.  Unlike previous evaluations, Oscar's survey recognizes the importance of novel recommendations.  The survey will offer you a number of  music recommendations (based upon your last.fm listening behavior) and ask you questions about the recommendations, including whether or not you've heard the artist or the track before, and to what degree do you like the music.   With this evaluation Oscar can learn which recommenders tend to recommend familiar music, which recommend novel music - as well as which recommenders are recommending relevant music (that is, music that the user will like to listen to).

Oscar has done a good job designing the survey - you can evaluate as many or as few recommendations as you'd like.  The more participants in the survey, the better the results, so I encourage all of my readers to take the survey.  As a reward, tt the end of it all, you may get a few novel recommendations from a state-of-the-art music recommender to expand your music horizons.

Take Oscar Celma's Music Recommendation Survey