Duke Listens!: Visit my main blog at MusicMachinery.com

There are lots of music recommenders out there, each one vying for
our attention, trying to connect us to our next favorite band. But not all
recommenders are created equal. Some have the insights and good taste of a
Bill Goldsmith or a John Peel, while others seem to have as much insight
about what I might like as an iPod on shuffle play. But which
recommenders are the best, and which are the worst? And how do these music
recommendations compare to what a professional would generate? Can machines
compete with humans in what is clearly a question of taste? A few months
ago, I decided to try to answer these questions - to find out what is the
best music recommender, and to find out if machines can compete with humans
when it comes to recommending music. Over the next few days, I shall be
blogging what I've discovered.

The tough question

The hardest question to answer is 'How to evaluate a recommendation?'.
A typical recommendaton starts with a seed artist, so for instance
a recommendation may be of the form "If you like The Beatles you may
like.. " followed by a list of recommended artists. Now, when I look a
music recommendation based on an artist that I really know, I can get
a feel for the quality of recommendation. In the list of recommended
artists, I expect to see some artists that I already am familiar with -
probably bands that I like. Even though these recommendations may
be obvious, they help me gain some trust in the recommender. If I
ask for a recommendation based on my affinity for Miles Davis and
a recommender suggests John Coltrane, I get a warm feeling that the
recommender is on the right track. It is a relevant (albeit somewhat
obvious) recommendation. I also expect to see some artists that I've
heard of but am not too familiar with, and I also expect to see some
artists that I've never heard of. So for me a good recommendation
contains a mix of the familiar (to help me gain trust), and the novel
(to help me find new music). Now of course, if the recommended artists
are off track ('if you like Miles Davis you might like Paris Hilton'),
no amount of familiarity or novelty is going to turn the recommendation
into a good recommendation. The recommendations need to be relevant.
To summarize, a good recommendation has three aspects:

• familiarity - to help us gain trust in the recommender
• novelty - without new music, the recommendation is pointless
• relevance - the recommended music has to match my taste
  

Lets look at a couple of recommendation as examples. Here's a set of
similar artist recommendations for The Beatles from last.fm. According to
last.fm, if you like the Beatles you might like:

• The Rolling Stones
• The Who
• John Lennon
• Led Zeppelin
• Queen
• Beach Boys
• Doors
• David Bowie
• Kinks
  

Well, last.fm certainly gets the 'familiar' and the 'relevant', but
there's nothing novel here. Any fan of the Beatles has no doubt heard of
all of these bands, no great discoveries will be found in this selection.

Compare those to this set from Dominique, a professional music critic.
According to Dominque, if you like the Beatles you might like:

• Paul McCartney/Wings
• John Lennon
• Harry Nilsson
• Queen
• George Harrison
• ELO
• Raspberries
• Badfinger
• XTC
• The Millennium
  

This list has the familiar and relevant McCartney, Lennon, Harrison
and Queen as well as some artists that a typical Beatles fan may not
have heard of: The Raspberries and Millennium. Dominique seems to have
restricted his recommendations to bands from the late sixties and early
seventies. Not an unreasonable choice, but it does perhaps reduce the
novelty aspect of the recommendations.

Compare those to this set from Chris, another professional music critic.
According to Chris, if you like the Beatles you might like:

• Chuck Berry
• Harry Nilsson
• XTC
• Marshall Crenshaw
• Super Furry Animals
• Badfinger
• The Raspberries
• The Flaming Lips
• Jason Faulkner
• Michael Penn
  

So there's the familiar with Chuck Berry and Harry Nilsson, but I have to
pause and think for a bit about relevance - there's no Rolling Stones or
The Who to give me a warm fuzzy feeling, Chris gets to the novel artists
right away - high on novelty scale but not high on the familiarity scale
(at least for me).

And finally here's a set of similar artist from the All Music Guide.
According to All Music, if you like the Beatles you might like:

• Hollies
• Searchers
• Peter and Gordon
• Monkees
• Gerry and the Pacemakers
• Bee Gees
• Zombies
• Dave Clark Five
• Remains
• Sorrows
  

This list seems fairly balanced, there's the familiar (Monkees, Bee Gees,
Dave Clark Five, and the Hollies), they've avoided the cliches (no Rolling
Stones, Queen or the Who) and have some bands that I'm not too familar
with including a band called 'The Sorrows' who are described as "one of
the most overlooked bands of the British Invasion" which certainly looks
like something that'd be fun to listen to if I like the Beatles.

With these four examples, I've tried to show the three elements
that I look for in a good music recommendation: familiarity, relevance
and novelty. The difficulty is, of course, that no two people will agree
exactly on what is familiar, relevant or novel. What I find novel, may be
familiar to a professional music critic, while a musician may find musical
relevance when I don't. The difficult job for a recommender, whether
it is human or a machine is to find the proper level of famililarity,
novelty and relevance for each person.

Finally, if we really want to evaluate a number of recommender systems,
to compare the quality of recommendations, we have to figure out a good
way to turn the very subjective measures that we've looked at here into
a set of objective measures that we can use to score recommendations.
Part 2 will look at some of the objective ways we will use to evaluate the
various music recommenders.