What's the point of artist recommendations?
In the comments section, Ian raises some questions about the relevance of artist-level recommendations and why I'm interested in this type of evaluation. Instead of burying my answers in the comments, I'll respond right here:
I think
there is room in the world for both artist-level recommendations and
track-level recommendations. Clearly for sites like Pandora that
are trying to give their users a traditional radio-like listening
experience, quite a bit of attention should be paid to making good
playlists, and this involves understanding music similarity at the track
level, otherwise the listeners will get iPod whiplash from the Eleanor
Rigby to Helter Skelter song transitions. There are many situations
where track level recommendations are going to be best.
But I
don't discount artist-level recommendations. These are important
for someone who is looking for new music. Every month I head over
to eMusic to try out a brand new artist - but I don't pick them
randomly, nor do I pick the artist at the top of the charts, I'll use an
artist level recommender to find an artist that I might like based on
artists I already like.
For this study, since my goal was to
compare (at least casually), various recommenders, I really had to use a
lowest-common-denominator approach. Most commercial recommenders
will give you artist-level recommendations, or 'similar-artist'
lists. Only a very few give track level recommendations. I
decided to start with the simplest of recommendation tasks: artist-level
recommendations based upon a single artist seed. There are a
dozen recommenders that support this type of recommendation, so it
seemed to be the best task for a first comparison. This does,
however, leave out a number of recommenders that work at the track level
(Pandora and Zukool for instance). This doesn't mean that I think
track-level recommendations are not important.
I am really
interested in exploring the question "What makes a good
recommendation?". When I look at the various recommendations
offered for the seed artist 'The Beatles' I am really forced to think
hard about that question. Is 'The Rolling Stones' a good answer to
"If you like The Beatles you might like ..."? Surely anyone who has
heard of the Beatles must also know about the Rolling Stones and has
already made up their minds about them, one way or another, so it can't
be a good recommendation. On the other hand, seeing "The
Rolling Stones" on a list of Beatles-like recommendations, gives me some
confidence that the recommender is giving reasonable
recommendations. Questions of popularity, serendipity,
familiarity, trust, and fashion all factor into people's sense of what
makes a good recommendation. I'd like to understand this at a
deeper level.
With that here's what I am doing in this study:
- Collecting artist-level recommendations from a number of commercial and academic machine-based recommender systems for 5 seed artists
- Collecting artist-level recommendations from a number of professional music critics, for the same 5 seed artists
- Comparing the machine-based systems to see how well they agree with each other
- Comparing the humans to see how well they agree with each other
- Comparing the machine-based recommendations to the human created recommendations
- Collect data via a survey that ranks that quality of each recommended artist.
- Score the various recommenders based upon the quality of recommendations as scored in the survey.
There
are all sorts of things that I am not measuring, mainly because it is
hard to figure out how to measure it. Of particular importance are:
- Serendipity - it is unlikely that a survey taker, when encountering a recommended artist that they don't recognize in the survey will actually go and listen to the artist and then rate the quality of the recommendation. No one is going to take the time to do that. But this is what we really want to measure - did we recommend something to you that was novel and interesting to you?
- Trust - to me, a good recommendation list will contain some artists that I already know, and some that I don't. For this survey, you are not seeing the individual artist lists, so you can't make an assesment of how reliable an recommendation list is.
The bottom line is that there are
many, many factors that make a good recommendation, and it is not always
easy to measure these things, but there are some things that we can do
to start to understand which factors are most important.
I hope I answer at least some of your questions - and I appreciate input from many of the smart minds that read this blog on how to do this better.
Paul, that makes lots of sense to me.
Btw, do you think the outcomes of your questionnaire would be different if you hadn't mentioned the word "recommendation", and instead asked participants to rate artist similarity? (I'm not saying that those two tasks are the same, but I'm curious if participants would treat them differently.)
Posted by elias on September 09, 2007 at 10:48 PM EDT #
It's a good question. In the survey so far, only 12% of participants think that "Beatles" is a poor recommendation for "The Beatles". I think that is some evidence to support the theory that many equate similarity with recommendation and don't care about obviousness or usefulness.
Posted by Paul on September 10, 2007 at 05:25 AM EDT #
Hey Paul,
I need to say that this is a topic that is near and dear to my heart and I am of the same mind as you are (in a lot of ways):
1. Track-based recommendations are ideal for playlisting and finding specific songs that match your request.
2. Artist-based recommendations are great for looser, more adventurous exploration when the listener has time to weed through more results.
==============================================
1. Track-based recommendations are ideal when the listener says "One of my favorite songs is 'You've Got to Hide Your Love Away' by The Beatles. I'd like to hear more songs like that." From a Track-based perspective, a great system would bring back "If You See Her, Say Hello" by Bob Dylan, "Pale Blue Eyes" by the Velvets and "As Tears Go By" by the Stones. Not just songs from the right era or by "Similar Artists" but songs with the same feel, mood and conceptual elements.
Artists like Miles Davis or Bruce Springsteen or The Beatles have gone through so many phases and iterations in their careers that using any links between these artists can return wildly different songs as a result ("Eleanor Rigby" to "Helter Skelter" indeed).
A solid "I Want To Find Music That Sounds Like This" system works best when track-level information is the core data set behind it.
===============================================
2. Artist-Based recommendations require a little more forgiveness and digging, but can be even more rewarding if you stumble across a gem.
The longer-lasting benefits can be when a listener/user is looking for "More Artists That Sound Like The Hold Steady" and they get a list back of names. Some they've heard before (which is vital, because that is our initial judge of how well the technology works) and some fresh faces. A Deerhoof example could be if it came back with The Fiery Furnaces, The Arcade Fire, Life Without Buildings and U.S. Maple (artists I am already familiar with, are roughly in the same style, and give credence to the technology behind it), as well as some names I'm not familiar with like Gunther, Lab Partners and Nervous Cop (I have no idea if they are good matches, I've never heard them).
Provided that the user gets to listen to some of these songs (I know I left my celestial jukebox around here somewhere...) the listener could sample some songs and make some decisions about what they want to dig further into.
The real power of something like this (which is also possible at the track level as well) is when a tune comes on and you say "My, but that is nice, who is this?" you make a note of it, and later that artist comes around again. Maybe a slightly different song, an acoustic version or something where you say "Ah, this is the band I liked before. Sounds different, but I still like it." The third time that artist comes around in your playlist again, you're already logging on to your favorite music provider (or on your way to the record store if it still exists) to buy that album. And when you're done having that album change your life, you go back and buy every other album in the discography.
Sloan, The Hold Steady, Nick Drake, Tortoise, Johnny Hartman, Iron & Wine, Patty Griffin...these are all artists where I discovered one album, then went back and devoured everything I could get my hands on.
This kind of musical spelunking is still possible using track-based recommendations but it often presents to you the songs that match the Seed Song you started from. Imagine if you like "Old man" by Neil Young. You Hear "Cowgirl in the Sand" and "A Man Needs a Maid" and say "Ah, a quiet acoustic folkie, I'd like to pick up an album of his" and you go out and buy "Arc/Weld" (all Crazy Horse madness and feedback). You might be pissed. But if you were looking for more artists that have a quiet acoustic side and a loud electric side (kinda grasping here but maybe Pearl Jam or Lou Reed) and stumbled across Neil Young's wide range of emotions, you might become a fan for life.
=============================================
(Man, this is longer than I intended)
I think the reason everybody who reads this blog has gravitated toward artist recommendation is because the APIs are out there. Artist standardization and normalization is a much easier beast than trying to find individual tracks out there on the web. A company/individual tags Deerhoof with Indie Rock and Experimental and that can cascade down to all of their songs. It is a much bigger task to dig through an entire discography (Hell, "Songography") and make those connections/attributions there. I can guarantee that if you poked around online you could find a Deerhoof song or two floating out there, but trying to find an exact match for the song "Halfmole Halfbird" is not going to be as easy.
The tools we all have at hand are bent toward Artist-Based recommendations, that's why we're seeing so much more of it being experimented with.
Posted by Zac on September 10, 2007 at 09:19 AM EDT #
Very interesting question, and possibly one with no correct answer (or at least, no correct answer that means practical results).
Not only is music recommendations a field where there's no official "good answers" which a computer can see (those would be dangerous anyway), but it's really a matter of opinion, interpretation and trust, highly volatile concepts for humans, and pretty difficult to gauge for an automatic system.
What is the recommender trying to say? On what (facts) is the recommender basing his recommendations. What metrics is the recommender using? Is the data strong, verified? Is this recommender good? for me? for anyone in general?
How do we know a recommender is trustworthy? A human, we can agree/disagree with based on the wording of the recommendation, the looks, the reputation, the format. Is she a friend, a friend of a friend of a friend, a professionnal critic, a radio host, a PR person, an anonymous internet user? Is that person only voicing her personal tastes (I like this and I like that), trying to impress by cleverly obscure or clichéd recommendations, trying to sell something?
For machine recommendations, we usually can't get a subtle wording at all. The best way (I can think of right now) is to use simple heuristics, based on good data (that users know is good, or think is), and be see-through with the recommendation. Amazon is the easiest example: "Customers Who Bought This Item Also Bought [Y]". People know what they get.
Sadly, for more clever solutions, mixing different sources of data, and different heuristics, the recommender will probably be more of a black box, with the wording of the recommendation being either absent (just a list of links), or arcane/overly technical (mysterious score calculated from 12% of this, 22% of that, with the juice of three lemons).
Cross-validation of recommendations is a good idea, but will possibly end up creating an echo chamber, where some concenssus of the most common recommendations will be considered good, and the rest erratic. And thus, the systems adapting to this "metric", will end up saying exactly the same thing. What is there to say about the recommender who disagrees with the norm, other that, well, he disagrees?
Posted by Marc-O on September 11, 2007 at 11:24 AM EDT #
I really tried to take the survey; I found that I couldn't, because I just couldn't give any sort of recommendation at all, based on the question, "if you like X, then you might like Y" format.
To me, I either like a band or I don't. My like or dislike of a band is not conditional. Given that I like Y, that like or dislike isn't conditioned on any X.
I guess I am just not sure how to think about doing conditional recommendation. With friends, whether or not they like the same bands that I do, I will often just recommend a band to them. No strings attached. No "you must like this other band, first, and therefore you will like the band I am telling you about". I just tell them about a band that I like.
Posted by jeremy on September 11, 2007 at 08:51 PM EDT #
Jeremy:
Let's say you are having dinner with your 10 year old niece. She doesn't perhaps share your sophisticated music tastes yet, but she tells you she really liked that Chris Daughtry on American Idol much better than Kelly Clarkson or Sanjaya. And she asks you what she should listen to next. You may not like Chris Daughtry or the whole post-grunge-pop thing, but you could still point her to Breaking Benjamen or Seether to help her out couldn't you?
Posted by Paul on September 11, 2007 at 08:59 PM EDT #
I'm not trying to be difficult.. but.. I think I would probably just recommend something to her that I liked at that age.
Er, wait. Maybe not. At that age, I was into an album called "Smurfing Sing Song", loved that old Moog Synthesizer album "Switched on Bach", listened to "The Sting" soundtrack over and over (Scott Joplin) and had a 45 of Sweet Georgia Brown (my parents had taken me to see the Harlem Globetrotters :-) So I probably wasn't (still am not) the best arbiter of musical taste.
Still, I think there is something to be said for pulling someone, even a 10 year-old, out of their narrowly-bounded world, and exposing them to new music, nothing like they've ever heard before. I loved that friends and family turned me on to everything from Bach Electronica to vaguely-Alvin-and-the-Chipmunks sounding cartoon characters (the Smurfs record album came out even before the cartoon itself hit the U.S.)
This is honestly honestly not meant as a criticism of anyone's work on recommender systems, because I very much like the idea of getting music recommendations. But sometimes I think that the mass aggregation of "people who bought this also bought".. co-occurrence approaches are not really capable of helping oneself break out of local music minima. It sometimes takes a single, independent voice saying to you, "hey, I know you like this stuff over there. But leave that behind for a moment and come listen to this other stuff, over here. It's like nothing you've ever heard before. You might like it."
I remember talking to Don Byrd, a few ISMIRs ago, about this. I think it was ISMIR Barcelona, actually. Or maybe Paris. Can't remember; time flies. Anyway, he said that what he wanted was a music retrieval system that gave him music that was not the most *similar* to what he'd heard before, but the most *dissimilar*. Analyze his collection, he asked, and then show him subsections of the music world that he'd never even considered before. It's still recommendation, but it the anti-"if X then Y" approach. It is more like the "if (X1 and X2 and X3) then Y4", where Y4 != Y1, Y2, or Y3. Does that kinda make sense?
A few years ago I bought an album.. not an original really.. but a compilation. Called "Under the Influence" by Paul Heaton. Heaton is (was) the lead singer of one of my all-time favorite groups, the Beautiful South. And the album is simply a compilation of songs that are some of his own favorites, some of the songs that influenced him, musically, and led him to be where he is.
You would think, right, that if I like Paul Heaton, then the songs he is essentially recommending to me are going to match my "Beautiful South" taste, right? They're going to be at least somewhat similar to all the other recommendations that I am getting from Last.fm, Yahoo Music, Amazon, etc. Groups like the Lightning Seeds, Crowded House, Ocean Colour Scene, etc.
Well, look up the track listing. Ok, there is an Elvis Costello in there. That's safe. But otherwise? Folk. Soul. Country. I'll say that again: Country. Not even alt-country. "Randy Travis" country. Something French that I'd never heard of.
Basically, with the exception of one, maybe two artists, I'll bet the wisdom of crowds would completely disagree with all of Heaton's recommendations. People who listen to Heaton *do not* recommend the groups that Heaton himself listens to.
And therein lies something I think is extremely interesting. Here you have Heaton as a lone pointer *out* of the local minimum of musical taste that surrounds him. These are not conditional recommendations, where Heaton is saying "If you like me, then you will like these other groups." These are Heaton simply saying "I like that and that. Period. Take it or leave it."
Those are the kind of recommendations I would like to see more of. Strong voices, pointing the way out of local minima. "DonByrd"-ian "show me something, good, that is completely different to anything I've never heard before" recommendations. (Don Byrd the ISMIR researcher, not Don Byrd the jazz musician :-)
Do you know of any work in this area?
Posted by jeremy on September 12, 2007 at 01:25 AM EDT #
@ Jeremy:
Here's what I like to do to discover something I've never heard of (using Last.fm)
1) listen to interesting tag radio stations. almost any country, instrument, mood, colour, or even theme works. e.g.: http://www.last.fm/group/The+Special+Interest+Tag+Radio+Collective
(Btw, if you are a Last.fm subscriber you can turn on the "discovery mode" which will only play you music you never heard of before.)
2) I also love to browse music profiles of other people who I find interesting (e.g. friends, or musical neighbours which Last.fm computes for me).
Posted by elias on September 12, 2007 at 08:19 AM EDT #