Take the Music Recommendation Turing Test
Here are two sets of music recommendations, one is created by a
machine (a music recommendation algorithm that we've created here in the
labs), and one created by a professional music critic. Can you
tell which is which? Which one is a better list? Post your answers in
the comments.
Recommendation List A:
If you like "Miles Davis" you might like:
- John Coltrane
- Duke Ellington
- Thelonious Monk
- Charlie Parker
- Herbie Hancock
- Bill Evans
- Charles Mingus
- Sonny Rollins
- Wayne Shorter
- Weather Report
Recommendation List B:
If you like "Miles Davis" you might like:
- John Coltrane
- Weather Report
- Herbie Hancock
- Wayne Shorter
- Charlie Parker
- Dave Douglas
- Chet Baker
- Tony Williams Lifetime
- Can
- Sly & the Family Stone
Well, this is too easy...
A = human
B = machine
no doubt in my mind.
Posted by Dan Trenz on September 14, 2007 at 08:46 AM EDT #
I don't know which is which.
I think A is better if you like Miles' stuff up to about 1968, and B is better if you like his stuff from 1968 onwards.
Ok, OK, that's not a straight answer, but Miles' career is so extensive and wide-ranging that it's hard to pin down. I like the later stuff, so B is better for me.
Posted by David Jennings on September 14, 2007 at 08:52 AM EDT #
If I had to guess I'd say B is the human, betting against Dan T, but I'd stress that this is a guess.
Give me their top 50 recommendations and I might be able to tell more confidently.
Posted by David Jennings on September 14, 2007 at 09:00 AM EDT #
This is where Artist-based recommendation gets tricky.
I am a big Miles Davis fan (as I have a pulse) but everything from "On the Corner" and beyond gets too fusion-y/Funky for me. I would say because I am a fan of Traditional Jazz, Bop and Cool, List A is the winner (minus Weather Report, thank you. Somebody once said "There's a fine line between the music of Weather Report and the music on The Weather Channel" and I have always agreed.)
BUT: My buddies who are into Phish, Umphrey's Magee and and all things experimental/jammy would gravitate toward the Can/Sly & the Family Stone/Tony Williams Lifetime/Dave Douglas list.
This is the double edged sword of the Artist recommendation: If I liked trad jazz and experimental jammy exploration, Miles Davis is right up my alley, but if I was just a fan of non-experimental stuff and heard Can into Sly into Weather Report as a representation of what Miles Davis is all about, I might get turned off of Miles for the rest of my life.
Posted by Zac on September 14, 2007 at 09:36 AM EDT #
Tell us, Paul, were these two lists specifically chosen for the Turing test? Seems to me both list are valid (though I know little of Jazz), and each could be generated by either a critic or a machine. But if these were chosen by design, it's not really a turing test, it's rather a question of what we think a machine can do. More specifically, can the machine link Can and Sly to Miles Davis? (Though I think perhaps all turing tests can be transformed into similar questions: what can we expect the machine to do).
I'd pick the critic to be B, mostly because A is (to me) more conservative, and that being conservative is probably a good heuristic to maintain the quality of machine recommendations, to sort out the noise.
Now, I'm ready to bet that the critic would be much more likely to explain why he picked those 10 artists for recommendation, and that the machine would have much trouble doing so.
Posted by Marc-O on September 14, 2007 at 10:01 AM EDT #
The recommendations in B remind me very much of the recommendations your system gave me for Le Volume Courbe. Both seem to come from the same recommendation system.
I wouldn't be surprised if the recommendations in list A were generated by a machine, too. It's really not hard to put together a list of similar artists for such popular artists.
So my guess is:
A = machine or human
B = Paul's machine
Posted by elias on September 14, 2007 at 10:24 AM EDT #
Marco: - Good point about the human in the loop affecting the nature of the test. I chose the two lists from a larger set of data. I picked Miles Davis as the seed artist by hand since I wanted a fairly recognizable artist (not too many people know about 'deerhoof' now do they). I chose the human list at random from the set of human lists. I chose one of our internal algorithms to represent the machines since no one else has the algorithm, and therefore can't check. If I chose last.fm or Mystrands or some other commercial recommender, some people would be able to recognize the results.
Posted by Paul on September 14, 2007 at 10:32 AM EDT #
List A is machine, List B is human
List B picks out connected artists like Can and Sly Stone which could only be done by a human who knows the cultural history of the music and aware of the inter-connections and influences that have occured between different artists. List B is more of a straight list of standard Jazz artists which has been generated because Miles is one of the key standard Jazz artists. I think List B is better because it is more interesting and potentially takes you on a more rewarding journey.
Posted by juniorbonner on September 15, 2007 at 03:19 AM EDT #
@ Paul: this is great! :-)
Maybe you should wait with your answer and give those attending your tutorial a chance to think about it, too? I'm very curious how your recommender works. The approaches I used for my own work (e.g. MusicSun) would have completely failed for artists like "Can".
@ Junior: You seem to know a lot more about music than I do, but have you tried a google search for "site:myspace.com miles davis sly family stone"? It really depends on what you tell the machine to do. For example, you could tell the machine to crawl MySpace profiles where people describe the music they like.
Posted by elias on September 15, 2007 at 08:37 AM EDT #
@elias: Indeed, this Turing test idea is straight from the music recommendation tutorial - I consider it a teaser to whet the your appetite. There will be a couple others to ponder as well.
Posted by Paul on September 15, 2007 at 08:43 AM EDT #
Good Work! Paul. How about Human vs Algorithm vs "Community". The "Community" being the tally of your most recent questionnare regarding how much is an artist like the seed artist.
Unless, of course, your algorithm is based on these questionnaires. Then in these two lists we are comparing a music expert vs the community/algorithm. However, your algorithm may be looking at other factors such as genre, musical instruments used, period of time, mood. If you algorithm is working like the latter, then my suggestion of Expert vs Algorithm vs Community would be correct.
Posted by Cesar on September 15, 2007 at 02:22 PM EDT #
i'm gonna go with A being the better set. i agree that this is time period dependent, but while I can see Can being a good rec for late Davis fans, Sly seems like a bit of a stretch...
Here's a question: if I told you the person receiving the recommendations currently owned more than one album by Miles Davis, but *NOTHING* from list A, does that make list A a better or the worse list?
Posted by stephen on September 18, 2007 at 01:06 PM EDT #
Yet another crazy angle: Since music from artists like Duke Ellington, Bill Evans (most music by) Sonny Rollins is more accessible or "listenable" than more Avant-Gardeish artists like Can or Dave Douglas, would the majority of people gravitate toward List A?
Despite the fact that List B may be more representative of the overall historical career of the artist, if people are hearing music they like in List A, has the system then made a better "Recommendation" (meaning suggesting music the person might like).
Is there any value in skewing the results toward more popular or widely "acceptable" songs? (I'm not advocating, just posing the question).
Posted by Zac on September 18, 2007 at 03:19 PM EDT #
A is just a straightforward and obvious jazz list.
B looks a bit more interesting since it throws in a joker or two.
It does not have to mean that A is machine generated and B generated by a human. If the B list has been created to drop in a couple of "semi-random, but still related" artists it could still be machine generated.
If the idea is "let me listen to what I already like, because I only dig jazz" then I guess it will do. But wouldn't you already know all of the acts?
B for better. At least it lets you expand your horizon a bit. Which I think Miles was a bit about anyways ...
Posted by Hal on September 21, 2007 at 11:54 AM EDT #
I think list A is probably the human. While both lists contain artists that a Miles Davis fan is statistically likely to enjoy, list A appears to have been constructed with reference to some sort of notion of 'genre' (specifically, jazz, in this case),which is a pretty vague taxonomy, and therefore more likely to be the result of human influence.
Posted by Tim on October 02, 2007 at 07:17 AM EDT #
I bet B is machine generated, although it's a tough one to call. I'm guessing the human expert would be more aware of genre boundaries, be they real or perceived.
Posted by eric casteleijn on October 02, 2007 at 07:23 AM EDT #
hello,
miles was and still is one of hot players, we miss his trumpeter! we was teh godfather of jazz!
A it´s Humam
B it´s Machine kind of intelligence =P
keep the good work,
my reagards
p.s.: want the books =P & not easy math so submit the post eheheh;)
Posted by André do Carmo on October 02, 2007 at 07:51 AM EDT #
Hi,
Here is my opinion:
A = human
B = machine
but just because I think B is better so it should be your system :-)
thanks for your tutorial with oscar at Ismir.
Posted by Cyril Laurier on October 02, 2007 at 08:01 AM EDT #
Dear Paul,
awesome posting :)
A = a conservative human music expert (maybe using a community/cultural metadata system :) )
B = an open-minded recommendation engine in a funky mood
PS I ordered already the give-away, so please excluded me from the selection process:)
Posted by Stephan on October 02, 2007 at 10:30 AM EDT #
A=Human, B=Machine.
Posted by Brian Utterback on October 02, 2007 at 12:01 PM EDT #
A = Machine. B = Human.
I think B's oddities indicate a single person's bias, whereas A would be the average list I would see from a bunch of people.
Posted by Jeremy Reed on October 02, 2007 at 01:46 PM EDT #
It's fifty-fifty, but maybe A could be human...
Posted by davide on October 02, 2007 at 02:29 PM EDT #
A: Human stuck more to the jazz genre
B: Machine - too many variables given by the machine which don't directly correlate with the question asked.
Posted by Terry on October 02, 2007 at 09:09 PM EDT #
I can't see that even a machine list listening to Miles Davis would include Tony Williams Lifetime, so I suspect that's the fabulosity of your algoritm, so I vote:
A: Human
B: Machine
However, if it was some other recommendation system, I would probably have reversed it. I'm probably second and third guessing completely incorrectly.
Mind you, it has made me dig out a Sly and the Family Stone album, so I would prefer B either way, just for the genre-merge.
Posted by Tim Caynes on October 03, 2007 at 05:38 AM EDT #
A: Human
B: Machine
B has differences that wouldn't show up in a human's list because a human will stick to the genre.
That said I think I like B more.
Posted by Steve on October 03, 2007 at 09:53 AM EDT #
a = human
b = machine
i also enjoy b's recs more. prepare for the rise of the machine!
Posted by mark on October 03, 2007 at 11:10 AM EDT #
A = Human
B = Machine
Posted by Carlos Gortaris on October 05, 2007 at 09:17 AM EDT #
A - human
B - machine
Posted by Liam on October 05, 2007 at 09:55 AM EDT #
I'd have to say that :
A= Machine
B = Human
Posted by Stephen on October 06, 2007 at 05:37 AM EDT #
A=Machine
B=Human
Posted by martin on October 20, 2007 at 08:50 AM EDT #
A=human B=machine
I know little jazz and nothing by the people listed but I could see (before reading the comments) that there was 50% overlap between the lists and this overlap occurred in the TOP HALF of list B.
That makes A probably someone's rambling mind and B the more googleicious.
Posted by peter on October 20, 2007 at 04:30 PM EDT #