Monday Nov 26, 2007

Over the Thanksgiving break, I read the Ghost Map The Story of London's Most Terrifying Epidemic--and How It Changed Science, Cities, and the Modern World by Stephen Johnson.  This is the story how John Snow  convinced the London health authorities that the Cholera epidemic was not caused by bad air (miasma), but was caused by contaminated drinking water.   Johnson does an excellent job describing what life was like in London during the middle of the 19th century.  Imagine a city of 2 million with no sewers - in fact much human waste would end up in the cellars of the poorer housing (and from the cellars, the waste could make its way to the public well.)  There was a scavenger class of night soil men, toshers, shoremen, pure collectors and others that made their living dealing with waste.  Despite the fact that Londoner's were surrounded by all of waste, it was not obvious that this was related to the spread of disease.  John Snow was convinced that all of this waste was polluting the drinking water.  His challenge was to collect enough evidence in order to convince the establishment that this was the case.  Snow used his familiarity with the Broad Street neighborhood, plenty of shoeleather,  and (eventually) visualizations (the ghost map) to make his case, and the authorities removed the handle from the Broad Street pump.  All in all,  a good read, highly recommended.

The book has a website with a number of good resources including a high resolution copy of the ghost map.


Wednesday Nov 21, 2007

Lately there's been quite a bit of attention being paid to making sure that the data that describes the things that we like,  our attention data, is portable.  With portable attention data, we could go to any music store and be directed to the music that we are most likely to want to listen to.  We won't have to spend any time rating tracks or artists, we'll just show the music store our taste data.  Of course, this taste data needs to be in some standard format so that everyone can understand it. One effort at standardizing our taste data is APML.  APML is an XML based language that allows users to share their own personal taste data in much the same way that OPML allows the exchange of reading lists between blog readers.   APML is new and not finished yet, but even in its infant state, it is garnering lots of support.  

I am particularly interested in how APML could be used to represent an individual's music taste.   One possibility is to have the APML file for the individual list the artists that a person likes (or vehemently dislikes).  Another approach is to have the preferences be more abstract - to list weighted affinities toward music genres or styles.   The latter approach seemed much more interesting to me - it offers some bit of privacy (instead of seeing Paris Hilton in my APML file, you would just see  Female Pop Singer). 

As an experiment, I've created a little APML generator web service for users.  If you give the web service your user name, the service goes to, retrieves data about your listening habits and generates an APML representation of your taste.  For example, to retrieve the APML for my listening tastes visit the following URL:

This yields:

<APML version="0.6">
<Title>music taste for lamere</Title>
<Generator>Created by in 2777 ms </Generator>
<Body defaultprofile="music">
  <Profile name="music">
       <Concept key="rock" value="1.0" from="" updated="2007-11-21T16:15:24"/>
       <Concept key="alternative" value="0.74616855" from="" updated="2007-11-21T16:15:24"/>
       <Concept key="indie" value="0.63257456" from="" updated="2007-11-21T16:15:24"/>
       <Concept key="alternative rock" value="0.38583755" from="" updated="2007-11-21T16:15:24"/>
   <!-- many lines omitted -->


To generate the implicit concepts, I gather the top 50 artists for the user, and for each of these artists I gather the top 50 tags that have been applied to each of those artists.  I adjust the weight of the tags based on the user's affinity for the associated artist. I then take top scores to generate the APML. Of course, all this chattering with can make the web service quite slow.  I do try to cache as much data as I can to try to speed things up, but if you have eclectic tastes, it can take up to a minute or so to generate your APML file.

The resulting APML seems to be a good representation of my taste.  I'm interested in hearing from others that might be users whether or not the generated APML file is a good map of their taste.  Feel free to try out the web service.  The URL is:

The next step is to see how well we can generate recommendations based upon these APML files.

Tuesday Nov 20, 2007

As Toby shuts down Musicmobs, he has left us with a gift - an archive of all of the playlists that Musicmobs has collected over the years.  It is a nice set of data for anyone looking to do research around artist or track co-occurrence in playlists.   Some stats about the data:


Unique Tracks

 This is really good data! Thanks Toby.

Toby Padilla, the creator of Musicmobs, one of the first social music sites has joined the team.  Musicmobs was clearly ahead of the curve - pioneering many aspects that are now seen as requirements in the 'Music 2.0' space. Social playlisting, recommendation, open web services,  implicit gathering of taste data, - the 'hipster <-> mainstream slider', excellent design, cool name: Musicmobs has it all.  It will be sad to see Musicmobs close its doors as Toby moves to, but I am really excited to see what happens when you connect the vision that Toby has for Music 2.0 with the resources that has.   Congrats to Toby and kudos to for building the social music dream team!

Sunday Nov 18, 2007

This week, Pandora opened access to their classical collection, and this morning I'm giving it a try. What better way to spend a Sunday morning than reading the paper and listening to some Vivaldi radio?  Pandora currently has about 20,000 classical recordings - more than I am likely to listen to even with a month of Sunday mornings.  Thanks Pandora!

Saturday Nov 17, 2007

iLike has an exclusive clip from U2 - a song intended for Joshua Tree that was never finished.  This is quite a coup for iLike ... this type of first tier music tie-in is usually reserved for the likes of Steve Jobs.  ValleyWag has the backstory about how this deal happened.

Friday Nov 16, 2007

I had the opportunity to check out a beta version of Spotify recently, and  I am really liking what I am seeing.    Spotify is a music service that lets you listen to music on demand from a large number artists.

Spotify lets you create Pandora-like radio stations based upon a favorite artist.  Instead of using the music genome to find similar tracks as Pandora does, and instead of using the wisdom of the crowds to populate their radio stations like does, they use the All Music 'similar artists' data to populate their radio stations.  Spotify makes good use of the AMG data too, they show artist bios, photos,  and album art.

Spotify also has a genre/era based radio.  You can choose from a set of 18 top level genres, and narrow down the year of interest to give you a focused, but artist independent radio station.


Any song that you see in Spotify, you can play instantly. Apparently Spotify has (or is working on) arrangements with the labels to allow for this playback on demand.  The playback was solid, good quality audio with minimal dropouts. It compared well to my experience with other streaming services like or Pandora. Spotify also lets you create playlists of tracks. Playlists are saved and you can play them back at anytime.  Spotify may also be able to provide persistent URLs for songs ... something that the web has been needing for quite a while.

The next generation of music services are going to be all about giving listeners music on demand.  Services like Spotify, Finetune, Deezer  no longer are constrained by internet streaming rules - you can play songs by any artist at anytime.  Soon the more established services like and Pandora will start to seem old fashioned - as they conform to the rather restrictive rules about internet radio enforced by SoundExchange. 

Thursday Nov 15, 2007

If you are not too far away from NYC,  consider attending the first meeting of NEMISIG on January 25th.  This gathering is being organized by Dan Ellis and Michael Mandel of Columbia. They hope to provide graduate students with an informal and safe environment in which to share current research and discuss ideas for future research. It will offer principals the opportunity to present their research interests and visions of the field, and to participate in discussions of the field and future directions it might take.  There's a good set of participants already including:

  • Adobe Labs
  • The College of New Jersey
  • Columbia
  • Connecticut College
  • Dartmouth
  • Drexel
  • The Echo Nest
  • Harvard / USC
  • McGill
  • NYU
  • Sun Microsystems
  • Université de Montréal
In particular there's a call for discussion topics for panel or breakout sessions so send the organizers your suggestions.



APML, the attention profiling mark-up language,  is a standard for representing and exchanging attention data - data about what I like and what I don't like. Anyone who's involved in recommendation to should be paying attention to APML.  A good place to start is this article: Attention Profiling: APML Beginner's Guide

Tuesday Nov 13, 2007

If I were in London tomorrow I would head over to Mile End road to hear Stephan Baumann give a talk on his ideas about the future of music recommendation. Stephan's talk: In Search of the Goosebump Factor - A Blueprint for Emotional Music Recommenders:   Here's the abstract:

Music Information Retrieval (MIR) as an interdisciplinary research discipline achieved impressive progress over the last decade. Pandora, or MyStrands are successful commercial webservices offering previously unavailable convenience for customers. Although such systems compute personalised recommendations based on relevance feedback on top of content-based, expert-based or community metadata, the embedding of emotional context is still a challenge. In my talk I will sketch a blueprint towards an architecture of an emotional music recommender in order to solve the abovementioned problem. The approach is in its infancy but we have already the core ingredients developed. Lifestream aggregation from Web2.0 platforms and the analysis of blog postings will be aligned with the analysis of song lyrics. Furthermore we propose an open Web2.0 platform in order to collect personal descriptions of "goosebump sensations" when listening to music. This collection will be available to researchers in the field to serve as a common ground for training emotional classifiers.

Here are the details. 

Sunday Nov 11, 2007

Cari at Niagara
Originally uploaded by PaulLamere.
This weekend, Cari and I took a road trip to Western New York state to visit a few colleges. Today we had some free time so we took a visit to Niagara falls. Neither of us had been there before. Since it was off season, most of the attractions were closed, which means there were not too many people there, so it was actually quite nice.

Wednesday Nov 07, 2007

NPR Music  has gone live.  NPR Music is a "a free, multi-genre web site that presents the best of public radio music". The site is organized by genre and by type of musical content including live concerts, studio sessions, artist interviews, profiles, reviews, blogs and podcasts.  There's a particularly nifty session featuring the Animal Collectiive.
The maker of Rip Guard is going to purchase  All Music, the music metadata company.  Details in this press release.  Something tells me that Macrovision, the king of copy protection, is not going to be building rich web services around  the AMG  data. Sigh. (Update - fixed the title, yep its Macrovision, not Macromedia).

Tuesday Nov 06, 2007

What do you do if you want to build a music search engine that will allow people to search for tracks from any artist and instantly play them, but you really don't want to pay those pesky music licensing fees?  Why stream music videos from YouTube of course. Just make sure that you hide the video.  This is what Songza is doing.  Yep ... it is illegal to stream music on demand, but it is perfectly OK to stream music videos. What a crazy world. - (via Tech Digest)

Everyone is excited that Apple will be releasing an SDK for the iPhone in February.  This could potentially open up many avenues for new types of interfaces to interact with one's music collection.  Imagine Ishkur's Guide to Dance music on your phone, or one of Masataka's and/or Elias's fantastic interfaces  like MusicReam, MusicRainbow or MusicSun, or Frederic Vavrille's Musicovery, or our own Search Inside the Music

My worry is that the when it comes to the music the iPhone SDK will be crippled - that the SDK will not provide for basic manipulation of the music queue and that we won't be able to write new music apps.   We will all know for sure in February, but in the mean time I want to make sure that Apple knows what it should include in the API to support music apps.  Here's the absolute minimum:

  •  Basic Player controls
    • Start playing a song
    • Stop playing
    • Volume up/down
    • Pause/Resume
  • Player events:
    • Song Started
    • Song Finished
    • User skipped, user pause/resume
    • User volume up, volume down
  • Basic Now Playing Queue controls
    • Enqueue a song
    • Dequeue a song
    • Next song
    • Previous song
  • Song database info
    • Get info for a song including:
      • title, artist, album, genre, composter
      • track number, year of release, length
      • playcount, skipcount, rating, BPM
      • bitrate, last played, format
      • album art
    • Get all artists
    • Get all albums
    • Get all tracks
      • by genre
      • by artist
      • by album
    • Add a track to the database
  • Playlist manipulation
    • Get all playlists
    • Create a playlist
    • Delete a playlist
    • Get tracks in a playlist
    • Add tracks to a playlist

With this minimal SDK support for music we should see some really interesting apps for exploring our personal music collection.  These apps will help prevent our iPod / iPhone from becoming the music graveyard - the place that our music goes to die.

This blog copyright 2010 by plamere