finding music with pictures: using visualization for discovery

86
Finding Music With Pictures: Data Visualization for Discovery Paul Lamere #musicviz The Echo Nest

Upload: paul-lamere

Post on 26-Jan-2015

106 views

Category:

Technology


0 download

DESCRIPTION

Slides from my SXSW 2011 talk. Here's the abstract:With so much music available, finding new music that you like can be like finding a needle in a haystack. We need new tools to help us to explore the world of music, tools that can help us separate the wheat from the chaff. In this panel we will look at how visualizations can be used to help people explore the music space and discover new, interesting music that they will like. We will look at a wide range of visualizations, from hand drawn artist maps, to highly interactive, immersive 3D environments. We'll explore a number of different visualization techniques including graphs, trees, maps, timelines and flow diagrams and we'll examine different types of music data that can contribute to a visualization. Using numerous examples drawn from commercial and research systems we'll show how visualizations are being used now to enhance music discovery and we'll demonstrate some new visualization techniques coming out of the labs that we'll find in tomorrow's music discovery applications.

TRANSCRIPT

Page 1: Finding Music With Pictures: Using Visualization for Discovery

Finding Music With Pictures: Data Visualization for Discovery

Paul Lamere #musicvizThe Echo Nest

Page 2: Finding Music With Pictures: Using Visualization for Discovery

Music Discovery Challenge

2

Page 3: Finding Music With Pictures: Using Visualization for Discovery

Music Discovery Challenge

3

Listener Study

Listeners 5,000

Average Songs Per User 3,500

Percent of songs never listened to 65%

Page 4: Finding Music With Pictures: Using Visualization for Discovery

Ishkur’s Guide to Electronic Dance Music

4

Page 5: Finding Music With Pictures: Using Visualization for Discovery

Why Visualize?

5

Visualizations can engage our pattern-matching brain

Page 6: Finding Music With Pictures: Using Visualization for Discovery

Human rendered visualizations

6

Page 7: Finding Music With Pictures: Using Visualization for Discovery

Visualizations as a tree

71939

Page 8: Finding Music With Pictures: Using Visualization for Discovery

Visualizations as a tree

71939

Page 9: Finding Music With Pictures: Using Visualization for Discovery

Visualizations as a tree

71939

Page 10: Finding Music With Pictures: Using Visualization for Discovery

The greatest visualization ever?

8

Page 11: Finding Music With Pictures: Using Visualization for Discovery

Visualizations as a Map

9Dorian Lynskey - The Guardian

Section:GDN RR PaGe:8 Edition Date:060203 Edition:01 Zone: Sent at 1/2/2006 20:06 cYanmaGentaYellowblack

The Guardian | Friday February 3 2006 98 The Guardian | Friday February 3 2006

Cover story

Going underground

Could we chart the branches and connections of 100 years of music using the London Underground map? Dorian Lynskey explains how a box of coloured crayons and a lot of swearing helped

It seems like a deeply implausibleproject: to plot the history of 20thcentury music on the LondonUnderground map devised byHarry Beck in 1933. Artist SimonPatterson transformed the tube

map into a constellation of famousnames in his 1992 work The Great Bear,but he didn’t have to make them all linkup. It is, after all, a tall order to find asaint who was also a comedian. But forthis one to work every interchange hadto be logical in the context of musicalhistory, an unlikely prospect.

I started out with a packet of colouredcrayons, four sheets of A4 taped to-gether and a big box of doubt, but thedifferent character of each line quicklylent itself to a certain genre. Pop inter-sects with everything else, so that had tobe the Circle Line; classical music for themost part occupies its own sphere,which made it perfect for the DocklandsLight Railway. There were a couple offalse starts but by the end of one after-noon I had assigned genres to almost allthe lines and thrashed out most of themajor intersections. The key stationsnaturally went to the most eclecticartists, not necessarily the most impor-tant: the Beatles may be more signifi-cant than Beck but even their most de-voted fan must admit that they nevertried rapping.

The system thus in place, the nextcouple of days were devoted to writingnames in, scribbling them out (sorry,Doug E Fresh and Lynyrd Skynyrd), ago-nising over certain omissions, askingclassical music critic Tom Service for in-valuable help with the DLR, and swear-ing just a little bit. Amazingly, therewere no calamitous blind alleys. It justseemed to make sense.

I tried as far as possible to be objec-tive. Some bands I cannot stand are inhere, while some that I love dearlyaren’t. I also followed chronology wher-ever the path of the line allowed it. Eachbranch line represents a sub-genre: rocksprouts off into grunge and psychedeliawhen it reaches South-West London;hip-hop diverges, north of Camden, intoold school and New York rap. If I was re-ally lucky, the band name echoed theoriginal station name: Highbury & Is-lington became Sly & the Family Stone.

Pedants, of course, will find flaws.Musical influences are so labyrinthinethat any simple equation will be imper-fect. Where, for example, does pop stopand rock begin? How can you draw a de-cisive line between soul and funk? Theseare problems that have plagued recordshop proprietors for decades and they’renot going to be solved here. But I thinkall of these choices are justifiable giventhe limitations of the form.

Other people will quibble with omis-sions – it’s a shame, for example, thatthe Circle Line constantly runs in tan-dem with either the District or Metro-politan lines, thus leaving no room forpure pop acts such as Kylie Minogue andthe Pet Shop Boys. I should also pointout that, to keep my head from explod-ing, I limited the remit to western, pre-dominately Anglo-American music.Then there are those changes necessi-tated by London Underground’s under-standable sensitivity to explosive refer-ences: arrividerci, Massive Attack. Forsome reason, they also took exception tothe late rapper Ol’ Dirty Bastard.

But this is not some definitive historyof music. It’s an experiment to see if oneintricate network can be overlaid on acompletely different one. The eleganceand logic of Harry Beck’s design – itscombination of bustling intersections,sprawling tributaries, long, slanting tan-gents and abrupt dead ends, all suckedinto the overturned wine bottle of theCircle Line – seems to spark other con-nections and appeal to the brain’s innatedesire for patterning and structure. Plusit’s fun, as any piece of music journalismcreated with coloured crayons shouldbe. I hope you like it.

Tell us what you think of the map atwww.guardian.co.uk/artsBuy the map atwww.ltmuseumshop.co.ukSpecial thanks to Chris Townsend atTransport for London and AndrewJones at London Underground.

Music History as a London Tube Map

Page 12: Finding Music With Pictures: Using Visualization for Discovery

Visualizations as a Map

9Dorian Lynskey - The Guardian

Section:GDN RR PaGe:8 Edition Date:060203 Edition:01 Zone: Sent at 1/2/2006 20:06 cYanmaGentaYellowblack

The Guardian | Friday February 3 2006 98 The Guardian | Friday February 3 2006

Cover story

Going underground

Could we chart the branches and connections of 100 years of music using the London Underground map? Dorian Lynskey explains how a box of coloured crayons and a lot of swearing helped

It seems like a deeply implausibleproject: to plot the history of 20thcentury music on the LondonUnderground map devised byHarry Beck in 1933. Artist SimonPatterson transformed the tube

map into a constellation of famousnames in his 1992 work The Great Bear,but he didn’t have to make them all linkup. It is, after all, a tall order to find asaint who was also a comedian. But forthis one to work every interchange hadto be logical in the context of musicalhistory, an unlikely prospect.

I started out with a packet of colouredcrayons, four sheets of A4 taped to-gether and a big box of doubt, but thedifferent character of each line quicklylent itself to a certain genre. Pop inter-sects with everything else, so that had tobe the Circle Line; classical music for themost part occupies its own sphere,which made it perfect for the DocklandsLight Railway. There were a couple offalse starts but by the end of one after-noon I had assigned genres to almost allthe lines and thrashed out most of themajor intersections. The key stationsnaturally went to the most eclecticartists, not necessarily the most impor-tant: the Beatles may be more signifi-cant than Beck but even their most de-voted fan must admit that they nevertried rapping.

The system thus in place, the nextcouple of days were devoted to writingnames in, scribbling them out (sorry,Doug E Fresh and Lynyrd Skynyrd), ago-nising over certain omissions, askingclassical music critic Tom Service for in-valuable help with the DLR, and swear-ing just a little bit. Amazingly, therewere no calamitous blind alleys. It justseemed to make sense.

I tried as far as possible to be objec-tive. Some bands I cannot stand are inhere, while some that I love dearlyaren’t. I also followed chronology wher-ever the path of the line allowed it. Eachbranch line represents a sub-genre: rocksprouts off into grunge and psychedeliawhen it reaches South-West London;hip-hop diverges, north of Camden, intoold school and New York rap. If I was re-ally lucky, the band name echoed theoriginal station name: Highbury & Is-lington became Sly & the Family Stone.

Pedants, of course, will find flaws.Musical influences are so labyrinthinethat any simple equation will be imper-fect. Where, for example, does pop stopand rock begin? How can you draw a de-cisive line between soul and funk? Theseare problems that have plagued recordshop proprietors for decades and they’renot going to be solved here. But I thinkall of these choices are justifiable giventhe limitations of the form.

Other people will quibble with omis-sions – it’s a shame, for example, thatthe Circle Line constantly runs in tan-dem with either the District or Metro-politan lines, thus leaving no room forpure pop acts such as Kylie Minogue andthe Pet Shop Boys. I should also pointout that, to keep my head from explod-ing, I limited the remit to western, pre-dominately Anglo-American music.Then there are those changes necessi-tated by London Underground’s under-standable sensitivity to explosive refer-ences: arrividerci, Massive Attack. Forsome reason, they also took exception tothe late rapper Ol’ Dirty Bastard.

But this is not some definitive historyof music. It’s an experiment to see if oneintricate network can be overlaid on acompletely different one. The eleganceand logic of Harry Beck’s design – itscombination of bustling intersections,sprawling tributaries, long, slanting tan-gents and abrupt dead ends, all suckedinto the overturned wine bottle of theCircle Line – seems to spark other con-nections and appeal to the brain’s innatedesire for patterning and structure. Plusit’s fun, as any piece of music journalismcreated with coloured crayons shouldbe. I hope you like it.

Tell us what you think of the map atwww.guardian.co.uk/artsBuy the map atwww.ltmuseumshop.co.ukSpecial thanks to Chris Townsend atTransport for London and AndrewJones at London Underground.

Music History as a London Tube Map

Page 13: Finding Music With Pictures: Using Visualization for Discovery

Genre Map

10Ben Discoe http://www.washedashore.com/music/genremap.html

Page 14: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011http://mapofmetal.com

Map of Metal

11

Page 15: Finding Music With Pictures: Using Visualization for Discovery

Visualization as a Time Series

12

Genealogy of Pop and Rock Music - Reebee Garofalo

Page 16: Finding Music With Pictures: Using Visualization for Discovery

Visualization as a Time Series

12

Genealogy of Pop and Rock Music - Reebee Garofalo

Page 17: Finding Music With Pictures: Using Visualization for Discovery

Visualization as a Time Series

12

Genealogy of Pop and Rock Music - Reebee Garofalo

Page 18: Finding Music With Pictures: Using Visualization for Discovery

Visualization as a Time Series

12

Genealogy of Pop and Rock Music - Reebee Garofalo

Page 19: Finding Music With Pictures: Using Visualization for Discovery

13

Visualizations as a graph

http://www.homeofmetal.com/

Page 20: Finding Music With Pictures: Using Visualization for Discovery

14

Seattle Band Map

http://www.seattlebandmap.com/

Page 21: Finding Music With Pictures: Using Visualization for Discovery

15

Seattle Band Map

http://www.seattlebandmap.com/

Page 22: Finding Music With Pictures: Using Visualization for Discovery

16

Hybrid visualization

franksreelreviews.comSchool of rock

Page 23: Finding Music With Pictures: Using Visualization for Discovery

16

Hybrid visualization

franksreelreviews.comSchool of rock

Page 24: Finding Music With Pictures: Using Visualization for Discovery

16

Hybrid visualization

franksreelreviews.comSchool of rock

Page 25: Finding Music With Pictures: Using Visualization for Discovery

Other types of visualizations

17

Page 26: Finding Music With Pictures: Using Visualization for Discovery

Other types of visualizations

18

Page 27: Finding Music With Pictures: Using Visualization for Discovery

Other types of visualizations

19

Page 28: Finding Music With Pictures: Using Visualization for Discovery

Other types of visualizations

20

Page 29: Finding Music With Pictures: Using Visualization for Discovery

Other types of visualizations

21

Page 30: Finding Music With Pictures: Using Visualization for Discovery

Computer rendered visualizations

22

Page 31: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011 23

Time Series

Page 32: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011 24

http://www.leebyron.com/what/lastfm/ - Lee Byron

Visualizing Listening History

Page 33: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011 24

http://www.leebyron.com/what/lastfm/ - Lee Byron

Visualizing Listening History

Page 34: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011

Maps and Graphs

25

Page 35: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011

http://www.stanford.edu/~dgleich/demos/worldofmusic/WorldOfMusic.htmlDavid Gleich, Matt Rasmussen, Leonid Zhukov, and Kevin Lang

z

The World Of Music

26

Page 36: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011

http://www.stanford.edu/~dgleich/demos/worldofmusic/WorldOfMusic.htmlDavid Gleich, Matt Rasmussen, Leonid Zhukov, and Kevin Lang

z

The World Of Music

26

Page 37: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011

The Landscape of Musicby Gansner, Hu, Kobourov and Volinsky

AT&T Labs-Research

The Landscape of Music

27

Page 38: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011

The Landscape of Musicby Gansner, Hu, Kobourov and Volinsky

AT&T Labs-Research

The Landscape of Music

27

Page 39: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011

The Landscape of Musicby Gansner, Hu, Kobourov and Volinsky

AT&T Labs-Research

The Landscape of Music

27

Page 40: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011

The Landscape of Musicby Gansner, Hu, Kobourov and Volinsky

AT&T Labs-Research

The Landscape of Music

27

Page 41: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011

http://playground.last.fm/iom - Islands of Music - Elias Pampalk

Islands of Music

28

self-organizing maps

Page 42: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011

http://www.cp.jku.at/projects/nepTune/

nepTune

29

Page 43: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011

http://sixdegrees.hu/last.fm/index.html

Last.fm Artist Map

30

Page 44: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011

http://sixdegrees.hu/last.fm/index.html

Last.fm Artist Map

30

Page 45: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011

The Goal - build an artist graph suitable for music exploration

Building an artist map

31

Problem: The artist space is very complex

Page 46: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011

The Goal - build an artist graph suitable for music exploration

Building an artist map

31

Problem: The artist space is very complex

Page 47: Finding Music With Pictures: Using Visualization for Discovery

Making a good graph

• Minimize edge crossings, bends and curves

• Maximize angular resolution, symmetry

32

Good Bad

BadGood

Cognitive measurements of graph aesthetics, Ware, Purchase, Colpoys, McGill

Page 48: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011

digraph {

"The Beatles" -> "Electric Light Orchestra";

"The Beatles" -> "The Rutles";

"The Beatles" -> "The Dave Clark Five";

"The Beatles" -> "The Tremeloes";

"The Beatles" -> "Manfred Mann";

"The Beatles" -> "The Lovin' Spoonful";

"The Beatles" -> "The Rolling Stones";

"The Beatles" -> "Creedence Clearwater Revival";

"The Beatles" -> "Bee Gees";

"The Beatles" -> "the Hollies";

"The Beatles" -> "Badfinger";

"The Beatles" -> "Paul McCartney";

"The Beatles" -> "Julian Lennon";

"The Beatles" -> "George Harrison";

"The Beatles" -> "John Lennon";

}

Graphing the Beatles Neighborhood

33http://www.graphviz.org/

Page 49: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011 Already interconnections start to overwhelm

Artist map - level 1 out

34

Page 50: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011

Artist map - level 1 in/out

35

Page 51: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011

Artist map - level 1 in/out

35

Page 52: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011

Artist map - level 1 in/out

35

Page 53: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011

Artist map - level 1 in/out

35

Page 54: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011

Artist map - level 1 in/out

35

Page 55: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011

Artist map - level 1 in/out

35

Page 56: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011

Artist map - level 1 in/out

35

Page 57: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011

Artist map - level 1 in/out

35

Page 58: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011

116 artists and 665 edges. We are totally overwhelmed.

Artist map - level 2

36

Page 59: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011

116 artists and 665 edges. We are totally overwhelmed.

Artist map - level 2

36

Page 60: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011

Eliminate the edges and use spring force embedding to position the artists

Simplifying the graph

37

Page 61: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011

Eliminate the edges and use spring force embedding to position the artists

Simplifying the graph

37

Page 62: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011

Attach artists to only one most similar artist

Reducing the edges

38

Page 63: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011

Attach artists to only one most similar artist

Reducing the edges

38

Page 64: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011

Attach artists to only one most similar artist

Reducing the edges

38

Page 65: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011

Attach artists to only one most similar artist

Reducing the edges

38

wtf?

Page 66: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011

Attach artists to the most similar artist that has greater familiarity

Putting ELO II in its place

39

Page 67: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011

Attach artists to the most similar artist that has greater familiarity

Putting ELO II in its place

39

Page 68: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011

Attach artists to the most similar artist that has greater familiarity

Putting ELO II in its place

39

Page 69: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011

Graph is not 100% connected.

One more problem ...

40

Page 70: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011

Graph is not 100% connected.

One more problem ...

40

Page 71: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011

Build a minimum spanning tree from the graph

Making the graph connected

41

Page 72: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011

Build a minimum spanning tree from the graph

Making the graph connected

41

Page 73: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011

A browsing network for music discovery

2K artists

42

Page 74: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011

A browsing network for music discovery

2K artists

42

Page 75: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011

Mirrored Radio dial - by thomwatson

Interactive

43

Page 76: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011

The SXSW Music Maze

44http://static.echonest.com/SxswMusicMaze/SxswMusicMaze.html

Page 77: Finding Music With Pictures: Using Visualization for Discovery

liveplasma

45http://liveplasma.com

Page 78: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011

Flow List

46http://bowr.de/flowlist/flowlist.html

Page 79: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011

MoodLogic

47

Page 80: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011

http://mpac.ee.ntu.edu.tw/~yihsuan/pub/MM08_MrEmo.pdf

Mr. Emo: Music Retrieval in the Emotional Plane

48

Page 81: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011 http://thesis.flyingpudding.com/ - Anita Lillie

Music Box

49

Page 82: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011

The Mufin Player

3D Track similarity

50

Page 83: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011

Discovr

51

Page 84: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011

www.hitlantis.com

Hitlantis

52

Page 85: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011

SongExplorer: Exploring Large Musical Databases Using a Tabletop InterfaceCarles F. Julià - Music Technology Group - Universitat Pompeu Fabra

SongExplorer

53

Page 86: Finding Music With Pictures: Using Visualization for Discovery

Finding music with pictures - SXSW 2011

http://mediacast.sun.com/share/plamere/sitm_final.pdf - Lamere, Eck

Search Inside the Music

54