Photographer Paths: Sequence Alignment of Geotagged Photos for Exploration-based Route Planning!

Abdallah  ‘Abdo’  El  Ali  Sicco  van  Sas  

Frank  Nack  

Feb.  26,  2013

h6p://staff.science.uva.nl/~elali/  

Outline!

I.  Introduc3on  

II.  Photographer  Paths  

III.  User  Evalua3on  

IV.  Results  

V.  Discussion  &  Future  Work  

2

3

Introduction

4

5

6

Pffttt…

   We  don’t  always  want  to  supply  user  preferences  

7

 Social  and  local  interpreta3on  of  city  places  and  routes  8

Off-­‐the-­‐beaten  track,  social  trails,    ≠  Lonely  Planet!    

9

 Assump3on:  

 Loca3ons  of  photographs  are  poten3ally  interes3ng  

10

 But  sequen3al  property  needs  to  be  captured!  

11

 Sequence  Alignment  methods  

12

 Analysis  of  mobility  behavior  of  city  photographers:  

  where  photographers  have  been  

  in  what  order  they  have  been  there  

  how  closely  their  movements  parallel  those  of  other  photographers  

13 By Keiichi Matsuda via supercolossal

Research Questions!

 How  can  walkable  route  plans  be  automaCcally  generated  for  residents  (and  tourists)  that  would  like  to  explore  a  city?  

 And  are  these  route  plans  desirable?  

14

 Three  factors:  

 1)  Which  data  sources?    2)  Which  methods  to  generate  routes?  

                   3)  User  percep3ons  compared  to  fastest  and  popular  routes?    

15

Photographer Paths!

Approach!

 1)  Crawl  Flickr    geotags,  3mestamps  

 2)  Map  each  geotag/loca3on  in  a  sequence  to  a  cell  in  a  par33oned  grid  map  

 3)  Mul3ple  Sequence  Alignment  on  photographer  routes  to  find  aligned  loca3on  sequences  

   These  alignments  are  Photographer  Route  Segments  (PRSs)  

16

Dataset!

  Flickr  geotagged  photos  within  Amsterdam,  The  Netherlands    

  Area:  17.3  km  N-­‐S  and  24.7  km  E-­‐W  (center)  

  5-­‐year  period  (Jan.  2006  -­‐  Dec.  2010)  

  Aeributes:    

  owner  ID  

  photo  ID  

  date  and  3me-­‐stamp  

  la3tude  and  longitude  (street  level  accuracy)  

 Database:  426,372  photos  

17

Preprocessing!

  Sequence  inference  with  following  constraints:  

  photo  taken  within  4  hours  from  previous  photo  and  in  same  order  

  minimum  2  or  more  different  loca3ons  (or  nodes)  

  early  experiments  determined  125  x  125m  cells  in  center  of  Amsterdam  grid  suitable  

 1691  routes  (average  length  of  9.92  loca3ons)  

 1130  unique  photographers  

18

!"#$%

##&%

!'(%

)'%

'!&%

!*!%

!*%

!"#$

+'% !%*%

+**%

"**%

#**%

$**%

!***%

!+**%

!"**%

!#**%

+% )% "% '%

%&'()*+$,*-.*)/*0$

1)'-.*$&2/342)0$

Photographer Route Segments (PRSs)!

+%,-./.012,-314%

)%,-./.012,-314%

"%,-./.012,-314%

Sequence Alignment!

  To  find  photographer  paths  from  Photographer  Route  Segments  (PRSs),  constraints  set:  

  PRS  has  minimum  4  photographers  with  minimum  2  aligned  nodes/loca3ons  

           231  PRSs    

 (average  length  of  2.61  nodes)  

19

PRSs in Amsterdam!

20

PRS Aggregation!

  Modified  Dijkstra’s  shortest  path  algorithm  

21

Start

P1N2

EndP2N2

P1N1 P2N1

!

"

#$%

&

!

P1N3

&

#

!"#

PRS Aggregation to Crude Routes!

22

WW Photographer Route

CM Photographer RoutePRSs

23

User Evaluation

Laboratory Study Design!

  ~45  min.  Quan3ta3ve/Qualita3ve  lab-­‐based  study  

  15  par3cipants  (10  m,  5  f)  aged  between  21-­‐35  (M  =  29.2;  SD  =  3.3)  

  Interac3ve  web-­‐based  prototype  route  planner  

  Expert  route  evalua3on  by  ‘city  residents’  (lived  in  Amsterdam  >  1  year)  

  Plain  routes  to  avoid  informa3on  type  bias  

24

Laboratory Study Design!

  Two  scenarios:  

  Route  1:  Central  Sta3on  to  Museumplein    anernoon  scenario  favoring  explora3on  

  Route  2:  Waterlooplein  to  Westerkerk    evening  scenario  favoring  efficiency  

  Baseline  comparisons:    

  Photo  Density  (PD)  route:  highest  density  of  photos  (over  5  year  period)  in  grid  cells  along  route    

  Google  Maps  (GM)  route:  shortest  route  between  two  loca3ons  

  Counterbalanced  within-­‐subject  design    

  Route  Varia3on  (IV):  Photographer  Paths  vs.  Photo  Density  vs.  Google  Maps  

25

Central Station to Museumplein (CM)!

26

Photographer Paths route (5.36 km)

Photo Density route (3.83 km)

Google Maps route (3.35 km)

Waterlooplein to Westerkerk (CM)!

27

Photographer Paths route (2.28 km)

Photo Density route (2.60 km)

Google Maps route (1.59 km)

Laboratory Study Design!

 Data  collected:  

1.  AerakDiff2  (Hassenzahl,  2003)  UX  ques3onnaire  responses  [7-­‐point  seman3c  differen3al  scale]:  Usability,  Hedonic  Quali3es  (Iden3ty,  S3mula3on),  Aerack3veness  

2.  Two-­‐part  semi-­‐structured  interviews  

 Part  1:  Route  preferences,  feedback  on  Photographer  Paths  

 Part  2:  Inves3ga3on  of  visualized  informa3on  types  (visualized  info  type  handouts):      a)  Google  maps      b)  Color  coded  PRSs  (PP  route)    

 c)  Density  geopoints  (PD  route)    d)  Thumbnail  photo  geopoints    

 e)  Foursquare  POIs  

28

29

a)

b)

c)

d)

e)

Web Survey Study!

  Short  web-­‐based  survey  for  CM  and  WW  routes  and  varia3ons  

  Basic  demographics  collected:  age,  gender,  years  in  Amsterdam  

  Sta3c  route  images,  no  counterbalancing  

  82  par3cipants  (55  m,  27  f)  aged  between  17-­‐62  (M=  27.6;  SD=  6.1)    

  Most  lived  in  Amsterdam  for  more  than  3  years  (44/82)    

  Some  between  1-­‐3  years  (15/82)  

  Less  than  a  year  (11/82)      Only  visited  before  (12/82)  

30

31

Results

AttrakDiff2 !

32

Central Station to Museumplein (CM) Route

!"#

!$#

!%#

&#

%#

$#

"#

'()*+),-#./)0123#4'.5# 6789:1-#./)0123#!#;87:,23#46.!;5#

6789:1-#./)0123#!#<,+/0),9:#46.!<5#

=>()-,?7:7@@#4=AA5#

!"#$%

&'("$)'*$%

'B929*()CB7(#')2B@# 'B929#D7:@123# E99*07#F)C@#** !"#"$%$&

******

******

**

* !"#"$%$'

AttrakDiff2 !

33

!"#

!$#

!%#

&#

%#

$#

"#

'()*+),-#./)0123#4'.5# 6789:1-#./)0123#!#;87:,23#46.!;5#

6789:1-#./)0123#!#<,+/0),9:#46.!<5#

=>()-,?7:7@@#4=AA5#

!"#$%

&'("$)'*$%

'B929*()CB7(#')2B@# 'B929#D7:@123# E99*07#F)C@#** !"#"$%$&

****

****

Waterlooplein to Westerkerk (WW) Route

Route Preference!

Lab  Study  

  CM  route:  most  chose  to  follow  the  PP  route  (9/15),  PD  route  (4/15),  GM  route  (2/15)  

  “One  of  the  routes  [PP]  was  long  and  took  many  detours,  and  I  thought  that  was  a  very  aFracHve  route!”  

  WW  route:  most  chose  to  follow  GM  route  (10/15),  PD  route  (4/15),  no  route  (1/15)    

  “You  are  going  for  coffee  so  you  just  want  to  get  there,  unlike  in  the  first  [CM]  scenario  where  it  is  a  nice  day  and  you  have  Hme.”  

34

Web  Survey  

•           CM  route:  GM  (40/82),  PD  (23/82),  PP  route  (10/82),  neither  (9/82)  •   No  experimenter  steering;  many  Amsterdam  residents  know  the  city  already  quite  well!  •   “I  would  not  easily  walk  these  routes...  who  in  Amsterdam  walks?  ;)”  

•           WW  route:  GM  (67/82),  PD  (6/82),  PP  route  (3/82),  neither  (6/82)  

Digital Information Aids!Lab  Study  

             Interview:  Part  1    “How  many  persons  (focus  on  city  photographers)  took  a  given  route  segment  over  a  certain  3me  

period  (e.g.,  1  year)?”  

  Useful  (8/15)  for  exploring  a  city  one  already  knows  

  Not  sure  (4/15)  

  Depends  on  which  photographers  (2/15)  

  Not  for  me  (1/15)  

 Interview:  Part  2    Found  PP  info  type  aerac3ve  (10/15),  but  combine  with  Photo  thumbnails  (3/10)  and  POIs  (3/10)  

35

Digital Information Aids!

Web  Survey  

  POIs  along  a  route  (51x)  

  Route  distance  (51x)  

  Comments  along  a  route  (ranked  by  highest  ra3ngs  or  recency)  (24x)  

  Expert  travel  guides  (22x)  

  Photos  of  route  segments  (17x)  

  No  digital  aids  (13x)  

  Number  of  photographers  that  took  a  given  path  over  a  Hme  period  (9x)  

  Number  of  photos  along  a  route  over  a  3me  period  (9x)  

36

37

Discussion

Discussion!

  Discrepancy  between  lab-­‐study  and  web  survey  

  Quick  web  survey  insufficient?      Visualiza3on/explana3on  of  digital  aids  important?  

  Proof-­‐of-­‐concept  approach  requires  real-­‐world  ‘outdoor’  evalua3on  

  Different  street  grid  network  

  Scalability  to  larger  ci3es  

  More  context-­‐awareness  

38

Take Home Message!

Going  towards  data-­‐driven  explora3on-­‐based  route  planners…  

  Some3mes  it’s  the  journey,  not  the  des3na3on  

  A  quan3ta3ve  approach  may  oversimplify  human  needs  for  explora3on  

  But  some3mes  we  want  an  automa3c  solu3on,  so  as  not  to  be  bothered  with  supplying  user  preferences  and  encounter  serendipity  

39

40

Questions

h6p://staff.science.uva.nl/~elali/  

References!  1.  Cheng,  A.-­‐J.,  Chen,  Y.-­‐Y.,  Huang,  Y.-­‐T.,  Hsu,  W.  H.,  and  Liao,  H.-­‐Y.  M.  Personalized  travel  recommenda3on  by  mining  people  aeributes  from  community-­‐contributed  photos.  In  Proc.  MM  ’11,  ACM  (2011),  83–92.    2.  M.  Clements,  P.  Serdyukov,  A.  P.  de  Vries,  and  M.  J.  Reinders.  Using  flickr  geotags  to  predict  user  travel  behaviour.  In  Proc.  SIGIR  ’10,  pages  851–852.  ACM  Press,  2010.  3.  M.  De  Choudhury,  M.  Feldman,  S.  Amer-­‐Yahia,  N.  Golbandi,  R.  Lempel,  and  C.  Yu.  Automa3c  construc3on  of  travel  i3neraries  using  social  breadcrumbs.  In  Proc.  HT  ’10,  pages  35–44.  ACM  Press,  2010.  4.  F.  Girardin,  F.  Calabrese,  F.  D.  Fiore,  C.  Ra|,  and  J.  Blat.  Digital  footprin3ng:  Uncovering  tourists  with  user-­‐generated  content.  IEEE  Pervasive  Compu3ng,  7:36–43,  October  2008.  5.  N.  Shoval  and  M.  Isaacson.  Sequence  alignment  as  a  method  for  human  ac3vity  analysis  in  space  and  3me.  Annals  of  the  Associa3on  of  American  Geographers,  97(2):282–297,  2007.  6.  A.  Vaccari,  F.  Calabrese,  B.  Liu,  and  C.  Ra|.  Towards  the  socioscope:  an  informa3on  system  for  the  study  of  social  dynamics  through  digital  traces.  In  Proc.  GIS  ’09,  pages  52–61.  ACM  Press,  2009.  7.  Hassenzahl,  M.,  Burmester,  M.,  and  Koller,  F.  AerakDiff:  Ein  Fragebogen  zur  Messung  wahrgenommener  hedonischer  und  pragma3scher  Qualit¨at.  Mensch  &  Computer  2003.  Interak3on  in  Bewegung  (2003),  187–196.  8.  Lu,  X.,  Wang,  C.,  Yang,  J.-­‐M.,  Pang,  Y.,  and  Zhang,  L.  Photo2trip:  genera3ng  travel  routes  from  geo-­‐tagged  photos  for  trip  planning.  In  MM  ’10,  ACM  (2010),  143–152.  9.  Wilson,  C.  Ac3vity  paeerns  in  space  and  3me:  calcula3ng  representa3ve  hagerstrand  trajectories.  TransportaHon  35  (2008),  485–499.  

41

Related Work!  Sequence  Alignment  (SA)  methods:  

  Borrowed  from  bioinforma3cs  and  later  3me  geography  

  Time  geography  systema3cally  analyzes  and  explores  the  sequen3al  dimension  of  human  spa3al  and  temporal  ac3vity  (Shoval  &  Isaacson,  2007).    

  Visualize  human  movement  on  2-­‐D  plane:  x-­‐  &  y-­‐  axis    longitude  and  la3tude;  z-­‐axis    3me  

   useful  for  analyzing  sequences  of  human  ac3vity  (in  this  case,  photo-­‐taking  behavior  of  photographers)  

  Photo-­‐based  City  Modeling:    Understand  tourist  site  aerac3veness  based  on  geotagged  photos  (Girardin  et  al.,  2008)    Construct  inter-­‐city  travel  i3neraries  (De  Choudhury  et  al.,  2010)    Generate  personalized  Point-­‐of-­‐Interest  (POI)  recommenda3ons  of  where  to  go  in  a  city  based  on  

the  user's  travel  history  in  other  ci3es  (Clements  et  al.,  2010)  

   Approaches  focus  on  describing  loca3ons,  not  on  fine-­‐grained  within-­‐city  routes  that  connect  them  

  Non-­‐efficiency  Driven  Route  Planners  

  Automa3c  genera3on  of  travel  plans  based  on  millions  of  photos  (Lu  et  al.,  2010)    Personalized  data-­‐driven  travel  route  recommenda3ons  (Cheng  et  al.  2011)  

   Systems  geared  towards  recommending  hotspots  and  popular  routes,  not  off-­‐beat  explora3on  routes  42

Sequence Alignment Overview!Input:  two  sequences  over  the  same  alphabet  Output:  an  alignment  of  the  two  sequences  

Example:     Source:  GCGCATGGATTGAGCGA  

   Target:    TGCGCCATTGATGACCA  

  A  possible  alignment:      -­‐GCGC-­‐ATGGATTGAGCGA      TGCGCCATTGAT-­‐GACC-­‐A  

Three  opera3ons  (each  with  cost):  

  Perfect  matches  (MATCH)  

  Mismatches  (DEL)  

  Inser3ons  &  dele3ons  (INDEL)  

  The  less  distance  cost,  the  higher  the  similarity  between  two  sequences  

43 (Shoval & Isaacson, 2007)

Multiple Sequence Alignment Overview!  Used  ClustalTXY  sonware  (Wilson  et  al.,  2008)  for  photo  alignment:  

  makes  full  use  of  mul3ple  pairwise  sequence  alignments,  where  alignments  are  computed  for  similarity  in  parallel  

  uses  a  progressive  heuris3c  to  apply  mul3ple  sequence  alignment  (MSA)  

  allows  elements  to  be  represented  with  up  to  12-­‐character  words,  which  allows  unique  representa3on  of  small  map  regions,  used  for  represen3ng  the  geotagged  photos  

  to  deal  with  differences  in  sequence  length,  ClustalTXY  adds  gap  openings  and  extensions  to  sequences.    

  MSA  in  3  stages:  

 1)  Pairwise  alignments  are  computed  for  all  sequences  

 2)  Aligned  sequences  are  grouped  together  in  a  dendogram  based  on  similarity  

 3)  Dendogram  used  as  a  guide  for  mul3ple  alignment  

44