Suppor&ng  Emergence:    Interac&on  Design  for  Visual  Analy&cs  

Approach  to  ESDA  William  Wong  

Head,  Interac&on  Design  Center  Middlesex  University  

London,  UK  15  September  2011    

1  

NSF  Workshop  on    

From  OpenSHAPA  to    Open  Data  Sharing  Arlington,  VA,  15-­‐16  Sep  2011  

2  

What  we  do  in  ESDA  •  Tool  usage  in  observa&on,  data  analysis  and  interpreta&on  

–  The  resolu,on  (wing  touch),  tool  differences  and  hence  what  can  be  done,  in  different  contexts  eg  development,  learning,  teaching  etc  

•  Sharing  of  collected  data  –  Why  would  I  want  to  share  –  If  I  could  share,  what  problems  and  hinderances  

•  Very  insighMul  of  the  specific  challenges  and  nuances  of  use  in  each  domain  of  use  

•  What  can  we  learn  from  a  different  form  of  “ESDA”  for  a  future  OpenSHAPA  /  OpenSHARE?  –  From  security  and  library  domains  –  Data  sharing  –  ‘common  source’  but  used  by  different  analysts  –  While  analysis  is  crucial,  sense-­‐making  to  draw  conclusions  based  on  assembled  

evidence  for  making  decisions  is  paramount  –  Use  Interac,ve  Visualisa,on  to  couple  intelligent  analysis  (e.g.  automa,c  en,ty  

extrac,on,  automa,c  thema,c  analysis)  with  emergence  driven  user  interface  design  

3  

Learning  from  a  Security  and  Library  Perspec&ve  

•  Making  Sense  –  EPSRC,  9  UK  Universi,es;  Imperial  College  PI,  MU  Deputy  PI  –   Mul,-­‐disciplinary  approach,  as  the  problem  cannot  be  addressed  by  a  single  

discipline  e.g.  image  analysis,  corpus  linguis,cs  and  automated  en,ty  extrac,on,  soVware  forensics,  systems  engineering,  representa,on  design,  psychology,  law    

•  UKVAC  Phase  2  –  US  DHS  and  UK  HM  Government,  5  UK  Universi,es,  Coordinator  MU  –  Mul,-­‐disciplinary  approach  to  Nobel  Laureate  and  FAA  Flight  Data  Problem  

•  INVISQUE  –  JISC  Rapid  Innova,on  Programme,  MU  PI  –  Conceived  as  next  genera,on  alterna,ve  to  difficult-­‐to-­‐use  library  e-­‐resources  =>  

tangible  reasoning  workspace  –  Taylor  and  Francis    

•  Visual  Analy&cs  

4  BLWWong©2011  

What  is  Visual  Analy&cs?  •  Visual  analy&cs  is  the  science  of  analy&cal  reasoning  facilitated  by  

interac&ve  visual  interfaces  (Thomas  and  Cook,  2005).  –  Integra,ng  tools  for  interac,ng    with  the  abstract  human  thinking  and  

reasoning  processes  –  Manipula,on  helps  in  reasoning  by  enabling  the  user  to  re-­‐arrange  the  

problem  space  (Maglio  et  al,  1999)  •  Data  graphics  or  info  vis  are  sta&c  •  VA  combines  interac5ve  visualiza5ons  based  on  analy5c  tools  to  

enable  rapid  querying  and  interroga&on  of  informa&on  …  –  Visual  form  includes  charts,  network  graphs,  rela,onships  over  ,me  and/

or  (geographical)  space  –  enables  explora,on  through  rapid  and  repeated  querying  –  access  to  original  data,  –  analysis  of  data    –  genera,on  of  hypotheses    –  construc,on  of  conclusion  pathways  

•  …  for  the  purpose  of  sense-­‐making  –  The  ability  to  rapidly  (and  visually)  process  and  assemble  evidence  to  

enable  genera,on  of  explana,ons  or  conclusions,  enabling  decisions   5  

Frame  of  Reference  

The  Visual  Analy&cs  Problem:  Emergence,  Search  and  Explana&on  

Lack  of  the  ‘big  picture’  

Jig-­‐saw  puzzle  (not  one,  but  many)  

Large  data  sets:  mul,-­‐sourced,  mixed-­‐format,  silo-­‐based,  sta,c/stream,  out  of  sequence,  uncertain  and  varying  quality  

Keyhole  problem  

Visually  supported  analy,c  reasoning  Varied  media,  varied  analysis  and  presenta,on  tools  

BLWWong©2011  

20  Representa&on  and  Analy&c  Problems  1.   The  problem  of  seeing  a  large  data  set  and  reasoning  space  through  a  small  keyhole.  2.   The  problem  of  handling  missing  data.  3.   The  problem  of  handling  decep&ve  /  misleading  data.  4.   The  problem  of  handling  contradictory  data.  5.   The  problem  of  aggrega&ng  and  reconciling  mul&ple  points  of  view  or  predic&ons.  6.   The  problem  of  evidence  colla&on  and  eviden&al  reasoning.  7.   The  problem  of  provenance  and  tracing  analy&c  reasoning.  8.   The  problem  of  integra&ng  data  space,  analy&c  space  and  hypothesis  spaces.  9.   The  problem  of  handling  strength  of  evidence  (including  subjec&ve  and  objec&ve  measures  of  strength)  +  

contribu&on  of  different  pieces  of  evidence  to  a  conclusion.  10.   The  problem  of  handling  uncertainty  in  data  and  /  or  informa&on.  11.   The  problem  of  represen&ng  and  handling  evidence  over  &me  and  space.  12.   The  problem  of  annota&ng,  remembering,  re-­‐visi&ng,  and  sehng  aside.  13.   The  problem  of  developing  a  sense  of  what  is  in  the  data  –  exploring  what  is  there.  14.   The  problem  of  predic&ng  and  represen&ng  emergent  behaviour.  15.   The  problem  of  Iden&fying  and  represen&ng  trends.  16.   The  problem  of  recognising  and  represen&ng  anomalous  data.  17.   The  problem  of  finding  the  needle  in  the  haystack  (or  knowing  what  is  chaff  –  i.e.  info  of  no  or  low  value)  18.   The  problem  of  predic&ng  the  path  of  cascading  failures  or  effects.  19.   The  problem  or  represen&ng  the  sta&c  and  dynamic  rela&onship  between  the  data  /  informa&on.  20.   The  problem  of  scalability  and  reusability.      

7  BLWWong©2011  

Visual  Analy&cs  Concept  

BLWWong(c)2010   8  

Visualiza,on  of  Output  

Seman,c  Extrac,on  

Data  integra,on  &  transforma,on  

Sensors  /  Surveillance  /  Data  collec,on  “SoV”  Data   “Hard”  Data  

Filters  

Palerns  and  commonali,es  

Social  networks,  interac,ons,  ac,vi,es  

Interac,ve  Dynamic  querying  

Many  tools  

Architecture:  Many  Tools  

Indexing,  Structuring  and  Theorizing:  Visual  Analy&cs  and  OpenSHAPA    

10  

Indexing   Structuring   Theorizing  

Automated  en,ty  extrac,on  

Analy,cal  tools  for  topical,  geospa,al,  temporal,  network  analysis  

Data  Sets  -­‐  Structured  

and  unstructured  text  

-­‐  Video  -­‐  Speech  Not  just  reports  and  video,  but  also  social  media  

Schema,za,on  

Search  and  query  

Colla,on  

Thema,c  analysis  

Explana,ons  Hypothesis  tes,ng  Eviden,al  reasoning  Conclusion  pathways  

Provenance  –  data,  processes,  and  reasoning:  Traceability,  how  did  we  get  here?  

 Wong©2004   11  

Activities Cues Knowledge Difficulties

Decision Strategies

Representation Design Concepts

Narratives

Transcripts

Broad Themes Related excerpts from transcripts

Specific themes Excerpts relating to specific

concepts in a theme, e.g. types of activities, examples of cues

e.g. Goals

e.g. Assessment

e.g. Planning

e.g. Control e.g. Assessment of

Situation

e.g. Assessment of Resources

Identify, Index & Collate

Structuring & Data reduction

Interpret & Conceptualise

Emergent  Themes  Analysis  

INVISQUE  demo:  Interac&on  Design  for  Suppor&ng  Emergence  

•  INterac&ve  VIsual  Search  and  QUery  Environment  –  Visual  forms  alempt  to  create  palerns  that  reinforce  relaIonships  (CSE)  

–  Interac,on  designed  to  support  emergence  in  themaIc  analysis  

•  INVISQUE  JISC  Library  Version  –  Suppor,ng  sense-­‐making  –  Data-­‐Frame  Model  –  Using  the  basic  interac,ve  visualiza,on  techniques  developed  here  to  support  sense-­‐making  in  inves,ga,ve  domains  

12  

The  Interac&ve  Visualiza&on  Approach  

•  Informa&on  Design  Principles  –  Focus+Context  –  Proximity-­‐Compa,bility  Principle  –  Gestalt  Principles  of  Form  Percep,on  –  Principle  of  Visual  Affordances  –  Ecological  Interface  Design  –  Representa,on  of  Func,onal  Rela,onships  

 

13  

The  Interac&ve  Visualiza&on  Approach  

•  Principles  implemented  in  design  by  –  Anima,on,  transparency,  informa,on  layering,  spa,al  layout,  palern  crea,on  

–  Emphasizing  the  representa,on  of  rela,onships  within  the  data  

–  Discovery  of  expected  and  un-­‐an,cipated  rela,onships  –  Interac,on  techniques  enable  rapid  and  con,nuous  itera,ve  querying  and  searching  while  keeping  visible  the  context  of  search  

– Minimizing  WWILF-­‐ing,  or  the  ‘What  Was  I  Looking  For?’  problem  

14  

The  Data-­‐Frame  Model    Guides  Interac&on  Design    

Klein  et  al,  2007  

15  

Reasoning  workspace  framework:  Mapping  and  design  and  of  reasoning  work  to  the  “keyhole”  

 

16  

Analysis  Space  -­‐ Tools  and  algorithms  -­‐ Behaviours,  rela,onships  and  palerns  -­‐ what’s  going  on  in  there?  

Hypothesis  Space  -­‐ Collate,  assemble,  marshal  -­‐ Formula,on    -­‐ Tes,ng  and  simula,on  -­‐ arguments,  conclusions,  evidence  

Transla&on  into  Design  

Conclusion  Pathways  

BLWWong(c)2010  

Data  Space  -­‐ what’s  available?    -­‐ What’s  changed?  -­‐ Awareness:  what’s  in  there?  

Depic,on  of  “Data  terrain”  

“brushing”  

Depic,on  of  “reasoning  and  search  process”  

Conclusion:  Some  Ques&ons  

•  What  can  we  do  for  a  future  OpenSHAPA  and  OpenSHARE?  –  indexing,  structuring,  bearing  in  mind  future  will  have  lots  of  “smart”  

analysis  technologies  that  can  support  the  lower  levels  of  analysis,  par,cularly  indexing  

•  What  System  Architecture?  –  that  combines  data  from  different  sources,  and  allows  a  variety  of  

tools  to  analyse  and  make  sense  of  data  •  Alterna&ve  designs  for  structuring  and  theorizing  that  more  

directly  support  sense-­‐making?  –  Adopt  /  adapt  an  interac,ve  visualisa,on  interface  design  –  Focus  on  emergence,  search  and  sense-­‐making  

•  Emergence  techniques  such  as  “Temporal  narra,ves”  •  Mul,ple  threads  /  parallel  lines  of  enquiry  and  finding  intersec,ng  storylines  

–  Reasoning  workspace  for  assembling  our  thoughts  and  conclusions  •  Future  work:  Collabora&ve  Sense-­‐making  environments   17  

End  

18