1

ImaginariumMerging science and practice

David Kirsh, UCSD Cognitive Science, Assoc. Director ACCHI

Erik Viirre M.D. Ph.D., UCSD School of Medicine

Sheldon Brown, UCSD Visual Arts, Director of ACCHI

Plus collaborators…

Objectives1. Observation: Study the nature of distributed creativity in

the natural setting of:

1. Brain storming

2. Design

3. Creative Expression

2. Theory: Build a more comprehensive theory of activity and sensory involvement during brainstorming

3. Design: Explore impact of changes in environment design on imagination, creativity and problem solving

Imaginarium Technology• Binocular eye tracking among groups• Video cameras coordinated with eye tracking• Kinects for gesture detection• Portable ERP caps• Precise sound localization• Whiteboard and other surface capture• Automated Annotation of design session• Remote interaction with other sites

Analytic Techniques

• Ethnographic coding, activity analysis,

• Datamining for patterns among different capture records

• Machine learning using VATIC and crowd sourcing to classify events defined via activity analysis

• Analyses using theories of distributed, situated and embodied cognition

Collaborators Relevant Research Basis• Kirsh: NSF on Distributed Creative Cognition

– In choreography – being expanded to music

• Makeig, Saygin: ERP (Event Related Potentials) during activity

• Trivedi: Video and Auditory awareness systems.• Viirre: Eye Tracking for wayfinding, medical diagnosis

and therapy• Brown: User analytics of virtual worlds

Observation Set-Up for Distributed Creative Cognition in Choreography - Kirsh

Mandeville Auditorium

Cal-IT2

Oculomotor Measuring Equipment

Video Engineering

• Wide Angle Camera to identify head• High resolution camera slaved to get

face/eye activity• Analysis of video image (visible/IR)

Body and Gaze Tracking

• Tool for Collaboration• Centered around whiteboard, display

table, workbench– Annotation– Command interaction (Kinect)

Medical Evaluation• Monitoring of Movement control

– Fall Risk– Movement control– ADL

• Evaluation of Disease progression– Vestibular– Parkinsons

• Diagnosis?– Cerebrovascular vs. Neurodegenerative

Research Questions

• Are binocular indices sensitive indicators of neural disruption

• What do “natural” eye movements look like?– Combined Eye-Head– Head Free

• Scan paths

Embedded Cognition Theory

• Interaction with physical objects– Mental planning, practice

• Learning theory of tasks

Principals

• Erik Viirre, Medicine• David Kirsch, Cog Sci• Mohan Trivedi, Engineering• Sheldon Brown, Visual Arts

Funding Sources

• Industry Partners• Medical Research

– DOD/VA– NIH

• NSF Computing, sensors, creativity, social sciences