![Page 1: Real-time interactions between attention and behavior in multimedia learning environments Susan Letourneau Postdoctoral Fellow, CREATE Lab NYU & CUNY Graduate](https://reader035.vdocuments.site/reader035/viewer/2022062321/56649e625503460f94b5da11/html5/thumbnails/1.jpg)
Real-time interactions between attention and behavior in multimedia learning environments
Susan LetourneauPostdoctoral Fellow, CREATE Lab
NYU & CUNY Graduate Center
LearnLab Summer WorkshopAugust 4, 2012
![Page 2: Real-time interactions between attention and behavior in multimedia learning environments Susan Letourneau Postdoctoral Fellow, CREATE Lab NYU & CUNY Graduate](https://reader035.vdocuments.site/reader035/viewer/2022062321/56649e625503460f94b5da11/html5/thumbnails/2.jpg)
How can multimedia technology be made more effective for learning?
CREATE Lab research includes:
• Systematic investigation of design principles that may support learning
• Iterative development of educational games and simulations
![Page 3: Real-time interactions between attention and behavior in multimedia learning environments Susan Letourneau Postdoctoral Fellow, CREATE Lab NYU & CUNY Graduate](https://reader035.vdocuments.site/reader035/viewer/2022062321/56649e625503460f94b5da11/html5/thumbnails/3.jpg)
Interactivity and Engagement
• Students interact and “engage” with multimedia materials in different ways:– By acting and doing– By looking and thinking– By reacting and feeling
• How can we capture attention, cognition, emotion, in addition to behavioral activity?
• Multiple measures:– Activity logs– Eye-tracking– Physiological responses
![Page 4: Real-time interactions between attention and behavior in multimedia learning environments Susan Letourneau Postdoctoral Fellow, CREATE Lab NYU & CUNY Graduate](https://reader035.vdocuments.site/reader035/viewer/2022062321/56649e625503460f94b5da11/html5/thumbnails/4.jpg)
Eye-tracking measures of visual attention
• Benefits– Remote, noninvasive– Objective– Continuous recording
• Measures include:– Location of gaze– Duration of fixations– Fixation Sequences
![Page 5: Real-time interactions between attention and behavior in multimedia learning environments Susan Letourneau Postdoctoral Fellow, CREATE Lab NYU & CUNY Graduate](https://reader035.vdocuments.site/reader035/viewer/2022062321/56649e625503460f94b5da11/html5/thumbnails/5.jpg)
Integrating Activity Logs & Eye-tracking
• Synchronized recordings of behavior and attention using common timestamp
• Data analysis approaches:– Behaviors as individual events– Behaviors as markers or dividers to parse eye-
tracking data– Sequences of gaze and behavior over time
![Page 6: Real-time interactions between attention and behavior in multimedia learning environments Susan Letourneau Postdoctoral Fellow, CREATE Lab NYU & CUNY Graduate](https://reader035.vdocuments.site/reader035/viewer/2022062321/56649e625503460f94b5da11/html5/thumbnails/6.jpg)
Study 1: Gaze and Activity in a Chemistry Simulation
• 26 high school students• Measures:
– Eye-tracking, activity logs– Pre/post-tests of
chemistry knowledge
![Page 7: Real-time interactions between attention and behavior in multimedia learning environments Susan Letourneau Postdoctoral Fellow, CREATE Lab NYU & CUNY Graduate](https://reader035.vdocuments.site/reader035/viewer/2022062321/56649e625503460f94b5da11/html5/thumbnails/7.jpg)
• Gaze transitions between multiple representations correlated with learning outcomes
– Controllers-Axes: =.54, t(20)=2.88, p=.01, Container-Graph: =.46, t(20)=2.38, p=.02
• Students often looked to these key areas immediately after changing a variable in the simulation
![Page 8: Real-time interactions between attention and behavior in multimedia learning environments Susan Letourneau Postdoctoral Fellow, CREATE Lab NYU & CUNY Graduate](https://reader035.vdocuments.site/reader035/viewer/2022062321/56649e625503460f94b5da11/html5/thumbnails/8.jpg)
Study 2: Using visual scaffolds to guide attention
• 28 high school students, using simulation with or without scaffolds
• Examined gaze patterns following interactions with the controllers
![Page 9: Real-time interactions between attention and behavior in multimedia learning environments Susan Letourneau Postdoctoral Fellow, CREATE Lab NYU & CUNY Graduate](https://reader035.vdocuments.site/reader035/viewer/2022062321/56649e625503460f94b5da11/html5/thumbnails/9.jpg)
• Scanpaths follow the path of the scaffolds. • Students with more transitions show higher learning outcomes
[Controllers-Axes, r=.56, p<.01]
![Page 10: Real-time interactions between attention and behavior in multimedia learning environments Susan Letourneau Postdoctoral Fellow, CREATE Lab NYU & CUNY Graduate](https://reader035.vdocuments.site/reader035/viewer/2022062321/56649e625503460f94b5da11/html5/thumbnails/10.jpg)
Study 3: Attention during experimentation.
• 32 high school students planned and executed experiments in a chemistry simulation
• Activity logs used to divide eye-tracking data into three types of activities:
o Adjusting variables (planning experiment)o Watching ongoing experimento Experiment completed
![Page 11: Real-time interactions between attention and behavior in multimedia learning environments Susan Letourneau Postdoctoral Fellow, CREATE Lab NYU & CUNY Graduate](https://reader035.vdocuments.site/reader035/viewer/2022062321/56649e625503460f94b5da11/html5/thumbnails/11.jpg)
Students directed attention to different parts of the simulation during different activities.
Attention to the graph area specifically while students planned an experiment was correlated with post-test scores [=0.49, t(22)=2.51, p=.02].
![Page 12: Real-time interactions between attention and behavior in multimedia learning environments Susan Letourneau Postdoctoral Fellow, CREATE Lab NYU & CUNY Graduate](https://reader035.vdocuments.site/reader035/viewer/2022062321/56649e625503460f94b5da11/html5/thumbnails/12.jpg)
Planning Watching End of Experiment
![Page 13: Real-time interactions between attention and behavior in multimedia learning environments Susan Letourneau Postdoctoral Fellow, CREATE Lab NYU & CUNY Graduate](https://reader035.vdocuments.site/reader035/viewer/2022062321/56649e625503460f94b5da11/html5/thumbnails/13.jpg)
Ongoing work: Physiological measures of cognitive and affective responses
• Cognition: – Eye-tracking– EEG
• Emotion: – Skin conductance– Heart rate
![Page 14: Real-time interactions between attention and behavior in multimedia learning environments Susan Letourneau Postdoctoral Fellow, CREATE Lab NYU & CUNY Graduate](https://reader035.vdocuments.site/reader035/viewer/2022062321/56649e625503460f94b5da11/html5/thumbnails/14.jpg)
Triangulating multiple measures
Physiological measurements can be synchronized with eye-tracking and behavioral recordings.
Measurements can be time-locked with any channel of information.
![Page 15: Real-time interactions between attention and behavior in multimedia learning environments Susan Letourneau Postdoctoral Fellow, CREATE Lab NYU & CUNY Graduate](https://reader035.vdocuments.site/reader035/viewer/2022062321/56649e625503460f94b5da11/html5/thumbnails/15.jpg)
Current Research Directions
• Controlled comparisons of responses to tasks
Behaviorally Engaging Cognitively EngagingAffectively Engaging
![Page 16: Real-time interactions between attention and behavior in multimedia learning environments Susan Letourneau Postdoctoral Fellow, CREATE Lab NYU & CUNY Graduate](https://reader035.vdocuments.site/reader035/viewer/2022062321/56649e625503460f94b5da11/html5/thumbnails/16.jpg)
Acknowledgments
• CREATE Lab
• PIs: Jan Plass, Bruce Homer, Catherine Milne
• Lizzie Hayward, Ruth Schwartz
• Institute of Education Sciences, IPORT Fellowship