expert estimation from multimodal features
DESCRIPTION
Grand Challenge on Multimodal Learning Analytics 2013TRANSCRIPT
![Page 1: Expert estimation from Multimodal Features](https://reader034.vdocuments.site/reader034/viewer/2022052505/5558b9f0d8b42aa52a8b4dd4/html5/thumbnails/1.jpg)
Expertise Estimation based on Simple Multimodal Features
Xavier Ochoa, Katherine Chiluiza, Gonzalo Méndez, Gonzalo Luzardo, Bruno Guamán and James Castells
Escuela Superior Politécnica del Litoral
![Page 2: Expert estimation from Multimodal Features](https://reader034.vdocuments.site/reader034/viewer/2022052505/5558b9f0d8b42aa52a8b4dd4/html5/thumbnails/2.jpg)
http://www.slideshare.net/xaoch
Download this presentation
![Page 3: Expert estimation from Multimodal Features](https://reader034.vdocuments.site/reader034/viewer/2022052505/5558b9f0d8b42aa52a8b4dd4/html5/thumbnails/3.jpg)
How to (easily) obtain multimodal features?
What is already there?
![Page 4: Expert estimation from Multimodal Features](https://reader034.vdocuments.site/reader034/viewer/2022052505/5558b9f0d8b42aa52a8b4dd4/html5/thumbnails/4.jpg)
Three Approaches
• Literature-based features
• Common-sense-based features
• “Why not?”-based features
![Page 5: Expert estimation from Multimodal Features](https://reader034.vdocuments.site/reader034/viewer/2022052505/5558b9f0d8b42aa52a8b4dd4/html5/thumbnails/5.jpg)
All approaches proved useful
Proof that we are in an early stage
![Page 6: Expert estimation from Multimodal Features](https://reader034.vdocuments.site/reader034/viewer/2022052505/5558b9f0d8b42aa52a8b4dd4/html5/thumbnails/6.jpg)
Video: Calculator Use (NTCU)
• Idea:– Calculator user is the one solving the problem
• Procedure:– Obtain a picture of the calculator– Track the position and angle of the image in the
video using SURF + FLANN + Rigid Object Transformation (OpenCV)
– Determine to which student the calculator is pointing in each frame
![Page 7: Expert estimation from Multimodal Features](https://reader034.vdocuments.site/reader034/viewer/2022052505/5558b9f0d8b42aa52a8b4dd4/html5/thumbnails/7.jpg)
Video: Calculator Use (NTCU)
![Page 8: Expert estimation from Multimodal Features](https://reader034.vdocuments.site/reader034/viewer/2022052505/5558b9f0d8b42aa52a8b4dd4/html5/thumbnails/8.jpg)
Video: Total Movement (TM)
• Idea:– Most active student is the leader/expert?
• Procedure:– Subtract current frame from previous frame– Codebook algorithm to eliminate noise-movement– Add the number of remaining pixels
![Page 9: Expert estimation from Multimodal Features](https://reader034.vdocuments.site/reader034/viewer/2022052505/5558b9f0d8b42aa52a8b4dd4/html5/thumbnails/9.jpg)
Video: Total Movement (TM)
![Page 10: Expert estimation from Multimodal Features](https://reader034.vdocuments.site/reader034/viewer/2022052505/5558b9f0d8b42aa52a8b4dd4/html5/thumbnails/10.jpg)
Video: Distance from center table (DHT)
• Idea:– If the head is near the table (over paper) the
student is working on the problem• Procedure:– Identify image of heads– Use TLD algorithm to track heads– Determine the distance from head to center of
table
![Page 11: Expert estimation from Multimodal Features](https://reader034.vdocuments.site/reader034/viewer/2022052505/5558b9f0d8b42aa52a8b4dd4/html5/thumbnails/11.jpg)
Video: Distance from center table (DHT)
![Page 12: Expert estimation from Multimodal Features](https://reader034.vdocuments.site/reader034/viewer/2022052505/5558b9f0d8b42aa52a8b4dd4/html5/thumbnails/12.jpg)
Audio: Processing
![Page 13: Expert estimation from Multimodal Features](https://reader034.vdocuments.site/reader034/viewer/2022052505/5558b9f0d8b42aa52a8b4dd4/html5/thumbnails/13.jpg)
Audio: Processing
![Page 14: Expert estimation from Multimodal Features](https://reader034.vdocuments.site/reader034/viewer/2022052505/5558b9f0d8b42aa52a8b4dd4/html5/thumbnails/14.jpg)
Audio: Features
• Number of Interventions (NOI)• Total Speech Duration (TSD)• Times Numbers were Mentioned (TNM)• Times Math Terms were Mentioned (TMTM)• Times Commands were Pronounced (TCP)
![Page 15: Expert estimation from Multimodal Features](https://reader034.vdocuments.site/reader034/viewer/2022052505/5558b9f0d8b42aa52a8b4dd4/html5/thumbnails/15.jpg)
Digital Pen: Basic Features
![Page 16: Expert estimation from Multimodal Features](https://reader034.vdocuments.site/reader034/viewer/2022052505/5558b9f0d8b42aa52a8b4dd4/html5/thumbnails/16.jpg)
Digital Pen: Basic Features
• Total Number of Strokes (TNS)• Average Number of Points (ANP)• Average Stroke Path Length (ASPL)• Average Stroke Displacement (ASD)• Average Stroke Pressure (ASP)
![Page 17: Expert estimation from Multimodal Features](https://reader034.vdocuments.site/reader034/viewer/2022052505/5558b9f0d8b42aa52a8b4dd4/html5/thumbnails/17.jpg)
Digital Pen: Shape Recognition
Stronium – Sketch Recognition Libraries
![Page 18: Expert estimation from Multimodal Features](https://reader034.vdocuments.site/reader034/viewer/2022052505/5558b9f0d8b42aa52a8b4dd4/html5/thumbnails/18.jpg)
Digital Pen: Shape Recognition
• Number of Lines (NOL)• Number of Rectangles (NOR)• Number of Circles (NOC)• Number of Ellipses (NOE)• Number of Arrows (NOA)• Number of Figures (NOF)
![Page 19: Expert estimation from Multimodal Features](https://reader034.vdocuments.site/reader034/viewer/2022052505/5558b9f0d8b42aa52a8b4dd4/html5/thumbnails/19.jpg)
Features Variation
• When the features were evaluated inside a group two variations were usually obtained:
– Percentage of the total (e.g. Calculator Use)
– Highest / Lowest (e.g. Faster Writer, Lowest Time)
![Page 20: Expert estimation from Multimodal Features](https://reader034.vdocuments.site/reader034/viewer/2022052505/5558b9f0d8b42aa52a8b4dd4/html5/thumbnails/20.jpg)
Next step: Find predictive features
![Page 21: Expert estimation from Multimodal Features](https://reader034.vdocuments.site/reader034/viewer/2022052505/5558b9f0d8b42aa52a8b4dd4/html5/thumbnails/21.jpg)
Prediction at two levels
Problem and Group
![Page 22: Expert estimation from Multimodal Features](https://reader034.vdocuments.site/reader034/viewer/2022052505/5558b9f0d8b42aa52a8b4dd4/html5/thumbnails/22.jpg)
Analysis at Problem levelSolving Problem Correctly
• All available problems were used• Logistic Regression to model Student Solving
Problem Correctly• Resulting model was significantly reliable• 60,9% of the problem solving student was
identified• 71,8% of incorrectly solved problems were
identified
![Page 23: Expert estimation from Multimodal Features](https://reader034.vdocuments.site/reader034/viewer/2022052505/5558b9f0d8b42aa52a8b4dd4/html5/thumbnails/23.jpg)
Analysis at problem level
![Page 24: Expert estimation from Multimodal Features](https://reader034.vdocuments.site/reader034/viewer/2022052505/5558b9f0d8b42aa52a8b4dd4/html5/thumbnails/24.jpg)
Analysis at Group LevelExpertise Estimation
• Data from group 2 was removed because there was no expert
• Features were feed to a Classification Tree algorithm
• Several variables had a high discrimination power between expert and non-experts
• Best discrimination (6.53) result in 80% expert prediction and 90% non-expert prediction
![Page 25: Expert estimation from Multimodal Features](https://reader034.vdocuments.site/reader034/viewer/2022052505/5558b9f0d8b42aa52a8b4dd4/html5/thumbnails/25.jpg)
Analysis at Group LevelExpertise Estimation
![Page 26: Expert estimation from Multimodal Features](https://reader034.vdocuments.site/reader034/viewer/2022052505/5558b9f0d8b42aa52a8b4dd4/html5/thumbnails/26.jpg)
Expert Estimation over Problems
Plateau reached after just 4 problems
![Page 27: Expert estimation from Multimodal Features](https://reader034.vdocuments.site/reader034/viewer/2022052505/5558b9f0d8b42aa52a8b4dd4/html5/thumbnails/27.jpg)
Main conclusion:
Simple features could identify expertise
![Page 28: Expert estimation from Multimodal Features](https://reader034.vdocuments.site/reader034/viewer/2022052505/5558b9f0d8b42aa52a8b4dd4/html5/thumbnails/28.jpg)
Conclusions
• Three strong features:– Faster Writer (Digital Pen)– Percentage of Calculator Use (Video)– Times Numbers were Mentioned (Audio)
• Each mode provide discrimination power to establish expertise
• These features maintain its discriminant power at lower levels (solving a problem correctly)
![Page 29: Expert estimation from Multimodal Features](https://reader034.vdocuments.site/reader034/viewer/2022052505/5558b9f0d8b42aa52a8b4dd4/html5/thumbnails/29.jpg)
Gracias / Thank you
Questions?
Xavier [email protected]://ariadne.cti.espol.edu.ec/xavierTwitter: @xaoch