2013 07 05 (uc3m) lasi emadrid jgzubia deusto learning analytics primeras experiencias weblab deusto
Post on 11-May-2015
395 Views
Preview:
DESCRIPTION
TRANSCRIPT
Learning analytics and remote experimentation
Universidad de Deusto (Bilbao, Spain)IQS ‐ Universidad Ramon Llull (Barcelona, Spain)
Aristos Campus Mundus
Presentation
• International Campus of Excellence: AristosCampus Mundus
• University of Deusto – DeustoTech : Pablo Orduña, Aitor Almeida, Mari Luz Güenaga, Susana Romero y Javier García Zubía (zubia@deusto.es)
• IQS – Ramón Llull University: Jordi Cuadros y Lucinio González
Agenda
• What is a remote laboratory?• Remote lab examples. Data mining.• Case of study: Analysis of opinion surveys of users of WebLab‐Deusto
• Case of study: Analysis of learning outcomeswith VISIR
• Conclusion and future work
What is a remote lab?
• It allows the user to experiment out of the lab, as being there.
• Internet are the hands and the eyes of the user: student, teacher, citizens.
• The experience can be worse than in the lab, but it can be similar, or even it can be better because the user controls more the experiment. But, what about the feeling and the inmersion?
What is a remote lab?
• Advantages: organizational, academical,….• It is cheap because it can be shared amongdifferent institutions.
• It is always available.• It is powerful because the teacher can add all thathe wants
• The teacher can track everything done by theuser.
• Disadvantages.
Examples of remote labs
• WebLab‐Deusto (Spain), VISIR (Sweden), iLAB(USA), labShare (Australia), etc.
• WebLab‐Deusto offers experiments with basicanalog circuits, basic digital circuits, microcontrollers, robots, incubator, Archimedes’ principle…..
• Weblab‐Deusto also offers the system toschools, etc.
Examples of remote labs
• WebLab‐Deusto Robot:• Move, upload, program…
Examples of remote labs
• VISIR allows to mount circuits and analyze them
Data mining in WebLab‐Deusto
Data mining in WebLab‐Deusto
• Tracking: who, when, where, how long, web browser, IP, what device, experiment, state of thequeue, Facebook…. And the most important
• What commands were executed and with whatresults
• The teacher can reproduce the session to see and analyze the profile of the user: knowledge and learning
• Since 2009 more than 55 000 sessions.
Data mining in WebLab‐Deusto
• Is it useful? Is it usual?
Analysis of the opinion survey
• At the end of the semester the studentsanswer to a survey with 18 questions.
• The first survey was designed on 2003 with 15 questions.
• In the year 2008‐2009 the current survey wasdesigned merging the previous survey withthe results of the eMERGE project (Bordeaux) and Ma&Nickerson.
Analysis of the opinion survey
Q1. WebLab helps me with the subject, the concepts, the exercises, projects, etcQ2. Using the WebLab, I fell that it is real and it is not a simulationQ3. It is a good idea to extend this WebLab to all the studentsQ4. I have enjoyed using the WebLabQ5. WebLab is easy to useQ6. The quality of the WebCam is goodQ7. The different I/O devices (switches, buttons, etc.) are easy to useQ8. I don’t have problems with the assigned timeQ9. The I/O devices implemented are well selectedQ10. Even being far from the WebLab, I have felt that I control itQ11. I would like to use the WebLab in others subjectsQ12. I am satisfied with the WebLabQ16. The user’s manuals are good and clear.Q17. I have been motivated by the WebLab to learn more about the subjectQ18. The WebLab is a high quality software (access, management, availability, etc.)
Analysis of the opinion survey
• Is it correct? • Is it useful and meaningful? • Does it contain information?• For what is it being used by WebLab‐Deusto?• Learning analytics
Analysis of the opinion survey
• X‐O:– Experience
• The students use the weblab in a regular way in twodifferent subjects since 2008 to 2012
– LP: Programmable logic (VHDL&CPLD) (123 surveys) Thirdyear.
– DE: Electronics design (VHDL&FPGA) (73 surveys) Fifth year.
– Opinion survey• 18 questions, Likert pseudo‐scale with 5 levels
Analysis of the opinion survey
• Fiability– Cronbach’s alfa
• Construct Validity– Is there an internal consistency?
• Item analysis– Homogeneity index
Analysis of the opinion survey
• FIABILITY• Cronbach’s alfa
• Result for the survey: 0.78• It is a medium to high value
Analysis of the opinion survey
• Construct validity• Designed substructure (by me)
– UTILITY: Q01, Q03, Q11, Q12, Q17– USABILITY: Q04, Q05, Q07, Q08, Q09, Q16, Q18– INMERSION: Q02, Q06, Q10– OTHERS: Q13, Q14, Q15
Analysis of the opinion survey
• FIABILITY– UTILITY α = 0,74– USABILITY α = 0,64– INMERSION α = 0,40– OTHERS α = 0,27
• Looking to the last two values it seems thatthese two factors do not exist
Analysis of the opinion surveyQ02 Q06 Q10 Q04 Q05 Q07 Q08 Q09 Q16 Q18 Q01 Q03 Q11 Q12 Q17 Q13 Q14 Q15
Q02 1,00Q06 0,20 1,00Q10 0,30 0,07 1,00Q04 0,25 ‐0,07 0,19 1,00Q05 0,24 0,11 0,29 0,19 1,00Q07 0,21 0,20 0,21 0,10 0,39 1,00Q08 0,15 0,09 0,19 0,21 0,08 0,14 1,00Q09 0,17 0,15 0,26 0,13 0,29 0,37 0,15 1,00Q16 0,32 0,21 0,25 0,15 0,34 0,15 0,09 0,16 1,00Q18 0,32 0,26 0,36 0,35 0,27 0,18 0,07 0,28 0,37 1,00Q01 0,43 0,21 0,34 0,37 0,41 0,19 0,11 0,24 0,35 0,58 1,00Q03 0,27 0,05 0,20 0,27 0,38 0,12 0,09 0,21 0,22 0,44 0,46 1,00Q11 0,04 ‐0,03 0,31 0,37 0,16 0,19 0,05 0,13 0,20 0,36 0,36 0,28 1,00Q12 0,25 0,17 0,46 0,34 0,39 0,37 0,15 0,33 0,29 0,59 0,61 0,44 0,43 1,00Q17 0,25 0,00 0,12 0,50 0,07 0,06 0,14 0,07 0,22 0,31 0,38 0,18 0,41 0,27 1,00Q13 0,05 ‐0,08 0,04 0,10 0,13 0,09 0,10 ‐0,01 ‐0,01 0,02 0,02 0,08 0,17 0,13 0,05 1,00Q14 0,08 0,09 0,12 ‐0,07 0,12 0,03 0,06 0,19 ‐0,02 0,13 0,04 0,13 ‐0,01 0,08 ‐0,12 0,35 1,00Q15 0,02 0,03 ‐0,01 ‐0,06 ‐0,07 ‐0,01 0,05 ‐0,09 ‐0,04 ‐0,14 ‐0,10 ‐0,10 ‐0,04 ‐0,18 0,01 ‐0,13 ‐0,09 1,00
Analysis of the opinion survey
• Items‐factor correlations
Q02 Q06 Q10 Q04 Q05 Q07 Q08 Q09 Q16r_IN 0,71 0,68 0,64 0,17 0,31 0,31 0,2 0,28 0,38r_US 0,42 0,23 0,44 0,56 0,63 0,57 0,47 0,58 0,56r_UT 0,33 0,1 0,4 0,53 0,37 0,26 0,15 0,26 0,35r_ZZ 0,08 0,02 0,09 0 0,13 0,07 0,11 0,08 ‐0
Q18 Q01 Q03 Q11 Q12 Q17 Q13 Q14 Q15r_IN 0,46 0,47 0,25 0,15 0,42 0,17 ‐0 0,14 0,02r_US 0,61 0,56 0,43 0,36 0,61 0,35 0,11 0,1 ‐0,1r_UT 0,62 0,76 0,62 0,74 0,75 0,68 0,13 0,02 ‐0,1r_ZZ 0,05 0 0,09 0,08 0,07 ‐0 0,78 0,79 0,16
Analysis of the opinion surveys
• Factorial analysis. Three factorsFour factorsFive factors
Factor 1 Factor 2 Factor 3 FactorQ01 0,50 0,27 0,58 4Q02 0,17 0,10 0,65 3Q03 0,40 0,29 0,34 4Q04 0,75 0,09 0,07 1Q05 0,08 0,61 0,31 2Q06 -0,28 0,11 0,65Q07 -0,05 0,75 0,08 2Q08 0,15 0,26 0,03Q09 0,01 0,70 0,09 2Q10 0,27 0,50 0,21 2Q11 0,69 0,28 -0,05 1Q12 0,46 0,57 0,32 4Q16 0,15 0,13 0,64 3Q17 0,74 -0,09 0,17 1Q18 0,46 0,27 0,55 4
Analysis of the opinion surveys
• Identified factors– Q04, Q11, Q17 ENJOYMENT– Q02, Q16 INMERSION?– Q01, Q03, Q12, Q18 CLIENT SATISFACTION– Q05, Q07, Q09, Q10 USABILITY
• Excluded questions– Q06, Q08, Q13, Q14 and Q15
Analysis of the opinion surveys
• Fiability of the survey excluding Q06, Q08, Q13, Q14 y Q15: 0,83
• Fiability of the new identified model– ENJOYMENT α = 0,69– ¿INMERSION? α = 0,49– CLIENT SATISFACTION α = 0,81– USABILITY α = 0,71
Analysis of the opinion surveysQ04 Q11 Q17 Q02 Q16 Q01 Q03 Q12 Q18 Q05 Q07 Q09 Q10 Q06 Q08 Q13 Q14 Q15
Q04 1,00Q11 0,37 1,00Q17 0,50 0,41 1,00Q02 0,25 0,04 0,25 1,00Q16 0,15 0,20 0,22 0,32 1,00Q01 0,37 0,36 0,38 0,43 0,35 1,00Q03 0,27 0,28 0,18 0,27 0,22 0,46 1,00Q12 0,34 0,43 0,27 0,25 0,29 0,61 0,44 1,00Q18 0,35 0,36 0,31 0,32 0,37 0,58 0,44 0,59 1,00Q05 0,19 0,16 0,07 0,24 0,34 0,41 0,38 0,39 0,27 1,00Q07 0,10 0,19 0,06 0,21 0,15 0,19 0,12 0,37 0,18 0,39 1,00Q09 0,13 0,13 0,07 0,17 0,16 0,24 0,21 0,33 0,28 0,29 0,37 1,00Q10 0,19 0,31 0,12 0,30 0,25 0,34 0,20 0,46 0,36 0,29 0,21 0,26 1,00Q06 ‐0,07 ‐0,03 0,00 0,20 0,21 0,21 0,05 0,17 0,26 0,11 0,20 0,15 0,07 1,00Q08 0,21 0,05 0,14 0,15 0,09 0,11 0,09 0,15 0,07 0,08 0,14 0,15 0,19 0,09 1,00Q13 0,10 0,17 0,05 0,05 ‐0,01 0,02 0,08 0,13 0,02 0,13 0,09 ‐0,01 0,04 ‐0,08 0,10 1,00Q14 ‐0,07 ‐0,01 ‐0,12 0,08 ‐0,02 0,04 0,13 0,08 0,13 0,12 0,03 0,19 0,12 0,09 0,06 0,35 1,00Q15 ‐0,06 ‐0,04 0,01 0,02 ‐0,04 ‐0,10 ‐0,10 ‐0,18 ‐0,14 ‐0,07 ‐0,01 ‐0,09 ‐0,01 0,03 0,05 ‐0,13 ‐0,09 1,00
Analysis of the opinion surveys
• items‐factor CORRELATIONS
Q04 Q11 Q17 Q02 Q16 Q01 Q03 Q12 Q18r_DI 0,78 0,77 0,82 0,23 0,24 0,47 0,31 0,44 0,43r_IN 0,25 0,15 0,28 0,8 0,82 0,48 0,3 0,33 0,43r_SC 0,42 0,45 0,36 0,4 0,39 0,83 0,72 0,83 0,81r_US 0,22 0,29 0,12 0,33 0,33 0,43 0,33 0,56 0,4
Q05 Q07 Q09 Q10 Q06 Q08 Q13 Q14 Q15r_DI 0,18 0,15 0,14 0,27 ‐0 0,16 0,13 ‐0,1 ‐0r_IN 0,36 0,22 0,2 0,34 0,25 0,15 0,02 0,03 ‐0r_SC 0,45 0,27 0,33 0,43 0,22 0,13 0,08 0,12 ‐0,2r_US 0,72 0,71 0,7 0,64 0,19 0,2 0,09 0,16 ‐0,1
Analysis of the opinion surveys
• Homogeneity index
Q04 Q11 Q17 Q02 Q16 Q01 Q03 Q12 Q180,58 0,59 0,54 0,52 0,53 0,75 0,57 0,75 0,70
Q05 Q07 Q09 Q10 Q06 Q08 Q13 Q14 Q150,57 0,46 0,47 0,56 0,19 0,21 0,12 0,08 ‐0,1
Q1. WebLab helps me in the subject concept, exercises, projects, etc
Q12. I am satisfied with the WebLab
Q18. The WebLab is a high quality software (access, management, availability, etc.)
Analysis of the opinion surveys
• Enjoyment: Q04, Q11, Q17 (196 surveys). 75%
Analysis of the opinion surveys
• Inmersion?: Q02, Q16. 92%
Analysis of the opinion surveys
• Client satisfaction: Q01, Q03, Q12, Q18. 97%
Analysis of the opinion surveys
• Usability: Q05, Q07, Q09, Q10. 97%
Analysis of the opinion surveys
• Does the subject/matter influence the results?
Analysis of the opinion surveys
• Kolmogorov‐Smirnov test for the two subjects
Analysis of the opinion surveys
• Mann‐Whitney test
Analysis of the opinion surveys
• Does the year influence the results?
Analysis of the opinion surveys
• Kruskal‐Wallis test
Analysis of the opinion surveys
Learning evaluation
• O‐X‐O: Around 50 students of two different and similar groups …. Analog electronics. 2012‐2013– Pre‐test
• 10 questions, multiple choice with 4 answers
– Activity/Treatment: 2 weeks• 2 working sessions with VISIR in the classroom• 1 reviewing session
– Post‐test• 10 questions, multiple choice with 4 answers• The same questions but in different order
Learning evaluation
Learning evaluation
• Fiability– Cronbach’s alfa
• Construct validity– Content– Construct, internal consistency?
• Items analysis– Difficulty index– Homogeinity index
Learning evaluation
• Cronbach’s alfa
• Result for the survey: 0,52• It is a low value, but the results of the nexttest are good. It means that the survey it isnot very good, so it should be improved.
Learning evaluation: Questions 1‐5P01 P02 P03 P04 P05
a 70 32 23 31 32b 8 29 2 23 2c 22 37 73 22 51d 6 7 8 30 21
P01 P02 P03 P04 P05a 34 12 12 12 13b 6 12 0 6 1c 9 25 38 9 25d 5 5 4 27 15
P01 P02 P03 P04 P05a 36 20 11 19 19b 2 17 2 17 1c 13 12 35 13 26d 1 2 4 3 6
ALL THE TESTS
PRE‐TEST
POST‐TEST
Learning evaluation: Question 2
• In the circuit with two resistors of the samevalue, the voltage in each resistor is:
A: it is equal, B: it is the half of E in each resistorC: serial or paralell D: it is null
Learning evaluation: Question 5
• Which of the following circuits measures thevoltage in R1?
Learning evaluation: Questions 6‐10P06 P07 P08 P09 P10
a 46 9 35 18 10b 17 19 18 41 18c 34 57 3 27 10d 9 21 50 20 67
P06 P07 P08 P09 P10a 17 7 15 9 6b 10 8 7 15 10c 21 20 3 18 6d 6 19 29 12 31
P06 P07 P08 P09 P10a 29 2 20 9 4b 7 11 11 26 8c 13 37 0 9 4d 3 2 21 8 36
ALL THE TESTS
PRE‐TEST
POST‐TEST
Learning evaluation: Question 7
• The total value of the resistor is:A: close to 1 kohm B: close to 100 ohmC: close to 10 kohm D: power the circuit
Learning evaluation: Question 8
• Which of the following circuits measures thetotal resistance?
Learning evaluation
• Difficulty index
P01 P02 P03 P04 P050,61 0,28 0,64 0,21 0,28
P06 P07 P08 P09 P100,41 0,50 0,34 0,37 0,60
Learning evaluation
• Difficulty index. The test is too difficult.• We should redesign it.
Difficulty index Interpretation Recommended number of items
< 0,25 Very difficukt 10% (1) 1
[0,25; 0,45) Difficult 20% (2) 5
[0,45; 0,55) Normal 40% (4) 1
[0,55; 0,75] Easy 20% (2) 3
> 0,75 Very easy 10% (1) 0
Learning evaluation
• Homogeneity index
P01 P02 P03 P04 P050,43 0,38 0,24 0,50 0,38
P06 P07 P08 P09 P100,47 0,53 0,50 0,49 0,43
Learning evaluation
• Question 3: Which of the following circuitsmeasures the current in R1?
Learning evaluation
Learning evaluation
Learning evaluation
Learning evaluation
• Total punctuation
pre post
02
46
810
Learning evaluation
• The difference between groups is statistically significant:– Paired t‐test, p < 0,001
Paired t-testdata: dfParelles[, 3] and dfParelles[, 2] t = 4.9189, df = 51, p-value = 9.474e-06
– Wilcoxon test (rangos con signo), p < 0,001Wilcoxon signed rank test with continuity correctiondata: dfParelles[, 3] and dfParelles[, 2] V = 825, p-value = 1.886e-05
Conclusion
• The survey designed to know the opinion of the users is useful, but it is clearly and easilyimprovable.
• The use of VISIR has a positive effect in thestudents’ learning with basic electronics.
Future work
• Remote Labs + Learning Analytics• Improve and redesign the survey for the users.• Analyse the inmersion concept when usingremote labs.
• Extend the analysis of the VISIR effect in thestudents’ learning.
• Better explotation of the available data: userprofiles, use profiles, session profiles
• Massive use of remote labs in MOOCs.• Data mining.
Who is copying?
Future work
• New network at spanish level fostering collaborations between groups working on LA in Spain
• First meeting at some point in autumn to be established
• 100% Open (contact snola@deusto.es)• http://www.snola.deusto.es/
Learning analytics and remote experimentation
Universidad de Deusto (Bilbao, Spain)IQS ‐ Universidad Ramon Llull (Barcelona, Spain)
Aristos Campus Mundus
top related