uw-ace survey results: students’ experiences of using technology in learning vivian schoner and...
TRANSCRIPT
UW-ACE Survey UW-ACE Survey Results: Students’ Results: Students’
Experiences of Using Experiences of Using Technology in Learning Technology in Learning
Vivian Schoner and Katrina StrampelVivian Schoner and Katrina Strampel
Centre for Learning and Teaching through Technology (LT3)Centre for Learning and Teaching through Technology (LT3)University of WaterlooUniversity of Waterloo
The Setting: Teaching, Technology, The Setting: Teaching, Technology, and UW-ACE and UW-ACE
Student Encounters with the Blended Student Encounters with the Blended Learning ExperienceLearning Experience
Technically Viable? YesTechnically Viable? Yes Pedagogically Sound? Yes, No, Maybe, Pedagogically Sound? Yes, No, Maybe,
Depends… Depends… Survey Research ResultsSurvey Research Results
The UW-ACE SurveyThe UW-ACE Survey
We asked students to tell us about their We asked students to tell us about their experiences withexperiences with
Online Learning ActivitiesOnline Learning Activities Online Collaborative Learning (small group)Online Collaborative Learning (small group) Technology FunctionTechnology Function Student SatisfactionStudent Satisfaction Online Instructor FeedbackOnline Instructor Feedback
Students Completing SurveyStudents Completing Survey
FacultyFaculty Quest Enrolment Quest Enrolment % Survey % Survey
AHSAHS 15251525 66
ArtsArts 30413041 66
ENGENG 739 739 55
ENVENV 14141414 77
MATHMATH 31703170 44
SCIENCESCIENCE 94749474 44
Comparisons of ScalesComparisons of Scales
0
10
20
30
40
50
60
70
80
90
Cu
mu
lati
ve P
osi
tive
%
Technology Function
Student Satisfaction
Online InstructorFeedback
Online LearningActivities
Online CollaborativeLearning
Technology Function: Access; Technology Function: Access; Ease of Use; NavigationEase of Use; Navigation
26. logging in to be… 26. logging in to be… 27. accessing my courses to be…27. accessing my courses to be…28. navigating around the UW-ACE course pages to be…28. navigating around the UW-ACE course pages to be…29. accessing the course tools/announcements to be… 29. accessing the course tools/announcements to be… 30. posting, reading, responding to messages in the online 30. posting, reading, responding to messages in the online
discussion forum to be…discussion forum to be…31. accessing feedback on submissions to be…31. accessing feedback on submissions to be…32. accessing my grades to be…32. accessing my grades to be…33. submitting assignments to be…33. submitting assignments to be…34. accessing quizzes to be… 34. accessing quizzes to be… 35. accessing team or small group resources to be… 35. accessing team or small group resources to be…
Scale ResultsScale Results
Technology (alpha .953)Technology (alpha .953)Faculty % Very
Difficult% Difficult
%Undecided
%Easy
%Very Easy
Cum. Positive%
SCIENCE 2 4 8 32 53 85
FES 1 4 12 35 38 83
ARTS 3 5 9 32 51 83
AHS 1 5 12 26 56 82
MATH 6 6 14 34 48 74
ENG 6 11 16 32 35 67
Av for %Combined
Neg 9 UnD 12 Pos 79
Technology: Accessibility and Ease Technology: Accessibility and Ease of Useof Use
““quick access to grades and assignment feedback, access to quick access to grades and assignment feedback, access to past lecture notes, announcements, additional resources”past lecture notes, announcements, additional resources”
““UW-ACE helped a lot in centralizing my courses online”UW-ACE helped a lot in centralizing my courses online”
““the fact that you can connect anytime/anywhere”the fact that you can connect anytime/anywhere”
““a safe online environment”a safe online environment”
““easy to use, helpful, good interactions with profs and peers”easy to use, helpful, good interactions with profs and peers”
““I could sign in and have everything at my fingertips”I could sign in and have everything at my fingertips”
Appreciation for Technical Potential Appreciation for Technical Potential
““it enhances the learning process by offering supplemental it enhances the learning process by offering supplemental resources for course related topics”resources for course related topics”
““it is certainly appropriate to post lecture notes and other it is certainly appropriate to post lecture notes and other resources, but the absolute most important thing to remember resources, but the absolute most important thing to remember is not to invent uses just because the UW-ACE software is is not to invent uses just because the UW-ACE software is there. Legitimate uses are great, but online activities just for there. Legitimate uses are great, but online activities just for the sake of doing them are frustrating”the sake of doing them are frustrating”
““it is desirable because it gives students an alternative way to it is desirable because it gives students an alternative way to find information about the course, also allows them to find information about the course, also allows them to communicate with others who need help and who can offer communicate with others who need help and who can offer help”help”
““if you have a question, but hate raising your hand and asking if you have a question, but hate raising your hand and asking in front of all your peers, it can be asked and answered in front of all your peers, it can be asked and answered online” online”
““I believe it’s an integral part of the university to have students I believe it’s an integral part of the university to have students using technology which will be useful once we are out there in using technology which will be useful once we are out there in the workforce. Developing this skill is essential for us to be the workforce. Developing this skill is essential for us to be able to learn to be technologically diverse”able to learn to be technologically diverse”
Overall Student Satisfaction: Overall Student Satisfaction: Course Organization; Instructions; Course Organization; Instructions;
Workload; Technical SupportWorkload; Technical Support
36. the organization of the course content and 36. the organization of the course content and activities?activities?
37. the instructions for completing learning 37. the instructions for completing learning activities?activities?
38. the number of course learning activities 38. the number of course learning activities through the term?through the term?
39. the distribution of course learning activities 39. the distribution of course learning activities through the term?through the term?
40. the technical support?40. the technical support?41. the professional look of the course web 41. the professional look of the course web
page(s) page(s)
Scale ResultsScale Results
Overall Student Satisfaction (alpha .916)Overall Student Satisfaction (alpha .916)Faculty Very
UnsatisfiedUnsatisfied Undecided Satisfied Very
SatisfiedCum Positive%
AHS 2 4 17 38 39 77
SCIENCE 3 4 16 42 35 77
ARTS 5 7 14 44 30 74
FES 3 6 17 4 30 74
ENG 3 8 22 35 32 67
MATH 5 9 24 427 15 62
Av for %Combined
Neg 10
UnD 18
Pos 72
Online Instructor Feedback: Online Instructor Feedback: Helpful; Informative; Appropriate; Helpful; Informative; Appropriate;
TimelyTimely
42. instructor feedback helped me learn and 42. instructor feedback helped me learn and understand difficult concepts understand difficult concepts
43. the frequency of instructor feedback met my 43. the frequency of instructor feedback met my information needsinformation needs
44. the quality of instructor feedback was helpful 44. the quality of instructor feedback was helpful and informativeand informative
45. the timeliness of instructor feedback was 45. the timeliness of instructor feedback was appropriate for my needs appropriate for my needs
46. the amount of instructor feedback was 46. the amount of instructor feedback was appropriate for my needsappropriate for my needs
Scale ResultsScale Results
Online Instructor Feedback (alpha .972)Online Instructor Feedback (alpha .972)
Faculty Strongly Disagree
Disagree Undecided Agree Strongly Agree
Cum. Positive %
ARTS 5 6 17 39 33 72
AHS 4 7 22 36 31 67
SCIENCE 5 10 21 26 38 64
FES 13 8 24 31 24 55
ENG 2 18 26 41 13 54
MATH 23 9 25 26 17 43
Av for %Combined
Neg 18 UnD 23 Pos 59
Online Instructor FeedbackOnline Instructor Feedback ““I got assignments back sooner and easier with the site”I got assignments back sooner and easier with the site”
““the professor always kept us informed of news by posting the professor always kept us informed of news by posting announcements frequently on the UW-ACE website”announcements frequently on the UW-ACE website”
““the immediate feedback from quizzes written as well as the the immediate feedback from quizzes written as well as the correct answers is a definite asset”correct answers is a definite asset”
““took too long for marks to be posted”took too long for marks to be posted”
““not being able to see answers on actual quizzes”not being able to see answers on actual quizzes”
““grades are not properly recorded”grades are not properly recorded”
““there was an announcement that marks for an important there was an announcement that marks for an important section would be posted which they never were”section would be posted which they never were”
Online Learning Activities: Learning Online Learning Activities: Learning Support Features, Concepts, Skills, Support Features, Concepts, Skills,
Applications; Integrated Parts; Applications; Integrated Parts; Study Time; Performance LevelStudy Time; Performance Level
1. Discussion forum activities1. Discussion forum activities2. Self-Assessment (un-graded)2. Self-Assessment (un-graded)3. Self-assessment (graded)3. Self-assessment (graded)4. Collaborative/Peer Review submissions4. Collaborative/Peer Review submissions5. Individual Submissions5. Individual Submissions6. Lecture notes6. Lecture notes7. Links to external resources7. Links to external resources8. Information literacy/library modules8. Information literacy/library modules9. Lab modules/activities9. Lab modules/activities10. helped me learn the new skills and concepts for this course10. helped me learn the new skills and concepts for this course11. helped in that it reduced the amount of time spent studying text or notes in order 11. helped in that it reduced the amount of time spent studying text or notes in order
to grasp the material to be learnedto grasp the material to be learned12. helped me to better analyze and apply key points in the material to be learned12. helped me to better analyze and apply key points in the material to be learned13. encouraged me to take responsibility for my own learning13. encouraged me to take responsibility for my own learning14. I found that working online improved my performance on graded course 14. I found that working online improved my performance on graded course
assessments assessments 15. I found that the online component(s) were well integrated with the other course 15. I found that the online component(s) were well integrated with the other course
components not offered online components not offered online
Scale ResultsScale Results
Online Learning Activities (alpha .965/.935/.919)Online Learning Activities (alpha .965/.935/.919)
Faculty Strongly Disagree
Disagree Undecided Agree Strongly Agree
Cum. % Positive
ENG 6 5 27 36 26 62
FES 6 11 21 35 27 62
AHS 7 13 22 30 28 58
SCIENCE 11 11 24 33 21 57
ARTS 11 12 20 33 24 57
MATH 12 14 25 34 15 49
Av for % Combined
Neg 20 UnD 23 Pos 57
Online Learning ActivitiesOnline Learning Activities ““I’m sure there are situations where it would be desirable to “use I’m sure there are situations where it would be desirable to “use
technology to support teaching and learning”, but that in itself is too technology to support teaching and learning”, but that in itself is too broad a goal. You should be using technology to solve a particular broad a goal. You should be using technology to solve a particular problem, or to compensate for a recognized weakness in existing problem, or to compensate for a recognized weakness in existing methods. In the case of my course, I can’t see how UW-ACE has methods. In the case of my course, I can’t see how UW-ACE has added anything to the course. I found the written quizzes far more added anything to the course. I found the written quizzes far more helpful than the online ones, and I prefer to read printed course helpful than the online ones, and I prefer to read printed course notes rather than getting course material from the web (you can’t notes rather than getting course material from the web (you can’t highlight/underline/scribble on a webpage)”highlight/underline/scribble on a webpage)”
““it helped me develop the habit of studying for the course regularly. it helped me develop the habit of studying for the course regularly. It allowed me to learn course material through practice questions. A It allowed me to learn course material through practice questions. A chance to do the same quizzes again and again which helped me chance to do the same quizzes again and again which helped me learn and understand course concepts better. Unlimited time to do learn and understand course concepts better. Unlimited time to do the quiz”the quiz”
““a good way to practice concepts learned in class. Particularly I a good way to practice concepts learned in class. Particularly I found the ‘fill in the blank’ type exercises to be very helpful”found the ‘fill in the blank’ type exercises to be very helpful”
““the information was very complete and organized. I feel that the the information was very complete and organized. I feel that the textbook for the course would not have been sufficient on its own to textbook for the course would not have been sufficient on its own to learn all the course material. The resources offered on UW-ACE learn all the course material. The resources offered on UW-ACE enhanced my ability to learn the material”enhanced my ability to learn the material”
Online Collaborative Learning: Online Collaborative Learning: Learning Activities; Peer Learning Activities; Peer
Interaction; Reflective Learning; Interaction; Reflective Learning; CommunityCommunity
16. Team or small group graded assignments16. Team or small group graded assignments17. Whole class discussion activities17. Whole class discussion activities18. Peer review submissions18. Peer review submissions19. Team or small group un-graded assignments19. Team or small group un-graded assignments20. Giving/receiving peer feedback 20. Giving/receiving peer feedback 21. I was able to develop a better understanding of the material 21. I was able to develop a better understanding of the material
because of my interaction with my peersbecause of my interaction with my peers22. I put more thought into my comments when interacting with 22. I put more thought into my comments when interacting with
others onlineothers online23. I felt part of a learning community when working online23. I felt part of a learning community when working online24. I found that working online encouraged me to take 24. I found that working online encouraged me to take
responsibility for my group contributionresponsibility for my group contribution25. I am more comfortable asking questions to members of my 25. I am more comfortable asking questions to members of my
group in an online settinggroup in an online setting
Scale ResultsScale Results
Online Collaborative Learning (alpha .961/.964/.895)Online Collaborative Learning (alpha .961/.964/.895)Faculty Strongly
DisagreeDisagree Undecided Agree Strongly
AgreeCum. % Positive
ARTS 10 14 22 33 21 54
AHS 10 17 20 34 19 53
SCIENCE 12 7 28 29 24 53
FES 8 16 27 43 17 49
ENG 19 13 18 35 14 49
MATH 14 12 35 35 4 39
Av for % Combined
Neg 25 UnD 25 Pos 50
Online Collaborative Learning: Online Collaborative Learning: BenefitsBenefits
““the group discussion page made it a lot easier to communicate the group discussion page made it a lot easier to communicate with my group members since we are all in different classes and with my group members since we are all in different classes and communication for projects is an issue”communication for projects is an issue”
““I don’t mind the discussions online, I’m more comfortable since it I don’t mind the discussions online, I’m more comfortable since it gives the sense of anonymity (privacy thing)”gives the sense of anonymity (privacy thing)”
““It depends on what type of activities: I think group tutorials are a It depends on what type of activities: I think group tutorials are a good way to learn, whereas group projects are hard to realize”good way to learn, whereas group projects are hard to realize”
““they allow me to communicate with others, evaluate their thoughts they allow me to communicate with others, evaluate their thoughts and compare them to my own”and compare them to my own”
““the addition would benefit learning because it would provide a the addition would benefit learning because it would provide a place for quick review and knowledge testing as well as the style place for quick review and knowledge testing as well as the style of the lecturers questions”of the lecturers questions”
Construct Validity: Pearson Construct Validity: Pearson Correlation MatrixCorrelation Matrix
Correlations (Unadjusted)
1 .573** .604** .572** .483** .581** .526**
. .000 .000 .000 .000 .000 .000
602 561 245 267 582 571 241
.573** 1 .493** .626** .371** .568** .598**
.000 . .000 .000 .000 .000 .000
561 726 251 287 710 693 266
.604** .493** 1 .623** .493** .477** .507**
.000 .000 . .000 .000 .000 .000
245 251 265 217 258 255 165
.572** .626** .623** 1 .460** .510** .556**
.000 .000 .000 . .000 .000 .000
267 287 217 296 294 289 180
.483** .371** .493** .460** 1 .642** .521**
.000 .000 .000 .000 . .000 .000
582 710 258 294 812 773 273
.581** .568** .477** .510** .642** 1 .661**
.000 .000 .000 .000 .000 . .000
571 693 255 289 773 778 274
.526** .598** .507** .556** .521** .661** 1
.000 .000 .000 .000 .000 .000 .
241 266 165 180 273 274 275
Pearson Correlation
Sig. (2-tailed)
N
Pearson Correlation
Sig. (2-tailed)
N
Pearson Correlation
Sig. (2-tailed)
N
Pearson Correlation
Sig. (2-tailed)
N
Pearson Correlation
Sig. (2-tailed)
N
Pearson Correlation
Sig. (2-tailed)
N
Pearson Correlation
Sig. (2-tailed)
N
S1OnlineEnvironment
S2OLEHelpedMe
S3CAExperience
S4CALearning
S5EaseOfUse
S6Satisfaction
S7FeedbackExperience
S1OnlineEnvironment
S2OLEHelped
MeS3
CAExperienceS4
CALearningS5EaseOf
UseS6
SatisfactionS7FeedbackExperience
Correlation is significant at the 0.01 level (2-tailed).**.
Identifying PatternsIdentifying PatternsScience:Science:
Technology Technology (85)(85) and Student Satisfaction and Student Satisfaction (77)(77) 81% 81%
Instructor Feedback Instructor Feedback (64)(64), Learning Activity , Learning Activity (57),(57), Collaborative Learning Collaborative Learning (53)(53) 57%57%
Applied Health Sciences:Applied Health Sciences:
Technology Technology (82)(82) and Student Satisfaction and Student Satisfaction (77)(77) 80%80%
Instructor Feedback Instructor Feedback (67)(67), Learning Activity , Learning Activity (58),(58), Collaborative Learning Collaborative Learning (53)(53) 59%59%
Environmental Studies:Environmental Studies:
Technology Technology (83)(83) and Student Satisfaction and Student Satisfaction (74)(74) 79%79%
Learning Activity Learning Activity (62),(62), Instructor Feedback Instructor Feedback (55)(55), Collaborative Learning , Collaborative Learning (49)(49) 55%55%
Arts:Arts:
Instructor Feedback Instructor Feedback (72)(72), Technology , Technology (83),(83), Student Satisfaction Student Satisfaction (74)(74) 76%76%
Learning Activity Learning Activity (57)(57) and Collaborative Learning and Collaborative Learning (54)(54) 56%56%
Math:Math:
Technology Technology (74)(74) and Student Satisfaction and Student Satisfaction (62)(62) 68%68%
Learning Activity Learning Activity (49),(49), Instructor Feedback Instructor Feedback (43)(43), Collaborative Learning , Collaborative Learning (39)(39) 44%44%
Engineering:Engineering:
Technology Technology (67),(67), Student Satisfaction Student Satisfaction (67),(67), Learning Activity Learning Activity (62)(62) 65%65%
Instructor Feedback Instructor Feedback (54)(54) and Collaborative Learning and Collaborative Learning (49)(49) 52%52%
All Faculties, All ScalesAll Faculties, All Scales
General PatternsGeneral Patterns
Technology function and student satisfaction scales Technology function and student satisfaction scales represent the more straightforward scales group represent the more straightforward scales group addressing technical support and student workload in all addressing technical support and student workload in all FacultiesFaculties
Online Learning Activity, Online Collaborative Learning Online Learning Activity, Online Collaborative Learning (Small Group) and Instructor Feedback scales represent (Small Group) and Instructor Feedback scales represent the teaching and learning pedagogy group in four of the the teaching and learning pedagogy group in four of the six Facultiessix Faculties
Exceptions Exceptions Instructor Feedback, a pedagogy scale, grouped with technical Instructor Feedback, a pedagogy scale, grouped with technical
support and student workloadsupport and student workload Learning Activity, a pedagogy scale, grouped with technical Learning Activity, a pedagogy scale, grouped with technical
support and student workloadsupport and student workload
Faculty ScalesFaculty ScalesAHS: All Categories
010
203040
5060
StronglyDisagree
Disagree Undecided Agree StronglyAgree
Ratings
Stud
ent R
espo
nses
Online Learning Activity
Collaborative Learning
Technology Function
Student Satisfaction
Online Instructor Feedback
Arts All Categories
0102030405060
StronglyDisagree
Disagree Undecided Agree StronglyAgree
Ratings
Stud
ent R
espo
nses
Online Learning Activity
Collaborative Learning
Technology Function
Student Satisfaction
Online Instructor Feedback
Faculty ScalesFaculty ScalesEngineering All Categories
0
10
20
30
40
50
StronglyDisagree
Disagree Undecided Agree StronglyAgree
Ratings
Stud
ent R
espo
nses
Online Learning ActivityCollaborative LearningTechnology FunctionStudent SatisfactionOnline Instructor Feedback
FES: All Categories
0102030405060
StronglyDisagree
Disagree Undecided Agree StronglyAgree
Ratings
Stud
ent R
espo
nses Online Learning Activity
Collaborative LearningTechnology FunctionStudent SatisfactionOnline Instructor Feedback
Faculty ScalesFaculty ScalesMath: All Categories
0
10
20
30
40
50
StronglyDisagree
Disagree Undecided Agree StronglyAgree
Ratings
Stud
ent R
espo
nses
Online Learning ActivityCollaborative LearningTechnology FunctionStudent SatisfactionOnline Instructor Feedback
Science: All Categories
0102030405060
StronglyDisagree
Disagree Undecided Agree StronglyAgree
Ratings
Stu
dent
Res
pons
es Online Learning Activity
Collaborative Learning
Technology Function
Student Satisfaction
Online Instructor Feedback
Survey Results: 865 Students; 113 Survey Results: 865 Students; 113 Courses; 6 Faculties Courses; 6 Faculties
UW-ACE All Faculties All Scales
05
10152025303540
StronglyDisagree
Disagree Undecided Agree StronglyAgree
Ratings
Stu
dent
Res
pons
es b
y Fa
culty
AHS
ARTS
ENG
ES
MATH
SCI
Percent Student Response Ratings Percent Student Response Ratings by Faculty by Faculty
Faculty StronglyDisagree
Disagree Undecided Agree Strongly Agree
Total # Responses
Total #Course
Total #StudentsTaking Survey
AHS 5 9 18 33 35 1924 12 90
ARTS 7 9 16 36 32 4013 42 185
ENG 7 11 22 36 24 904 6 37
ES 6 9 29 36 29 1836 16 96
MATH 12 10 25 35 18 2628 9 120
SCI 7 7 20 32 34 6299 28 337
Av. % 7.33 9.16 20.17 34.67 28.67 17,604 113 865
% Rounded
7 9 20 35 29
Overview of Student CommentaryOverview of Student Commentary
Students are aware of the benefits of technology in the Students are aware of the benefits of technology in the classroom and commented frequently on instructors classroom and commented frequently on instructors under useunder use of this resource. of this resource.
They responded positively to technology use perceived They responded positively to technology use perceived to add real value to learning outcomes as evidenced in to add real value to learning outcomes as evidenced in improved performance on quizzes, tests, assignments or improved performance on quizzes, tests, assignments or projects.projects.
Students provided suggestions for improvements and/or Students provided suggestions for improvements and/or described new features they would like to see developed described new features they would like to see developed for UW-ACE. for UW-ACE.
They did not see value-added in working in small groups They did not see value-added in working in small groups or teams online when they could readily interact with or teams online when they could readily interact with their peers in person. their peers in person.
Recommendations for Online Recommendations for Online PedagogyPedagogy
We need to further explore ways to improve We need to further explore ways to improve instructional design and teaching techniques for instructional design and teaching techniques for online learning and collaborative learning online learning and collaborative learning activities. (LT3 Centre interface)activities. (LT3 Centre interface)
Some of the needed changes suggested by the Some of the needed changes suggested by the students should be considered for students should be considered for implementation either in the technology itself, implementation either in the technology itself, using the technology more fully to support using the technology more fully to support learning, and reducing the ambiguity learning, and reducing the ambiguity (undecided) noted in all the Scales except (undecided) noted in all the Scales except Technology Function.Technology Function.
Looking AheadLooking Ahead We need to think how to reduce the “undecided” We need to think how to reduce the “undecided”
rates, rates, Rethink providing this as a response choice Rethink providing this as a response choice Forced choice is one optionForced choice is one option Use “intermediate” instead of “undecided”Use “intermediate” instead of “undecided” Provide instructors with key information on the Provide instructors with key information on the
pedagogy of online teaching and learning pedagogy of online teaching and learning Encourage instructors to direct their students to Encourage instructors to direct their students to
the online survey available in students’ course the online survey available in students’ course pagespages
Analysis of the features of individual course Analysis of the features of individual course profiles should be undertaken (Liaisons)profiles should be undertaken (Liaisons)
THE Capstone StatementTHE Capstone Statement
Learning technologies should be used to Learning technologies should be used to support, enhance, supplement and/or support, enhance, supplement and/or expand learning opportunities; Technology expand learning opportunities; Technology should NEVER be used to replace should NEVER be used to replace teaching and learning in the classroom teaching and learning in the classroom setting. setting.