evaluating educational technology brian mcnurlen & chris migotsky university of illinois at...
TRANSCRIPT
![Page 1: Evaluating Educational Technology Brian McNurlen & Chris Migotsky University of Illinois at Urbana-Champaign](https://reader034.vdocuments.site/reader034/viewer/2022051621/5697bf811a28abf838c856b9/html5/thumbnails/1.jpg)
Evaluating Educational Technology
Brian McNurlen & Chris Migotsky
University of Illinois
at Urbana-Champaign
![Page 2: Evaluating Educational Technology Brian McNurlen & Chris Migotsky University of Illinois at Urbana-Champaign](https://reader034.vdocuments.site/reader034/viewer/2022051621/5697bf811a28abf838c856b9/html5/thumbnails/2.jpg)
Perspective & Context
“Like every shining new gift, sensory devices may dazzle one into all kinds of wishful thinking…Audio-visual materials must be understood in their relationship to teaching as a whole and to the learning process as a whole.”
Edgar Dale, 1946
![Page 3: Evaluating Educational Technology Brian McNurlen & Chris Migotsky University of Illinois at Urbana-Champaign](https://reader034.vdocuments.site/reader034/viewer/2022051621/5697bf811a28abf838c856b9/html5/thumbnails/3.jpg)
Hold Your Horses!
Student Preference for Use of IT in Courses
0
5
10
15
20
25
30
35
40
45
No IT Limited IT Moderate IT Extensive IT Exclusive IT
Preference
Res
po
nd
ent
Per
cen
tag
e
![Page 4: Evaluating Educational Technology Brian McNurlen & Chris Migotsky University of Illinois at Urbana-Champaign](https://reader034.vdocuments.site/reader034/viewer/2022051621/5697bf811a28abf838c856b9/html5/thumbnails/4.jpg)
Who Should Evaluate?
• Instructors who want to try something new with educational technology
• Instructors who want to decide if what they currently do is effective
• Classroom or online (or both)
• Others?
![Page 5: Evaluating Educational Technology Brian McNurlen & Chris Migotsky University of Illinois at Urbana-Champaign](https://reader034.vdocuments.site/reader034/viewer/2022051621/5697bf811a28abf838c856b9/html5/thumbnails/5.jpg)
Why Evaluate?
• Systematic documentation of your teaching
• What is working?
• How can it be improved?
• Other reasons?
![Page 6: Evaluating Educational Technology Brian McNurlen & Chris Migotsky University of Illinois at Urbana-Champaign](https://reader034.vdocuments.site/reader034/viewer/2022051621/5697bf811a28abf838c856b9/html5/thumbnails/6.jpg)
What is the Focus of the Evaluation?
• What is the product or process that needs to be resolved, improved?
• Is it something that can be addressed as an instructor, with limited resources?
• What counts as “success”?
![Page 7: Evaluating Educational Technology Brian McNurlen & Chris Migotsky University of Illinois at Urbana-Champaign](https://reader034.vdocuments.site/reader034/viewer/2022051621/5697bf811a28abf838c856b9/html5/thumbnails/7.jpg)
Activity #1
• Select a program, product, or service to be evaluated
• Define how the program/product/service fits into your course (goals/objectives)
• Discuss with partner
• Share with group
![Page 8: Evaluating Educational Technology Brian McNurlen & Chris Migotsky University of Illinois at Urbana-Champaign](https://reader034.vdocuments.site/reader034/viewer/2022051621/5697bf811a28abf838c856b9/html5/thumbnails/8.jpg)
Special Issues Associated with Educational Technology
• Continuity with course objectives
• Access issues
• Technical support
• Time
• Costs
• Ongoing technology development
![Page 9: Evaluating Educational Technology Brian McNurlen & Chris Migotsky University of Illinois at Urbana-Champaign](https://reader034.vdocuments.site/reader034/viewer/2022051621/5697bf811a28abf838c856b9/html5/thumbnails/9.jpg)
Defining Key Evaluation Terms
• Evaluand
• Formative vs. summative
• Evaluation vs. assessment
• Evaluation vs research
• Logic model
• Resources for further study– Evaluation Thesaurus (Scriven)– Web sites on last slides
![Page 10: Evaluating Educational Technology Brian McNurlen & Chris Migotsky University of Illinois at Urbana-Champaign](https://reader034.vdocuments.site/reader034/viewer/2022051621/5697bf811a28abf838c856b9/html5/thumbnails/10.jpg)
Evaluation Process
1. Define evaluand and connection to course goals
2. Methodology & data collection
3. Data analysis
4. Conclusion & implications
![Page 11: Evaluating Educational Technology Brian McNurlen & Chris Migotsky University of Illinois at Urbana-Champaign](https://reader034.vdocuments.site/reader034/viewer/2022051621/5697bf811a28abf838c856b9/html5/thumbnails/11.jpg)
1. Define Evaluand and its Fit with Course
• Boundaries
• What, how, & why…logic model
• Intended and unintended consequences
![Page 12: Evaluating Educational Technology Brian McNurlen & Chris Migotsky University of Illinois at Urbana-Champaign](https://reader034.vdocuments.site/reader034/viewer/2022051621/5697bf811a28abf838c856b9/html5/thumbnails/12.jpg)
2. Methodology & Data Collection
• Institutional Review Board (IRB)• Qualitative
– Interviews– Observations/VT– Focus Groups– Quick writes
• Quantitative– Surveys– Tests– Participation, attendance, server hits, etc
• Pilot test & timing
![Page 13: Evaluating Educational Technology Brian McNurlen & Chris Migotsky University of Illinois at Urbana-Champaign](https://reader034.vdocuments.site/reader034/viewer/2022051621/5697bf811a28abf838c856b9/html5/thumbnails/13.jpg)
Activity #2
• Outline an evaluation plan– Evaluation questions– Tools or methods you’re going to use– Timeline– Expected outcomes
• Keep it simple– What can be accomplished in a semester?– What can be accomplished with limited resources and
staff?
• Share in small groups
![Page 14: Evaluating Educational Technology Brian McNurlen & Chris Migotsky University of Illinois at Urbana-Champaign](https://reader034.vdocuments.site/reader034/viewer/2022051621/5697bf811a28abf838c856b9/html5/thumbnails/14.jpg)
Clicker study: Evaluation Questions
• Will implementing this clicker program affect how students perform academically?
• Will implementing this clicker program affect the overall student feedback scores for my course?
![Page 15: Evaluating Educational Technology Brian McNurlen & Chris Migotsky University of Illinois at Urbana-Champaign](https://reader034.vdocuments.site/reader034/viewer/2022051621/5697bf811a28abf838c856b9/html5/thumbnails/15.jpg)
Design
• Comparison groups– Two different lecture sections
• Independent variable – Use of clickers, along with instructional plan– Other section just uses the instructional plan
• Dependent variable – Some measure of academic performance,
perhaps a cumulative final
![Page 16: Evaluating Educational Technology Brian McNurlen & Chris Migotsky University of Illinois at Urbana-Champaign](https://reader034.vdocuments.site/reader034/viewer/2022051621/5697bf811a28abf838c856b9/html5/thumbnails/16.jpg)
Timeline
1. July– Select clicker technology, training2. August-- TA training3. Late August– Student training, pre-test4. Sept.-December-- 16 weeks of treatment5. Reliability observations-- October, December6. December-- post-test7. April-- follow-up reliability observation
![Page 17: Evaluating Educational Technology Brian McNurlen & Chris Migotsky University of Illinois at Urbana-Champaign](https://reader034.vdocuments.site/reader034/viewer/2022051621/5697bf811a28abf838c856b9/html5/thumbnails/17.jpg)
3. Data Analysis
• Results…what does it mean?– Level of analysis– Practical significance– Outliers/extremes– Baseline/benchmark data– Response rates
![Page 18: Evaluating Educational Technology Brian McNurlen & Chris Migotsky University of Illinois at Urbana-Champaign](https://reader034.vdocuments.site/reader034/viewer/2022051621/5697bf811a28abf838c856b9/html5/thumbnails/18.jpg)
4. Conclusion & Implications
• What do your results tell you?– Make changes– Sharing your results (IRB?)– Follow-up evaluation– Share with next generation of students
![Page 19: Evaluating Educational Technology Brian McNurlen & Chris Migotsky University of Illinois at Urbana-Champaign](https://reader034.vdocuments.site/reader034/viewer/2022051621/5697bf811a28abf838c856b9/html5/thumbnails/19.jpg)
Evaluating Ed Tech: Common Issues
• Too much data
• Evaluating something that could be done by other means, or is impossible to do without the technology
• Technology is a constantly moving target so results will be quickly outdated
• Any other issues?
![Page 20: Evaluating Educational Technology Brian McNurlen & Chris Migotsky University of Illinois at Urbana-Champaign](https://reader034.vdocuments.site/reader034/viewer/2022051621/5697bf811a28abf838c856b9/html5/thumbnails/20.jpg)
Evaluation Resources
• Teaching Center or Survey Center on your campus• Western Michigan University Evaluation Center
http://www.wmich.edu/evalctr/• Key Evaluation Checklist
http://www.wmich.edu/evalctr/checklists/checklistmenu.htm#models• An Educator's Guide to Evaluating The Use of Technology in
Schools and Classrooms
http://www.ed.gov/pubs/EdTechGuide/index.html• Evaluation Springboard
http://www.evaluationspringboard.org/• Survey Monkey
http://www.surveymonkey.com
![Page 21: Evaluating Educational Technology Brian McNurlen & Chris Migotsky University of Illinois at Urbana-Champaign](https://reader034.vdocuments.site/reader034/viewer/2022051621/5697bf811a28abf838c856b9/html5/thumbnails/21.jpg)
Teaching & Technology Resources
• Principles of Good Practicehttp://fc.byu.edu/tpages/tchlrn/7princip.htm
http://www.tltgroup.org/programs/seven.html
• The Technology Sourcehttp://technologysource.org/article/
toward_more_effective_instructional_uses_of_technology/
• ECARwww.educause.edu/ecar
![Page 22: Evaluating Educational Technology Brian McNurlen & Chris Migotsky University of Illinois at Urbana-Champaign](https://reader034.vdocuments.site/reader034/viewer/2022051621/5697bf811a28abf838c856b9/html5/thumbnails/22.jpg)
Teaching & Technology Resources
• An analysis of technology enhancements in a large lecture course.http://www.educause.edu/ir/library/pdf/
eqm0335.pdf
• Planning and Sustaining Evaluation of Instructional Technology Support Programshttp://www.educause.edu/ir/library/powerpoint/
EDU04110.pps
![Page 23: Evaluating Educational Technology Brian McNurlen & Chris Migotsky University of Illinois at Urbana-Champaign](https://reader034.vdocuments.site/reader034/viewer/2022051621/5697bf811a28abf838c856b9/html5/thumbnails/23.jpg)
Questions?
• Brian McNurlen(217) 244-5724
• Chris Migotsky(217) 333-3490