aura programme: collect learning assessment data
TRANSCRIPT
Engaged Excellence in Research & Teaching Practicesaura
Collect Learning Assessment Data
Teaching Course 1, Day One, Session 4
Mood Monitor
Session’s Learning OutcomesBy the end of this session you will:
1. Recognise why data collection is important
2. Explain that data collection can:
1. Take place at different times (when);
2. Be used in different ways (how);
3. Measure different things (what)
3. Identify best practices in designing data collection questions
When is Formative Assessment Carried out?A. Before an intervention
B. During an intervention
C. Before and during an intervention
D. After an intervention
How do you feel about the prospect of a student evaluating your classes?A. It is the academics job to assess the student, not the other way
around
B. I collect student feedback only because I am asked to.
C. I incorporate student feedback into my teaching strategies and
curriculum design
D. I am unsure
What is your average class size?
A. 0-25
B. 26-50
C. 51-75
D. 75-100
E. 100 Plus
Group Activity
What forms of learning assessment data do you collect?
Discuss in groups and feed into the class
(5 minutes)
When, How, What?The 3 questions are interrelated
When: before training, after training or several months/years after
What: attainment; satisfaction rates, attitudes and long term behaviour and impact
How: qualitative or quantitative
When are you Measuring?
Formative assessment Summative assessment
Pre-training assessment
During training Post-training
How are you Measuring?
Pavel Ševela / Wikimedia Commons Nicholas/Wikimedia Commons
Qualitative Quantitative
What are you Measuring?Before Training After Training Long Term
Applicant suitability Satisfaction with course content
Change in skills/attitudes (was it retained?)
Participant profile Satisfaction with course delivery
Change in behaviour
Participant’s prior experience
Change in skills Impact
Expectations Change in attitudes ‘Externalities’
Group Activity 2: Design a SurveyPlease get into groups and design 5 questions that you would like to ask your students – use the following table:
Please try to get a mix of the When; What; and How
Question When? What? How?E.g. What are you hoping to gain from this course? Pre-course Background Qualitative
E.g. What industry are you working in now?
Post-course Career Quantitative
Group Activity 3: Critique a Survey1. Please get into groups and take a look at the
survey handout (5 mins)
2. Go over the survey and make amendments to existing questions where you feel you can improve them (10 mins)
Good Practices in Setting Data Collection Questions
Avoid Vanity Metrics
Vanity metrics include questions that participants feel compelled to
give a positive answer to
E.g. Just how satisfied were you with the our course?
• Often participants tell you what they think you want to hear
• This question is not explicit about the quality of the course
• Be specific and well directed in your questions to avoid vanity metrics
Use Likert ScalesA Likert Scale to score responses:1. Use 1-5 or 1-10
• Note: research shows individuals tend to choose the median score (e.g. 3 in 1 to 5 scale)
• Always indicate which is the high and low value (they are usually in ascending order but be explicit about this)
2. Use balanced and meaningful supporting text:• Strongly disagree• Disagree• No opinion• Agree• Strongly Agree
Likert Scales: Video
https://www.youtube.com/watch?v=q3FnFTJ528s
Use Exhaustive & Plausible QuestionsWhen supplying multiple-choice options ensure:
• Plausibility (more applicable to tests)
• Cater for all responses:• By ensuring that the categories are comprehensive• Allow for a free response option (i.e. ‘other’)
Plausibility: Video
https://www.youtube.com/watch?v=NXvmkTC9rcA
Collect Demographic Data
Collect profile data about the contributor, such as:• Gender: • Age: • Preferred contact:
• Email • Telephone / Mobile
• Organisation: • Job Title:
Demographic Data: Video
https://www.youtube.com/watch?v=hdRaMg3hBtk
Good Practice List1. Be specific – defining any technical words2. Avoid leading questions3. Avoid any ambiguity4. Avoid vanity metrics5. Be consistent in your questions6. Be careful with your sequence of questions7. Strike a balance between qualitative and quantitative8. Avoid asking too many questions (this is context specific)9. Avoid asking too few questions (also, context specific)10. Be creative
This work is licensed under a Creative Commons Attribution-Non-commercial ShareAlike 3.0
The content is authored by:Jagdeep ShokarMonitoring and Evaluation Advisor,Institute of Development Studies, [email protected]