project evaluation: measuring learning impact diane salter, vice provost teaching and learning

Project Evaluation: Measuring Learning Impact

Diane Salter, Vice Provost Teaching and Learning

Why measure ‘learning impact’?

To determine whether a pedagogical intervention has influenced student learning as intended

To monitor progress and change to enhance student learning

To provide empirical data to support the pedagogical viability of the learning/teaching method

When might this be important? As part of a scholarly approach to

teaching – ongoing revision and course/ programme review

Projects/grants applications and

reporting to show:Does this project/instructional strategy promote

active learning-centred teaching? Does this project/instructional strategy lead to

significant improvements in quality or outcomes of student learning?

How might the approach to this subject be improved?

To measure impact we need to know:

How do we measure impact on learning?

What are the objectives/expected outcomes

of the intervention?

What are the indicators of success?

How can these be observed/measured?

What is the baseline (what was happening before the intervention) against which changes can be judged?

Measuring impact

Level 1 – Reaction

Level 2 – Knowledge and Skills Gained

Level 3 – Behaviour

Level 4 – Performance Competence *

Measuring impactReaction – How did it feel ? Did they like it?

Do they think it made a difference?

Knowledge and Skills Gained – What can they do that they couldn’t do before, or do better than they could do before?

Behaviour – What do they do now that they didn’t do before?

Performance competence – Has their performance improved as a result of increased learning?

Examples of indicators

Student Feedback

Perceived usefulness and benefitsSelf report on motivation, engagement or study approaches Self-reported learning gains

Instructor Feedback

Observed changes in students’ motivation, engagement and study behaviourTeachers’ reflections, etc.

Classroom Observations

Nature/amount of classroom interactionsStudents’ engagement and participationOther classroom study behaviours

Student performance

Assessment grade/marksQuality of student work Performance in practice settings

How do we know our course design works?

What was the effect of an experience the learners? (learning impact):

Knowledge (e.g. know what, know how and know why)

Skills (e.g. cognitive, meta-cognitive, social and psychomotor)

Attitude (e.g. beliefs, disposition and values)

What’s in a name?

Scholarly Teaching

Scholarship of Teaching

9

Scholarly Teaching

Evidence based approaches to teaching and course design based on best practice.

10

Scholarship of Teaching

Sharing Presentations Publications Reflection Revision

11

What is the difference?

Research for scholarship and publication Ethical approval required

Research/feedback for the purposes of improvement of studies and programs.

See Article 2.5 Tri-council agreement

12

Tri-council policy statement

The Tri-council policy statement on Ethical Conduct for Research Involving Humans guides research done at university and provides clear direction regarding doing research/gathering feedback for the purposes of improvement of studies and programs.

See article 2.5 (next slide)

13

Tri-council policy statement

Article 2.5 states:

Quality assurance and quality improvement studies, program evaluation activities, and performance reviews, or testing within normal educational requirements when used exclusively for assessment, management or improvement purposes, do not constitute research for the purposes of this Policy, and do not fall within the scope of the REB. 

14

More information needed?

If you would like a seminar in your department about SoTL and measuring learning impact

please contact:

diane.salter@kpu.ca

15