JISC LLL WFD Assembly

Maximising Impact – assembling & leveraging the

evidence baseDr Rachel HarrisInspire Research Ltd

10th January 2011

Inspire Research Ltd

Curriculum Delivery & Evaluation

The programme invited projects to “transform how they deliver and support learning across a curriculum area through the effective use of technology”.

13 projects funded by JISC, and 2 by Becta– What did they want to know?– How did they do it?– What impact did they have?

(And, what was the evidence?)

Project outputs: http://jiscdesignstudio.pbworks.com/

Project blogs aggregated at: http://www.netvibes.com/circlejisc#Curriculum_Delivery_Projects

Inspire Research Ltd

Delivery evaluation mapLive map

Inspire Research Ltd

Impact on the bottom line

• eBiolabs – no more breakages or how to balance a centrifuge

• Cascade – reduced admin time & the evidence to prove it

• eReflect – students engage with assessment feedback

Inspire Research Ltd

eBiolabs• Since introduction of eBiolabs, student lab prep

time increased from 20 to 35 minutes (average).• 54% students agreed they “felt well prepared

when they entered the lab” compared to 18% prior to introduction of eBiolabs.

• Designed to support one 40 credit unit in one School. Now supports ten units across the Faculty of Medical and Veterinary Sciences.

• Approx 120 teaching staff impacted by use of eBiolabs at Bristol. Currently, 900 student users.

http://ebiolabs.learnsci.co.uk/

Inspire Research Ltd

Cascade – Oxford University• “We forecast savings equivalent to over 38

weeks of productive admin time a year.”(Savings identified via historical data and time motion studies.)

• “These efficiency savings, and our ability to quantify them, have been useful when we have needed to make the case to fully embed and sustain these services going forward.”

http://cascade.conted.ox.ac.uk/

Inspire Research Ltd

eReflect – Westminster University

https://sites.google.com/a/staff.westminster.ac.uk/mac/Home

Inspire Research Ltd

“The e-Reflect feedback suggested that I speak to the module leader … I wouldn’t normally have thought to do this and it helped. He gave me ways I could improve and suggested a different way of approaching some of the subject matter. It has helped I think because since then my marks have shot up.”Link to video

Inspire Research Ltd

Evaluation: things to think about

• What are your evaluation questions – what do you want to know?

• A bit on reporting

Inspire Research Ltd

Questioning the project

• How well was the project implemented?• To what extent were the objectives met?• What were the barriers and enablers that

made the difference between successful and disappointing outcomes?

• What else was learned? (why or how did outcomes happen?)

Adapted from Davidson, E.J. (2009) Improving evaluation questions and answers: Getting actionable answers for real-world decision makers. Presented at AEA conference.

Inspire Research Ltd

Value or So what?• How valuable are the outcomes to stakeholders

(students, lecturers, admin staff, the sector)?• Was the project worth doing, given the

resources used? (Consider the balance of resources to outcomes.)

• Why would the project, or aspects of it, be worth implementing elsewhere? (How reusable is it?)

• How sustainable is the project? (Could it continue or expand with limited external funding?)

Adapted from Davidson, E.J. (2009) Improving evaluation questions and answers: Getting actionable answers for real-world decision makers. Presented at AEA conference.

Inspire Research Ltd

Saving a millionHow do they know?

Average cost of commercial alternatives to MyStudyBar = £115

Estimated saving based on MyStudyBar downloads, multiplied by ave cost = £1M

Inspire Research Ltd

What is valuable about your project?In ten words outline a• Benefit to particular stakeholders

(learners, lecturers, admin staff, senior managers, the sector)

Inspire Research Ltd

Telling the evaluation story• Executive Summary - with headings linked to

your key evaluation questions (or objectives).• Background – contextual issues that influenced

the project – what the project evaluated and why, including Key project evaluation questions.

• Methodology – the evaluation approach and techniques used (& why).

• Findings/Results - sections that address evaluation questions, summarising what was found, followed by the evidence.

Inspire Research Ltd

But!

How policy makers & practitioners access research

• Very short time to access project outputs• More likely to trust information received face-to-

face (or from peers)• Reluctant to pay for access to academic journals

(or ‘pay’ time to look for project findings)• Access new research mostly via the internet

Adapted from Beckmann, K., & Mason, C. (2010). An investigation of the knowledge exchange practices of the end-users of sustainability research.

Inspire Research Ltd

And, it is worth remembering

• Institutional processes are really complicated.• ‘Project’-based evidence usually plays a very

minor role (in institutional decision-making).• It is possible to influence policy/your institution,

but you need:– a holistic understanding of the context you are

working in,– additional skills,– to really want to do it.

Adapted from Young, J. (2008). Strategies to Enhance Research Impact. ODI, London. Retrieved from http://www.slideshare.net/RECOUP/strategies-for-impact-and-policy-relevance-v2

Any questions and thank you!

Dr Rachel A Harristel: 07779 980333email: rachel@inspire-research.co.uk web: www.inspire-research.co.uk