testa seda keynote spring 2016
TRANSCRIPT
![Page 1: TESTA SEDA Keynote Spring 2016](https://reader036.vdocuments.site/reader036/viewer/2022070517/58d0cc011a28ab866c8b6483/html5/thumbnails/1.jpg)
Changing the culture of assessment and feedback
through TESTA
Tansy Jessop@tansyjtweets @TESTAwin
13 May 2016
![Page 2: TESTA SEDA Keynote Spring 2016](https://reader036.vdocuments.site/reader036/viewer/2022070517/58d0cc011a28ab866c8b6483/html5/thumbnails/2.jpg)
![Page 3: TESTA SEDA Keynote Spring 2016](https://reader036.vdocuments.site/reader036/viewer/2022070517/58d0cc011a28ab866c8b6483/html5/thumbnails/3.jpg)
The TESTA Methodology
75 PROGRAMME AUDITS
Programme Team
Meeting
![Page 4: TESTA SEDA Keynote Spring 2016](https://reader036.vdocuments.site/reader036/viewer/2022070517/58d0cc011a28ab866c8b6483/html5/thumbnails/4.jpg)
Sustained growth
![Page 5: TESTA SEDA Keynote Spring 2016](https://reader036.vdocuments.site/reader036/viewer/2022070517/58d0cc011a28ab866c8b6483/html5/thumbnails/5.jpg)
TESTA….
“…is a way of thinking about assessment and feedback”
Graham Gibbs
![Page 6: TESTA SEDA Keynote Spring 2016](https://reader036.vdocuments.site/reader036/viewer/2022070517/58d0cc011a28ab866c8b6483/html5/thumbnails/6.jpg)
TESTA shifts in perspective from…
• ‘my’ module to ‘our’ programme
• from teacher-focused on module delivery to student experience of whole programme
• from individualistic modular design to coherent team design
• from the NSS to enhancement strategies
![Page 7: TESTA SEDA Keynote Spring 2016](https://reader036.vdocuments.site/reader036/viewer/2022070517/58d0cc011a28ab866c8b6483/html5/thumbnails/7.jpg)
TESTA addresses three problems
Problem 1: Knee-jerk problem
Problem 2: Curriculum design problem Problem 3: Evidence to action problem
![Page 8: TESTA SEDA Keynote Spring 2016](https://reader036.vdocuments.site/reader036/viewer/2022070517/58d0cc011a28ab866c8b6483/html5/thumbnails/8.jpg)
Problem 1: The knee-jerk
![Page 9: TESTA SEDA Keynote Spring 2016](https://reader036.vdocuments.site/reader036/viewer/2022070517/58d0cc011a28ab866c8b6483/html5/thumbnails/9.jpg)
Wow! Our students love History! Fantastic!
![Page 10: TESTA SEDA Keynote Spring 2016](https://reader036.vdocuments.site/reader036/viewer/2022070517/58d0cc011a28ab866c8b6483/html5/thumbnails/10.jpg)
Whoops there’s a little problem here
![Page 11: TESTA SEDA Keynote Spring 2016](https://reader036.vdocuments.site/reader036/viewer/2022070517/58d0cc011a28ab866c8b6483/html5/thumbnails/11.jpg)
Fix it!
Ok, we’ll look especially at polishing up our feedback. Students seems to
find that the least best thing.
![Page 12: TESTA SEDA Keynote Spring 2016](https://reader036.vdocuments.site/reader036/viewer/2022070517/58d0cc011a28ab866c8b6483/html5/thumbnails/12.jpg)
Apply spit and polish
![Page 13: TESTA SEDA Keynote Spring 2016](https://reader036.vdocuments.site/reader036/viewer/2022070517/58d0cc011a28ab866c8b6483/html5/thumbnails/13.jpg)
Anyone for the feedback sandwich?
I cushion the blow!
The hard truths are nicely disguised!
Me too - nice and soft!
![Page 14: TESTA SEDA Keynote Spring 2016](https://reader036.vdocuments.site/reader036/viewer/2022070517/58d0cc011a28ab866c8b6483/html5/thumbnails/14.jpg)
Problem 2: Curriculum design problem
![Page 15: TESTA SEDA Keynote Spring 2016](https://reader036.vdocuments.site/reader036/viewer/2022070517/58d0cc011a28ab866c8b6483/html5/thumbnails/15.jpg)
Does IKEA 101 work for complex learning?
![Page 16: TESTA SEDA Keynote Spring 2016](https://reader036.vdocuments.site/reader036/viewer/2022070517/58d0cc011a28ab866c8b6483/html5/thumbnails/16.jpg)
Curriculum privileges ‘knowing’ stuff
“Content is often the most visible aspect for students, the control of which is frequently devolved to individual academics, who receive little or no training in curriculum design and planning”
(Blackmore and Kandiko 2014, 7).
![Page 17: TESTA SEDA Keynote Spring 2016](https://reader036.vdocuments.site/reader036/viewer/2022070517/58d0cc011a28ab866c8b6483/html5/thumbnails/17.jpg)
Blunt instrument curriculum
![Page 18: TESTA SEDA Keynote Spring 2016](https://reader036.vdocuments.site/reader036/viewer/2022070517/58d0cc011a28ab866c8b6483/html5/thumbnails/18.jpg)
Problem 3: Evidence to action gap
![Page 19: TESTA SEDA Keynote Spring 2016](https://reader036.vdocuments.site/reader036/viewer/2022070517/58d0cc011a28ab866c8b6483/html5/thumbnails/19.jpg)
Problem 3: Evidence to action gap
Three misguided assumptions:1. Problem is a lack of high
quality data.
2. Analysis and findings a key mechanism for change.
3. Academics’ intellectual approach will facilitate change.
http://www.liberalarts.wabash.edu/study-overview/
![Page 20: TESTA SEDA Keynote Spring 2016](https://reader036.vdocuments.site/reader036/viewer/2022070517/58d0cc011a28ab866c8b6483/html5/thumbnails/20.jpg)
Proving is different from improving
“It is incredibly difficult to translate assessment evidence into improvements in student learning”
“It’s far less risky and complicated to analyze data than it is to act”
(Blaich & Wise, 2011)
![Page 21: TESTA SEDA Keynote Spring 2016](https://reader036.vdocuments.site/reader036/viewer/2022070517/58d0cc011a28ab866c8b6483/html5/thumbnails/21.jpg)
Paradigm What it looks like
Technical rational Focus on data and tools
Relational Focus on people
Emancipatory Focus on systems and structures
![Page 22: TESTA SEDA Keynote Spring 2016](https://reader036.vdocuments.site/reader036/viewer/2022070517/58d0cc011a28ab866c8b6483/html5/thumbnails/22.jpg)
TESTA themes and impacts
1. Variations in assessment patterns2. High summative: low formative3. Disconnected feedback4. Lack of clarity about goals and standards
![Page 23: TESTA SEDA Keynote Spring 2016](https://reader036.vdocuments.site/reader036/viewer/2022070517/58d0cc011a28ab866c8b6483/html5/thumbnails/23.jpg)
1. Huge variations
• What is striking for you about this data?
• How does it compare with your context?
• Does variation matter?
![Page 24: TESTA SEDA Keynote Spring 2016](https://reader036.vdocuments.site/reader036/viewer/2022070517/58d0cc011a28ab866c8b6483/html5/thumbnails/24.jpg)
Characteristic Range
Summative 12 -227
Formative 0 - 116
Varieties of assessment 5 - 21
Proportion of examinations 0% - 87%
Time to return marks & feedback 10 - 42 days
Volume of oral feedback 37 -1800 minutes
Volume of written feedback 936 - 22,000 words
![Page 25: TESTA SEDA Keynote Spring 2016](https://reader036.vdocuments.site/reader036/viewer/2022070517/58d0cc011a28ab866c8b6483/html5/thumbnails/25.jpg)
And some patterns…Characteristic Low Medium High
Volume of summative assessment
Below 33 40-48 More than 48
Volume of formative only Below 1 5-19 More than 19
% of tasks by examinations Below 11% 22-31% More than 31%
Variety of assessment methods
Below 8 11-15 More than 15
Written feedback in words Less than 3,800 6,000-7,600 More than 7,600
![Page 26: TESTA SEDA Keynote Spring 2016](https://reader036.vdocuments.site/reader036/viewer/2022070517/58d0cc011a28ab866c8b6483/html5/thumbnails/26.jpg)
Actions based on evidence
a) Reduction in summative b) Increase in formative c) Streamlined varieties d) More or less feedback depending…e) Quantifiable f) Every time a coconut with each feature
![Page 27: TESTA SEDA Keynote Spring 2016](https://reader036.vdocuments.site/reader036/viewer/2022070517/58d0cc011a28ab866c8b6483/html5/thumbnails/27.jpg)
Theme 2: High summative: low formative
• Summative ‘pedagogies of control’
• Circa 2 per module in UK
• Ratio of 1:8 of formative to summative
• Formative weakly understood and practised
![Page 28: TESTA SEDA Keynote Spring 2016](https://reader036.vdocuments.site/reader036/viewer/2022070517/58d0cc011a28ab866c8b6483/html5/thumbnails/28.jpg)
What students say…
• A lot of people don’t do wider reading. You just focus on your essay question.
• In Weeks 9 to 12 there is hardly anyone in our lectures. I'd rather use those two hours of lectures to get the assignment done.
• It’s been non-stop assignments, and I’m now free of assignments until the exams – I’ve had to rush every piece of work I’ve done.
![Page 29: TESTA SEDA Keynote Spring 2016](https://reader036.vdocuments.site/reader036/viewer/2022070517/58d0cc011a28ab866c8b6483/html5/thumbnails/29.jpg)
What students say: the barriers
• If there weren’t loads of other assessments, I’d do it.
• If there are no actual consequences of not doing it, most students are going to sit in the bar.
• It’s good to know you’re being graded because you take it more seriously.
• The lecturers do formative assessment but we don’t get any feedback on it.
![Page 30: TESTA SEDA Keynote Spring 2016](https://reader036.vdocuments.site/reader036/viewer/2022070517/58d0cc011a28ab866c8b6483/html5/thumbnails/30.jpg)
Assessment Arms Race
![Page 31: TESTA SEDA Keynote Spring 2016](https://reader036.vdocuments.site/reader036/viewer/2022070517/58d0cc011a28ab866c8b6483/html5/thumbnails/31.jpg)
Actions based on evidence
1. Rebalance summative and formative2. Shared language: programme approach3. Formative in the public domain4. Linking formative and summative5. Risky, creative, challenging tasks6. Students reading and producing more7. Deeper understanding of value of formative
![Page 32: TESTA SEDA Keynote Spring 2016](https://reader036.vdocuments.site/reader036/viewer/2022070517/58d0cc011a28ab866c8b6483/html5/thumbnails/32.jpg)
Theme 3: Disconnected feedback
![Page 33: TESTA SEDA Keynote Spring 2016](https://reader036.vdocuments.site/reader036/viewer/2022070517/58d0cc011a28ab866c8b6483/html5/thumbnails/33.jpg)
Take five
• Choose a quote that strikes you.
• What is the key issue?
• What strategies might address this issue?
![Page 34: TESTA SEDA Keynote Spring 2016](https://reader036.vdocuments.site/reader036/viewer/2022070517/58d0cc011a28ab866c8b6483/html5/thumbnails/34.jpg)
What students say…
The feedback is generally focused on the module.
It’s difficult because your assignments are so detached from the next one you do for that subject. They don’t relate to each other.
Because it’s at the end of the module, it doesn’t feed into our future work.
I read it and think “Well, that’s fine but I’ve already handed it in now and got the mark. It’s too late”.
![Page 35: TESTA SEDA Keynote Spring 2016](https://reader036.vdocuments.site/reader036/viewer/2022070517/58d0cc011a28ab866c8b6483/html5/thumbnails/35.jpg)
Students say the feedback relationship is broken…
Because they have to mark so many that our essay becomes lost in the sea that they have to mark.
It was like ‘Who’s Holly?’ It’s that relationship where you’re just a student.
Here they say ‘Oh yes, I don’t know who you are. Got too many to remember, don’t really care, I’ll mark you on your assignment’.
![Page 36: TESTA SEDA Keynote Spring 2016](https://reader036.vdocuments.site/reader036/viewer/2022070517/58d0cc011a28ab866c8b6483/html5/thumbnails/36.jpg)
Actions based on evidence
• Conversation: who starts the dialogue?• Iterative cycles of reflection across modules• Quick generic feedback: the ‘Sherlock’ factor• Feedback synthesis tasks• Technology: audio, screencast and blogging• From feedback as ‘telling’…• … to feedback as asking questions
![Page 37: TESTA SEDA Keynote Spring 2016](https://reader036.vdocuments.site/reader036/viewer/2022070517/58d0cc011a28ab866c8b6483/html5/thumbnails/37.jpg)
Theme 4: Confusion about goals and standards
• Consistently low scores on the AEQ for clear goals and standards
• Alienation from the tools, especially criteria and guidelines
• Symptoms: perceptions of marker variation, unfair standards and inconsistencies in practice
![Page 38: TESTA SEDA Keynote Spring 2016](https://reader036.vdocuments.site/reader036/viewer/2022070517/58d0cc011a28ab866c8b6483/html5/thumbnails/38.jpg)
What students say…
We’ve got two tutors- one marks completely differently to the other and it’s pot luck which one you get.
They have different criteria, they build up their own criteria.
It’s such a guessing game.... You don’t know what they expect from you.
![Page 39: TESTA SEDA Keynote Spring 2016](https://reader036.vdocuments.site/reader036/viewer/2022070517/58d0cc011a28ab866c8b6483/html5/thumbnails/39.jpg)
What students say…
There are criteria, but I find them really strange. There’s “writing coherently, making sure the argument that you present is backed up with evidence”.
I get the impression that they don't even look at the marking criteria. They read the essay and then they get a general impression, then they pluck a mark from the air. I don’t have any idea of why it got that mark
![Page 40: TESTA SEDA Keynote Spring 2016](https://reader036.vdocuments.site/reader036/viewer/2022070517/58d0cc011a28ab866c8b6483/html5/thumbnails/40.jpg)
Caught in a paradigm war…
Scientific Paradigm Naturalistic paradigmNeutrality Interpretation
Written and traceable Free, ephemeral, incidental, gaps
Convergent Divergent
Standardised Varied
Final word Dialogic, provisional
Accountability and evidence Social practice
![Page 41: TESTA SEDA Keynote Spring 2016](https://reader036.vdocuments.site/reader036/viewer/2022070517/58d0cc011a28ab866c8b6483/html5/thumbnails/41.jpg)
Taking action: internalising goals and standards• Regular calibration exercises• Discussion and dialogue• Discipline specific criteria (no cut and paste)
Staff Team
• Rewrite/co-create criteria• Marking exercises (ASKE CETL)• Design and value formative
Staff and students
• Enter secret garden - peer review• Engage in drafting processes• Self-reflection
Students
![Page 42: TESTA SEDA Keynote Spring 2016](https://reader036.vdocuments.site/reader036/viewer/2022070517/58d0cc011a28ab866c8b6483/html5/thumbnails/42.jpg)
It’s about educational paradigms…
![Page 43: TESTA SEDA Keynote Spring 2016](https://reader036.vdocuments.site/reader036/viewer/2022070517/58d0cc011a28ab866c8b6483/html5/thumbnails/43.jpg)
Transmission Model
![Page 44: TESTA SEDA Keynote Spring 2016](https://reader036.vdocuments.site/reader036/viewer/2022070517/58d0cc011a28ab866c8b6483/html5/thumbnails/44.jpg)
Social Constructivist Model
![Page 45: TESTA SEDA Keynote Spring 2016](https://reader036.vdocuments.site/reader036/viewer/2022070517/58d0cc011a28ab866c8b6483/html5/thumbnails/45.jpg)
Impacts at Winchester
• Upwards trajectory on A&F scores on NSS on TESTA programmes – ‘Top 4’ University
• TESTA ‘effect’ - people talk about formative• Experimentation in co-creation• Team approach to designing curricula• Design cycle for periodic review includes TESTA
![Page 46: TESTA SEDA Keynote Spring 2016](https://reader036.vdocuments.site/reader036/viewer/2022070517/58d0cc011a28ab866c8b6483/html5/thumbnails/46.jpg)
ReferencesBlaich, C., & Wise, K. (2011). From Gathering to Using Assessment Results: Lessons from the Wabash National Study. Occasional Paper #8. University of Illinois: National Institution for Learning Outcomes Assessment.Boud, D. and Molloy, E. (2013) ‘Rethinking models of feedback for learning: The challenge of design’, Assessment & Evaluation in Higher Education, 38(6), pp. 698–712. doi: 10.1080/02602938.2012.691462.Gibbs, G. & Simpson, C. (2004) Conditions r which assessment supports students' learning. Learning and Teaching in Higher Education. 1(1): 3-31.Harland, T., McLean, A., Wass, R., Miller, E. and Sim, K. N. (2014) ‘An assessment arms race and its fallout: High-stakes grading and the case for slow scholarship’, Assessment & Evaluation in Higher Education, 40(4), pp. 528–541. doi: 10.1080/02602938.2014.931927.Hughes, G. (2014) Ipsative Assessment. Basingstoke. Palgrave MacMillan.Jessop, T. and Maleckar, B. (2014). The Influence of disciplinary assessment patterns on student learning: a comparative study. Studies in Higher Education. Published Online 27 August 2014 http://www.tandfonline.com/doi/abs/10.1080/03075079.2014.943170Jessop, T. , El Hakim, Y. and Gibbs, G. (2014) The whole is greater than the sum of its parts: a large-scale study of students’ learning in response to different assessment patterns. Assessment and Evaluation in Higher Education. 39(1) 73-88.Nicol, D. (2010) From monologue to dialogue: improving written feedback processes in mass higher education, Assessment & Evaluation in Higher Education, 35: 5, 501 – 517.O'Donovan, B , Price, M. and Rust, C. (2008) 'Developing student understanding of assessment standards: a nested hierarchy of approaches', Teaching in Higher Education, 13: 2, 205 — 217Sadler, D. R. (1989) ‘Formative assessment and the design of instructional systems’, Instructional Science, 18(2), pp. 119–144. doi: 10.1007/bf00117714.