research leads network event: brighton 18 april 2015
TRANSCRIPT
Developing research tools for teacher inquiry
Nick Rose
Research Leads Network Event: Brighton 18 April 2015
What will I be talking about?
• What’s my role as research lead?
• Further thoughts …
• What tools might help teacher inquiry?
• What (little) do we know about improving teaching?
1. What’s my role as a research lead?
Leading practitioner for psychology and research
Janus-faced role
Outward looking: Find psychological and educational research which might inform development of teaching practice; summarise and disseminate this within the school (e.g. via the blog, coaching groups and twilight INSEDs).
What’s my role as a research lead?
Inward looking: Co-ordinating the whole-school coaching programme; working with small groups and individuals looking to develop their teaching through inquiry projects; developing teacher inquiry and evaluation tools to provide useful feedback.
2. What do we know about improving teaching?
What should teachers be trying to improve?
A strong understanding of the subject being taught – including the ways students think about the subject and the common misconceptions they may possess.
Effective questioning and use of assessmentReviewing previous learningProviding model responses for studentsGiving adequate time for practice to embed skills securelyProgressively introducing new learning (scaffolding)
Constantly demanding more, but still recognising students’ self-worth. Attributing student success to effort rather than ability.
Behaviour management: the relationships and routines which lead to efficient use of lesson time
Subject knowledge
Instructional techniques
Classroom climate
What feedback helps teachers improve?
Schools often use Ofsted style grading structures – which are principally summative. We need formative observation protocols to provide challenging yet supportive feedback.
Value-added data appears too infrequently in the academic year to provide regular feedback on teaching. Assessment data not often used as a formative form of feedback.Problem of Goodhart’s Law: “When a measure becomes a target, it ceases to be a good measure.”
A cheap and easy source of reasonably valid feedback. Must avoid the ‘popularity’ trap in order to provide useful insight.
Cognitive biases means pure reliance on ‘reflection’ is likely to have minimal impact on teaching practice.
Qualitative analysis is difficult and therefore tends to be superficial.
Observations
Student data
Student surveys
Teacher self reports
Work scrutiny
We need a greater range of research tools for teachers to use
We want teachers to focus on inquiry projects which will have an impact on student outcomes.
Teacher inquiry will only be effective if the evaluation of these outcomes is rigorous.
Teachers need access to range of fairly easy-to-use yet reasonably valid evaluation tools which can be adapted to their inquiry focus.
We need formative evaluation tools – we have mainly summative
Observations, value-added data and student surveys have been investigated as measures of effective teaching.
Problem is that they were all developed as summative measures for the purposes of accountability – not to inform professional development.
Summative measures are designed to discriminate – i.e. to identify effective versus less effective teachers.
Formative measures are designed to inform improvement – i.e. to explicitly identify ways in which a teacher could improve on their teaching.
There will be some cross-over between the two – but we need university-based researchers to help create and validate formative measures of teaching effectiveness so that they provide more developmentally useful feedback.
3. What tools might help teacher inquiry?
Some ideas …1. Coaching logs as a scaffold for teacher inquiry2. Focused and formative observations3. Using student data to inform development4. Classroom climate logs5. Student surveys as formative feedback
Coaching logs as a scaffold for teacher inquiry
Can we embody the principles of effective teacher feedback within the scaffolds we use to guide teacher inquiry?
Six principles of teacher feedbackSustained professional learning is most likely to result when:• the focus is kept clearly on improving student outcomes;• feedback is related to clear, specific and challenging goals for the
recipient;• attention is on the learning rather than to the person or to
comparisons with others;• teachers are encouraged to be continual independent learners;• feedback is mediated by a mentor in an environment of trust and
support;• an environment of professional learning and support is promoted
by the school’s leadership.
Whole-school peer coaching
Peer Coaching Cycle12 Weeks - 6 sessionsCross-curricular teams• Challenge for all• Classroom climate• Engagement and motivation• Evidence into practice
Professional ReviewIdentify key developmental priorities.Connect to goals related to student outcomes.
Faculty FeedbackShare outcomes of coaching project with others in subject area.
Subject T&LSubject-based pedagogy.
Getting started: What will you try? Which students will you focus on? How will you measure outcomes?
Work in progress: Where have you got to? What are the obstacles? Have you started to measure any difference it makes for your target pupils?
Reflection and sharing: What were the outcomes for students? Why did / didn’t it work? What have you learnt? How will you share your work?
Coaching logs as a scaffold for teacher inquiry
Scaffolding inquiry using coaching questions:
What is your focus for this coaching cycle? Will coaching be supporting a goal identified in your professional review?
Which students will you focus upon? Why did you select these students? Where are your students now?
What is the intervention / innovation you intend to try to improve the outcomes for these students? Why have you chosen this?
How might you measure / evaluate whether the intervention makes a difference for those students?
How will you share your work with other teachers? What elements do you think other teachers could learn from within / outside your subject area?
Focused and formative observations
Can we structure observation protocols to encourage more focused and developmental feedback?
Focused and formative observations
Explicit message about purpose
Lesson observations should be a supportive and developmental process designed to ensure that all teachers have the skills and support they need to carry out their role effectively and to continue to improve their professional practice and development as teachers.
Focused and formative observations
Using reflection questions to frame observation feedback in a developmental way:
How did the teacher consolidate previous skills / knowledge?
How did the teacher introduce new learning?
How did teacher make the links between previous and new learning?
How did the teacher challenge / support individual students?
Focused and formative observations
Creating reflection prompts focused on specific developmental areas
e.g. Developing behaviour management
Routines
What were the routines the teacher used in the lesson?
• Entering the classroom:
• Seating plan:
• Taking the register:
• Distributing equipment / books:
• Transitions between learning activities:
• Others:
Using student data to inform development
Can we encourage teachers to use student assessment data in developmentally useful ways?
Using student data to inform development
Summative data
Pupil attainment and value-added isn’t very useful for developing my teaching.
It doesn’t tell me why certain students underperformed or what I might do about it.
TARGET A2 summaryStudent ALIS (50) ASP PSYA1 PSYA2 PSYA3 PSYA4 UMS Grade PointsA B A 84 90 87 96 357 A* 300B C B 66 58 87 94 305 B 240C C B 64 75 77 67 283 B 240D C B 72 83 64 58 277 C 210E B A 58 75 64 68 265 C 210F D C 62 63 73 48 246 C 210G D C 34 60 64 67 225 D 180
APS 227.1
Using student data to inform development
Question analysis
Given most students performed well on this question, variation in outcomes is likely to be consolidation rather than the way I taught it.
What could I do to ensure lower PA students revise this topic thoroughly?
Using student data to inform development
Question analysis
None of my students did well on this question.
Does this indicate a weakness in the way I taught it?
Have I missed a misconception here that’s holding students back?
Am I confident in my understanding of how this topic is assessed?
How could I teach this differently?
How helpful is the textbook – do I need to find a better resource to support this topic?
Classroom climate logs
Can we use a self-report system to track changes in classroom climate over time?
Hayden (2014)
Level 10 You feel completely relaxed and comfortable; able to undertake any form of lessonactivity without concern. ‘Class control’ not really an issue—teacher and pupilsworking together, enjoying the experiences involved.
Level 7 You can undertake any form of lesson activity, but the class may well be rather ‘bubbly’ and rowdy; there may be minor instances of a few pupils messing around on the fringes of the lesson but they desist when required to do so. No one goes out of their way toannoy you or challenges your authority.
Level 2 The pupils largely determine what will go on in the lesson. You take materials into thelesson as a manner of form, but once distributed that will be ignored, drawn on ormade into paper aeroplanes. When you write on the board, objects will be thrown atyou rather than round the room. You go into the room hoping that they will be ina good mood and will leave you alone and just chat to each other.
Classroom climate logs
Classroom climate logs
Used the log as part of induction for all new members of teaching staff. A check on whether the behaviour management system of the school was working as intended.
Continued use periodically with NQTs and RQTs. Logs used alongside specific CPD on developing the routines and relationships that form behaviour management.
Classroom climate logs
Helped to identify challenging groups / times of day where the teacher might need additional support or where the class allocation / membership needed reviewing.
Structured self-report system for teacher to see whether the behaviour management strategies they were trying were having the positive effect they wanted with the most challenging classes.
Student surveys as formative feedback
Can we use student surveys to identify areas to improve in our teaching?
Student surveys as formative feedback
“Not every survey will produce meaningful information on teaching.
“Not to be confused with popularity contests, well-designed student perception surveys capture important aspects of instruction and the classroom environment.”
Some measure of validity:
Teachers’ student achievement gains for those identified as being in the top 25 percent Student Perception Surveys the equivalent of about 4.6 months of schooling more in math—over the course of the year and measured using state tests—than the students of teachers whose survey results were in the bottom 25 percent.
Student surveys as formative feedback
“… even a good instrument, implemented poorly, will produce bad information. Attending to issues such as student confidentiality, sampling, and accuracy of reporting takes on greater urgency as systems look toward including student surveys in their evaluation systems.”
Important to have a consistent protocol for the use of surveys.
Vital that teachers have confidence in how the data from their students will be used. In our context, we made the decision at the start that teachers owned the data produced.
Student surveys as formative feedback
We’ve been trialling the MET project student survey as a coaching tool.
“To measure teaching, the survey groups items under seven constructs, called the “7 Cs”: Care, Control, Challenge, Clarify, Confer, Captivate, and Consolidate. For each, the survey poses a series of statements, asking students’ level of agreement on a five-point scale.”
e.g. Control• Student behaviour in this class is under control.• I hate the way that students behave in this class.*• Student behaviour in this class makes the teacher angry.*• Student behaviour in this class is a problem.*• My classmates behave the way my teacher wants them to.• Students in this class treat the teacher with respect.• Our class stays busy and doesn’t waste time.
Student surveys as formative feedback
Student surveys run within coaching groups.
November survey used by teacher to identify areas they want to work on.
Student surveys as formative feedback
This teacher was keen to work on areas relating to consolidation.
Identified some strategies (e.g. questioning and verbal feedback) which the teacher would trial. Conducted a follow up survey in January to look at whether students perceived progress in this area.
Consolidate [48%] 50% 11 We get helpful comments to let us know what we did wrong on assignments 39% 16 My teacher takes the time to summarise what we learn each day 43% 18 My teacher checks to make sure we understand what s/he is teaching us 61% 35 The comments that I get on my work in this class help me understand how to improve
JD 9-1 Science 11/14 – 03/15
Consolidate [66%] 63% 11 We get helpful comments to let us know what we did wrong on assignments 50% 16 My teacher takes the time to summarise what we learn each day 88% 18 My teacher checks to make sure we understand what s/he is teaching us 63% 35 The comments that I get on my work in this class help me understand how to improve
4. Further thoughts …
Can we use structured observation techniques to investigate changes in behaviour to evaluate mentoring interventions?
Structured observation of pupil behaviour in lessons
Problems measuring effectiveness of mentoring interventions
Discipline points: Problem is that a reduction may simply reflect the teacher ‘losing faith’ with the behaviour management system rather than improvements in behaviour.
Reward points: Where intrinsic motivation is lacking we use extrinsic motivators. Where intrinsic motivation is ok, we shouldn’t (and usually don’t) use extrinsic motivators (which is why ‘naughty’ kids tend to get more reward points than ‘good’ kids).
Effort grades : Attitudes to learning scores may simply reflect teacher bias. e.g. Rausch, Karing, Dörfler and Artelt (2013) Personality similarity between teachers and their students influences teacher judgement of student achievement.
Attitude surveys: Can be easily distorted by social desirability bias. There is also a big gap between a change in reported attitudes and actual changes in behaviour.
Structured observation of pupil behaviour in lessons
Learning is invisible …
Rob Coe’s suggestion at the December Research Leads event:
‘Time on task’ analysis for selected students in lessons.
A question like: ‘What fraction of a lesson does a student appear to be thinking hard about the learning material?’
Poor proxies for learning.
1. Students are busy: lots of work is done (especially written work)2. Students are engaged, interested, motivated3. Students are getting attention: feedback, explanations4. Classroom is ordered, calm, under control5. Curriculum has been ‘covered’ (i.e. presented to students in some form)6. (At least some) students have supplied correct answers (whether or not they really understood them or could reproduce them independently)
… but are there behaviours ‘proximate enough’ for teacher inquiry?
Structured observation of pupil behaviour in lessons
Aren’t some behaviours more proximate to learning than others?• Looks at teacher when they are speaking?• Taking notes – jots down ideas?• Distracted by other student / external event?• Diverts attention from the teacher?
• Engages in discussion about learning topic?• Engages in off-topic conversation (e.g. last night’s football)?
• Completes activities within time agreed with teacher?• Procrastination techniques employed (e.g. finding a pen)?
• Makes mistakes and attempts to correct errors? • Waits for teacher to model answer then copies into book?
We need a method for providing valid / comparable feedback on whether a mentoring intervention is having a positive / neutral / negative effect for a student.
Whilst accepting the inevitable limitations of behavioural proxies for learning, can we make structured observation techniques valid enough for teacher inquiry?
Structured observation of pupil behaviour in lessons
What are structured observation techniques?
Event sampling:
With Event Sampling a list of the behaviours under investigation is devised (usually based on a pilot study), and the number of observed instances is recorded on a checklist, without taking into account the chronological order.
Time Sampling:
If the chronological order of the events is useful to know, then Time Sampling could be used instead. In this case, the units of observations are entered at standard intervals of time (for example, every 2 minutes), and coding happens as the observation unit occurs.
Content analysis:
Content Analysis could be developed to examine children’s work in books to provide a comparable quantitative description. The information is broken down into categories and then summarised; allowing us to characterise the material in a way that is potentially reliable.
