Dr Tanya Vaughan, 26 May, CiTEL Curriculum Conference

2

We acknowledge the traditional custodians of the land on which we meet, the

Turrbal & Jagera people, and pay respect to elders past and present.

Confidential. For the use of E4L’s client only. Written permission required for any other use.

3

• Introducing Evidence for Learning

• Evidence ecosystem

• How we work

• Building, sharing and using evidence

• Teaching & Learning Toolkit

• Feedback Implementation Materials

• Assessment

• How to measure your impact

• Workshop with the Toolkit

4

5

Confidential. For the use of E4L’s client only. Written permission required for any other use.

11Confidential. For the use of Evidence for Learning's client only. Written permission required for any other use.

12evidenceforlearning.org.au/evidence-informed-educators/evidence-ecosystem

13

Three common barriers to accessing and using research:

• Shortage of time to engage with research

• Overload of information to process

• Insufficient contextualised information for practice.

Source: Sharples, J (2013), Evidence for the Frontline: A report for the Alliance For useful Evidence.

.

14Adapted from Sharples J Evidence Chain for the Frontline (2013). evidenceforlearning.org.au/evidence-informed-educators/evidence-ecosystem

15evidenceforlearning.org.au/evidence-informed-educators/evidence-ecosystem

16evidenceforlearning.org.au/evidence-informed-educators/impact-evaluation-cycle/

17

School leaders need to be continually working with their

staff to evaluate the impact of all on student progression.

Leaders need to create a trusting environment where

staff can debate the effect they have and use the

information to devise future innovations… Schools need

to become incubators of programs, evaluators of

impact and experts at interpreting the effects of

teachers and teaching on all students.

(Hattie J, What Works Best in Education: the Politics of Collaborative Expertise,

2015)

18

Share

Use

Build

School decisions

informed by

evidence

We are an Evidence

Intermediary; we play a

broking role between

research and practice.

We specialise in translating

evidence and then help

implementing it in real world

settings.

evidenceforlearning.org.auevidenceforlearning.org.au

Share knowledge

Drive use

Build evidence

Trials Unit (RCT)

Philanthropy + Govt $ on

school programs

(4 underway)

Teaching & Learning Toolkit

Global evidence summaries and

practice guides from international

partners

Tailored

evidence

toolkits (17)

Local Practice

Guides on key

topics

Supporting

school networks

(68 events to

date)

We are an Evidence

Intermediary; we play

a brokering role

between research and

practice

Better School

Decisions

informed by

evidence

We specialise in translating

evidence and then help

implementing it in real world

settings.

20

Program name Program description Evaluator Scale

Quick Smart Maths Intensive 30-week one-to-one

tutoring intervention to increase

fluency and automaticity in maths

for students performing in the

bottom third of their peers.

Teachers and Teaching

Research Centre at the

University of Newcastle

23 schools (primary

and secondary

schools), 293

students

Thinking Maths A two-term program of professional

learning for maths teachers in upper

primary and lower secondary South

Australian schools to stimulate

deeper and more engaging

instruction

Australian Council for

Educational Research7,367 students

across 167 schools

Mini Lit Testing a small-group reading

intervention for struggling Year 1

students, focusing on five keys: (1)

phonemic awareness; (2) phonics;

(3) fluency; (4) vocabulary; and (5)

comprehension.

Centre for Program

Evaluation at the

Melbourne Graduate

School of Education

and Murdoch

Children’s Research

9 schools, 237

students and 67

teachers/paraprofessi

onals

21

Resilient families was developed from Deakin

University. In a partnership between Vic Health and

Evidence for Learning, Western Sydney University is

evaluating the program in 14 Victorian Schools.

The Resilient Families program works with parents to

increase their engagement with schools and supports

students to develop their social skills and emotional

intelligence both at school and at home.

A previous trial of the program found students

became closer with their parents and teachers. Binge

drinking reduced by 25 per cent across the whole

student population, and depression reduced in at-risk

students whose parents attended program activities.

http://evidenceforlearning.org.au/news/

Fairhills High School acting assistant

principal Adrienne Tanner with year 7

students. The school is testing out a

program to help students resist peer

pressure. Photo: Eddie Jim

22

The Toolkit aims to:

• Support evidence-informed decision-making in Australian schools;

• Provide guidance for educators on how to use their resources to improve

educational outcomes for their students;

• Act as an introduction to educational research.

23

Average months’ worth of learning progress;

Cost to implement; and

The security of evidence.

24

25evidenceforlearning.org.au

Tablet or phoneLaptop

26

LaptopTablet or phone

evidenceforlearning.org.au/the-toolkit/

27

Average cost

The approximate cost of

implementing an approach.

Evidence security

Based on the quantity and

the methodological quality of

the available evidence, and

the reliability or consistency

of impact estimates.

Months’ impact

The additional months'

progress you can expect

students to make as a result

of an approach being used.

evidenceforlearning.org.au/the-toolkit/

28

29

LaptopTablet or phone

evidenceforlearning.org.au/the-toolkit/feedback

30evidenceforlearning.org.au/the-toolkit/feedback

31

LaptopTablet or phone

evidenceforlearning.org.au/the-toolkit/feedback

32evidenceforlearning.org.au/Australasian-research-summaries/

Summary of Australian and New Zealand

Research

References

Databases searched

Search terms

33

LaptopTablet or phone

34http://www.evidenceforlearning.org.au/the-toolkit/approaches-by-organisation/citel-curriculum-conference/citel/

35

36

37

The use of assessment information to monitor learning requires a shift

in mindset. It requires an expectation that assessment results –

whether for individuals or groups will provide more that uninterpreted

scores and grades for the purposes of ranking and comparing.

Teachers, students, parents, school leaders, system managers and

governments should expect assessment to provide useful, substantive

information about what students know, understand and an do, as a

basis for informed decision making.

Source: Masters, G (2017) Monitoring Learning. Bentley, T., & Savage, G. C. (2017). Educating Australia:

Challenges for the decade ahead. Melbourne University. Publishing

.

38

39

How would you define

feedback?

What are the different

categories of feedback you

can give to students (e.g.

what are the different areas

feedback can focus on?)

Pair and share with the

educator next to you.

40

In the end, it all comes down to the relationship between the teacher and the

student. To give effective feedback, the teacher needs to know the student—

to understand what feedback the student needs right now. And to receive

feedback in a meaningful way, the student needs to trust the teacher—to

believe that the teacher knows what he or she is talking about and has the

student's best interests at heart. Without this trust, the student is unlikely to

invest the time and effort needed to absorb and use the feedback.

The only thing that matters is what the student does with the feedback.

If the feedback you're giving your students is producing more of what you

want, it's probably good feedback. But if your feedback is getting you less of

what you want, it probably needs to change.

Source: Wiliam, D. (2016). The secret of effective feedback. Educational Leadership, 73(7), 10-15. http://www.ascd.org/publications/educational-leadership/apr16/vol73/num07/The-Secret-of-Effective-Feedback.aspx

.

41

• Feedback is information given to the learner and/or the teacher about the

learner’s performance relative to learning goals.

• It should aim to producing improvement in students’ learning.

• Feedback redirects or refocuses either the teacher’s or the learner’s actions

to achieve a goal.

• It can be about the learning activity, the process, the student’s management

of their learning or self-regulation or about them as individuals. It can come

from a teacher or someone taking a teaching role, or from peers.

evidenceforlearning.org.au/the-toolkit/feedback

42

Research suggests that it should be:

• specific, accurate and clear (e.g. “It was good because you...” rather than just “correct”);

• compare what a learner is doing right now with what they have done wrong before (e.g. “I

can see you were focused on improving X as it is much better than last time’s Y…”);

• encourage and support further effort and be given sparingly so that it is meaningful;

provide specific guidance on how to improve and not just tell students when they are

wrong;

• and be supported with effective professional development for teachers.

evidenceforlearning.org.au/the-toolkit/feedback

43aitsl.edu.au/feedback/

• Students who are praised for intelligence, ‘You did a brilliant job,

you are very smart’ – can develop a fixed mindset, seeing their

intelligence as fixed. These students can become:

• less confident

• less resilient

• less motivated

• decrease effort in response to set backs.

• Alternatively if students receive praise for effort, or process

feedback show greater engagement, motivation and

improvement. This focus on the process of achievement is a key

tenet of the growth mindset.

44aitsl.edu.au/feedbackaitsl.edu.au/feedback/

45aitsl.edu.au/feedbackaitsl.edu.au/feedback/

46

Black and Wiliam emphasise student self-regulation which is consistent with

the most powerful level of feedback identified by Hattie and Timperley.

http://evidenceforlearning.org.au/news/effective-feedback-in-action/

Level Description Example Impact

Self level personal evaluations and affect

(usually positive) about the

learner

Overall you did a good

job

Ineffective

Task how well tasks are understood

and performed

You need to include

appropriate, scientific

language

Useful

Process the main process needed to

understand/perform tasks

How did you evaluate

your research sources

Powerful

Self-

regulationself-monitoring, directing and

regulating actions

Can you think of a

different strategy to try?

Powerful

47

48evidenceforlearning.org.au/toolkit/meta-cognition-and-self-regulation

• Meta-cognition and self-regulation approaches (sometimes

known as ‘learning to learn’ approaches) aim to help learners

think about their own learning more explicitly.

• This is usually by teaching students specific strategies to set

goals, and monitor and evaluate their own academic

development.

• Self-regulation means managing one’s own motivation towards

learning. The intention is often to give students a repertoire of

strategies to choose from during learning activities.

49

Strategy Effect size Definition Description

Organising and

transforming

0.85 Rearrangement of

instructional materials,

analogies, cognitive

organisers.

Making an outline

before writing a paper

or making a concept

map.

Self-

consequences

0.70 Student arrangement or

imagination of rewards or

punishment for success or

failure.

Putting off pleasurable

events until work is

completed.

Self-instruction 0.62 Self verbalising the steps to

complete a given task.

Verbalising the steps in

solving a mathematics

problem

Self-evaluating 0.62 Setting standards and using

them for self-judgement.

Check quality of own

work prior to handing in

to teacher

Adapted from Lavery, l. (2008) as cited by Hattie, J (2009) Visible learning: A synthesis of over 800 meta-analyses relating to achievement. Routledge and Tan, K., Dawson, V., & Venville, G. (2008). Use of cognitive

organisers as a self regulated learning strategy. Issues in Educational Research, 18(2), 183-207.

50

Strategy Effect

size

Definition Description

Help-seeking: peers,

teachers, adults

0.60 Explaining to someone else, asking

questions, answering questions

Using a study partner.

Keeping records,

monitoring

0.59 Recording of information related to

study tasks.

Note taking,

summarising.

Rehearsing,

memorising

0.57 Memorization of material by overt or

covert strategies

Writing a mathematics

formula down until it is

remembered.

Goal setting / planning 0.49 Goals, sub-goals, timeline Making lists to

accomplish during

studying.

Self monitoring 0.45 Observing and tracking one’ own

performance and outcomes, often

recording the,

Keeping records of

study outputs.

Adapted from Lavery, l. (2008) as cited by Hattie, J (2009) Visible learning: A synthesis of over 800 meta-analyses relating to achievement. Routledge and Tan, K., Dawson, V., & Venville, G. (2008). Use of cognitive

organisers as a self regulated learning strategy. Issues in Educational Research, 18(2), 183-207.

51

52

Evaluating the impact of adopting a change is crucial to ensure that your school

is using its resources to best improve student learning. Importantly, the

evaluation needs to focus on two levels:

1) Measuring the impact by answering the question, Has there been an

improvement in students’ learning? (Hattie, 2015)

2) Identifying the active ingredients involved in the implementation of the

approach that worked in your setting and how they worked (Sharples, 2013).

Adapted from Hattie, J 2015, What works best in education: The politics of collaborative expertise, Pearson, London and Sharples, J. (2013). Evidence for the Frontline. London: The Alliance for Useful Evidence: http://www.alliance4usefulevidence.org/assets/EVIDENCE-FOR-THE-FRONTLINE-FINAL-5-June-2013.pdf

aitsl.edu.au/feedback

53

When evaluating impact, it is important to draw on both quantitative and

qualitative data (Caldwell & Vaughan, 2012).

• Quantitative data measures an amount or level and is usually represented by

numbers. e.g. assessment data such as grades, NAPLAN, PAT and attendance

rates.

• Qualitative data assesses reasons, opinions and motivations and is usually

represented by statements and illustrations. Example measures are

observational assessments or notes and samples of student work.

Adapted from Caldwell & Vaughan (2012). Transforming Education through the Arts. Routledge: New York aitsl.edu.au/feedbackaitsl.edu.au/feedback

54

What are useful measurements of impact to triangulate your

data?

• achievement data (using individual and group level results and/or effect

sizes)

• NAPLAN results (at individual and group level)

• student feedback through a survey

• teacher feedback through a survey

• interview data.

aitsl.edu.au/feedback

55

Vaughan, T., Harris, J., & Caldwell, B. J. (2011). Bridging the Gap in School Achievement through the Arts:

Summary report. In. Retrieved from http://www.songroom.org.au/wp-content/uploads/2013/06/Bridging-the-Gap-

in-School-Achievement-through-the-Arts.pdf

56

• Pick one or a few of the Toolkit approaches to help you address a specific

challenge within your school.

• From the Toolkit approach page and drawing from the Australasian Research

Summary, identify three teaching practices and three things to consider to

determine the impact of this approach (what will you measure?) (20 min).

For example targeting feedback to students at the self-regulation and

process level.

• Be ready to share your approach with the rest of the group.

evidenceforlearning.org.au/the-toolkit/approaches-by-organisation/acer/evidenceforlearning.org.au/the-toolkit/full-toolkit/

57

• Join our Evidence Informed Educator Network

evidenceforlearning.org.au/evidence-informed-educators/join/

• Subscribe to our newsletter for updates evidenceforlearning.org.au/

• Follow us on Twitter @E4Ltweets and Facebook Evidence for Learning

• Comments and feedback please tvaughan@evidenceforlearning.org.au

info@evidenceforlearning.org.au | evidenceforlearning.org.au | @E4Ltweets