© curriculum foundation1 part 2 from theory to practice part 2 from theory to practice

© Curriculum Foundation 1

Part 2

From theory to practice


Evaluation in action

We have looked at different forms of evaluation:

• Establishing a benchmark before instituting changes• Evaluating against outcomes• Evaluating the process• Using evaluation formatively and summatively

We shall look at what these mean in our schools – and we also need to consider quality assurance and quality control.


Establishing a starting point

Noorland’s analysis points to the need to establish a benchmark before instituting change. This could relate to the process or to the outcomes – or both!

In terms of outcomes, it is essential that these include the full range of outcomes the change is designed to bring about. If the intention is to develop competencies such as critical thinking or problem-solving, then it is essential that these are included.

It is also essential to include other aspects of learning that we do not want to get worse!

We shall look at the process benchmark later.

In their ‘Curriculum Design Handbooks’, Brian Male and Mick Waters suggest that:“as evaluation is both formative and summative, we do not have to wait until the end of a programme to see if it is successful; we can check as we go along. In addition to the subject assessments we are making in terms of the national curriculum or syllabus, we have also to consider the Skills and Competencies that form the ‘roots’ of learning."

© Curriculum Foundation


Male and Waters distinguish between ‘Quality Assurance’ and ‘Quality Control’

Quality Assurance

Making sure that your programme is likely to

be as successful as possible before starting

to teach it.

Quality Control

Checking that the programme is working well as it is being used.


Quality Assurance takes many forms in schools. At one important level, it involves subject leaders or headteachers checking curriculum or lesson plans before they are implemented. But what are they looking for when they scrutinise these plans? What questions do they ask or what criteria do they apply in order to assure quality?

Brian Male has set out a ‘Design Checklist’ which can be amended to suit different levels of planning and design.

This obviously looks at the process rather that the outcomes – but can be part of setting the expected outcomes.

A Design Checklist

1) At the end of this learning experience, what will the students have learned?

2) Is it really necessary to learn this at all?3) If they learn this, what level will they attain in terms of the subject? Is

this high enough in terms of their age and prior learning?4) What skills and competencies will they be developing?5) Will the planned learning experiences actually bring about this

intended learning?6) How long will this take? (Does it really need to take half a term?)7) Where should it take place? (Classroom, school grounds, wider

locality?)8) Who should be involved? (Teacher, parents, others?)9) Couldn’t we make it even more interesting?


1. At the end of this learning experience, what will the students have learned?If it is not clear what the key learning is, then this needs to be clarified. What is the point if doing this at all? The clarification often alters the design for the better.

2. Is it really necessary to learn this at all?Why is this piece of learning being included in the curriculum? Is it a requirement of the national curriculum, an essential experience of childhood or the locality, or an attempt to widen the students’ understanding? Or is it something we do not need to do at all. We need to be rigorous here.

3. If they learn this, what level will they attain in terms of the subject? Is this high enough in terms of their age and prior learning?If it does not appear in the present national curriculum levels or the new national curriculum year prescription, you need to wonder why you are including it. If you cannot find it in your school’s list of skills and competencies, you are not addressing important key skills. This is a key question in terms of challenge and rigour. If you can find it, then you need to ensure it is at the right level. And what if our assessment records show that some students have already reached this level? How can we amend the design to take account of this?

Evaluating our curriculum design: Quality AssuranceSome explanations of the questions


4. What skills and competencies will they be developing?Have we included roots as well as leaves? Are these building on skills already acquired?

5. Will the planned learning experiences actually bring about this intended learning?This point is often missed. There is often a clear learning objective, and an engaging experience – but the two do not always match. However long the students were to engage in this experience, they would never achieve the objectives! There needs to be rigour here before the design is implemented. We also need to ask whether all the experience is necessary. Does some seem unnecessary or irrelevant?

6. How long will this take? (Does it really need to take half a term?)We need some flexibility here if we are really focusing on the outcome, but the suggested time does need to be challenged. Time sometimes seems to come first!

7. Where should it take place? (Classroom, school grounds, wider locality?)This is an opportunity for widening the experience. Could we take the students out of school – to a museum, on a field trip, or just engage in some practical first-hand experiences? Is this best as a series of lessons, or could it be an event or part of the routines of the school? (Do you remember lessons, events and routines?)


8. Who should be involved? (Teacher, parents, others?)This could also widen the experience. Does it have to be just the teacher? Could parents be involved, or local experts, outside visitors or teachers from other classes or schools?

9. Couldn’t we make it even more interesting?All curriculum design sessions and evaluations should end (or start!) with questions about how we can make the learning as exciting as possible for the students.


Evaluating against outcomes: Quality Control

This is where we use the assessment data from Unit 5. There are several factors to consider when evaluating against the expected outcomes:

• Did we specify the right outcomes in the first place? (This is where the quality assurance process is so important)

• Are we able to determine the whole range of outcomes? Do we include skills, competencies and personal development? (This is Stenhouse’s concern – are we able to take account of the full richness and depth of the learning)

• Are we able to link the outcomes directly to the programme we designed, or do we need to consider other factors as well?

• Do we know how well the students were doing before we started this programme? (This is Norland’s point about establishing a benchmark)


Because it is always difficult to take account of the whole range of learning that goes on in a school, we end up looking for some key aspects that we can take account of and focus on these. These then become ‘Key Performance Indicators (KPIs)’.

The danger of KPIs is that people sometimes forget that they are only indicators and so they become regarded as the performance itself.

An analogy is to think about measuring the flow of a river. At the centre it seems to be flowing quickly, but it’s slower at the edges and in some places is eddying backwards. (Does it sound like your school?) So how do we describe its flow overall? Well, we take two or three key points across the river and take those to stand for the whole.

The problem comes when we try to improve school performance by focusing solely on the KPI – forgetting that they only stand for the performance overall. Even if we made the river flow a bit quicker at the three measuring points, it would not necessarily be flowing faster overall – the eddies might be going backwards even faster.

The other issue is that KPIs tend to focus on the easily measured academic outcomes, and seldom the skills and competencies.


Process and Product Evaluation

We are so used as the teaching profession to being judged against outcomes that we forget that in other professions it is the process that is key to evaluation. For example, doctors can be held responsible if they do not administer the right treatment, but not if they do all the right things and the patient dies anyway (their patients always die anyway, eventually!). The judgement of professionalism focuses on the process, and it is accepted that other factors can impact on the outcome. The same is true of defence lawyers who cannot be held responsible if their client is found guilty – only if they did not present the case properly. It is accepted that there are all sorts of reasons why a jury might bring in a ‘guilty’ verdict that are beyond the lawyer’s control. This focus on process is a professional approach.

Yet in schools we are frequently being held responsible for outcomes that are beyond our control. The reason a student does not learn may well lie in our lessons or curriculum, but it might also lie at home or in the 18 hours a day they are not at school.

Our evaluations must take account of this – even if the judgements of some other agencies’ do not do so!


Principles of a World Class Curriculum A world-class curriculum will:• be based on clear, shared values, aims, principles that put the learner at the heart

of the curriculum and recognise their role as citizens of the world• provide exciting opportunities for the intellectual, physical, emotional, social,

scientific, aesthetic and creative development of every learner• ensure the development of competencies for learning and life and a sense of hope

and agency in every learner• encourage independence of mind and action and the development of individual

interests and talents• excite the imagination, encourage curiosity and develop creativity• secure learners’ knowledge, skills and understanding of the world's major

branches of learning, disciplines and subjects• ensure understanding of how learning in different disciplines is interconnected

and relevant to life, global issues and world events past, present and future• provide clear and relevant pathways for learning and the flexibility to respond to

developing needs, interests and contexts• locate learning in the context of the learner’s life and local community and also

within a national and international dimension• address contemporary issues as well as the big ideas that have shaped the world

A World Class Curriculum

The Curriculum Foundation, working with the Centre for the Use of Research and Evidence in Education (CUREE), has developed a process approach to evaluating the whole curriculum against a set of criteria. (You may detect echoes of Stenhouse here!)

These give schools a way of taking stock of the curriculum before planning any developments (Noorland’s first stage) or of using a process-based approach to evaluate a more mature curriculum.

There are more details about this at: www.curriculumfoundation.org

And here are the main criteria.

http://www.curriculumfoundation.org/


Here are two more faces identified:

Alexander and Hedberg (1994) summarised the representative approaches used in educational research over the past 50 years and gave the following four key paradigms with their perceived advantages and disadvantages:


Objective-based: Evaluation as a process of determining the degree to which educational objectives are being achieved. This follows the scientific tradition and is straightforward to apply, but does not take account of unintended outcomes, and takes no account of students as individuals with all their differences.

Decision-based: Focuses on the decisions made during development and improvements that could be made. It is useful for programs with a large scope or multiple levels, but needs the co-operation of decision makers. It has proved difficult to put into practice and expensive to maintain.

Value-based: Evaluation is not only concerned with goals, but also whether the goals are worth achieving. Formative and summative evaluation is used, and the evaluator considers major effects, achievements and consequences of the program. This acknowledges the importance of unintended outcomes, and learners' perceptions of the learning experience, and evaluation can be made without the need to know about the objectives. Its perceived disadvantages are that it may leave important questions unanswered.

Naturalistic approach: Organises evaluations around the participants' key concerns and issues. Uses qualitative data collection such as journals, observations and interview. The advantages are that it acknowledges context and can be used to benefit those being studied, but participants may identify criteria with little educational worth.


In Slide 7, we posed a ‘Big Question’ (actually, there were three questions!) about how we are going to find out whether curriculum change has been successful, what counts as success and how long it has to run before we can tell.

Are we any closer to answering these questions?

To answer the first, we could look at both process and outcomes. If we are looking at outcomes, then we must be sure that the range of outcomes we are considering is sufficiently wide to encompass our overt goals and also the wider aspirations of education. (Back to Unit 1 where we set out our aims – and asked what the School Aims say)

If we are looking at the process, then we must have some regard for the outcomes. It is no use having what looks like a good process if no-one is learning anything. (Of course, it would hardly be a good process if no-one was learning anything!)

Stenhouse would argue that is it only through taking account of the participants’ accounts that you can truly discover how successful a curriculum change has been. This is not just about process but involves taking the widest view about outcomes. How do the students see the outcomes? Can we rely solely upon assessment to provide the whole picture of outcomes?


The second question, what counts as success, will always depend upon your professional judgment and on the extent of your initial aims and goals. No evaluation can be effective unless we are very clear at the outset what it is we are trying to achieve.

This takes us back to Unit 1 again. This really is the first step in design and also in evaluation.

And how long does it have to run? Well, Prof Emma Norland answered that rather neatly with her chart outlining what sort of data can be considered at each stage. There is always something to evaluate – but there are not always summative outcomes.


The first ‘Big Issue’ (Slide 9) question concerned finding out about complex outcomes. Stenhouse and Ruddock have pointed to the need to consider a range of information broader than just test scores to achieve this.

The Big Issue also asked at what level success would be achieved. The notion that there can be a set arithmetical threshold for success is rather fanciful. (If 60% gain a particular score on a test then that’s successful, 59% is not.) The government can set targets for percentages of Level 4s or 5A* to Cs, but this is both arbitrary and unreasonable because it ignores context and the myriad of complex factors that affect learning. Progress measures are clearly an improvement on these performance thresholds but they still rely on figures, valid assessments of start and end points. Both approaches suggest that a very complex process can be reduced to a single number.

As a profession, we know that simplistic criteria of this kind are unhelpful and can be damaging to our students. We must be strong enough as a profession to say so. There are very many effective and professional methods of evaluating education success, and they should be used. An arbitrary percentage of scores on a single test should only ever be taken as part of a wider array of information.


That leaves us with the ‘Side Issue’ on slide 10. How can we be sure that success is the result of our curriculum change? How do we know that they would not have learned even more anyway? Of course, we can never be entirely sure in education. Our students are not being taught in laboratory situations with control groups.

However, we must be as thorough as we can in terms of the benchmarking we carry out before making a change and in the range of factors that we consider in our evaluation.

And if the change seems not to be successful, is that the curriculum’s fault? Again, we can never really be sure – but we can be as thorough as possible in the methods we use. It is also where a range of data beyond test outcomes is essential.

20

Do you remember Andreas Schleicher from Unit 1? He is the head of the ‘PISA’ international educational comparisons.

He points out that:

‘PISA tests students’ ability to:

• apply their learning• think critically• solve problems• make judgements’

You can find more about PISA evaluations at: http://www.ted.com/talks/andreas_schleicher_use_data_to_build_better_schools.html

His PISA organisation sets out to use this data to evaluate whole national education systems. This is educational evaluation on a vast scale.

You will recognise by now that this is mainly an outcomes evaluation – and the outcomes are those that PISA decides itself. But those outcomes are compared with a host of other factors such as class size, amount spent on education and socio-economic status.

There are many valuable lessons to learn about evaluation from PISA – both positive and negative!


So, that’s it. So it must be homework time again.

This time you might like to go back to the unit that you planned and this time give consideration to the benchmarking that you will carry out.

You need to think about the range of the learning expectations you have set so that they can all be taken account of in the evaluation. You also need to think back to Unit 5 and consider how you are going to establish a benchmark for skills and competencies and well as subject learning.

And if improvements in personal development are among your aims, these will need to be benchmarked as well.

As always, you can share your work at [email protected]

mailto:[email protected]

© curriculum foundation1 part 2 from theory to practice part 2 from theory to practice

Documents

process benchmark

range of outcomes

terms of outcomes

different forms of evaluation

quality assurance

quality control

curriculum foundation3

curriculum foundation1