university of exetersocialsciences.exeter.ac.uk/media/universityofexeter/colle… · web...

PGCE (PCE)

Curriculum Evaluation

Derek Wise February 2008

Content

Assignment 6b 3Definitions of Syllabus, Curriculum and Evaluation 4Curriculum Models 5HOW to Evaluate: A Simple Model for Evaluating Specifics 7Success Criteria 8WHAT to Evaluate: the CIPP Model of Programme Evaluation 9WHAT to Evaluate: the OFSTED of Course Evaluation 12Curriculum Evaluation – The Stakeholders and Interested Parties 16The 4 levels of training evaluation 17Curriculum evaluation – an exam board perspective 19Competence – a semantic differential 20Curriculum Evaluation - Less Reputable Evaluations 21Bennet’s Seven Curriculum Issues 23Curriculum Evaluation – Gathering Data 24Change - the five categories of product adopters: 27Reading List 27

Progress & Presentation of assignment 6b

‘Carry out an evaluation of curriculum, teaching and learning issues in the teaching of your subject specialism; measure the effectiveness of teaching and learning and make informed conclusions and recommendations as to the future development of the programme’

Feb 7: brief informal discussions with tutor on proposed curriculum evaluation

Feb 28: max 30 minute presentation of your proposed curriculum evaluation in 3 or 4 groups of 4/5 course members

Presenters should provide a seminar paper (max 2 A4 sides) to support their presentation – this will probably be bullet points responses to the 6 seminar issues above. Make 3 copies - 1 copy per 2 members and 1copy for tutor. OHTs should NOT be used. NB presenters will be stopped after 30 minutes to ensure equal time for all members

June 5 : max 30 minute final presentations of your completed curriculum evaluation. In 3 or 4 groups of 4/5 course members.

Presenters should provide a seminar paper (max 2 A4 sides) to support their presentation -this will probably be bullet points responses to the 8 seminar issues above (which could of course be an updated version of the earlier seminar paper). Make 3 copies - 1 copy per 2 members and 1copy for tutor. OHTs should NOT be used. NB presenters will be stopped after 30 minutes to ensure equal time for all members

(Curriculum Evaluation (approx 2500words) in a plastic project folder with name clearly indicated should be handed to Derek at the end of the session for marking). A copy of the seminar papers for February and June should be included in the folder

Assignments will be returned within 1 week and any required developments (hopefully there won’t be’) can be emailed to Derek. Thoughtful and carefully presented seminars will be positively taken into account in the light of any weaknesses in the final assignments!

Exeter PGCE (PCE) - Curriculum Evaluation Handbook:

Assignment 6b Evaluation of a Curriculum

Aims and Learning Outcomes

By the end of this assignment you will be able to: select and describe a course programme in your specialist subject analyse specific issues raised by the programme specification, course design,

teaching, learning or assessment methods in relation to the needs of typical learners of this subject and level

use evaluation techniques to identify and analyse the responses of learners and colleagues to aspects of the curriculum and its delivery

explain and apply models of curriculum to their evaluation identify strengths and weaknesses of the current programme, propose and justify improvements to the design or delivery of the programme justify your analysis and recommendations with reference to current research into

teaching and learning, with specific reference to you specialist subject present your evaluation in a concise, well-written and referenced report

Task

Candidates are required to:

1. carry out an evaluation of curriculum, teaching and learning issues in the teaching of your subject specialism

2. measure the effectiveness of teaching and learning 3. make informed conclusions and recommendations as to the future development of the

programme.

Suggested activities could include:

identifying your current weaknesses in and updating your subject knowledge gathering information about syllabuses, standards and national developments in your

subject reviewing the resources available for the teaching of your subject critically reviewing current issues in assessment related to your subject area undertaking a comparative examination of the teaching of your subject in another

establishment discussion with experienced practitioners on proven effective strategies for planning

and delivering teaching and learning.

Word count 2500 maximum, excluding bibliography and appendices

Assessment CriteriaThis assignment is marked on a pass/refer basis.

In completing this assignment, you should be able to demonstrate evidence of StandardsAS1-7, AK1.1-5.1, 6.1-7.3, BK1.1-2.7, 3.3, 4.1-5.2, CS1-4, CK1.1-4.2, DS1-3, DK1.1, 1.3, 2.1, 3.1-3.2, ES1, 5, EK1.1-2.1, 2.3, 3.1-3.2, 5.1-5.2, FS3,


Definitions of Syllabus, Curriculum and Evaluation

Evaluating Education- A Warning!

‘With the institutionalisation of ‘new wave’ evaluation, I think we have produced a situation in which there is some danger that evaluators create their own establishment and glamorise it as an elite. Let’s keep hold of the idea that it is mostly a matter of common sense and learning from experience. That is not entirely true but it keeps us from going technical or theological; and a little modest oversimplification is better than a lapse into jargon or pretentiousness’Stenhouse, L. (1975) An Introduction to Curriculum Research and Development Heinemann p116

Syllabus

A syllabus is a list of indicative content that is usually provided by the awarding body. This may come in forms such as competence statements, programmes of study, directives, learning outcomes etc. It is essentially static and in this sense prescribed and is only brought alive through description by the teacher and by the students where it becomes transformed to a curriculum.

Curriculum

A curriculum is dynamic. It incorporates the whole of the learner’s experience, both formal and informal. Curriculum is not the intention or prescription but what happens in real situations. It is about perceiving, understanding and describing what happens

.. a ‘course; especially a regular course of study at a school or university’(Shorter Oxford Dictionary)

In 1984 Sir Keith Joseph (then Secretary of State for Education) gave the annual North of England speech on the curriculum stating that ‘The curriculum must have relevance to the world of work ... it must be balanced, there must be breadth and differentiation’21 years later these principles are still fundamental to any curriculum

Evaluation

‘To judge or assess the worth of…..’Collins Concise English Dictionary

‘The process of evaluation is essentially the process of determining to what extent the educational objectives are actually being realised by the program of curriculum and instruction. However, since educational objectives are essentially changes in human beings, that is, the objectives aimed at are to produce certain desirable changes in the behaviour patterns of the student, then evaluation is the process of determining the degree to which these changes in behaviour are actually taking place.’ Tyler’ (1949) Basic Principles of Curriculum and Instruction Chicago: University of Chicago (pp. 105-106)

Evaluation is the process of conceiving, obtaining and communicating information for the guidance of educational decision-making with regard to a specified programme (MacDonald, 1975)

One of the important considerations in any systematic evaluation is to make clear what the criteria are for the evaluation.

It is the insistence on the explicitness about criteria which distinguishes evaluation … from the ‘everyday’ use of the term. (Harlen)


Curriculum Models

There are a number of 'models' of curriculum or instructional systems. Remember a model is an abstraction from reality - it doesn’t exist in itself but can provide a useful ‘ often visual - basis for analysis. Consider a curriculum with which you are currently engaged – does it conform to one or more of the following models?

1 linear curriculum:

Useful for instruction related to sequential topic areas containing objectives of roughly equal weighting.

Knowledge 1aSkill 1a

Knowledge 1bSkill 1b

Knowledge 1cSkill 1c

Required Knowledge & Skill for topic 1

Knowledge 2aSkill 2a

Knowledge 2bSkill 2b

Required knowledge & Skill for topic area 2

Example of use:particular value:Any inherent problems:Style(s) of teaching/learning best suited by this model:

2 Spiral Curriculum

Topics are covered in enough depth to provide basic understanding and prerequisite knowledge for 2nd time round and then represented in greater depth for 3rd time etc etc

Example of use:particular value:Any inherent problems:Style(s) of teaching/learning best suited by this model:

3 Pyramid Curriculum

Clear move from general (broad base) to particular

Year 3 - options

Year 2 - some specialism Year 1 - broad introduction

Example of use:particular valueAny inherent problems:Style(s) of teaching/learning best suited by this model:


4 Common Core

The common core is often linked to other models (eg pyramid with common core + options) It requires ensuring that the common element is necessary & sufficient

Example of use:particular valueAny Inherent problemsStyle(s) of teaching/learning best suited by this model:

5 Project centred curriculum

Leads out from an initial central theme and provides network of topics as a map to show where the learner should get to over a period of time - can be accredited as accumulated/transferable units. Very popular in primary education and syllabi for adult learners with particular needs


6 Research/Inquiry centred

Centred around substantive issue(s) or question(s) which the student has to inquire into in order to provide a coherent, in depth and analytical research report or dissertation or thesis


If the curriculum with which you engaged does not fit to any of the above models – how would you define it?


Curriculum Evaluation – HOW to Evaluate: A Simple Model for Evaluating Specifics

Evaluation is EXTREMELY important to the teacher as it provides the opportunity to make an informed opinion about something - and helps reflection. Evaluation is concerned with looking at the strengths & weaknesses of something; the efficiency & effectiveness of how it is undertaken and the value to those who are involved. This can be remembered by the acronym SWEET (nb – the acronym SWEET was created by Derek Wise and is not published). Evaluation should be based on evidence rather than just gut feelings.

This evidence can be quantitative (ie quantifiable as assessment results, numbers or physical evidence such as retention on courses). You can observe learner participation and interest, for example the time they spent on-task (doing the work they should be doing) as against being off-task over a period of time and recording your results. Your evaluation can also be qualitative (ie about the quality or feeling or perception of something for example learner’s feelings, tutor’s feelings or the reaction to the learners to the lesson).

Strengths What were the good things about it – how can they be used in the future? What worked well? why? – can this be applied to other lessons? What were the good results? – can they be replicated in the future?

Weaknesses What were the weak things about it? – how can they be overcome in the future? What did not work well? - why? – How could they be developed for the future? What were the poor results? – how can they be improved?

EfficiencyEfficiency is concerned with competence; the ability to do something well or achieve a desired result without wasted energy or effort, or the degree to which this ability is used

What things showed that the lesson was efficient eg (re-usable learning resources, recap of previous work to ensure all learners were ready to start etc). how can this efficiency be applied to other situations?

What things suggested that the lesson was not efficient? – how can this be improved?

EffectivenessEffectiveness is concerned with producing or causing a result; especially the desired or intended result. To be truly effective the result should be striking especially in producing a strong or favourable impression on people

What things showed that the lesson was effective? eg (learners were involved and appeared to be learning; appropriate learning resources, recap of work at the end of the session which showed learners had learnt) – what lessons can I learn for my teaching?

What things suggested that the lesson was not effective?

The value to those involved (value added) Sometimes social / personal things like teaching cannot be easily evaluated in terms

of efficiency and effectiveness or based on qualitative or quantitative data. The ‘extent to which the lesson (or more probably whole programme of study) has contributed to the growth of the learner although this may not appear in examination results is very important. This is sometimes called ‘value added’

Evaluation often requires comparison with similar events eg another lesson – it is essential that this evaluation is based on like events.


Curriculum Evaluation – HOW to Evaluate: Success Criteria

For a valid and reliable evaluation (particularly a course evaluation) it is essential to have a clear idea of the criteria for what would constitute a successful outcomes (ie the success criteria).

Success criteria are statements of what should reasonably be expected as a result of the action taken

For example if the student achievement on a particular course is 70% is this good or bad? If the success criteria established at the outset of the course was that the course should have an 80% student achievement rate (with grade stated if necessary) based on previous years, or on national benchmark figures from other providers, then clearly 70% is unsatisfactory.

Writing success Criteria

It is essential to ensure that the success criteria fair and honestly capture the true situation – not an adaptation to suit the evaluation!

A course evaluation of a GCSE course states that one success criteria will be the % of students who gain a grade B qualification. Results shows that 60% of the course members who took the exam gained a grade B. It appears pretty good!

But

The course started with 24 students and only 12 completed and took the exam - there was a 50% drop-out! Of these 12 students 60% - ie 7 students - gained a grade B.

So

in fact only 30% of the students who started the programme gained a grade B

The success criteria did not make it clear whether it was using the start number or the number taking the exam.

For each of the issues considered in the CIPP or OFSTED models it is important to establish a success criteria based on previous years, or national benchmark figures or on what the course team feels is appropriate. It is important to state how the success criteria has been reached

Success criteria should be observable and measurable

Success criteria should be based on what the objective reasonable and legitimately requires

Success criteria should be attainable – they should be designed to attain success rather than prevent it.


Curriculum Evaluation – WHAT to Evaluate: the CIPP Model of Programme Evaluation

A useful model for teaching programme evaluation is the Context-Input-Process-Product (CIPP) model (Daniel Stufflebeam 1966, 1967, 1971,1972,2000) which proposes a four part evaluation for any programme and suggests a number of questions which should be asked within those headings

In general, these four parts of an evaluation respectively ask:

1 What needs to be done? (context)2 How should it be done? (Input)3 Is it being done? (process)4 Did it succeed? (product)

Context - The setting of the course or subject ie the aims. This can be seen in terms of the validity of what is being taught External factors which impinge on the programme and which may affect its outcomes might be considered here (eg course is requirement for Government New Deal) .

Input - The input element relates to the factors which contribute to the course, ie the tutor(s), the students, the resources, the environment

Process - The process element concerns the appropriateness of what happens on the course, how the input elements are used to achieve the aims and objectives.

Product - The product concerns the outcomes - what has been gained (ie value added) and what has been achieved, (ie qualifications)

Product element – sub divisionsMore recently the ‘Did it succeed?’ or product evaluation part is itself divided into

Impact Were the right beneficiaries reached? Was there a possible audience who were not reached – if so why (nb LLUK values of open access)

Effectiveness - Were their needs met? Was there an in-effective prioritization of one aspect of the curriculum over another. Could the programme have been delivered in a more cost efficient way with equal effectiveness? Do the results of the effectiveness evaluation suggest that changes should be made in specific areas?

Sustainability - Were the gains for the beneficiaries sustained? Was there a short term improvement which was not evidence some time after the programme. Possible interview of past course members to ascertain the long-term programme value

Transportability - . Did the processes that produced the gains prove transportable and adaptable for effective use in other settings? How will early adopters be reached? How will the more reticent early and late adopters be persuaded? (see note on Everett Rogers at back of handbook)

Thus the model can be remembered by the acronym CIPPIEST!

For the purposes of general curriculum evaluation the initial CIPP model should suffice although evaluators could consider the 4 product aspects for greater insight.

CIPP in course evaluation

The four CIPP main headings provide a range of questions to be asked (for each one consider – how do I know this (data)?)


Context (The setting of the course or subject)

Meeting Needs

Are the aims consistent with the needs of the learner, industry or society?Does the curriculum develop links with the world of work?Does the curriculum meet current Government initiative eg basic skills/key skills?

Proposed Course Structure

Is the course structure the best way of achieving the desired outcomes?Are the aims updated regularly/has the course been updated in the light of national developments (eg basic skills/key skills, vocational developments

Encouraging Progression

Do the aims encourage progression in further or higher education, or transferability to other programmes?

Input (The input factors which contribute to the course)

Students

Do the students possess the entry ability required by the course contentDid the students original wish to do this course?

Tutors

Are tutors qualified (an area of current interest in post compulsory education)Are the tutors competent in the subject; in current teaching methods and where necessary, in current industrial requirements?Are there adequate tutors for all aspectsDo tutors have adequate time for all the requirements of the course eg IV/double marking/tutorials

Delivery

Is there an appropriate amount of time in line with awarding body specificationsAre resources appropriate, adequate and available?Is the environment conducive to learning and appropriate to the world of work?


Process (The appropriateness of what happens on the course,)

Students Do students have a ‘voice’ (eg staff/student forums) Is there appropriate course group support and interactivity facilities Is there an effective student guidance/counselling system

Tutors Do tutors receive/access, understand and use, all available documents related to the curriculum, ie information from examining bodies regarding syllabus and assessment

requirements Are there regular and useful course team meetings Do team members have access to necessary professional development?

Delivery Are teaching strategies appropriate? Are appropriate teaching aids and learning resources used? Is there appropriate/adequate general access to information technology? Is there appropriate/adequate general access to library and academic support ? Are course accommodation & timetabling facilities maintained (eg no constant room or

timetable changes)

Monitoring & Assessment Is the monitoring/tutoring/support system appropriate and adequate? Is the assessment system valid and reliable? Is there an effective system of feedback to learners ?

Product (what has been gained (ie value added) and achieved)

Students

Attendance rate (and possible comparison to achievement)What is the staying-on rate as a trend (this year compared to the last 2or 3)?What is the qualification achievement rate as a percentage of starters?Has the course contributed to the students ability to meeting the needs of work (eg presentation skills?What is the progression rate to further industry/FE/HE?What was the students response to the course – (eg results of student perception questionnaires

TutorsWhat staff development was identified and satisfied at individual and course team level?Are tutors motivated by their involvement in the course?

DeliveryTo what extent has the course been modified in the light on on-going feedback?If appropriate, what do employers feel about the course?Is there an effective system for evaluating and gaining feedback about the programme?

Developed by Derek Wise from Stufflebeam D. (1971). Educational Evaluation and Decision Making. In Stufflebeam, D.L. Madaus, G. F. & Kellaghan, T. (Eds.), (2000) Evaluation models (2nd ed.). (Chapter 16). Boston: Kluwer Academic Publishers.

(note: the CIPP model was developed over many years with specific aspects added at each review and development.)


Curriculum Evaluation – WHAT to Evaluate: the OFSTED of Course Evaluation

The Office for Standards in Education (Ofsted) is a non-ministerial government department, established in 1993 under the Education (Schools) Act 1992. It is responsible for inspecting the standards of independent and state schools, local education authorities, child day care and childminding in England; it also monitors the work of the Independent Schools Inspectorate. Ofsted is also required to provide independent advice to the United Kingdom government and parliament on matters of policy and to publish an annual report to parliament on the quality of educational provision in England. The legal name is the "Office of Her Majesty's Chief Inspector", reflecting the fact that Ofsted's powers are vested in its Chief Inspector, appointed by the Queen-Council. In September 2005 a new system of short notice inspections came into being. From April 2007 OFSTED will merge with the Adult Learning Inspectorate (ALI) to provide an inspection service which includes all post 16 government funded education

A college of course Evaluation undertaken by Ofsted

‘A college inspection is a process of evidence gathering to evaluate the effectiveness of the college in supporting the learning and achievement of students. The emphasis is on judging the degree to which rigorous self-assessment leads to quality improvement in the college”

The inspectors carry out inspections using the Common Inspection Framework, details of the criteria can be found on the Ofsted website. www.ofsted.gov.uk . Once the inspection has been conducted a report will be published, this will list the names of the qualification reviewed (including retention and achievement data). OFSTED’s Common Inspection Framework as a model for course evaluation

What are the school's main strengths? What are you working on to improve? How successful are your policies and practice? Does what you offer help children to reach the highest standards? Does what you do help to make the children safe and healthy?

Ofsted considers five issues

Effectiveness of provision Capacity to improve Achievements and standards Quality of provision Leadership and management

When observing courses and classes they ask 4 questions

How well do learners achieve?

How effective are teaching and learning?

Do activities meet the needs of learners?

How well are learners guided and supported?

(A curriculum evaluation could be based on these four questions)

These are assessed against four standards Outstanding Good Satisfactory


http://www.ofsted.gov.uk/

http://en.wikipedia.org/wiki/September_2005

http://en.wikipedia.org/wiki/Order-in-Council

http://en.wikipedia.org/wiki/United_Kingdom_parliament

http://en.wikipedia.org/wiki/United_Kingdom_government

http://en.wikipedia.org/wiki/United_Kingdom_government

http://en.wikipedia.org/wiki/Independent_Schools_Inspectorate

http://en.wikipedia.org/wiki/Independent_Schools_Inspectorate

http://en.wikipedia.org/wiki/England

http://en.wikipedia.org/w/index.php?title=Childminding&action=edit

http://en.wikipedia.org/wiki/Day_care

http://en.wikipedia.org/wiki/Day_care

http://en.wikipedia.org/wiki/Local_education_authority

http://en.wikipedia.org/wiki/School

http://en.wikipedia.org/wiki/State_school

http://en.wikipedia.org/wiki/Independent_school_(United_Kingdom)

http://en.wikipedia.org/wiki/1992

http://en.wikipedia.org/w/index.php?title=Education_(Schools)_Act_1992&action=edit

http://en.wikipedia.org/wiki/1993

http://en.wikipedia.org/wiki/Non-ministerial_government_department

http://en.wikipedia.org/wiki/Non-ministerial_government_department

Inadequate


OFSTED – SCHOOL SELF-EVALUATION

Schools now receive an on-line self evaluation form which is used by Inspectors. This form can be a useful basis for a curriculum evaluation

1. CHARACTERISTICS OF YOUR SCHOOL

1a Please outline the main characteristics of the learners, including:

- their attainment on entry and how you know this;- their social and economic backgrounds, indicating the level of prosperity or deprivation.

1b Please summarise briefly your distinctive aims and describe any special features of your school, for example:

- whether you intend to become a specialist school, or school with special status, and ifit is one already, the main changes that have occurred because of this;- whether you are a school with a religious character;- any special units, additional community services or extended provision;- significant partnerships with other providers or agencies (such as shared arrangementsfor the curriculum, federal arrangements, or partnerships with employers).

1c Please outline specific contextual or other issues that act as aids or barriers to raising performance, for example:

- any difficulties in recruiting and retaining staff, or governors;- recent or impending reorganisation;- mobility of learners;- particularly important facts in your recent history, such as change of leadership.

1e Please outline briefly the main priorities in your improvement/developmentplan, and how they reflect the context in which you work.

2. VIEWS OF LEARNERS, PARENTS/CARERS AND OTHER STAKEHOLDERS

What are the views of learners, parents/carers and other stakeholders and how do you know?

2a How do you gather the views of learners, parents/carers and other stakeholders, such as those accessing additional services, how often do you do this, and how do you ensure the impartiality of the information?

2b What do the views of learners, parents/carers and other stakeholders tell you about the learners’ standards, personal development and well-being, and the quality of your provision?

2c How do you share with parents/carers and other stakeholders the collated findings about their views?

2d Can you give examples of action you have taken based on the views of learners, parents/carers or other stakeholders, with an evaluation of the effectiveness of what you did?

- Are there examples of actions you decided not to take (with the reasons for this)?- Are there examples of ways in which your stakeholders have influenced the prioritiesnoted in section 1e?


3. ACHIEVEMENT AND STANDARDS

How well do learners achieve?

In answering the following questions, please make clear the main evidence, such as performance data, assessments and records of learners’ progress, on which your evaluation is based (but please use data selectively, avoiding the copying out of tables of descriptive information).

3a What are learners’ achievement and standards in their work?

- the standards learners reach as indicated by their test and examination results, taking account of: any significant variations between groups of learners, subjects, courses and key stages; trends over time; comparisons with other schools; whether learners reach challenging targets- the standards of learners’ current work in relation to their learning goals (noting any significant differences between current work and recent results)- learners’ progress relative to their starting points and capabilities, with any significant variations between groups of learners (making clear whether there are any groups that are underachieving and could be doing better)

3c On the basis of your evaluation, what are your key priorities for development?

4. PERSONAL DEVELOPMENT AND WELL-BEING

How good is the overall personal development and well-being of the learners? In answering the following questions, please make clear the main evidence on which your evaluation is based.

4a To what extent do learners adopt healthy lifestyles?- whether learners take adequate physical exercise, and eat and drink healthily- learners’ growing understanding of how to live a healthy lifestyle

4b To what extent do learners feel safe and adopt safe practices?- whether learners feel safe from bullying and racist incidents- the extent to which learners have confidence to talk to staff and others when they feel at risk

4c How much do learners enjoy their education?- take account of learners’ attitudes, behaviour and attendance- learners’ spiritual, moral, social, emotional and cultural development

4d How well do learners make a positive contribution to the community?- learners’ growing understanding of their rights and responsibilities, and of those of others- how well learners express their views and take part in communal activities

4e How well do learners prepare for their future economic well-being?- how well learners develop skills and personal qualities that will enable them to achieve future economic well-being- learners’ understanding of career options, and the acquisition of workplace skills

4g On the basis of your evaluation, what are your key priorities for development?


5. THE QUALITY OF PROVISION

Your evaluation of the quality of provision should take account of the impact on the standards achieved and the personal development and well-being of learners. In answering the following questions, please make clear the main evidence, such as monitoring of teaching, on which your evaluation is based.

5a How good is the quality of teaching and learning?

- how well teaching meets the needs of the full range of learners and course requirements- the suitability and rigour of assessment in planning learning and monitoring learners’ progress- the diagnosis of, and provision for, individual learning needs- the involvement of parents and carers in their children’s learning and development

5b How well do the curriculum and other activities meet the range of needs andinterests of learners?

- the extent to which the curriculum or activities match learners’ needs, aspirations and capabilities, building on prior attainment and experience- how far the curriculum meets external requirements and is responsive to local circumstances- the extent to which the provision enables and encourages learners to be healthy and stay safe- the extent to which learners have opportunities to develop enterprise, financial skills and work in teams- the extent to which enrichment activities and, where appropriate, extended services contribute to learners’ enjoyment and achievement- where appropriate, the extent to which employers’ needs are met through developing work-related skills

5c How well are learners guided and supported?

- the care, including as appropriate integrated day care, advice, guidance and other support provided to safeguard welfare, promote personal development and make good progress in their work- the quality and accessibility of information, advice and guidance to learners in relation to courses and programmes, and, where applicable, career progression- the extent to which the school and any additional services contribute to the learners’ capacity to be healthy, including vulnerable groups, such as looked after children

5e On the basis of your evaluation, what are your key priorities for development?

Quality of teaching and learningQuality of the curriculum and other activitiesQuality of care, guidance and support for learners


Curriculum Evaluation – The Stakeholders and Interested PartiesFormal & Informal Evaluations

The learner – (primarily in terms of value to self)informal but may be involved in formal (through personal reflection and value judgements about the worthwhileness of the course). Primarily qualitative

The tutor - (for monitoring programme benefits)– formative informal but may be involved in summative formal. Probably primarily qualitative


The curriculum/programme team - (through the internal verifier/moderator)Primarily formal and qualitative (standardisation of assessment and possibly quantitative (eg pass rates)

The College / Educational Organisation (through the quality unit)Primarily formal and quantitative (eg pass rates) possibly qualitative (student perceptions of course)

The Awarding Body (through the external verifier/ moderator)Eg 3 main groups (OCR; Edexel; AQA/CGLI) plus ABC Awards; OCNPrimarily formal and quantitative (eg pass rates; standards of assessment; portfolios); possibly qualitative (student perceptions of course)

Learning & Skills Council (LSC) / local LSC (LLSC)(Through OFSTED and ALI (ALI ceasing 2006)Primarily formal and quantitative (eg completion/achievement rates; retention rates); possibly qualitative (student perceptions of course gained during inspections)

DfESPrimarily formal and quantitative gained from OfStEd reports

The General PublicPrimarily formal and quantitative gained from OfStEd reports available on-line (open.gov.uk)


Kirkpatrick's Four Levels of Evaluation in Training

In Kirkpatrick's (1998) four-level model, each successive evaluation level is built on information provided by the lower level. According to this model, evaluation should always begin with level one, and then, as time and budget allows, should move sequentially through levels two, three, and four. Information from each prior level serves as a base for the next level's evaluation. Thus, each successive level represents a more precise measure of the effectiveness of the training programme, but at the same time requires a more rigorous and time-consuming analysis.

Level 1 Evaluation - Reactions

Evaluation at this level measures how participants in a training programme react to it. It attempts to answer questions regarding the participants' perceptions - Did they like it? Was the material relevant to their work? This type of evaluation is often called a ‘smilesheet.’ According to Kirkpatrick, every programme should at least be evaluated at this level to provide for the improvement of a training

programme. In addition, the participants' reactions have important consequences for learning (level two). Although a positive reaction does not guarantee learning, a negative reaction almost certainly reduces its possibility.

Level 2 Evaluation – Learning

To assess the amount of learning that has occurred due to a training programme, level two evaluations often use tests conducted before training (pretest) and after training (post test).

Assessing at this level moves the evaluation beyond learner satisfaction and attempts to assess the extent students have advanced in skills, knowledge, or attitude. Measurement at this level is more difficult and laborious than level one. Methods range from formal to informal testing to team assessment and self-assessment. If possible, participants

take the test or assessment before the training (pretest) and after training (post test) to determine the amount of learning that has occurred.


Level 3 Evaluation - Transfer

This level measures the transfer that has occurred in learners' behaviour due to the training programme. Evaluating at this level attempts to answer the question - Are the newly acquired skills, knowledge, or attitude being used in the everyday environment of the learner? For many trainers this level represents the truest assessment of a programme's effectiveness. However, measuring at this level is difficult as it is often impossible to predict when the change in behaviour will occur, and thus requires important decisions in terms of when to evaluate, how often to evaluate, and how to evaluate.

Level 4 Evaluation- Results

Level four evaluation attempts to assess training in terms of business results. In this case, sales transactions improved steadily after training for sales staff occurred in April 1997.

Frequently thought of as the bottom line, this level measures the success of the programme in terms that managers and executives can understand -increased production, improved quality, decreased costs, reduced frequency of accidents, increased sales, and even higher profits or return on investment. From a business and organizational perspective, this is the overall reason for a training programme, yet level

four results are not typically addressed. Methods for Long-Term Evaluation include:

Send post-training surveys Offer ongoing, sequenced training and coaching over a period of time Conduct follow-up needs assessment Check metrics (e.g., scrap, re-work, errors, etc.) to measure if participants achieved

training objectives Interview trainees and their managers, or their customer groups (e.g., patients, other

departmental staff)

Kirkpatrick, D. (1998) Evaluating Training Programs: The Four Levels.


Curriculum Evaluation

Comparison of the roles of the internal and external moderators/verifiers as a part of curriculum/programme quality control

Internal VerificationJob Specifications

External VerificationJob Specifications Appointed by awarding body

Occupational competence Occupational competenceNeed `V1' unit for NVQs Need V2 unit issued by awarding body

Can be marker/assessors, but not IV their own work

Not in their familiar area, conflicts of interest must be declared

Need knowledge of using current IV systems and procedures and needs of the awarding body documents and syllabus/standards of award

Attend regular meetings for currency Suitably qualified and experienced in the area

Responsibilities ResponsibilitiesProvide support, feedback and guidance for marker/assessors

Be familiar with the syllabus/standards of awards

Identify and support training needs for assessors

Reports to and receives advice from the awarding body (principal verifier or head office)

Set up and complete an IV system in accordance with the awarding body regulations

Provide support, feedback and guidance for marker/assessors

Check that assessor/markers are working to a common standard

Make approval visits to organisations wishing to become an accredited centre

Sample marked work (All learners/ units/ methods) to check standard and to give feedback to the assessors

Make visits to approved centres who wish to offer further qualifications

Hold standardisation meetings to share ideas of good practice with assessor team, information from the awarding body

Make routine verifier visits to centres to check that all aspects of the centre delivery is `safe' (H/S, equal ops, procedures, paperwork, recording systems etc)

Check that the recording of all results and enrolments are completed

Interview candidates, assessors, verifiers, (observe assessment or IV taking place in NVQs)

Check that any action plans from the EV are implemented in the timescale agreed.

Discuss ideas for improvement and make SMART action plans for the centre to work through

Ensure that the equal ops, anti-discriminatory policies are in place and upheld.

Maintain records for each centre as required by awarding body.

Maintain safe and secure storage of the students' work

Check that formative and summative assessor verification is taking place

Record and deal with any appeals from the learners using centre systems

Check dates for enrolments, certification and timescale of the, awards


Competence assessment

1 2 3 4 5Once a person is competent in an area they are always competent in that area

Competence is very short lived and needs constant practice - you can be competent one minute and not competent the next

Competence indicates the ability to perform a skill with basic levels of safety and achieving a reasonable outcome (if you pass your driving test you are therefore a competent driver)

True competence indicates a high level of creative skill which only occurs after years of constantly refined practice

There are different levels of competence - you can be more or less competent

Competence is competence - you are either competent or you are not – there are no half measures

A competent performance does not require a large amount of underpinning knowledge

A competent performance requires a large amount of underpinning knowledge

A true standard of performance which could be called competent only exists in the mind of the person assessing the performance and cannot really be communicated to the learner

A true standard of competent performance can be accurately written down and communicated to the learner

It is possible to teach a person to be totally competent at a job

True competence can be only be acquired by the individual practising the activity for themselves

Competence can be accurately assessed by a few observations

Competence can be accurately assessed by a large number of observations

To accurately assess competence you have to be extremely skilled in the activity yourself

To accurately assess competence you really only need to know what the performer has to achieve

A performance without thorough understanding of all of the possible implications cannot be seen as competent

A performance can be competently carried out even if the performer doesn't really know all of the implications

True competence requires significant individual creativity

True competence requires the performer to repeat the actions which they have been taught accurately

Within an occupational area you can be extremely competent in one job and totally incompetent in another, although the jobs may be related

To be truly competent you must be able to perform all related jobs to a high level

True competence cannot really be assessed since it requires accurate performance in all possible conditions

True competence can be easily assessed if performance criteria are determined

True competence can be demonstrated in simulated conditions if they are appropriately testing

True competence can only be demonstrated in a realistic work environment with all of the pressures that that entails

Based on The Semantic differential Osgood C & Tannenbaum P (1957) The Measurement of Meaning University of illinois press


Curriculum Evaluation - Less Reputable Evaluations

Consider the following ‘less reputable’ types of evaluation identified by Patton (1981) - do you recognise any of them? Where might these be used in education or training?

Less reputable types of evaluation

Where they might be used - and prevented!!

Quick-and-Dirty Evaluation

Weighty Evaluation

Guestimmate Evaluation

Personality-Focused Evaluation


Pseudo-evaluation’ identified by Suchman (1967)

Pseudo-evaluation Where they might be used - and prevented!!

Eyewash Evaluation

Whitewash Evaluation

Submarine Evaluation

Posture Evaluation

Postponement Evaluation


Bennet’s Seven Curriculum Issues

Thinking about evaluation

There is a saying in education –‘if it moves teach it; if it doesn’t move evaluate it’. There is a danger of evaluating for evaluation sake. Always consider the following:

1 What is the evaluation trying to achieve – what am I doing it for?

2 What would be appropriate indicators of success? If you are evaluating something then you must have some idea of a standard or benchmark which means that the thing evaluated is successful or not. If you evaluated a new car you would have ‘success criteria’ eg speed, comfort, in-car facilities, petrol consumption, visibility, price, look etc (NB consider these factors related to a car which ones are efficiency factors and which ones are effectiveness factors)

3 How should data be gathered and processed? What evidence will I gather to evaluate a lesson -–observation; discussion with teacher, discussion with learners, homework, later class work etc

4 With what can the results be validly compared? You may not expect the results of say an ‘A’ level session with students who have struggled to gain a couple of GCSEs grades C/D to be the same as the results of an ‘A’ level undertaken with learners who have gained a number of grade A GCSEs.

5 What other information is necessary to put the results into context?

6 What conclusions can legitimately be drawn?

7 What action follows?

Reference Bennet N et al (1994) Improving Educational Management through Research and Consultancy Chapman

TaskWhat do you understand by Bennet’s 7 points? Make notes on each point and think about these when you come to your programme evaluation.


Curriculum Evaluation – Gathering Data

What evidence can we use in evaluating courses and curricula?

QUANTITATIVE EVIDENCE [ so-called robust evidence]

enrolments & patterns of enrolments (gender, ethnicity, widening participation etc.) attendance & partial attendance course completion and retention achievements and public outcomes evidence of progression to further courses evidence of progression to employment three year trends for any of the above institutional comparisons for any of the above (e.g. attendance) national benchmarks with other institutional sectors (6th-form, tertiary, general FE etc)

How many of these would be useful to you in your course evaluations? Would you be able to get access to this information?

QUALITATIVE QUANTITATIVE EVIDENCE [evidence from human sources that you can turn into quantities if you chose to do so]

evidence from the learners themselves evidence from employers evidence from parents evidence from assessors [markers, examination boards, accreditation bodies] evidence from moderators, internal and external verifiers evidence from observers of teaching: formal teaching observations [including those

by inspectors, consultants and ‘critical friends’] and more informal peer observations and team-teaching

evidence from the course team and the teachers and themselves

How many of these would be useful to you in your course evaluations? Do you use any of these at present? Which, for your purposes, would you consider the most valuable?

Nevo’s 10 Major Issues in Curriculum EvaluationDavid Nevo (1986) has attempted to clarify the meaning of evaluation by identifying 10 questions that represent the 'major issues addressed by the most prominent evaluation approaches in education'

1 How is evaluation defined? Educational evaluation is a systematic description of educational objects and/or an assessment of their merit or worth.

2 What are the functions of evaluation?Educational evaluation can serve four different functions:

(a) formative (for improvement); (b) summative (for selection and accountability);(c) political (to motivate and gain public support); and (d) administrative (to exercise authority).

3 What are the objects of evaluation?Any entity can be an evaluation object. Typical evaluation objects in education are students, educational and administrative personnel, curriculum, instructional materials, programs, projects, and institutions.

4 What kinds of information should be collected regarding each object?


Four groups of variables should be considered regarding each object. They focus on

(a) the goals of the object; (b) its strategies and plans; (c) its process of implementation; and (d) its outcomes and impacts.

5 What criteria should be used to judge the merit of an object?The following criteria should be considered in judging the merit or worth of an educational object:

(a) responding to identified needs of actual and potential clients; (b) achieving national goals, ideals, or social values; (c) meeting agreed-upon standards and norms; (d) outdoing alternative objects; and (e) achieving important stated goals of the objects.

Multiple criteria should be used for any object.

6 Who should be served by an evaluation?Evaluation should serve the information needs of all actual and potential parties interested in the evaluation object ('stakeholders'). It is the responsibility of the evaluator(s) to delineate the stakeholders of an evaluation and to identify or project their information needs.

7 What is the process of doing an evaluation?Regardless of its method of enquiry, an evaluation process should include the following three activities:

(a) focusing the evaluation problem; (b) collecting and analysing empirical data; and (c) communicating findings to evaluation audiences.

There is more than one appropriate sequence for implementing these activities, and any such sequence can (and sometimes should) be repeated several times during the life span of an evaluation study.

8 What methods of enquiry should be used in evaluation?Being a complex task, evaluation needs to mobilise many alternative methods of enquiry from the behavioural sciences and related fields of study and utilise them according to the nature of a specific evaluation problem. At the present state of the art, an a priori preference for any specific method of enquiry is not warranted.

9 Who should do evaluation?Evaluation should be conducted by individuals or teams possessing

(a) extensive competencies in research methodology and other data analysis techniques; (b) understanding of the social context and the unique substance of the evaluation

object; (c) the ability to maintain correct human relations and to develop rapport with individuals

and groups involved in the evaluation; and (d) a conceptual framework to integrate the above-mentioned capabilities.

10 By what standards should evaluation be judged?Evaluation should strike for an optimal balance in meeting standards of

(a) utility (to be useful and practical); (b) accuracy (to be technically adequate); (c) feasibility (to be realistic and prudent); and (d) propriety (to be conducted legally and ethically).


Empiricist v Rationalist approaches to the curriculum

For centuries and throughout the world, education has been dominated by either the Empiricist tradition (ie physically – or empirically - observing by the senses) or the Rationalist tradition(recognising humanities intellectual abilities which may not by observable).

England & the USA have had a strong empiricist tradition (eg Hume & Dewey) and curricula have tended towards the empiricist idea with evidence for achievement being extremely important (for examples portfolios of evidence in NVQs

Europe has tended towards the rationalist tradition (eg Voltaire) and therefore baccalaureate type curricula with value placed upon reasoned argument have been strong Consider a curriculum with which you are involved. Plot it against each of the empiricist-rationalist criteria by ticking a box.

The Empiricist Tradition1 2 3 4 5

The Rationalist Tradition

Proof by evidence of the senses eg portfolios of evidence!

Proof by logic & understanding eg rationally argued essays!

Behaviourist Principles [lessons based on objectives and outcomes!]

Gestalt Principles linking ideas to form patterns[experiential reflective learning]

Constant Reinforcement[Extrinsic motivations!]

Stimulus to gain Insight ‘Eureka’[intrinsic motivations!]

Mastery Learning[Competence assessment]

Conceptual Learning [assessment of understanding]

Ability demonstrated by performance

Ability demonstrated by explanation/ critical evaluation

Motivation assumed to be External (Extrinsic)

Motivation assumed to be Internal (Intrinsic)

Lots of direct instruction (the teacher as expert informing the unknowing learners)

student centred learning (learners creating their own understanding

NVQ & GNVQs /key skills type programmes

Academic/recreational type programmes

Lots of Internal and external verification of assessor’s marking

Internal and external examinations of students’ understanding


Change - the five categories of product adopters:

Innovators – venturesome, educated, multiple info sources; Early adopters – social leaders, popular, educated; Early majority – deliberate, many informal social contacts; Late majority – skeptical, traditional, lower socio-economic status; Laggards – neighbours and friends are main info sources, fear of debt.

Rogers E (1962) Diffusion of innovations

Rogers stated that adopters of any new innovation or idea could be categorized asinnovators (2.5%),early adopters (13.5%), early majority (34%), late majority (34%) laggards (16%)

Reading List

Bennet N, Glatter R and Levacic R (1994) Improving Educational Management through Research and Consultancy Chapman

Hopkins, D. (1993) A Teacher’s Guide to classroom Research OUPress

Kirkpatrick, D. (1998) Evaluating Training Programs: The Four Levels. 2nd ed.

Osgood, C. & Tannenbaum, P. (1957) The Measurement of Meaning University of illinois press

Stenhouse, L. (1975) An Introduction to Curriculum Research and Development Heinemann

Stufflebeam, D. (1971). Educational Evaluation and Decision Making

Stufflebeam, D.L. Madaus, G. F. & Kellaghan, T. (Eds.), (2000) Evaluation models ( 2nd ed.). Boston: Kluwer Academic Publishers.


http://en.wikipedia.org/wiki/Early_adopter

http://en.wikipedia.org/wiki/Diffusion_of_innovations

http://en.wikipedia.org/wiki/Everett_Rogers

university of exetersocialsciences.exeter.ac.uk/media/universityofexeter/colle… · web...

Documents