from exams to school- based assessment: opportunities and challenges gordon stanley, phd honorary...

From Exams to School-based Assessment:Opportunities and

Challenges

Gordon Stanley, PhDHonorary Professor of Education

The University of SydneyEmeritus Professor of Psychology

The University of Melbourne

Global reform trendHow we collect data about student learning and

achievement is at the centre of education reform.

Are our practices fit for the national purpose of human development?

Traditional assessment focused on sorting function.

Over-reliance on external system wide tests and exams.

Traditional school-based practiceTeachers give quiz or test to chart progress of

students.

Teachers mark and grade performance using implicit standards based on experience.

Often marks given based on relative performance rather than with alignment to common standards.

Standards-based assessment Reporting required to be based on credible

evidence about developmental progress and achievement.

Reported with reference to a series of standards or verbal descriptions of what students ‘know and can do.’

Not sufficient to say: ‘You are an ‘A’ student”.

Comprehensively analyses the key features of specific periods of twentieth century history and evaluates the role of key individuals, groups, events and ideas.

Evaluates the relative significance of factors contributing to change and continuity in the modern world. Displays a sophisticated understanding of historical terms and concepts.

Utilises a variety of relevant historical sources and evaluates their reliability. Assesses different historical interpretations and perspectives.

Communicates high level argument through well-structured and detailed texts.

Modern History ‘A’

Giving meaning to the standardThe verbal description helps to create an

‘image’ of student capability.

A clearer image emerges when examples of student work typical of a student who has achieved an ‘A’ grade are available.

A standards-based curriculum and reporting along a clear developmental (growth) pathway allows new thinking about how best to collect evidence.

The assessment for learning movement has highlighted the way in which assessment and pedagogy are interconnected in classroom activities.

Recognition of the opportunities in school-based assessment also raises challenges about how it is to be managed

The process of assessmentAssessment involves professional judgment

based upon an image formed by the collection of information about student performance.

This definition draws attention to two important facts.

Fact oneAll assessment involves professional judgment

i.e. judgment based on deep understanding of the curriculum and reporting standards and the images of student performance associated with progress along the developmental pathway described by these standards.

External exams and so-called ‘objective tests’ require a considerable amount of professional judgment in their design, construction and interpretation.

Fact twoAssessment is based on collection of information,

which can range from unstructured classroom observations and short quizzes to standardized tests and exams.

This information provides the judge, be it the teacher or examiner, with an on-balance picture (the ‘image’) of where the student is with respect to the developmental pathway.

Importantly all this information should be scrutinized for any inconsistency. All sources of information provide a window on development of the individual.

Two questionsHow can we ensure that professional judgment

with respect to standards is consistent across an education system?

How much information do we need to collect?

No easy answersAssessment cannot be a precise activity. What

we must ensure is that our processes are as reliable and fair as we can make them given the constraints of time and effort available.

We need to avoid being over-zealous in the collection of information.

Information needs to be robust enough for the task at hand.

Design requirementA major design requirement for quality

assessment, whether involving external tests or school-based assessment, is the need to ensure alignment to curriculum and syllabus goals and expected outcomes.

Alignment helps define the standards.

We need assessment to report developmental progress towards and/or attainment of the curriculum outcomes

External exams

Role of external examsExternal exams are common in Commonwealth

and European countries, especially at the point of graduation from school and for entry to college and university.

They are used for certifying student achievement and provide the basis for competitive selection.

Students and their teachers, as well as tutors and coaching colleges, all take an intense interest in the process and outcomes

Traditional external exam scene

Strength of external examsDeveloped by exam authorities who have

subject specialists produce the papers and ensure their security until students sit the same exam at the same time and under common conditions of supervision.

As all students respond to a common set of questions at the same time it is possible for a direct comparison of results.

Student responses are marked by trained markers in a process independent from the school in which the student has been taught.

Commonly exam authorities are subject to quality audits and generally have high public confidence.

Weaknesses of external examsThe time available as well as the length of

external exams is limited by cost factors.

Samples of domain knowledge and skills need to be made, thereby limiting coverage of the curriculum in the exam.

Additionally there are often significant parts of the curriculum, which cannot be assessed by traditional exams

Pressure is placed on examination authorities to avoid ‘surprise’ questions based on fine detail and to ensure that questions are ‘accessible’ to all students.

As a result papers are often fairly predictable. With high stakes being associated with public exams teachers and tutors put a lot of effort into preparing students for particular types of exam questions.

‘Though fairly reliable tests of narrow textbook content, Indian school board exams are rarely valid tests of desired competencies and broader curricular objectives, even within the cognitive domain.’

(From National Council of Educational Research and Training (2006) Position Paper: National Focus Group on Examination Reforms, p 7.)

Similar comments can be made about exams in a number of Commonwealth countries.

Weakness of only external exams.High stakes leads to tutoring or coaching

industry.

Focus on ‘teaching to the test’.

One-shot approach can lead to considerable stress.

Sometimes performance can be affected by chance factors associated with lucky choice of exam preparation.

Thus the external exam may be measuring a personal best due to last minute coaching and lucky breaks rather than representing a stable achievement.

In other cases test anxiety or tiredness may make the student performance less than expected.

Stress reducers

CheatingThe high stakes with external exams have often

lead to students seeking ways to get an advantage by having unauthorised material available to them.

Most authorities have had to develop procedures to minimise such behaviour but there are always new attempts being made.

Need for reliable informationIf we wish to be confident about the assessment

of knowledge and skills attained we need to have reliable information.

More information available should increase the reliability of our assessment as well as the validity of our judgment.

Hence assessment can be enhanced considerably through incorporating school-based assessment into the process.

School-based assessmentA renewed interest in school-based assessment

recognises its everyday role as part of the teaching/learning process.

Teachers use a variety of ways to get an understanding of their students and to provide feedback and guidance to improve learning.

As a critical source of information, school-based assessment provides many opportunities, but also poses many challenges to education systems as they manage change.

Today in most public policy goals for modern education there is concern about evidence of depth of understanding, and evidence of 21st century and employability skills.

The traditional external exams using pencil and paper tests relying on essays and multiple-choice questions are thought to be inadequate by themselves for these current demands.

Laboratory skills can be assessed effectively by observing students conducting practical work in the laboratory.

Research skills may be best assessed by having the student carry out a project and defending the outcome in the classroom.

Working collaboratively is best assessed through observation of the students at work in groups.

Observing, questioning and interviewing are not typically incorporated in current external exams, but are part of everyday classroom activity.

School-based assessment in high stakes exams

Many countries have responded to the criticisms levelled at the traditional one-shot external exams by incorporating school-based assessment into their high stakes exam systems.

Some have given equal weighting to school-based and external exam results in determining the level of achievement.

Other systems have not found the journey of implementation completely successful and consequently have given a lower weighting to such assessment.

It cannot be assumed that school-based assessment necessarily avoids problems associated with external testing without some assurance of consistent professional standards for effective assessment by classroom teachers.

Teachers may teach a narrow curriculum by selecting their favourite bits for emphasis and may employ classroom assessments, which create similar pressures to external exams.

Problems with high stakes assessment

It may be in the nature of national and system level assessment that such corruption of the intent of curriculum delivery occurs, especially when student results are used for performance management of teachers and schools.

To counteract such trends requires a strong professional ethic and system level investment in developing and maintaining high quality teaching and assessment regimes.

Need for teacher assessment skills

School systems need to support processes to develop assessment competence in teachers, a task that is often not maintained effectively beyond an initial implementation cycle.

The professional status of teachers is enhanced when assessment occurs at the level of the classroom.

Teacher involvement in assessment moderation and standard setting are invaluable in helping them to assign performance levels correctly according to national standards.

Requirements for school-based assessment

For effective school-based assessment in an education system there are a number of issues that need to be addressed, ranging from workload to ensuring consistent and quality engagement by teachers and students.

Comparing education systems in different countries one can see common challenges emerging.

There is agreement that professional development and moderation of practice are vital to gain acceptance and fairness.

Manageability and evidence records

Tensions often arise between the activities performed by teachers in embedding assessment into their daily teaching and the requirements for quality assessment.

Different classroom teaching and learning situations provide varying opportunities to observe and record information to inform judgments about student achievement.

The amount of evidence collected and how it is recorded is a significant issue.

Practices differ in the extent of data collection and recording ranging from detailed assessment protocols to ‘on-balance’ judgments of attainment of assessment criteria.

Tradeoffs need to be made with respect to the degree of task standardization, the number of tasks on which assessment is based, the degree of recording and the storage of information to justify the teacher judgment and the impact on teachers’ workloads.

Since a prime focus is to provide an opportunity to make judgments about student progress and to guide the next step in learning it is important that the assessment process is manageable in terms of workload and provides timely feedback

EffectivenessThe effectiveness of school-based assessment

models needs to be evaluated in the policy context in which they are operating so that effectiveness can be measured against the purpose/s required by the education system.

Much of the literature on the reliability or validity of assessment by teachers has been based on relatively small scale studies embedded in contexts which may not fit well into a system wide reporting and accountability framework.

Where teacher training in assessment has been strongly embedded school-based assessments have found to be more reliable than when they are not.

It is reasonable to expect student’s levels of skill to develop over time as a function of instruction and practice and the rate of such development may vary significantly across individuals.

Usually exam authorities advise schools to weight school assessment components closer to the summative external exam higher than earlier assessed components.

Alignment to standards

MacCann and Stanley (2010) compared school assessment grades in Yr 10 science with grades on an external test for science.

The school assessment programme was reported with respect to performance descriptors for five levels as was the external test.

The test performance descriptors are similar to those for school reporting though somewhat more narrowly focused on scientific literacy.

The alignment of scores on school assessment to grade descriptors was made at school level and submitted to the examination board whereas the external test score alignment to grades was made centrally by a small team of experienced judges.

With an annual candidature of around 80,000 one might expect reasonably stable results.

% at Grade ‘A’

% at Grade ‘C’

Natural tendency for the classroom teacher to see their students in the best possible light, despite the training received in aligning portfolios to the system-wide standards.

Clearly there is a need for external moderation of some sort, depending on the degree to which the assessments based on classroom teacher judgments are high stakes.

Moderation of school-based assessment

To ensure comparable outcomes across schools with school-based assessment a moderation process is usually invoked.

The accuracy required of a given moderation process bears a practical relationship to the degree to which the measures are high stakes.

Programmes which determine tertiary entrance are based most commonly on both public examinations and moderated school assessments.

Having external examinations reassures the public concern about fair comparison across schools.

The combination of external plus moderated internal results improves both the reliability and validity of the scores or grades.

Moderation of the internal results involves making sure that the school assessments are consistent with system-wide reporting in relation to standards.

Moderation can involve a statistical process or a social (consensus) process.

Statistical moderation

In statistical moderation the criterion for moderation is either an external test or public exam that is attempted by the students in addition to being assessed within the school.

When the distribution of criterion marks obtained by the school group on the external exam has a similar distribution shape to that of the school assessments, it is common to use a linear conversion line.

The main advantage of statistical moderation is its transparency and reproducibility.

Social (consensus) moderation

Social moderation involves a number of procedures for aligning school assessments based on professional judgment.

School assessments are compared with work samples and compared across different schools by referring them to a standards-referenced scale.

Work samples are commonly in portfolios of student work that are assigned to a certain performance level by the school.

These are then referred to external moderators who determine whether the judgments are accurate or too lenient or too harsh.

Comments on social moderation

For high stakes competitive assessments most systems rely on external testing to moderate or monitor the assessments rather than social moderation.

Using external exams to moderate has public confidence because of external marking and the identity of the students not being known to the markers.

It is cost effective because it does not require extensive deployment of moderators and can be accomplished in a shorter time-frame

ConclusionSchool-based assessment to be respected in a

standards world has to demonstrate that it is based on evidence that is consistent with appropriate alignment to the curriculum standards expected by end-users.

School-based assessment ensures that evidence about student performance is well-grounded and supported by the teaching and learning programme at school level.

For quality assessment there needs to be real investment in developing assessment competency and ensuring common standards are being implemented.

In India the Continuous and Comprehensive Evaluation (CCE) programme implemented by CBSE provides an excellent example of a system level approach to implementing school-based reform.

The programme of assessment training developed by the Centre for Assessment, Evaluation, and Research will provide support for schools and teachers to build confidence that their assessment will assist student learning and progression towards each student achieving their potential.

Thank you for listening

Email: gordon.stanley@sydney.edu.au