lesson 5a evaluation basic concepts

8/14/2019 Lesson 5a Evaluation Basic Concepts

1/25

EVALUATION

Basic Concepts

Sylvester Saimon Simin

Keningau Teachers Training College


2/25

2

You should be able to.

T & L Resources:

Kementerian Pendidikan

Malaysia (1995d)

Past year PMR and SPM

Examination papers

Values:

To be aware of the

importance of planningand preparation before

administering a test.

To be aware of the

accountability of the test.

Prepare a test blue print

based on KBSM Science

Syllabus

Construct objective,

structure, and essay

questions based on the

test blue printPrepare marking schemes

for the above questions

Study and analyze the

format of the examination

papers on the followingaspects: distribution of

multiple choice, structure

and essay questions.

Analyze each MCQ and

classify it based on

Blooms Taxonomy

6. Evaluation

6.1 Test Blue Print and

Question

Construction (10

hours)

6.2 Centralised

examinations (PMRand SPM)

Format of PMR and

SPM examination

papers

Values / RemarksSkillsKnowledge


3/25

3

I Have a Dream About Assessment

Roger Farr

I have a dream that assessment... ...will be accepted as a means to help teachers plan

instruction rather than as a contrivance to force teachers tojump through hoops;

...will be based on trust in a teachers judgment as much asnumbers on a page are trusted;

I have a dream that assessment... ...will become a helpful means to guide children to identify

their own literacy strengths rather than a means toconveniently label them;

...will support each child in becoming the best he or she canbe rather thana means to sort children into groups of the best and theworst;

And I have a dream that assessment... ...will be put to use to honor what children can do rather

than destroying them for what they cant do.

If we all work together we can make such dreams

become a reality as we work to help each child grow.


4/25

4


5/25

5

Purpose of Evaluation

to determine the students achievement of certainknowledge and skills as specified by the syllabus of thesubject

to measure students progress over time,

to rank students in terms of their achievement, to diagnose the main difficulties faced by the studentsin the areas of study,

to determine how effective are the teachersinstructional strategies,

to determine the effectiveness of the curriculum, itsstrengths and weaknesses, to encourage good study habits, to motivate students


6/25

6

Assessment Terms

Performance Assessments

Assessment requiring students to

demonstrate their acheivement of

understandings and skills by actuallyperforming a task or set of tasks (eg. Writing a

story, giving a speech, conducting an

experiment, operating a machine)


7/25

7

Assessment Terms

Alternative Assessment

A title for performance assessments that

emphasizes that these assessment methods

provide an alternative to traditional paper-and-pencil testing.


8/25

8

Assessment Terms

Authentic Assessment

A title for performance assessments that

stresses the importance of focusing on the

application of understandings and skills to realproblems in real-world contextual settings


9/25

9

TERM WORKING DEFINITION

accreditation The official endorsement of the procedures and/or standards of aninstitution by an authority. For example, an examination board may accredita center for the assessment of course work.

aim (educational aim) A long-term goal which may or may not be achievable within the teachingprogram.

appeal A challenge by a candidate or a school to the results awarded by anexamining authority.

assessment General term used for the 'measurement' of a behavior or characteristic

assessment component One part of an assessment package - e.g., a written paper, a practical test,an oral exam, a piece of coursework.

assessment objective A statement of an expected learning outcome which will be assessed.

assessment package The total assessment scheme which may be composed of one or morecomponents

aural examination Listening test (not to be confused with an 'oral test' i.e., a test of speaking.)

backwash effect

(occasionally 'washbackeffect')

The effect (positive or negative) of the scheme of assessment on the

teaching/ learning program which precedes it.

bias Tendency of a test, or an item, to place one group at an advantage overanother on the basis of a factor (e.g., gender, ethnicity, language) otherthan that which the test purports to assess.

camera-ready copy (CRC) Final proof of an examination paper as it will appear, after printing, on thecandidate's desk.

centralized marking Administrative arrangement where all answer scripts are brought to acentral location for marking. Where markers remain at the centerthroughout the marking period, this may be referred to as 'residentialmarking'.

certification Use of examination results to provide individuals with documentaryevidence of achievement (i.e., a certificate).

classical item statistics Statistics describing the behavior of a test item (typically its level ofdifficulty and its discriminatory power) by analysis the responses of aparticular group of test-takers. Note that such statistics are dependent onthe group taking the test. (See also IRT).

coaching Special preparation of candidates for an examination typically by practicingthe techniques of test taking, rote learning of past questions and answers,'question spotting' etc.

code of practice Set of guidelines and/or regulations controlling the procedures ofassessment authorities in the conduct of public examinations. Where

examination bodies have constitutional autonomy, this may have to be avoluntary code of practice.


10/25

10

curriculum All educational aspects of an institution and its teaching programs -including non-examined subjects

cut-off point Test score at which students are deemed successful (and below whichthey are deemed unsuccessful). See also grade threshold.

double marking Procedure in which answer scripts are independently scored by two raters.Where there is a discrepancy between scores, set procedures apply forreaching the final score. Typically these include averaging small

differences and using an 'expert marker' as an arbiter where differencesare large.

end-users Individuals or institutions who use examination results for their ownpurposes e.g., universities, schools, employers.

equity An equitable examination ensures that all students who possess the samedegree of ability receive the same result. Where there are inequities, anindividual or group gains an unfair advantage over others. It follows thatinequity places some individuals and/or groups at a disadvantage due tofactors other than the ability that the examination purports to assess.

evaluation Assessment for the purpose of making a value judgment, e.g., to judge the

effectiveness of a teaching program

examination center Place officially recognized for the conduct of examinations. Typicallycenters are state schools, private schools, university halls or privatebuildings hired for examination purposes.

feedback The systematic flow of information gained from an assessment toeducationists, policy makers, and others e.g., examiner reports forteachers.

formative assessment Assessment which takes place as an integral part of the teaching-learningprogram (see also summative assessment).

grade threshold Test score between two reporting grades. For example, if the A-gradethreshold is 81%, students scoring 80% will be awarded grade B and thosescoring 81%, grade A.

group certificate Examination system which requires candidates to take a prescribednumber and combination of subjects. The award of the certificate isdependent on the candidate meeting pre-determined criteria for success.


11/25

11

high-stakes examination An examination where students, parents and teachers invest a great dealof effort, and perhaps money, in preparing because success can potentiallybring great rewards whilst failure may damage the candidate's life-chances.

impersonation Form of malpractice where someone takes an examination in place of theregistered candidate.

invigilator Person who supervises and is responsible for the conduct of anexamination in a particular examination room/hall.

IRT Item Response Theory (sometimes IRM - Item Response Modeling).

Psychometric tool which, in its simplest form, uses a mathematical modelto link a student's chance of being successful on an item with the student'sabilityand the item's difficulty. This allows items to be calibrated on anabsolute measurement scale.

item bank A collection of items categorized according to their characteristics e.g., typeof item, topic, skill being assessed, level of difficulty, etc. Items are thendrawn from the bank to build a test according to predetermined testspecifications.

league table A table which ranks schools on the basis of examination results and otherindicators (see also 'value added').

leakage Unauthorized release of examination materials and/or information prior to

the official release date.

localization Where an independent country takes responsibility for the maintenanceand further development of an examination system introduced by a formercolonial authority.

malpractice Any deliberate act of wrongdoing, contrary to the rules of the examination,designed to give a candidate an unfair advantage or, albeit less frequently,to place a candidate at a disadvantage.

marker One who marks/scores candidate responses (also rater).

marking scheme Instructions as to how marks are to be allocated to student responses(answers). These may be detailed for objective and semi-objective tasks.

For open-ended and subjective tasks, they may take the form of generaldescriptions ('band descriptors').

measurement An assessment made using the concept of a well-defined ability scale to

quantify a behavior or characteristic e.g. mathematical ability.

moderation General term used by examining authorities for the process of checkingquality. Question paper moderation typically involves the review of draftquestion papers by an expert panel. Moderation of school-basedassessmentmay involve a Board representative visiting the school to lookat work and interview teachers and students. Alternatively, samples ofstudent work may be sent for review by a Board moderator.

National Assessment Assessment designed to determine national standards usually conducted

using a representative sample of students.


12/25

12

objective item Item that can be scored without the marker making a personal judgment asto the quality of the response e.g., multiple-choice.

OMR Optical Mark Reader - scanning device for reading marks from specialforms thereby allowing the automatic input of student responses to, forexample, multiple-choice question papers.

parastatal Term applied, especially in Africa, to an organization established by agovernment but which, through its constitution and budgetaryarrangements, enjoys a great degree of operational freedom and insulationfrom direct political interference.

pedagogy The science of teaching including both theory and practice.

private candidate Candidate who enters, and pays for, his/her own entry to a publicexamination as compared with a candidate who is entered by the institution(school) in which he/she is studying and which is recognized by theexamining authority as an authorized center.

psychometry(psychometrics)

Field concerned with the measurement, and hence quantification, of humanbehaviors and characteristics. Psychometric strategies are built onstatistical models of measurement and human behavior.

public examination An examination offered by a national or provincial (state) authority, or onbehalf of such an authority, to students at a particular level of an educationsystem. The primary purpose is to certify the level of achievement ofindividual students and/or to select students for the next level of theeducation system.

quota system Form of selection system where the share of available opportunities to beawarded to a particular group is pre-determined. For example, in order toensure gender balance in a selective secondary school system, 50% ofplaces may be awarded to boys and 50% to girls. As a consequence, someboys may be selected with lower examination scores than those achievedby girls who are rejected (or vice versa).

rater One who marks/scores candidate responses - a marker.

registration Key process whereby the details of individuals (students) are entered intothe administrative database as candidates for forthcoming examinations.

regular candidate Term used, particularly in the Asian sub-continent, for candidatesregistering through recognized centers for a series of examinations for thefirst time. Private candidates and those re-sitting examinations areconsidered irregular.

reliability A measure of the stability of the results produced by an examination. Thisincludes the stability of scores on re-testing, the stability of scores with re-marking, and the correlation of scores for sub-sections within the test(homogeneity).

school-based assessment Any assessment of student performance which takes place in a school andis incorporated into the public examination result. Note that the degree offreedom allowed to the school will depend on the regulations and

moderation procedures of the examining authority.


13/25

13

script General term for an answer booklet or sheets produced by a candidate inresponse to an assessment task.

selection Use of examination results to select individuals for educational oremployment opportunities where the number of such opportunities islimited. In many developing countries, examination results are used toselect students for the next phase of education e.g. primary-secondary,lower secondary-higher secondary, secondary-tertiary.

specification grid A plan or 'blueprint' giving the format of a question paper or otherassessment component.

stakes (of an examination) The importance of an examination as judged by what may be gainedthrough success - and what may be lost through failure. Therefore, a 'high-stakes' examination will typically be highly competitive because thesuccessful will enjoy greatly enhanced opportunities.

structured question Task composed of a number of sub-questions (items) linked by a commoncontext or piece of stimulus material. The sub-questions may beindependent of each other or may be sequenced to lead candidatesthrough a more complex task (progressive).

subjective item Item that requires the marker (rater) to make a personal judgment as to thequality of the response e.g. the literary merits of an essay or the artisticmerits of a painting. Note that in order to minimize variation, raterjudgments may be guided and constrained by marking schemes and

descriptors of performance.

summative assessment Assessment which takes place at the end of the teaching-learning programto record 'final achievement' (see also formative assessment).

supplementary examination A follow-up examination allowing students to retake subjects in which theyhave not reached the required level. This issue is of particular importancein systems awarding group certificates.

syllabus (examinationsyllabus)

A document formally specifying what will be assessed by the examinationand how the assessment will be carried out.

tamper-evident packaging Plastic envelopes for examination materials which cannot be resealedwithout showing obvious signs of being opened.

teaching objective(curriculum objective)

A specific short-term goal of the teaching program.

teaching program The program of instruction.

teaching/learning program The instruction delivered by a teacher coupled with the learning that takesplace during the program.

transparency Extent to which the processes involved in the examination system arevisible to the public - especially schools, teachers and students.

validity A measure of the extent to which an examination measures what it

purports to measure.


14/25

14

Reflective practice in which students make observations about the

performance of their peers

Peer assessment

A specific statement of intentObjective

Judgement of performance against the norm for the groupNorm-referenced

A general statement which describes an observable result by which a

student demonstrates knowledge, skill or attitude

Learning outcome

The measure of student growthIpsative assessment

Assigning a letter, percentage or scoreGrading

Ongoing feedback on a student=s performance throughout the learning

process

Formative

assessment

The application of judgement to the data in the form of a grade orcomment, placing a value on that work

Evaluation

Determines the level of achievement/performance prior to entering aDiagnostic

assessment

Judgement of performance against a previously agreed standardCriterion-referenced

The standard against which performance is measuredCriterion

The gathering of data about students or program, often used as a formative

process to guide instruction

Assessment

Any and all assessments that differ from multiple choice, one word answer,

timed items that characterize standard tests

Alternative

assessment

A demonstration of learning at a particular moment in timeAchievement


15/25

15

A report on the final achievement -- given at the end of a unit or work or

semester or year

Summative

Evaluation

A point of reference against which judgements can be madeStandard

Reflective practice in which students make observations about their ownperformance

Self-referenced

Reflective practice in which students make observations about their own

performanceSelf-assessment

A set of quality criteriaRubric

Communicating process or achievement to the student or his/her parents

or guardianReporting

Focuses on the end product of a learning processProduct

assessment

Focuses on the variety of strategies, thinking skills and processes that a

student uses to complete a task

Process

assessment

The assessment of a representative collection of a students work over timePortfolio

assessment

Usually an alternate or authentic assessment, where a student completes a

relevant task which demonstrates learning by using or applying

knowledge

Performance

assessment


16/25

16

What is performance assessment?

A performance assessment is an assessment

activity that requires students to construct aresponse, create a product or demonstrate askill they have acquired. Rubrics, based on theselected criteria, are given to students to ensurethat they know what they need to do to meet or

exceed the learner outcomes. Well-constructed performance assessments:

are the most authentic types of assessment sincethey replicate out of school experiences, encourageself-evaluation and demonstrate what students knowand can do;

put students in a role (e.g. scientist, newspapereditor) and provide an audience for their task

provide degrees of proficiency based on criteria andmake public the criteria.


17/25

17

A few things to know

Blooms taxonomy

Difference between;

Testing, measurement, evaluation objective & subjective items

formative & summative evaluation

critrion reference test & norm reference test

Validity & Reliability


18/25

18


19/25

19


20/25

20


21/25

21


22/25

22

The Assessment Process

Preparation (including Test / Task Blueprint) Determine the kind of information needed and

decide how and when to obtain it.

2. Information gathering

Obtain a variety of information as accurately aspossible

3. Forming judgements Judgements are made by comparing the

information to selected criteria.

4. Decision making and reporting Record significant findings and determine

appropriate courses of action.


23/25

23

INFORMATION GATHERING

Information gathering techniques

Procedure for obtaining information Inquiry (asking), observations (senses), analysis

(performance, product), testing (common situation to whichall students respond,common set of instructions governingresponse, set of rules for scoring responses & descriptionof performance ie score)

Information gathering instrument Tools used to gather information

3 basic types : tests, rubrics and questionnaires Teacher made test / classroom tests vs standardized tests

Rubric : set of rules for scoring student products orperformance. Typically take the form of a checklist or a ratingscale

Questionnaires : useful for getting opinions, feelings andinterests


24/25

24

Information Gathering Techniques

Most expensive,

but most

information gained

per unit of time

Fairly

inexpensive

Preparation

time is

somewhat

lenghty but

crucial

Inexpensive but

time-consuming

Inexpensive but can

be time consuming

cost

Most objective

and reliable

Objective but

not stable over

time

Subjective, but can

be objective if care

is taken in the

construction and

use of the

instruments

Least objective

Highly subject to

bias and errorObjectivity

Attitude and

acheivement

Terminal goals

Cognitive

outcomes

Maximum

performance

Learning

outcomes during

the learning

process

(intermediate

goals)

Cognitive and

psychomotor

skills

Some affectiveoutcomes

Performance or

end products of

some performance

Affective

(especially

emotional

reactions)

Social interaction

Psychomotor skills

Typical behavior

Opinions

Self-perceptionsSubjective

judgements

Affective (especially

attitudes)

Social perceptions

Kind of

information

obtainable

testinganalysisobservationinquiry


25/25

25

Information Gathering Instrument

Take time and effort to

construct. Difficult to score.

No right or wrong answers.

Data difficult to summarize.

Keep inquiry focused and

help teacher obtain the

same information from

each student.

To inquire about

feelings, opinions,

and interestsQuestionnaires

Take time and effort to

construct. Can be clumsy to

use if too complex.

Allow observational data

to be used in making

qualitative as well as

quantitative judgements

Rating Scales

To judge quality of

performance

Measure only presence or

absence of a trait or behavior.

Helpful in keeping

observations focused on

key points or critical

behaviors.

Checklists

To determine the

presence or absence

of specific

charateristics of

performance

Rubrics

To assess the

quality of

studentperformance

No norms beyond the class are

available. Often unreliable.

Require quite a bit of time to

construct.

Usually measure exactly

what has been taught.

Inexpensive. Can be

constructed as need

arises.

Routinely as a way

to obtain

achievement

information

Teacher made

test

Often not measuring exactly

what had been taught.

Expensive. Limited in what ismeasured.

Usually well developed

and reliable. Include

norms for comparing theperformance of a class or

an individual

when accurate

information is

neededStandardized

test

DisadvantageAdvantageUsedType

lesson 5a evaluation basic concepts

Documents