performance assessment osi workshop june 25 â€“ 27, 2003 yerevan, armenia
Post on 16-Jan-2016
Embed Size (px)
DESCRIPTIONPerformance Assessment OSI Workshop June 25 – 27, 2003 Yerevan, Armenia. Ara Tekian, PhD, MHPE University of Illinois at Chicago. Presentation Outline. Characteristics, types, strengths, and limitations Five factors to consider when making performance assessment Checklists and rating scales - PowerPoint PPT Presentation
OSI WorkshopJune 25 27, 2003Yerevan, ArmeniaAra Tekian, PhD, MHPE University of Illinois at Chicago
Presentation OutlineCharacteristics, types, strengths, and limitationsFive factors to consider when making performance assessmentChecklists and rating scalesPortfoliosVideo presentation & exercise (workshop)
Performance Assessment (PA)PA can be distinguished from the traditional paper-and-pencil test by a number of characteristics:Greater realism of tasksGreater complexity of tasksGreater time needed for assessmentGreater use of judgment in scoringLinn & Gronlund, 1995
Performance tasks appear in many forms:Solving realistic problemsOral or psychomotor skills without a productWriting or psychomotor skills with a product
(Various types of performance may be restricted or extended)
Restricted PerformanceHighly structured and limited in scopeExamples:Write a one-page reportGive a one minute speechConstruct a graph form a given set of dataDemonstrate how to sep up lab equipment
Extended PerformanceLess structured and broad in scopeExamples:Design and conduct an experiment on a selected topic, present and defend the findingsTake the history of a patient, perform a physical examination, and diagnose and write a management plan.
Performance Assessment - StrengthsCan evaluate complex learning outcomes and skillsProvides a more natural, direct, and complete evaluation of some types of reasoning, oral, and physical skillsProvides greater motivation for students by clarifying goals and making learning more meaningfulEncourages the application of learning to real life situations
Performance Assessment - LimitationsRequires considerable time and effort to useJudgment and scoring performance is subjective, burdensome, and typically has low reliabilityEvaluation must frequently be done individually, rather than in groups
Need for Perf. AssessmentThe shift from norm-referenced measurement to criterion-referenced measurementThe need for focusing on more complex learning outcomes (reasoning and thinking skills), on using more comprehensive student projects based on real life problems, and on engaging students in the activities and in the construction of meaning from them
Main factors to consider when making PA1. Specifying the performance outcome2. Selecting the focus of the assessment (procedure, product, or both)3. Selecting an appropriate degree of realism4. Selecting the performance situation5. Selecting the method of observing, recording, and scoring
1. Specifying the performance outcomeE.g. A research project might include intended learning outcomes as follows:Selects an appropriate research taskDesigns and conducts an experimentStates valid conclusionsWrites a critique of the procedure of findings
Typical Action Verbs for restricted performance outcomesIdentify, locate, select, describeConstruct, design, draw, prepareDemonstrate, measure, perform, set up
2. Selecting the focus of the assessmentPerformance assessment can focus on the procedure, the product, or some combination of the two. The nature of the performance frequently dictates where the emphasis should be placed.
Assessing the ProcedureFocus the PA on the procedure when:There is no product, or product evaluation is infeasible (e.g., unavailable or too costly)The procedure is orderly and directly observableCorrect procedure is crucial to later successAnalysis of procedural steps can aid in improving a product
Assessing the ProductPA should be focused on the product when:Different procedures can result in an equally good productThe procedure is not available for observationThe procedural steps have been masteredThe product has qualities that can be clearly identified and judged
3. Selecting an appropriate degree of realismAlthough we cannot expect the duplicate the natural situation in which the learning will later be used, we can strive for performance assessments that approximate real world conditions.
4. Selecting the Performance SituationPA can be classified by the type of situation or setting used:Paper-and-pencil performanceIdentification testStructured performance testSimulated performanceWork sampleExtended research project
5. Selecting the Method of Observing, Recording, and ScoringWhether judging procedures, products, or some combination of the two, some type of guided observation, method of recording and scoring the results is needed.Systematic observation and anecdotal recordsChecklistsRating scales
Systematic observation and anecdotal recordsObservations are frequently guided by checklists or rating scales.An anecdotal record is a brief description of some significant event. It typically includes the observed behavior, the setting in which it occurred, and a separate interpretation of the event.
Anecdotal RecordsAre likely to be most useful when:They focus on meaningful incidentsThey are recorded soon after the incidentThey contain enough information to be understandable laterThe observed incident and its interpretation are kept separate
Checklists (Advantages)List of performance for which raters indicate presence or absence of what they have observedTwo purposes:Descriptors direct the attention of the raterBoxes provide a means of recording performance judgment
Checklists - FocusFocus on the procedure when there is NO productFocus on product when:You have a choiceThe procedure is not available for observation
Checklists - LimitationsChecklists are for all-or-none decisionsThey create a conflict for respondents when the item contains more than one performance attributeCan force a judgment when there is no basis for the judgment
Rating Scales (Advantages)Contain a list of attributes with a range of responses: e.g. VG, G, ..Poor; SA, A, D, ..Rating scales direct raters attention to certain performance dimensions & provide a way for recording judgmentCovers areas that are not covered well by other methods (e.g. MCQs)Flexibility of useLow costUnobtrusiveness
Rating Scales - LimitationsOften filled out retrospectivelyLack of agreement. Get many raters for one student.Associated with certain types of errorsLeniencyRange restrictionHalo effect
My AdviceUse checklists rather than rating scales when you have a choiceUse checklists and rating scales immediately after observationsTake care in describing performance attributesUse tailored descriptive phrases to anchor the points on scaleInclude a category like Not able to rate
More AdviceProvide room for comments to encourage constructive feedbackDont ask raters to provide a gradeUse 5 7 rating points on the scaleGet multiple ratings of the same student (7 or more)Use a scoring guide or product scale when rating holisticallySupplement Checklists and rating scales
PortfoliosA useful method of collecting, organizing, and evaluating samples of students work.Advantages:Learning progress over timeComparing work to past workSelf-assessment skillsClear communication of learning progress
Effective Use of PortfoliosDeciding what to includeDeciding on criteria and standardsCollecting the work samplesMaintaining and using the portfoliosGetting started
In conclusionPAs can provide useful information concerning student achievement but they are subject to all the errors of observation and judgment, such as personal bias, generosity error, and halo effect.If PAs are to provide valid information, special care must be taken to improve the objectivity, reliability, and meaningfulness of the results.