EDU 385Classroom

Assessment

Session 6Preparing and Using Achievement Tests

Bell Work

Content Objectives

•Understand how to prepare and use an achievement test

•Understand when to use an achievement test

Language Objectives• List steps to prepare an Achievement Test andPrepare

a set of Specifications for a Test

• Describe merits of Selection-Type and Supply-Type Tests

• Match test items to specific Learning Outcomes they measure

• Describe factors to consider in preparing items for test

• Write Clear Directions for a test

• Review and Evaluate an assembled test

• Administer a test properly

• Evaluate the effectiveness of a test or test items

Preparing Achievement Tests

•Specify the Instructional Objectives

•Prepare Test Specification

•Construct relevant Test Items

•Arrange the items in the Test

•Prepare Clear Directions

•Review and Evaluate the Assembled Test

•Administer the Test and Make an Item Analysis

Specify Instructional Objectives

• At End of Unit the Student will:

• Remember the meaning of Common Terms

• Remember Specific Facts

• Remember Basic Procedures

• Understand Relevant Principles

• Apply the Principles

Define General Outcomes in

Specific Termspe

•Next Step is to list Specific Types of Student Performance Accepted as Evidence that Outcomes have been Achieved

•Specific Observable Performance Terms (Action Verbs)

•Identifies, Distinguishes between

•Selects, Describes, Matches

•Explains, Predicts, Formulates

Vague and Indefinite Terms

•Avoid Using Terms As:

•Learns

•Sees

•Realizes

TaxonomyCategories

Sample Verbs for Stating Specific Learning Outcomes

RememberIdentifies, names, defines, describes, lists, matches, selects, outlines

UnderstandClassifies, explains, summarizes, converts, predicts, interprets, illustrates

ApplyDemonstrates, computes, solves, modifies, arranges, operates, relates, uses

AnalyzeDifferentiates, diagrams, estimates, separates, orders, subdivides, distinguishes between

Evaluatejudges, criticizes, compares, justifies, concludes, discriminates, detects

CreateCombines, creates, formulates, designs, composes, constructs, rearranges, revises

Illustrative Action Verbs for Defining Objectives in the Cognitive Domain of Blooms Revised Taxonomy

Preparing Test Specifications

• The writing of Test Items Should be Guided by set of Test Specifications

• Describe the Achievement Domain to be Measured

• Link the Achievement Domain to Course Subject Matter

• Provide Guidelines for Obtaining Representative Sample of Test Tasks

Table of Specifications

•Select the Learning Outcome to be Tested

•Outline the Subject Matter

•Make a Two-Way Chart (describes the sample of items to be included in the test)

Select Learning OutcomesFrom Following Items

•Recall of Knowledge

•Intellectual Abilities and Skills

•General Skills (laboratory, performance, communication, work-study)

•Attitudes, Interests, and Appreciations

Outlining Subject Matter

•Content of a Course may be outlined in detail for teaching purposes

•For Test Planning only the major categories need be listed

Making 2-Way Chart or Table of Specifications

•Purpose helps provide assurance that the test will measure a representative sample of learning outcomes

•It relates outcomes to content

•Indicates relative weight given to various areas

ContentTerm

sFact

sProcedur

es

Understands

Principles

AppliesPrinciple

s

Total # of Items

Role of Tests in Instruction

4 4 2 10

Principles of Testing

4 3 2 6 5 20

Norm Referenced vs. Criterion Referenced

4 3 3 10

Planning the Test

3 5 5 2 5 20

Total Number of Items

15 15 10 10 10 60

Remembers

Table of Specifications for a Summative Test on a Unit for Achievement Testing (example)

Factors to Consider on Weighting Test Items

• Importance of area in learning outcome

• Importance in total learning experience

• Which outcomes have greater retention and transfer value

• What importance does CORE curriculum place on each area

• Should reflect the importance placed during instruction

Constructing Test Items

•Selection-Type Items• Multiple Choice

• True-False

• Matching

• Interpretive Exercise

•Supply-Type Items• Short Answer

• Essay (restricted Response)

• Essay (Extended Response)

Which to Use

•Guiding Principle

•Use the item types that provide the most direct measures of student performance specified by the intended learning outcomes

Measures Factual Information

Yes Yes Yes(*)

Measures Understanding Yes No(**) Yes

Measures Synthesis No(**) No(**) Yes

Easy to Construct No Yes Yes

Samples Broadly Yes Yes No

Eliminates Bluffing Yes No No

Eliminates Writing Skill Yes No No

Eliminates Blind Guessing No Yes Yes

Easy to Score Yes No No

Scoring is Objective Yes No No

Pinpoints learning Errors Yes Yes No

Encourages Originality No No Yes

Characteristic Selection-Type Items

Short Answer

Essay

Supply-Type Items

(*) The essay Test can measure knowledge of facts, but because of scoring and sampling problems it probably should not be used for this purpose(**) These items can be designed to measure limited aspects of these characteristics

Matching Items (examples)

• Specific Learning Outcome: Defines terms in a student’s own words

• Directions: Define each of the following terms in a sentence or two

• Taxonomy

• Cognitive

• Measurement

• Evaluation

Matching Items (examples)Specific Learning Outcome: Identifies procedural steps in planning for a test

1. Which one of the following steps should be completed first in planning for an achievement test?

A. Select the types of test items to use B. Decide on the length of the test*C. Define the intended learning outcomes D. Prepare the test specifications

Matching Items (examples)Specific Learning Outcome: Distinguish between sound and unsound principles in achievement testing

Directions: Read each of the following statements. If the statement indicates a sound principle of achievement testing, circle the S; if it indicates and unsound principles circle the U.

*S U 1. The specific learning outcome to be tested should be stated in terms of student performance

S U 2. Achievement testing should be limited to outcomes that can be measured objectively

S U 3. Each achievement test item should measure a clearly defined type of student performance

Cautions in Phrasing Test Items

•Vocabulary that is unnecessarily difficult

•Sentence structure that is unnecessarily complex

•Statements containing ambiguity

•Unclear pictorial materials

•Directions that are vague

•Material reflecting race, ethnic, or sex bias

Common Clues to Avoid• Verbal associations that give away the answer

• Grammatical inconsistencies that eliminate wrong answers

• Specific determiners that make certain answers probable (e.g., sometimes) and others improbable (e.g. always)

• Stereotyped or textbook phrasing of the correct answer

• Length and location of correct answers

• Material in an item that aids in answering another item

Deciding Number of Test Items

•Age of Students Tested

•Time Available for Testing

•Type of Test Items Used

•Type of Interpretation to be Made

Checklist for Evaluating the Test Plan

1. Is the purpose of the test clear?2. Have the intended learning outcomes been identified & defined?3. Are the intended learning outcomes stated in performance (measurable terms?4. Have test specifications been prepared that indicate the nature and distribution of items to be included in the test:5. Does the specified set of items provide a representative smaple of the tasks contained in the achievement domain?6. Are the types of items appropriate for the learning outcomes to be measured?7. Is the difficulty of the items appropriate for the students to be tested and the nature of the measurement (e.g., mastery or survey)8. Is the number of items appropriate for the students to be tested, the time available for testing and the interpretations to be made?9. Does the test plan include built-in features that contribute to valid and reliable scores?10. Have plans been made for arranging the items in the test, writing directions, scoring, and using the results?

Guidelines for Writing Items• Select type of test item that measures

intended outcome most directly

• Write test item so the performance it elicits matches the performance of the learning task

• Write test item so test take is clear and definite

• Write test item so it is free from nonfunctional material

• Write test item so irrelevant factors do not prevent an informed student from performing correctly

Guidelines for Writing Items

• Write test question so irrelevant clues do not enable the uniformed student to respond correctly

• Write test item so difficulty level matches the intent of the learning outcome, age group tested, & use made of results

• Write test item so there is no disagreement concerning the answer

• Write the test items in advance so they can be reviewed and modified

• Write more test items than called for in test plan

Arranging Items in Test

• Guidelines

• For instructional purpose, it is usually desirable to group together items that measure the same outcomes

• Where possible, all items of the same type should be grouped together

• Items should be arranged in terms of increasing difficulty

Preparing Directions

• Directions should be simple and concise

• Yet contain information concerning each of following:

• Purpose of Test

• Time allowed to complete the test

• How to record the answers

• Whether to guess when in doubt about an answer

Checklist for Evaluating the Assembled Test

1. BalanceDo the items measure a representative sample of the learning tasks in the achievement domain?

2. Relevance Do the test items present relevant tasks?

3. ConcisenessAre the test tasks stated in simple, clear language?

4. SoundnessAre the items of proper difficulty, free from defects, and do they have answers that are defensible

5. independence

Are the items free from overlapping, so that one item does not aid in answering another?

6. Arrangement

Are items measuring the same outcome grouped together? Are items of the same type grouped together? Are items in order of increased difficulty?

7. NumberingAre the items numbered in order throughout the test?

8. Directions

Are there directions for the whole test and each part?Are the directions concise and at proper reading level?Do directions include time limts and how to record answers?Do directions tell what to do about guessing?

9. SpacingDoes the spacing on the page contribute to ease of reading and responding?

10. Typing Is the final copy free from typographical errors?

Administering Tests

•Provide space with desks far enough apart to prevent cheating

•Keep interruptions at a minimum

•May want to give oral directions in addition to written

•Make sure all students know exactly what to do and provide them with most favorable conditions possible

Scoring Test

•Scoring is facilitated by having answers on left side of each test page

•If using a separate answer sheet may want to use a scoring stencil (punch out letters of correct answers and lay it over the answer sheet)

Item Analysis• Provide students a copy of the test and review

each question with them, allow discussion may find test items which can be improved plus students will learn from review

• After tests are scored tally student responses on a master copy (number right and number wrong for each question)

• Analyze top 1/3 and bottom 1/3 (Did students miss same items or were they random. Review will help next time you teach the course

Summary• Preparing and using achievement test includes

specifying the instructional objectives in performance terms, preparing table of specifications, constructing relevant test items, arranging items in test, preparing clear directions, reviewing and evaluating the assembled test, administering the test, and making an item analysis

• Test specifications typically consist of twofold table of specifications that indicates the sample of performance tasks to be measured

• Type of test items used in a test are determined by how directly they measure intended learning outcomes and how effective as measuring instruments

Summary• Each item should provide a task that matches

student performance described in specific learning outcomes

• Functioning content of test items can be improved by eliminating irrelevant barriers and unintended clues

• Difficulty of test item should match the difficulty of the learning task to be measured. Beware of irrelevant sources of difficulty (e.g., obscure material)

• Achievement test should be short enough to permit all students to attempt all items during time available

Summary• Test should contain sufficient number of test items for

each type of interpretation to be made. (Interpretation of fewer that 10 should be considered highly tentative)

• Following general set of guidelines during item writing will result in higher quality items that contribute to validity and reliability of test results

• Item arrangement within test varies with type of test used. When possible items should be grouped by major learning outcomes (e.g., knowledge, understanding, application) similar items should be grouped together and arranged in increasing difficulty

• Test directions should indicate purpose of test, time allowed, how to record answers, and answer all items

Summary

• A correction for guessing should be used for speed tests only.

• Review and evaluate the assembled test before using

• Administer the test under controlled conditions

• Item analysis provides a means of evaluating the items and is useful in reviewing the test with students

Now Go Forth and

Do Good Things