1

Developing Tests for Developing Tests for Departmental AssessmentDepartmental Assessment

Deborah Moore, Assessment SpecialistInstitutional Research, Planning, & Effectiveness

University of Kentucky

2

Session OverviewSession Overview

Steps in Test Development and Steps in Test Development and Test AdministrationTest Administration

Process for Evaluating Test and Process for Evaluating Test and ItemsItems

Common DifficultiesCommon Difficulties Strategies for Analysis and ReportsStrategies for Analysis and Reports

3

Improve Student LearningImprove Student Learning

4

Exercise 1 Exercise 1 (10 minutes)(10 minutes)

How do you decide you are going to How do you decide you are going to give a test?give a test?

What do you do to develop a new What do you do to develop a new test?test?

How do you decide what types of How do you decide what types of item formats to use?item formats to use?

How do you develop items?How do you develop items? How long does it take to put the test How long does it take to put the test

together?together?

Under What Conditions Do You Give a Test?Under What Conditions Do You Give a Test?

5

Using a Test for Using a Test for Classroom AssessmentClassroom Assessment

Generally, instructors develop their own Generally, instructors develop their own classroom tests, making all decisions about classroom tests, making all decisions about when and how to construct, administer, score, when and how to construct, administer, score, and report results of tests. and report results of tests.

Construction is often done without formality or Construction is often done without formality or documentation. documentation.

The most frequent use of tests by instructors is The most frequent use of tests by instructors is to assign grades related to individual student to assign grades related to individual student learning.learning.

6

A Test Is a Good Choice When…A Test Is a Good Choice When…

the student must demonstrate acquisition of knowledge the student must demonstrate acquisition of knowledge or ability to process and use knowledgeor ability to process and use knowledge

the student's knowledge about a wide range of content the student's knowledge about a wide range of content is to be evaluated (e.g., survey and capstone courses)is to be evaluated (e.g., survey and capstone courses)

multiple observations of the content-related knowledge multiple observations of the content-related knowledge are needed (e.g., math and foreign languages)are needed (e.g., math and foreign languages)

more resources are available for constructing the test more resources are available for constructing the test than for scoring and reportingthan for scoring and reporting

a large group is being assesseda large group is being assessed

7

Using a Test for Using a Test for Departmental AssessmentDepartmental Assessment

Planning, implementing, and using results Planning, implementing, and using results become a group effort – a shared set of become a group effort – a shared set of decisions and responsibilities. Consensus is decisions and responsibilities. Consensus is emphasized. emphasized.

Additional planning time, communication, and Additional planning time, communication, and record keeping will be needed. record keeping will be needed.

When used for program assessment, test When used for program assessment, test performance is generally used along with other performance is generally used along with other information to describe group achievement information to describe group achievement and is independent of grading.and is independent of grading.

8

Advantages of Advantages of Selected-Response Item Selected-Response Item FormatsFormats

Better content coverageBetter content coverage Higher reliabilityHigher reliability Greater efficiencyGreater efficiency ObjectivityObjectivity Mechanical scoringMechanical scoring

9

Test Development ProcessTest Development Process Define purpose/useDefine purpose/use Outline curriculum & consult goals/objectives Outline curriculum & consult goals/objectives Create test plan Create test plan (aka blueprint or specifications)(aka blueprint or specifications)

Create a pool of itemsCreate a pool of items Critique and revise itemsCritique and revise items Pilot/field test Pilot/field test (e.g. item analysis, reliability/validity studies)(e.g. item analysis, reliability/validity studies)

Set guidelines for test administration, scoring Set guidelines for test administration, scoring procedures, interpretation of scores procedures, interpretation of scores (e.g. develop (e.g. develop norm tables and standard setting procedures)norm tables and standard setting procedures)

10

Exercise 2 Exercise 2 (10 minutes)(10 minutes)

What role does the exam play in the What role does the exam play in the assessment plan for this department?assessment plan for this department?

What type of item formats should be What type of item formats should be used on the comprehensive exam?used on the comprehensive exam?

Review a Sample Departmental PacketReview a Sample Departmental Packet

11

ObjectivesObjectivesRespond in an informed way Respond in an informed way to the form, structure and to the form, structure and aesthetic qualities of artistic aesthetic qualities of artistic and literary works.and literary works.

Explore the interrelationships Explore the interrelationships among historical events and among historical events and intellectual, artistic, literary and intellectual, artistic, literary and philosophical or religious philosophical or religious movements and works.movements and works.

Identify and analyze Identify and analyze similarities, differences and similarities, differences and interrelationships among the interrelationships among the fine arts.fine arts.

Articulate central philosophical Articulate central philosophical and religious questions and the and religious questions and the varying responses to them varying responses to them within different cultures.within different cultures.

Apply appropriate vocabulary Apply appropriate vocabulary and concepts for the and concepts for the description and analysis of description and analysis of artistic, literary, historical and artistic, literary, historical and philosophical or religious philosophical or religious works.works.

Explain how artistic and literary Explain how artistic and literary works from the past and works from the past and present civilizations are present civilizations are individual expressions of individual expressions of cultural, historical and cultural, historical and intellectual forces.intellectual forces.

12

Obj 1 Obj 2 Obj 3 Obj 4 Obj 5 Obj 6 Totals

Theater 1 1 11 3 2 18

Art 5 3 3 4 3 18

Music 5 1 6 1 1 14

Philosophy/ Religion

1 2 3 9 8 23

Anthropology 2 1 1 2 6

Literature 5 4 6 1 4 20

Totals 17 6 30 12 20 14 99

Test Blueprint Test Blueprint (Content X Objective)(Content X Objective)

13

Test Plan Test Plan (Items Within Cells)(Items Within Cells)

Obj 1 Obj 2 Obj 3 Obj 4 Obj 5 Obj 6 Totals

Theater 7 18 1-6,9-12, 16

8, 13, 17

14-15 18

Art 48,50-51, 58, 64

47, 49, 61

53-54, 59

52, 57, 60, 63

56-57, 62

18

Music 42-46 41 35-40 33 85, 89-90, 99

14

Philosophy/ Religion

66 24-25 26, 78-79

65, 67-68, 72-77

19-23, 69-71

18

Anthropology 29, 32 31 27 28, 30 6

Literature 86-88, 94, 98

47, 49, 61

53-54, 59

52, 57, 60, 63

56-57, 62

20

Totals 17 6 30 12 20 14 99

14

Administrative ApproachesAdministrative Approaches

Common exams Common exams

Course-embedded testingCourse-embedded testing

Assessment Center/Assessment Assessment Center/Assessment DaysDays

15

Sources for ItemsSources for Items

ResearchResearch CurriculumCurriculum Real lifeReal life Professional Professional

development development materials & materials & textbookstextbooks

Own final exam Own final exam filesfiles

Comprehensive Comprehensive exams from exams from other other universitiesuniversities

Retired Retired test/item bankstest/item banks

16

Common Difficulties with Test Common Difficulties with Test QualityQuality

Push for higher order thinking often Push for higher order thinking often weakweak

Insufficient consensus on test Insufficient consensus on test plan/blueprintplan/blueprint

Discomfort with various item-writing Discomfort with various item-writing approaches (writer’s block)approaches (writer’s block)

Insufficient editingInsufficient editing

Little or no piloting of itemsLittle or no piloting of items

17

Critiquing ItemsCritiquing Items

Match with test Match with test planplan

Item valueItem value Item qualityItem quality

18

Typical Item Review Typical Item Review QuestionsQuestions

Has each item received a quality appraisal? Has each item received a quality appraisal? Has each item’s content been verified? Has each item’s content been verified? Has each item been classified?Has each item been classified? Has the key been identified?Has the key been identified? Have you edited the items? Have you edited the items? Have items been checked for Have items been checked for

bias/insensitivity?bias/insensitivity? Have you field tested the items?Have you field tested the items?

19

Planning A Departmental Planning A Departmental TestTest

Develop testDevelop test Plan administrationPlan administration Analyses test dataAnalyses test data Report and use results Report and use results Agree on leadership/support rolesAgree on leadership/support roles

20

Exercise 3 Exercise 3 (30 minutes)(30 minutes)

Review the headings of the checklist formReview the headings of the checklist form

List additional headings neededList additional headings needed

List activities that need to be completed List activities that need to be completed under the various headingsunder the various headings

Add a number to the activities to Add a number to the activities to represent the order of completionrepresent the order of completion

Developing a Planning Developing a Planning ChecklistChecklist

21

AnalysesAnalyses

Examine quality of your assessment Examine quality of your assessment tools and procedurestools and procedures

Describe the test resultsDescribe the test results By test plan featuresBy test plan features By meaningful subgroupsBy meaningful subgroups

Combine with other datasetsCombine with other datasets Execute and summarize planned Execute and summarize planned

comparisons linked to research comparisons linked to research designdesign

22

ReportsReports

Determine Determine stakeholders/constituentsstakeholders/constituents Identify specific interestIdentify specific interest

Create a timeline and shareCreate a timeline and share Include a vetting procedureInclude a vetting procedure

23

Testing in the FutureTesting in the Future

Computer based testingComputer based testing Adaptive (CAT)Adaptive (CAT) Self-Adaptive (SAT)Self-Adaptive (SAT)

Computer assisted item generationComputer assisted item generation Automated essay scoringAutomated essay scoring

24

Thank you for your Thank you for your attention.attention.

Deborah Moore, Assessment SpecialistDeborah Moore, Assessment Specialist

101B Alumni Gym101B Alumni GymOffice of Institutional Research, Office of Institutional Research, Planning, & EffectivenessPlanning, & Effectiveness

dlmoor2@email.dlmoor2@email.ukyuky..eduedu

859/257-7086859/257-7086

http://www.http://www.ukyuky..eduedu//LexCampusLexCampus//