Assessing When Numbers Assessing When Numbers Don’t CountDon’t Count

Binghamton UniversityBinghamton University

March 23, 2007March 23, 2007

Today’s ObjectivesToday’s Objectives

Define what a discourse-based, or Define what a discourse-based, or qualitative assessment method isqualitative assessment method is

Review a few of these methodsReview a few of these methods Work with some case studies to Work with some case studies to

better understand how these better understand how these methods might be usedmethods might be used

Discuss the uses and limitations of Discuss the uses and limitations of these methods, especially with these methods, especially with regard to assessmentregard to assessment

The Pressure to AssessThe Pressure to Assess This week—Spellings Commission meeting This week—Spellings Commission meeting

to discuss several issues. One of them is to discuss several issues. One of them is student learning outcomesstudent learning outcomes

There is consistent pressure to use There is consistent pressure to use standardized tests and surveys; we are standardized tests and surveys; we are currently using one of them, the NSSEcurrently using one of them, the NSSE

Through the use of standardized tests and Through the use of standardized tests and surveys, we gain perspective on how we surveys, we gain perspective on how we compare to other institutionscompare to other institutions

We might be able to gain a “value added” We might be able to gain a “value added” perspectiveperspective

Weaknesses of the Standardized Weaknesses of the Standardized Test/Survey ApproachTest/Survey Approach

We obtain a “macro” perspective, but may We obtain a “macro” perspective, but may not gain a “micro” perspectivenot gain a “micro” perspective

We have little control over the questionsWe have little control over the questions ““Value added” is still difficult to establish Value added” is still difficult to establish

with standardized testswith standardized tests Faculty see little value in using Faculty see little value in using

standardized tests, especially in standardized tests, especially in interdisciplinary studies and the liberal interdisciplinary studies and the liberal artsarts

Standardized tests/surveys often assume Standardized tests/surveys often assume they can measure most of what concerns they can measure most of what concerns faculty in respect to teaching and learningfaculty in respect to teaching and learning

Forging the Middle Ground:Forging the Middle Ground:Discourse-Based Assessment MethodsDiscourse-Based Assessment Methods Allow for the discovery of the unanticipatedAllow for the discovery of the unanticipated Relevant to interdisciplinary studyRelevant to interdisciplinary study Maximizes faculty/staff input when Maximizes faculty/staff input when

discourse is highly valueddiscourse is highly valued Good to use when the number of objectives Good to use when the number of objectives

outweigh the amount of time available to outweigh the amount of time available to assess student learningassess student learning

Great contextualizerGreat contextualizer When performed carefully and compared When performed carefully and compared

with other methods, great return on with other methods, great return on investmentinvestment

Types of Qualitative AssessmentTypes of Qualitative Assessment

Focus groupsFocus groups Expert panelsExpert panels Open-ended surveysOpen-ended surveys Ethnographic studies (participant Ethnographic studies (participant

observations)observations) Portfolio reviewsPortfolio reviews Primary trait scoringPrimary trait scoring Delphi panelsDelphi panels

Qualitative Assessment is an Qualitative Assessment is an Inductive ProcessInductive Process

Defined Focus

Observations

Analysis

Summary Report

Comparison with Other Assessments

Action

Expert PanelExpert Panel

Type of focus groupType of focus group Focus is on (for assessment purposes) a Focus is on (for assessment purposes) a

particular assignment or performanceparticular assignment or performance Not a simple conversation; is done Not a simple conversation; is done

methodically, with precision, and is methodically, with precision, and is systematicsystematic

Often involves convenience or random Often involves convenience or random samples of a homogeneous populationsamples of a homogeneous population

Must include carefully written questionsMust include carefully written questions Might include a survey or other assessment Might include a survey or other assessment

technique as part of the processtechnique as part of the process

Krueger’s 10 Quality Factors in Focus Krueger’s 10 Quality Factors in Focus Group (Expert Panel) ResearchGroup (Expert Panel) Research

Clarity of purposeClarity of purpose Appropriate environmentAppropriate environment Sufficient resourcesSufficient resources Appropriate participantsAppropriate participants Skillful moderatorSkillful moderator Effective questionsEffective questions Careful data handlingCareful data handling Systematic and verifiable analysisSystematic and verifiable analysis Appropriate presentationAppropriate presentation Honoring the participant, clarity, and Honoring the participant, clarity, and

methodmethod

Expert Panel ProcedureExpert Panel Procedure

Select Focus Select ExpertsWrite Questions

Use Additional Method?

LogisticsConduct Expert Panel—How to Assess Agreement?

Report Results

Final Thoughts on Expert PanelsFinal Thoughts on Expert Panels

Good method of assessing “ineffable Good method of assessing “ineffable outcomes”outcomes”

Better when expert panel questions and Better when expert panel questions and conversations are grounded in standards conversations are grounded in standards and/or student learning objectivesand/or student learning objectives

Also good when specific focus is on a Also good when specific focus is on a particular assignment or performanceparticular assignment or performance

It is advisable to use a secondary method It is advisable to use a secondary method either prior to or during expert paneleither prior to or during expert panel

Does not control for anonymity among Does not control for anonymity among respondentsrespondents

Primary Trait ScoringPrimary Trait Scoring

Focus is upon one particular Focus is upon one particular assignment, performance, etc., that assignment, performance, etc., that is reflective of several aggregate is reflective of several aggregate student learning outcomesstudent learning outcomes

Rate each outcome according to a Rate each outcome according to a scale—ex., proficient, satisfactory, scale—ex., proficient, satisfactory, unsatisfactoryunsatisfactory

Idea is to look at trends, not Idea is to look at trends, not numbers, that spark discussionnumbers, that spark discussion

Primary Trait Scoring--ProcedurePrimary Trait Scoring--Procedure

Choose an assignment in which students Choose an assignment in which students demonstrate summative knowledge, skills, or demonstrate summative knowledge, skills, or competenciescompetencies

Carefully rate student performance according to Carefully rate student performance according to the scalethe scale

Place checkmarks in each columnPlace checkmarks in each column Look for visual trendsLook for visual trends Discuss why these trends occur, what basis these Discuss why these trends occur, what basis these

rating occurred, and what specific issues are rating occurred, and what specific issues are revealed through the analysisrevealed through the analysis

Combine with other findings, or make plans for Combine with other findings, or make plans for actionaction

Example of Primary Trait AnalysisExample of Primary Trait Analysis

ElementElement ExcellentExcellent SatisfactorySatisfactory UnsatisfactoryUnsatisfactory

Demonstrate an ability Demonstrate an ability to research the to research the financial integrity financial integrity of a business plan of a business plan through through accounting/financaccounting/financial analysisial analysis

√√√√√√√√ √√√√√√√√√√√√ √√√√

Demonstrate an ability Demonstrate an ability to assess a plan’s to assess a plan’s organizational organizational integrity; that is, a integrity; that is, a business plan business plan demonstrates how demonstrates how an organization an organization will be built and will be built and sustained to under sustained to under gird the success gird the success of the business of the business being organizedbeing organized

√√√√ √√√√√√√√ √√√√√√√√√√√√

Primary Trait Analysis—Final Primary Trait Analysis—Final ThoughtsThoughts

Method to get faculty or staff to talk Method to get faculty or staff to talk about what assessment results meanabout what assessment results mean

A good starting point toward A good starting point toward developing a rubricdeveloping a rubric

Enables discussion, which can lead to Enables discussion, which can lead to further discoveryfurther discovery

““Simple, stress free, and easy”Simple, stress free, and easy”

Delphi Panel Introductory ExerciseDelphi Panel Introductory Exercise

Divide into 3 groupsDivide into 3 groups Get out piece of paper and individually Get out piece of paper and individually

write down, “what do students have write down, “what do students have the most difficulty with when they first the most difficulty with when they first come to college (as first-year come to college (as first-year students)?”students)?”

Try to create frequency counts—Try to create frequency counts—combine like answers and tally themcombine like answers and tally them

DiscussDiscuss

Questions for GroupsQuestions for Groups

What do these say about the difficulty What do these say about the difficulty students might have when they start?students might have when they start?

Take a look at the most “popular” Take a look at the most “popular” answer—do these ordinarily achieve answer—do these ordinarily achieve “majority vote status?“majority vote status?

Even in cases where “majority vote Even in cases where “majority vote status” is achieved, might less popular status” is achieved, might less popular answers indicate group consensus?answers indicate group consensus?

Introduction to the Delphi MethodIntroduction to the Delphi Method

Combination of at least 3 methods—open-Combination of at least 3 methods—open-ended survey, closed-ended survey, and ended survey, closed-ended survey, and expert panelexpert panel

Unlike expert panel, attempts to maximize Unlike expert panel, attempts to maximize anonymity of respondents to control for anonymity of respondents to control for power dynamics among these respondentspower dynamics among these respondents

Assumes highly motivated groups of Assumes highly motivated groups of experts (faculty or staff) willing to experts (faculty or staff) willing to participate in more than one round of participate in more than one round of questionsquestions

Introduction to the MethodIntroduction to the Method1.1. Find homogeneous group of experts who can Find homogeneous group of experts who can

comment either on one assignment or specific comment either on one assignment or specific student learning outcomesstudent learning outcomes

2.2. Create an open-ended survey in which respondents Create an open-ended survey in which respondents are asked to identify strengths and weaknesses in are asked to identify strengths and weaknesses in student performance in reference to specific student performance in reference to specific standard or student learning outcomestandard or student learning outcome

3.3. Content analyze responses by combining like Content analyze responses by combining like responses, placing how many times each was responses, placing how many times each was mentioned in parenthesesmentioned in parentheses

4.4. ““Cut and paste” these onto a survey, and ask Cut and paste” these onto a survey, and ask respondents to indicate to what extent they agree respondents to indicate to what extent they agree with each on a 4 or 5 point scalewith each on a 4 or 5 point scale

5.5. Report those responses that indicate consensusReport those responses that indicate consensus6.6. If needed, move to 3If needed, move to 3rdrd round, in which respondents round, in which respondents

rank these consensus itemsrank these consensus items

Strengths of Delphi MethodStrengths of Delphi Method

A way of addressing “ineffable A way of addressing “ineffable outcomes”outcomes”

Can be used to designate the most Can be used to designate the most agreed-upon student learning objectives agreed-upon student learning objectives that faculty have communicatedthat faculty have communicated

Can be used to gather information from Can be used to gather information from employers, internship supervisors, employers, internship supervisors, alumni, etc. about specific items of alumni, etc. about specific items of interestinterest

Limitations of Delphi MethodLimitations of Delphi Method

Can be time consumingCan be time consuming Takes some knowledge of statisticsTakes some knowledge of statistics Not a method that can be used by Not a method that can be used by

itself; usually results need to be itself; usually results need to be compared with direct assessments of compared with direct assessments of student learningstudent learning

Today’s ActivitiesToday’s Activities

Separate into three groups; select Separate into three groups; select group note takergroup note taker

If you have not already, read the case If you have not already, read the case study packetsstudy packets

As a group, discuss questions at end As a group, discuss questions at end of case study—debate, applaud, etc.—of case study—debate, applaud, etc.—do something activedo something active

Write answers on provided sheet of Write answers on provided sheet of paperpaper