Review: Alternative Assessments IReview: Alternative Assessments I

Describe the two epistemologies in ch. 3 (o/s)Describe the two epistemologies in ch. 3 (o/s)

Compare the two principles for assigning value Compare the two principles for assigning value (util/int-pl)(util/int-pl)

Identify pros/cons of the two evaluation Identify pros/cons of the two evaluation approaches we discussed last weekapproaches we discussed last week

Alternative Approaches to Alternative Approaches to Evaluation IIEvaluation II

Dr. Suzan Ayers Dr. Suzan Ayers

Western Michigan UniversityWestern Michigan University(courtesy of Dr. Mary Schutten)(courtesy of Dr. Mary Schutten)

Consumer-Oriented ApproachConsumer-Oriented Approach

Typically a summative evaluation approach Typically a summative evaluation approach

This approach advocates consumer education This approach advocates consumer education and independent reviews of productsand independent reviews of products

Scriven’s contributions based on groundswell of Scriven’s contributions based on groundswell of federally funded educational programs in 1960sfederally funded educational programs in 1960sDifferentiation between formative/summative eval.Differentiation between formative/summative eval.

Consumer-Oriented ChecklistConsumer-Oriented Checklist(Scriven, 1974, p. 102)(Scriven, 1974, p. 102)

NeedNeed MarketMarket PerformancePerformance

True field trials [tests in a “real” setting]True field trials [tests in a “real” setting] True consumer tests [tests with real users]True consumer tests [tests with real users] Critical comparisons [comparative data]Critical comparisons [comparative data] Long term [effects over the long term]Long term [effects over the long term] Side effects [unintended outcomes]Side effects [unintended outcomes] Process [product use fits its descriptions]Process [product use fits its descriptions] Causation [experimental study]Causation [experimental study] Statistical significance [supports product effectiveness]Statistical significance [supports product effectiveness] Educational significanceEducational significance

Cost effectivenessCost effectivenessExtended support [in service training]Extended support [in service training]

Producer’s efforts to meet these standards Producer’s efforts to meet these standards improve product effectivenessimprove product effectiveness

Key Evaluation Checklist Key Evaluation Checklist developed to evaluate developed to evaluate program evaluationsprogram evaluations

Educational Products Information ExchangeEducational Products Information Exchange (EPIE)(EPIE): Independent product-reviewer service: Independent product-reviewer service

Curriculum Materials Analysis SystemCurriculum Materials Analysis System (CMAS) (CMAS) checklist: Describe product, analyze rationale, checklist: Describe product, analyze rationale, consider: antecedent conditions, content, consider: antecedent conditions, content, instructional theory & teaching strategies, form instructional theory & teaching strategies, form overall judgmentsoverall judgments

Uses of Consumer-OrientedUses of Consumer-OrientedEvaluation ApproachEvaluation Approach

Typically used by gov’t. agencies and Typically used by gov’t. agencies and consumer advocates (i.e., EPIE)consumer advocates (i.e., EPIE)

What does one need to know about a product What does one need to know about a product before deciding whether to adopt or install it?before deciding whether to adopt or install it?Process informationProcess informationContent informationContent informationTransportability informationTransportability informationEffectiveness informationEffectiveness information

Consumer-Oriented Pros/ConsConsumer-Oriented Pros/ConsStrengthsStrengths: valuable info given to those who : valuable info given to those who

don’t have time to study, advance consumers’ don’t have time to study, advance consumers’ knowledge of appropriate criteria for selection knowledge of appropriate criteria for selection of programs/productsof programs/products

WeaknessesWeaknesses: can increase product cost, : can increase product cost, stringent testing may “crimp” creativity, local stringent testing may “crimp” creativity, local initiative lessened b/c of dependency on initiative lessened b/c of dependency on outside consumer servicesoutside consumer services

Consumer-Oriented QsConsumer-Oriented Qs

What educational products do you use?What educational products do you use?

How are purchasing decisions made? How are purchasing decisions made?

What criteria seem to most important in the What criteria seem to most important in the selection process?selection process?

What other criteria for selection does this What other criteria for selection does this approach suggest to you?approach suggest to you?

Expertise-Oriented ApproachExpertise-Oriented ApproachDepends primarily upon professional expertise Depends primarily upon professional expertise

to judge an institution, program, product, or to judge an institution, program, product, or activityactivity

This is the first view that relies heavily on This is the first view that relies heavily on subjective expertise as the key evaluation toolsubjective expertise as the key evaluation tool

Examples: doctoral exams, board reviews, Examples: doctoral exams, board reviews, accreditation, reappointment/tenure reviews accreditation, reappointment/tenure reviews etc…etc…

Expertise-Oriented TypesExpertise-Oriented TypesFormal Review Systems Formal Review Systems (accreditation)(accreditation)

Existing structure, standards exist, set review Existing structure, standards exist, set review schedule, experts, status usually affected by resultsschedule, experts, status usually affected by results

Informal Review systems Informal Review systems (grad S committee)(grad S committee)

Existing structure, no standards, infrequent Existing structure, no standards, infrequent schedule, experts, status usually affectedschedule, experts, status usually affected

Ad hoc panel review Ad hoc panel review (journal reviews)(journal reviews)

Multiple opinions, status sometimes affectedMultiple opinions, status sometimes affectedAd hoc individual review Ad hoc individual review (consultant)(consultant)

Status sometimes affectedStatus sometimes affected

Expertise-Oriented Pros/ConsExpertise-Oriented Pros/ConsStrengthsStrengths: those well-versed make decisions, : those well-versed make decisions,

standards are set, encourage improvement standards are set, encourage improvement through self-studythrough self-study

WeaknessesWeaknesses: whose standards? (personal : whose standards? (personal bias), expertise credentials, can this approach bias), expertise credentials, can this approach be used with issues of classroom life, texts, and be used with issues of classroom life, texts, and other evaluation objects or only with the bigger other evaluation objects or only with the bigger institutional questions?institutional questions?

Expertise-Oriented QsExpertise-Oriented Qs

What outsiders review your program or What outsiders review your program or organization? organization?

How expert are they in your program’s context, How expert are they in your program’s context, process, and outcomes?process, and outcomes?

What are characteristics of the most/least What are characteristics of the most/least helpful reviewers? helpful reviewers? (list brainstorms on board)(list brainstorms on board)

Participant-Oriented ApproachParticipant-Oriented ApproachHeretofore, the human element was missing Heretofore, the human element was missing

from program evaluationfrom program evaluation

This approach involves all relevant interests in This approach involves all relevant interests in the evaluationthe evaluation

This approach encourages support for This approach encourages support for representation of marginalized, oppressed representation of marginalized, oppressed and/or powerless partiesand/or powerless parties

Participant-Oriented Characteristics Participant-Oriented Characteristics Depend in inductive reasoning Depend in inductive reasoning [observe, discover, [observe, discover,

understand]understand]

Use multiple data sources Use multiple data sources [subjective, objective, [subjective, objective, quant, qual]quant, qual]

Do not follow a standard plan Do not follow a standard plan [process evolves as [process evolves as participants gain experience in the activity]participants gain experience in the activity]

Record multiple rather than single realities Record multiple rather than single realities [e.g., [e.g., focus groups]focus groups]

Participant-Oriented ExamplesParticipant-Oriented ExamplesStake’s Countenance FrameworkStake’s Countenance Framework

Description and judgmentDescription and judgmentResponsive EvaluationResponsive Evaluation

Addressing stakeholders’ concerns/issuesAddressing stakeholders’ concerns/issuesCase studies describe participants’ behaviorsCase studies describe participants’ behaviors

Naturalistic EvaluationNaturalistic EvaluationExtensive observations, interviews, documents and Extensive observations, interviews, documents and

unobtrusive measures serve as both data and unobtrusive measures serve as both data and reporting techniquesreporting techniques

CredibilityCredibility vs. internal validity vs. internal validity (x-checking, triangulation)(x-checking, triangulation)

ApplicabilityApplicability vs. external validity vs. external validity (thick descriptions)(thick descriptions)

AuditabilityAuditability vs. reliability vs. reliability (consistency of results)(consistency of results)

ConfirmabilityConfirmability vs. objectivity vs. objectivity (neutrality of evaluation)(neutrality of evaluation)

Participatory EvaluationParticipatory EvaluationCollaboration between evaluators & key organiz-Collaboration between evaluators & key organiz-

ational personnel for practical problem solvingational personnel for practical problem solvingUtilization-Focused EvaluationUtilization-Focused Evaluation

Base all decisions on how everything will affect useBase all decisions on how everything will affect useEmpowerment EvaluationEmpowerment Evaluation

Advocates for societies’ disenfranchised, voiceless Advocates for societies’ disenfranchised, voiceless minoritiesminorities

Advantages: training, facilitation, advocacy, Advantages: training, facilitation, advocacy, illumination, liberationillumination, liberation

Unclear how this approach is a unique participant-Unclear how this approach is a unique participant-oriented approachoriented approach

Argued in evaluation that it is not even ‘evaluation’ Argued in evaluation that it is not even ‘evaluation’

Participant-Oriented Pros/ConsParticipant-Oriented Pros/ConsStrengthsStrengths: emphasizes human element, gain : emphasizes human element, gain

new insights and theories, flexibility, attention to new insights and theories, flexibility, attention to contextual variables, encourages multiple data contextual variables, encourages multiple data collection methods, provides rich, persuasive collection methods, provides rich, persuasive information, establishes dialogue with and information, establishes dialogue with and empowers quiet, powerless stakeholdersempowers quiet, powerless stakeholders

WeaknessesWeaknesses: too complex for practitioners : too complex for practitioners (more for theorists), political element, (more for theorists), political element, subjective, “loose” evaluations, labor intensive subjective, “loose” evaluations, labor intensive which limits number of cases studied, cost, which limits number of cases studied, cost, potential for evaluators to lose objectivitypotential for evaluators to lose objectivity

Participant-Oriented QsParticipant-Oriented QsWhat current program are you involved in that What current program are you involved in that

could benefit from this type of evaluation?could benefit from this type of evaluation?

Who are the stakeholders?Who are the stakeholders?

Alternative Approaches SummaryAlternative Approaches Summary

Five cautions about collective evaluation Five cautions about collective evaluation conceptions presented so farconceptions presented so far

1) Writings in evaluation are not models/theories1) Writings in evaluation are not models/theoriesEvaluation is a transdiscipline Evaluation is a transdiscipline (not yet a distinct discipline)(not yet a distinct discipline)

““Theoretical” underpinnings in evaluation lack Theoretical” underpinnings in evaluation lack important characteristics of most theoriesimportant characteristics of most theories

Information shared is: sets of categories, lists of Information shared is: sets of categories, lists of things to think about, descriptions, etc.things to think about, descriptions, etc.

2) “Discipleship” to a single ‘model’ is dangerous2) “Discipleship” to a single ‘model’ is dangerousUse of different approaches as heuristic tools, each Use of different approaches as heuristic tools, each

appropriate for the situation, recommendedappropriate for the situation, recommended

3) Calls to consolidate evaluation approaches into 3) Calls to consolidate evaluation approaches into a single model are unwisea single model are unwiseThese efforts based in attempts to simplify These efforts based in attempts to simplify

evaluationevaluationApproaches are based on widely divergent Approaches are based on widely divergent

philosophical assumptionsphilosophical assumptionsDevelopment of a single omnibus model would Development of a single omnibus model would

prematurely close a divergent phase in the fieldprematurely close a divergent phase in the fieldJust because we can does not mean we should; Just because we can does not mean we should;

would evaluation be enriched by synthesizing the would evaluation be enriched by synthesizing the multitude of approaches into a few guidelines?multitude of approaches into a few guidelines?

4) The choice of an evaluation approach is not 4) The choice of an evaluation approach is not empirically basedempirically basedSingle most important impediment to development Single most important impediment to development

of more adequate theory and models in evaluationof more adequate theory and models in evaluation

5) Negative metaphors underlying some 5) Negative metaphors underlying some approaches can cause negative side effectsapproaches can cause negative side effectsMetaphors shared in ch. 3 are predicated on Metaphors shared in ch. 3 are predicated on

negative assumptions in two categories:negative assumptions in two categories:Tacitly assume something is wrong in system Tacitly assume something is wrong in system

being evaluated (short-sighted indictment)being evaluated (short-sighted indictment)Based on assumptions that people will lie, evade Based on assumptions that people will lie, evade

Qs or withhold information as a matter of courseQs or withhold information as a matter of course

Alternative Approaches’ ContributionsAlternative Approaches’ ContributionsApproaches shared in ch. 4-8 influence evaluation Approaches shared in ch. 4-8 influence evaluation

practices in important wayspractices in important ways Help evaluators think diverselyHelp evaluators think diversely Present & provoke new ideas/techniquesPresent & provoke new ideas/techniques Serve as mental checklists of things to consider, Serve as mental checklists of things to consider,

remember, or worry aboutremember, or worry about Alternative approaches’ heuristic value is very high, Alternative approaches’ heuristic value is very high,

but their prescriptive value is less sobut their prescriptive value is less so Avoid mixing evaluation’s philosophically incompatible Avoid mixing evaluation’s philosophically incompatible

‘oil/water’ approaches; eclectic use of alternative ‘oil/water’ approaches; eclectic use of alternative approaches can be advantageous to high-quality approaches can be advantageous to high-quality evaluation practicesevaluation practices

Table 9.1Table 9.1

ExerciseExerciseClearly identify your evaluandClearly identify your evaluand

Is it a program, policy, product, service, other?Is it a program, policy, product, service, other?Who does it (or should it) serve?Who does it (or should it) serve?Who is in charge of it?Who is in charge of it?

Find a partner and explain what you have writtenFind a partner and explain what you have writtenDoes it make sense?Does it make sense?Does it match what you wrote?Does it match what you wrote?Does it avoid specifying criteria?Does it avoid specifying criteria? Is it simple enough?Is it simple enough?Did you avoid commenting on the merits of the Did you avoid commenting on the merits of the

evaluand?evaluand?