Download - Meta-Evaluation
![Page 1: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/1.jpg)
Meta–Evaluation Theory:Definitions, Processes,
& Systems
David PassmoreWorkforce Education & Development
![Page 2: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/2.jpg)
Meta — from Greek: μετά = “after", "beyond", "with", “adjacent”, “self”
A prefix used…indicate a concept which is an abstraction from another concept.
What is meta–evaluation
![Page 3: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/3.jpg)
Educational Evaluation and Decision Making ◦1971◦Out of print
Conceived in reaction to the difficulty and practical inconsequence of:◦Applying experimental and quasi–experimental designs◦In educational and work settings that rarely afford
opportunities for attaining the gold standard of random assignment of subjects to experimental conditions
From Phi Delta Kappa Committee on Evaluation
![Page 4: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/4.jpg)
PDK Committee Definition
A process for delineating, obtaining, and providing useful information for judging decision alternatives.
Evaluation is….
![Page 5: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/5.jpg)
PDK Committee DefinitionA dissection of the definition
A process for delineating, obtaining, and providing useful information for judging decision alternatives.
Evaluation is….
![Page 6: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/6.jpg)
An activity subsuming many methods and involving a number of steps or operations.
A process…
A process for delineating, obtaining, and providing useful information for judging decision alternatives.
Evaluation is….
![Page 7: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/7.jpg)
Two or more different actions that might be taken in response to some situation requiring altered action.
Decision alternatives…
A process for delineating, obtaining, and providing useful information for judging decision alternatives.
Evaluation is….
![Page 8: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/8.jpg)
Situations include: (a) unmet needs exist; (b) some barrier to fulfillment of needs exists; or (c) opportunities exist which ought to be exploited.
Decision alternatives…
A process for delineating, obtaining, and providing useful information for judging decision alternatives.
Evaluation is….
![Page 9: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/9.jpg)
Quantitative or qualitative data about entities (tangible or intangible) and their relationships in terms of some purpose.
Information…
A process for delineating, obtaining, and providing useful information for judging decision alternatives.
Evaluation is….
![Page 10: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/10.jpg)
Information is derived from many sources and methods. Could be from scientific data, precedence, or experience.
Information…
A process for delineating, obtaining, and providing useful information for judging decision alternatives.
Evaluation is….
![Page 11: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/11.jpg)
Information is more than a collection of facts and data. Rather, facts and data must be organized for intelligibility and must reduce the uncertainty that surrounds decision making.
Information…
A process for delineating, obtaining, and providing useful information for judging decision alternatives.
Evaluation is….
![Page 12: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/12.jpg)
Information often is limited and imperfect in evaluation situations in comparison with experimental situations in which controls are possible and are formalized by rigorous design.
Information…
A process for delineating, obtaining, and providing useful information for judging decision alternatives.
Evaluation is….
![Page 13: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/13.jpg)
Identifying evaluative information required through an inventory of the decision alternatives to be weighed and the criteria to be applied in weighing them.
Delineating…
A process for delineating, obtaining, and providing useful information for judging decision alternatives.
Evaluation is….
![Page 14: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/14.jpg)
Knowing at least two elements is essential: (a) decision alternatives and (b) values or criteria to be applied. These are obtained only from clients for evaluations who must make or ratify decisions.
Delineating…
A process for delineating, obtaining, and providing useful information for judging decision alternatives.
Evaluation is….
![Page 15: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/15.jpg)
Therefore, evaluation has an “interface” role as well as a “technical” role because its worth is defined by meeting client needs.
Delineating…
A process for delineating, obtaining, and providing useful information for judging decision alternatives.
Evaluation is….
![Page 16: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/16.jpg)
Collecting, organizing, and analyzing information through such formal means as observation, review of artifacts, measurement, data processing, and statistical analysis.
Obtaining…
A process for delineating, obtaining, and providing useful information for judging decision alternatives.
Evaluation is….
![Page 17: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/17.jpg)
Ways of obtaining information are guided by scientific criteria and disciplinary preferences.
Obtaining…
A process for delineating, obtaining, and providing useful information for judging decision alternatives.
Evaluation is….
![Page 18: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/18.jpg)
Fitting information together into systems and subsystems that best serve the purposes of the evaluation, and reporting the information to the decision maker.
Providing…
A process for delineating, obtaining, and providing useful information for judging decision alternatives.
Evaluation is….
![Page 19: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/19.jpg)
Involves interaction between the evaluator and the various audiences for information. Multiple audiences are common.
Providing…
A process for delineating, obtaining, and providing useful information for judging decision alternatives.
Evaluation is….
![Page 20: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/20.jpg)
The direct audience for information is the decision maker. Also important are people and organizations who must ratify decisions made as are others who have strong stakeholder positions.
Providing…
A process for delineating, obtaining, and providing useful information for judging decision alternatives.
Evaluation is….
![Page 21: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/21.jpg)
The information delivery preferences of these audiences vary in specificity, modality, and timing.
Providing…
A process for delineating, obtaining, and providing useful information for judging decision alternatives.
Evaluation is….
![Page 22: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/22.jpg)
Satisfying criteria for evaluation and matched to the judgment criteria to be employed in choosing among decision alternatives.
Useful…
A process for delineating, obtaining, and providing useful information for judging decision alternatives.
Evaluation is….
![Page 23: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/23.jpg)
Criteria to be imposed on evaluations include: scientific criteria (internal and external validity, reliability, objectivity);
Useful…
A process for delineating, obtaining, and providing useful information for judging decision alternatives.
Evaluation is….
![Page 24: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/24.jpg)
practical criteria (relevance, importance, scope, credibility, timeliness, and pervasiveness);
Useful…
A process for delineating, obtaining, and providing useful information for judging decision alternatives.
Evaluation is….
![Page 25: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/25.jpg)
and prudential criteria (efficiency).
Useful…
A process for delineating, obtaining, and providing useful information for judging decision alternatives.
Evaluation is….
![Page 26: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/26.jpg)
Must meet the client’s identified information needs.
Useful…
A process for delineating, obtaining, and providing useful information for judging decision alternatives.
Evaluation is….
![Page 27: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/27.jpg)
The act of choosing among decision alternatives.
Judging…
A process for delineating, obtaining, and providing useful information for judging decision alternatives.
Evaluation is….
![Page 28: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/28.jpg)
Judging is central because evaluation is meant to serve decision making.
Judging…
A process for delineating, obtaining, and providing useful information for judging decision alternatives.
Evaluation is….
![Page 29: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/29.jpg)
Although judging is central to evaluation, the act of judging is not central to an evaluator’s role.
Judging…
A process for delineating, obtaining, and providing useful information for judging decision alternatives.
Evaluation is….
![Page 30: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/30.jpg)
PDK Committee Definition
A process for delineating, obtaining, and providing useful information for judging decision alternatives.
Evaluation is….
![Page 31: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/31.jpg)
Evaluation process serves decision–making
![Page 32: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/32.jpg)
Types of decisions are linked to types of evaluation
![Page 33: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/33.jpg)
Planning decisions Context evaluation—This type of evaluation determines goals and objectives; defines the relevant environment, its actual and desired conditions, its unmet needs and unused opportunities, and why needs and opportunities are not being met and used; examines contingencies that pressure and promote improvements; and assesses congruities between actual and intended performance.
Types of decisions are linked to types of evaluation
![Page 34: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/34.jpg)
Structuring decisions Input evaluation—Essentially, this type of evaluation helps state objectives operationally and whether their accomplishment is feasible.
Types of decisions are linked to types of evaluation
![Page 35: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/35.jpg)
Implementing decisions Process evaluation—This type of evaluation is used to: identify and monitor potential sources of failure in an activity; to service preprogrammed decisions that are to be made during the implementation of an activity; and to record events that have occurred so that “lessons learned” can be delineated at the end of an activity. Process evaluation assesses the extent to which procedures operate as intended.
Types of decisions are linked to types of evaluation
![Page 36: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/36.jpg)
Recycling decisions Product evaluation—This type of evaluation measures criteria associated with the objectives for an activity, compares these measurements with predetermined absolute or relative standards, and makes rational interpretations of these outcomes using recorded context, input, and process information. Product evaluation investigates the extent to which objectives have been, or are being, attained.
Types of decisions are linked to types of evaluation
![Page 37: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/37.jpg)
C = Context
I = Input
P = Process
P = Product
PDK model is often called “CIPP” model
![Page 38: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/38.jpg)
General work breakdown for evaluation
![Page 39: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/39.jpg)
Delineating informationWho needs what when?
![Page 40: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/40.jpg)
Obtaining informationHow?
![Page 41: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/41.jpg)
Providing informationTo whom and in what form?
![Page 42: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/42.jpg)
Providing informationTo whom and in what form?
![Page 43: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/43.jpg)
Scientific criteria—
Practical criteria
Prudential criteria
Evaluating the evaluation…More expansive than evaluating quality of research
Evaluation is, itself, evaluated….
![Page 44: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/44.jpg)
These criteria primarily assess “representational goodness;” that is, they assess how well the evaluation depicts a situation. These criteria, in their detailed and technical forms, are quite familiar to most people with research training.
Scientific criteria…More expansive that evaluating quality of research
Evaluation is, itself, evaluated….
![Page 45: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/45.jpg)
Internal validity
External validity
Reliability
Objectivity
Scientific criteria…More expansive that evaluating quality of research
Evaluation is, itself, evaluated….
![Page 46: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/46.jpg)
Extent to which evaluation findings can be attributed to the activity evaluated rather than to some other factor or artifact.
Internal validity…
Evaluation is, itself, evaluated….
![Page 47: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/47.jpg)
The extent to which evaluation findings are generalizable over people, places, and time.
External validity…
Evaluation is, itself, evaluated….
![Page 48: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/48.jpg)
Simply said, the extent to which evidence is measured accurately.
Reliability…
Evaluation is, itself, evaluated….
![Page 49: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/49.jpg)
Intersubjectivity in the sense of agreement on conclusions between independent observers of the same evidence.
The “Do you see what I see?” phenomenon.
Objectivity…
Evaluation is, itself, evaluated….
![Page 50: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/50.jpg)
Intersubjectivity in the sense of agreement on conclusions between independent observers of the same evidence.
The “Do you see what I see?” phenomenon.
Objectivity…
Evaluation is, itself, evaluated….
![Page 51: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/51.jpg)
Practical criteria…
Relevance
Importance
Scope
Credibility
Timeliness
Persuasiveness
Evaluation is, itself, evaluated….
![Page 52: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/52.jpg)
Evaluation information is obtained to serve decision making.
If the information obtained does not match these decisions, then this information, no matter how well obtained scientifically, is useless.
Relevance…
Evaluation is, itself, evaluated….
![Page 53: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/53.jpg)
Information needs to be more than nominally relevant to a decision.
Importance…
Evaluation is, itself, evaluated….
![Page 54: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/54.jpg)
Not all information is important. Information with the highest importance (relevance graded by quality) must be obtained and provided, within budget constraints.
Importance…
Evaluation is, itself, evaluated….
![Page 55: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/55.jpg)
The evaluator must determine what the client believes is important. In some cases, the evaluator might suggest to the client what information ought to be considered important because the evaluator often has considerable experience with obtaining and using some types of information.
Importance…
Evaluation is, itself, evaluated….
![Page 56: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/56.jpg)
This is a “completeness” criterion.That is, is all information obtained and provided that
is needed to make a decision?
Scope…
Evaluation is, itself, evaluated….
![Page 57: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/57.jpg)
Clients often are not in a position to judge whether information obtained and provided by an evaluator meets scientific criteria. Therefore, the trust invested in the evaluator by the client is an important dimension of the quality of the outcomes of an evaluation project.
Credibility…
Evaluation is, itself, evaluated….
![Page 58: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/58.jpg)
Just like the scientific criterion of internal validity pertains to a particular evaluation situation, credibility is never generalizable and always refers to a particular situation.
Credibility…
Evaluation is, itself, evaluated….
![Page 59: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/59.jpg)
◦Information from evaluation projects is provided to identified audiences. And, this information is meant to be used.
◦The criterion of pervasiveness is met if all of the people and organizations who should, do, in fact, know about and use evaluative information.
Persuasiveness…
Evaluation is, itself, evaluated….
![Page 60: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/60.jpg)
Proper application of practical criteria of relevance, importance, and scope should remedy many inefficiencies in evaluation.
However, the conduct of an evaluation project must be weighed against alternative evaluation designs that could have achieved the same outcome with different time, financial, and personnel resources.
Prudential criterion…
Evaluation is, itself, evaluated….
![Page 61: Meta-Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022070519/58f3443d1a28ab223d8b464f/html5/thumbnails/61.jpg)
Meta–Evaluation Theory:Definitions, Processes,
& Systems
David PassmoreWorkforce Education & Development