Chapter 8:Participant-Oriented Evaluation ApproachesBy: Jason Artz, Andrea Hogentogler, Stacey Knerr, and Robert Shaw

SAK

Orienting Question #1:Who were the historic leaders in establishing participant-oriented

approaches and what did they contribute?

• Robert Stake: Responsive Approach– 1967 – The Countenance of Educational Evaluation

• Two basic acts of evaluation are description and judgment• Created evaluation framework on p. 192

– 1973 – Responsive Evaluation• “What people do naturally in evaluating things. They

observe and react” (p. 193)– 1975a, 1975b, 1978, 1980 – subsequent conceptions

of responsive evaluation• Flexible, changing methods & approaches• Multiple realities• Local is key

SAK

Orienting Question #1 cont.• Egon Guba and Yvonna Lincoln: Naturalistic and

Fourth-Generation Evaluation– 1985 – Naturalistic Inquiry

• Consider qualitative, naturalistic methods, including interviews, observations, and case studies

• “The naturalistic evaluator…observes the program and its actions, its participants and staff, in its natural environment and, through observations, documents and records, interviews, and unobtrusive measures, comes to understand and describe the program.” (p. 198)

– 1989 – Fourth-Generation Evaluation• Evaluators no longer simply measuring, describing, judging• Argued for more active roles for stakeholders & less neutral

roles for the evaluator

SAK

Orienting Question #2:What prompted their work?

• 1967 – Several evaluation theorists began to react to what they considered to be the dominance of mechanistic and insensitive approaches to evaluation in the field of education– Argued that human element was missing

• 1970s – Stressed first hand experience with program activities and settings and involvement of program participants, staff, and managers in evaluation

SAK

Orienting Question #3:How do the practical participative approaches differ from the

transformative ones?

Orienting Question #4:Contrast the Practical Participatory Evaluation, Empowerment

Evaluation, Development, and Deliberative Democratic Evaluation approaches. How do their purposes and methods

differ?

SAK

Practical participative approaches involve stakeholders in an attempt to increase use of the evaluation results based on the view that involvement gives stakeholders, the potential users, a sense of ownership and understanding that increases the likelihood of use. • Stakeholders: managers, staff, and policymakers• Balanced control• Practical participative approaches include:

• Stakeholder-based evaluation (evaluator retains control, and seeks input from a variety of stakeholders

• Practical Participatory Evaluation (Shared decision making, or partnerships between evaluators and key stakeholders)

• Developmental Evaluation (Shared decision making, or partnerships between evaluators and key stakeholders)

SAK

Transformative participatory approaches involve stakeholders for political reasons: to empower those stakeholders by providing them with tools in evaluation and self-determination or insights into power arrangements in their community or society, and tools to change those power arrangements. • Stakeholders: Program managers and staff• Involve stakeholders for political reasons

• To empower stakeholders

• Transformative participatory approaches include:• Empowerment Evaluation (Primary decision-making power given

to the stakeholders)• Deliberative Democratic Evaluation (Uses democratic principles to

empower each stakeholder group, and encourages full participation to help the evaluator make informed decisions)

SAK

Orienting Question #5:Under what conditions would you use each?

• Conditions to use Practical Participatory Evaluation– To achieve goals, may be used: • Primarily at the beginning and ending stages of

evaluation• Intensively throughout entire process• Helps with formative decision making through

balanced control

SAK

Orienting Question #5 cont.

• Conditions to use Empowerment Evaluation– To increase the probability of program success• Helps to assess the planning and

implementation of the program• Allows for self-evaluation of the program• Used for mainstreaming evaluation as part of

planning and management

SAK

Orienting Question #5 cont.

• Conditions to use development evaluation–Development activities–Changing the environment

SAK

Orienting Question #5 cont.

• Conditions to use deliberative democratic evaluation–When there is a need for moderation• need for dialogue among stakeholders•power differentials among participants•many stakeholders

SAK

Orienting Question #6:How are participatory approaches used in practice?

• Used to:– empower stakeholders– Build evaluation capacity in the organization– Increase organizational learning and data-based

decision making

• Can be used at beginning and end, or throughout the process

SAK

Orienting Question #7:What are the major strengths of participant-oriented approaches?

• Involving stakeholders has advantages:– They have knowledge and perspectives that evaluators do not have:

• Policy makers know what programs they are considering, the budgetary and time constraints involved, and the factors (political or otherwise) that may affect their decisions

• Program managers and staff know the details of the program and some of the struggles and successes of the students or clients. They know what they have tried before and why it did not work. They know the logic behind the current program, why they adopted it, and what elements of it they think may work. They have expertise in working with these types of clients or problems.

• The clients know the details of their own lives, the problems with which they struggle, as well as the solutions they find, and how they respond to and think about the program or policy

– Evaluators’ communication with stakeholders increases the evaluators’ understanding of the concept and ability to develop more valid measures for evaluation

– Involving stakeholders helps them gain trust in the evaluation

SAK

Orienting Question #8:What are the limitations of participant-oriented approaches?

• The feasibility, or manageability, of implementing a successful participative study– Increased time and costs– Difficulty in obtaining participation from the most

disadvantaged stakeholders– Few evaluators have extensive training or skills in

facilitation, particularly with groups that may have political differences (Need to develop “people skills” )

SAK

Orienting Question #8 cont.• The credibility of the results to those who do not participate in the

process– Other audiences wonder if stakeholders are able to be objective

about their own programs (potential for bias). • Evaluators can even become biased as they work with

stakeholders. • To avoid bias, many organizations hire external evaluators for

important, controversial, or costly programs– Competence of stakeholders to perform tasks that some

approaches require• Encouragement of organizations to improve their

performance and evaluative ways of thinking is positive; however, Peter Dahler-Larson “bemoans the ‘expectation that lay persons will become good evaluators based on a short introduction and rudimentary training in methodology’” (p. 226).

SAK