Suki Goodman

School of Management Studies

University of Cape Town

Evaluating Programme Theory

a case illustration of the value and challenges in conducting a

theory evaluation

Evaluating Programme Theory:

Outline of Presentation• Aim

• A little about theory evaluations: what we hope to achieve

• Brief introduction to programme

• The logic model/causal logic

• Brief examples of attempts at theory evaluation

• Practical problems and challenges

AimThe aim of the presentation is to give

snapshot of some the processes, results, relevance and challenges of attempting to conduct a theory evaluation of a judicial education programme

Theory Evaluations Rossi, Lipsey & Freeman, 2004

Evaluation at this level of the evaluation hierarchy assesses if the causal logic implicit in the programme is practically realistic and theoretically sound (Donaldson, 2003)

The programme theory is analysed to assess the feasibility of the stated objectives of the intervention

Evaluating the programme theory entails addressing a variety of evaluative questions such as:

1. Are programme goals and objectives well defined?

2. Is there alignment between the programme theory and a documented social need?

3. How well does the programme theory compare with research and practice

4. Is the programme logic feasible and plausible? (Rossi et al., 2004)

There are a number of steps in a theory evaluation

1. evaluator has to extract a logic model of the training programme

2. assess if it represents the key stakeholders’ understanding of the underlying causal processes

3. outline historical conflicts/problems in the conceptualising of the programme which might be reflected in the final product

Brief introduction to programme

• Law, Race and Gender (LRG) Unit’s programme of social context training.

• The LRG Unit housed at the University of Cape Town’s Law Faculty

• Unit has been conducting social context training with South African magistrates since the mid 1990s

Law, Race & Gender Unit

Established 1994 as a research and training unit

Social context training:

Provide different views and experiences of the world that people have

Explore ways of understanding different attitudes and social realities

Enable participants to recognise and deal more sensitively and fairly with cases

LRG Approach

Judicial education in three dimensions to be effective:

Content – substantive lawCraft – Skills enhancementSocial Context

InputProximal outcomes

Distal outcomes: Individual

Distal Outcomes: Organisation

Workshop/training intervention

Change in knowledge, awareness, attitudes and understanding

Change in work behaviour:Using new knowledge on the job

Improved administration and delivery of fair and equal justice appropriate

Fig. 1 Programme Logic Model

Is the programme logic feasible and plausible?

The feasibility and plausibility of the sequence linking or causal logic implied in the programme will be analysed by

(1) a critical discussion of how key concepts are operationalised in the training and

(2) interrogating the causal relationships implied in the programme with social psychological knowledge about the attitude behaviour relationship  

The Operational Definitions of Key Concepts

The programme’s documentation gives scant attention to how racism or prejudice is defined or understood bar a relatively crude explanation related to :

1) redressing past inequality and prejudicial treatment

2) discriminatory laws associated with the apartheid regime

What we know about modern-day racism is that it operates on two levels:

1. traditional blatant racism 2. more subtle negative feelings of

which the individual is unaware (Akrami, Ekehammar & Araya, 2000; Dovidio, 2001, McConahay, 1986; Meertens & Pettigrew, 2001; Sears, 1998)  

• Latter that causes theory problems for the programme

• Training attempts to raise magistrates’ levels of consciousness about their stereotypes, biases and discriminatory practices

• Not structured around, or designed to penetrate, seemingly intractable deep-seated implicit attitudes

The attitude-behaviour relationship

The causal logic of the theory suggests: training will bring about change in magistrate’s prejudicial attitudes which will in turn bring about changes in their behaviour  

The relationship between attitude and behaviour is complex and nuanced (Ajzen, 1985,1987, 1988, 1991; Ajzen & Fishbein, 1980; Albarracin & Wyer, 2000; Bagozzi, 1992, Bohner & Wanke, 2002; Eagly & Chaiken, 1993; Eiser, 1994; Eiser & van der Pligt, 1988; Foster & Nel, 1991, Kraus, 1995).

Historically there has been ongoing controversy about the casual links in this relationship

The Causal Debate

• A contemporary conceptualisation of the attitude-behaviour relationship posits that it can best be understood as a “substantive relationship of interest” (Kraus, 1995, p.71)

• A relationship which will vary in strength according to context and is influenced by a variety of variables

Modern theorists agree that there is no straightforward answer to the theoretically, and practically, complicated question, does attitude predict future behaviour? (Bohner & Wanke, 2004; Eagly, 2004)

Using theory to evaluate the programme

• Theory of Reasoned Action (Ajzen & Fishbein, 1980; Ajzen & Fishbein, 1975)

• Theory of Planned Behaviour(Ajzen, 1985,1987, 1988, 1991, Ajzen & Madden, 1986)

Attitude

Subjective norm

Perceived behavioural control

Intention Behaviour

Fig. 2 Theory of Planned Behaviour (Version 1 without broken arrow, Version 2 with broken arrow)

Challenges

• Evaluability assessments• Eliciting programme theory:

Defining boundaries• Identifying causal logic• Academic pursuit• Practicality for “real” world evaluationsImplications of flawed logic for programme

staff