exit presentation program logic an introduction next exit presentation

Exit presentation

Program LogicAn introduction

Next

Exit presentation

Exit presentation

This guide to program logic is designed for managers in government and community organisations, and for new evaluators.

Program managers can use program logic to plan and clarify how program activities lead to intended outcomes. Evaluators can use it to define key evaluation questions and shape methods for data collection. This guide covers:

• what is program logic• developing outcomes hierarchies • how to use program logic in evaluation

(c) 2009. This guide was first published in 1994 as a computer-based tutorial. We have refined it and release it again because people have told us that they still find it useful. The original drew on the work of the NSW Premier’s Department and in particular Sue Funnell in designing program logic for public sector programs in NSW. We also acknowledge the support of Jerome Winston.

Next

Exit presentation

Exit presentation

Menu

1. What is program logic?

2. Outcomes hierarchies

3. Using generic outcomes hierarchies

4. A program logic framework

5. Identifying evaluation questions

6. References

Back to introduction

Exit presentation

Exit presentation

1. What is program logic?

We need to start with some terminology.

“Program” refers to a set of coordinated activities and resources designed to achieve a desired result. A program can be large or small, simple or complex, and have a name such as “program”, “sub-program”, “project”, “initiative” and so on. However they all have the common feature of aiming to reach a goal, objective, result or outcome.

“Program logic” refers to the causal links between the elements of a program and its intended result or outcome. It shows the assumed chain of cause and effect between the activities and the desired result. By drawing out the logic as a diagram, we have a clearer picture of how the program is assumed to work.

Next Back to menu

chris

arrows pls

Exit presentation

What is program logic?

In the evaluation literature, program logic is a form of program theory, which has been described as: • ‘a description of the mechanisms by which the program is expected to achieve its effects’ (Davidson, 2005)

• ‘an explanation of the causal links that tie program inputs to expected program outputs’ (Weiss, 1998)

• ‘mapping out a program’s inputs, activities, outputs, initial outcomes, intermediate outcomes, and longer term outcomes’ (Donaldson, 2007).

Sometimes it is called a logic model: • ‘... a way of portraying such theories concretely and visually’ (Frechtling, 2007)

• ‘...describe[s] a bounded project or initiative: both what is planned (the doing) and what results are expected (the getting). They provide a roadmap to a specific end’ (Knowlton and Phillips, 2009).

See references at the end of this guide.

Next Back Back to menu

Exit presentation


Program logic is used to represent a program to help in planning or evaluation.

It is an analytical tool and not an end in itself. A program logic can be revised and refined to reflect changing focus on aspects of a program, or actual changes to the program over time.

For evaluation, drawing a program logic diagram can help indicate the key questions to ask about a program's performance at different times in the life of the program. It can be used to shape evaluative arguments about the success or otherwise of a program.

???????????????

√√√√√√√√√√√√√


Exit presentation


Where does program logic fit in evaluation?

The evaluation of a program typically has four steps.

Identifying the program logic is usually part of the evaluation design (step 2).

It can also be used to support the conclusions in the reporting (step 4). 1.Plan the

evaluation

2.Design the

evaluation

3.Collect data

4.Analyse data and write a report


Exit presentation


Analysing the program logic can help answer these questions : •how is the program meant to work – is it feasible?• to what extent is the program able to be evaluated at this time?• is the program strategy logical and appropriate?•what are key questions for the evaluation?•what information on program performance should be collected?

1.Plan the evaluation

2.Design the

evaluation

3.Collect data

4.Analyse data and write a report


chris

change colour

Exit presentation


As a starting point for developing a program logic, it can be useful to think about a program as a simple system with the stages illustrated below.

Environment

Inputs Processes Immediate results Long-term

results


chris

Is diagram still ok or should it include activities and outputs? its OK as is

Exit presentation


Throughout this guide we will use an office skills program as a case study.

This can be represented by the diagram below.

Environment

Funds

Suitable courses

Young people

Community groups funded

Courses are run

Young people attain skills

Young people get jobs

Unemployment in the region is

reduced

Inputs Processes Immediate results

Long-term results


Exit presentation


There are many reasons why a program may not be producing the intended results, for example

•it is being implemented incorrectly or •it does not meet the needs of the target group•the measurements being used are not able to show the results•the program logic is wrong

Environment

Funds

Suitable courses

Young people


Courses are run




reduced


Long-term results


Exit presentation


Here we focus on the last reason - the program logic is wrong. In other words, some of the assumed links represented by the arrows are wrong.

In this case, attaining skills does not lead to young people getting jobs.

Environment

Funds

Suitable courses

Young people


Courses are run




reduced


Long-term results

XX

“Even if these young people have the skills, I wouldn’t give them a job unless their attitude was right.”

Employer


Exit presentation


Program logic can be visually presented in different ways.

One useful way is an outcomes hierarchy.

It shows the “outcomes” a program is intended to produce at different levels, and the assumed links between them. The causal chain flows from the bottom outcome to the top. Outcomes hierarchies are explained in the next section.

Appropriate information gathered/

developed

Information organised into accessible form

Clients exposed to information

Increased awareness

Changed knowledge

Changed attitudes

Changed behaviour

Program specific outcomes


Exit presentation


Program logic can also be presented in different ways to reflect different focus. Stakeholders may have their own ideas about how a program should work and describe the program in different ways.

Minister for Community Services

Course coordinatorProgram Manager – Department

“If we put more money into it we should get better results.”

“If these kids just turned up and did some work, they would have no trouble getting a job.”

“If the courses were run more efficiently, then more kids would be trained and get jobs.”


Exit presentation


In practice, stakeholders have their own interests which can influence the way they think about the program’s logic. Working with stakeholders in designing and refining the program’s logic can help to build a better understanding of the program’s intended goals and the strategies needed to reach them.


Program Manager

Next section Back Back to menu

Exit presentation

2. Developing outcomes hierarchies

A program can be described in terms of its intended outcomes. These outcomes can form a logical hierarchy, which can be used as a framework for evaluating the program.

Next Back to menu

Exit presentation

Developing outcomes hierarchies

Firstly consider a program’s intended results or outcomes, rather than its objectives or processes.

For example, for the office skills program (used as a case study in this guide):

An objective

To teach participants office skills

can be written as an outcome

Graduates have suitable office skills

An outcome is not a process... but a result that can be measured and observed

How many graduates have which skills?


Exit presentation


While the office skills program was originally described by the Department’s manager in terms of aims, objectives and strategies, you can see that it aims to achieve a range of outcomes at different levels.

Unemployment is reduced

Aim:

Objectives:

Strategy:

To reduce long-term unemployment for target

group

To teach participants appropriate office

procedures

To assist course graduates to gain office

work positions

Design suitable course

Fund providers

Select students

Implement course

Graduates have suitable knowledge

and skills

A suitable office skills course is prepared

Participants react favourably to course


Exit presentation


In fact, the office skills program aims to produce a whole series of outcomes. And we can see the outcomes will have logical links with each other based on cause and effect.

Graduates have suitable knowledge

and skills


For example,we have assumed this outcome has to be achieved first...

...in order to achieve the higher outcome.


Exit presentation


The outcomes for the office skills program can be drawn up and arranged into a logical chain of cause and effect, like in this diagram.

The cause and effects chain is called an outcomes hierarchy. The logic is that each lower outcome has to be achieved in order to achieve the next higher outcome.

Long-term unemployment is reduced

Graduates have suitable knowledge and skills


Appropriate people apply for the course

Graduates gain office positions

Office skills course for target group is prepared

Information about training needs of target group is available


Exit presentation


The outcomes hierarchy forms the basis of a program logic framework. Vertically, it shows the sequence of outcomes. Horizontally, it can be used to work out the details of outcomes. We will look at defining attributes, factors and performance information later in the guide (see section 4).

Outcomes Attributes of success

Factors Performance indicators

7. Long-term unemployment is reduced

6. Graduates get office jobs

5. Graduates attain office skills

4. Participants react favourably to the course

3. Appropriate people apply for course

2. Office skills course for target group is prepared

1. Information on training needs of target group is available


Exit presentation3. Generic outcomes hierarchies

Starting from a blank page to draw up an outcomes hierarchy for a program is not an easy task. Instead, a generic outcomes hierarchy can be a useful starting point.

A generic outcomes hierarchy can represent a whole range of different programs that have a common underlying logic or strategy.

This section describes some common generic outcomes hierarchies and how they can be used to develop an outcomes hierarchy for a specific program.

Next Back to menu

Exit presentation

Generic outcomes hierarchies

The Office Skills Program (the case study in this guide) can be classed as an educative program:

• it aims to influence people’s behavior to achieve its overall results

• it uses strategies involving communication and learning to change a target group’s knowledge, skills or attitude.

Outcomes









Exit presentation


The Quit Smoking Program represented by the outcomes hierarchy on the right is a very different educative program. It is a public education campaign conducted through the media. However, like the office skills program, it aims to influence people’s behaviour using educational strategies.

Outcomes

6. Decrease in smoking related disease

5. People quit smoking

4. Changed attitude to smoking in target group

3. Target group is exposed to the campaign

2. Media advertising

1. Information on smokers’ responses available


Exit presentation


Although the outcomes for each program are very different, because they use the same kind of strategy, they can be represented by the same generic outcomes at each level.

Outcomes

6. Decrease in smoking related disease

5. People quit smoking

4. Changed attitude to smoking in target group

3. Target group is exposed to the campaign

2. Media advertising

1. Information on smokers’ responses available

Outcomes








Program outcomes

Changed attitudes, skills or knowledge

Target group is exposed to the

information

Information is organised in

accessible form

Changed behaviour

Appropriate information is

available


Exit presentation


These generic outcomes at each level can be used to form a generic outcomes hierarchy, which broadly represents all educative programs.

Program outcomes

Changed attitudes, skills or knowledge

Target group is exposed to the

information

Information is organised in

accessible form

Changed behaviour

Appropriate information is

available

Program outcome

Application of new skills, attitudes and knowledge

New skills, attitudes and knowledge

Program is implemented

Workable program is designed

Worthwhile goals are established


Exit presentation


Generic outcomes hierarchies have been developed for seven types of programs, based on their overall strategy. These are:

• Educative• Regulatory• Case management• Advisory• Provision of a service/ product• Policy development• Revenue raising

These seven cover most public sector programs. Select from the list above to view a specific generic hierarchy.

Click here to skip the generic outcome hierarchies


Exit presentation


EducativeAn educative program aims to influence people’s behaviour to achieve its overall objective.

It uses strategies which expose the target group to information and educational processes.

The exposure is intended to produce changes in the target group’s knowledge, skills or attitude, and, as a result, to change their behaviour.

Appropriate information gathered/ developed


Clients exposed to information

Increased awareness

Changed behaviour


Changed knowledge, skills and/ or attitudes

Next Back to list of generic hierarchies Back to Menu

chris

make arows straight

Exit presentation


RegulatoryA regulatory program aims to influence people’s behaviour to achieve its overall objective.

It uses strategies based on deterrence which require a regulation prohibiting unwanted behaviour.

The assumption behind the program is that potential offenders will be deterred if the target group is aware of both the prohibition and the risk of being caught for violating it.

Legislation

Licensing/ registration

Appropriate inspection and investigation

Identification of infringements

Appropriate behaviour, compliance


Potential offenders are aware of the

program

Conciliation

Prosecution of people who fail

to comply

Potential offenders are

deterred


Exit presentation


Case managementA case management program aims to influence people’s behaviour to achieve its overall objective.

It treats each case (person or group) individually, and sets objectives for them on a case-by-case basis.

The assumption behind a case management program is that as each individual achieves their own objectives, the circumstance of all the target group will improve and the overall program objective will be achieved.

Reduced long-term dependence and cost to government

Life circumstances improved

Short-term objectives progressively achieved

Selected individual programs satisfactorily implemented

Realistic objectives set on a case-by-case basis

Accurate identification of needs and capacity on a

case-by-case basis

Target group access program


chris

reduce font - same as prev

Exit presentation


AdvisoryAn advisory program aims to influence people’s behaviour to achieve its overall objective.

It uses strategies that encourage individuals in the target group to seek advice about an issue or concern.

The advice is intended to change their knowledge, skills or attitudes, and as a result, to change their behaviour.

Advisory programs differ from educative programs in that the target groups seek out the advice, rather than being exposed to it.

Appropriate information available


Target group aware of advisory service

Target group seek advice

Changed behaviour


Changed knowledge

Changed attitudes


chris

ditto font

Exit presentation


Service delivery A service delivery program aims to achieve its overall objectives through providing clients with a service or product.

A service provision program may be a non-competitive monopoly where the government is the sole provider, or the government may be in competition with other providers. The government may also fund NGOs to provide the service.

The clients of the program may vary from customers in the marketplace to clients with specific entitlements. The program outcomes could range from raising revenue to improved health for a target group.

Product/ service designed to match clients’ needs

Clients know about product/ service

Clients satisfied


Clients use product/ service

Product/ service delivered efficiently

and effectively


Exit presentation


Policy developmentA policy development program aims to change government policies to achieve its objectives.

It uses consultation and research to produce policy proposals which are realistic and acceptable to government and major stakeholders.

Information about issue is available

All stakeholders identified

Policies are approved by government

Desired change occurs

Process perceived as credible by government

Stakeholders support policy

proposals

Stakeholders’ views taken into account

Proposed policies are workable

Process perceived as credible by stakeholders


Exit presentation


Revenue raisingA revenue raising program usually involves a legislative mandate, with strategies to make the target group aware of its responsibilities, and to collect the revenue.

Increased revenue

Monies collected

Licensing/registration

Clients aware of rights/ responsibilities

Legislation


Exit presentation


Once the initial hierarchy is designed, it can be improved by talking with stakeholders.

Outcomes








For example, in the case study office skills program, one stakeholder said: “Well these kids might pick up the skills, but half of them don’t want to work anyway.” And the outcomes hierarchy was refined (see diagram on the right).

Outcomes







Graduates attain office skills

Graduates have suitable attitudes


Exit presentation

Outcomes


6. Graduates develop their careers







Outcomes









Some of the participants in the office skills program said: “Now I’ve done the course, I never want to work in an office.” And the outcomes hierarchy was refined (see diagram on the right).

Refining the outcomes hierarchy helps build up a more realistic model of the program.



Exit presentation


A generic hierarchy can be a useful starting point for developing your outcomes hierarchy and for identifying the basic assumptions behind your program. Once you have talked to program staff and reviewed original program documents, you can se a generic hierarchy as a reference to help you begin drawing up your hierarchy.

In practice, many programs are more complex than a generic outcomes hierarchy.

You might find your program can be described by using two or more of the generic outcomes hierarchies.

The next section shows how to use an outcomes hierarchy to build a program logic framework


Exit presentation

4. Program logic framework

Outcomes hierarchies can be used to build evaluation frameworks by looking at the details of the outcomes at each level.










Next Back to menu

Exit presentation

A program logic framework

For example, the details of the Level 5 outcome for the Office Skills Program could be:



Level 5.Graduates have suitable knowledge, skills and attitudes

All graduates demonstrate satisfactory competencies

All feel confident to apply for office jobs

Cost per graduate is less than $700

Participants’ literacy levels

Teaching standards vary

Patterns of graduate test scores (gender, background)

Cost per graduate


Exit presentation




Attributes of successWe will look at all the elements of the evaluation framework, starting with attributes of success. In the outcomes hierarchy, each outcome has only been described by a brief statement. It is helpful to identify specific characteristics of a successful outcome. These characteristics are called attributes of success for each outcome. They describe what the outcome is expected to look like if the program is operating successfully.


Exit presentation




Attributes are defined by the needs of different stakeholders.

EmployerProgram Manager - Department

“Our pilots showed that 60% of this target group should learn the skills.”

“I’ll only employ people who know how to do the job properly.”

60% of graduates achieve satisfactory competencies


Exit presentation




Key stakeholders will look for different attributes of success.

Students attending courses

Community administrator

“Our budget assumes that cost per graduate will be less than $700.”

“We want to feel confident when we go for job interviews.”



Graduates build up their confidence in job seeking


Exit presentation


FactorsFactors are the next element in the framework – factors affect how well the outcome is achieved. One way to consider the factors affecting the program is to do a SWOT analysis.

Strengths

Weaknesses

Opportunities

Threats


Exit presentation


Outcomes Factors


Stakeholders can identify factors. For example, in the office skills program, teachers identify literacy as a weakness.

“You can’t expect all the kids to get the same office skills when half of them can’t read or write properly.”

Literacy levels of students

Course coordinator


Exit presentation


Outcomes Factors


Different stakeholders are likely to suggest different factors. For example, the participants of the office skills program identify the teachers as a factor.

Students attending courses

“The teachers in the course were no good.”


Suitability of teachers


Exit presentation


Factors


It is important to be aware of factors when it comes to evaluating programs. It enables realistic targets to be set and provides a better understanding of the program’s results. Also, once important factors are identified, program managers can plan strategies to deal with them. In this way, they can improve the program and change their assumptions about the program’s logic.



Outcomes


Exit presentation


Outcomes


In the case study office skills program, some changes might be made based on factors.

“We want to know which teachers are producing the best results. In the future we might have to select our teachers more carefully.”

Community administrator

Factors




Exit presentation


Outcomes


Another change might relate to the literacy level of students.

Factors



“To account for individual differences, we record each students’ progress separately. We are considering introducing a basic literacy course as well.”

Teacher


Exit presentation


Outcomes Factors

Level 7. Long-term unemployment is reduced

For higher level outcomes, there may be many more factors outside the control of the program. Direct links between these outcomes and the program may be difficult to see.

Rapid technological changes

Economic restructuring

Increased migration


Exit presentation


Performance informationThe attributes of success and factors identified in the last two steps give a more detailed picture of each outcome. These can be used to work out what information is needed to see whether the outcome is being achieved – the performance information. The attributes of success and factors identified in the case study suggest a need for information on:

• how well graduates learnt about office work• the quality of teachers in the program• the cost effectiveness of the course.





Graduates build up their confidence in job seeking


Participants’ literacy levels

Teaching standards vary

Patterns of graduate test scores (gender, background)

Profile of teachers

Cost per graduate


Exit presentation


Outcomes Factors

Level 6. Course graduates get jobs

The program logic framework indicates what performance information is needed to assess how well the program is achieving a specific outcome. This information can come from program monitoring or an evaluation study. However, data collection can raise issues. For example, collecting certain data could be costly, unethical, or ultimately not very worthwhile because of the impact of other factors.

“That information would be great to know, but who has time to chase up the kids?”

Patterns of graduate employment after the course

Data: % of graduates who remain in the job for 3 months (by type)Course coordinator


Exit presentation

5. Identifying key questions

In the previous section we built a program logic framework based on an outcomes hierarchy. Collecting information on all outcomes could be expensive and inefficient, and priorities need to be set. When we design an evaluation or monitoring system, the framework can be used to decide which outcomes to focus on, what questions to ask, and what information is needed. The issues to consider in deciding on the key evaluation questions to ask include: the purpose of the evaluation the balance between efficiency, effectiveness, appropriateness and equity the interests of stakeholders any issues that have arisen from monitoring the program.

Next Back to menu

chris

change "key" to "evaluation"

Exit presentation

Identifying key questions

While evaluations are used for many different purposes, most of them are a balance between:

Program improvement

Program justificationand

What changes are needed? Should the program continue?


Exit presentation


Evaluations may include questions that focus on:

Efficiency

Effectiveness

Appropriateness

Equity

Adequacy

Outputs achieved related to inputs.

The extent that outcomes match what was intended.

How well the program aimed to meetthe problem it set out to address;the needs of the client group;and how well it fit government priorities.

How fair the program was to all people in the target group.

Whether the response was sufficient given the problem or need.


Exit presentation


Stakeholders might see different purposes for an evaluation. For example, in the office skills program, the program manager wants the evaluation to answer the following questions.

Program Manager - Department

“I want to know how well the program is being run, and how we can do better next year.”

“Are my budget targets being met?”

“Are all the kids getting a fair chance?”

Program improvement

Efficiency

Equity


Exit presentation


While the course coordinator wants to know about:

Effectiveness

“I wonder if the course structure is right. Would it be improved by more work experience time?

Course coordinator

Program improvement


Exit presentation


The Minister for Communities wants to know about:

Effectiveness

“Most of the cabinet thinks that these programs are a waste of money. Some say that government should not be involved at all.

I want to know whether these programs actually have any impact on unemployment.

And I want some good success stories to show the media.”

Minister for Community Services Appropriateness

Program justification


Exit presentation


The educational consultant wants to know about:

Effectiveness

“Classroom-based programs can’t give people real life skills, and using an educational model just won’t work with that target group.”

Educational consultant Appropriateness

Program justification


Exit presentation


The purpose of the evaluation will direct you to relevant questions. At the same time, different stakeholders will have their own interests.

“What is our cost for each person who gets a job?”


Outcomes








“Are the right people applying?”


Exit presentation


Outcomes








“Does our course cover all of the competencies you need for office work?”

Course coordinator


Exit presentation


Outcomes








“Will these kids have the right skills and attitudes?”

Employer


Exit presentation


Outcomes








“What are the changes in youth unemployment levels?”


“How many got jobs?”


Exit presentation


Stakeholders’ interests may vary, and be at different levels of the hierarchy.

Outcomes








Politicians and senior management are usually interested in the higher level outcomes where they are seen as accountable.


Program staff may want to focus on lower level outcomes where they are familiar and have more control over the results.

Course coordinator


Exit presentation


Outcomes








When you are planning an evaluation, the

framework makes it easier to consider a range of stakeholder

viewpoints


Exit presentation


Outcomes Performance information

Level 6. Course graduates get jobs

Regular program monitoring can also raise further questions for an evaluation.

“Our records show that 80% of the graduates who get jobs are men. Yet, equal numbers of men and women enrol. What’s going on?”

% of graduates who remain in the job for 3 months


End Back Back to menu

Exit presentation

You have reached the end of this tutorial – you can return to the main menu to review of the sections, continue to a list of references, or exit the presentation.

References Back to menu

Exit presentation

References

Frechtling, J.A, 2007, Logic Modeling Methods in Program Evaluation, Wiley, Inc., Jossey-Bass, San Francisco Torvatn, H., 2008, Book review (Frechtling, Logic Modeling Methods in Program Evaluation), Evaluation and Program Planning

W.K. Kellogg Foundation, 2001, Logic Model Development Guide, Using Logic Models to Bring Together Planning, Evaluation & Action

Wyatt Knowlton, L and Phillips, C 2008, The Logic Model Guidebook, SAGE

Rogers, P.J., Hacsi, T.A., Petrosino, A. & Huebner, T.A (editor), 2000. Program Theory in Evaluation Challenges and Opportunities: New Directions for Evaluation, No. 87, Jossey-Bass.

Weiss, Carol H. 1998, Evaluation: Methods for Studying Programs & Policies - 2nd edition, Prentice Hall

Davidson, D.E.J., 2004. Evaluation Methodology Basics: The Nuts and Bolts of Sound Evaluation 1st ed., Sage Publications, Inc.

Back to menu

exit presentation program logic an introduction next exit presentation

Documents