evaluating behaviour change programs

Evaluating behaviour change programs

Liz AmptConcepts of Change

WasteMinz Roundup 2014was WasteMINZ Roundup 2014

What are we measuring?

- Whether people are

doing things differently

http://www.google.com.au/url?sa=i&rct=j&q=&esrc=s&source=images&cd=&cad=rja&uact=8&docid=cvTi_WJPxJ432M&tbnid=8ERVQCkt7PRh3M:&ved=0CAUQjRw&url=http://harmony1.com/recycling-waste-reduction-equipment/&ei=KPswU73PDcW8kQWao4DQDQ&bvm=bv.63587204,d.dGI&psig=AFQjCNHcd83oWHZN7_Yj6m-wN2O_njqQDA&ust=1395805299255430


How can we measure it?

- Measuring changes in levels of reducing/ reusing/ recycling

- Observing/recording behaviours- Asking about change

- Each is a form of survey or data collection exercise



SampleDesign

PilotSurvey

Conductof

Survey

DataCoding Data

Editing

Data Managementand Analysis

Data Correctionand Expansion

Presentationof ResultsTidying-Up

Selection ofSurvey Method

SurveyInstrument

Design

PreliminaryPlanning

The survey processRichardson, Ampt, Meyburg 1995



Each is a form of survey Method of measurement Aspects of survey design needed

Measuring changes in levels of reducing/ reusing/ recycling

- Sample selection- Pilot- Survey- Expansion/weighting- Analysis

Observing/recording behaviours - Sample selection- Pilot- Survey- Expansion/weighting- Analysis

Asking about change - Sample selection- Survey design- Pilot- Survey- Expansion/weighting- Analysis



Key elements of survey design

1. Preliminary planning2. Selection of method3. Sample design4. Survey instrument design5. Pilot6. Survey implementation7. Expansion/weighting• Analysis – over to you..



1. Preliminary planning

• Define survey objectives– Very specific: what, by whom, over what period, where

• Review of existing information– Useful methodologies from elsewhere; use of stated

preference?• Define terms

– From your objectives and from respondent’s perspective• Survey content

– Dot points

7



Exercise

• Think of 2 terms you would want to use in a survey related to your work– Write down clear definitions for both– Ask another person (not from your organisation) to do the

same with your terms– Compare



2. Selection of a Survey Method for Measurement

• Observation surveys• Intercept surveys• Self-administered surveys• Telephone surveys• Personal interview surveys• Internet/online surveys

9



Observation methods

• Chosen when – Possible to count accurately– Possible to count all or select a representative sample

• Can be manual, automatic, video

10



Intercept Surveys

• Intercepting people – At an activity centre (e.g. workplace, transfer station,

shopping centre)• Possible methods

– distribution - mail-back/on-line– personal interview– collect phone no.

• Always needs a total classification count

11



Intercept Surveys

• Advantages– Able to reach specific populations– Can combine with observational counts– Can use multiple survey methods

• Disadvantages– Generally low response rates (20-30% for self-completion)– Hurried conditions– Must allow for non-random sampling– No follow-up possible in most cases

12



Self-administered Surveys

• Possible targets– households– activity centres/workplaces/transfer stations

• Method of Distribution– mail-out vs. hand delivered

• Method of Collection– mail-back vs. hand collection

13



Self-administered Surveys

• Advantages– Can get extensive geographical coverage– No interviewer effects– Can obtain considered responses– Hand-collection to good response rate

• Disadvantages– Layout and wording must be clear - hard to design– No probing possible– Answers not independent– Response rates lower than face to face

14



Telephone Interviews

• Advantages– Wide geographic coverage– Intermediate costs – Good supervision - CATI– Multilingual capabilities– Computerised



Telephone Interviews

• Disadvantages– Sample usually weak

• Low phone ownership for some groups • Answer-phones, mobile phones, screening devices• Hard to know how it represents the population

– Credibility of interviewer (confusion with telemarketing)– Low response rate– No follow-up for refusals



Personal Interviews

• Can be paper or computer• Advantages

– Generally higher response rates (60-90%)– Flexibility of information– Presence of interviewer– Maintain interest– Spontaneous answers



Personal Interview

• Disadvantages– High costs– Interviewer influence

• personal characteristics• interrupt household/work routine• opinions of interviewers• interpretation of vague answers

– Considered response difficult



On-line Surveys

• Advantages– Low costs– Can use elaborate visual effects– Can use adaptive techniques (can give different scenarios

for different responses)– Good for workplaces if sufficient follow-up

• Disadvantages– Usually very biased sample– Low response rate– Hard to get all people in household if needed



Exercise

• Think of a behaviour you would like to measure• Discuss with a partner

– Best method of collection• Strengths• Weaknesses



3. Sample Design in the Survey Process

SampleDesign

PilotSurvey

Conductof

Survey

DataCoding Data

Editing





SurveyInstrument

Design

PreliminaryPlanning

Sampling Methods



Preliminary Concepts

• What is a sample?– a collection of things which is some part of a larger

population and which is selected so as to be representative of some or all of that population

• Target Population– who are we trying to survey?

• Sampling Units– what are we going to sample?

• Sampling Frame– where are we going to get a list of these things?

Sampling Methods



Sampling Frame

• a base list to identify the sampling units• should contain all the sampling units• examples,

– all households on a street (e.g. Council records)– telephone directories– mailing lists– maps– electoral rolls– blocklists

Sampling Methods



Sampling Frame Problems

• inaccuracy• incompleteness• duplication• inadequacy• out-of-date

• Must check the reason for which the list was originally compiled to understand likely deficiencies.

Sampling Methods



Sampling Error & Sampling Bias

• Sampling Error– due to the simple fact that we are taking a sample, and not

the population. – can minimise error by taking larger sample.

• Sampling Bias– due to systematic omission of some elements from our

final sample.– cannot minimise error by taking larger sample.

Sampling Methods



Random Sampling

• Each unit is selected independently and each unit in the population has an equal probability of being selected.

• Must use random sampling to avoid sampling bias.

Sampling Methods



Random Sampling Methods

• Simple Random Sampling

• Stratified Random Sampling

• Variable Fraction Stratified Random Sampling

• Multi-stage Sampling

• Cluster Sampling

• Systematic Sampling

• Note quotas not on list

Sampling Methods



Sample Size

• How much data do we need?

• Too much data >>> too expensive• Not enough data >>> not able to draw conclusions

• Somewhere in the middle is a sample size which enables us to draw sufficient conclusions at a reasonable cost

Sampling Methods

Stopher, P. (2012) Collecting, Managing and Assessing Data Using Sample Surveys , Cambridge.



Exercise

• Hand out random sampling sheet– Explain– Questions?



4. Survey design in the survey process

SampleDesign

PilotSurvey

Conductof

Survey

DataCoding Data

Editing





SurveyInstrument

Design

PreliminaryPlanning



Instrument design for reliable measurement

• Question content

• Question types

• Physical design - also for observation/counting



Question Content

• Reliability– repeatable– easy to answer

• Accuracy – no question bias– measures what we want

• Relevance– must appear relevant to respondent



Question Types

• Factual– “What did you do?”

• Classification (e.g. socio-demographic)– for comparing with secondary data

• Opinion and attitude questions– “What do you think about ……?”

• Stated Response Questions– “What would you do if ……?”



Physical Design of Forms/Apps

• Observational surveys– Ergonomic– Size/format – not too big or small– Weather-proof– Need log forms– Test under actual conditions



Physical Design of Forms• Self-administered forms

– Layout vital– Minimal writing should be required– No coding aids should appear– Instructions very clear– Professional appearance– Include ID number

35



5. Pilots in the survey process

SampleDesign

PilotSurvey

Conductof

Survey

DataCoding Data

Editing





SurveyInstrument

Design

PreliminaryPlanning



Pilot Surveys

• Why not do a pilot survey?– too expensive– not enough time

• Why do a pilot survey?– too expensive to omit it– not enough time to omit it



Pilot Surveys

• pilot survey is a test of ALL aspects of design• scope for experimental design• saves expensive mistakes



Uses of the Pilot Survey

• "Skirmishing" of wording• Adequacy of questionnaire

– definitions clear?– too many "don't knows“?– too long?– open to closed questions

• Efficiency of interview/surveyer training• Non-response rate• Analysis• Cost and duration



6. Survey implementation

SampleDesign

PilotSurvey

Conductof

Survey

DataCoding Data

Editing





SurveyInstrument

Design

PreliminaryPlanning



Conducting the Surveys

• Need high response rate for validity

• Consider– Announcement letter/message >> higher response– Follow-up regime >> higher response rate

Gross sample . . . . . . . . . . . . . . . . . . = 100 (houses who are eligible to put out bin) Sample loss (vacant, invalid phone no.) = 5 (vacant houses) Net sample . . . . . . . . . . . . . . . . . . . . . . = 95 Responses . . . . . . . . . . . . . . . . . . . . . . = 50 (put out green bin) Response rate . . . . . . . . . . . . . . . . . . . . = response/net sample = (50/95) = 53%



Exercise – dot points for a survey you need• What method?• How to get a sampling frame?• What questions? Need any for weighting?• Other issues/questions?

– Richardson, Ampt, Meyburg (1995)– http://www.geog.ucsb.edu/~deutsch/geog111_211a/

code_books/Survey_Methods_For_Transport_Planning.pdf



7. Weighting (correction)/Expansion

SampleDesign

PilotSurvey

Conductof

Survey

DataCoding Data

Editing





SurveyInstrument

Design

PreliminaryPlanning



Weighting & Expansion of Data

• Getting the sample data to represent the population from which it was drawn, as nearly as possible

• Why? – systematic errors– Non-response >> weighting certain type of respondents

higher– Missing Data >> can make assumptions – or note– Inaccurate Reporting >> e.g. social desirability bias



Example of weighting

• Your response is 50/95 – what about the 45?Say 30 males (67%) 15 females (33%)

- Your secondary data (e.g. counts, other data) males 50% females (50%)– missing responses from females

- Responding females are therefore ‘weighted’ with a slightly higher value (1.5) males (.75)

Stopher, P. (2012) Collecting, Managing and Assessing Data Using Sample Surveys , Cambridge.



Summary

• To measure behaviour change it is essential to understand the data collection and survey process

• In particular need to understand:– Survey method– Sampling principles– Importance of a pilot– Implementation options– Need for weighting

• Vital for future funding as well as for sharing methodologies



• [email protected]


evaluating behaviour change programs

Documents

termscompare wasteminz

wasteminz midyear roundupmay

survey processrichardson

representative sample

specific populationscan

transfer station

observational countscan

activity centre