Choosing measures and evaluation tools in social intervention

Methodologies for a new era summer school

School of Applied Social Studies, University College Cork

Wednesday 22nd June 2011

Matthew Morton, Jennifer Burton

Purposes of Measures

Implementation Adherence, quality, responsiveness, exposure

Outcome Changes (or baseline screening) in attitudes,

behaviors, wellbeing

From inputs/Outputs to Outcomes/Impacts

Theory of change

“Blueprint of the building blocks needed to achieve the long-term goals of a social change initiative.”

Informed by… Existing evidence Psychosocial Theory Formative research Stakeholder input

Standard results chain

Main Components of a Theory of Change

Component Description

Inputs Resources that go into a project/program (e.g., funding, staffing, equipment, curriculum materials)

Activities What we do.

Outputs What we produce. Tangible and can be counted. (e.g., # trainings, attendance, partnerships created, quality processes)

Outcomes Why we do it. Behavioral/attitudinal changes, result from outputs. (e.g., quit smoking, reduce depression, better parenting)

Impacts Long-term changes that result from accumulation of outcomes. Strategic. (e.g., reduce infant mortality)

Kusek & Rist 2004

ToC Assumptions

• External: The program is feasible, addresses real problems/needs, built on accurate beliefs of human nature, and culturally/politically appropriate.

• Internal: The program’s process and implementation will be implemented as intended by the TOC.

• Evaluative: The data captures the intended indicators/outcomes and is accessible. The methodology credibly assesses the TOC.

Exercise – 1. Topic clusters

1. Children2. Disabled3. Discrimination4. Elderly5. Environment6. Health7. Homelessness8. Immigration9. Youth

Exercise – Outcomes Mapping4 months 1 year 5 years

Level I (e.g., child)

Level II (e.g., family)

Level III (e.g., neighborhood)

Think: (a) changes, (b) specific, (c) realistic, (d) measureable

Types of MeasuresStrengths Weaknesses

Self-report (e.g., q’airre) Relatively easy to implement; helps with feelings/attitudes

Self-report biases

Third party (e.g., parent, teacher, psychologist)

Avoids self-report biases; alternative perspective

Limited perspectives; more complicated than self-report to implement

Observational (e.g., observer, video)

More objective; less subject to participant bias

Difficult & costly; harder to standardize coding

Institutional data (e.g., school or gov’t records)

Often directly policy-relevant; objectivity; can avoid burden of surveys

Can be limited; may not capture important “softer” psychosocial outcomes

What Makes a Good Measure?

• Matches Theory of Change• Appropriate for population (age, culture, language, etc.)• Well-tested

• Reliability – The extent to which a test can discriminate between one subject and another

• Validity - The extent to which a test measures what is is intended to measure

Reliability

INTERNAL CONSISTENCY

Measures level of agreement among individual test items

INTER-RATER RELIABILITY

Measures agreement between different raters

TEST-RETEST RELIABILITY

Measures agreement between successive ratings over time

Inter-rater Reliability

For continuous variablesThe Intra-Class Correlation

Coefficient (ICC) measures the extent to which overall test variance is attributable to variability between patients as opposed to random error variability (ie signal-to-noise ratio)

Inter-rater Reliability

For binary (dichotomous variable)Kappa measures the proportion of

agreement above that by chanceFor ordered categorical data,

weighted kappa quantifies degree of disagreement

Values range 0-1 (1=perfect agreement) 0.4-0.6 = Moderate >0.80 = Very high

Internal ConsistencyMeasures the extent to which items of the same

scale (or subscale) are inter-correlatedCronbach increases as (a) average inter-item

correlation rises or (b) number of items risesNo formal test statistic for

>0.5 = moderate >0.8 = excellent

Types of Validity

CONTENT (FACE) To what extent do the items reflect the construct being assessed?

CRITERION (CONCURRENT)

How does the test perform against the ‘gold standard’?

CRITERION (PREDICTIVE)

Do results on the test predict future outcomes? (i.e., why it’s important)

CONSTRUCT Do all the items (in a subscale) relate to the same concept?

CONVERGENT Do the test results correlate with those of other theoretically related tests?

DIVERGENT Is the correlation low between the test and others which should be unrelated?

Benefits of community based parenting groups for hard-to-manage

children: Findings from the Family Nurturing

Network Trial

Jenny BurtonFrances Gardner

University of OxfordCentre for Evidence Based Intervention

Funded by the Esmee Fairbairn Foundation

DEMOGRAPHIC INFORMATION

Parent How old? Status? Married, living as married, single, divorced etc. Employed? Educational achievement?

Child Gender? Age? Siblings?

PRIMARY AIMS OF INTERVENTION

Parents Help provide parents with necessary skills with which to deal with

their hard to manage children

Target Child Decrease child negative behaviour

eg. Non-compliance, tantrums, yelling,destructive, aggressive behaviours.

SECONDARY AIMS OF INTERVENTION

Parent

Improve sense of parenting competence Improve mood Improve their relationships Worst Sibling Decrease child negative behaviour in other siblings

MEASURES USED TO ASSESS PARENT OUTCOME AT BASELINE AND POST INTERVENTION (I)

Questionnaires

Parenting Scale(Arnold, O’Leary et al, 1993) Beck Depression Inventory (Beck, 1972)

Parent Sense of Competence (PSOC; Johnston & Mash, 1989) Dyadic Adjustment Scale (DAS; Spanier, 1976)

MEASURES USED TO ASSESS CHILD OUTCOME AT BASELINE AND POST INTERVENTION (II)

Parent-report questionnaires:

Eyberg Child Behavior Inventory (ECBI; Robinson et al,1980)

Parent interview: Semi-structured interview re conduct & hyperactivity: Parent Account of

Child Symptoms (PACS; Taylor et al,1986)

MEASURES USED TO ASSESS CHILD OUTCOME AT BASELINE AND POST INTERVENTION (III)

Teacher

Strength and Difficulties Questionnaire (SDQ, Goodman et al, 1999)

WHY USE DIRECT OBSERVATIONS? (I)

Direct observations

Planning interventions

Evaluate outcomes

Examine research questions about the mechanisms involved in parent-child interaction

WHY USE DIRECT OBSERVATIONS? (II)

Provide window on real processes and outcomes of interest, eg. parent skills and strategies and child problem behaviour in the home.

Behaviours defined by researcher rather than parent (therefore less easily influenced by parent mood or expectations of intervention)

Self-reported outcomes problematic as parents tend to overestimate change following intervention.

WHY USE DIRECT OBSERVATIONS? (III)

Bias due to expectancy effects less likely (distressed parents poor at manipulating their child into behaving well during home observation)

Observational measures can be particularly sensitive to change in parent and child behaviour following intervention

Observed behaviour a better predictor of long-term hard outcome measures such as arrest rates and incarceration.

DIRECT OBSERVATIONS IN PRACTICE (I)

Choose a visit protocol and coding system which address the questions which need answering:

Do parents use more positive parenting strategies? Does negative child behaviour decrease? In there an increase in positive interaction between parent

and child Do measures assess treatment change or group

differences?

DIRECT OBSERVATIONS IN PRACTICE (II)

Different observational tasks designed to study parent-child interaction in home:

Mildly stressful events eg: mealtimes

Problem solving tasks and tidy up tasks

Tasks where mother busy and child has nothing to do

Exposure (eg. how many sessions)

Adherence

Quality

Reponsiveness (Therapy Attitude Inventory)

IMPLEMENTATION FIDELITY

PARENTING INTERVENTION: THEORY OF CHANGE

• Hire staff• Train staff to implement tested intervention

• Carry out 14 week parenting intervention•Ensuretreatmentintegrity

• Maximiseparent Participation

• Decreasedchild negativebehaviour• Increasedparenting skills and strategies

• Less delinquency• Less incarceration

PARENTING INTERVENTION: THEORY OF CHANGE

• Hire staff• Train staff to implement tested intervention

• Carry out 14 week parenting intervention•Ensuretreatmentintegrity

• Maximiseparent Participation

• Decreasedchild negativebehaviour• Increasedparenting skills and strategies

• Less delinquency• Less incarceration

Exercise 2 - ToC

• Construct a basic TOC of an intervention you’ve worked with/have interest in, and focus on key outcomes

• Identify potential measures or types of measures for implementation and outcomes that match your ToC

Example TOC template

Youth Program Assessment Tools

Implementation Guide to various instruments for Measuring Youth Program Quality by the Forum for Youth Investment

http://forumfyi.org/content/measuring-youth-program-quality-guide-assessment-tools-2nd-edition

E.g., Youth Program Quality Assessment (PQA)

OutcomesCART: Compendium of various psychosocial measures (esp., for youth) http://cart.rmcdenver.com/index.cgi?screenid=seldomain&autoid=91867 CORC: http://www.corc.uk.net/index.php?contentkey=81

Select references

Church, A. T. (2010). Measurement Issues in Cross-cultural Research. In G. Walford, E. Tucker & M. Viswanathan (Eds.), The Sage Handbook of Measurement (pp. 152-175). London: SAGE. (validity across cultures)

Streiner, D. L. (2003). Starting at the Beginning: An Introduction to Coefficient Alpha and Internal Consistency. Journal of Personality Assessment, 80(1), 99-103.

Imas, L. G. M., & Rist, R. C. (2009). The Road to Results: Designing and Conducting Effective Development Evaluations. Washington, D.C.: The World Bank. (fesp. for Th of Chg)

Weiss, C. H. (1995). Nothing as Practical as Good Theory: Exploring Theory-based Evaluation for Comprehensive Community Initiatives for Children and Families. In J. P. Connell, A. C. Kubisch, L. B. Schorr & C. H. Weiss (Eds.), New Approaches to Evaluating Community Initiatives: Concepts, Methods, and Contexts. Washington, DC: Aspen Institute.