charleston presentation v4

8/2/2019 Charleston Presentation v4

1/35

Click to edit Master subtitle style

Propensity Score Matching to Evaluate theImpact of Substance Abuse Treatment

Services for Child Welfare Clients

Richard Barth, Claire Gibbons, & Shenyang GuoJordan Institute for FamiliesSchools of Social Work (1) and Public Health (2)

University of North Carolina

Presented to the Annual Grantees MeetingSubstance Abuse Policy Research ProgramCharleston, South CarolinaDecember 2, 2004

PSM analyses were funded by the Robert Wood Johnson Foundations Substance Abuse PolicyResearch Program NSCAW data used to illustrate PSM were collected under funding by theAdministration on Children, Youth, and Families of the U.S. Department of Health and HumanServices. Findings do not represent the official position or policies of the U.S. DHHS. Results arenot quotable in print. Contact [email protected].
mailto:[email protected]:[email protected]


2/35

Todays Presentation

Introduction to the research questions

Brief explanation of the data sources and

reasons to use PSM to answer the researchquestions.

Description of post PSM analyses

Discussion of the findings


3/35

Background for the Study

Next to poverty, substance abuse is considered mostcentral reason for CWS involvement Linda GordonHeroes of Their Own Lives (1800s)

ACF Building Common Ground (1999)

SAMSHA and ACF National Resource Center on ChildWelfare and Substance Abuse (2003)

Largest population of children who receive child

abuse reports remain at home, but almost all CWSand SAT research is about placement into out-of-home care


4/35

Research Questions for this Sub-study

Does participation in SAT result in an increasein child safety, as reflected in a reduction inthe rate of subsequent child abuse reports

Are there other mechanisms operating that help toexplain the relationship between SAT andsubsequent child abuse reports?


5/35

First National Random Sample Study Of CWS

Extended Research Team includes:

Research Triangle Institute

University of North Carolina

Caliber Associates

San Diego Childrens Hospital, CASRC

CSRD, Pitt Medical Center

Duke Medical Center

National Data Archive on Child Abuse andNeglect, Cornell

92 Local Child Welfare Agencies Admin. For Children and Families

Children and Families


6/35


NSCAW Cohort at Baseline

Total6,231

Enter throughinvestigation

5,504

No services1,725

Ongoing services3,779

In home2,312

Out-of-home1467

Other gateways600

Long-termfoster care

727


7/35


NSCAW Child Sampling is Complex

Stratified: 8 Certainty States and remainder (28) states

92 PSUs (basically, county agencies)

Over Sampled on the basis of:

Children/Families Receiving Services Infants

Sexually Abused Children

Not Sampled on the basis of:

Substantiated Reports (cases are included whether abuse allegations aresubstantiated or not)

ALL

ANA

LYSES

AREWEIGHTED


8/35

Data Collection TimelineTarget population: Childreninvolved in investigationsclosed between October 1, 1999and December 31, 2000

W

ave1:BaselineNov,1999

Apr,2001

Wav

e2:12Month

Follow-upO

ct,2

Wave3:

18MonthFoll

ow-

1999 2000 2001 2002 2003 2004

Wave4:36Mont


9/35

Achieved Samples

Wave 1 (Baseline) Current caregiver : (CPS) 70% (OYFC) 73%

Child welfare worker: (CPS) 86% (OYFC) 80%

Children: (CPS) 66% (OYFC) 70%

Wave 2 (12-Months) Current caregiver: (CPS) 83% (OYFC) 89%

Child welfare worker: (CPS) 84% (OYFC) 85%

Wave 3 (18-Months) Current caregiver : (CPS) 85% (OYFC) 87% Child welfare worker: (CPS) 94% (OYFC) 95%

Children: (CPS) 82% (OYFC) 84%

Differential

attrition wasvery modestand adjustedwith W3weights


10/35

Counterfactuals and Propensity ScoreMatching

Theory of Counterfactuals The fact is that some people who could use substance

abuse treatment (SAT) receive it and some do not. The key assumption of the counterfactual framework is

that individuals selected into treatment andnontreatment groups have potential outcomes in both

states: the one in which they are observed and the onein which they are not observed (Winship & Morgan,1999).


11/35

Counterfactuals and Propensity ScoreMatching II

Counterfactuals cannot be observedwe can onlycreate an estimate of them

These estimates are generally viewed as biased unlessgroups are randomized

Yet, some interventions, like SAT, are considered sovital that randomization is highly improbable

PSM is one correction strategy that endeavors toreduce selection biases and permit comparisonsbetween outcomes of observed and unobservedtreatment when randomization has not been achieved


12/35

Types of Selection Bias Likely at Playin this Study

Self selection (clients influencing the service theyreceive),

Initial attendance

Quitting treatment

Bureaucratic selection (staff in service programs makedecisions based on agency guidelines or more implicitdecision rules [e.g., the availability of treatment]),

Geographic selection (some services are only availablein some areas).


13/35

Two Types of Selection Bias EspeciallyGermane to SAT and CWS Research

Creaming: Best cases are selected intoprograms because most disorganized clientelecannot manage program participation

Souring (triaging): Most difficult cases aretriaged into services--sometimes required by

law to participateso that the treatment groupmay be likely to look worse at the outset andafter treatment


14/35

PSM in a Nutshell

Employs a predicted probability of groupmembershipe.g., treatment vs. control group--based on observed predictors, usually obtained

from logistic regression to create acounterfactual group

Propensity scores may be used for matching and

as covariates in secondary analysis Time to first re-reportevent history analysis


15/35

Creating the Propensity Score:

Dependent Variable AOD service receipt

Stayed overnight in AOD treatment program Clinic or doctor

CWW reports CG received treatment

CG reported currently receiving treatment


16/35

Creating the Propensity Score:

Independent Variables Marital status

Education

Employment Poverty

Case status

Child race/ethnicity Child age

Caregiver age

Trouble paying for basic necessities

CG mental health CG arrest/jail time

Prior AOD treatment

Maltreatment type


17/35

Independent Variables (contd)

Need for treatment

Risk assessment

CIDI-SF Screen, Dependence

CG report of need


18/35

Data & Study Sample

NSCAW data from two waves: baseline (1999-2000) andthe 18-month follow-up.

The sample for this study was limited to families where:

children lived at home (n=4034)

primary caregivers were female (n=3670)

the primary caregiver had at least one indicator of a substanceabuse problem (n=1472)

there was a non-missing value of AOD service receipt and anon-missing value of time to first re-report (n=1074)

Of these children, 276 (26%) formed the treatmentgroup, and 798 (74%) comparison group.


19/35

Selecting the Best Model

12 matching schemes (using Mahalanobis distance and nearestneighbor matching) were used with different tolerances andwith and without including the propensity score as a covariate. Differences in KS survivor functions were all in the same direction;

that is, the treated group has a faster rate of re-report than the

nontreated group.

Did not use Heckmans difference-in-difference because theresults are not amenable to event history analysis

Nearest neighbor matching within a caliper of of thestandard deviation of the propensity score Retained the most cases

Had the fewest remaining statistical differences between items

Had broad common supportarea.


20/35

0

2

4

6

0

2

4

6

0 .5 1

Nontreated

Treated

Density

Predicted ProbabilityGraphs by aodserv

Histogram of Predicted Probabilities of Using SATBy Treatment Group Before Matching


21/35

Sample Comparison: Demographics

Before matching(n=1074)

After matching(n=276)

Child race p=.02 p=.50

Child age p


22/35

Sample Comparison: Caregiver Risks and Need

Before matching(n=1074)

After matching(n=276)

Trouble paying for basicnecessities

p=.05 p=.90

Mental health problems p


23/35

Research Question

Whether or not children of caregivers receivingsubstance abuse services are living in a safeenvironment?

Operationalized as: Does substance abuse treatment for

caregivers affect the risk of child

maltreatment re-reports by caregiver and family?

by caregiver alone?


24/35

Kaplan Meier and Cox Regression

Dependent variable(s) Time to first re-report (18 month follow-up) by:

Caregiver or relative

Caregiver alone

Independent variables for Cox Regression AOD service receipt

Child age

CG age

Prior child welfare services

Open child welfare case Family cumulative risk

Urban/rural


25/35

Findings

EHA of the best PSM resample shows that thegroup difference on survivor functions arestatistically significant with recipients of SAThaving a greater hazard of re-reports than thenontreatment comparison group (roughly 20%vs. 10% during the 18-months).

Time to re-report by caregivers or relatives (p < .02)

Time to re-report by caregivers (p < .05)


26/35

Percent Children RemainingWith No Re-report on Caregivers

Substance Abuse Sample (n=1,074) Resampled with PSM (n=276)

The proportion of children re-reported within 18-months was higher for SAT

after the PSM correction (19 % vs. 10%) but not before (18% vs. 18%)

0 . 0 0

0 . 2 5

0 . 5 0

0 . 7 5

1 . 0 0

d u r 2

0 . 0 2 . 5 5 . 0 7 . 5 1 0 . 0 1 2 . 5 1 5 . 0 1 7 . 5 2 0 . 0

S T R A T A : A O D S E R V _ = 0 C e n s o r e d A O D S E R V _ = 0 A O D S E R V _ = 1 C e n s o r e d A O D S E R V _ = 1

0 . 0 0

0 . 2 5

0 . 5 0

0 . 7 5

1 . 0 0

d u r m

0 . 0 2 . 5 5 . 0 7 . 5 1 0 . 0 1 2 . 5 1 5 . 0 1 7 . 5 2 0 . 0

S T R A T A : a o d s e r v _ = 0 C e n s o r e d a o d s e r v _ = 0 a o d s e r v _ = 1 C e n s o r e d a o d s e r v _ = 1


27/35

Estimated Hazard Ratios:Cox Regression based on the Resample

________________________________________________________

Variable Re-report Re-reportRelative or CG Primary CG

SA Service Receipt (No)

Yes 2.02 * 2.03 *

Child Age (11+)

0-2 .60 .58

3-5 .76 .72

6-10 1.44 1.49

Caregiver Age (35 or Older)

< 35 1.51 1.66

Prior Child Welfare Service(No)

Yes 2.44 ** 2.51 **

Open Child Welfare Case (No)

Yes .77 .74Family Cumulative Risk (High)

Low .68 .75

Moderate .85 .89

Residence (Nonurban)

Urban 1.9 1.93

________________________________________________________

** p


28/35

Limitations of PSM^

Large samples are required

The sample size did drop, but is still substantial (unweighted n=276)

Group overlap must be substantial before matching

There were significant differences between the groups prior to PSM

About half of the SAT cases were excluded from the final models

Hidden bias may remain because matching only controls for observedvariables (to the extent that they areperfectly measured)

We based our variable selection on a thorough review of predictorsof service use

Although we have 14 indicators of the need for substance abuse

services, this information is still notperfectly measured^Shadish, Cook, & Campbell, 2002


29/35

Conclusions

Caregivers with allegations of child abuse andneglect against them, and with substance abuse

problems, often remain at home following the CWSinvestigation.

Children whose female caregivers receive substanceabuse treatment are more likely to receive asubsequent child maltreatment report in thefollowing 18-months: Whether the source of the report includes all relatives or

is restricted to the caregiver, and Whether or not the selection bias into SAT is controlled

for using PSM methods.


30/35

Possible Mechanisms

The higher rate of re-reports may be attributable tosome of the following mechanisms, each of which wastested and found not to differbetween the SAT andnontreatment group:

The type of maltreatment at re-report

The average CIDI-SF score for alcohol and drug use, acrossthe two waves

Childrens behavior on the CBCL (ages 2 and up)

Childrens reports (ages 11 and up) with regard to harsh orsevere parenting


31/35

Alternate Explanations for Findings

PSM did not match for important unobserved covariates Services may not have been used in full

Substance abuse services may interfere with parentaladequacy

Focus is on parents recovery not childs welfare

Time and effort for SAT can be burdensome to parent

Services may result in greater surveillance which resultsin more observed behaviors that might place children atrisk, thus more reports

But there were also more placements into foster care (p < .05).


32/35

Implications

Children whose caregivers receive SAT may be saferif the early re-reports were for failure to participate insubstance abuse treatment and prevented abuse orneglect

Children whose caregivers receive SAT may be lesssafe if the re-reports followed abuse or neglect Caregivers who are involved with SAT may increase

their likelihood of remaining involved with CWS, atleast for 18-months

Much more needs to be known about the impact ofSAT on child welfare caregivers and children This study suggests that it is not unambiguously beneficial

for reducing maltreatment and preserving families


33/35

Partial References

Gregoire, K. A., & Schultz, D. J. (2001). Substance-abusing child welfare parents: Treatment

and child placement outcomes. Child Welfare, 80(4), 433-452.

Leathers, S. J. (2002). Parental visiting and family reunification: Could inclusive practicemake a difference? Child Welfare, 81(4), 595-616.

Rosenbaum, P. R., & Rubin, D. B. (1985a). Constructing a control group using multivariatematched sampling methods that incorporate the propensity score.American Statistician, 39,

33-38.Semidei, J., Radel, L. F., & Nolan, C. (2001). Substance abuse and child welfare: Clearlinkages and promising responses. Child Welfare, 80(2), 109-128.

Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002).Experimental and quasi-experimentaldesigns for generalized causal inference. Boston: Houghton Mifflin.

Sosin, M. R. (2002). Outcomes and sample selection: The case of a homelessness and

substance abuse intervention.British Journal of Mathematical and Statistical Psychology, 55,63-91.

U.S. Department of Health and Human Services. (1999, April).Blending perspectives andbuilding common ground. A report to Congress on substance abuse and child protection.Retrieved May 23, 2000, http://aspe.hhs.gov/hsp/subabuse99/subabuse.htm
http://aspe.hhs.gov/hsp/subabuse99/subabuse.htmhttp://aspe.hhs.gov/hsp/subabuse99/subabuse.htm


34/35

PSM and Project Resources

This Presentation

Substance Abuse and Child Welfare Services: Research Update andNeeds (2003)

Introduction to PSM: A New Device for Program Evaluation (2004)

Substance Abuse among Caregivers Involved with Child WelfareServices: Prevalence and Identification by Child Welfare Workers(2004) http://sswnt5.sowo.unc.edu/VRC/Lectures/index.htm

Substance Abuse Needs and Services for Families Involved in the Child(brief project description and link to other project materials) http://sswnt5.sowo.unc.edu/JordanIF/jif_map/projects_family_nonframe.cfm
http://aspe.hhs.gov/hsp/subabuse99/subabuse.htmhttp://aspe.hhs.gov/hsp/subabuse99/subabuse.htmhttp://aspe.hhs.gov/hsp/subabuse99/subabuse.htmhttp://aspe.hhs.gov/hsp/subabuse99/subabuse.htmhttp://aspe.hhs.gov/hsp/subabuse99/subabuse.htmhttp://aspe.hhs.gov/hsp/subabuse99/subabuse.htmhttp://aspe.hhs.gov/hsp/subabuse99/subabuse.htmhttp://aspe.hhs.gov/hsp/subabuse99/subabuse.htmhttp://aspe.hhs.gov/hsp/subabuse99/subabuse.htmhttp://aspe.hhs.gov/hsp/subabuse99/subabuse.htmhttp://aspe.hhs.gov/hsp/subabuse99/subabuse.htmhttp://aspe.hhs.gov/hsp/subabuse99/subabuse.htmhttp://aspe.hhs.gov/hsp/subabuse99/subabuse.htmhttp://aspe.hhs.gov/hsp/subabuse99/subabuse.htmhttp://sswnt5.sowo.unc.edu/VRC/Lectures/index.htmhttp://sswnt5.sowo.unc.edu/JordanIF/jif_map/projects_family_nonframe.cfmhttp://sswnt5.sowo.unc.edu/JordanIF/jif_map/projects_family_nonframe.cfmhttp://sswnt5.sowo.unc.edu/JordanIF/jif_map/projects_family_nonframe.cfmhttp://sswnt5.sowo.unc.edu/JordanIF/jif_map/projects_family_nonframe.cfmhttp://sswnt5.sowo.unc.edu/VRC/Lectures/index.htm


35/35

Thank you very much

Questions?

?

charleston presentation v4

Documents