mark w. fraser, pi, school of social work, unc-chapel hill steven h. day, school of social work
DESCRIPTION
Social and Character Development in Elementary School: The Effectiveness of the Making Choices Program. Mark W. Fraser, PI, School of Social Work, UNC-Chapel Hill Steven H. Day, School of Social Work Shenyang Guo, School of Social Work - PowerPoint PPT PresentationTRANSCRIPT
Social and Character Development in Elementary School:
The Effectiveness of the Making Choices Program
Mark W. Fraser, PI, School of Social Work, UNC-Chapel Hill Steven H. Day, School of Social Work Shenyang Guo, School of Social Work Alan Ellis, Sheps Center and School of Social Work Roderick A. Rose, School of Social Work Maeda J. Galinsky, School of Social WorkKim Dadisman, Co-PI, Center for Developmental Science, UNC-CH Dylan Robertson, Center for Developmental Science Tom Farmer, School of Education, Pennsylvania State University
This presentation was given at the School of Social Work, University of Maryland, Baltimore, MD, on April 9, 2009. Portions of this report were presented at the annual meeting of SACD Project grantees on June 13, 2008 in Washington, DC
Preliminary Findings
Agenda• Theoretical Bases and programs
• Design and challenges
• Analytic strategies
• Analytic methods (skim – see slides)
• Findings
Acknowledgments
• This project was support by a cooperative agreement (R305L030162) with the Institute of Education Sciences at the U.S. Department of Education (US DOE). Funding for the project was appropriated by the US DOE and the Centers for Disease Control and Prevention.
• We thank Paul Rosenbaum (U Penn), Ben Hansen (U of Michigan), and Matthias Schonlau (Rand Corp) for their consultation on methodological issues related to this presentation.
Teachers Talk about Making Choices
• Changes in Classroom Atmosphere
• Observable Differences in Student Behaviors
• Measurable Academic Achievement
Classroom Atmosphere
“I noticed that the classroom
started working more as one big group instead of
individuals.”
Gr.5 Sandy Grove Elementary,Hoke County
Observable Behaviors
“The students tend to be less critical of each other and
more understanding of
each other’s differences.”
Gr. 5 Sandy Grove Elementary,Hoke County
Academic Achievement
“ The program uses excellent books to support the goals of being a good friend and not hurting others.… I use them during Language Arts time.” Gr. 4 Tommy’s Road Elementary, Wayne County
“It provided a way for students to put their feelings into words.”
Gr. 2, Bunn Elementary, Franklin County
Observable Behaviors …
I am feeling really mad!
Academic Achievement
“My students spend more time on task. They seem less distracted by annoying behavior.”
Gr. 5 Scurlock Elementary, Hoke County
Children are actually stopping and thinking about making the right choices, and I have heard a lot of children say to themselves,
“Make the right choice.” It is great to hear.
Kdg. Bunn Elementary, Franklin County
Make the right choice!
“It made a difference with teaching children how to deal with their feelings using better methods rather
than having tantrums or hitting.”Kdg. Bunn Elementary, Franklin County
Oh, boy! I need that Making Choices program.
Classroom Atmosphere
“This program provided a
foundation on which we
could build a classroom
community.” Gr. 1 North Drive,
Wayne County
…a program designed to reduce disruptive behavior and promote academic achievement.
…lessons that teach children respect toward others and responsibility for their own actions.
…social skills to make friends and deal with interpersonal problems.
Children in my school need…
Making Choices
Does social and character education work?
Research Question:
(i.e., is Making Choices effective?)
Intervention Research Perspective:The Design and Development Approach
1. Specify the problem and develop a program theory
2. Create and revise program materials
3. Refine and confirm program components (sequential experimentation perspective)
4. Assess effectiveness in a variety of practice circumstances and settings
5. Disseminate findings and program materials
Source: Fraser, M. W., Richman, J. M., Galinsky, M. J., & Day, S. H. (2009). Intervention research: Developing social programs. New York, NY: Oxford University Press.
PROBLEM THEORY:Perspectives on Conduct Problems and Academic
Achievement in Elementary School
•Developmental risk perspective
•Ecological theory
•Social information processing theory
Social and Character Development in Childhood:A Risk and Resilience Orientation
Biological RisksParentingFamily-School Pre-School ClimateNeighborhood
School ReadinessProcessing SkillsParentingFamily-SchoolSchool ClimateNeighborhood
Peer RejectionAcademic FailureParentingFamily-SchoolSchool ClimateNeighborhood
Increasingly Broad Repertoire of Potentially Damaging and
Aggressive Behaviors
Eco-Developmental Risk CascadePOINT: Risk factors for poor developmental outcomes vary over time. Lacking effective
intervention, the potential for poor outcomes increases – and cascades – as function of complex bio-social processes. To promote positive outcomes, we must disrupt malleable risk mechanisms.
Cognitive Mediation Model(in Developmental Sciences)
Biological Biological PredispositionPredisposition
Biological Biological PredispositionPredisposition
Biological Biological PredispositionPredisposition
Biological Biological PredispositionPredisposition
Sociocultural Sociocultural ContextContext
•Stress/povertyStress/poverty•RacismRacism•Street codesStreet codes•Acute/chronic stressAcute/chronic stress
Sociocultural Sociocultural ContextContext
•Stress/povertyStress/poverty•RacismRacism•Street codesStreet codes•Acute/chronic stressAcute/chronic stress
Adapted from: Dodge, K. A., & Pettit, G. S. (2003, p. 351). A biopsychosocial model of the development of chronic conduct problems in adolescence. Developmental Psychology, 39(2), 349-371.
PeersPeers•Deviancy trainingDeviancy training•Contagion effectContagion effect•False consensusFalse consensus effecteffect
PeersPeers•Deviancy trainingDeviancy training•Contagion effectContagion effect•False consensusFalse consensus effecteffect
ParentingParenting•MonitoringMonitoring•BondingBonding
ParentingParenting•MonitoringMonitoring•BondingBonding
Mental Mental ProcessesProcesses
•Social knowledgeSocial knowledge•ScriptsScripts•Schema/skillsSchema/skills
Mental Mental ProcessesProcesses
•Social knowledgeSocial knowledge•ScriptsScripts•Schema/skillsSchema/skills
Conduct Conduct ProblemsProblems
•Conduct disorderConduct disorder•FightingFighting•Drug useDrug use
Conduct Conduct ProblemsProblems
•Conduct disorderConduct disorder•FightingFighting•Drug useDrug use
Sociocultural Sociocultural ContextContext
•Stress/povertyStress/poverty•RacismRacism•Street codesStreet codes•Acute/chronic stressAcute/chronic stress
Sociocultural Sociocultural ContextContext
•Stress/povertyStress/poverty•RacismRacism•Street codesStreet codes•Acute/chronic stressAcute/chronic stress
Social Information Processing Theory:SIP Skills and Emotional Regulation as Malleable
Mediators?State the
problem
Generate
potentialsolutions
Evaluate potentialsolutions
Select &enact the
best solution(s)
Assessoutcomes
Encode social cues
Interpretsocial cues
Arousal, Emotions,Social Knowledge
Setgoal(s)
Social Knowledge: Life experiences producing scripts, schemata, skills, and beliefs
PROGRAM THEORY(specifies how a program is to work)
InterventionProgram StructureSuch as:Targeting Unit: Classroom Entire school Other (after school, family) Curriculum Structure: Distinct activities Embedded in curriculumActivities to address SACD GoalsSuch as:Character educationViolence prevention/peace promotionSocial and emotional developmentTolerance and diversityRisk prevention and health promotionBehavior management
Social - Emotional
Competence (mediator)
Attitudes about aggressionSelf-efficacyEmpathy
School Climate (mediator)
School connectednessVictimizationFeelings of safety at schoolParent involvement
BehaviorPositive BehaviorResponsible behaviorProsocial behaviorSelf-regulationCooperationNegative behaviorAggressionMinor delinquencyDisruptive classroom behavior
Moderating FactorsChild Family CommunityGender Parenting practices Community risk factorsSocioeconomic status Home atmosphere Social capitalRace/ethnicityRisk status Program SchoolPrior test scores/grades Fidelity Activities to promote social and character development
Intensity and dosage Organizational structure
Social and Character Development: Social and Character Development: Prevention ModelPrevention Model
AcademicsAcademic competenceSchool engagementGradesStandardized test scores
Social Development Model PerspectiveInstruction in social & emotionalskills . . .
EmpathyAnger managementProblem solving & impulsecontrol
Opportunity to . . .Discuss and identify feelingsAcquire language andcommunication skillsPractice solving problemsObserve models
Reinforcement andgeneralization of learningthrough . . .
Naturally occurringopportunities in schoolHome discussion of materials
Engaged school behavior. . .
Focusing on workPaying attentionFollowing instructions
School success
Reductions in . . .Classroom disruptivebehaviorAnxietyAnger
Reductions in . . .Problem behaviorAggressionPeer rejection
Social & emotional skills . . .EmpathyAnger managementProblem solving &impulse control
Intervention Immediate outcomes Knock-on outcomes Outcomes
PROGRAMS
*Developed by the program investigators, the intervention simultaneously focuses on the characteristics of children and on the classrooms in which they learn. The intervention combines three components.
Social Skills Training for students
ClassroomBehavior
Management Training and Consultation
for teachers
Social Dynamics Training for teachers
Group randomization:
Cohort 1: Hoke and Wayne Counties (10 schools randomized to 5 intervention; 5 control)
Cohort 2: Franklin County (4 schools randomized to 2 intervention; 2 control)
The Competence Support Program*
Program Elements• Making Choices: Skills Training curriculum for students in
elementary school. In-service training introduced teachers to the risks of peer rejection and social isolation, including poor academic outcomes and conduct problems. Throughout the school year, teachers received consultation and support (2 times per month) in providing lessons designed to enhance children’s social information processing and other skills. As a part of the Standard Course of Study, the program was integrated into routine class instruction.
• Classroom Behavior Management provided teacher consultation on classroom management strategies designed to strengthen engagement in instructional activities.
• Social Dynamics Training provided teacher consultation on classroom contexts, social groupings, and interactional patterns that can be used to reinforce academic achievement and prosocial behavior.
Theory of Change: Theory of Change: Making ChoicesMaking Choices
Training the Teacher or Counselor
Application of Making Choices by Teacher or Counselor
SIP skills of the Children in the School
Impact on Social Engage-ment and Peer Rejection
Impact on Disruptive Behavior and Academics
Characteristics of the Teacher or Counselor
Characteristics of the Children and the Classroom
Ran
dom
Ass
ignm
ent
Core #1
Core #2
Core #3
Core #4
Core #5
Note. In a randomized trial, you must figure out a way to measure each of the core elements.
Treatment as Usual Control Condition
•Assess implementation of training•Assess if teacher acquires skills from training/supervision
•Test the degree to which the intervention is delivered as intended, e.g., specific activities
Make Program Manuals• From risk mechanisms, mediators, and
logic models to the design of a program
• Specifying program activities that target the malleable mediators and have cultural congruence
• Example: Making Choices
For a discussion of issues in the development and use of treatment manuals, see: Galinsky, M. J., Terzian, M. A., & Fraser, M. W. (2006). The art of group work practice with manualized curricula. Social Work with Groups, 29(1), 11-26.
Warning: It is easy to under estimate the difficulty of developing a program manual.
“That Sunk Feeling”
Source: Don Moyer, Harvard Business Review (October, 2004, p. 160)
If you start in the wrong place, it usually does not help to dig deeper!
Start with theory and research, plus practice
experience…
Develop a template for each lesson or session
How to begin in the right place…
Recognizing Your Feelings
Objectives:
The learner will recognize that certain situations bring out feelings in all of us. The learner will practice recognizing their own feelings. The learner will use personal experiences and knowledge to interpret written
and oral messages. (SCS- LA 3.01) The learner will write structured, informative presentations and narratives
when given help with organization. (SCS- LA 4.08)
Materials:
Penguin Facts page, Response Sheets, Write About It worksheets A and B
Introduction
Review the idea that we all experience a variety of emotions and responses to emotions. Even when we experience the exact same situation, we may have different responses to the situation. Our responses to our feelings can cause us to do good things, but at times they can also cause us to do things that are not helpful.
Activity I: Pete the Penguin
Using two columns, list on the board the emotions presented in Lesson 1 of the book, The Way I Feel. Column I- Emotions that Feel Good: happy, silly, excited, proud, or
thankful Column II- Emotions that Don’t Feel Good: scared, sad, disappointed, bored, angry, or jealous Introduce the students to Pete the Penguin using the penguin puppet. Pass out the Penguin Facts page and discuss the factual information about penguins. Explain to the students that Pete has experienced events that have brought out many different emotions. Sometimes his emotions feel good, but at other times they don’t feel very good at all.
Review the emotions listed in the columns on the board. Then give each student four small pieces of paper (about the size of a note card). Read aloud the following events involving Pete the Penguin. After reading each event, ask the students, “How would you feel?” Give the students enough time to record their responses on one of
Grade 2Grade 2 Lesson Lesson 22
Activity 1
Overview
Review
PropAnswers
Process Tip
Standard Course
of StudyPrep Material
s
their blank pieces of paper. They can use the emotions on the board to express how they would feel or they may provide their own responses.
After you read each situation, collect a few responses randomly and read them aloud (so as not to bring attention to specific student responses). As you read through each response, discuss whether the event brought out a good feeling or a not-so-good feeling. The texts are ambiguous so that students can develop their own interpretations—not all students will feel the same way about each situation. Discuss the idea that everyone heard the same event, yet the feelings were different in many instances.
Today Pete walked in the classroom. As he walked to his desk, Pete noticed Susan and Tony talking quietly and laughing. They both looked up at Pete and giggled. If you were Pete, how would you feel?
When Pete was on the playground, he saw a group of students playing ball. He went to join them, and they told him he could play as soon as they started the next game. If you were Pete, how would you feel?
At lunch, Pete was sitting next to Jermaine. Jermaine opened his lunch and Pete looked inside. All he saw was two cookies and a drink box. If you were Pete, how would you feel?
Pete’s teacher told him he could play a game with Juan as soon as he finished his writing assignment. If you were Pete, how would you feel?
After discussing the above events, ask the students how they recognize when they are feeling certain emotions. “What happens when you start to feel angry?” “Happy?” “Frustrated?” and so on. (Example response: When you are getting angry- you might get hot, start to shake, get tense, grit your teeth, etc.)
Leave the list of emotions on the board to use in Activity II .
Activity II: Write About It
Give the students the Write About It page. On the top of the sheet, have students write about an event in their life that caused them to experience an emotion that made them feel good. On the bottom of the sheet they can write about an experience that caused an emotion that didn’t feel good. Each
narrative should describe the emotion, what caused it, and how they responded to the emotion. Students can refer to the columns on the board to choose the emotions they want to write about. Share the following examples aloud or on a transparency: Example 1: Once I felt excited when I was going to my friends party. I knew I felt this way because I was smiling and jumping around.
Avoid labeling
Scenarios
Activity 2: Write About It!
Develop all worksheets and artwork
G r a d e 2 - L e s s o n 2
A good feeling: Once I felt ___________________ when ____________________________________ ____________________________________ ____________________________________ ____________________________________ I knew I felt this way because__________ ____________________________________ ____________________________________
Activity II: Sheet A NAME: _________________
A not so good feeling: Once I felt _____________________ when ___________________________________ ___________________________________ ___________________________________ ___________________________________ I knew I felt this way because__________ ___________________________________
___________________________________
Activity II: Sheet B NAME: ____________________
““Pete the Penguin” Poster for Pete the Penguin” Poster for Grade 2Grade 2
Sample Lesson
Activities
from
Making Choices
Gr. 3 Lesson - Intentions
SYMBOLS
FRIENDLY
Grrrrr!
MEAN CAN’T TELL
Grade 3 Lesson 10
Intentions: Mean or Friendly?
LAUGHSWITHYOU
HITSYOU
SHARESWITHYOU
MAKESA FACEAT YOU
HUGSYOU
IGNORESYOU
TALKSABOUT
YOU
BITESYOU
HELPSYOU
Grrrrr!
MEAN
FRIENDLY
FRIENDLY
FRIENDLY
CAN’T TELL
GOAL SETTING
GOAL: Something a person wants or something a person wants to see happen.
RELATIONSHIP GOAL: Goals that involve wanting to get along with another person.
Grade 4 Lesson 6
Are these Relationship Goals?
• I want to make an “A” on my math test.• I want to play more often with my friend.• I want a new video game for my birthday.• I want to eat out at a restaurant for dinner.• I want to become friends with the new student.• I want to join in the basketball game at recess. • I want to sit with Jose on the bus.• I want to be in the class play this fall.• I want to stop getting upset when friends ignore me.
(thumbs up or thumbs down)
GOAL SETTING
• Set a relationship goal for these situations:
I was playing basketball at recess with some friends. Terrell, who is not very good at basketball, asked if he could play with us.
Set a Relationship Goal
Denise just made me really upset. She tried to pick a fight with me by saying things that are not true. I am feeling angry with her right now.
Set a Relationship Goal
Yesterday, my mom gave me a really cool pen that writes in all different colors. When I brought it to school this morning, Stacey asked me if she could borrow it. Last time I let Stacey borrow something she lost it, but if I say no she might get angry with me.
EVALUATION DESIGN:Cluster Randomized Trial
with Ten Schools Randomly Assigned to Treatment (j=5) and Control (j=5) Conditions
Cohort Design: Intervention provided in grades 3, 4, and 5
Prior Studies 1.Single-group qualitative trial of MC intervention (8th grade girls)2.Two-group cluster randomized trial at classroom level in one middle school (6th grade only)3.Two-group cluster randomized trial at classroom level in one school (3rd grade)4.Two-group, MC+SF intervention randomized trial (11 sites, 3rd – 4th grade)5.Cohort sequential study by classroom in two schools (3rd grade)6.(Current) Two-group cluster randomized trail at 14 elementary schools
SAMPLE
Two Overlapping Samples
* One treatment school was reorganized into a different building and dropped the program between 4th and 5th grade; students from that school were excluded from the 3-4-5 sample.
Grade 3-4-5 Sample• 3rd, 4th, and 5th graders • 9 schools*• Only consented students on a 5th grade roster• Change=addition of entrants
n=370
n=414
Grade 5n=433
Grade 3-4 Sample• 3rd and 4th graders • 10 schools• Any consented students on a 3rd or
4th grade roster• Change=entrants-leavers
Grade 4n=557
Grade 3n=571
N Total Control Intervention p-value
Child (analysis sample)
Male child, % 618 47.7 50.0 45.3 .243
Age of child, years 618 7.92 7.94 7.89 .235
Black child, % 618 34.5 21.9 48.0 .001
White child, % 618 48.1 59.3 35.8 .001
Hispanic child, % 618 10.0 13.7 6.1 .001
Family (analysis sample)
Primary caregiver, not a HS graduate , %+ 564 13.5 15.0 11.7 .259
Two biological parents not in household, %+ 563 48.5 43.7 54.0 .014
Income-to-needs < = 1, %+ 550 26.0 24.6 27.6 .416
School (aggregate, school level)
Black, %* 10 41.3 28.4 51.4 .001
Free Lunch, %* 10 44.7 37.4 50.7 .001
Adequate Yearly Progress*# 10 81.9 84.7 79.0 .068
Pupil/teacher ratio, mean* 10 16.3 16.2 16.4 .825
Note. + MPR data from baseline child level data file. * NCES school level data (CCD 2003-2004) across all schools. # AYP Performance Composite score, Year 1 of the SACD study.
Equivalence of Intervention and Control Groups on Selected Child, Family, and School Attributes: Grade 3 Cohort 1
Difference in School-Level Academic Performance: Percentage at Grade Level
Cohort 1 AYP Performance Composite
50
55
60
65
70
75
80
85
90
02-03 03-04 04-05 05-06 06-07
Per
cen
t of s
tud
ents
ach
ievi
ng
at g
rad
e le
vel
Treatment
Control
Test results for 2005-06 and 2006-07 are based on a revised accountability model and are not comparable to those from previous years.
Equivalence of Intervention and Control Groups on Selected Site Specific Outcomes Grade 3 Sample (Pretest): Cohort 1 N Total Control Intervention p-value
SIP Skill Level Assessment - Child
Encoding 420 44.2 43.0 45.6 .149
Goal formulation 412 67.6 65.5 70.1 .074
Response decision making 410 66.0 64.1 68.3 .225
Carolina Child Checklist - Teacher
Social contact 549 3.8 3.7 3.8 .121
Cognitive concentration 549 3.2 3.2 3.2 .952
Social competence 549 3.3 3.3 3.3 .945
Social Aggression 549 4.1 4.1 4.0 .183
Interpersonal Competence Scale - Teacher
Aggression 548 2.5 2.4 2.6 .095
Academic competence 548 5.1 5.1 5.1 .948
Popularity 548 4.9 4.9 4.8 .798
Peer Interpersonal Assessment
Aggression 502 38.7 42.6 34.4 .152
Prosocial skills 502 82.8 84.8 80.6 .552
Sample sizes vary because pretest measures were collected from different respondents (teachers, students) at different times. SLA and Peer assessment pretest were collected from students at the end of 2nd grade. CCC and ICST were collected from teachers at the beginning of 3rd grade. SLA=Skill Level Assessment (SIP skill – HOME Scale adaptation by Dodge, 1980). CCC=Carolina Child Checklist (Macgowan et al. 2002 – Research on Social Work Practice). ICST-Interpersonal Competency Scale – Teacher (Xie et al., 2002, Social Development)
Teacher and Classroom Characteristics by Intervention Status
Characteristic
Intervention Teachers (n = 21)
Non Intervention Teachers (n = 23) Difference p-value
Demographics White %+ 73.7 95.5 -11.8 .051 Greater than a bachelor degree %+ 28.6 4.6 24.0 .033 Years of teaching, mean+ 11.6 14.1 -2.5 .385 Years teaching at current school, mean+ 4.1 8.0 -3.9 .040 Has regular/advanced teaching certificate, %+ 85.0 90.9 -5.9 .566
SACD classroom activities
N of SACD classroom strategies, mean+ 14.8 13.7 1.1 .267 Violence prevention hours, mean+ 7.8 2.7 5.1 .039 Social and emotional development hours, mean+ 10.9 2.9 8.0 .002
Classroom observations Number of feedback and structure exemplars in place, mean# 6.2 4.2 2.0 .019
Note. +MPR baseline data (spring). #Observations were conducted with the Classroom Observation Form (COF) by an intervention specialist blind to the treatment or control condition of each school. The COF assesses seven domains relevant to the intervention: daily routines, time and task management, consequences and follow-through, teaching alternative behaviors, communication and feedback, and group processes and peer support.
Causation in Research Design:Randomization Is Supposed to Produce the Counterfactual
Note. We let the control group serve as evidence for what would have happened counter to the fact of participation in intervention (the stat class). Randomization is supposed to create equivalence or balance between the intervention (taking the stat class) and control (not taking the stat class) groups. But it didn’t. On several observed and an unknown number of unobserved measures, the intervention and control group schools differ.
Four evaluation challenges
Selection Bias: Covariates are not balanced between treated and control groups
Missing Data: No baseline data on enterers and lost data on leavers = constant churning of sample
Rater Effects: Outcome ratings were made by the same teachers within grades, but different teachers over grades 3, 4, and 5• Piecewise analyses – change scores within grade level
Treatment Contamination/History: High intervention content in control schools*
Note. Student Citizen Act (SL 2001-363) was passed into law by the Legislature in 2001. The Act required local boards of education to develop and incorporate character education instruction into standard curricula. Local boards of education began implementation in the 2002-2003 school year.
MEASURES
Site-Specific Outcomes
• Skill Level Assessment Activity (SLA): Based on the Dodge Home Scale (1980), the SLA uses students’ responses to questions about hypothetical social situations. After viewing picture scenarios, students answer questions measuring different aspects of social information processing skill: encoding (α=.78), goal formulation (α=.76), and response decision making (α=.80).
• Carolina Child Checklist (CCC): The CCC is a 35 item teacher questionnaire that yields factor scores on children’s behavior including social contact (α=.90), cognitive concentration (α=.97), social competence (α=.90), and social aggression (α=.91).
• Interpersonal Competence Scale-Teacher (ICST): The ICST is an 18-item teacher questionnaire that yields factor scores on children’s behavior including aggression (α=.84), academic competence (α=.74), and popularity (α=.78).
• Peer interpersonal assessments: Peer interpersonal assessments were used to examine classmates’ perceptions of participants’ social and behavioral characteristics including aggression (α=.92), prosocial skills (α=.84 ), and internalizing behavior (α=.67 ).
Grade 2 Grade 3 Grade 4 Grade 5 Wave 1 Wave 2 Wave 3 Wave 4 Wave 5 Wave 6 Wave 7 Wave 8 Wave 9 Time of year March Sept March April Sept March April Sept April Cohort 1 (C1) 2003-04 2004-05 2004-05 2004-05 2005-06 2005-06 2005-06 2006-07 2006-07 Cohort 2 (C2) 2004-05 2005-06 2005-06 2005-06 2006-07 2006-07 2006-07
Instrument (in-house data)
Making Choices Student report (SLA)
C1 C2
C1 C2
C1 C2
C1 C2
C1 C2
C1 C2
C1 C1
Friends and Groups Student report (Social Cognitive Maps)
C1 C2
C1 C2
C1 C2
C1 C2
C1 C2
C1 C2
C1 C2
C1 C1
Peer groups Teacher report (PG)
C1 C2
C1 C2
C1 C2
C1 C2
C1 C2
C1 C2
C1 C2
C1 C1
Carolina Child Checklist Teacher report (CCC)
C1 C2
C1 C2
C1 C2
C1 C2
C1 C2
C1 C1
Interpersonal Competence Scale Teacher report (ICST)
C1 C2
C1 C2
C1 C2
C1 C2
C1 C2
C1 C1
Peer Interpersonal Assessments Student report (PNOMS)
C1 C2
C1 C2
C1 C2
C1
Ratings of school adjustment Teacher report (TASS)
C1 C2
MPR data (across-site data)
Student reported C1
C2 C1
C2 C1
C1
C2 C1
Teacher reported C1
C2 C1
C2 C1
C1
C2 C1
Parent reported C1
C2 C1
C2 C1
C1
C2 C1
Summary of Data Collection Occasions
Minutes of Skills Training Instruction in 3rd and 4th Grades by Student
Benchmark=1,140 minutes
Below benchmark: 19%Above benchmark: 81%
(overall n=571)
ANALYTIC PROCEDURES
Analytic Procedures: Flow Chart for Use of “Bias-Correcting” Statistical Methods
Multiple Imputation of Missing Data(The imputation models employed both predictor and outcome variables, but the analysis models employed imputed missing values for predictor variables only). 50 imputations for each outcome variable
Piecewise change score HLM analysis using propensity score weighting (propensity scores estimated by gbm)
Estimation of propensity scores using Generalized Boosted Modeling (gbm) -- aims to optimize balance on observed covariates between treated and control groups
Heckman sample selection Model (Predictors of the selection equation are similar to the input of gbm)
Optimal pair matching using propensity scores estimated by gbm
Optimal full matching using propensity scores estimated by gbm
Post-pair-matching with regressionadjustment
Post-full-matching with Hodges-Lehmann aligned rank test
Dose (efficacy subset) analysis) using Abadie et al. Matching estimator
Procedures for multiple imputation of missing data
Test for MCAR (Little, 1988) confirms models are not MCAR.
Assumption of MCAR not required if imputation model is informed (i.e., data may be missing at random)
A diagnostic stage identified models that resulted in 99% relative efficiency for all analysis variables.
50 simulations (copies of the raw data set) generated using MI.
DVs and predictors both used in imputation; imputed DVs discarded after imputation (MID procedure; von Hippel, 2007).
Missing Data Diagnostics: Proportion without Missing Data and Proportion of Missing Data Points
Step 2: MatchingGreedy match (nearest neighbor with or without calipers) Mahalanobis with or without propensity scores Optimal match (pair matching, matching with a variable number of controls, full matching)
Step 1: Logistic regression Dependent variable: log odds of receiving treatmentSearch an appropriate set of conditioning variables (boosted regression, etc.) Estimated propensity scores: predicted probability (p) or log[(1-p)/p].
General Procedure for Propensity Score Analysis
Step 2: Analysis using propensity scores: Multivariate analysis using propensity scores as weights
Step 2:Analysis using propensity scores Analysis of weighted mean differences using kernel or local linear regression (difference-in- differences model of Heckman et al.)
Step 3: Post-matching analysis Multivariate analysis based on matched sample
Step 3: Post-matching analysis Stratification (subclassification) based on matched sample
Estimating propensity scores Need relevant conditioning variables Obtain “best” logistic regression (i.e., best
functional forms); however, no way to know Used Multiple Additive Regression Trees (MART)
to run logistic regression. Rand Generalized Boosted Modeling (gbm): Aims for best balance on observed covariates between treated and controlled groups. Iteration stops when the sample average standardized absolute mean difference (ASAM) is minimized.
Example of gbm output: Does gbm reduce the difference between treated and control schools?
STR=treatment group; LTR=control group; ASAM= average standardized absolute mean difference between treatment and control cases; pretreatment covariates: red solid diamonds= p-values before use of gbm weights (if below line then significant); black diamond outline = p-values after weights applied
Point: After using gbm propensity score weights, all pretreatment differences are ns.
Predictors of the propensity score model __________________________________________________________________________________________________________________________________________________________________
Outcome__________________________________________________________________________________________________________________________________________________________ICSTAGG ICSTACA ICSTINT CCCSCOM CCCPROS CCCEREG CCCRAGG
Predictor AggressionAcademic
Competence InternalizingSocial
Competence ProsocialEmotion
RegulationRelational
Aggression
Age at baseline (year) Gender female (male is reference) Race (Other is reference)
African American White Hispanic Primary caregiver's education (years of schooling) Ratio of income to poverty threshold Primary caregiver full-time employment (part-time is referrence) Father's presence in family: Yes (absence is reference) ICST-aggrestion at baseline ICST-academic competence at baseline ICST-internalizing at baseline CCC-emontion regulation at baseline CCC-social competence at baseline CCC-prosocial at baseline CCC-relational aggression at baseline __________________________________________________________________________________________________________________________________________________________________
Note. Predictors vary by outcome variable. Following the convention of propensity score analysis, we did not include predictors that are highly correlated with the outcome variable.
Propensity score weighting When estimating the treatment effect, can use propensity
scores as sampling weights.(Hirano & Imbens, 2001; McCaffrey et al., 2004; Rosenbaum, 1987)
Suppose p is the propensity score of receiving treatment. Then: Average treatment effect for the treated (ATT):
control weight = p/(1-p) treatment weight = 1
Average treatment effect for the population (ATE): control weight = 1/(1-p) treatment weight = 1/p
Post-optimal-matching analysis For the matched sample created by optimal pair
matching, regress pairwise differences in Y between treated and control cases on pairwise differences in X vector between treated and control cases (Rubin, 1979). In doing so, use the intercept of the regression to estimate the treatment effect and its p-value as a significance test.
For matched sample created by optimal full matching or optimal variable matching, use the signed-rank test of Hodges and Lehmann (1962) to estimate the average treatment effects.
Dose analysis using Matching estimator
• The dose analysis evaluates the outcome difference between a dose group (i.e., low, benchmark, or high) and a comparison group using Matching estimator developed by Abadie et al. (2004).
• Under the exogeneity assumption, this method imputes the missing potential outcome by using average outcomes for individuals with “similar” values on observed covariates.
• The estimator uses the vector norm (i.e., ||x||v=(x’Vx)1/2 with positive definite matrix V) to calculate distances between one treated case and each of the matched multiple nontreated cases, and chooses the outcome of the nontreated case whose distance is the shortest among all as the predicted outcome for the treated case.
Comparing model features
___________________________________________________________________________________________________________________________________Model The Model Controls for:
____________________________________
Level at which treatment was tested
Multiple imputation of missing data
Rater's effect
Selection bias Clustering
____________________________________________________________ ________ ________ ________ ________
Piecewise change with propensity score weighting (ATE) School Yes Yes Yes YesPiecewise change with propensity score weighting (ATT) School Yes Yes Yes YesOptimal pair matching with regression adjustment School Yes Yes Yes YesOptimal full matching with Hodges-Lehmann test Student Yes Yes Yes NoEfficacy subset analysis using Matching estimator Student Yes Yes Yes No___________________________________________________________________________________________________________________________________
Note. Regression models include covariates age at baseline, female, black, white, latino, primary caregiver education, income-to-poverty ratio, primary caregiver fulltime employment, father in household, and midyear change in teacher.
FINDINGS
Findings: Treatment effects measured by changes in the 3rd Grade (g34)
Outcome VariableHypothetical
Sign
Piecewise Change
with Propensity
Score (gbm)
Weighting Model 1:
ATE Grade 3
Piecewise Change
with Propensity
Score (gbm)
Weighting Model 2:
ATT Grade 3
Change Score Using
Optimal Pair-Matching (gbm) and Regression Adjustment
Grade 3
Change Score Using Optimal Full-
Matching (gbm) and Hodges-
Lahmann Aligned
Rank Test Grade 3
Approximate Sample Size Used in Analysis ≈571 ≈571 ≈542 ≈571ICSTAGG - Aggression - -.10 -.04 -.10 -.01ICSTACA - Academic competence + .12+ .08 .11 -.08***ICSTINT - Internalizing - .14+ .13+ .14 .17CCCSCOM - Social competence + -.22 -.25+ -.25 -.25CCCPROS - Prosocial + -.25 + -.26 + -.25 -.19+CCCEREG - Emotion regulation + -.16 -.18 -.20 -.24CCCRAGG - Relational aggression - .17 .21* .18 .28__________________________________________________________________________________________________________________________________________________________________________________
*** p <.001, ** p < .01, * p < .05, + p < .1, two-tailed.
Findings: Treatment effects measured by changes in the 4th Grade (g34)
Outcome VariableHypothetical
Sign
Piecewise Change
with Propensity
Score (gbm)
Weighting Model 1:
ATE Grade 4
Piecewise Change
with Propensity
Score (gbm)
Weighting Model 2:
ATT Grade 4
Change Score Using
Optimal Pair-
Matching (gbm) and Regression Adjustment
Grade 4
Change Score Using
Optimal Full-
Matching (gbm) and Hodges-
Lahmann Aligned
Rank Test Grade 4
Approximate Sample Size Used in Analysis ≈557 ≈557 ≈550 ≈557ICSTAGG - Aggression - -.13 -.14 -.14 -.17ICSTACA - Academic competence + -.13+ -.10 -.11 -.08ICSTINT - Internalizing - -.00 .04 -.02 .18CCCSCOM - Social competence + -.01 -.02 .06 .05CCCPROS - Prosocial + -.00 -.03 .05 .07+CCCEREG - Emotion regulation + -.00 -.01 .06 .03CCCRAGG - Relational aggression - -.12 -.12 -.13 -.06__________________________________________________________________________________________________________________________________________________________________________________
*** p <.001, ** p < .01, * p < .05, + p < .1, two-tailed.
Findings: Treatment effects measured by changes in the 3rd Grade (g345)
Outcome VariableHypothetical
Sign
Piecewise Change
with Propensity
Score (gbm)
Weighting Model 1:
ATE Grade 3
Piecewise Change
with Propensity
Score (gbm)
Weighting Model 2:
ATT Grade 3
Change Score Using
Optimal Pair-
Matching (gbm) and Regression Adjustment
Grade 3
Change Score Using Optimal Full-
Matching (gbm) and Hodges-
Lahmann Aligned
Rank Test Grade 3
Approximate Sample Size Used in Analysis ≈370 ≈370 ≈314 ≈370ICSTAGG - Aggression - -.15 -.12 -.12 -.08ICSTACA - Academic competence + .15 .12 .03 .28ICSTINT - Internalizing - .09 .09 .13 .19+CCCSCOM - Social competence + -.02 -.03 -.04 -.03CCCPROS - Prosocial + -.03 -.05 -.04 -.01CCCEREG - Emotion regulation + -.01 -.01 -.04 .03CCCRAGG - Relational aggression - .09 .09 .07 .07__________________________________________________________________________________________________________________________________________________________________________________
*** p <.001, ** p < .01, * p < .05, + p < .1, two-tailed.
Findings: Treatment effects measured by changes in the 4th Grade (g345)
Outcome VariableHypothetical
Sign
Piecewise Change
with Propensity
Score (gbm)
Weighting Model 1:
ATE Grade 4
Piecewise Change
with Propensity
Score (gbm)
Weighting Model 2:
ATT Grade 4
Change Score Using
Optimal Pair-Matching (gbm) and Regression Adjustment
Grade 4
Change Score Using
Optimal Full-
Matching (gbm) and Hodges-
Lahmann Aligned
Rank Test Grade 4
Approximate Sample Size Used in Analysis ≈414 ≈414 ≈380 ≈414ICSTAGG - Aggression - -.17 -.21 -.18 -.21+ICSTACA - Academic competence + -.08 -.08 -.06 -.13ICSTINT - Internalizing - -.07 -.03 -.02 .13CCCSCOM - Social competence + .15+ .18* .20 .13CCCPROS - Prosocial + .15 + .14 + .17 .11+CCCEREG - Emotion regulation + .17+ .20+ .22 .21+CCCRAGG - Relational aggression - -.15 -.18 -.18 -.15+__________________________________________________________________________________________________________________________________________________________________________________
*** p <.001, ** p < .01, * p < .05, + p < .1, two-tailed.
Findings: Treatment effects measured by changes in the 5th Grade (g345)
Outcome VariableHypothetical
Sign
Piecewise Change
with Propensity
Score (gbm)
Weighting Model 1:
ATE Grade 5
Piecewise Change
with Propensity
Score (gbm)
Weighting Model 2:
ATT Grade 5
Change Score Using
Optimal Pair-Matching (gbm) and Regression Adjustment
Grade 5
Change Score Using
Optimal Full-
Matching (gbm) and Hodges-
Lahmann Aligned
Rank Test Grade 5
Approximate Sample Size Used in Analysis ≈433 ≈433 ≈350 ≈433ICSTAGG - Aggression - -.08 -.08 -.12 -.01ICSTACA - Academic competence + .20* .20* .16 .16ICSTINT - Internalizing - -.17+ -.18+ -.17 -.24+CCCSCOM - Social competence + .27** .25** .29 .28*CCCPROS - Prosocial + .29*** .27*** .29 .28*CCCEREG - Emotion regulation + .24* .22* .27 .27+CCCRAGG - Relational aggression - -.20 -.22 -.24 -.24+__________________________________________________________________________________________________________________________________________________________________________________
*** p <.001, ** p < .01, * p < .05, + p < .1, two-tailed.
Findings: Growth curve and dose models (g345)
Outcome VariableHypothetical
Sign
Growth curve
(intervntn*months)
ATE
Growth curve
(intervntn*months)
ATT
Dose (cum. minutes)Grade 3
ATE
Dose (cum. minutes)Grade 4
ATE
Dose (cum. minutes)Grade 5
ATE(9 months) (9 months) (8 hours) (8 hours) (8 hours)
Approximate Sample Size Used in Analysis ≈472 ≈472 ≈370 ≈414 ≈433ICSTAGG - Aggression - -.01 -.02 -.00 -.09 .12ICSTACA - Academic competence + .03 .03+ .03+ .08 -.03ICSTINT - Internalizing - -.13*** -.14*** .02 -.06 -.08CCCSCOM - Social competence + .07** .06** .03 .10 -.05CCCPROS - Prosocial + .06*** .05* .02 .07+ -.10CCCEREG - Emotion regulation + .07*** .07*** .02 .07 -.10CCCRAGG - Relational aggression - -.05* -.05* .00 -.19** .22+__________________________________________________________________________________________________________________________________________________________________________________
*** p <.001, ** p < .01, * p < .05, + p < .1, two-tailed.
Low Confidence:Do Not Cite
Findings of Efficacy Subset Analysis: Treatment effects measured by changes in the 3rd and 4th Grades (g34)
__________________________________________________________________________________________________________________________________________________________________________________
Outcome VariableHypothetical
Sign
Low Exposure
(<900) versus
Comp. (0) Grade 3
Benchmark Exposure (900-1044)
versus Comp. (0) Grade 3
High Exposure (1045+) versus
Comp. (0) Grade 3
Adequate Exposure
(240+) versus Comp. (0) Grade 4
Approximate Sample Size Used in Analysis 343 372 446 545ICSTAGG - Aggression - .11 -.09 .01 -.25**ICSTACA - Academic competence + -.10 .10 .06 -.12ICSTINT - Internalizing - -.04 .10 .15 .09CCCSCOM - Social competence + -.50*** -.17+ -.23* .08CCCPROS - Prosocial + -.51*** -.20* -.24** .06CCCEREG - Emotion regulation + -.39** -.21* -.16+ .08CCCRAGG - Relational aggression - .38*** .10 .21* -.22**__________________________________________________________________________________________________________________________________________________________________________________
*** p <.001, ** p < .01, * p < .05, + p < .1, two-tailed.
Findings of Efficacy Subset Analysis: Treatment effects measured by changes in the 3rd, 4th, and 5th Grades (g345)
Outcome VariableHypothetical
Sign
Low Exposure
(<900) versus
Comp. (0) Grade 3
Benchmark Exposure (900-1044)
versus Comp. (0) Grade 3
High Exposure (1045+) versus
Comp. (0) Grade 3
Adequate Exposure
(240+) versus Comp. (0) Grade 4
Adequate Exposure
(240+) versus Comp. (0) Grade 5
Approximate Sample Size Used in Analysis 240 260 295 354 354ICSTAGG - Aggression - .16 -.08 -.10 -.30*** -.10ICSTACA - Academic competence + .10 .14 .03 -.06 .10ICSTINT - Internalizing - -.17 .19 .14 .04 -.24*CCCSCOM - Social competence + -.28 -.08 -.02 .22* .29**CCCPROS - Prosocial + -.43* -.08 -.07 .20* .30**CCCEREG - Emotion regulation + -.27 -.09 .03 .23* .27**CCCRAGG - Relational aggression - .36+ .05 .11 -.31** -.20+__________________________________________________________________________________________________________________________________________________________________________________
*** p <.001, ** p < .01, * p < .05, + p < .1, two-tailed.
Findings of Efficacy Subset Analysis: Treatment effects measured by changes in the 4th & 5th Grades using subsets of Grade 3 exposure (g345)
Outcome VariableHypothetical
Sign
G3 Low Exposure
(<900) versus
Comp. (0) Grade 4
G3 Benchmark Exposure (900-1044)
versus Comp. (0) Grade 4
G3 High Exposure (1045+) versus
Comp. (0) Grade 4
G3 Low Exposure
(<900) versus
Comp. (0) Grade 5
G3 Benchmark Exposure (900-1044)
versus Comp. (0) Grade 5
G3 High Exposure (1045+) versus
Comp. (0) Grade 5
Approximate Sample Size Used in Analysis 221 246 285 234 252 288ICSTAGG - Aggression - -.09 -.45*** -.32** .34 -.05 -.26*ICSTACA - Academic competence + .61** -.09 -.22* .19 .17 .02ICSTINT - Internalizing - -.16 -.02 .12 -.50+ -.12 -.17CCCSCOM - Social competence + .38* .09 .19* .27 .29* .28**CCCPROS - Prosocial + .36* .12 .15 .33 .25+ .28**CCCEREG - Emotion regulation + .39* .10 .22* .09 .29* .28**CCCRAGG - Relational aggression - -.19 -.24+ -.32** .09 -.09 -.33**__________________________________________________________________________________________________________________________________________________________________________________
*** p <.001, ** p < .01, * p < .05, + p < .1, two-tailed.
SummaryFrom different methods of analysis, a pattern of small, cumulative program effects emerges across grades 3, 4, and 5. These analyses exclude one poorly performing that was dissolved in third year of the study.
Positive cumulative effects on: Social competence – including
• Prosocial behavior and • Skill in regulating emotions
Internalizing behavior Relational aggression
By HLM and efficacy subsets, promising effects observed on: Academic competence Aggression
Focuses on Program Development and Steps in Intervention Research
Focuses on (Selection) Bias-Correction Statistical Methods
For a description of Making Choices and copies of sample lessons, see http://ssw.unc.edu/jif/makingchoices/
ReferencesAbadie, A., Drukker, D., Herr, J. L., & Imbens, G. W. (2004). Implementing matching estimators for average
treatment effects in Stata. The Stata Journal 4(3), 290-311.
Fraser, M. W., Day, S. H., Galinsky, M. J., Hodges, V. G., & Smokowski, P. R. (2004). Conduct problems and peer rejection in childhood: A randomized trial of the Making Choices and Strong Families programs. Research on Social Work Practice, 14(5), 313-324.
Fraser, M. W., Galinsky, M. J., Smokowski, P. R., Day, S. H., Terzian, M. A., Rose, R. A., & Guo, S. (2005).Social information-processing skills training to promote social competence and prevent aggressive behavior in third grade. Journal of Consulting and Clinical Psychology, 73(6), 1045-1055.
Fraser, M. W., Nash, J. K., Galinsky, M. J., & Darwin, K. E. (2000). Making choices: Social problem-solving skills for children. Washington, DC: NASW Press.
Fraser, M. W., Richman, J. M., Galinsky, M. J., & Day, S. H. (2009). Intervention research: Developing social programs. New York, NY: Oxford University Press.
Galinsky, M. J., Terzian, M. A., & Fraser, M. W. (2006). The art of group work practice with manualized curricula. Social Work with Groups, 29(1), 11-26.
Guo, S., & Fraser, M. W. (in press). Propensity score analysis: Statistical methods and applications. Thousand Oaks, CA: Sage Press.
Hansen, B. B. (2007). Optmatch: Flexible, optimal matching for observational studies. R News, 7, 18-24.
Heckman, J. J. (1979). Sample selection bias as a specification error. Econometrica, 47, 153-161.
Heckman, J. J. (2005). The scientific model of causality. Sociological Methodology, 35, 1-97.
Hirano, K., & Imbens, G. (2001). Estimation of causal effects using propensity score weighting: An application to data on right heart catheterization. Health Services and Outcomes Research Methodology, 2, 259-278.
Hodges, J. L., & Lehmann, E. L. (1962). Rank methods for combination of independent experiments in the analysis of variances. Annals of Mathematical Statistics, 33, 482-497.
McCaffrey, D. F., Ridgeway, G., & Morral, A. R. (2004). Propensity score estimation with boosted regression for evaluating causal effects in observational studies. Psychological Methods, 9, 403-425.
Rosenbaum, P. (1987). Model-based direct adjustment. Journal of the American Statistical Association, 82, 387-394.
Rosenbaum, P. (2002). Observational studies (2nd ed.). New York: Springer-Verlag. Rubin, D. B. (2008). For objective causal inference, design trumps analysis. The Annals of Applied
Statistics, 2(3), 808-840.
Rubin, D. B. (1979). Using multivariate matched sampling and regression adjustment to control bias in observational studies. Journal of the American Statistical Association,74(366), 318-328.
Rubin, D. B. (1979). Using multivariate matched sampling and regression adjustment to control bias in observational studies. Journal of the American Statistical Association,74(366), 318-328.
Von Hippel, P. T. (2007). Regression with missing Ys: An improved strategy for analyzing multiply imputed data. Sociological Methodology, 37(1), 83-117.
Note. CCC=Carolina Child Checklist; ICST = Interpersonal Competency Scale - Teacher
S t e p s i n I n t e r v e n t i o n R e s e a r c h
Step 1:
Specify Problem and Develop Program
Theory
Step 2:
Create and Revise Program Materials
Step 3:
Refine and Confirm Program Components
Step 4:
Assess Effectiveness in Variety of Settings and Circumstances
Step 5:
Disseminate Findings and Program Materials
Stage 1
Formulation
Stage 3
Differentiation
Stage 2
Revision
Stage 4
Translation/Adaptation
Four Stages in the Development of Program Manuals Integrated across the Five Steps in Intervention Research
Source: Fraser, M. W., Richman, J. M., Galinsky, M. J., & Day, S. H. (2009). Intervention research: Developing social programs. New York, NY: Oxford University Press.