First Step to Success: An Evidence-based Secondary

Level InterventionW. Carl Sumi, Ph.D. and Darcey Edwards, Ph.D.

SRI International

Northwest PBIS ConferenceMarch 8, 2010Corvallis, OR

National Effectiveness Study of First Step to SuccessNational Center for Special Education Research (NCSER)

Institute of Education Sciences (IES)

U.S. Department of Education

NBRCC Staff: SRI InternationalW. Carl Sumi, Ph.D. Mary Wagner, Ph.D.Frances Bergland, Hal Javitz, Ph.D.Michelle Woodbridge, Ph.D. Patrick Thornton, Ph.D.

NCSER Project OfficersJacquelyn A. Buckley, PhD

3

Outline of Presentation

Provide description of First Step to Success.Describe National Effectiveness Study of

First Step and present preliminary outcome data.

Introduce and discuss Practice Guide: “Reducing Behavior Problems in the Elementary School Classroom.”

The information presented here does not necessarily reflect the views or policies of the U.S. Department of Education or Institute of Education Sciences nor does mention of trade names, commercial products, or organizations imply endorsement by the U.S. government.

4

Social skill ratings of elementary/ middle school students

Sources: Wave 1 SEELS parent interviews, 2000 and 2001, Social Skills Rating Scale (Gresham and Elliott, 1990) national norms.

Statistical significance: *p < .05.

Percentage with low social skills rating by parents

5

School suspensions and expulsions

Percentage ever suspended or expelled

Sources: Wave 1 SEELS and NLTS2 parent interviews, 2000 and 2001; National Household Education Survey, 1999.

Statistical significance: * p <.05.

6

Examples of problem behaviorsArguingStealingDisturbing othersHyperactive, inattentiveTantrumsPhysical aggression- hitting, kicking, biting,

scratching, spittingImpulsive, attention problemsUncontrollable cryingNot complying with teacher instructions or directivesOppositional behaviors

7

Consequences of problem behaviors

Less “time-on-task”Poor grades, poor achievementIncreased negative interactions with

peers, family members, teachersPeer rejectionDelinquencySubstance abuseNegative cycle of poor achievement and

problem behavior

8

Typical responses to problem behaviors

PunishmentExclusion/isolationLack of positive interactionsIneffective, inconsistent consequencesNegative consequencesInadvertent reinforcing of problem

behaviors

9

10

11

Where do we go from here?

Since a large percentage of children in our schools exhibit behavior problems to a varying degree what should teachers, administrators, families, school staff, and other professionals do?

=> Implement evidence-based interventions and programs that have proven positive effects for school-aged students!

12

First Step to SuccessA short-term (approximately 3 months),

secondary level early intervention that targets children exhibiting problem behaviors (e.g., aggression, antisocial, oppositional, impulsive, hyperactive, etc).

Goals of First Step are to enhance children’s social competence skills and school engagement in an effort to decrease problem behaviors and prevent children from developing more serious antisocial conditions.

First Step relies on parents, teachers, and children in order to modify and, in turn, reward behavior both at school and at home.

13

First Step to Success

Major components of the program include:A proactive, class-wide screening to

nominate a student for the program.CLASS: A classroom-based curriculum used

to teach and reinforce appropriate behaviors.A family-based component called

“homeBase,” a brief child-focused program for the caregiver(s).

14

First Step: Class-wide Screening

Teachers complete the Systematic Screening for Behavior Disorders (SSBD).

Teachers are asked to evaluate all students on various measures of antisocial behavior.

Ultimately, teachers identify those children at risk for, or already exhibiting, internalizing or externalizing behavior problems.

15

First Step: Class-wide Screening

SSBD Stage 1 Review class roster, identify five children in the class who exhibit

externalizing behaviors, and rank order them from 1 to 5. SSBD Stage 2: For the top 3 ranked students the teacher

completes the: Critical Events Index – 32 yes/no items (e.g., steals, sets fires, is

physically aggressive). Adaptive Behavior Index – Positive, prosocial behaviors rated on a

1-5 scale (e.g., follows rules, cooperates with peers). Maladaptive Behavior Index – Antisocial behaviors rated on a 1-5

scale (e.g., manipulates others, refuses to participate in activities with other students at recess).

Implement First Step with child with highest Stage 2 score.

16

First Step to Success: CLASSCLASS – Contingencies for Learning Academic

and Social Skills Intended to target and correct behavior problems.One child per classroom can receive the intervention at

a time.Children learn how to:

Attend to the teacher.Get along with others.Participate in activities .

Three phases which last 30 program days:Coach/consultant phase Teacher phase Maintenance phase

17

First Step: CLASS- Coach phase

The coach:Can be school psychologist, social worker,

counselor, behavior specialist.Observes the focus child.Meets with caregiver and teacher.Provides the materials for the teacher and child.Begins the program with the child.Teaches acceptable behavior 1:1 with child.Monitors student progress.Phones parent regularly.Implements homeBase starting Day 10.

18

First Step: CLASS- Coach phase

The coach meets with the child and classConducts first meeting with the child.

Child agrees to play the Green/Red card “game.” Child chooses appropriate and meaningful rewardsRole-play appropriate behaviors.

Introduces the Green/Red card game to the class.Introduce “volunteer.”Obtain cooperation from class (positive comments,

model “doing the right thing”).Class can participate in reward activity.

19

First Step: CLASS- Coach phase

Days 1-5: The coach leads programCoach operates Green/Red card in class.

Days 1-4 last 20 minutes and Day 5 is 30 minutes.Coach awards points during game.Child earns reward if 80% of points earned.Coach announces reward to class.

Student must get opportunity for class reward immediately after session.

Child brings Green/Red card home. Coach contacts caregiver each day first five days.

Caregiver rewards child for successful day.

20

First Step: CLASS- Teacher phase

Days 6-20 the teacher takes the leadTeacher operates the Green/Red card game. Monitors student behavior and awards points.

The Green/Red card must be visible during the game.Communicates with coach and caregiver regularly.

Sends the Green/Red card home daily. Immediately delivers reward when earned.Provides child and parent with encouragement and

support. “Catches” the child doing the right thing.As the child progresses through the program the length

of the Green/Red card game increases daily.

21

First Step: CLASS- Maintenance

Days 21-30 are considered the Maintenance phaseTeacher continues to lead program but time is

dedicated to reducing dependence on the intervention and transferring involvement to families.

Rewards and privileges for appropriate behavior are used steadily less by teachers and adopted in the home environment.

Parents are encouraged to substitute praise for these privileges.

During the whole program a child must meet daily performance criteria in order to proceed to the next day of intervention.

22

First Step: CLASS

Additional information:If the child fails to meet criteria one day,

they must repeat that program day. Most children take approximately two

months to complete the school intervention.

23

First Step to Success: homeBase

Purpose is to help parents improve their child’s school adjustment, competence and performance.

Starts on Day 10 of CLASS programSix lessons, approximately 1 hour each

Involve guide and parent-child activities.Coaches emphasize ways in which parents can help

their child with communication and sharing, cooperation, limits-setting, problem-solving, friendship skills and confidence.

Can meet at home, school, or comfortable location.

24

First Step: homeBase- Parent role

Participate in homeBase lessons.Play activity cards for 10 minutes each night with child.

Reward and acknowledge child when s/he brings home the Green/Red cardDo something fun with child when s/he earns

reward at schoolRead a bookTake a walkBake a cakePlay a gameSpecial snack/treatHelp mom/dad on project

25

First Step to Success: homeBase

Week 1: CommunicationChild practices giving informationParent listens and gives encouragement

=>Information gives parents the ability to help children be successful outside the home.

Week 2: CooperationParent & child learn strategiesSticker chart or chart used at home

=>Being cooperative allows a child opportunities to avoid problems.

26

First Step to Success: homeBase

Week 3: Limit settingGiving effective directions and encouragementTime-out procedures

=>Teaching children to follow limits at home leads to self-control and accepting limits.

Week 4: Problem solving Stay calm and brain-storm Parents help to guide, encourage and suggest steps to

achieve goal=>Children who see problems as opportunities rather than

obstacles feel capable.

27

First Step to Success: homeBase

Week 5: Friendship skillsLearning to initiate Empathy and self-controlCooperation

=>Learning friendship skills now provides a base for friendship throughout life.

Week 6: Confidence building=>Self-confidence developed at home provides a

foundation for success outside the home.

28

National Effectiveness Study of First Step to Success: Introduction

Goal 4 grant from the U.S. Department of Education, Institute of Education Sciences, National Center for Special Education Research

Well-developed evidence base for the efficacy of First Step“Manualized” off-the-shelf interventionSolid evaluation framework

Randomized controlled trial in 48 schools in 5 diverse elementary school districts across the country

Evaluators (SRI) independent of program developers (ORI)

29

Study participants

8 to 10 schools in each district Matched on basic demographics and randomly assigned Half in the intervention condition receive First Step Half in the usual-care condition receive typical services

Teachers trained in First Step at conclusion of data collection6 first- through third-grade students in each school

All students screened with SSBD 1 student per class participates in each condition each year In year 2, intervention teachers implement First Step again with

another student

30

First Step to Success Effectiveness Study:Core research questions

Continued…

EffectivenessTo what extent does First Step improve the behavior at

school and the academic performance and participation of students with severe behavior problems?

For what kinds of students does First Step work best and less well?

In what contexts (classroom and school level) does First Step work best and less well?

How do variations in effectiveness relate to variations in implementation fidelity?

31

First Step to Success effectiveness study:Core research questions

MaintenanceAre the effects of First Step sustained

for 1 year? For 2 years?Does maintenance of effects relate

to variations in students or contexts?

32

Student-level information

Student Enrollment SurveyBasic demographics (gender, ethnicity, primary language,

free or reduced-price lunch status)Student Record Survey

School records information:IEP/504 Plan status Instructional settings (i.e., percentage of

instructional time in general education classes)AbsencesOffice Discipline Referrals (ODRs)

Continued…

33

Student-level assessments

Social Skills Rating System (SSRS) –Teacher and Parent versionsSocial skills, problem behaviors, and academic

competenceWoodcock-Johnson III Letter-Word Identification

SubtestReading skills

Oral Reading Fluency (ORF)Ability to read aloud expressively

Academic Engaged Time (AET)Active engagement in relevant academic material over

two 15-minute observations

34

Classroom-level information

Classroom Atmosphere Rating Scale (CARS)30-minute observation of intervention classrooms (e.g.,

student compliance, cooperation, problem solving)Classroom / Teacher Survey

Classroom characteristics (e.g., student enrollment)Teacher characteristics (e.g., years experience, degrees)Teacher support (e.g., training, classroom aides)Teacher self-reported skills to work with students with

behavior problems

35

School-level information

School Characteristics SurveyStudent characteristics (e.g., mobility rate)School climate (e.g., total number of ODRs)Staff and program resources (e.g., number of FTEs)

NCES Common Core of Data (CCD)Extracted data describing participating schools and

districts (e.g., enrollment, teacher/student ratio)

36

Implementation measures

Fidelity Integrity of program (monitored three times throughout

intervention for each participant)Social validity (teacher and parent’s perspective)

Acceptability–General support for interventionPositive effects–for participating student(s) and classroom

AllianceStrength of the relationship between coach and teacher

37

Study timeline

• Screening: SSBD• Baseline: WJIII, ORF, AET, SSRS, Student Record Survey,

Teacher Survey, CARS, School Survey• Posttest: WJIII, ORF, AET, SSRS, Satisfaction (parent),

Social Validity, Alliance• Follow-up: WJIII, ORF, AET, SSRS, Student Record Survey,

Teacher Survey, CARS, School Survey

Screen/baseline Posttest Follow-up 1 Follow-up 2

~10 weeks 1 year 1 year

Intervention

Preliminary Analysis of Intervention Effects:

Academic and Behavior Outcomes

39

Analysis method

Repeated measures: significance reflects effect by time and group.

Effect size of the difference scores. Effect sizes calculated by dividing the

estimated treatment effect by the standard deviation of posttest measurements (using both comparison and intervention students).

40

Preliminary behavior outcomes: Teacher reportedMeasure/Group Assignment¹ Baseline (M) Posttest ESSSRS-Social Skills .93* Intervention 83.4 92.7 Comparison 84.5 85.1SSRS-Problem Behaviors -.54* Intervention 120.1 115.4 Comparison 119.7 119.0SSBD-Adaptive Behavior .63* Intervention 33.6 38.5 Comparison 34.8 35.1SSBD-Maladaptive Behavior -.57* Intervention 32.9 28.1 Comparison 30.4 29.7

1 Intervention Group sample size ranged from 156 to 183; Comparison Group sample size ranged from 127 to 136.* = p < .05.SOURCE: Social Skills Rating System (Gresham and Elliott 1990), Woodcock-Johnson III.

41

Preliminary behavior outcomes: Parent reported

Measure/Group Assignment Baseline (M) Posttest ESSSRS-Social Skills¹ .43*

Intervention 88.8 94.7 Comparison 90.4 89.9SSRS-Problem Behaviors² -.17 Intervention 113.6 108.6 Comparison 114.3 111.7

1 Intervention Group sample size = 138; Comparison Group sample size = 115.2 Intervention Group sample size = 138; Comparison Group sample size = 115.* = p < .05.SOURCE: Social Skills Rating System (Gresham and Elliott 1990).

42

Preliminary academic outcomesMeasure/Group Assignment¹ Baseline (M) Posttest ESAET percent engaged¹ .21 Intervention 0.59 0.73 Comparison 0.60 0.67ORF mean words read correctly .32* Intervention 59.0 71.5 Comparison 64.7 71.5SSRS Academic Competence .40* Intervention 87.2 88.4 Comparison 87.4 86.4WJ III Letter-Word ID .12 Intervention 100.0 101.1 Comparison 100.3 101.61 Intervention Group sample size ranged from 150 to 183; Comparison Group sample size ranged from 123 to 136.

* = p < .05.SOURCE: Social Skills Rating System (Gresham and Elliott 1990), Woodcock-Johnson III.

43

Teacher SSRS Social Skills

p < .05es = .84

44

Teacher SSRS Problem Behavior

p < .05es = -.52

45

SSBD – Adaptive Behavior Index

p < .05es = .59

46

SSBD – Maladaptive Behavior Index

p < .05es = -.57

47

Parent SSRS Social Skills

p < .05es = .44

48

Parent SSRS Problem Behavior

p = .08es = -.21

49

Academic Engaged Time

p = .054es = .22

50

Oral Reading Fluency

p < .05es = .32

51

Teacher SSRS Academic Competence

p < .05es = .35

52

Woodcock Johnson III Letter-Word Identification

Social Validity and Alliance Summary

54

Social Validity and Alliance SummarySocial ValidityTeachers: Mean rating of 3.56

Would recommend the intervention to other teachers. Program had positive effect on child relationships.

Parents: Mean rating of 4.21 I would recommend the program to other parents. I liked getting daily feedback from the green/red card.

AllianceTeachers: Mean rating of 4.58

“The coach is approachable.” “The coach and I trust one another.”

Coaches: Mean rating of 4.53 “The teacher is approachable.” “The teacher and I trust one another.”

55

Next StepsData collection will continue until spring 2011.To follow the progress of the National Effectiveness

Study of First Step to Success or to find out more information visit, http://firststeptosuccess.sri.com.

The What Works Clearinghouse Practice Guide: “Reducing Behavior Problems in the Elementary School Classroom”

57

Purpose of The What Works Clearinghouse

The mission of the WWC is to “be a central and trusted source of scientific evidence for what works in education.”

WWC generates several different products:Intervention reportsTopic reportsQuick reviewsPractice guides

58

Practice Guide: Purpose

Supply discrete recommendations that are intended to be actionable.

Provide a coherent approach to a multifaceted problem.

Explicitly connect each recommendation to the level of evidence supporting it (strong, moderate, or low).

Continued…

59

Practice Guide: Purpose

Use expertise and judgment of a panel to identify the most important research relevant to the recommendations.

Bring the best available evidence to bear on challenges that cannot currently be addressed by single interventions or programs.

60

Released in 2007

61

Released in 2008

62

Practice Guide: Scope

Primary audiences include:General education elementary school

teachers who will implement practicesElementary school/district administrators

who will promote practicesFinal product is “more like a consensus

panel report than a meta-analysis” in terms of breadth and complexity of topic addressed.

63

Reducing Behavior Problems in the Elementary School Classroom Recommendations

Broader points about improving practiceDrawn from evidence about the effectiveness

of specific programs and practices, including:Suggestions for how to implement in school settings

( checklist)• Provide district-, school-, and teacher-level guidelines

Descriptions of potential roadblocks• Refute myths with evidence• Suggest solutions

64

Identify specifics of problem behavior and conditions that prompt and reinforce it. Level of Evidence: Moderate Implementation Guidelines

Observe and record frequency of problem behavior.Identify what prompts and reinforces behavior.

Potential Roadblocks and SolutionsProblem behaviors may persist even after intervening.

• Interventions must be implemented with sufficient time and consistency.

• A single behavior may stem from multiple triggers.• Intervene for at least one month, and continue to observe

and record behavior. Try new approach if necessary.

Recommendation 1

65

Modify the classroom learning environment to decrease problem behavior. Level of Evidence: Strong Implementation Guidelines

Reinforce classroom expectations.Adapt/vary instruction

to increase engagement.Potential Roadblocks and Solutions

Teachers do not want to disrupt routines.• Time used to practice new routines will increase quality of

instructional time in the end.• Prepare students well for change; ask students to model

new behaviors as reward for appropriate behavior.

Recommendation 2

66

Teach and reinforce new skills to increase appropriate behavior and preserve a positive classroom climate.Level of Evidence: Strong Implementation Guidelines

Teach replacement skills explicitly.Reinforce appropriate behavior;

withhold reinforcement for inappropriate behavior.

Potential Roadblocks and SolutionsTeachers fear extrinsic rewards undermine student

motivation.• Tie reinforcement to student competence.• Reward students with behavior-specific praise.

Recommendation 3

67

Draw on relationships with colleagues and families for guidance and support.Level of Evidence: Moderate Implementation Guidelines

Build collaborative professional partnerships.Encourage families to participate in reinforcing

appropriate behavior.Potential Roadblocks and Solutions

Faculty meetings can be a waste of teachers’ time.• Administrators should encourage a culture of professional

learning.• Use time together productively to joint problem-solve.

Recommendation 4

68

Implement schoolwide strategies to reduce negative and foster positive interactions.Level of Evidence: Moderate Implementation Guidelines

Involve school improvement team in collecting data about school “hot spots.”

Adopt program that aligns with school climate.Potential Roadblocks and Solutions

Packaged programs may be too costly.• Consider evidence-based programs that meet school needs.• If too costly, encourage school staff to observe patterns of

problem behavior to assist in formulating an intervention.

Recommendation 5

69

Principles

Trusting and supportive relationships lay the foundation for positive behavior.

There is increased need for building cultural competence among school communities.

Collecting data is critical in targeting resources and changing strategies to improve behavior.

70

To download and printGo to:http://ies.ed.gov/ncee/wwc/publications/practiceguides/or go to whatworks.ed.gov and find the Practice Guide under the Publications link.

71

Q & A

Final Thoughts and Wrap up