using early warning indicators to identify students at highest risk of dropping out

Using Early Warning Indicators to Identify Students at Highest Risk of Dropping Out

Ruth Curran NeildCenter for Social Organization of Schools, Johns Hopkins University

Copyright © 2010. All rights reserved.

Relationship to college and career readiness

• Earning high school diploma is a critical juncture in the pipeline

• Provides a conceptual model for tracking likelihood of young adult success in postsecondary education and/or work


The Checklist Manifesto


All professions in the 21st century face increasing

complexity in their work


The Modern Challenge: Keeping Track of a Lot of “Moving Pieces”

• It’s easy for well-trained professionals to:• Forget• Fail to communicate• Overlook• Fail to “connect the dots”


Most errors of are those of ineptitude

Mistakes that occur because we do not make proper use of what we know

(As contrasted with errors of ignorance – mistakes that arise from not knowing what to do)


Like other professions, teachers operate in an environment of increasing complexity

•Increased responsibility for outcomes of all students, including those who are disengaging from school

•Increased responsibility to “individualize” education - to find the “right solution” or “right fit” for each student

•Substantial amounts of data about students collected and made available to teachers


Early Warning Indicator Systems enable teachers to:

• Use empirically-developed data indicators that are most predictive of a given outcome as a “flag” that a student are in trouble

• Track interventions that have been assigned to particular students

• Systematically track associations between interventions and outcomes for students at their school


Logic of Early Warning Indicators Of High School Dropout


There are many underlying reasons for dropping out of school

• Social science research using secondary data sets has helped us to understand correlates of dropping out and…most importantly, that…

• Dropping out is the culmination of a gradual process of disengagement from school


For educators, this research has real limitations

How do we know which specific individuals are most likely to drop out so that we can target interventions to them?

How early in students’ careers can we reliably identify those on the path to dropping out?

Can we identify students with readily-available data or do we need specialized assessments?


Developed in the Context of “Dropout Factory” Schools

• At these schools, 40% or more of the students fail to graduate

• Family income, family structure, race/ethnicity, scores on nationally-normed tests are usually the same or within a narrow band in these schools


Four questions about EWIs

• What are the characteristics of a good EWI system?

• What are the signals?

• What technological and organizational infrastructure is needed to “capture” the signal?

• What can schools and districts do once the signals are identified and captured?


Characteristics of a good EWI system

High accuracy: A high percentage of students with the “signals” drop out. Conversely, a low percentage of students without the “signals” graduate.

High yield: These “signals” capture most of the dropouts (avoiding the “1% problem”).

Accessible data: Data that provide the “signals” are readily available and relatively inexpensive to access.

Empirically developed: The “signals” are identified through analysis of longitudinal data for prior cohorts of students.


How did we identify the “signals” of eventual dropout?

• Empirical analysis of cohorts in Philadelphia, starting with 6th graders (Balfanz, Herzog, & MacIver), and 8th graders (Neild & Balfanz, 2006)

• Data scan of longitudinal student record data– Test scores– Report card grades– Attendance– Special education and ELL status– Gender– Age– Race/ethnic background


Looked for a 75% threshold – why?

• Choosing a “strong signal” – students who are at highest risk of dropping out

• By not making the net too broad, scarce resources can be targeted at those students who are greatest risk


The Big Four in 6th grade

• Failing Math• Failing English• Attendance <80%• At least one poor behavior mark

(Balfanz, Herzog, & MacIver)


8th grade warning signals

• Three factors gave students at least a 75% probability of dropping out:

1. Failing math in 8th grade2. Failing English in 8th

grade3. Attending less than 80%

of the time


54% of the dropouts sent one or more of these signals

in 8th grade


Had an 8th grade “signal”

Did not have an 8th grade signal:

Passed 8th grade EnglishPassed 8th grade Math

Attended at least 80% of the time


9th Grade signals

• Three factors gave students at least a 75% probability of dropping out:

1. Earning fewer than 2 credits 2. Not being promoted to 10th grade

3. Attending less than 70% of the time


80% of the dropouts sent one or more of these signals in 8th or 9th grade


Technological Infrastructure: Real time data

PSSA

LAST FIRSTAtt

05-06Att to Date

FINAL 05-06 Math

FINAL 05-06 Rdg

MP1 06-07 Math

MP1 06-07 Rdg

MP1 Behavior Marks 2006-07

Mar-06EXIT

Jun-06Dec-06

Change, Jun-Dec

Student A 88% 95% D D NA NA NABELOW BASIC Rdg 927 Math

11574 NA NA

Student B 96% 99% B C C C NoneBASIC Rdg

1259 PROF Math 1335

5 5.5 0.5

Student C 85% 90% C D C CFails to be attentive

(Math)

BELOW BASIC Rdg 968

PROF Math 1335

4 4 0

Student D 97% 100% A B B B NonePROF Rdg

1280 ADV Math 1544

5.5 6 0.5

COURSES READING LEVEL

2006-07 School A 6th g., Teacher A

Does NOT: complete work on time; follow school

rules; make appropriate transitions; organize self

ALL AREAS

None

STUDENTS ATTENDANCEBEHAVIOR

Where Student Needs Support

Final Behavior Marks 2005-2006

Does NOT: accept responsibil. for choices; complete work on time;

follow school rules; handle conflict; show positive

attitude


Conceptual frame for intervention

Whole school interventions

Targeted Interventions

Intensive Interventions

More labor intensive

More specialized

More costly


Organizational Infrastructure

TEAMS of Teachers, ideally all teaching the same group of

students

Supported by…

“Near-peers” to nag and nurture

Links to social services


Implications for College and Career Readiness

Possibility of using indicators across systems to address readiness

Connecting school district and local college data to identify high school predictors of key postsecondary outcomes, such as:

Placement out of remedial coursesOverall credit accumulation and in key areasReturn for a second semester or a second year

EXAMPLE


New York City

EXAMPLE



Possibility of using indicators across systems to address readiness

There is a great deal that is unknown about whether there are readily accessible, high accuracy, high-yield high school predictors of postsecondary outcomes



Possibility of using indicators within a higher education system to identify students at-risk of course failure

Survey data about study habits in high school and other non-cognitive predictors, combined with data on class attendance and interim grades

EXAMPLE


The Checklist Manifesto

The purpose of a good checklist is not to fill out paperwork or to prove to others that we’ve “covered our bases,” but to help well-trained professionals cope with the complexity and detail of their work in the modern world.

EWI System

teachers

keeping students on track


Ruth Curran NeildCenter for Social Organization of SchoolsJohns Hopkins University

[email protected]

using early warning indicators to identify students at highest risk of dropping out

Documents