contract no.: ed-98-co-0073 matheiv atica policy research...

Contract No.:

ED-98-CO-0073MPR Reference No.: 8571-010 MATHEIV ATICA

Policy Research, Inc.

Evaluation of theFederal TalentSearch Program:Phase II and PhaseIII FeasibilityReport

Final Report

October 2000

Myles MaxfieldMaggie CahalanTim Silva

Justin HumphreyMelissa Thomas

Submitted to:

Submitted by:

U.S. Department of Education

Mathematica Policy Research, Inc.Planning and Evaluation Service

600 Maryland Avenue, S.W.400 Maryland Avenue, S.W.

Suite 550Room 6W200

Washington, DC 20024Washington, DC 20202

Project Officer:

Project Director:Sandra Furey

Maggie Cahalan

ii

DISCLAIMER

The views, opinions, and approaches presented herein represent the opinions of the authorsand do not necessarily represent the policies or positions of the U.S. Department ofEducation.

Disclaimer

ACKNOWLEDGEMENTS

The authors would like to acknowledge the contributions of Sandra Furey and DavidGoodwin of the Planning and Evaluation Service, U.S. Department of Education. Theircomments on the technical approaches presented herein were very helpful. Any errors andomissions are the sole responsibility of the authors.

Acknowledgements

CONTENTS

Chapter

Page

EXECUTIVE SUMMARY xv

INTRODUCTION 1

A. THE CONGRESSIONAL MANDATE FOR THE STUDY 1

B. TALENT SEARCH PROGRAM BACKGROUND 2

1. Goals and Services 32. TS in the Context of Other TRIO Programs 4

C. CONTEXT FOR THE STUDY: CHANGES SINCE THE 1960S INNATIONAL STATISTICS RELATED TO EDUCATIONENROLLMENT 6

D. PREVIOUS STUDIES OF TALENT SEARCH 7

E. STUDY OBJECTIVES, COMPONENTS AND RESEARCHQUESTIONS 8

1. Components of the Study 92. Study Questions 9

F. THE CONCEPTUAL FRAMEWORK FOR THE STUDY 10

G. STRUCTURE OF THE REMAINDER OF THE REPORT 13

II

STUDIES USED TO INFORM THE FEASIBILITY REPORT 15

A. PERFORMANCE REPORTS AND TARGET SCHOOL ANALYSIS 15

1. Project and Participant Demographics 162. Target Schools and Percentage of Eligible Served 193. Services Provided by Talent Search 234. Objectives and Outcomes 25

B. Project Feedback On The Study 27

1. Project Director Feedback Discussions and COE Fax Responses 28-2. Project Survey 29

C. CASE STUDIES 32

vi

CONTENTS

Chapter

Page

D. EXPLORATION OF THE POTENTIAL USE OF FEDERAL AIDRECORDS AS AWAY TO TRACK PARTICIPANT OUTCOMES 34

1. Using the Aid Files as an Indicator of Performance 34

E. USING STATE-LEVEL DATABASES AND ADDITIONALSOURCES OFPOST-SECONDARY DATA 37

F. CONCLUSIONS FROM FEASIBILITY WORK THUS FAR 37

II

STUDIES USED TO INFORM THE FEASIBILITY REPORT 15

A. PERFORMANCE REPORTS AND TARGET SCHOOL ANALYSIS 15

1. Project and Participant Demographics 162. Target Schools and Percentage of Eligible Served 193. Services Provided by Talent Search 234. Objectives and Outcomes 25

B. Project Feedback On The Study 27

1. Project Director Feedback Discussions and COE Fax Responses 282. Project Survey 29

C. CASE STUDIES 32

D. EXPLORATION OF THE POTENTIAL USE OF FEDERAL AIDRECORDS AS AWAY TO TRACK PARTICIPANT OUTCOMES 34

1. Using the Aid Files as an Indicator of Performance 34

E. USING STATE-LEVEL DATABASES AND ADDITIONALSOURCES OFPOST-SECONDARY DATA 37

F. CONCLUSIONS FROM FEASIBILITY WORK THUS FAR 37

Contents

vii

CONTENTS

Chapter

Page

IV

PHASE II NATIONAL-LEVEL IMPACT STUDY DESIGN 73

A. POLICY QUESTIONS 73

B. AGGREGATE SCHOOL DISTRICT APPROACH 74

1. Experimental Design 742. Propensity Model 743. Database Construction 754. Sampling 755. Analysis : 75

C. ALTERNATIVE APPROACHES INVESTIGATED 76

1. NELS 88/1994 762. Geo-Coding Individual FFA Records 773. Lists of HS Completers from School Districts 78

D. SUM IARY 78

E. REPORT AND BRIEFING 79

V

PHASE III SERVICE MIX DEMONS .RATION EVALUATIONDESIGN 81

A. PURPOSE OF THE STUDY AND HOW IT RELATES TO PHASE II 81

B. ES'IED SUPPLEMENTAL SERVICE MIXES 82

C. EXPERIMENTAL DESIGN 84

D. TIMELINE 85

E SAMPLE DESIGN AND RANDOM ASSIGNMENT 87

F. DATA ACQUISITION PLAN 91

G. IMPLEMENTATION STUDY 93

Contents

viii

CONTENTS

Chapter

Page

ESTIMATES OF COST AND TASK SCHEDULE FORPROPOSED PHASE II AND PHASE III STUDIES 95

A. OVERVIEW OF STUDIES AND COSTS 96

1. State impact studies using state secondary and postsecondary databases,project participant level records, and federal aid application and receiptfiles 98

2. Demonstration studies of alternative innovative services 98

B. SUMMARY OF COSTS 99

C. LIST OF TASKS AND DELIVERABLES FOR PHASE II 100

D. DETAIL OF TASKS FOR THE PROPOSED PHASE IIIDEMONSTRATION STUDIES 100

REFERENCES 103

APPENDIX A: HIGHLIGHTS OF HIGH SCHOOL COMPLETIONAND COLLEGE ENROLLMENT STATISTICS:1960-1999 107

APPENDIX B: 1992 TALENT SEARCH DESIGN CONFERENCESSUMMARY OF CONCLUSIONS 111

APPENDIX C: SUMMARY OF FIRST TECHNICAL REVIEWPANEL MEETING FOR THE NATIONALEVALUATION OF TALENT SEARCH 117

APPENDIX D: MINU IES OF PROJECT DIRECTORTELECONFERENCE 129

APPENDIX E TALENT SEARCH DATABASES IN FIVE STATES 133

Contents

TABLES

Table

Page

1.1

TRIO FUNDING, NUMBER OF GRANTS, AND AVERAGEAWARD, AMOUNT PER PERSON SERVED AND NUMBERFUNDED TO SERVE 1999 5

1.2

TRIO FUNDING LEVELS IN CONSTANT 1999 DOLLARS 5

1.3

STUDIES OF TALENT SEARCH AND EOC 8

11.1

NUMBER OF TALENT SEARCH PROJECTS AND NUMBER OFPARTICIPANTS BY TYPE OF HOST INSTITUTION: 1998-99 16

11.2

NUMBER OF TALENT SEARCH TARGET SCHOOLS AND ESTIMATEDNUMBER OF SCHOOLS SERVING TARGET AGE GROUP WITHIN THESTATE 19

11.3

NUMBER OF MIDDLE AND HIGH SCHOOL STUDENTS SERVED BYTALENT SEARCH, NUMBER ELIGIBLE FOR THE FREE LUNCHPROGRAM IN GRADES 6-12 IN TARGET SCHOOLS AND ALLSECONDARY SCHOOLS, AND NUMBER SERVED BY TALENTSEARCH AS A PERCENTAGE OF THE NUMBER ELIGIBLE FOR THEFREE LUNCH PROGRAM IN THE TARGET SCHOOLS AND ALLSECONDARY SCHOOLS, BY STATE: 1998-99 21

11.4

ESTIMATES OF THE PERCENTAGE OF INCOME-ELIGIBLECHILDREN SERVED BY TALENT SEARCH IN THE UNITED STATESUSING CENSUS DATA ON POVERTY 23

11.5

PERCENTAGES OF PROJECTS OFFERING SERVICE TOPARTICIPANTS, BY GRADE LEVEL: 1998-99 24

11.6

PERCENTAGES OF PARTICIPANTS RECEIVING SERVICES, ANDAVERAGE NUMBER OF SESSIONS PER PROJECT: 1998-99 24

11.7

PERCENTAGES OF PARTICIPANTS RECEIVING EACH TYPE OFSERVICE, COMPARING PERFORMANCE REPORT DATA FROM 1990-91 AND 1998-99 25

11.8

COMPARISON OF APPROVED OBJECTIVES AND ACTUALACHIEVEMENTS FOR MIDDLE SCHOOL AND HIGH SCHOOLOUTCOMES : 1998-99 25

TABLES

Table

Page

11.9

COMPARISON OF APPROVED OBJECTIVES AND ACTUALACHIEVEMENTS FOR POSTSECONDARY OUTCOMES: 1998-99 26

11.10

DISTRIBUTION OF POSTSECONDARY PLACEMENT OFPARTICIPANTS BY SECTOR: 1998-99 27

11.1 1

TALENT SEARCH PROJECT DIRECTOR RESPONSES TO QUESTIONASKING WHICH SERVICES THEY WOULD STUDY 30

11.12

TYPE OF RECORDS ON PARTICIPANTS KEPT BY TALENT SEARCHPROJECTS TALENT SEARCH PROJECT 31

11.13

VARIABLES PRESENT ON THE FEDERAL AID FILES 35

11.14

RESULTS OF MATCHING STUDENT SUPPORT SERVICES NATIONALSTUDY STUDY WITH FINANCIAL AID FILES 36

111.1

TALENT SEARCH PROJECTS IN TEXAS 47

111.2

VARIABLES INCLUDED IN THE TEXAS PUBLIC EDUCATIONINFORMATION MANAGEMENT SYSTEM 49

111.3

VARIABLES INCLUDED IN THE TEXAS HIGHER EDUCATIONDATABASE 50

111.4

TALENT SEARCH PROJECTS IN INDIANA 50

111.5

DATA COLLECTED ON THE STATE OF INDIANA'S ICPACPROGRAM ENROLLMENT FORMS 54

111.6

TALENT SEARCH PROJECTS IN FLORIDA 55

111.7

FLORIDA DEPARTMENT OF EDUCATION (DOE) INFORMATIONDATABASE REQUIREMENTS — AUTOMATED STUDENT DATAELEMENTS 58

111.8

THE STATE UNIVERSITY SYSTEM BOARD OF REGENTS DATABASE(FLORIDA) - STUDENT DATA ELEMENTS 59

Contents

xi

TABLES

Table

Page

111.9

THE STATE BOARD OF COMMUNITY COLLEGES DATABASE(FLORIDA) — STUDENT DATA ELEMENTS 60

111.10

TALENT SEARCH PROJECTS IN MASSACHUSETTS 61

111.11

SUMMARY OF 11EMS INCLUDED IN THE MASSACHUSETTSSTATEWIDE SCHOOL AND STUDENT MANAGEMENTINFORMATION SYSTEM 64

111.12

TALENT SEARCH PROJECTS IN NORTH CAROLINA 65

111.13

SUMMARY OF 11'EMS INCLUDED IN THE NORTH CAROLINASTUDENT INFORMATION SYSTEM 67

111.14

SUMMARY OF ITEMS INCLUDED IN THE NORTH CAROLINAHIGHER EDUCATION DATABASE 67

V.1

EXPERIMENTAL DESIGN OF THE PLAN VARIATION STUDY 86

V.3

MINIMUM DETECTABLE DIFFERENCES (MDD) 91

V.4

SERVICE MIX DEMONSTRATION DATA COLLECTION PLAN 94

VI. 1

SUMMARY OF COST ESTIMATES 99

VI.2

TASKS AND DELIVERABLE SCHEDULE FOR PHASE II 100

VI.3

TASK STRUCTURE AND SCHEDULE FOR PHASE III 102

Contents

FIGURES

Figure

Page

I.1

EDUCATIONAL TALENT SEARCH WITHIN THE EDUCATIONALSYSTEM 11

1.2

POTENTIAL PATHWAYS FOR TALENT SEARCH TO INFLUENCESTUDENTS' PURSUIT OF POSTSECONDARY EDUCATION 12

II.1

PARTICIPANT DISTRIBUTION BY ELIGIBILITY STATUS, 1999 17

11.2

PARTICIPANT DISTRIBUTION BY GRADE LEVEL: 1998-99 18

11.3

PARTICIPANT DISTRIBUTION BY STATUS AND TYPE OF HOSTINSTITUTION: 1998-99 18

III.1

TEXAS TALENT SEARCH PROJECTS BY NUMBER OFPARTICIPANTS AND TARGET LOCATIONS 48

111.2

INDIANA TALENT SEARCH PROJECTS BY NUMBER OFPARTICIPANTS AND TARGET LOCATIONS 51

111.3

FLORIDA TALENT SEARCH PROJECTS BY NUMBER OFPARTICIPANTS AND TARGET LOCATIONS 56

111.4

MASSACHUSETTS TALENT SEARCH PROJECTS BY NUMBER OFPARTICIPANTS AND TARGET LOCATIONS 62

111.5

NORTH CAROLINA TALENT SEARCH PROJECTS BY NUMBER OFPARTICIPANTS AND TARGET LOCATIONS 66

111.6

OUTLINE OF THE STATE-LEVEL IMPACT REPORT 72

IV.1

OUTLINE OF THE NATIONAL IMPACT REPORT 80

V.1

DEMONSTRATION TIMELINE 87

VI. 1

LISTING OF TASKS AND TIME SCHEDULES 96

EXECUTIVE SUMMARY

This report assesses the feasibility of three studies of the Talent Search (TS)program's impact on its three primary outcomes — high school (HS) graduation,application for federal financial aid (FFA), and enrollment in a postsecondaryeducational institution (PSE). The three studies are:

n The impact of TS within each of several states.

The impact of TS nationally.

n The impact of two specific supplementary TS services.

The first two studies fall into Phase II of the Evaluation of the Talent Search program(Contract No. ED-98-CO-0073 with Mathematica Policy Research, Inc.), and the third studywould fall into Phase III of that contract.

In summary, two of the three studies are found to be feasible. To assess their feasibility,we prepared a design for each study, including a careful statement of the policy and researchquestions addressed by the study, an experimental design, a sampling plan, a data collectingplan, an analysis plan, and a reporting plan. We assessed the feasibility of each componentof the design. For example, for the state impact study, we collected information on thekindergarten through 12 th grade (K-12) database and the higher education databasemaintained by each state. If the state databases contained the needed data—covering theappropriate population and time period-- and those databases were available to researchers,we deemed this part of the design feasible.

Each feasibility assessment produced as a by-product the outline of a design for eachstudy. These designs should serve as the guidelines for conducting the studies in the future.

xvi

A. STATE IMPACT STUDY

Both the state and national impact studies are designed to focus on the impact of TS as awhole with limited analysis of the impact of specific services. They address the question,"Does TS improve the likelihood of graduation, of applying for FFA, and of its participants'enrolling in PSE, relative to the likelihood for students who did not participate in TS butmay have received similar services from other programs?"

The state impact study estimates TS's impact in each of several states. We examined theK-12 and higher education databases in Massachusetts, Indiana, Texas, Florida, and NorthCarolina. Those states were among 14 states cited by the report on state data systems by theState Higher Education Executive Officers (SHEEO, May 1999) as possibly havingdatabases sufficient to support these studies. We called the state education agency in each ofthe 14 states, identifying the five listed above as appearing to have sufficient K-12 and highereducation databases.

A review of the database documentation from the five states revealed that the databasesin Texas, Florida and North Carolina appear able to support an impact study. Massachusettsimplemented an automated statewide K-12 database just this year. Since it will not cover therelevant time period, we do not recommend conducting an impact study there. Indiana'sdatabase is missing some variables pertaining to HS performance and family income.However, we do not view these shortcomings as fatal, and propose to estimate impacts inIndiana.

We found the state impact study to be feasible, although not without some risk. The riskis that we may not be able to detect the true impact of TS because of limiting ourselves toadministrative data. We believe the risk to be of an acceptable magnitude, and thus thestudy is feasible.

B. NATIONAL IMPACT STUDY

The national impact study addresses the same research and policy questions as does thestate impact study and is intended for the same audience. The reason for conducting boththe state and the national impact studies is that they have offsetting advantages anddisadvantages. The national impact study has the limitations of having a narrow range ofoutcome variables and less statistically power than the state study but has the advantage ofbeing national in scope. The state impact study is more powerful--meaning it can detect thetrue impact of TS even if it is small--but it cannot be generalized to the nation as a whole.

Under this approach, student addresses are culled from the FFA database and linked toschool districts by geo-coding school district boundaries, which we are now doing.

Once FFA filings are linked to school districts, we will link TS projects to districts. Ourrecent survey of project directors recorded the number of TS participants by target school.

Executive Summary

xvii

We will use the Common Core of Data (CCD) to link TS target high schools to schooldistricts and will assume all other high schools have no TS presence.

The dependent variable is the proportion of graduating seniors who submit an FFAapplication. We compute the numerator of the proportion for an individual district bycounting the number of FFA applications linked to the school district. The denominator ofthe proportion is the number of HS completers, which we will extract from the CCD.

We found the national study not to be feasible in the sense that the risk of not detectingthe true impact of TS is unacceptably large. This risk stems from the fact that we are limitedto national administrative data.

C. SERVICE MIX DEMONSTRATION

The service mix demonstration has a somewhat different purpose than the state ornational impact studies do. It does not produce an estimate of the impact of TS as a whole.Instead, it estimates the impact of one TS service mix relative to another service mix. Thismeans the audience for the service mix demonstration is TS project directors. The findingsof the demonstration will provide - directors guidance on how best to design their servicepackages.

The demonstration estimates the impact of one service enhancement for programparticipants and one service enhancement for eligible nonparticipants. One can think of theservice mix demonstration as two demonstrations, one for TS participants and one foreligible nonparticipants.

We propose that the final choice of service mixes be decided in consultation with TSprojects, PES, the Office of Federal TRIO programs and the MPR study team. Forpurposes of discussion, we propose that the service enhancement be mentoring/tutoring.We combine mentoring and tutoring because collectively they represent servicesadministered to an individual, as distinct from services administered to groups of students.Providing such intensive, one-on-one services to participants with special needs or barrierspresent is the program design issue most often mentioned by TS project directors in oursurvey, and is suggested as a critical component of youth programs by researchers in a widerange of disciplines, including education, employment and training, juvenile justice, andadolescent psychology.

One of the key aspects of the mentoring/tutoring study is that both mentoring andtutoring will be defined in detail by ED's Planning and Evaluation Service (PES) and will beconsistent across projects. For the purposes of this study, mentoring/tutoring will beprovided by a single mentor/tutor to a single student, delivered over an extended period,and comprehensive, that is, it addresses in some fashion whatever barriers the youth faces,including health, substance abuse, delinquency, housing, and family problems.

Executive Summary

xviii

Typically, TS projects do not provide services to eligible nonparticipants. The eligiblenonparticipant demonstration estimates the impact of mailing to such youth a packet ofinformation on FFA and college, and following up with a phone call. The project wouldprovide this service to eligible nonparticipants in target schools. Thus, this portion of thedemonstration assesses the efficacy of projects adopting this more aggressive outreach tostudents who are eligible for TS but have not applied.

The notion of TS projects possibly providing a low level of services to eligiblenonparticipants in target schools was suggested by project directors, concerned that manyeligible youth were not benefiting from what the projects had to offer.

Unlike the state and national impact studies, the service mix demonstration does not relyon existing administrative data. It involves at least two surveys, a baseline survey and afollow-up survey. Data from these surveys will be supplemented by FFA data and schoolrecords.

A second difference between the demonstration and the state and national impactstudies is that the demonstration uses random assignment, rather than a comparison groupdesign based on propensity score matching. If possible, we will draw the control group fromthe same target schools that treatment students are drawn from. If that is not possible,either because of high TS penetration rates in target schools or because the project directorsdetermine such a procedure to be logistically infeasible, we will identify a comparison targetschool for each treatment school and draw the control students from the comparisonschool.

The service mix demonstration estimates the impact of enhanced services provided byTS projects and does not estimate the impact of TS as a whole, that is, operating TS versusnot operating TS. Unlike the state and national studies, the demonstration uses survey dataand relies on a true control group generated by random assignment. Such an experimentaldesign will give the demonstration the statistical power to detect even weak impacts of thealternative services.

We find that the demonstration is feasible, is the most statistically powerful of the threestudies, and is also the most costly of the three studies.

Executive Summary

CHAPTER I

INTRODUCTION

alent Search is one of the oldest of the federal higher education programsdesigned to increase college access for low-income youth and on a yearly basis, TSreaches out to more students than any of the TRIO-programs. Yet, until thecurrent planned evaluation, there has not been an impact assessment of Talent

Search (TS). In this document we: (1) summarize results from our feasibility studies, and (2)outline the designs for a three-pronged approach to the outcomes evaluation. The threeparts of the proposed study are:

n State-level impact studies using project, state, and federal records

n National impact study using federal databases and performance reports

n Selected site evaluations of effectiveness of alternative services focused on aninformation and mentoring/tutoring-based initiative for juniors and seniors

We describe our rationale for the proposed approaches and outline plans for implementingthe study.

A. THE CONGRESSIONAL MANDATE FOR THE STUDY

Congress has authorized and requested the U.S. Department of Education (ED) toevaluate TRIO programs, and this study is being planned in response to this congressionalmandate. The National Study of Talent Search is the first study of Talent Search sponsored by

2

the Department of Education in 20 years and as indicated above it will be the first study toconduct an outcomes or impact evaluation.

The Higher Education Act of 1998, the authorizing legislation for the TRIO programs,contains the following statement concerning evaluations:

In General—For the purpose of improving the effectiveness of the programs and projectsassisted under this chapter, the Secretary may make grants to or enter into contracts withinstitutions of higher education and other public and private institutions and organizationsto evaluate the ffectiveness ofthe programs and projects assisted under this chapter.

Practice—The evaluations described in paragraph (1) shall identify institutional,community, and program or project practices that are particularly effective in enhancing theaccess of low-income individuals and first-generation college students to postsecondaryeducation, the preparation of the individuals and students forpostsecondary education, andthe success of the individuals and students in postsecondary education. Such evaluationsshall also investigate the effectiveness of alternative and innovative methods within FederalTRIO programs of increasing access to, and retention of, students in postsecondaryeducation (H. R. 6, Sec. 402H).

Evaluations that focus on program improvement are not new to TRIO programs, butthe 1998 reauthorization added language calling for investigation of the effectiveness of"alternative and innovative methods within TRIO programs." This language has influencedthe designs we have put forth. The studies described in this report taken together address(1) the estimated impact of Talent Search in increasing college access, (2) the effectiveness ofexisting services in increasing college going, and (3) the potential impact of introducinginnovative services on college-going outcomes.

B. TALENT SEARCH PROGRAM BACKGROUND

In the 1965 legislation that first authorized the TRIO programs, Congress noted thatfinancial aid alone would not ensure equal educational opportunity for disadvantagedstudents. So Congress sponsored the development of supplemental services to preparedisadvantaged students for college and help them to succeed once there. In addition,Congress called for the development of higher education institutional policies designed toserve a more diverse population. In this regard, TRIO ' s role was to be not only to providedirect services but also to serve as a model and catalyst for the development of other similarservices at the state and local level.

Talent Search is one of the original three TRIO programs, beginning in 1967 along withUpward Bound. The third program, Student Support Services, began in 1970. EducationalOpportunity Centers (EOCs) started in 1974, and staff-training grants were initiated in 1978.The McNair Fellowship Program, which began in 1989, is the most recent direct service

I. Introduction

3

TRIO program. In addition, ED funds staff training and program innovation disseminationprograms.

1. Goals and Services

The goals of Talent Search are to:

Identify qualified youths with potential for postsecondary education, to encourage them tocomplete secondary school and to enroll in postsecondary education programs, to publicizethe availability of student financial aid, and to encourage secondary and postsecondaryschool dropouts to reenter an educational program (Department of Education AnnualReport 1997).

EOCs were begun in 1974 with goals and services similar to Talent Search but with afocus on serving an adult population including those that had left high school beforecompletion. While the descriptive component of this study includes a survey of EOCprojects, the planned impact and demonstration studies are limited to Talent Search.

In 1998 new grants were made, increasing the number of Talent Search projects from319 to 361. During the 1999-2000 program year, 361 Talent Search projects nationwideprovided services to about 300,000 participants, with total federal funding of $98.5 million.In the same period there were 83 Educational Opportunity Centers serving about 157,000participants with funding of about $30 million. Talent Search projects and EducationalOpportunity Centers are operated by two- or four-year colleges, public or private nonprofitagencies or organizations, or combinations of these sponsors. Talent Search (TS)participants must be 11 to 27 years of age' and have completed the fifth grade.

In each local TS project, two-thirds of the participants must be low-income studentswho are potentially first-generation college students. Unlike the other TRIO programs, theother one-third, of participants need not meet the low-income or first-generation criteria.

TS services include:

n Academic and personal counseling

n Career exploration and aptitude assessment

n Assistance with re-entry to high school and college

n Information on postsecondary education

p Projects may serve clients aged 28 or older if no Educational Opportunity Center is available to servethem and doing so will not dilute the services they provide to the main target group.

I. Introduction

4

n Information on financial assistance

n Assistance in completing college and financial aid applications, and inpreparing for admissions tests

n Exposure to a range of career options

n Tutorial services

n Cultural events

The major Department of Education Performance Indicator goal for TS is to increasethe "ratio of TS participants who apply to college or apply for financial aid compared tonumber served." In recent years, two major initiatives have been undertaken to reshapeTalent Search. First, there has been an increased emphasis on project accountability asmanifest by new performance reporting requirements, while allowing projects greaterflexibility in deciding how to deliver services. Second, TS proejcts have been encouraged byED to place more emphasis on increasing the participation of younger students in the"college pipeline."

2. TS in the Context of Other TRIO Programs

In preparing a design for this study, it is useful to look at TS in the context of the otherTRIO programs because TRIO programs are designed to work together. TS and EOCs arethe least intensive of the TRIO programs and serve the most participants. Talent Search wasdesigned to be an outreach program that cast a wide net to complement other TRIO andnon-TRIO services. The central features of TS are the emphasis on school and communityoutreach and the heavy use of personal, academic, and financial aid counseling. As shown byED summary statistics, TS serves more students a year than any other TRIO program. In1999, the average TS project was funded to serve 896 students, and the average EOC projectserved 2,117 people (Table 1.1). TS funding averages $304 per student. Within TS, thetypical staff-to-student ratio is about 1 to 183 (COE 1993). As such, the outreach quality ofthe program both challenges the designers of an impact evaluation and presents anopportunity for creative approaches.

I. Introduction

5

Table 1.1. TRIO Funding, Number of Grants, and Average Award, Amount Per Person Served andNumber Funded to Serve 1999

1999Number of

Grant amount

grantsNumberserved

Averageaward

Amount perperson served

Averagenumber funded

to serve perproject

Talent Search $98,450,697 361 323,541 $272,717 $304 896EducationalOpportunity Centers $29,794,380 82 158,063 $363,346 $188 1928McNair $32,114,068 156 3,734 $205,859 $8,600 24Student SupportServices $178,916,836 796 178,099 $224,770 $1,005 224Upward Bound $220,500,637 772 52,960 $285,623 $4,164 69Upward Bound MathScience $29,276,284 124 6,200 $236,099 $4,722 50

The funding history of TRIO programs (Table 1.2) shows that while TS started from thesmallest base compared with the other programs, funding has increased at a rate as high orhigher than the other TRIO programs. In constant dollars, TS funding increased more thanfour-fold between 1970 and 1999.._ The periods of increased funding for TS are similar tothose of other TRIO programs. However, the increases around 1991, when serving youngerstudents in the seventh and eighth grades became a priority, are especially apparent. TS andEOC programs doubled funding in constant dollars in 1991. In 1993, TS regulations werefurther revised to allow students who had completed at least the fifth grade to be eligible forparticipation. While TS and EOC do not offer intensive services, practitioners haveobserved that these smaller interventions can be critical to a disadvantaged student.

Table 1.2: TRIO Funding Levels in Constant 1999 DollarsYear

19671970197519801985199019951999

TS$12.43$21.5$18.6$30.9$32.1$34.5$85.7$98.5

EOC

$9.3$15.6$14.2$15.2$26.9$29.8

UB$139.7$127.1$118.6$126.4$114.0$128.2$208.3$220.5

SSS

$42.9$71.2

$121.3$108.5$115.9$156.9$178.9

McNair

$3.8$20.9$32.1

Training

$4.0$2.0$2.0$2.2$4.5

UB Math

$29.1SOURCE: Calculated from information in U. S. Department of Education Annual Report,1997; CPI 1997 US Statistical Abstract, Table 752.

I. Introduction

6

C. CONTEXT FOR THE STUDY: CHANGES SINCE THE 1960S IN NATIONAL STATISTICS

RELATED TO EDUCATION ENROLLMENT

Since the implementation of the TRIO programs in the 1960s, a number of social anddemographic changes have occurred in the nation, and these will constitute the largercontext in which the TS impact evaluation is conducted. Viewed from a broad societalperspective, the ultimate goal of TRIO and TS is to reduce the gap between low-income andhigh-income students in high school graduation, college enrollment and graduation rates.Appendix A summarizes national statistics on high school completion and college entrancesince the initiation of the TRIO programs. These data show that the rate of collegecontinuation of high school graduates has increased substantially among all socioeconomicstatus (SES) groups since the late 1960s, from about 45 percent to a record high of 66percent by the end of the in 1990s (Postsecondary Educational Opportunity June 1999).

While the percentage increase in college-going rates among low-income youth has beenlarger than among high-income youth, the gap between the rates of college attendance forlow-income and high-income youth remains large. Data from NELS88 show a gap of about42 percentage points between low and high SES youth in college enrollment within twoyears of high school graduation. Using NELLS88 data, the reported enrollment rates were49 percent for low SES youth and .91 percent for high SES youth in 1994. Data from theHigh School and Beyond Study in the 1970s reported rates of 42 percent for low SES youthand 85 percent for high SES youth.

Census Bureau data from 1998 indicate that among high school graduates, about 86 percentof high-income youth and about 56 percent of low-income youth attended college withintwo years. Differences were larger if one considers the total age cohorts and the chances ofcollege completion by age 24. Estimates are that about 5 percent of those in the lowesteconomic quartile will receive a bachelor 's degree by age 24, while about 50 percent of thosein the highest quartile will receive a bachelor ' s degree by age 24 (Postsecondary EducationOpportunity December 1998). Similarly the type of college one attends is highly correlatedwith income. There is also evidence that the type of college attended has become morestratified by income over time (McPherson and Schipiro 1998).

At a time when less than 10 percent of low-income students can expect a bachelor'sdegree, surveys of college aspirations repeatedly find that low-income students report highaspirations for college. A recent survey of all 11 th graders in Indiana found that more thantwo-thirds expected to obtain an bachelor ' s degree and only 5 percent had no expectationsto obtain any postsecondary education (Indiana Guidance Report for 11 th Grade Survey1998). At the same time, analyses have shown that low-income students are much less likelyto take college preparatory courses and that these courses are highly correlated with enteringand succeeding in college (National Center for Education Statistics 1997; Adleman 1999).

Significant shifts have, however, occurred in the college continuation rates of women,and in the increased college participation of racial and ethnic minorities. Collegecontinuation rates for women are now at about 70 percent compared with 61 percent for

I. Introduction

7

men. Racial and ethnic minorities taken together now make up about 30 percent ofundergraduate enrollments compared with about 17 percent in the 1970s (HEGIS 1974;IPEDS 1996). These trends mean that the demographic composition of college enrolleeshas changed significantly since the initiation of the TRIO programs.

An equally important change in the last 20 years is the rapid introduction of personalcomputers and Internet communication, that are significantly changing the nation'seducational system and are affecting how TS and EOCs provide services. The number ofcomputers in the nations' public schools went from 125 students per computer in 1983 to 9students per computer in 1996 and is dropping rapidly (QED, Glennan and Melmed 1996).The proposed TS evaluation will be conducted in the context of these trends and changes.

Because there has not been a national impact evaluation of the TS program, we do notknow what role the program has played in these statistics. Common sense and the testimonyof student participants tell us that it has played a positive role—but they also tell us that theprogram is acting within a system in which there are other similar programs and otherfactors that exert both positive and negative influences on college-going rates. It may be thatthe impact of TS and other, similar programs has been to help keep the gap between thehigh- and low-income rates of college attendance within a certain range. It may also be thatthe role played by TS in the previous 35 years needs to be reexamined to ensure that it is themodel best suited for the future.

D: PREVIOUS STUDIES OF TALENT SEARCH

Table 1.3 lists the major studies conducted of Talent Search over its history. The plannedstudy is unique in that it would be the first national study of TS that includes the possibilityof an impact study. Previous national studies of TS have been descriptive studies, or theyhave been combined with studies of other programs, such as the evaluation of UpwardBound and Talent Search conducted in the 1970s (Burkheimer, Riccobona, Wisenbaker1976) and the College Board study of the mid-1980s (Franklin 1985).

In 1992, ED commissioned six design papers looking at the potential for an impactstudy of Talent Search. A summary of these papers is included in Appendix B. These studiespointed out the difficulty in implementing a random assignment study for Talent Search, butnoted the potential for a comparison group design focused on short-term impacts. In 1994,ED contracted with Decision Information Resources (DIR) to conduct a small descriptivestudy of TS to look at feasible measures of program performance criteria (Jackson, Arbona,and Branch 1994) and to prepare a review of target population needs and effectiveinterventions (Arbona and Jackson 1994). This study concluded that it would be possiblefor projects to keep records of participants ' services and do limited tracking of outcomes. Inaddition, the National Council of Educational Opportunity Association (NCEOA)sponsored a literature review and a survey of Talent Search and Upward Bound in the early1990s (NCEOA 1993; Nettles and GetzFeld 1992). Results of these studies are summarizedin the literature review document prepared for the national study (Silva and Kim 1998).

I. Introduction

8

Table 1.3: Studies of Talent Search and EOC

Author(s)And Date Brief Description of Study

Pyecha et al. 1975

This was the first national evaluation of Talent Search, conducted for the U.S.Office of Education by Research Triangle Institute. It included a survey of all114 project directors (response rate 92 percent), a survey of the postsecondaryinstitutions on the enrollment status of a sample of former program participants(response rate 93 percent), and case studies of 20 projects.

Franklin 1985

Primarily focused on 11 purposively selected Talent Search projects, using amail survey, telephone interviews, and document review. Also drew on alimited amount of national level data--Annual Performance Reports--from EDfor 1979-83 (percentages obtained unreported). Conducted for the CollegeEntrance Examination Board.

Coles 1992 In-depth interviews with the directors of 19 purposively selected Talent Searchprojects. Prepared for the Design Conference for the Evaluation of the TalentSearch Program, hosted by the U.S. Department of Education, Office of Policyand Planning, September 30, 1992.

Eisner 1992

A review of data from Annual Performance Reports for 1986-87 and 1990-91.Data obtained for an estimated two-thirds of the Talent Search funded in 1986-87 and for 92 percent of the 177 projects funded in 1990-91. Included in thereport from the Talent Search Design Conference.

Lee and Clery 1993

A mail survey of all 294 Talent Search projects operating in 1992 (responserate 72 percent). Conducted for the National Council of EducationalOpportunity Associations.

Decision Information Case studies of seven purposively chosen Talent Search projects, focusing onResources (DIR)

current and potential program performance measures. Conducted for the U.S.1994 Department of Education, Office of Planning and Evaluation.

E. STUDY OBJECTIVES, COMPONENTS AND RESEARCH QUESTIONS

As stated in the statement of work for the project, the study has the following overallobjectives:

n To contribute to strategic thinking of how best to improve high schoolgraduation rates and access to college for disadvantaged students

n To understand what role federal policy can play in improving and promotingeducational opportunity for disadvantaged students

I. Introduction

9

n To provide updated information on the operations and services that TS andEOC provide.

1. Components of the Study

To achieve these objectives the overall study involves three major elements:

1) A descriptive study of TS and EOC utilizing information from:

- Descriptive surveys of all TS and EOC projects that are ongoing andexpected to be completed in September of 2000

- Student-centered case studies of 14 sites done in the period of Spring1999 to March 2000

Analysis of the most recent performance reports currently being finalized

Analysis of the characteristics of Target Schools from matching withCommon Core of Data (in process)

Interviews with former_ project directors from sites that are no longerfunded

2) A study to look at the feasibility of conducting an impact assessment for TSutilizing information from the above noted components (for which thisdocument is the report)

3) If feasible, an impact study of the program.

The study is cumulative in that the descriptive phase informs the impact study phase.Both phases will build on the findings of related work on serving disadvantaged students.The focus of the impact study will be the implementation of an impact design to observeshort-term impacts.

2. Study Questions

The study's design would address the following questions, which are derived from thecongressional mandate and from our own assessment of what might be the best use of theevaluation resources set aside for the program:

n What role does the Talent Search Program now play in increasing collegeaccess and success of low-income youth?

n How can the program be improved? How effective are selected alternativeservices in increasing college access? What is the most effective timing andmix of services?

I. Introduction

10

n How do Talent Search projects act in consort with other college accessinitiatives and programs?

F. THE CONCEPTUAL FRAMEWORK FOR THE STUDY

A key component of the study's framework is recognizing that Talent Search operateswithin a system and that the impact study must look at the role of TS within that system.Figure 1.1 graphically locates Talent Search within the educational systems in which it

operates. This figure lists other services and activities that may also affect a student's collegedecision. TS projects are designed to operate in consort with these services. The figure alsolists obstacles that TS and other programs face in attempting to foster increasedpostsecondary attendance. Figure 1.2 portrays the pathways through which Talent Searchinfluences a student's pursuit of postsecondary education, including family factors, regularschool guidance, other precollege programs, peers, community organizations, andpostsecondary institutions themselves.

This conceptual framework relates to selected metaphors from a systems theoryapproach. These include viewing TS programs as part of an interacting system withrecognition that:

n Relatively small changes in conditions may result in significant differences inoutcome.

The timing of service may be an important determinant of outcome.

n The targeting of services to specific groups (for example those who areundecided in later years in high school as to whether they wish to attendcollege or not) may be a way to increase the impact of TS.

n The matching of services to unmet needs may be a way to increase theimpact of Talent Search.

n There is a need to address the interaction of TS with other pre-collegeservices and to assess the role that TS has played in fosteringinstitutionalization of services within the schools served.

I. Introduction

11

Figure 1.1: Educational Talent Search Within the Educational System

PostsecondaryEducation System

Related Services

TRIO (UB, SSS)

4Other pre-college

services

Educatonal Talent Search Program

Academic skillsDecision making skills

Motivation developmenlResources information

4-YearCollege

11th and 12th Grades2-YearCollege 4—

LocalSchoolSystem

Preschool

ElementarySchool

Middle School

Academic skillsCollege awareness

MentoringTutoring

Counseling

Early High School

Transition to HSCourse selectionDecision makingAcademic skills

SAT/ACT prepApplications for financial

aidCareer goals

College matchingCounseling

Course selection

Other Options

rr

Obstacles

Low self esteemNegative school experiencesTracking outDrugs/alcoholFamily factorsPregnancyNeed to work

Lack of skillsLack of motivationLack of resourcesPoor mental healthPoor physical healthBehavior problemsLack of informationOther competing options

EOC Program

Academic skillsStudy skills

Decision making skillsReturn counseling

Motivation developmentResources

InformationGED

I. Introduction

12

Figure 1.2: Potential Pathways for Talent Search to Influence Students'Pursuit of Postsecondary Education

SHORT-TERM STUDENT OUTCOMES

Educational Expectations

Attitudes Toward School

Knowledge of Prerequisite Coursesand Skills

Identification of College Optionsand Career Interests

Peer Group Factors -

Enrollment in Academic Program,Receipt of Academic Credits/H.S.Degree

Receipt of Special Services toImprove Skills, (Tutoring, APClass, SAT/ACT Prep.)

Awareness of Financial AidOptions and Procedures

Application for Financial Aid

Community Groups/Organizations

Application for College Admission,Taking College Admission Tests

Acceptance by College/Receiptof Financial Aid

Family Factors

Regular SchoolGuidance and

Assistance

Other PrecollegeServices

Students and Out-of-School Youth

I. Introduction

13

We discuss these points in more detail in Chapter II when we describe the conclusionsof the feasibility studies.

G. STRUCTURE OF THE REMAINDER OF THE REPORT

Chapter II continues the introductory material by summarizing results from thefeasibility studies that have been used to inform the designs presented in Chapters III to V.Chapters III and IV present designs for a Phase II of the study calling for an impact studythat utilizes existing records data. Chapter III presents plans for a Phase II study in fourselected states. Chapter IV discusses a plan for an exploratory national impact study usingfederal databases and existing national data from performance reports and the projectsurvey. This is referred to as the National Study component of Phase II

Chapter V presents a plan for a Phase III evaluation of the impact of alternative servicesdesigned to increase the targeting outreach of TS and the intensity of services for a randomlyassigned groups of eligible students.

I. Introduction

CHAPTER II

STUDIES USED TO INFORM THEFEASIBILITY REPORT

To inform the feasibility assessment and the designs presented in Chapters III to V,MPR collected and utilized information on Talent Search from a number ofsources. In this chapter we summarize this information and indicate how theseinvestigations led us to the proposed research designs. The sources for this

chapter are:

n Analysis of Performance Reports for 1999 and analysis of Talent Searchtarget schools

n Consultation with Talent Search project directors and Project DirectorSurvey results

n Case studies of 14 projects

n Exploratory matching of TRIO participant samples from the NationalEvaluations of Student Support Services and National Evaluation of UpwardBound with national federal aid files

n Exploration of state-level transition databases that track the progress ofstudents from high school to postsecondary education.

A. PERFORMANCE REPORTS AND TARGET SCHOOL ANALYSIS

In 1999 Talent Search Projects completed a newly revised performance report form.These reports provide information on Talent Search participants, services, objectives andself-reported achievement of outcomes. In addition, projects submitted names of each

16

Target School served, and Mathematica Policy Research (MPR) matched this informationwith the U.S. Department of Education (ED) Common Core of Data (CCD). Thismatching enabled us to look at the characteristics of schools served and allowed us toestimate the proportion of eligible students being served. A full report has been prepared byMPR (Kim and Cahalan 2000) and is under ED review. Here, we note some highlights thathave helped inform the feasibility study.

1. Project and Participant Demographics

In 1999, there were 361 Talent Search projects. Forty-four percent were hosted by4-year-and-above degree-granting postsecondary institution and 34 percent were located at2-year degree-granting postsecondary institutions. The remaining 19 percent were hosted bycommunity organizations not affiliated with a college or university (Table 11.1). Together theprojects served 307,451 participants in 1999.

Table 11.1: Number Of Talent Search Projects and Number ofParticipants By Type Of Host Institution: 1998-99

SectorTotal projects

in 1998

Percentage oftotal TSprojects

Projectresponse

rate

Number ofparticipants

reportedPercent

distributionPublic 4-year 122 34% 98% 104,372 34%Private 4-year 48 13% 96% 36,480 12%2-year 124 34% 97% 90,217 29%Communityorganizations 68 19% 98% 76,382 25%All projects 362 100% 98% 307,451 100%

from United States Department of Education, Office of Federal TRIO Programs, Talent

1998-99 and Integrated Postsecondary Education System, Institutional

Eligibility Status. As discussed in Chapter I, Talent Search eligibility regulationsrequire that that two-thirds of participants be low-income and potentially first-generation-college or disabled. Unlike the other TRIO programs that require that the other one-thirdbe either low-income or first generation, Talent Search does not have specific requirementsfor the last one-third of participants. Nevertheless, examination of the performance reportsindicates that overall 95 percent of Talent Search participants were low-income or potentiallyfirst-generation college students. In 1999, 74 percent of participants were both low incomeand first generation. Seven percent of TS students were low-income only and 14 percentwere first-generation only. Only 5 percent met neither of the criteria (Figure 11.1).

Source: Calculated

Search Performance Reports,

Characteristics File, 1998.

II. Studies Used to Inform the Feasibility Report

17

Figure 11.1. Participant Distribution by Eligibility Status, 1999

Firstgeneration

only

Other14%

5%

Low-income& first

generationcollege

74%Source: Calculated from United States Department of Education, Office of Federal TRIO Programs, Talent

Search Performance Reports, 1998-99.

Sex and Ethnicity. As with the other TRIO programs, more females participated thanmales. In 1999, 60 percent of those served were female. Just over one-third (36 percent)were black or African American, 32 percent were white, 22 percent Hispanic or Latino, 4percent Asian, 4 percent American Indian or Alaska Native and 1 percent Native Hawaiianor Other Pacific Islander.

Grade Level of Participants. In the last decade, ED has encouraged Talent Searchprojects to extend services to middle schools, and in 1999 about one-third of participantswere from grades 6 to 8. The majority of the participants, however, continue to be in highschool. In 1999 two-thirds of participants were in senior high and almost one-fourth were inthe 12`h grade (22 percent). About 5 percent were high school or postsecondary dropouts orhigh school graduates (Figure 11.2).


18

Figure 11.2: Participant Distribution by Grade Level: 1998-99

Postsecondarydropout

High school (orGED) graduate

Secondary schooldropout

12th grade only

9th-11th grade

6th-8th grade

0%

10%

20%

30%

40%

50%

Source: Calculated from United States Department of Education, Office of Federal TRIO Programs, Talent

Search Performance Reports, 1998-99

Participant Status. Just under half of the participants in 1999 were new participants (48percent) and just over half (52 percent) were continuing participants, meaning they had beenin the program at some time prior to the reporting year (Figure 11.3).

Figure 11.3: Participant Distribution By Status and Type of Host Institution:1998-99

Public 4- Private 4-

2-year Community All projectsyear

year

orgs.

Source: Calculated from United States Department of Education, Office of Federal TRIO Programs, Talent

Search Performance Reports, 1998-99

100%

80%

60%

40%

20%

0%


19

2. Target Schools and Percentage of Eligible Served

Using information from the Performance Reports, we were able to determine whichschools nationally were served by Talent Search (see Table 11.2). In 1999, Talent Searchserved about 5,094 high schools across the nation—an average of 14 per project. Thisnumber represents about 13 percent of the schools in the nation serving the target agepopulation of students (defined here as any school serving any students in the 7th grade orhigher.)

Table 11.2: Number of Talent Search Target Schools and Estimated Number ofSchools Serving Target Age Group Within the State

StateNumber of Talent

Search target schools

Estimated number ofmiddle and senior highschools in the state*

Estimated percentage of middleand senior high schools served by

Talent Search*Alabama 286 716 40%Alaska 12 338 4%Arizona 60 692 9%Arkansas 154 495 31%California 306 3,536 9%Colorado 102 621 16%Connecticut 28 427 7%Delaware 29 86 34%District of Columbia** 149** 53** 281%**Florida 123 1,294 10%Georgia 161 734 22%Hawaii 38 87 44%Idaho 71 301 24%Illinois 196 2,289 9%Indiana 124 725 17%Iowa 183 691 26%Kansas 83 715 12%Kentucky 165 630 26%Louisiana 176 703 25%Maine 20 338 6%Maryland 56 461 12%Massachusetts 65 687 9%Michigan 91 1,466 6%Minnesota 34 944 4%Mississippi 77 440 18%Missouri 38 959 4%Montana 27 491 5%Nebraska 28 632 4%Nevada 16 174 9%New Hampshire 20 211 9%New Jersey 60 946 6%New Mexico 84 295 28%New York 150 1,721 9%North Carolina 205 910 23%North Dakota 38 283 13%Ohio 134 1,613 8%Oklahoma 185 1,083 17%Oregon 38 560 7%


20

StateNumber of Talent

Search target schools

Estimated number ofmiddle and senior highschools in the state*

Estimated percentage of middleand senior high schools served by

Talent Search*Pennsylvania 147 1,200 12%Rhode Island 12 93 13%South Carolina 125 455 27%South Dakota 20 505 4%Tennessee 140 783 18%Texas 283 3,312 9%Utah 69 295 23%Vermont 20 155 13%Virginia 141 705 20%Washington 36 927 4%West Virginia 96 324 30%Wisconsin 45 970 5%Wyoming 20 177 11%

Outlying Areas 128 569 22%Total 5094 39,817 13%*All public schools that have any enrollment at the 7th grade or higher were included in this count derivedfrom the Common Core of Data for 1998. The count includes K-8 and K-12 schools. Nationwide there wereabout 20,000 secondary schools, 10,000 middle schools, 51,000 elementary schools about 5,000 combined andother schools in the nation (Digest of Education Statistics, 1998).

**Figures for the District of Columbia reflect that a program administered in DC has target schools across thenation

Percentage of Low-Income Students Served. Matching the information on TargetSchools from the performance reports with the CCD allowed us to estimate the percentageof low-income students (defined as eligible for free lunch) served by Talent Search withinthe Talent Search Target Schools. We also estimated the percentage of low-incomestudents served by Talent Search in all schools serving the target age groups within the stateand the nation (Table 11.3).

Table 11.2: Number of Talent Search Target Schools and Estimated Number ofSchools Serving Target Age Group Within the State


21

Table 11.3: Number Of Middle And High School Students Served By Talent Search, Number EligibleFor The Free LunchProgram In Grades 6-12 In Target Schools And All Secondary Schools, AndNumber Served By Talent Search As A Percentage Of The Number Eligible For The Free LunchProgram In The Target Schools And All Secondary Schools, By State: 1998-99

State

Numberserved by

Talent Searchin grades 6through 12

Numbereligible forfree lunchprogram in

target schoolsin grades 6-12

Numbereligible forfree lunch

program in allsecondaryschools in

grades 6-12

Numberserved as

percentage ofnumber

eligible forfree lunchprogram in

target schools

Numberserved as

percentage ofnumber

eligible forfree lunch

program in allsecondary

schoolsAlabama 18,944 58,151 65,859 32.6% 28.8%Alaska 797 na na na naArizona 3,083 na na na naArkansas 7,442 16,928 46,343 44.0% 16.1%California 24,564 203,604 717,546 12.1% 3.4%Colorado 4,978 21,180 40,470 23.5% 12.3%Connecticut 1,475 15,930 29,744 9.3% 5.0%Delaware 1,348 6,577 6,243 20.5% 21.6%District of Columbia 14,662 na na na naFlorida 5,354 57,165 274,862 9.4% 1.9%Georgia 9,048 60,083 142,930 15.1% 6.3%Hawaii 1,699 10,078 13,779 16.9% 12.3%Idaho 2,511 7,703 15,201 32.6% 16.5%Illinois 10,796 na na na naIndiana 4,876 18,019 64,235 27.1% 7.6%Iowa 6,581 13,049 28,592 50.4% 23.0%Kansas 4,446 na na na naKentucky 6,470 31,191 67,207 20.7% 9.6%Louisiana 10,075 53,385 96,085 18.9% 10.5%Maine 739 1,798 18,409 41.1% 4.0%Maryland 2,882 18,501 60,078 15.6% 4.8%Massachusetts 5,498 18,089 38,244 30.4% 14.4%Michigan 3,922 21,599 131,995 18.2% 3.0%Minnesota 3,012 10,460 58,198 28.8% 5.2%Mississippi 5,220 23,193 101,911 22.5% 5.1 0/0Missouri 2,193 10,828 79,855 20.3% 2.7%Montana 1,754 4,524 9,915 38.8% 17.7%Nebraska 1,561 8,901 15,115 17.5% 10.3%Nevada 1,074 3,046 17,349 35.3% 6.2%New Hampshire 1,217 1,366 8,125 89.1% 15.0%New Jersey 5,714 17,758 85,228 32.2% 6.7%New Mexico 4,366 na na na naNew York 14,606 56,635 337,041 25.8% 4.3%North Carolina 8,276 42,037 107,214 19.7% 7.7%North Dakota 2,344 2,184 8,436 107.3% 27.8%Ohio 8,240 33,302 119,778 24.7% 6.9%Oklahoma 7,033 30,067 62,400 23.4% 11.3%


22

Table 11.3: Number Of Middle And High School Students Served By Talent Search, Number EligibleFor The Free Lunch Program In Grades 6-12 In Target Schools And All Secondary Schools, AndNumber Served By Talent Search As A Percentage Of The Number Eligible For The Free LunchProgram In The Target Schools And All Secondary Schools, By State: 1998-99

State

Numberserved by

Talent Searchin grades 6through 12

Numbereligible forfree lunchprogram in

target schoolsin grades 6-12

Numbereligible forfree lunch

program in allsecondaryschools in

grades 6-12

Numberserved as

percentage ofnumber

eligible forfree lunchprogram in

target schools

Numberserved as

percentage ofnumber

eligible forfree lunch

program in allsecondary

schoolsOregon 2,017 8,513 45,948 23.7% 4.4%Pennsylvania 10,249 na na na naRhode Island 865 6,393 10,324 13.5% 8.4%South Carolina 6,879 28,378 78,604 24.2% 8.8%South Dakota 827 1,353 12,666 61.1% 6.5%Tennessee 7,974 na na na naTexas 14,010 125,809 453,934 11.1% 3.1%Utah 4,958 6,746 27,668 73.5% 17.9%Virginia 1,300 2,168 5,583 60.0% 23.3%Vermont 6,747 17,709 83,255 38.1% 8.1 %C

Washington 1,637 na na na naWisconsin 2,325 15,079 33,634 15.4% 6.9%West Virginia 3,296 16,188 68,097 20.4% 4.8%Wyoming 622 1,026 6,519 60.6% 9.5%Outlying areas 10,688 40,652 161,116 26.3% 6.6%Overall 293,194 1,147,343 3,855,734 25.6% 7.6%

Source: United States Department of Education, Office of Federal TRIO Programs, Talent SearchReports, 1998-99; National Center for Education Statistics, Elementary/Secondary and Libraries StudiesCommon Core of Data, 1998-99.

These statistics indicate that Talent Search serves, on average, about 26 percent of thenumber reported eligible for free lunch in the Target Schools. Looking at all age-appropriateschools, the number served by Talent Search in a given year represents about 8 percent ofthe number eligible for free lunch in the nation. Within states, estimates of the percentageof eligible served in the Target Schools range from 9 percent in Connecticut and Florida tomore than 100 percent in North Dakota. For the potentially eligible population in all ageappropriate schools, the estimates range from 2 percent in Florida to 29 percent in Alabama.

The estimates based on free lunch numbers are similar to those obtained by looking atthe percentage of children in the age groups 11 to 17 at 150 percent of the poverty level whoare served by Talent Search in a given year. There were an estimated 8.2 million childrenaged 11 to 17 living in households at or below 150 percent of poverty-about 28 percent ofthe total children in these age groups in 1997 (estimated from Povery in the United States: 1997,Bureau of Census 1998). The number served by Talent Search in a given year is about 3.6percent of the total number estimated in 150 percent of poverty in a given year (Table 11.4).If one assumed that a there was no overlap in those served from year to year and each child

PerformanceDivision,


23

could be served for only one year, the percentage who Talent Search might serve in somegrade between age 11 and 17 would be about 25 percent (the maximum percentage served).

This information is useful for the feasibility study because it tells us that there is apotential population of eligible and non-served students from which comparison groupsmight be drawn in both the Target Schools and the states.

Table 11.4: Estimates of the Percentage of Income-Eligible Children Served byTalent Search in the United States Using Census Data on PovertyEstimates U.S.Estimated number of children aged 11 to 17 in thousands 29,387Estimated number of children aged 11-17 at 150% of poverty in thousands 8,228Percent of children at 150% of poverty 28.0%Percent in free or reduced lunch in nation 34.4%Number of TS participants in year 302,034Estimated percent of 11 to 17 year old income eligible served each year (assuming 3.6%same students served each year)Estimated percent of 11 to 17 year old income-eligible students served each year 25.2%(assuming different students served each year)

3. Services Provided by Talent Search

The performance reports also provide input concerning key feasibility questions relatedto type and intensity of project services. Table 11.5 shows the percentage of projectsoffering services listed on the performance report, and Table 11.6 shows the percentage ofparticipants who were reported as having received the service in 1999. For most of theservices listed on the form, projects were also asked to report the number of sessions given.Group sessions were often counted as one session even though several participants attended.For this reason a calculation of the number of sessions per participant underestimates thenumber of grouped sessions that might have been received. Table II. 7 compares the resultsfor service categories present on both the old performance form and the new form. Thistable enables some observation of service trends in Talent Search over the last decade.Because the forms were different in the two periods, it is not possible to conclude that theservices newly listed on the 1999 form are necessarily new services. Keeping in mind theselimitations, we observe the following relevant to our feasibility study:

n Since 1990 there appears to have been an increase in the percentage ofstudents receiving academically oriented services. For example 20 percentreported receiving tutoring in 1999 compared with 8 percent in 1990. (Table11.7)

n Mentoring was not listed on the form in 1990. Among the services listed onthe form in 1999, mentoring was the least frequently offered service (withabout 58 percent of projects having anyone reported as receiving this


24

service). About 17 percent of participants were reported as receiving theservice. (Table 11.5 and Table 11.6).

n Counseling was the dominant service in 1990 and remains the mostfrequently offered and received service. (Table 11.5 and Table 11.7)

Table 11.5: Percentages of Projects Offering Service to Participants, byGrade Level: 1998-99Service Middle School High School Adult OverallTutoring 72% 65% 13% 80%Assisted (computer) labs 48% 52% 12% 63%Test-taking & study skills development 84% 88% 24% 92%Counseling 86% 90% 43% 92%Academic advising/course selection 80% 90% 39% 91Mentoring 50% 50% 9% 58%Cultural activities 87% 80% 16% 90%College orientation activities 81% 93% 36% 95%Family activities 76% 79% 20% 84%Referrals 39% 59% 32% 68%Source: Calculated from the United States Department of Education, Office of Federal TRIO Programs,Talent Search Performance Reports, 1998-99.

Table 11.6: Percentages of Participants Receiving Services, and Average Number of

Average number ofPer Project: 1998-99Sessions

Percentage of participantsService

receiving service sessions per projectTutoring 20% 259Assisted (computer) labs 13% 94Test-taking & study skills development 47% 133Academic advising/course selection 64% 379Counseling 76% 662Mentoring 17% 127Cultural activities 33% 34College orientation activities 54% 129Family activities 30% 54Referrals 12% 18

Source:

United States Department

of Education, Office of Federal TRIO Programs, Talent SearchPerformance Reports, 1998-99.


25

Table 11.7: Percentages of Participants Receiving Each Type of Service,Comparing Performance Report Data From 1990-91 and 1998-99Service 1990-91 1998-99Tutoring 8% 20%Study Skills 5% 47%Counseling 78% 76%Cultural Activities 9% 33%College Orientation 19% 54%

Source: United States Department of Education, Office of Federal TRIO Programs, Talent SearchPerformance Reports, 1998-99.

4. Objectives and Outcomes

The most recent performance reports also contain information on Talent Searchobjectives and outcomes. In grant applications, each project specifies outcome objectives in

listed areas and enters into a partnership agreements with ED. Table II. 8, and Table II. 9show the objectives, the average targets set by projects, and the calculated percentagemeeting outcomes.

Table 11.8: Comparison of Approved Objectives and Actual Achievements forMiddle School and High School Outcomes: 1998-99Outcome Approved Objective ActualRetentionMiddle School 89% 100%High School 89% 98%

GraduationMiddle School *High School 88% 96%

Re-entryMiddle School 64% **High School 64% 85%

Source: Calculated from information in the United States Department of Education, Office of Federal TRIOPrograms, Talent Search Performance Reports, 1998-99.

Not applicable ,for middle school participants. Although promotion from middle to high school was notincluded among the approved objectives, projects were asked to provide the number of middle school studentswho went on to high school. Projects reported 37,027 participants, or 40 percent of all middle-school-agedparticipants, were promoted from middle to high school. Performance report data on the number of middle-school students eligible to be promoted were not available, so we were not able to determine the percentage ofeligible students promoted.

**The performance report did not ask projects to provide the number of participants who were middle schooldropouts at the beginning of the reporting period. As a result, we were not able to determine the percentage ofmiddle school dropouts that re-entered middle school. Projects reported that 1,385 participants re-enteredmiddle school during the 1998-99 program year.


26

As shown in Table 11.8, Talent Search projects set high school retention goals of 89percent and these goals were exceeded with 98 percent of TS participants reported retainedin high school. Projects targeted assisting 90 percent of participants with postsecondaryadmissions and federal aid applications. They reported achieving this objective for 85percent of applicable participants (those in 12 th grade). Projects targeted a goal of 75 percentenrollment in postsecondary and reported that 72 percent of applicable participants hadenrolled (Table 11.9).

Table 11.9: Comparison of Approved Objectives and Actual Achievements forPostsecondary Outcomes: 1998-99Outcome Approved Objective ActualPre-collegeAssistance in applying for post-secondary admissions 90% 85%Assistance in applying for student financial aid 90% 84%

Postsecondary OutcomePostsecondary admissions 75% 72%Postsecondary re-entry 65% 100%

Figure II. 3 showed that a small percentage of participants in Talent Search, (about 2percent) in a given year were classified as high school "dropouts." For this group, TalentSearch projects averaged a target goal of 64 percent school reentry (Table 11.8). On average,they reported that 85 percent of dropouts served had re-entered high school.

Type of Institution Attended. Overall, about 52 percent of participants who enrolledin postsecondary schools were reported as having enrolled in a 4-year institution, 41 percentin a 2-year institution, and 6 percent in some "other postsecondary." As shown in Table II.10, the type of host institution has a relationship to type of institution attended. Participantswere somewhat more likely to attend institutions of a similar type as the host project. Forexample, among projects hosted by 2-year institutions, 55 percent of the participants whoenrolled were reported enrolled at a 2-year college, while among those hosted at a 4-yearprivate institution about 28 percent of those who enrolled attended a 2-year college. Overall41 percent enrolled in a 2-year institution, about the same as the national average.


27

Table 11.10: Distribution of Postsecondary Placement ofParticipants by Sector: 1998-99

Sector

Percentageadmitted to

public 4-year


private 4-year


2-year*


other types**Public 4-year 48% 9% 35% 8%Private 4-year 44% 20% 28% 9%2-year 31% 8% 55% 6%Community organizations 41% 12% 41% 5%All projects 41% 11% 41% 6%

Source: Calculated from information in the United States Department of Education, Office of Federal TRIO

Programs, Talent Search Performance Reports, 1998-99.

B. PROJECT FEEDBACK ON THE STUDY

Direct feedback from projects on their interests with regard to the study was a secondmajor source of input into the impact study feasibility design. We took several steps toobtain input from the projects:

n We held discussions on the study with project directors at the nationalmeetings held by Council on Opportunity in Education (COE) in Septemberof 1999 and held by ED in June of 2000.

n We have an advisory panel for the study that includes several Talent Searchproject directors as well as academic experts in the field of college access. Inaddition to an in person panel meeting (held in January 1999), we have heldperiodic consultations with these directors and will be seeking their review ofthis design document. (Appendix B is a copy of the minutes and agendafrom this meeting.)

n As a followup to the September 1999 COE meeting, COE sent a fax to allProject Directors asking for input about the issues and topics of mostinterest to them for the study. The fax also asked about their willingness toparticipate in a study of alternative services .

n We held a teleconference with eight project directors and others interested inthe study. We discussed study design issues in depth (Appendix D is a copyof the minutes of the Project Directors Teleconference)

n We included several questions on the project survey that were designed toget specific feedback on the capacities and interests of the projects inparticipating in impact studies


28

The questions we put to projects included:

n What topics were of most interest?

n What information would be most helpful to them in providing services?What did they most want to get out of the planned study?

n How willing and interested were project directors in participating in anevaluation focused on program improvement?

1. Project Director Feedback Discussions and COE Fax Responses

The discussions and the fax identified a number of issues of current concern to projects.We enumerate them here:

n Maintaining and developing access to students within the schools

n High stakes state and district testing and its impact on low-income studentsand Talent Search Programs

n Parental involvement and the best way to work with families

n Effective ways to use computers and technology

n Maintaining attendance and keeping interest of participants especially seniorhigh students

n Having sufficient resources to serve students and meet reportingrequirements of Department of Education

n Effectively tracking highly mobile students

n Effectively serving the high school drop out population

n The role of summer programs

n Establishing the most effective staffing configuration; use of part-time andvolunteer staff

The role of ACT/SAT preparation in TS

n Integrating Service Learning concepts

About 35 of the Talent Search projects responded to the fax or filled out similar forms atthe fall meeting of COE. While those who returned the forms might be considered thosewith higher-than-average interest in the study, more than two-thirds of this group indicated


29

that they would like to participate in a study of alternative impacts in which students receiveddifferential services. A few projects had specific services and designs that were shared onthe form. Those directors who indicated they did not want to participate usually cited recentstaffing losses or instability in the program as the major reason.

The Project Director Teleconference allowed for more in-depth discussion of thesetopics of concern and also more in-depth discussion of study design goals. In theteleconference, we asked project directors to identify, from their perspective, goals ordesirable characteristics for the impact study. The directors said the following factors wereimportant:

n The study would need to maintain a dual focus or purpose, (1) to address theoverall impact of Talent Search and (2) to look at the impact of specificservices.

n It should focus on short-term impacts.

n It would need to take into account the low-intensity nature of programs.

n The study ought to look at Talent Search as it operates within the school andcommunity systems.

n It should identify "best practices" and ideas for program improvement.

n It should help directors find out which service models work best for projectswith similar populations and situations.

n It should obtain information at a more detailed level than overall impactevaluations.

2. Project Survey

The descriptive survey of all Talent Search Projects operating in 1999 is another majorsource of feasibility information. As of the middle of August, about 86 percent of 361projects had returned the project surveys. We note responses to two key feasibility items-willingness to participate in an alternative service evaluation and types of records kept onprogram participants.

a. Willingness to Participate in an Alternative Service Evaluation

The Project Survey included a question similar to that on the COE fax " Would you beinterested in participating in a study designed to assess the effectiveness of new or alternativeservice models?" Of those who returned the survey and answered this question, 39 percentindicated they would be interested in having their project participate in a study designed toassess the effectiveness of new or alternative service models.


30

When asked about assessing the effectiveness of new or alternative service models,Talent Search project directors noted the following five services as the items they wouldmost like to assess (see Table 11.11):

Tutoring

n Mentoring

n Parental and family involvement

College preparatory activities

n General effectiveness of and participant satisfaction with services

Other frequent choices included assessing a technology component of the program orassessing outreach activities targeted towards rural and disadvantaged populations.

Table 11.11: Talent Search Project Director Responses to Question Asking WhichServices They Would Study

If yes, which services would you assess?N=108

Number PercentageTutoring 12 11Mentoring 9 8Parental/family involvement 9 8College preparatory activities 8 7Effectiveness of/satisfaction with services 8 7Follow-up 6 6Technology component 6 6Outreach 6 6Workshops 5 5Counseling 5 5Summer programs 5 5Recruitment 4 4Financial aid 4 4Student progress 4 4Assessment 4 4Staffing issues 3 3Career services 2 2Collaboration with others 2 2Retention and re-entry 2 2Other 5 5

Source: Data calculated from the National Study of Talent Search, Project Directors Survey: 2000, with 85 percent ofprojects responding by the time of this calculation


31

a. Project Recordkeeping

To conduct an impact study, it is necessary to have clearly identified group of programparticipants, and it is most helpful to have information on the extent to which theparticipants have received services and on the type of services received. The surveycontained several questions designed to assess project capacity to provide information onindividual participants and the degree and type of participation in the project.

The survey responses indicate that almost all (99 percent) of Talent Search projectsmaintain some individual-level records on participants, and 90 percent of projects have someparticipant-level information in a computerized database. More than three-fourths (77percent) indicated that they had service records in a computer database, and virtually allprojects (99 percent) indicated that they kept some form of paper service records.

Table 11.12: Type of Records on Participants Kept by Talent Search ProjectsTalent Search ProjectType of record Paper records Computer databaseDemographic information 98% 90%SAT scores 58 29ACT scores 63 33Other standardized 59 24Attitude scale profiles 29 7Career survey results 83 26High school or postsecondary transcripts 85 23Recommendations 77 15Diagnostic tests 38 10Postsecondary applications 68 28Financial aid applications 71 32Postsecondary enrollment 82 61Individual participant contact sheets 98 69Records of service received 99 77Follow-up information on former participants 74 52Projects assessment records 90 45Source: Data calculated from the National Study of Talent Search, Project Directors Survey: 2000 with 85 percent ofprojects responding by the time of this calculation

The five most frequent types of participant-related information collected and maintainedin paper records are:

n Records of services received (99 percent)

n Individual, participant contact sheets (98 percent)

n Demographic information (98 percent)

n Project's assessment records (90 percent)


32

n High school and postsecondary transcripts (85 percent)

The five items most frequently collected and maintained in a computer database include:

n Demographic information (90 percent)

n Records of services requested (77 percent)

Individual participant contact sheets (69 percent)

n College or postsecondary enrollment (61 percent)

n Follow-up information (52 percent)

As with the paper records, an attitude scale profile is the item that is least frequentlymaintained in a computer database. A lower percentage of computer-based records thanpaper records are maintained for each participant-related item.

More than four-fifths of Talent Search (85 percent) project directors replied that theyretained participant-related information in their project files for more than two years. Fewerthan 10 percent of Talent Search programs (7 percent) retain information for less than ayear.

We also asked projects whether they tracked certain outcomes for participants. The fiveitems that Talent Search programs most frequently track for all or some of their participantsare:

n High school graduation (100 percent)

n Year-to-year progression through high school (98 percent)

n Enrollment in college (97 percent)

n Completion of college application (96 percent)

n Grades (92 percent)

The item that fewest number of Talent Search programs track is graduation fromcollege; only 40 percent of the programs track college graduation.

C. CASE STUDIES

In all 16 Talent Search projects have been visited as part of the study over the past 18months. In the 14 case study sites, we conducted one-week visits. The one-week site visits


33

were student-centered and included focus groups with current and past participants as wellas observations of workshops, tutoring and lab sessions held at various sites. The visits alsoincluded extensive interviews with Talent Search staff and Target School staff. Visitors alsoexamined project databases. A detailed report of the case studies and survey results is underpreparation. The preliminary observations and insights reached from the site visits used toinform the feasibility study were the following:

Overall Objectives

n Target schools and participants hold Talent Search programs in high regard.

n Schools, parents, and students consistently report that there are not enoughservices for the need.

n College trips were among the most popular services among students asindicated in the focus groups and interviews with students and alumni.

n Participants, teachers and counselors in the Target Schools perceive thatmajor benefit is Talent Search program's communication to the participantthat he or she is "special" and that they it is possible to attend collegedespite obstacles

n Projects visited were well-organized in terms of service delivery and recordkeeping

n Technology use is of considerable interest to the projects and is a majorfocus of attention.

Issues Among Projects

n Some project directors noted that maintaining staff was an issue for theirprojects given the low salaries.

n Projects differed in the extent to which they tried to serve academic lowperformers and most did not target those with lower than average GPAs.

n Projects differed in whether they considered academic support a proper rolefor Talent Search, with some noting that they did not have the resources todo this.

n Some projects expressed that they were struggling to know how best toengage parents.

n Some projects noted issues related to difficulties with a model of service thatpulls students out of classes. This issue became more significant in senior


34

high school. The introduction of high-stakes testing was also seen as areason to make pull-out programs unpopular among teachers.

n Treatment intensity and issues of how wide to cast services were importantconcerns.

n Projects differed in the extent to which and the manner in which they usedcollege student paid staff and student or community volunteers.

Feasibility of Impact Study

n Record systems could support the study

n Projects differed in their interest and willingness to consider testinginnovative or alternative services in their projects.

n Most projects had eligible students in target schools they were not serving.

D. EXPLORATION OF THE POTENTIAL USE OF FEDERAL AIDRECORDS AS A WAY TO TRACK PARTICIPANT OUTCOMES

Our approach to the study is to make full use of data that has been or will be collected tohelp us answer the study questions. In the sections above, we have discussed use ofperformance reports, CCD and IPEDS. The yearly federal aid application and award files areanother major source of existing data that can inform this study.

1. Using the Aid Files as an Indicator of Performance

The federal aid files are potentially strong outcome indicators for Talent Search becausethey address the major outcome goals of the project. They are a complete representation ofthose who applied for federal aid and who received a Pell award. In this way they avoidsurvey non-response bias from the tendency to obtain survey responses from those mostlikely to be enrolled in college. Surveys also frequently have higher response rates from thosewho are in the treatment group compared with the comparison group.

The aid files contain student family background information, information on aid status,and for enrolled students information on enrollment status and institution attended.Table 11.13 summarizes information contained on the files.


35

Table 11.13: Variables Present on the Federal Aid Files

Student Background Financial Aid Academic Status

n Social Security number

n Eligibility

n Institution attendedn Street address

n Cost of attending

n Plan to attend samen State of legal residence institution collegen Date student became legal

n Type of aid received

n Season enrolled andresident of state

n Amount of Pell grant enrollment statusn Marital status

n Expected family

n ` Program of studyn High school graduation contribution

n Program completion datedate

n Student income

n Degree/certificaten GED date

n Eligibility for Pell grant

n Year in collegen n Undergraduate/graduate

student status

The major limitation of using this information as an indicator of college outcomes is thatit covers only those who applied for federal financial aid and does not include those who donot seek federal aid while attending college. It cannot directly tell us the percentage ofstudents who enter or persist in postsecondary education.

Despite this limitation, it is possible that the aid application and award information canbe used as an indicator of project success for the population served by Talent Search. Anindicator does not mean that all participants are tracked—only that the measure is consistentand unbiased and that it indicates something meaningful about the topic of interest.Potentially it allows changes over time to be noted and comparisons among subgroups to bemade.

Because there is no existing national sample of Talent Search participants and theDepartment of Education does not collect individual records on Talent Search participants,it was not possible to look directly at the match potential or use of these files for trackingTalent Search participants. However, the Office of Federal TRIO Programs under thePerformance Report Technical Support and Analysis Task Order Contract asked MPR toexplore use of the federal files using the samples from the two existing TRIO evaluations,Student Support Services (SSS) and Upward Bound (UB). Our work involves two activities:

1) Matching the SSS and UB samples with the files for applicable years

2) For SSS, attempting to observe the extent of difference in outcomeconclusions that would have been reached had we used the aid files ratherthan surveys and transcripts as indicators of outcomes

A detailed report of the results of these using federal aid indicator data as a proxy for theoutcome measures is under preparation (Cahalan, Kim, and Humphrey 2000). Preliminaryresults indicate that such matching is feasible. The preliminary results of the second analysisalso indicate that using aid file measures yielded similar results to the analysis of outcomesrecently conducted for SSS.


36

Table 11.14 shows results from matching the files with the samples from the NationalEvaluation of Student Support Services. For the Student Support Services matching there isa one-year difference in the survey data being compared and the aid year matched (1993-94survey and 1994-95 aid file) because aid information for the survey year is not available.Despite this difference there is high correspondence between the findings from the file andthe survey reports. Eighty percent of the SSS participant sample who stated on the surveythat they received financial aid from any source in 1993-94 were found on the file a year laterin 1994-95. Those least likely to be found on the federal aid files were those not respondingto the survey, confirming the fact that follow-up surveys are more likely to reach thoseenrolled in college and may overstate the percentage enrolled. The federal files are also lesslikely to have those students reporting they are enrolled part-time, and those at two-yearinstitutions and those with higher family incomes (Table 11.14). Overall the match rates arehigher than we expected, given that the financial aid data were collected a year after thesurvey information. The rate may have been even higher, had we been able to merge surveydata and financial aid data from the same year.

Table 11.14: Results of Matching Student Support Services National StudyStudy With Financial Aid Files

On 1994-95 Federal

Not On 1994-95Survey Response

Aid File

Federal Aid File

Reported receiving financial aid in 1993-94 80.2% 19.8%

Enrolled in 1993-94 70.0% 30.0%

Enrolled Full-time in 1991-92 57.0% 43.0%

Attending Two-year Institution in 1991-92 39.7% 60.3%

Attending Doctorate Granting Institution 1991-92 56.5% 43.5%

Family Income Less than $20,000 in 1991-92 59.9% 40.1%

Family Income Greater than $20,000 in 1991-92 48.3% 51.7%

Did Not Respond to Survey in 1993-94 32.1% 67.9%Note: The data used in this analysis were collected through the baseline and longitudinal study component ofthe National Study of Student Support Services. The longitudinal study, sponsored by the U. S. Department ofEducation, Planning and Evaluation Service, was designed to measure the effects of SSS programs on thecollege success of participants. The study followed about 2,500 participants and 2,500 nonparticipantsbelonging to the comparison group for six years from 1991-92 to 1997. We use information from the 1993-94a third-year follow-up combined with baseline information (at the time of this work the six year follow-upinformation was not available). Students were asked to answer questions about their background, secondaryand post-secondary education decisions, career aspirations, sources of funding for education, and collegeexperiences

The results of these matching efforts and replications of analyses of impacts for StudentSupport Services (SSS) led us to conclude that the federal aid files were a valuable source ofinformation that should be used in TRIO evaluations.


37

E. USING STATE-LEVEL DATABASES AND ADDITIONAL SOURCES OF POST-

SECONDARY DATA

Student transition data systems that track students from high school to postsecondaryeducation are an additional source of existing data that we have explored using for the TalentSearch study. Recently the State Higher Education Executive Officers (SHEEO) publisheda report on the . prevalence and content of these databases based on a survey of states itconducted. Using this information as a starting point, MPR made calls to those states thatappeared from the SHEEO report to have databases with the most potential for our study.Based on calls to 14 states, we narrowed our search to the five states that are discussed indetail in Chapter III.

The process for obtaining access to files varies from state to state and only a few states(for example, Texas, North Carolina, and Florida) currently have state-unit-level databasesfor K-12 students that are potentially linkable to postsecondary databases. One of thedifficulties in using the state higher education databases is that they are not comprehensivelists of students. Unless there is a link with a K-12 database, it is not possible to develop acomparison group. For example, the higher education databases do not include high schoolstudents who did not enroll in college. There are also serious data access, security, andconfidentiality issues that need to be considered in each of the states. Chapter III detailsthese issues.

F. CONCLUSIONS FROM FEASIBILITY WORK THUS FAR

In summary, our feasibility work has led us to the following conclusions that haveinformed the designs put forth in the next chapters of this report.

Design Needs

n There is a need expressed by projects and the congressional mandate forTRIO evaluations to develop a design that will address issues of nationalimpact as well as the question of exploring the relative utility of alternativeservices and exploring service improvement.

n Given the intensity and nature of Talent Search services, the original focuson short-term outcomes continues to seem most appropriate for this study.

n Given the outreach quality of TS and its integration into school andcommunity systems there is a need to take a design approach that considersTS as part of a system and looks at the role of TS within the school andcommunity systems.

n There is a need to have a design that is consistent with observing systemicchanges that are occurring related to school reform, technology, servicelearning, and other programs such as GEAR UP.


38

Characteristics of Projects that Impact the Design

n Almost all of the Talent Search projects have clearly identified participantsfor whom the projects have SSN numbers and some background and serviceinformation that is in an electronic database.

n Typically, Talent Search projects serve only a portion of the income-eligiblestudents in a given school indicating that in most cases some form ofmatched in-school comparison groups would be possible.

n Selection bias remains an issue. There are also contamination issues becausesome of the Talent Search services are provided to the entire school. Thiswould suggest the utility of having comparison groups from non-TalentSearch schools as well as the Talent Search schools.

n In most projects there is some natural variation in the intensity of servicesreceived; however, the variation may be based on selection criteria related tooutcomes. For example, those who are having trouble with a subject may bethe ones to get tutoring or those most motivated go to the technologycamp—issues we are familiar with from the SSS study.

Thoughts and Promising Avenues for Evaluation

n While there is not overwhelming interest, a substantial portion of TalentSearch projects have a serious interest in evaluating innovative services.

n Increasing the rate of application and receipt of federal aid for college amonglow-income high school students has been a major goal of the TS projectfrom its initiation. Hence the federal application and aid files represent anexcellent outcome measure and one that can be obtained at a relatively lowcost. The major issue is obtaining valid comparison groups and developingvalid ways to estimate effects.

n A number of states are looking into K-12 postsecondary transition issues.Some are developing transition databases and feedback reports to highschools on postsecondary attendance in their states. Linking with theseefforts is a promising approach.

n Given the need to look at Talent Search as part of a system with a focus oninteractions, some of the concepts or metaphors from complexity and chaostheory appear to have some relevance. This points to the need to includeprocess studies as part of the impact studies.

Chapters III to V present a three-pronged approach to looking at the impact of TalentSearch addressing state, national and alternative service impacts.


CHAPTER III

PHASE II STATE-LEVEL IMPACTSTUDY DESIGN

his chapter presents the design of a study to estimate the impact of Talent Search(TS) on high school (HS) completion and postsecondary enrollment (PSE)enrollment in several states using existing records. Our decision to proposestatewide records based studies is based on the potential for this type of study to:

Maximize use of state and project records

n Address the question of impact of the TS program in a statistically powerfulmanner

n Include a process study to inform the statistical impact study

n Facilitate observation of participation of sample members in statewideprograms similar to TS

n Allow observation of how TS operates within the state education systemconsidered as a whole

The first section discusses the policy and research questions addressed by this study.The second section presents the study design, including the experimental design, propensitymodel technical approach, data collection plan, and sampling plan. The third sectiondescribes the setting and the databases for each of the recommended states. The finalsection outlines the process study to be conducted in each study state.

A. POLICY AND RESEARCH QUESTIONS

This study measures the TS program's impact on:

40

Taking college preparation courses in HS

n HS graduation

n Taking the SAT or ACT

n Applying for and receiving federal financial aid (FFA)

PSE enrollment, and type of PSE institution.

The study also measures the impact of participating in TS for a long period, and to theextent that project service records allow addresses the impact of different mixes of serviceson college outcomes.

The statistical impact of TS is measured relative to a world without TS but with otherprograms that provide similar or related services. It determines what the TS participant'sexperience is relative to what her experience would have been had she not participated in TSbut may have received services similar to those of TS.

We have worded this statement of purpose to highlight its "compared to what," orcounterfactual, aspect. The counterfactual is a world that is identical to a world with TSexcept that there is no TS program. The counterfactual world has other programs, asdocumented in the section on individual state environments below, offering services similaror related to those of TS. For example, the state education office in Indiana mails a packetof information on college and financial aid to every HS student. That means that in thecounterfactual world, a student who would be a TS participant in the real world couldpossibly receive TS-like services.

A second example of control students receiving TS-like services is that some controlgroup members may have received services from the TS project itself. This can happen inseveral ways: While we will know from project records who has participated in Talent Searchwithin the states, the student could have participated in TS in an other state or in a projectthat no longer operates in the State. Second, the student could have attended a single TSworkshop sponsored by a project that enrolls a student only after she attends two projectevents. Third, some projects occasionally deliver a workshop to an entire class, grade level,or school, not just official TS enrollees.

This means that in the state records study, we are estimating the impact of the TSprogram, not the impact of TS-like programs. The counterfactual for estimating the impactof TS-like programs is a world in which HS students receive no information on college orfinancial aid from programs dedicated to that purpose. In effect, students would not receivesuch information from any source other than their parents or HS counselor.

III. Phase II State-Level Impact Study Design

41

Another component of the statement of purpose is that we are measuring the impact ofTS on students, not dropouts or adults.

Within the broad policy goal of estimating the impact of TS, this study has these specificobjectives and limitations:

First, the study will estimate the impact of TS in selected states. This means the resultingimpact estimates will not be nationally representative. However, accepting this limitationenables us to use the complete state K-12 and higher education, databases. Because the statedatabases' unit of analysis is the individual student, they support statistical models that arestatistically powerful. Using records instead of survey will allow the study to be completedwith less time and financial resources.

In contrast, the impact estimates in the next chapter are national in scope. However,since national estimates require us to use the school district as the unit of analysis, thoseestimates are less powerful—meaning they may not detect the true impact of TS if the effectis small.

Second, the study focuses on the impact of TS in its entirety. The state records basedstudy will address the impact of specific services but only in a limited manner. We plan tocollect service records from all projects within the state. We also have access to the annualperformance records that include detailed information on the service mix and intensity ofservices within the project. This information may allow us to observe if there are differencesin impact related to:

n Length of time in Talent Search

n Specific types of services received

n Intensity of services received

However, the quality of the service record information available will vary and this may limitthese analyses. The Phase III demonstration, discussed in Chapter V, is specifically designedto estimate the impact of several specific TS services.

Third, the study uses records data from TS projects, state education agencies, and theFFA database. It does not require the collection of survey data or records data from schooldistricts or schools. In contrast, the Phase III demonstration requires surveys of researchsample members.

Fourth, the study uses statistical matching, rather than random assignment, to simulatethe counterfactual. The national study, presented in Chapter IV, also uses this technique,whereas the Phase III demonstration uses random assignment. Generally, statisticalmatching is less powerful for detecting impacts than is random assessments.


42

B. STUDY DESIGN

This section presents the study design, beginning with the experimental design. Thenext section reviews the propensity score model, and the final two sections present the datacollection plan and the sample plan.

1. Experimental Design

As mentioned above, the unit of analysis for this study is the individual student. Thestudy will observe outcomes for cohorts in the 11`h and 12th grades in 1998-99.

The study estimates the impact of TS on these outcomes: taking college preparationcourses in HS, taking the SAT or ACT, HS graduation, applying for and receiving FFA, PSEenrollment, and type of PSE institution. A treatment student is one who is in the studycohort and participated in TS at some point as determined by project records within thestate. A control student is one who is statistically identical to the TS students but did notparticipate in TS during HS.

Ideally, we would generate a control group that is statistically identical to the treatmentgroup through random assignment. We would draw a sample of 11`h and 12`h graders andrandomly assign each to either the treatment (TS participation) or the control groups (no TS

participation).

It is not possible to use random assignment for the Phase II studies and so we must useanother method for generating a statistically identical comparison group, specifically apropensity scoring approach (Rubin, 1997). This approach involves matching each treatmentstudent with another student who is as similar as possible other than the fact that she did notparticipate in TS in 11th and 12 th grade. Similarity is measured by a propensity score. Propensity isthe likelihood of participating in TS at any time during those two years. The propensityscore is assigned using a propensity model, a regression model of the likelihood of participatingin TS. This approach is more powerful and less prone to bias than are other statisticalapproaches used in the absence of random assignment.

2. A Propensity Model Approach to Estimating the Impact of TS

We begin by developing a list of 11 `h and 12`h graders, as described in the next section.We will use project data to determine whether each student is or has been a TS participant.Students who have participated in TS at some point as determined by the records form thetreatment group. The rest are candidates to be matched comparison students.

The propensity model's dependent variable indicates whether the student participated inTS. The independent variables include demographics, school, performance in school, andpoverty status. We will estimate the parameters of the propensity model using all the


43

students in the sample. We will use an econometric technique appropriate for a nonlineardependent variable, such as a logistic or probit regression technique.

The next step is to predict a propensity score for each student in the study sample. Thisis done by substituting the specific characteristics of each student in the sample into thepropensity regression equation, multiplying the characteristics by the estimates coefficients,and adding up the resulting products. The penultimate step, before estimating the impact ofTS, is to match each TS participant to the non-TS student with the closest propensity score.

The impact of TS for the individual TS participant is the mean value of the differencebetween her outcome variable and that of her matched non-TS student. We will computethe impact for each outcome variable. We will also estimate the impact for key subgroups,defined by race/ethnicity and gender. And we will estimate the impact separately forparticipants who received TS services over a long period versus those who participated forshorter periods. To the extent possible we will also observe the impact of different services.

Unfortunately, estimating the impact estimate's precision using propensity scoremethods is typically not straightforward. The impact estimate's variance comes from twosources, sampling and modeling. Sampling variance results from the fact that the impact iscomputed for a sample of students, rather than the population of students. Modelingvariance comes from the fact that the propensity score model almost certainly omits somerelevant but unmeasured variables. We are not aware of an analytical method of combiningthese two sources of variance.

In lieu of an analytical method, we will estimate the impact estimate's precision usingreplicate sample techniques. We will draw several replicate samples of treatment studentswith replacement, match each replicate sample with comparison students using the estimatedpropensity model, and compute an impact estimate for each replicate. Then we will calculatethe variance of the distribution of impact estimates. Finally, we will use the estimatedvariance to test the hypothesis that the initial estimated impact is significantly different fromzero.

3. Data Requirements and Data Acquisition Plan

This section presents a generic data acquisition plan for the state studies. It is generic inthe sense that it is approximately correct for each state in the study. The relevant databasesin the five states We have investigated — Texas, North Carolina, Florida, Massachusetts, andIndiana — are not identical. This means that the generic data acquisition plan will beadapted to each states' idiosyncrasies. The most important adaptations are presented in thesections describing each state.

The initial list of 11 'h and 12th graders will be drawn from the state K-12 database.Treatment students will be identified by matching the resulting 11`'' and 12`h grade data withlists of TS participants from the TS project management information systems (MISS). In


44

order to ensure that we know whether a student in the cohort of students in the 11th and 12th

grades in 1998-99 has ever been a Talent Search participant, we plan to obtain lists of allparticipants from each project from 1993 to 2000. We will also know from the statewidedatabase whether a student has been a participant, for example, in a statewide program suchas 21 51 Century Scholars Program in Indiana or CROP in Florida. We will consider whetherthis is a reason to remove the student from eligibility in the comparison group. Theremaining students will be considered candidates for matched comparison students.

The dependent variables for measuring impacts is taking college prep courses, taking theSAT or ACT, HS graduation, applying for and receiving FFA, PSE enrollment, and type ofPSE institution. The source of data for HS course-taking and graduationis the state K-12database. The source of data for FFA application and receipt is FFA data. This source hasthe advantage of including students enrolled in colleges in other states. The source of datafor SAT/ACT, PSE enrollment, and type of PSE is the state education agency's highereducation database. This source has the advantage of providing data on college enrolleeswho did not submit a Free Application for Federal Student Aid (FAFSA) and more detail onthe student's college experiences. Unfortunately, it does not include students who go tocollege in another state.

We will estimate the impact of TS on each of the outcome variables. The dependentvariable for the propensity model is whether the student participated in TS. The source ofthese data is the TS projects' MIS databases. The independent variables includedemographics, school performance, achievement test scores, and eligibility for school lunch.The source of these data is the state K-12 database.

4. Sample Design

We plan to estimate the impact of TS in each of four selected states. The study is limitedto selected states because state K-12 and higher education databases are the sources of muchof the required data. We investigated five states — Texas, Indiana, North Carolina, Florida,and Massachusetts — because of their capacity and willingness to participate in the study.We will estimate a TS impact for each state. This means that the impact estimates will notbe statistically representative of the nation, as will be the impact estimate presented in thenext chapter. It is likely that the state impact estimates will not be identical, so we willinclude an in-depth study of each state, outlined in the final section of this chapter, todetermine why.

For states whose K-12 databases provide information sufficient for providing a list ofcomparison students and the independent variables of the propensity model, we plan to useall 11 th and 12th grade TS students in the state as the treatment group and all non-TS studentsat all target schools (or at all comparison schools, depending on the TS participation rates intarget schools) in the state as candidate comparison students. We will not draw a sample ofprojects, schools, or students. In states for which all the needed data are available instatewide electronic databases, the marginal cost of including one more student in the


45

analysis is small. Indeed, it is sufficiently small that the benefit of increasing the model'sstatistical power by using the additional student outweighs the cost of processing theadditional data record.

Using all available data, as opposed to drawing a sample, will increase the impact model'spower by reducing the impact estimate's variance. Even if we use all treatment students inthe state, the impact will still be a statistical estimate because of the error from thepropensity model. Thus, we will estimate the impact estimates' variance using the replicatesample techniques described above.

Our preliminary analysis suggests that the K-12 databases in Florida, Texas, and NorthCarolina meet these requirements. This means we can conduct the impact study in thesestates without sampling.

Indiana does not have a statewide K-12 database in a manner similar to Florida, Texasand North Carolina. However, they have a statewide survey of 9th and 11 `h graders. Thissurvey contains information on academic grades, college aspirations, and parent education,but does not include a parent income indicator. These omissions may reduce the power ofthe impact model, relative to those of Texas, North Carolina, and Florida, and will changethe interpretation of the estimates. -

Finally, Massachusetts is in the process of developing a new K-12 database. Conductinga study there might require collecting records on students in the research sample from eachtarget school in the study. There are also issues of obtaining access that appear moredifficult than in the other states we are considering. Since this will require substantially moretime and money than will the study in states with a K-12 database, and since several otherstates with such databases appear to be available for the study, at this time we recommendagainst conducting the study in Massachusetts.

5. Feasibility and Risks

The design presented above illustrates that the state studies are feasible in two senses.First, it is logistically feasible. We have identified several states that have data sufficient tofund meaningful comparison sites and to support reasonably precise impact estimates of awide variety of outcome measures.

The second sense of feasibility focuses on the risk that we will be unable to detect thetrue impact of TS. Such an inability would come from a statistical model that is notpowerful enough to detect the true impact of TS. In the setting of a demonstration, we candesign a survey sample with a size and structure sufficient to detect even small impactsreliably. When-we are limited to administrative data, as opposed to survey data, we cannotbe certain that the statistical model will be sufficiently powerful.


46

Our assessment of the data in the states reviewed below is that the risk of not detectingthe true impact of TS is acceptably small. Therefore, we deem the state studies to be

feasible.

However, this risk is not zero, and ironically may be enhanced by using states with highlydeveloped databases. There may be a correlation between states that invest in an advanceddatabase and states that also operate TS-like programs which comparison group youth mayparticipate in. If any comparison group youth participate in similar programs, the impact ofTS may be small, and thus difficult to detect.

C. STATE STUDIES

In this section we present an overview of talent search projects within five states (Texas,Indiana, Florida, Massachusetts, and North Carolina). We also describe their K-12 andhigher education databases and note the presence of other statewide college preparationprograms for low income students.

1. Texas

a. Overview of the Talent Search Project and Related State Programs

Texas has 16 Talent Search programs, with a total of 14,775 participants (see Table 111.1and Figure 111.1). Projects serve an average of 678 participants who are both low incomeand potential first-generation college students. An average of 339 participants are HS seniorsand an average of 314 are in grades 9-11.

A review of web sites for the Texas Educational Agency and the Texas HigherEducation Coordinating Board revealed no statewide programs with goals similar to TalentSearch. Texas has a statewide GEAR-UP program, which was appropriated $120,000 for thefirst set of grants in 1999. Because these grants were aimed at middle school students, itwould not have been part of the larger service context experienced by the individuals in thisstudy — those who were HS juniors or seniors in 1998-99.

b. Project Capacity to Participate

Eleven of the 16 Talent Search sites had completed the project director survey by June.We contacted all 11 with a follow-up phone call to ask additional questions concerning datathey maintain on program participants. Ten of the 11 sites maintain a database onparticipant information that contains service records. The extent of other data varies (seeAppendix E, Table E-1). For example, while all 10 track participants' demographic data,only five record participants' SAT and ACT scores. The eleventh site does not have a singledatabase that contains every TS participant. Instead, information about participants ishoused at the schools the participants attend.


47

These ten sites collect participants' Social Security numbers that could be matched withSSNs from other files. Every project director also stated that their project could extract datafor all students who were juniors and seniors during the 1998-99 school year.

Table 111.1: Talent Search Projects in Texas

GranteeApproved Number

of Participants 1999 Federal GrantBoys & Girls Club of Greater Fort Worth 730 $220,860Coastal Bend College 700 $232,222Communities in Schools of San Antonio 600 $272,407Dallas County Community College District 600 $193,800Learn, Inc. 1,450 $431,824Paris Junior College 800 $296,765Project Stay, Inc. 1,600 $421,740Southwest Texas State University 1,050 $278,311Southwest Texas State University 600 $220,860Texarkana College 600 $220,860Texas Southern University 1,200 $339,216University of Texas-Brownsville 600 $220,539University of Houston/Downtown 1,100 $288,910University of North Texas 950 $313,506University of Texas-El Paso 600 $215,177University of Texas-Pan American 950 $317,644University of Texas-San Antonio 645 $314,583Total 14,775 $4,799,224


48

Figure 111.1: Texas Talent Search ProjectsBy Number of Participants and Target Locations

Texas Talent Search Projectsby Number of Participants and Target Locations.

University of North - TexasBoys;& Girls Cluh Of

rGreater Fort Worth

, Dallas County CC. District

Paris'Junior College'

Texarkana College

Educational Talent Search Program

University of Texas at San Antomo

* Texas citiesTX Talent Search Projeds

e (0 - 758) Participants! (759 1150) Parti aromas• (1151-2300) Participants

TX Target School Locationsn I target schools

2 target schools

® 3 target schools— 4 target schools

5 or more target schoolsCountiesshp

Univ of HoustonlDowntownTexas Southern University

Southwest Texas State Univ

Southwest Texas State UnivCommunities In Schools

Project Stay Inc.

Note: Unbersliy of Texas Par Americanproject it not shown- msing zip code.


49

c. State Record System

The Public Education Information Management System (PEIMS) is the Texas state databasethat houses K-12 student data as well as information on elementary and secondary schoolorganization, budgets, actual financial data, and staffing. The database tracks all K-12students in public and charter schools. Students are identified by their SSN, which allowsthe state to track students over time. The database was implemented in 1998, and revisionsto reporting data- have been made each year. Data recorded for each student includesidentification, demographics, enrollment, attendance, and reasons for leaving (see Table111.2).

Table 111.2: Variables Included in the Texas Public EducationInformation Management SystemIdentification

ID ChanqeSocial Security number

Prior IDFull nameDistrict ID

Special Program – Career and TechnologyGeneration code

Education

DemoqraphicsGenderEthnicityDate of birthCampus ID of residenceGrade-level

School Leaver (qrades 7-12)Campus IDLeaver reason codeDate of graduationGraduation typeCollege entry

EnrollmentCampus enrollment

AttendanceADA eligibility

District IDLEP indicator

Student IDHome languageParental permission codeEconomic disadvantage codeAt-risk indicatorTitle I, Part ASpecial Education CodeBilingual indicatorESL indicatorGifted and talented indicatorCareer and technology educator indicatorLocal student ID

The Texas Higher Education database tracks all students enrolled in two- and four-yearpublic institutions. The data are updated annually, and the student's SSN is used as anidentifier. The Educational Data Center tracks variables that include demographic data,academic course load, and post-baccalaureate enrollment (see Table 111.3).


50

Table 111.3: Variables Included in the Texas Higher Education DatabaseSocial Security number

Tuition exemptionEthnicity

Financial aidGender

CompletionsDate of birth

Assessment/PerformanceTuition status

Academically disadvantagedResidence

Economically disadvantagedMajor

Individual with disabilitiesDegree intended

Limited English proficiencyTransfer/First-time-in-college

Remedial courseworkContact hour load in academic courses

Cumulative GPAContact hour load in technical courses

Post-Baccalaureate enrollment

2. Indiana

a. Talent Search and State Program Overview

Indiana has eight TS projects, approved for 5,870 participants (see Table 111.4 and Figure111.2). The projects serve participants and an average of 479 and participants who are bothlow income and potential first-generation college students. An average of 81 participants areHS seniors and an average of 282 are in grades 9-11.

Table 111.4: Talent Search Projects in Indiana

GranteeApproved Number of

Participants1999 Federal Grant

Indiana State University 850 $191,625Ivy Technical State College/Northwest 600 $193,800National League of Cuban American Community 820 $329,730Oakland City University 750 $312,375Purdue University, West Lafayette 700 $277,389Purdue University, Westville 700 $232,350University of Notre Dame 800 $321,538Vincennes University 650 $288,910Total 5,870 $2,147,717


51

Figure 111.2: Indiana Talent Search ProjectsBy Number of Participants and Target Locations

Indiana Talent Search Projectsby Number of Participants and Target Locations.

University of:Noire Dame

East Chicago

MunsteNational League of£uban elmencan .Community-Based.

Vincennes Washington'

Jeffersonville

* Indaria citiesIN Talent Search Projects

Oakland City Unive. (0 - 758) Participants! (759 -1150) Partidpants• (1151 - 2300) Partidpants

IN Target Scrod Locations1 target schools2 target schools

® 3 target schods— 4 target schools▪ 5 or more target schools1

Counti esshp


52

A search of the Indiana Commission for Higher Education's web site revealed two stateprograms that may help disadvantaged students learn about and prepare for college:

• Indiana Career and Postsecondary Advancement Center (ICPAC). ICPAC wascreated in 1986 to inform, encourage, and support the education and careerdevelopment of state residents. It claims to be the largest state-supportedpostsecondary encouragement initiative in the country. The program hosts aweb site and produces information packets and brochures on careerpreparation, postsecondary education, and financial aid. Students enroll inthe program by completing a four-page form, typically at school.Information is currently sent to students in over 300,000 households eachyear. This figure represents virtually all students in the relevant age range,not just students who express an interest or explicitly request theinformation.

n Twenty-first Century Scholars Program. This program began in 1990 as a way toraise the educational aspirations of low- and moderate-income families.Income-eligible students ($30,895 for a household of four) can enroll in theprogram in 8th grade. School officials distribute program information andapplications to students they believe may be eligible. The programguarantees to cover the cost of four years of tuition at virtually any public orprivate college (including proprietary schools) in Indiana for all programparticipants who fulfill the program pledge. In taking the pledge, studentspromise to (1) graduate from an Indiana HS with a diploma, (2) achieve anHS GPA of 2.0 or better, (3) abstain from alcohol or illegal drugs, orcommitting crimes, (4) apply for college admission and financial aid as an HSsenior, (5) meet regular college admission requirements, and (6) take a full-time load in college. However, unlike TRIO programs, the Twenty-firstCentury Scholars Program does not offer any direct services to studentsduring middle or HS. Students with questions about college can contactICPAC, described above. The program currently enrolls about 8,500 8thgraders per year, which is about 50 percent of the eligible population, but hasa goal of increasing that figure over the next few years to at least 11,000,which would represent about 70 percent of eligible students. However, thenumber enrolled represent only about 10 percent of all 8th graders in thestate. The Twenty-first Century Scholars Program is Indiana's statewideGEAR-UP program. (Indeed, the program may have been a model forGEAR-UP legislation.)

In addition to these programs, the State Student Assistance Commission of Indianahosts a web site with information on various state and federal financial aid programs,including a page of "helpful hints" for completing the FAFSA. A search of the IndianaDepartment of Education's web site (the central resource for K-12 education) revealed noadditional state-wide or state-based programs similar to TS.


53

b. Project Data

All eight Indiana-based TS projects have returned the project director survey by June.We contacted all eight projects to ask a few additional questions concerning the data theymaintain on program participants. Appendix Table E-2 presents detailed information, fromthe survey and our follow-up calls, on each project's data systems. The table reveals that thetype of information available on individual participants varies substantially among projects.Fortunately, for our purposes, seven of the eight projects maintain participants' SSNs intheir computerized databases. However, two of the eight do not maintain computerizeddata on the services participants receive. In addition, two of the eight projects we contactedare new grantees that started operating in 1998-99 and served only 15 HS seniors in its firstyear. These projects may have to be excluded from our study because they will not have thekind of historical data on participants that we would hope to obtain from other projects.

c. State Data

We have identified two potentially useful sources of data on students in Indiana:

n ICPAC Surveys. Every year, ICPAC collects student name, parent name,address, phone number, sex, birth date, parent educational attainment,languages spoken at home, and race/ethnicity on virtually all 9th and 11thgrade students. Only Twenty-first Century Scholars Program participants areasked for their SSN. The form also asks a few questions about career andeducation interests/aspirations. (See Table 111.5) A few schools do notcooperate with the data collection effort, and perhaps eight percent ofstudents are typically absent on the day the form is filled in. But officialsestimate that each year, they get basic information on at least 85 percent ofthe target population. Since students are asked to complete the form in both9th and 11th grade, records can be matched by individual across the twosurvey points. Using name and date of birth, officials can match records for80 percent to 90 percent of the students.

n Commission for Higher Education's Student Information System (SIS). The purposeof Indiana's SIS is to provide comparable, accurate enrollment and financialaid information on all students enrolled in public and private colleges in thestate. Each institution files an annual report that contains a variety of datafields on each enrolled student, including a unique identifier (expected to beSSN, unless the student has requested that it not be used in the recordssystem) and basic demographics; a host of HS related items, such asgraduation date, class rank, cumulative GPA, and SAT scores; informationon many specific types of federal, state, and private aid sources (e.g., amountreceived), plus income and the student's and parents' expected contribution;and data on college credit hours taken and degrees earned. We downloadeda copy of the SIS instructions for the 1998-99 academic year.


54

We had one preliminary conversation with two state officials about the possibleavailability of these data for our study. They reported a longstanding interest in merging thetwo data sets to conduct their own research on state students. But they have not been ableto do this because of a lack of resources (funds and programmer time) and bureaucraticinertia. A colleague of theirs recently estimated the cost of merging the databases and doingsome initial analysis programming at about $30,000. We discussed the possibility of usingTS evaluation funds to pay for the merging effort in exchange for a copy of the resultingdatabase, an offer they found very appealing. However, they warned that whether theresponsible agency officials can be moved to make it happen remains an open question.' Ifthings went smoothly, they said, the merged database might be available by December 2000.

TABLE 111.5: DATA COLLECTED ON THE STATE OF INDIANA'S ICPACPROGRAM ENROLLMENT FORMS

9th Grade

11th GradeVersion

Version

Student last name, first nameParent last name, first nameAddress, phone numberSexDate of birthHighest level of school expect to achieveGrades usually earnThings that might limit education/training pursuits afterhigh schoolAcademic programs have an interest inAcademic programs currently enrolled inInterest in volunteering, mentoring, job shadowingThings would like to do first year after high school

xCareer areas of interest

xPlans concerning working while in high school

xWho student lives with during school year

xDoes a parent have four-year degree

xLanguages spoken at home

xRace/ethnicity

x

xMembership in Twenty-first Century Scholars ProgramSocial Security number (if member of Twenty-first

xCentury Scholars Program)Activities undertaken to prepare for college/career

xAssistance received toward getting a job

xIndiana colleges interested in

x

'They also cautioned that it would likely be very difficult for us to get student-level data, such as

transcripts, from state school districts, because officials are very protective of such information.

xxxxxx

xxx


55

3. Florida

a. Overview of Talent Search Projects and Related State Programs

Florida has eight Talent Search programs that serve a total of 5,451 participants (seeTable 111.6 and Figure III.3)and an average of 550 participants who are both low-income andpotential first-generation college students (see Table 111.6). An average of 128 participantsare HS seniors and an average of 378 participants are in grades 9-11.

TABLE 111.6: TALENT SEARCH PROJECTS IN FLORIDA

GranteeApproved Number of

Participants 1999 Federal GrantAspira of Florida, Inc. 750 $256,698Bethune-Cookman College 1,000 $280,932Florida A&M University 800 $265,524Lake-Sumter Community College 101 $193,800Pensacola Junior College 700 $241,484Polk Community College 600 $193,800Rollins College 800 $256,683Santa Fe Community College 700 $231,903Total 5,451 $1,920,824

Though Florida has numerous education programs for high-need students, the CollegeReach Out Program (CROP) is the program whose purpose is most similar to that of TalentSearch. CROP was established in 1983 to motivate and prepare educationally disadvantaged,low-income students in grades 6-12 to pursue and complete postsecondary education.CROP serves about 8,100 students through all 10 state universities, 27 community colleges,and six independent postsecondary institutions. CROP's 1999-2000 funding is about $3million. The Post Secondary Education Planning Commission evaluation has found that93% of HS seniors served by CROP received a standard diploma, compared with 71% ofthe random sample. Of those CROP graduates, 77% pursued a postsecondary education,compared with 49% of the random sample.


All eight Florida-based Talent Search projects completed the project director survey. Wecontacted these programs to ask additional questions concerning the data they maintain onproject participants. Appendix Table E-3 presents the type of information available at eachsite. All of the project databases contain the students' SSN, and most of the projects saidthey could extract data on participants who were juniors and seniors during the 1998-99school year. Lake-Sumter Community College is a new program and cannot supply thisinformation.


56

Figure 111.3: Florida Talent Search Projectsby Number of Participants and Target Locations

Florida Talent Search Projectsby Number of Participants and Target Locations

Santa FeCommunity College

* Florida citiesFL Talent Search Projects

. (0 - 758) Participants(759 - 1150) Participants

• (1151 - 2300) PartidpantsFL Target School Locationsn 1 target schoolsp3 2 target schools®3 target schools

4 target schools- 5 or more target schoolsn Counties.shp


57


The Florida Department of Education (DOE) Education Information and AccountabilityServices/Educational Data Systems division maintains a database on all K-12 students inpublic schools. (See Table 111.7) Students are identified using their SSNs, and data on HSgraduation, college attendance, date of birth, race, disciplinary action, and classes taken arecollected for each student.

There are many higher education databases available in Florida. The PostsecondaryEducation Planning Commission (PEPC) uses data from the Division of Public Schools, theState University System, the Community College System, the Office of Student FinancialAssistance, and the Florida Education and Training Placement Information Program(FETPIP) to conduct an annual evaluation of the CROP program. Students' SSNs in theCROP database are matched with SSNs in these databases, and annual reports are producedon HS graduates for each HS in Florida's 67 districts. The PEPC report includes informationon continuing postsecondary education by level (community college, state university system,private university, or postsecondary program within the Department of Public Schools),employment, military enlistment, public assistance participation, and incarceration.

The State University System database contains information on students attendingFlorida's 10 state universities. (See Table 111.8) Data elements include student demographics,academic and scholastic history, and current coursework. The Community College Systemdatabase offers similar information on students at Florida's community colleges. (See Table111.9). FETPIP provides follow-up data on former participants in Florida's educational andworkforce development programs.


58

Table 111.7: Florida Department of Education (DOE) InformationDatabase Requirements — Automated Student Data Elements

IdentificationSocial Security numberStudent number identifier (Florida)Student's legal nameDistrict number (current enrollment)

DemographicsSexRacial/ethnic categoryEconomically disadvantagedBirth dateBirth placeMailing / residence addressLunch statusHomeless studentLimited English proficiencyMigrant statusResident statusHome language

EnrollmentDate of entry / re-entryDaily / period attendanceClass rankCourseGrade levelGrade promotion statusCumulative earned creditsFinancial assistanceGrade point averageScholarship award or recognitionTest scoresReason not testedFirst-time student indicator

Participation in .special programsDropout preventionExceptional studentEarly admission studentEven Start Family Literacy ProgramFederal Impact Aid Eligibility indicatorFederal Compensatory Evaluation — TargetedAssistance indicatorSupport ServicesFederal /State Project indicatorProgram participation prior to kindergartenTeenage Parent programTech Prep participantCareer Academy participant

Incident dataDisciplinary / referral action code / dateCommunity service hoursIncarcerated studentDuration of disciplinary action

Exit dataCertificate of completion (date / type)Diploma date / typeWithdrawal informationVocational termination codeCollege entrance exam board placement test dataHigh school competency test (date passed forgraduation purposes)Minimum Exceptional Student EducationPerformance Standards (mastered for graduationpurposes)


59

Table 111.8: The State University System Board of Regents Database (Florida) -Student Data Elements

IdentificationPerson identification numberPerson name — lastPerson name — firstPerson name — middle initialStudent birth yearReporting institution

DemographicsRacial/ethnic groupGender

EnrollmentType of student at time of applicationStudent's classification levelStudents' date of entryFinal admission actionAmerican College Testing scoreQuantitative scoreVerbal/Other scoreDate of last attendanceHigh school GPACollege GPAAdmissions application program categoryAdmissions eligibility scaleInstitution granting highest degreeDegree- highest level held

Additional variables*Test determining remediationRemediation completion methodRemediation incompletion codeCommon course prefixCommon course numberCommon course outside indicatorCourse section numberCourse program categoryKey — College prepDegree program categoryDegree program — fraction of degree grantedNumber of degree program changesKey — hours to degreeHigh school graduation dateStudent section creditStudent budget entityDual enrollment identificationStudent section funding flagAccount/department CIDDoctoral candidacy flagTermination codeAdvanced placement examination — creditNational Merit Scholar indicatorTerm degree grantedTotal fundable student credit hours

*Note: The State University System Board of Regents database is split into several smaller databases. Thevariables listed here are found only in some of those databases.


60

TABLE 111.9: THE STATE BOARD OF COMMUNITY COLLEGESDATABASE (FLORIDA) – STUDENT DATA ELEMENTS

IdentificationStudent identification numberName – firstName – lastName – middle initialStudent birth date

DemographicsEthnic originGenderLimited English ProficiencyNation of citizenshipDisabled classification

Special programsSchool-to-work indicatorFinancial aid amountFinancial aid award typeOccupational Completion point indicatorAdult literacy completion point indicatorAdvanced technical certificate completionCollege prep. completion indicatorIncarceration status

hours

EnrollmentStudent date of entryTransfer student flagTransfer institutionEntry level/exit level test (score, site, subtest,

type, date)First-time student flagHigh school codeHigh school graduation dateInstitutional class level of studentTerm course load (clock and credit hours)Term hours earned (clock and credit)Term institutional grade pointsTerm institutional hours for GPATerm part-time/full-timeTotal institutional grade pointsProgram of study (hours, level, title)Course grade awardedCourse section hours

4. Massachusetts


Massachusetts has seven TS projects, approved to serve a total of 5,894 participants (seeTable 111.10 and Figure 111.4). The projects serve an average of participants of which anaverage of 559 participants are both low income and potential first-generation collegestudents. An average of 180 participants are HS seniors, and an average of 392 are in grades9-11.


61

TABLE I11.10: TALENT SEARCH PROJECTS IN MASSACHUSETTSApproved Number of

Grantee Participants 1999 Federal GrantColleges of Worcester Consortium, Inc. 800 $247,037The Education Resources Institute 625 $237,964Hispanic Office of Planning and Evaluation, Inc. 1,800 $480,481Middlesex Community College 617 $249,556Mount Wachusett Community College 600 $242,432University of Massachusetts, Amherst 852 $246,985University of Massachusetts, Boston 600 $288,801Total 5,894 $1,993,256

A review of the Massachusetts Board of Higher Education's web site revealed noinformation on state programs similar to TS. Massachusetts has a statewide GEAR-UPprogram. However, because the program is new (grants were awarded in fall of 1999) and isinitially aimed at middle school students, it would not have been available to the individualswe would be studying--those who were HS juniors or seniors in 1998-99. A review of theweb site for the Massachusetts Department of Education (covering the K-12 system) alsorevealed no information on state programs similar to TS.


Four of the seven Massachusetts-based TS projects had returned the project directorsurvey by June. We contacted all four projects to ask a few additional questions concerningthe data they maintain on program participants. Appendix Table E.4 presents detailedinformation, from the survey and our follow-up calls, on each project's data systems. Thetable reveals that the type of information available on individual participants varies widelybetween projects. All of the projects have a computerized database that contains some basicinformation on students, such as name, race/ethnicity, low-income/first-generation status.However, getting additional participant-level data could, in many cases, require project staffto consult hard copy files. Only one of the four projects appears to have SSNs on all theirstudents. One program does not use them at all, and the other two have SSNs only for somestudents. This could hamper our ability to match projects' participant-level data with federalfinancial aid records. Where SSNs are not available, the next best option will be to use nameand date of birth.


62

Figure 111.4: Massachusetts Talent Search ProjectsBy Number of Participants and Target Locations

Massachusetts Talent Search Projectsby Number of Participants and Target Locations.

Middlesex Community College

Education Resources InstituteHissp attic Office of Planning

and Evaluation, Inc.

* Massachusetts citiesMA Talent Search Projects

• (0 - 758) Participants(759 - 1150) Participants

* (1151 - 2300) ParticipantsMS Target School Location(I 1 target schools

2 target schools— 3 target schools

— 4 target schools— 5 or more target schoolsn Counties.shp

Note: U'ersilyofMarsaelunette/Bostonienotshown onnep -mieaingz code.


63


A review of the Massachusetts Department of Education's web site did not showinformation describing any existing statewide student-level database for the K-12 system. Agood deal of aggregated information is collected at the school level, but the only individual -level data reporting form we identified pertains to students suspended for more than 10consecutive school days.2 However, one of the school-level data sources, the Year-EndSchool Indicator Report, may be of some use to our proposed study. The electronicallysubmitted form collects data on attendance, absences, retentions in grade, suspensions andexpulsions, dropouts, HS graduates, and their post-HS plans. These data may permit someinteresting comparisons between TS target schools and schools not served by the program,assuming we could control for other key factors influencing outcomes.

Massachusetts is developing a comprehensive K-12 student-level database, known asthe Student Information Management System (SIMS). The system will contain eachstudent's name and date of birth but will use unique, locally assigned identification numbers,not SSNs. Table 111.1 1 lists the data elements in this system. Unfortunately, this system willcome online in fall of 2000, and thus will not contain data on the subjects of our proposedstudy--students who were HS juniors or seniors in 1998-99. In addition, an official told usthat state law might prohibit the Department of Education from releasing any data thatcould be identified with specific students. This suggests that Massachusetts may have to beexcluded from our state-level study, because individual-level data with student identifierswould be critical to our matching exercise.

Massachusetts Board of Higher Education publishes data from 28 public colleges. Thereports explore the relationship between performance in HS and college among graduates ofstate high schools. For example, they look at high school GPA and credits earned in college.However, this information is based on data from postsecondary files and the files onlyinclude those who enrolled in the colleges. Unless the timeframe and access issues can besolved, we do not anticipate that the state records based studies will be possible inMassachusetts.

2lncidentally, this form includes a unique, school-assigned identification number and date of birth, but not

name or SSN.


64

Table 111.11: Summary of Items Included in the Massachusetts StatewideSchool and Student Management Information SystemSchool level information Student Level InformationEnrollment by:

Locally assigned ID, State assigned IDGrade

Name City/town of residenceGender

School ID numberRace/ethnicity

Date of birth, City town of birthFirst language

GenderLimited English Proficiency

Race/ethnicityResidence

Reason reporting; Enrollment status at time of collectionMigrant

Reason for enrollmentLow income

Grade levelSpecial education

Days in attendance in yearTransitional bilingual

Days in membership in yearAttendance

Low income status (any of eligible for free or reduced lunch,Plans of high school

family receives Transitional Aid to Families, student eligible forgraduates

food stamps)Drop out report

Perkins low income status (annual income below federal povertyEmergency immigrant

guidelines, plus indicators mentioned above)Migrant statusImmigrant status and Country of originFirst languageLEP and Bilingual Education Program statusTBE tests and scoresPostgraduate plansSpecial Education programCareer and Technical program enrollment

5. North Carolina


Eleven organizations in North Carolina administer TS programs, with a total of 8,725students (see Table 111.12 and Figure 111.5). These projects serve an average of 793participants of which an average 545 participants are both low income and potential first-generation college students. An average of 146 participants are HS seniors and an average of334 are in grades 9-11.


65

Table 111.12: Talent Search Projects in North CarolinaApproved Number of

Grantee Participants 1999 Federal GrantCentral Piedmont Community College 875 $231,903Elizabeth City State University 1,000 $263,575Fayetteville State University 600 $193,800James Sprunt Community College 600 $193,800Lees-McRae College 650 $220,860North Carolina Commission of Indian Affairs 650 $220,860North Carolina State University 800 $285,544Saint Augustine College 1,150 $309,274Southeastern Community College 850 $282,841Western Carolina University 900 $265,056Western Piedmont Community College 650 $220,860Total 8,725 $2,688,373


Ten of the 11 TS sites had returned the project director survey by June. We contactedeight of the 10 with a follow-up phone call to ask additional questions concerning data theymaintain on program participants Each site has a database of participant information,although the extent of the information varies among sites (see Appendix Table E-5).

All of these sites collect participants ' SSNs, which could be matched with SSNs fromother files. Five of the eight sites also stated that they could extract data for all students whowere juniors and seniors during the 1998-99 school year. One site does not maintain recordson students once they have completed the TS program.

c. State Record Systems

The North Carolina Student Information Management System (SIMS) (see Table 111.13)tracks all students in public K-12 schools in the state. Most students are identified by SSN,and the data are updated annually.

North Carolina is moving to a new data system called the North Carolina Window ofInformation on Student Education (NC WISE). The system will collect a wider array ofdata and allow for more functionality and support in school operations, classroom andinstructional management, and decisions support. The system is being implemented inphases and will not be completed until 2008.


66

Figure 111.5: North Carolina Talent Search ProjectsBy Number of Participants and Target Locations

North caralina Talent Search Projectsby Number of Participants and Target Locations,

Elizabeth City State University

North Carolina Commission of Indian .AffSaint Augustine Colle e

HendersonNorth Carolina State Univer

Fayetteville State UniversityWestern Carolina UniversityCentral Piedmont Community College

* North Carolina doesNC Talent Search Projects

. (0 - 758) Participants(759 - 1150) Participants

pY (1151 -2300) Partid pantsNC Target School Locations

1 target schools2 target schools

® 3 target schools_ 4 target schools00 5 or more target schoolsn Counties.shp

Southeastern Conununity CoIlege


67

Table 111.13: Summary of Items Included in the North CarolinaStudent Information SystemStudent ID (or secondary school ID)

HomeroomFull name

CounselorGender

Suspension/Drop-outRace/Ethnicity

Test date, code, and scoreDate of birth

Post-graduation plansGrade level

Course of studySchool name

College choiceAttendance/Tardiness

The state of North Carolina also tracks all students enrolled in both two- and four-yearpublic institutions of higher education. The data is updated each semester and the student'sSocial Security Number is used as an identifier. The state has been collecting data since1981, although the information has been more complete each year it has been maintained.Variables maintained in the database include demographic background, college major, andcollege GPA (see Table 111.14).

Postsecondary school majorCourse enrolled and completedGrades for courses completedUndergraduate GPAClass rankSAT Scores

D. STATE CONTEXT STUDIES

1. Rationale and Objective

As mentioned earlier in this chapter, we will conduct in-depth studies of each state in ourstate-level analysis to provide insight on our statistical findings. Gaining a detailedunderstanding of the context of each state will help explain the impact analysis results.

If our analysis finds that the impacts of TS differ among states, an in-depth study canexplore the possible reasons for the differences. Likewise, if the analysis finds similarities, anin-depth study can explore the possible reasons for that, including why the impacts aregenerally large or small.

In the end, we will not be able to offer definitive explanations for various impactfindings. Rather, our in-depth research will be limited and exploratory, yielding onlytentative conclusions. Nonetheless, the information we collect will help interpret and frame

Table 111.14: Summary of Items Included in the North CarolinaHigher Education DatabaseSocial Security NumberGender and ethnicityHigh school attended (if in North Carolina)ResidencyEnrollment statusTransfer student records


68

the impact findings by providing a picture of what was going on when the students understudy were close to completing HS.

2. Research Focus

The focus of the in-depth state studies will be on contextual factors that hypotheticallycould affect the TS program impact estimates we observe within and among states. We areconcerned with factors that would have a differential effect on TS participants andnonparticipants.

Some factors might affect outcomes in general, such as the likelihood that HS graduatesenroll in college, but would not lead to different outcomes for participants andnonparticipants. For example, state economic factors that might push students toward oraway from college would presumably affect both TS participants and nonparticipants in astate similarly. Thus, if State A has a worse economy than State B, it does not follow thatthe difference in the college-going rate of TS participants and nonparticipants in State A willbe higher than in State B. By the same logic, if State C historically has a higher college-goingrate than State D, then that traditional pattern would affect both participants andnonparticipants equally in State C and could not explain why the impact on postsecondaryenrollment in State C is greater than in State D.

Our chief focus will be on students' potential exposure to programs and services similarto TS. If students have relatively few opportunities to learn about how to prepare forcollege other than through TS, then TS participants would have a decided advantage overtheir nonparticipating peers. Conversely, if there are multiple, widespread opportunities forstudents to receive precollege services other than through TS, then TS participants wouldhave little or no advantage over their nonparticipating peers.

However, one key limitation of our study is that by relying on existing records, we willnot be able to collect individual-level data. Because we will not know the extent to whichparticipants, as compared with nonparticipants, actually received precollege services fromprograms other than TS, we will focus on students' potential exposure to such assistanceduring their HS years. Our in-depth study also will be limited to an exploration of external,observable factors. We will not be able to explore whether nonparticipants in TS in State Aare more or less motivated toward educational achievements than are their peers in State B,compared with participants in their respective states.

Different patterns of statistical impact findings have different logical implications for thein-depth study:

n If program impacts are consistently small within or across states, potentialexplanations include (1) how the program operates and (2) the extent towhich nonparticipants receive similar services. Program structure andoperations can affect impacts in several ways. For example, TS projects


69

might not be serving enough of the types of students who stand to benefitthe most from program services. Or projects might not be having enoughintensive contact with participants over a long enough time to make asubstantial difference on important outcomes. As mentioned above, anotherreason could be that the students have ready access to precollege servicesoutside of TS.

n If program impacts are consistently large within or across states, we wouldexpect to find that TS programs are serving students who stand to benefit agood deal from services and are providing a substantial amount of importantservices over a reasonable period of time. Large impacts could also beexplained, as mentioned above, by a relative dearth of similar services forstudents who do not join TS.

n If program impacts differ markedly across states, the most likely explanationis the variation in precollege service environment outside of TS; we wouldnot expect TS program operations to vary across states. We mighthypothesize that TS program impacts would be smaller in states that havewidespread college-information and assistance programs, such as in Indiana'sICPAC and Twenty-first Century Scholars Program. The existence of thoseprograms should mean that nonparticipants there are closer to participants(in their ability to get information and confidence about ability to pay forcollege) than is the case in a state without such programs.

=The way the nonparticipant comparison students are drawn for the statistical analysisalso-.has implications for our in-depth study:

n If nonparticipants come from the same schools as do participants, then wecan rule out looking for any school-level differences among states. Bothgroups would have had the same exposure to school-level precollegeassistance

n But if nonparticipants are drawn from different schools, then a host ofschool-level differences could explain differential outcomes between states.One example may be student-counselor ratios. Say in State A all the schoolshave low student-to-counselor ratios, but in State B all the schools have highstudent-to-counselor ratios. We would expect TS to have less impact in StateA than in State B, because in State A, TS services may not give studentsmuch more assistance than they could get already from school. But in StateB, TS offers a bigger advantage because other students find it harder to getindividual attention from someone who can offer advice on college.


70

3. Research Process

Gathering information on state context will be an iterative process, relying on diversepotential sources. These sources will be both quantitative and qualitative. However, we willnot conduct surveys of TS projects, schools, alternative programs, or students, so ourmethods will be limited to state education systems and economic demographic information,document reviews, and in-person and telephone interviews.

In general, we will begin the process by reviewing available documentation on each ofthe TS projects in the study, such as annual performance reports and questionnairescompleted for Phase I of the national evaluation. This will give us a basic understanding ofTS program structure and operations, especially the similarities and differences among sites.

We will also call each TS project to clarify any uncertainties and gather additionalinformation about program operations. During these calls, we will also ask about otherprograms that might be providing services similar to TS in the target area and in other partsof the state. Each time we hear about a new program, we will pursue that lead to seek moredetailed information through telephone interviews with officials at that alternative programand any literature they can send us. For each such program, we will develop short, narrativedescriptions. Key information will include targeted participants, the area served, the numberof participants, and the types of services provided. These officials will also be asked aboutother programs like theirs, and follow-up calls will be made. Thus, we will be building anexpanding set of program profiles that will give a broad picture of precollege servicesavailable in each state.

Another initial source of information on alternative precollege programs in the fourstudy states will be a database developed by the College Board, in association with TheEducational Resource Institute (1'ERI), based in Boston. Over the past year, the CollegeBoard has conducted a "National Survey of Outreach Programs" in an attempt to build aweb-based, searchable directory of programs "whose goal is increasing access to and successin college for educationally or economically disadvantaged elementary and secondary schoolstudents." In addition to federal programs such as TS, Upward Bound, and GEAR-UP, stateand community-based programs were encouraged to access the College Board's web site andcomplete the questionnaire, which contained sections on the area served, program goals andservices, operations and staffing, budget, student characteristics, program needs, andprogram outcomes.

As for timing, the in-depth studies will take place, for the most part, while impact datacollection and analysis are still under way. The proposed time frame for the state-level study,given the amount of time we expect to spend collecting and analyzing quantitative data, istoo short to allow us to postpone the in-depth research component until the statisticalanalysis is complete. We will be looking for relevant contextual information withoutknowing in advance what types of findings we will be trying to explain. This will requirecasting a wide net. For example, we may not know whether nonparticipants in a given statewill be drawn from the same schools that TS participants attended or different schools, thus


71

requiring that we consider investigating a host of school-level differences to explaindifferential outcomes among states. Another complication of this work is that we will haveto focus on the context that existed at the time our sample of individuals were completingHS and preparing for college--that is, during the late 1990s--not on what is happening now.

Although a good deal of information can be gathered through phone calls and documentreviews, inevitably it will be necessary and advisable to visit each state, to see up-close andin-person what is going on that could affect the statistical impact results. All four statesmost likely to be included in our study (Florida, Indiana, North Carolina, Texas) are so largethat it obviously will be impossible--within reasonable budget/time constraints--to visit all ofthe TS projects (let alone all their target schools), all the other schools from whichcomparison students may be drawn, and all the alternative precollege programs in whichstudents may be participating. Therefore, the only practical choice is to visit a small sampleof places (perhaps two or three cities) in each state.

Our choice of sites to visit will depend a great deal on what we learn from the earlierphone calls, etc. The point will be to visit places that can give us detailed insight on therange of student exposure to precollege services. We will choose locations where we canlearn' about more than one major program in addition to TS. While on site, we will conductinterviews and observe programs in- action. Where relevant, we will meet with state programofficials in the capital city to discuss state-wide programs, such as the ICPAC and Twenty-first Century Scholars programs in Indiana, and collect state-level information on the publicHS system.

Once the basic field work is done, a core team of researchers will sift through andanalyze the information on each state.

E. REPORT AND BRIEFING

We will prepare a single draft report covering all of the impact estimates for review bythe Department and a final report incorporating its comments. We will also briefappropriate PES staff in-person regarding the study's key findings. The report will beorganized as shown in Figure 111.1.


72

Figure 111.6: Outline of the State-Level Impact Report

Toward the end of Phase II, when PES has reviewed the findings of both the state andnational studies, we will, at the discretion of PES, brief the Council for Opportunity inEducation (COE) and project directors on the study's key findings.

I. Executive Summary

II. Introduction

A. Policy Background

B. Purpose of the Study

C. Policy and Research Question

D. Limitation of the Study

Ill.

State TS Impacts

Course Selection

HS Graduation

C. SAT/ACT

D. Application for FFA

E. PSE Enrollment

IV. What the Findings Mean for TS

A. How the Impacts Should Be Interpreted

B. Programmatic Implications

V.

Methodology

A. Experimental Design

B. Data

C. Analysis

VI. Summary and Conclusions

A.

B.


CHAPTER IV

PHASE II NATIONAL-LEVEL IMPACTSTUDY DESIGN

This chapter discusses the feasibility and design of a national impact study to beperformed in Phase II. The national study is, by its nature, more representative ofthe impact of TS throughout the entire nation. On the other hand, withoutprimary survey data, the data limitations of the national study cause it to be a less

powerful statistical tool for detecting TS impacts.

The first section presents the policy and research questions addressed in this study. Theremainder of the chapter reviews four alternative study designs, concluding that one of thedesigns is feasible (the aggregate school district approach) and the rest are not. The secondsection presents the feasible approach, and the third section outlines the remainingapproaches and why they are not feasible.

A. POLICY QUESTIONS

The national impact study addresses the same policy question that the state impactstudies does: What is the impact of Talent Search (TS) relative to a world without TS butwith other programs that provide similar or related services?

The national study differs from the state studies in that it provides an impact estimatethat represents all projects in the nation, whereas the state studies cannot be generalized tothe entire population of TS projects. The national study is similar to the state studies in thatit relies on records data and does not involve any new surveys.

The national study differs from the service-mix demonstration, described in the nextchapter, in four ways. First, it estimates the impact of operating the TS program versus notoperating the TS program, whereas the demonstration estimates the impact of one service

74

mix relative to another service mix. Second, it uses records data, whereas the demonstrationuses a mix of surveys and records data. Third, it uses statistical matching for itsexperimental design, while the demonstration uses random assignment. Fourth, it has anarrower array of outcome variables.

B. AGGREGATE SCHOOL DISTRICT APPROACH

In this approach, the unit of analysis is the school district, as opposed to the student,school, or project. The outcomes are the proportion of graduating HS seniors in the schooldistrict who apply for Federal Financial Aid (FFA) and the proportion who receive financialaid. The impact of TS is measured by comparing districts that have a substantial TSpresence to similar districts that do not.

This approach is deemed feasible because it provides national impact estimates, mayhave adequate statistical precision, and is affordable. The rest of this section discusses theexperimental design, propensity model, database construction plan, sampling plan, andanalysis plan.

1. Experimental Design

The impact of TS is conceptually measured by comparing school districts with a TS

presence to those same districts if they did not have any TS presence. Ideally, one wouldsimulate that conceptual experiment using random assignment of districts to a TS treatmentgroup or a non-TS comparison group. Since random assignment of school districts to TSprojects is not feasible, we compare, in the simplest sense, each TS district to a similardistrict with no TS presence.

More realistically, we will categorize districts by the amount of TS penetration — none,small, medium, large relative to eligible participants. We will examine the extent to whichcollege outcomes differ from those expected for districts with Talent Search programs.

2. Propensity Model

The success of this study hinges on our ability to identify similar school districts. Weidentify similar non-TS school districts using a propensity model. A propensity model isdesigned to identify school districts that are most similar to TS districts but do not have TSservices do not have any TS target schools. The model's dependent variable is the extent ofTS presence relative to eligible participants whether the district has any TS schools. Theindependent variables are district characteristics associated with the propensity of containingTS schools, such as district size, proximity to an urban center percentage of students eligiblefor free and reduced lunch, percentage of students who are racial/ethnic minorities, dropoutrate, and state or region, all of which are available from the Common Core of Data (CCD).

IV. Phase II National-Level Impact Study Design

75

We may also include variables representing state or metropolitan poverty rates andunemployment rates. The model will be estimated using a logistic or probit regressiontechnique.

Once the propensity model is estimated, we will assess the quality of the match. Theconcern is that non-TS districts may not be similar to TS districts. We will then use it topredict a propensity score for both the TS and non-TS school districts. An individual schooldistrict's specific characteristics will be substituted into the estimated propensity model andmultiplied by the estimated coefficients. The sum of the resulting products is the predictedpropensity to be, or likelihood of being, a TS district.

The final step is to match each TS district with a similar non-TS district. Similar is nowdefined to be the non-TS district that has the propensity score closest to that of the TSdistrict.

3. Database Construction

Beginning with the FFA database, we will geo-code the addresses of FFA applicants andthose of Pell grantees. We are already geo-coding school district boundaries. From thesetwo sets of data, we will link each person in the FFA database to her school district.

Once FFA data are linked to school districts, we will link TS projects to districts. Ourrecent survey of project directors recorded the number of TS participants by target school.We will use the CCD to link TS target high schools to school districts and will assume allother high schools have no TS presence. We will define TS presence over one or two years,rather than at a point in time. This accounts for schools that recently transitioned frombeing a target school to being a non-TS school.

The outcome variable is the proportion of graduating seniors who submit a FAFSAform. We compute the numerator of the proportion for an individual district by countingthe number of FFA records linked with the district. The denominator of the proportion isthe number of HS completers, which we will extract from the CCD.

4. Sampling

This study will use all districts in the nation, without sampling. Impact estimates willhave measurement errors and modeling errors, but they will not have sampling error.

5. Analysis

We will subtract the proportion of graduating seniors who file a FAFSA form for thenon-TS district from that of the TS district. The impact estimate is the weighted average of


76

all these differences, where the weight is a measure of the size of the pair of districts. Wewill estimate the TS impact by year for the past two or three years.

If the propensity models and the magnitudes of the impact estimates are similar acrossyears, we will pool the data for the several years, perhaps add year variables to the propensitymodel, and estimate the impact of TS with the pooled data. The purpose of such pooling isto produce a more precise impact estimate.

We will compute the impact for each outcome variable. As outlined in the chapter onstate studies, we will estimate the precision of the impact estimate using replicate sampletechniques. We will draw several replicate samples of treatment districts with replacement,match each replicate sample with comparison districts using the estimated propensity model,and compute an impact estimate for each replicate. Then we will calculate the variance ofthe distribution of impact estimates. Finally, we will use the estimated variance to test thehypothesis that the initial estimated impact is significantly different from zero.

C. ALTERNATIVE APPROACHES INVESTIGATED

Before proposing the approach -described above, we considered the use of several otherapproaches. We note their strengths and weaknesses below.

1. NELS 88/1994

The National Educational Longitudinal Study (NELS) has the advantage of recordingboth participation in TS and enrollment in a postsecondary institution. This would enable usto compare TS participants to nonparticipants nationally. Unfortunately, the NELS 88/1994sample appears to be too small for estimating the impact of TS.

If there were no turnover among program participants, we estimate we will find about 25TS schools and about 168 TS high school participants in the NELS database. Combiningthese figures, and assuming we use an unbalanced design with a comparison group that isfour times as large as the treatment group, yields a minimum detectable difference (MDD) ofover 19 percentage points. This means that TS would have to increase postsecondaryenrollment rates by over 19 percentage points before one could detect the impact fromNELS data.

It is not realistic to assume there was no turnover among TS participants. Doubling thesizes of the treatment and comparison groups to account for such turnover yields an MDDof 13.5 percentage points. This means that if TS increases the postsecondary enrollment ratefrom 55 percent to 65 percent, we would not be able to detect the impact using NELS data.Thus, the NELS sample size is not sufficient to support impact estimates that are preciseenough for our purpose. Previous research (Myles 1993) using the databases NELS has alsonoted problems with student self rights of participation.


77

2. Geo-Coding Individual FFA Records

This approach involves linking FFA applications to the HS from which the studentgraduated by geo-coding the student's address. The unit of observation is the HS. Theconceptual experiment is to compare that proportion for TS schools to what that proportionwould have been without TS.

The outcome variable is the proportion of seniors completing HS who submit a FAFSAform. The denominator of this proportion is recorded in the CCD. The numerator of theproportion is problematic because the FFA data do not identify the student's HS. Thismeans that we must find a link between the student in the financial aid database and the listof high schools.

We have tested a geo-coding approach to such linkage. The test used a sample ofUpward Bound students in Los Angeles whose addresses and high schools were obtainedfrom survey data. We geo-coded the address of each student, in a sense converting theaddress into a latitude and longitude. Then we geo-coded the addresses of schools in thearea. The next step was to link each student to the school she was nearest to. Finally, wecompared the linked school to the school the student actually attends.

The result was a 45 percent success rate. This means that over half of the students werelinked to the wrong school. This inaccuracy would certainly make the impact estimateimprecise and may introduce a bias toward zero into the estimate. The lack of precisionmeans that the impact model will be less likely to detect the true impact of TS. Even thoughTS has a positive impact, the model will find the estimated impact to be virtually zero.

The possibility of bias may be seen by recognizing that many mismatches are caused byseveral factors other than the fact that HS catchment area boundaries are not always drawnto minimize the distance between the student's residence and the school. These factorsinclude the following:

n The student attends a private or a magnet school.

n The nearest HS is a type of school that the student does not wish to go to,such as a vocational or technical school.

n The district has adopted a policy of school choice, and the student attends aschool outside her neighborhood.

n The student is bussed to school outside the community for purposes ofachieving racial balance.


78

Each of these factors results in students associating themselves to high schools in asystematic, or nonrandom, way. The only way to ensure that such a systematic sorting ofstudents does not bias the TS impact estimates would be to estimate a comprehensive modelof that sorting process, which would be beyond the scope of this study.

3. Lists of HS Completers from School Districts

This approach is similar to geo-coding except that it uses lists of graduating seniorsprovided by school districts, rather than geo-coding, to link financial aid data to schools.The approach has two advantages. First, it generates a sample large enough to detectrealistic impacts of the TS program. Second, it avoids the measurement error problemassociated with geo-coding. However, it costs more than the other approaches discussed inthis chapter because it requires collecting data from school districts.

The approach involves selecting a sample of school districts, requesting a list of allgraduating seniors for the previous several years, and matching the lists to the FFA database.If we ask for the past three years of HS completers and use a comparison group four timesthe size of the treatment group, we estimate the MDD for 100 districts would be 8.4percentage points, for 200 districts would be 6.0 points, and for 300 districts would be 4.9points.

Thus, to achieve a customary MDD, this approach requires collecting data from over ahundred school districts and would cost several hundred thousand dollars, substantiallymore than any other approach considered in this chapter.

D. SUMMARY

The school district approach to measuring national TS impacts is the most feasible of theother approaches considered in this chapter. It is not limited to a small sample of TSschools and participants, as is the NELS approach. It avoids the substantial measurementerror in linking individual students to schools in the geo-coding approach. And it can becompleted relatively quickly and inexpensively, unlike the student list approach.

'While a national impact study based on school districts is logistically feasible, it is not asstatistically powerful as the state impact studies or the service mix demonstration outlined inChapter V, primarily because it uses the aggregate school district as the unit of analysis. Thismeans that, if the true impact of TS is small, the national impact is less likely to detect thetrue impact than are the state studies, which use the individual student as the unit of analysis.The national study is more likely to generate an impact estimate that is statisticallyinsignificant than are the state studies. We assess this risk to be unacceptably large. Thismeans that, while the national study is logistically feasible, it is ultimately deemed not to befeasible because the risk of not detecting the true impact of TS is too great.


79

E. REPORT AND BRIEFING

We will prepare a draft report on the national impact estimate for review by theDepartment, and a final report incorporating its comments. We will also brief appropriatePES staff in-person regarding the key finding of the study. The report will be organized asshown in Figure IV.1


80

Figure IV.1: Outline of the National Impact Report

Executive Summary

II. Introduction

A. Policy Background

B. Purpose of the Study

C. Policy and Research Question

D. Limitation of the Study

E. How This Study Relates to the State Impact Studies

III. The National TS Impact

A. Impact on HS Graduation

B. Impact on Application for FFA

C. Impact on PSE Enrollment

IV. What the Findings Mean for TS

A. How the Impacts Should be Interpreted

B. Programmatic Implications

C. How the National Impacts Relate to the State Impacts

V. Methodology

A. Experimental Design

B. Data

C. Analysis

VI. Summary and Conclusions


CHAPTER V

PHASE III SERVICE MIX DEMONSTRATIONEVALUATION DESIGN

This chapter investigates the feasibility of a service mix impact demonstration andpresents a preliminary design for it. This type of study enables PES to measurethe impacts of several service mix enhancements, as opposed to measuring theimpact of the current TS program as a whole. This would occur in Phase III of

the TS evaluation. This chapter's first section presents the purpose of the demonstrationand how it relates to the Phase II evaluations. The second section lists the alternative servicemixes to be tested. The third section presents the experimental design, demonstrationtimeline, sample design, and data collection plan.

A. PURPOSE OF THE STUDY AND How IT RELATES TO PHASE II

This study will identify the mix of services provided to 11 th and 12th grade participantsthat are most effective in achieving the goals of TS. We take the goals of TS to be increasingthe likelihood of taking college preparation courses in HS, taking the SAT or the ACT,graduating from HS, applying for and receiving FFA, and enrolling in a PSE.

By mix of services we mean whether a service that supplements the standard TS services isoffered by the project and received by participants. Standard and supplemental services aredefined below-. By effective we mean the impact of the supplemental service relative to thestandard service mix in achieving TS goals. This is an assessment of service mix impacts, asopposed to service mix outcomes. The impact of the supplemental service is the increasedlikelihood that a TS participant will achieve each of the successful outcomes listed above,relative to a similar participant who received only the standard services.

Thus, the service mix study measures the incremental effectiveness of the supplementalservice relative to the effectiveness of the basic TS program. The study does not estimate

82

the impact of the TS program. In this sense, the research objective of the service mix studydiffers from that of the Phase II evaluations, which seek to estimate the impact of the TSprogram as a whole.

This same distinction may be stated in program design and policy terms. The servicemix study's programmatic and policy objective is to inform the decisions of project directorsregarding the mix and timing of the services they offer. The service mix study will notinform the Department or the Congress regarding the effectiveness of TS relative to that ofother programs in which public funds could be invested.

B. TESTED SUPPLEMENTAL SERVICE MIXES

In this section, we suggest two supplemental service mixes to test. We arrived at thesesuggestions by reviewing the literature on TS-like programs and the results of the survey ofproject directors, while considering the logistical difficulties of random assignment.

These are suggestions in the sense that we plan to propose these to the Department,Council on Opportunity in Education (COE), and the TRIO regional associations such asthe Southeastern Association of Educational Opportunity Program Personnel (SAEOPP).We will discuss with each organization the programmatic interest in the suggested services,as well as the logistics of applying random assignment to them. Upon consideration of theircomments, we will finalize the experimental treatments.

We define standard services to be the services currently offered by TS projects. We will testsupplemental intensive mentoring/tutoring and the providing of information services toeligible nonparticipants.

Mentoring/Tutoring

While many projects provide both mentoring and tutoring, those services may beconsidered supplemental because we define mentoring/tutoring differently than mostprojects. By mentoring/tutoring we mean life counseling and academic assistance provided ona proactive one-on-one basis over an extended period.

Life counseling means discussing and helping the youth with any barriers or problemsshe faces in achieving the desired TS outcomes of HS graduation, application for financialaid, and PSE enrollment. The range of barriers may be very broad, including problems suchas mental health, substance abuse, poverty, criminal activity, an abusive caretaker, and gangpresence in the neighborhood. No barrier is outside the scope of the mentor's role. Thementor's responsibility is to assist the participant directly or refer her to an organization thatcan address the barrier.

V. Phase III Service Mix Demonstration Evaluation Design

83

Academic assistance means help with course homework; remedial instruction in reading,writing, and math; monitoring the participant's performance in school; and acting as thestudent's advocate with respect to school administration when necessary.

The key components of the definition are proactive, one-on-one, and extended. Proactivemeans that the mentor/tutor would actively persuade a TS participant to accept theseservices. The mentor/tutor would call and meet with the student to "sell" the services, andwould persevere even if the participant initially declined to accept the services. One-on-onemeans the services are tailored to the individual's particular needs and are delivered in a one-on-one setting, as opposed to a group setting. Workshops and other group activities are notper se part of this supplemental service. Extended means the one-on-one relationshipcontinues with the same mentor/tutor as long as possible throughout the participant's 11 `hand 12th grade years. A series of one-on-one activities each with a different mentor/tutordoes not count as part of the supplemental service.

We recommend the mentoring/tutoring service for two reasons. First, in our informalsurvey of TS project directors on what issues would be important to study in this TSevaluation, testing the efficacy of mentoring and tutoring was mentioned more often thanany other single issue.

Second, policy analysts and social scientists in several disciplines believementoring/tutoring to be an important component of programs targeting disadvantagedyouth. Such disciplines include secondary and postsecondary education, juvenile justice,employment and training for youth, welfare, and adolescent psychotherapy. Both Callahan(1999) and Levine and Nediffer (1996) conclude that a one-on-one mentoring/tutoringrelationship is the most effective service a program can offer for increasing the odds of adisadvantaged youth graduating and going to college. Mincy (1994) argues that mentoring isparticularly essential for young African-American males.

Public/Private Ventures (P/PV) has evaluated a number of mentoring programs. In itsmost visible evaluation, the Big Brothers/Big Sisters program, P/PV found that mentoringhas a significant impact on a wide variety of positive outcomes. The Employment andTraining Administration of the U.S. Department of Labor, the Office of Juvenile Justice andDelinquency Prevention of the U.S. Department of Justice, and The Ford Foundationteamed up to conduct the Quantum Opportunity Project (QOP) demonstration, the resultsof which are not yet available. The cornerstone of QOP is mentoring and tutoring.Gulliotta, Adams, and Montemayor (1998) and Durlack (1995) adopt a psychoanalyticalapproach to stress the importance of mentoring for violent delinquent youth. Other recentstudies find that mentoring and tutoring are the most important components of programsfor this target group include Sum, et al. (1997), Barnow and King (2000) and the YouthPolicy Forum (1997 and 1999).

In sum, mentoring and tutoring represent the state of the art for promoting positiveoutcomes for, and preventing risky behaviors of, disadvantaged youth.


84

Information services to eligible nonparticipants.

The second supplemental service to be tested in the demonstration is the providing ofinformation services to eligible nonparticipants. This involves providing information,perhaps by mail followed by a phone call, on applying to college and financial aid to youthwho are eligible for, but do not formally participate in, TS. This treatment does not involvealtering the project's academic eligibility rules or testing alternative target groups. Rather, ittreats eligible nonparticipants as a second type of project participant receiving a low level ofservice.

The informational service for eligible nonparticipants is motivated by a concern that theparticipation rate among students eligible for TS is low, between 4 percent and 25 percent,depending on one's turnover assumptions. Some project directors indicated a desire toincrease the program's overall effectiveness by viewing eligible nonparticipants asparticipants in a hypothetical "TS-lite" program. TS-lite services would be very inexpensiveand would be delivered to a large number of students.

The policy and research question is whether a fixed budget allocated to less intensiveservices that are delivered to large numbers of students is more or less effective than thesame budget allocated to more intensive services delivered to smaller numbers of students.

C. EXPERIMENTAL DESIGN

The key feature of the service mix demonstration is random assignment of TSparticipants and eligible nonparticipants in the 11 `h and 12`'' grades to alternative servicemixes. Table V.1 lists the control groups and treatment groups required to measure theimpact of the suggested supplemental services.

Table V.1: Experimental Design of the Plan Variation StudyTreatment/Control

Group Population Services

Control Group 1 TS

participants

in

the Standard servicestarget group

Treatment Group 1 TS

participants

in

the Standard plustarget group mentoring/tutoring

Control Group 2 Eligible nonparticipants None

Treatment Group 2 Eligible nonparticipants Information on PSE andFFA

Treatment Group 1 is compared with Control Group 1. Treatment Group 2 iscompared with Control Group 2. The outcomes are HS graduation, enrollment in PSE, andapplication for FFA.


85

The population eligible for random assignment to Control Group 1 and TreatmentGroup 1 is all 11`' and 12 `h grade TS participants in the mentoring/tutoring target group.We suggest a target group within the population of TS participants, rather than using theentire population of TS participants, for two reasons. First, we believe that not all TSparticipants need individualized mentoring/tutoring. Second, individualized services arerelatively expensive, resulting in projects having to ration such services to those who needthem most.

We suggest a target group of TS participants whose GPA in the previous academic yearis less than 3.0. The disadvantage of such a simple rule is that there may be participants withgood grades who are experiencing a life crisis and would benefit substantially from suchindividualized services. The advantage of such a rule is that it is objective, that is, it cannotbe affected by the subjective judgments of TS staff who may have an interest in thedemonstration's outcome. It is easy to measure. If possible, we will consult with projectdirectors about how best to define the target group, although all sites must use the samedefinition.

The population eligible for random assignment to Control Group 2 or Treatment Group2 is all 11 th and 12th graders who are eligible for but not participating in TS at the beginningof the school year in the high schools associated with participating projects.

D. TIMELINE

The timeline for the demonstration, illustrated in Figure V.2, is complex because we areproposing two cycles of the demonstration, and each cycle has two grade cohorts.

Selection into the research sample. We will select the members of the researchsample from lists of 11 `h and 12th grade TS participants and eligible nonparticipants at eachHS in the study. This will be done twice, once near the end of academic year 1 for bothgrade cohorts of cycle 1 and again near the end of year 2 for both grade cohorts of cycle 2.In each case, this is the spring before the beginning of service delivery in the fall.

Baseline survey. Collecting baseline data on research sample members is useful forimproving the statistical efficiency of our impact estimates. For the eligible nonparticipantdemonstration, it is also necessary for determining who is eligible for TS. The baselinesurvey is administered immediately after selection into the research sample at the end of year1 for cycle 1 and year 2 for cycle 2.

Random assignment. We will randomly assign each member of the research sample toa treatment or control group at the beginning of the academic year following the completionof the baseline survey. For cycle 1, this is the beginning of year 2, and for cycle 2 it is thebeginning of year 3.


86

Service delivery. Projects begin providing the enhanced service to treatment groupsimmediately after random assignment. Both 12 th grade cohorts receive enhanced services forup to one year, and both 11`h grade cohorts receive enhanced services for up to two years.

Outcome data collection. The follow-up survey is administered in the middle of thestudent's freshman year of college. This is year 3 for the 12 th grade cohort of cycle 1, year 4

for the 11 th grade cohort of cycle 1 and the 12 th grade cohort of cycle 2, and year 5 for the11`" grade cohort of cycle 2.

Analysis and report. In order to be cost-effective, the analysis will be conducted andthe report will be preparedonly once, covering both cycles, both cohorts of each cycle, andboth the participant and eligible nonparticipant demonstrations. The analysis will beconducted at the end of year 5 and beginning of year 6, and the report will be prepared inyear 6.


87

Figure V.1: Demonstration Timeline

Academic YearActivity Year 1 Year 2 Year 3 Year 4 Year 5 Year 6

Cycle 112th Grade Cohort

Sample SelectionBaselineRandom AssignmentService Delivery _Follow-up

11th Grade CohortSample SelectionBaselineRandom AssignmentService Delivery _Follow-up

Cycle 212th Grade Cohort

Sample SelectionBaselineRandom AssignmentService Delivery ♦_Follow-up

11th Grade CohortSample Selection ♦_BaselineRandom AssignmentService DeliveryFollow-up

AnalysisReport and Briefing

E. SAMPLE DESIGN AND RANDOM ASSIGNMENT

We propose conducting the demonstration in the states with which we conduct Phase IIstudies, assumed in this sample design to be four states. This approach has the disadvantageof not producing impact estimates that are externally valid, that is, the estimated impactscannot be generalized to the national population of TS projects. However, it has a practicaladvantage. By the time we complete the Phase II studies, we will have developedpartnerships with the relevant state agencies and will be familiar with their databases. We willbe using state databases for some of the demonstration outcome variables.


88

Within each state, we will invite all TS projects, assumed to be eight per state, toparticipate in the demonstration. We will attempt to recruit projects that allow randomassignment of participants and eligible nonparticipants to alternative service mixes. We

anticipate that half of the project directors in each state will accept our invitation.

Ideally, we would like to randomly assign participating and eligible nonparticipatingyouth to the treatment and control groups from the population of eligible students withinthe school. This procedure means that some participants and eligible nonparticipants ineach school will receive the tested service and some will not. Randomizing within the schoolallows the treatment students and control students to experience the same schoolenvironment. Drawing the control group and the treatment group from different schoolscould result in biased impact estimates because the school environment's impact could

erroneously be attributed to TS.

If project directors indicate that within-school randomization is not feasible, we willdraw the control group from statistically matched target schools, presumably within thesame TS project. Matching will be based on similarity of standard TS services, proportion ofstudents eligible for TS in the population of 11 `h and 12th graders, similarity of otherprograms offering TS-like services, and similarity of neighborhood characteristics. We willconsult with the project director to obtain this information.

Table V.3 shows the minimum detectable differences (MDD) for three participantsample designs as well as MDDs for three alternative eligible nonparticipant sample designs.All samples are designed on the assumption that the control group and the treatment groupare drawn from the same school.


89

Table V.3: Minimum Detectable Differences (MDD)

SubgroupTotal

SampleSchools Treatments Controls Size MDD Size Notes

160 14 14 1.00 0.060 4480 Participant Demo120 11 11 1.00 0.073 264060 22 22 1.00 0.091 2640

60 94 94 1.00 0.080 11280

EligibleNonparticipantDemo

24 75 75 1.00 0.128 360060 30 30 1.00 0.087 3600

Table notes:

Intraclass correlation (schools) =

Intraclass correlation (school districts) =

Probability of rejecting hypothesis of no impact by chance alone =

Probability of rejecting hypothesis of no impact when there is an impact =

Sample attrition =

Proportion of variation accounted for by baseline measures =

The top panel in Table V.2 presents the sample size and MDDs for the participantdemonstration. These figures are based on the following assumptions about the participatingTS projects. We are limited to such approximations because we do not yet know whichstates we will be working with.

n We conduct the demonstration in four states.

n Each state has eight TS projects, half of which agree to participate in thedemonstration. This means that 16 projects participate in the demonstration.

n Each project has 14 schools, 10 of which are high schools. This means thereare 160 high schools.

n Each project has 800 participants, 35 percent of whom are in the 11`h or 12`''grades. This means that each project has an average of 280 participants inthose grades, all participating projects taken together have 4,480 participantsin those grades, and each school has 28 participants in those grades onaverage.

0.05

0.00

0.05

0.8

20%

0.1


90

The MDDs for the participant demonstration presented in Table V.2 are also based onthe following statistical assumptions:

n The experimental design is balanced, that is, participants in a school aredivided equally among the control and two treatment groups.

n The outcome variable is measured as a percentage, such as the percentagewho enrolled in college.

n The sample is a two-stage clustered sample in which the first stage issampling schools and the second stage is sampling participants.

n Significance and power are set at the conventional standards of 95 percentand 80 percent, respectively.

n Twenty percent of the initial sample leaves the sample during thedemonstration.

n The correlation between the values of the outcome variable among studentswithin a single school is 5 percent.

n The proportion of the variance in the outcome variable that is explained bybaseline data is 10 percent.

The first row of the top panel indicates that if every participant in every school joins thedemonstration research sample, the MDD for each of the two tested service mixes is 6.0percentage points. The column labeled Total Sample Size presents the size of the researchsample including both treatment and control students for all the schools, or 4,480 for thisfirst design.

The second row assumes that 25 percent of the schools and 21 percent of the studentswithin each school choose not to participate in the demonstration. This results in an MDDof 7.3 percentage points, and a total sample size of 2,640.

The third and final row of the top panel recognizes that working with 120 schools maynot be logistically feasible. We propose an alternative strategy of recruiting half this number,or 60 schools, and drawing two cohorts, 11 and 12` h grade TS participants in the first yearof the demonstration, and the 11`h and 12`h grade TS participants in the second year of thedemonstration. In other words, the demonstration involves half the number of schools, butwe select twice as many students from each school, and the demonstration lasts nearly twiceas long. This results in an MDD of 9.1 percentage points, with a total sample of 2,640students. This last sample design is the most logistically feasible of the three alternatives andis the plan we recommend.


91

The bottom panel of Table V.2 presents a different story with respect to the eligiblenonparticipant demonstration, primarily because there are likely to be many more eligiblenonparticipants in the schools than TS participants. While we do not have precise data onthe TS participation rate-- the ratio of participants to eligible students-- we believe it isbetween 4 percent and 25 percent, depending on the rate of turnover among participants(MPR, December 1998). For this analysis, we have assumed a 15 percent participation rate,indicating that there are 6.7 times as many eligible nonparticipants than there are participantsin each school, or 188 eligible nonparticipants in each school.

If we allocate the eligible nonparticipants in all 60 schools taking part in the participantdemonstration equally to the control group and the treatment group, the first row of thebottom panel shows an MDD of 8.0 percentage points and a total sample of 11,280 youth.However, the cost of a demonstration with a sample of that size is likely to be prohibitive.The second and third rows of the lower panel present two alternative strategies that reducethe sample size to 3,600 students. The second row assumes that 20 percent of the eligiblenonparticipants choose not to join the research sample, and that we draw the sample from24 schools. This results in an MDD of 12.8 percentage points.

The third row of the lower panel shows a strategy in which we draw a random sample of30 eligible nonparticipants from each of the 60 schools. The resulting MDD is 8.7percentage points, with a sample size of 3,600. We recommend this strategy because theresulting impact estimates are substantially more precise than the strategy involving fewerschools and more students at each school.

F. DATA ACQUISITION PLAN

This section outlines the sources of each type of data needed for the service mixdemonstration, summarized in Table V.4.


92

Table V.4: Service Mix Demonstration Data Collection Plan Type of Data Source/Mode Typesof VariablesBaseline

Telephone survey

Demographics, income,attitudes toward school,educational expectationsIntensity of received services:hours of participation in thebasic and supplementalservicesOutcomes: graduation, GPA,courses takenOutcome: postsecondaryenrollmentOutcome: federal financial aid

Outcomes: self-reportedpostsecondary attendance,receipt of financial aid,importance of TS, success inpostsecondaryeducation

n Sample selection. Treatment and control TS participants are drawn fromthe project MIS, and eligible nonparticipants are drawn from school districtrecords. We will have to approximate eligibility for the eligiblenonparticipant sample using data on grades and eligibility for free andreduced lunch.

n Baseline. Baseline will be collected via a telephone survey of all researchsample members, that is, members of all the control and treatment groups.

n Service Mix Participation. We will extract from the participating projects'MIS records variables pertaining to the amount of each type of servicesreceived by each demonstration sample member. These variables are neededto measure the intensity of the services received.

n High School Graduation. We will request transcripts of sample membersfrom participating high schools. The transcripts will show whether thesample member graduated on time, the grade point average, and the types ofcourses taken.

n Postsecondary enrollment. Each state agency participating in thedemonstration operates a statewide higher education database. We or thestate agency will match the demonstration research sample with thatdatabase. Variables pertaining to enrollment will be extracted from thedatabase.

Service Mix

Project MIS recordsParticipation

High SchoolGraduationPostsecondaryEnrollmentFederal Financial Aid

Follow-up (optional)

HS transcript

State highereducation databaseFAFSA and Pell grantdatabasesTelephone survey


93

n Federal Financial Aid. We will match research sample members to theFFA database and retrieve variables indicating application for and receipt offederal financial aid.

n Follow-up. We will interview each sample member by telephone to collectself-reported data on postsecondary attendance, success in the postsecondarysetting, receipt of financial aid, and importance of TS in her path to college.This survey is optional because we have records-based sources of keyoutcome variables. A survey has the advantage of supplying informationabout enrollment in colleges outside the home state, receipt of nonfederalfinancial aid, and the youth's subjective assessment of the importance of TS.

G. IMPLEMENTATION STUDY

We suggest an implementation study as a component of the Phase III demonstration.The implementation study has three purposes. The first is to measure how thesupplementary services were implemented on the ground, which will help us interpret theimpact -findings. For example, if a supplemental service is delivered in a half-hearted andhaphazard way, we might use that information to interpret a finding of small impacts. If theimpact varies substantially by site; we might examine differences in how the service wasimplemented across sites as an explanation.

The second purpose is to monitor the project and school activities that pertain to theexperimental design's integrity. Monitoring includes checking the quality of the lists theyprovide for drawing the research sample, checking that they offer the supplementary serviceto every treatment student and not to any control student, and checking that the servicesoffered to treatment students are offered with the appropriate amount of aggressiveoutreach.

A third purpose of the implementation study is to obtain the opinions of treatmentstudents, participating target schools' administrations, and project staff about thesupplementary service's effectiveness. Such individuals' feelings and intuitions providevaluable lessons for future implementations of the supplementary service, as well as guidesto understanding the impact study's results.

The implementation study will consist of the following activities:

n Measuring the implementation of the supplemental service

Review the process leading to a mentoring/tutoring session.

- Observe mentoring/tutoring sessions.

- Review MIS data on the number of mentoring/tutoring sessions eachtreatment student receives.


94

Review the procedure used to generate the mailing list of eligiblenonparticipants.

- Review the information packets mailed to eligible nonparticipants.

Determine how many packets are returned by the post office asundeliverable.

n Monitoring demonstration integrity

- Check the quality of the lists provided by projects and school districtsfor drawing the research sample.

- Checking for crossovers, that is, treatment students who are notoffered the supplementary service and control students who are offeredthe supplementary service.

- Checking that the services offered to treatment students are offeredwith the appropriate amount of aggressive outreach.

- If control students are drawn from comparison schools rather thantarget schools, checking the similarity of school policy andmanagement culture, the standard TS services, and students'characteristics, relative to those of the matched school from which thetreatment sample was drawn.

n Lessons for future implementations

Executive interviews with project management and staff, and involvedschool personnel.

- Focus groups with treatment students.


CHAPTER VI

ESTIMATES OF COST AND TASK SCHEDULEFOR PROPOSED PHASE II AND

PHASE III STUDIES

This chapter presents a task schedule and estimated costs associated with Phase II andPhase III of the Talent Search Project.

The task schedule and costs estimates are for the following activities:

n Records based impact studies in 4 states (Phase II-Oct 2000 January 31-2002)

n A study of national impacts using records data (Phase II-Oct 2000 January31-2002)

n A Phase III of the evaluation that includes demonstrations of alternativeservices (Phase III January 1-2001-October 2006)

Figure N.1 is a listing of tasks and time schedules for the proposed Phase II and PhaseIII studies.

96

Figure VIA: Listing of Tasks and Time Schedules

ID Task Name1

Talent Search Phase II

6 Analysis

Thu 2i1/071 --"Thu 6131101

Refine design

N-1 National estimates

4. Report

Tue 5115101

2. File work

Tue 10131100

lion iAik

Start FinishFri 911100

Fri 12/28101

Mon 9/4tOO Tue i0/3-1/66-

Wed OR

_Fri'ii/30101

Fri 11/30101

HI HI[ H1 1H2 '{

Thu 1131162

w

11

12

13

14

15

16

_State studies

1. Refine

ign and access

Project records

3. State records

4. Process systems studies

5. Propensity and analysis

6. Report

Draft report

Final report

Mon 1012100

Mon 1

^m1ouxm----b^1

0o'

Mon 10/2/001

Thu 3115101

.11

5"01

Mon 1012100

Tue

Mon 1115101

Fri 6/29101

Mon 10116100

Thu 1131102

Mon 1011TC1

Mon 1/15/01

_^^^'

Mon 10/16100

m

7. Management

Mon 10/2100

Thu 1131/02

19 Talent Search Phase III Mon 1115101

Mon 1115/01

_Fri W31R16

20 1. Refine design Thu 2115/01

o/ 2. Instrument developmentlpretesting Thu 211/01

Tue 511/0

22 i. OM-6 packne Mon 4116101

Wed 9/12/0

Project: ipedschedule2Date: Thu 917/00

Task

Progress

Milestone

Summary

Lj1i1EaJ72jJ Road Up Task

11111010111

Rolled Up Milestone

Rolled Up Progress INIIIMMII•PMIIIINIIII

lit

External Tasks

Project Summary

External Milestone

VI. Estimates of Cost and Task Schedule forProposed Phase II and Phase III Studies

97

Figure VIA: Listing of Tasks and Time Schedules

P° 2001

2002

2003

2004

20

200605ID Task Name Start Finish H2 H1

i

H2

i HI

1 H2

HI

I H2

H1

I H2

H7

rH2

HI

I H2I23 4. Recruit projects schools Thu 311101 Fri 12128101

24 5. Informed consent Fn i r11/02 Thu 2128x02

25 6. Train project/school staff Fri 2115/02 Thu 2128/0226 7. Baseline survey (Cycle 1} Fri 3/1/02 Fri 8/30102

27 8

Random assignment (Cycle i) Mon 9/2102 Fri 11/15/02

28 9

Initial site visits Mon 213/03 Wed 4130103

29 10. Baseliine survey (Cycle 2) Mon 3/3/03 I

Fri 8129103

30 11. Random assignment (Cycle 2) Wed 1011/03 i Fri 11/28/03

31 12. Round 2 site visits Mon 212/04 I Fri 4130104

32 13. Followup Survey (Cycle 1-12th Wed 10/1103 Wed 3131104

33 14. Collect transcripts (Cycle 1-11th Cycle 2 .12) Fn 1013103 Wed 3/31104

34 15. Followup survey (Cycle 1 11th and Cycle 2 12) Fn 10/15/04 Thu 3131/05

35 18. Transcripts (Cycle 1-11 graders and Cycle 2-12th Fri 1011104 Thu 3131/05

36 17. Followup survey (Cycle 2-11) Mon 1013105 Fri 3131106

37 18. Collect transcripts (Cycle 2-11th) Mon 10/3105 Fn 3131106

38 19

Analysis Thu 411104 Wed 6/30/04

39 20. Report and briefings Fri 7130/041

Tue 11130/04

40 21

Project management

! Mon 1012100 Thu 11130108 s

c:w

y

Task

^^

j u<wRolled Up Task

External Tasks

Project ipedschedufe2 Progress

Rolled Up Milestone

Project Summary

sez9Date: Thu 917/00 Milestone

.

Rolled Up Progress

External Milestone

Summary

Split


98

A. OVERVIEW OF STUDIES AND COSTS

We briefly describe the studies and costs below.

1. State impact studies using state secondary and postsecondary databases, projectparticipant level records, and federal aid application and receipt files.

As described in Chapter III, this will be a records-based study in selected states utilizingproject, state, federal, and some district level databases. Plans are to work with a smallnumber of states that are developing statewide student transition databases (Texas, Indiana,North Carolina, Florida). These states' records will be used to conduct propensity samplingof Talent Search and comparison group schools and students. The federal aid files and statelevel databases for college tracking will be used to obtain outcome measures. Project fileswill also be used to obtain demographic and service information on TS participants. Thework will also include process studies in each of the states involving site visits andobservation of the role of TS within the overall state system of pre-college services fordisadvantaged students. The cost of these studies in 4 states is estimated to be$880,000. Of this $650,000 would be in the period of Oct 2000- Sept 2001 and $230,000in the period of Oct 2001 to Sept 2002.

2. Study of national impacts using federal financial aid files, project performancereport data, school district data from the Common Core of Data (CCD) andCounty demographic information.

This will be an exploratory study of the extent to which federal financial aid applicationand award files can be used to track outcomes and assess national impact. This componentaddresses the question of whether the rate of federal aid application and receipt within aschool district is impacted by the level of Talent Search service within the district. Theanalysis involves modeling the factors related to differential rates of federal aid applicationamong eligible students within school districts. Project performance reports and existingsurvey data on the numbers served by school and intensity of service will be used to createan index of service level for each school district. The strength of the study is that it addressesnational impacts. The estimatedcost for thisstudy is $260,000. Of this $190,000would be in the period of Oct 2000-Sept 2001 and $70,000 would be in the period ofOct 2001 to Sept 2002.

3. Demonstration studies of alternative innovative services

This component addresses the Congressional Mandate to evaluate the relativeeffectiveness of alternative innovative services. A random assignment design will be used inwhich students receive differential services of varying type and intensity. The final servicesevaluated will be determined by working with projects to determine the services in whichthey have the most interest. A review of the literature and input from the projects throughthe project survey suggest an interest in two types of services (information and


99

mentoring/tutoring). The first involves taking a more extensive approach and servingjuniors and seniors who have not previously been in the program with a service aimed atpushing undecided students toward college as an option through provision of increasedinformation and counseling toward college at critical times. The second service involvesproviding a more intensive one-on-one mentoring/tutoring service to randomly assignedgroups. The personal mentors will take the role of coaching the student toward deciding toattend college and.; assistance in completion of the steps necessary to do so (guidancediscussion, college visits, entrance tests, aid applications, enrollment).

The plans for this component involve implementing the demonstrations in 60 schools.There would be a sample of about 6000 students. Study tasks would involve siterecruitment, school coordination, training of site staff, school file procurement andsampling, baseline survey, transcript collection, and follow-up survey. Talent Search siteswould be given additional program funding to implement the additional services. Weanticipate that the demonstration would be done for two years for two cohorts of students.

The estimated cost of this study excluding payments to projects is about $3 millionover a6-year period (70 months). The funding needed would be as follows: $120,000(Oct 2000-2001); $700,000 (Oct 2001-2002); $500,0000 (Oct 2002-2003); $650,000 (Oct2003-2004); $500,000 (Oct 2004-2005) 520,000 (2005-2006).

B. SUMMARY OF COSTS

Table VI.1 gives a summary of costs for the proposed Phase II and Phase III of theproject.

Table VI.1: Summary of Cost EstimatesPhase II Year

OctYearOct

YearOct

YearOct

YearOct

YearOct

Total

2000-2001 2001-2002 2002-2003 2003-2004 2004-2005 2005-2006Records basednational impactstudyState impact

$190,000 $70,000 $260,000

studies in 4 states $650,000 $230,000 $850,000

Total phase II

Phase IIIInnovative service

$840,000 $300,000 $1,110,000

demonstrations

Total Phase II and

$120,000 $700,000 $500,000 $650,000 $500,000 $520,000 $2,990,000

Phase III $960,000 $1,000,000 $500,000 $650,000 $500,000 $520,000 $4,100,000


100

C. LIST OF TASKS AND DELIVERABLES FOR PHASE II

Table VI.2. lists the subtasks and a deliverable schedule for the proposed Phase II of theTalent Search project.

Table VI.2:Tasks and Deliverable Schedule for Phase II

TASK AND ASSOCIATED DELIVERABLES SCHEDULE OF TASKS AND DELIVERABLESExploratory National Impact Study

Sub-task 1: Refine DesignOct 30, 2000Design Memo

Sub-task 2: File PreparationJanuary 30File preparation memo

Sub-task 3: Model building and analysisMay 30Detailed analysis plan

Sub-task 4: Report of Findings

Report Outline-July 30Draft ReportSeptember 30Final ReportJanuary 30

State Process and Records Based ImpactStudies

Subtask 1: Refine design and gain accessJanuary 15, 2001Memo on state participation status and plans

Subtask 2: Collect and process projectparticipant records

February 15, 2001Memo on participation records collection

Subtask 3: Prepare state files, merge dataMarch 15, 2001Memo on state file and merge status

Subtask 4: Process system studiesJuly 20, 2001State Process summaries

Subtask 5: Conduct propensity modeling andanalysis of outcomes

Sept 15 2001Status of analysis report

Subtask 6: Prepare study report

October 20, 2001Draft of reportJanuary 30 2002Final report

D. DETAIL OF TASKS FOR THE PROPOSED PHASE III DEMONSTRATION STUDIES

Phase III includes a service mix evaluation and an implementation study. The purpose ofthe service mix study is to identify the mix and timing of services provided to 11 th and 12`hgrade participants that are most effective in achieving the goals of Talent Search. Data willbe collected from a total of 2,640 TS participants (60 schools x 44 students) and 3,600


101

eligible non-participants (60 schools x 60 students) at high schools associated with theTalent Search projects. Of the 6,240 students selected for the study, 60 percent will be in the12th grade and 40 percent will be in the

grade.

Data will be collected in 2 cycles. Cycle 1 will include half of the TS participants(n=1,320) and all of the non-participants (n=3,600). Cycle 2 will include the remaining halfof the TS participants (n=1,320). Each cycle will have an 11 th and a 12 th grade cohort for atotal of 4 cohorts of students. The sample sizes for each cohort are as follows:

Cohort 1 (Cycle 1 — 11`s graders) n = 1,9681,320 TS participants (60 schools x 22 students) x .4 = 5283,600 non-participants (60 schools x 60 students) x .4 = 1,440

Cohort 2 (Cycle 1 12th graders) n = 2,9521,320 TS participants (60 schools x 22 students) x .6 = 7923,600 non-participants (60 schools x 60 students) x .6 = 2,160

Cohort 3 (Cycle 2 — 11 th graders)

n = 5281,320 TS participants (60 schools x 22 students) x .4 = 528

Cohort 4 (Cycle 2 - 12th graders)

n = 7921,320 TS participants (60 schools x 22 students) x .6 = 792

Data collection will include:

A baseline survey of all TS participants and eligible non-participants. Thissurvey will be conducted in the spring prior to Academic Year 1 for eachcohort. For the cycle 1 cohorts, the baseline will be conducted in spring2002. For the cycle 2 cohorts, the baseline will be conducted in spring 2003.

n A telephone follow-up survey of all TS participants and eligible non-participants. This survey will be conducted in the fall/winter following highschool graduation. For cohort 2, a CATI survey will be conducted in thefall/winter of 2003/2004. For cohorts 1 and 4, a CATI survey will beconducted in the fall/winter of 2004/2005. For cohort 3, a CATI survey willbe conducted in the fall/winter of 2005/2006.

n The collection of high school transcripts for all TS participants and eligiblenon-participants. Transcripts will be collected in the fall/winter followinghigh school graduation, around the same time as the CATI survey. Forcohort 2, transcripts will be collected in the fall/winter of 2003/2004. Forcohorts 1 and 4, transcripts will be collected in the fall/winter of 2004/2005.For cohort 3, transcripts will be collected in the fall/winter of 2005/2006.


102

The cost estimates include the implementation study. This study involves conductingtwo rounds of site visits to 12 Talent Search projects (3 projects in each of 4 states). Theinitial site visits will be conducted in the spring 2003 and the second round in the spring of2004. Each site visit will include 5 high schools and will last 5 days.

The task structure and schedule for Phase III is outlined below.

Table VI.3: Task Structure and Schedule for Phase III

Task Description Duration(Months)

Date(s)

Task 1 Kickoff meeting/design refinement 2 January– February 2001Task 2 Instrument development/pretesting 5 February – June 2001Task 3 OMB package 3 July - September 2001Task 4 Recruit projects/schools 10 March– December 2001Task 5 Informed consent 2 January– February 2002Task 6 Train project/school staff 1 February 2002Task 7 Baseline survey (Cycle 1) 6 March – August 2002Task 8 Random assignment (Cycle 1) 2 October-November 2002Task 9 Initial site visits 3 February – April 2003Task 10 Baseline survey (Cycle 2) 6 March – August 2003Task 11 Random assignment (Cycle 2) 2 October – November 2003Task 12 Round 2 site visits 3 February - April 2004Task 13 Follow-up survey (Cycle 1 – 12th) 6 October 2003 – March 2004Task 14Task 15

Collect transcripts (Cycle 1 – 12th)Follow-up survey (Cycle 1 – 11 th

and Cycle 2 – 12th)

66

October 2003 – March 2004October 2004 - March 2005

Task 16 Collect transcripts Cycle 1 -11 thand Cycle 2 – 12t )

October 2004 - March 2005

Task 17 Follow-up survey (Cycle 2 -11 th) 6 October 2005 - March 2006Task 18 Collect transcripts (Cycle 2 -11 tn) 6 October 2005 – March 2006Task 19 Analysis 3 April - June 2006Task 20 Final report and briefings 4 July – October 2006Task 21 Project management 70 January 2001 – October 2006


REFERENCES

Adelman; Clifford, Answers in the Tool Box: Academic Intensity, Attendance Patterns, and Bachelor'sDegree Attainment. U.S. Department of Education, Office of Educational Research andImprovement, June 1999.

Akerhielm, Karen, Berger, Jacqueline, Hooker, Marianne, Wise, Donald. Factors Related toCollege Enrollment.- Final Report. Office of the Under Secretary, U.S. Department ofEducation. 1998

American Youth Policy Forum. More Things That Do Make A Difference for Youth: ACompendium of Evaluations of Youth Programs and Practices. Volume II. Donna Walker James,editor, with Sonia Jurich. Washington, DC. 1999.

American Youth Policy Forum. Some Things Do Make A Difference for Youth: A Compendium ofEvaluations of Youth Programs and Practices. Washington, DC.

Braum Linda and Brickman Alan. Information and Counseling Services and the Impact on CollegeAccess: A Preliminary Literature Review and Guide for Further Research Prepared for theEducational Resources Institute (I ERI). January 2000

Callahan, Mary-Mack. The Intentional Community--Colleges and Community GroupsHelping Low-Income Youth Prepare for College. Editorial assistance from RonaldA. Wolk. Washington, DC: Council of Independent Colleges. 1999.

Callahan, Mary-Mack. The Intentional Community, Colleges and Community Groups Helping Low-income Youth Prepare for College, Council of Independent Colleges and Dollars for Scholars,1999

Cochran, William G., and Rubin, Donald B. "Controlling Bias in Observational Studies: AReview." The Indian Journal of Statistics, Series A, vol. 35, part 4, 1973.

104

Delinquent Violent Youth: Theory and Interventions. Thomas, P. Gullotta, Gerald R.Adams and Raymond Montemayor, editors. Advances in Adolescent Development AnAnnual Box Series Volume 9. Sage Publications. 1998.

Durlak, Joseph A. School-Based Prevention Programs for Children and Adolescents.Developmental Clinical Psychology and Psychiatry, Volume 34. Sage Publications.1995.

Frangakis, Constantine E., and Rubin, Donald B. "Addressing Compilations of Intention-to-Treat Analysis in the Combined Presence of All-or-None Treatment-Noncomplianceand Subsequent Missing Outcomes." Cambridge, MA: Harvard University Departmentof Statistics, 1999.

Gladieux, Lawrence and Swail Scott. Financial Aid is Not Enough. College Board Review,Summer 1998.

Horn Laura J, Chen Xianglei, MPR Associates. Toward Resiliency: At Risk Students Who Make itto College. United States Department of Education Office of Educational Research andImprovement, May 1998

Improving the Odds: Increasing the Effectiveness of Publicly Funded Training. Barnow, Burt S. andChristopher T. King, editors. Washington, DC: Urban Institute Press. 1999.

Kazis, Richard and Hilary Kopp. Both Sides Now: New Directions in Promoting Work andLearning for Disadvantaged Youth. Jobs for the Future for the Annie E. CaseyFoundation. Boston, MA. July 1997.

King Jacqueline E. The Decision to Go to College. The College Board. 1996

Levine, Arthur and Jana Nidiffer. Beating the Odds--How the Poor Get to College. Higherand Adult Education Series. Jossey-Bass Publishers. San Francisco, CA. 1996.

McPherson and Schipiro, The Student Aid Game, Princeton New Jersey, Princeton UniversityPress, 1998

Merisotis, Jamie. A Review of the Level and Quality of Information, Resources, and Programs Availableto Students and Parents. Symposium on Information Resources, Services, and Programs.Background Paper Number Two. Advisory Committee on Student Financial Assistance,1992

National Center for Education Statistics Access to Posteseconday Education for the 1992 HighSchool Graduates. Washington, DC, 1997

Neild Ruth Curran and Frank Furstenberg. The 1996-97 Freshman Cohort: A Report to CollegeAccess. University of Pennsylvania, October, 1998

References

105

Nurturing Young Black Males--Challenges to Agencies, Programs and Social Policy. RonaldB. Mincy, editor. The Urban Institute Press. Washington, DC: 1994.

Rosenbaum, Paul R., and Rubin, Donald B. "Constructing a Control Group UsingMultivariate Matched Sampling Methods That Incorporate the Propensity Score." TheAmerican Statistician, vol. 39, no. 1, February 1985.

Rosenbaum, Paul R., and Rubin, Donald B. "Reducing Bias in Observational Studies UsingSubclassification on the Propensity Score." Journal of the American Statistical Association,vol. 79, no. 387, September 1984.

Rosenbaum, Paul R., and Rubin, Donald B. "The Bias Due to Incomplete Matching."Princeton, NJ and Cambridge, MA: Educational Testing Service and HarvardUniversity Department of Statistics, March 1985.

Rosenbaum, Paul R., and Rubin, Donald B. "The Central Role of the Propensity Score inObservational Studies for Casual Effects." Madison, WI and Chicago, IL: University ofWisconsin Departments of Statistics and Human Oncology, and University of Chicago,1983.

Rubin, D. B. "The Use of Matched Sampling and Regression Adjustment to Remove Bias inObservational Studies." The Biometrics Society, vol. 29, no. 1, March 1973.

Rubin, D.B. "Matching to Remove Bias in Observational Studies." The Biometrics Society,vol. 29, no. 1, March 1973.

Rubin, Donald B. "Assignment to Treatment Group on the Basis of a Covariate." Journal ofEducational Statistics, vol. 2, no. 1, pp. 1-26, spring 1977.

Rubin, Donald B. "Estimating Casual Effects From Large Data Sets Using PropensityScores." Cambridge, MA: Harvard University, May 29, 1997.

Rubin, Donald B. "Estimating Casual Effects of Treatments in Randomized andNonrandomized Studies." Journal of Educational Psychology, vol. 66, no. 5, pp. 688-701.

Rubin, Donald B. "Practical Implications of Modes of Statistical Inference for CasualEffects and the Critical Role of the Assignment Mechanism." Cambridge, MA:Harvard University Department of Statistics, December 1991.

Rubin, Donald B. "Using Multivariate Matched Sampling and Regression Adjustment toControl Basis in Observational Studies." Journal of the American Statistical association, vol.74, no. 366, June 1979.

Rubin, Donald B., and Thomas Neal. "Characterizing the Effect of Matching Using LinearPropensity Score Methods with Normal Distributions." Cambridge, MA and Princeton,NJ: Harvard University Department of Statistics and Educational Testing Service, 1992.

References

106

Rubin, Donald B., and Thomas Neal. "Matching Using Estimated Propensity Scores:Relating Theory to Practice." Cambridge, MA and Chapel Hill, NC: HarvardUniversity Department of Statistics and University of North Carolina, March 1996.

Rubin, Donald B., and Thomas, Neal. "Combining Propensity Score Matching withAdditional Adjustments for Prognostic Covariates." Cambridge, MA and Chapel Hill,NC: Harvard University and University of North Carolina, February 16, 1999.

Russell, Alene Bycer. The Status of Statewide Student Transition Data Systems: A Surveyof SHEEO Agencies. State Higher Education Executive Officers. May 1999.

Sum, Andrew, Stephen Mangum, Edward deJesus, Gary Walker, David Gruber, MarionPines and William Spring. A Generation of Challenge--Pathways to Success for UrbanYouth: A Policy Study of The Levitan Youth Policy Network. Policy IssuesMonograph 97-03. Sar Levitan Center for Social Policy Studies. June 1997.

Thomas, Ramona S. Black and Latino College Enrollment: Effects of Background, High School

Preparation, Family and Peer Influence, and Financial Aid. Paper presented to the AnnualMeeting of the American Educational Research Association, 1998

Tierney, Joseph P., Jean Baldwin Grossman with Nancy L. Resch. Making a Difference: AnImpact Study of Big Brothers/Big Sisters. Public/Private Ventures. Philadelphia, PA:November 1995

References

APPENDIX A

HIGHLIGHTS OF HIGH SCHOOLCOMPLETION AND COLLEGE ENROLLMENT

STATISTICS: 1960-1999

High School Completion

n Event high school dropout rates were about 6 percent in 1995. Low SESstudents' rates were about 13.3 percent, and high SES rates were about 2.0percent. By 1997, there were few observable differences between white (5percent) and black (6 percent) rates. However, Hispanic rates were about 12percent.

n The Census Bureau reported that the gap in high school completion ratesbetween Whites and African-Americans in the 25-29 age group narrowed by1997 to the point where there was no statistical difference (EducationalAttainment in the United States, March 1997).

n Since the 1970s blacks are the only ethnic group for whom the rate of eventdropouts has declined significantly. (Current Population Survey as reportedin Condition of Education, 1997, p.60).

College Continuation

n In 1996 college continuation rates for recent high school graduates reached arecord high of 65 percent. In 1967, college continuation rates were at 55percent, up from 45 percent in 1960. Since 1960, there has been a 44 percentincrease in the percentage of high school graduates enrolling in college in theyear after high school graduation (Current Population Survey, various years).

108

n The gap in college entrance rates of almost 43 percentage points between lowSES students and high SES students has not lessened since 1970. Between1974 and 1994, postsecondary enrollment rates of low and highsocioeconomic status (SES) student increased by similar amounts.Enrollment rates within 2 years for those who were seniors in 1992 increasedfrom 42.0 to 48.9 percent, and the rates for higher-income students in thesame years increased from 85.1 to 91.3 percent (High School and Beyondand NELS:88).

n High SES students remain about twice as likely to enroll in college as lowSES students. Low SES students are much more likely to enroll in 2-yearthan 4-year institutions and this difference is increasing (Condition of Education,1997, High School and Beyond, and NELS data).

College Continuation by Race/Ethnicity

n The rate of 18- to 24-year olds in college among whites was 33 percent in1972 and 43.7 percent in 1995. Among blacks, the rate was 27.2 in 1972 and35.2 in 1995. In the mid-1970s the gap between white and black enrollmentrates among recent graduates was less than it was at any time since. TheHispanic rate was 25.8 in 1972 and 34.9 in 1995 (US Department ofCommerce, Bureau of Census, October, Current Population Surveys,Condition of Education, 1997, p. 66).

n Examining the high school graduation rate by age 19 and the collegecontinuation rate of high school graduates by race/ethnicity reveals thefollowing about chances for college entrance among the population:

Chance for College Entrance by Age 19High school

graduation rateCollege continuation

rateChance for college

White, non-Hispanic 87.7% 67.7% 59.4%

Black 78.9% 55.3% 43.6%

Hispanic 68.4% 50.7% 34.6%

Other 88.4% 80.3% 70.9%Total 84.3% 65.0% 54.8%

Source: Postsecondary Educational Opportunity, August, 1997, number 62

College Costs and Family Income

n After a period of decline in the 1960s and 1970s, college costs at' publicinstitutions as a percentage of median family income, rose to about 15percent by the mid-1990s from a low of 10 percent in 1979 (IPEDS data andCurrent Population Surveys, March).

Appendix A: Highlights of High School Completion and

College Enrollment Statistics: 1960-1996

109

College Retention Rates

n Freshman-to-sophomore retention rates have averaged between 68 percent(1983) and 67 percent (1997) among Institutions of Higher Education (IHE).The ACT reported averages range from 91 percent in highly selective schoolsto 54 percent in open admissions schools. Most of the variation in rates isexplained by student entrance differences. However, differences in studentcollege entrance tests are controlled for, private and Catholic schools havehigher persistence rates (unpublished tabulations).

Demographic composition of the Postsecondary Student Population

n In 1974, whites made up about 83 percent of the postsecondary studentpopulation. In 1996, they accounted for 70 percent of this population. Theother statistics for 1976 and now, respectively, are: blacks, 11 and 16 percent;Hispanics, 5 and 9 percent; and the other races, 0.9 and 6 percent.

Gender

n Between 1959 and 1996, the college continuation rate for females increasedby 44 percent--from 39 percent to 70 percent; for males, the rate increasedfrom 54 to 60 percent--about 11 percent.

n Female enrollment increased by 23 percent and male enrollment by 14percent between 1976 and 1996.

Appendix A: Highlights of High School Completion andCollege Enrollment Statistics: 1960-1996

APPENDIX B

1992 TALENT SEARCH DESIGN CONFERENCESSUMMARY OF CONCLUSIONS

Report Title Author Focus Conclusions and Recommendations

Designing a Usefuland AppropriateEvaluation ofTalent Search:What are the MostImportant DesignQuestions toConsider?

Thomas A.Angelo

What an evaluationof Talent Searchshould entail.Outlines a plan for aneffective and usefulevaluation.

Angelo cautions that the Talent Search Evaluation should evaluate whatprojects actually do, not what they are assumed to do and accept thediversity in local projects. He also stresses that those most likely to beaffected by the evaluation should be involved in carrying out the evaluationand that the evaluation should build on already occurring evaluations. Healso notes that processes need to be evaluated in addition to performance-indeed the two are intimately linked. Angelo calls for using multiple methodsand measures and notes the importance of providing information onimproving and best practices. Angelo also brings up a key question thatneeds to be addressed concerning what is a successful TS project? To thisquestion, can be added how this definition of success has changed overtime?

Talent Search:Issues for anEvaluation

Alvia Y. Branch Branch notes the importance of documenting current practices of typical andexemplary projects. She stresses that TS must be examined in theinstitutional and community context looking at relationships that existbetween TS and schools. Branch notes the difficulty in achieving a validcomparison group and collecting impact data on college readiness andentrance. Any evaluation will need to determine upon which groups ofstudents the projects are focusing and develop appropriate outcomemeasures for that group. A random assignment design would work best butshe notes may not be feasible.


Perspectives on anEvaluation ofTalent Search:Interviews withTalent Search Staff

Ann Coles Interviews TalentSearch projectdirectors to see whatthey think should beincluded in anevaluation of TalentSearch and whatthey think would beuseful from such anevaluation. Alsosought projectdirectors views aboutthe ways TalentSearch impactsstudents, theschools, and thecommunity.

Coles approach was to ask 19 TS project directors what they most wantedgain from an evaluation and also what types of participant information theycollect. They confirmed the basic outcomes question—does TS increase HSgraduation and postsecondary enrollment? In addition project directors wantto know which service delivery models work best and under whatcircumstances. They want to know whether particular program strategiesaffect outcomes and what are the most effective program practices. Inaddition they want to know about the college experience of their students andprogram impact on parents. She stresses the differences in TS dependingon the institutional context, noting the differences between the way urban andrural projects must operate. She recommends interviewing students, schooland TS staff, and parents. She especially recommends developing profiles ofTS students. She also notes the importance of having TS project directorson the advisory panel.

MeasuringProgramOutcomes: WhatImpacts areImportant toAssess, and WhatImpacts arePossible toMeasure?

Amaury Noraand Alberto F.Cabrera

Focuses onoutcomes that areassociated withdifferent stages in the• college choice.process

The authors focus on college choice as a process that involves four stagesand notes that there are different outcomes that should be measured at eachstage. They recommend evaluating TS interventions at each stage. Thestages are: (1) predispositional stage that involves development ofoccupational and educational aspirations; (2) the search phase that involvesactually gathering information about possible options; (3) the choice phasethat involves deciding which option to pursue, and (4) the enrollment phasethat involves actually taking steps to enroll such as applying for financial aid.A TS evaluation would have to be conducted in stages looking at these stepsand should use qualitative as well as quantitative means.


Review of TwoStudies of TalentSearch: TheResearch TriangleInstitute. s 1975Study of TalentSearch and PaulFranklin* s 1985Study for theCollege Board

James E.Rosenbaum

Lessons learnedfrom Earlier TalentSearch evaluations

Rosenbaum presents a comprehensive review of the previous studies.Based on this review, he notes that for an evaluation to be useful, theconcerns of project directors must be taken into account. He recommendsthat the evaluation look at both process and outcomes. He recommendsevaluating what service or information works best for participants. He alsopoints up the need to have a clear definition of client or in this case• participant..

He also recommends looking at the interaction of TS with theschools. Does TS displace or supplement school activities? He alsosuggests looking at project records.

MeasuringProgram Impact:What Impacts areImportant toAssess and WhatImpacts arePossible toMeasure? AProposal forResearch

William T.Trent

A former TalentSearch projectdirector discussesimportant impacts forfuture evaluations

Trent notes the importance of establishing a baseline for a TS evaluationand determining the comparability or lack thereof between projects. Hestates that TS may be necessary but not sufficient to determine collegeparticipation. He identifies separate outcome measures for middleschool/early high school and senior high school. Impacts for students ingrades 8-10 could include: involvement in activities that provide informationon importance of staying in school, information on postsecondary education,information on preparing for postsecondary education; exposure to post-secondary institutions and their representatives; involvement in activities thatinform students how to be a better student. Impacts for students in grade 11-12 could include: direct access to representatives of postsecondaryinstitutions, counseling on test preparation and application procedures for testtaking (i.e. SAT); counseling, advising and assistance in applications foradmission to postsecondary institutions; counseling and assistance with thefinancial aid process; involvement in activities that inform students aboutpractices to be better students.


Analysis of TalentSearchPerformanceReports: 1986-87and 1990-91

ElizabethEisner

Examined annualperformance reportsto compare 1990-91program year to the1986-87 programyear.

She found that the size of projects range from 404 to 17, 470 in 1990-91.Most Talent Search grantees are institutions of higher education (most 4-yearpublic universities and 2-year public colleges). About 3/4 of participants meetthe income and first generation requirements in both years. In 1990-91,Black students comprised abut 33% of participants, Whites about 31% andHispanics about 26%. A majority (52%) of participants are female (in both1986-87 and 1990-91). The vast majority of participants are youths aged 12to 18 (83%).About 3/4 of participants were in high school

and another 11% weremiddle-school students. No one activity or service was offered at all projectsor that all participants received. But counseling (academic, personal, groupand/or peer) was the most common service received by participants (78%).Career awareness and orientation services were the next most commonservices with 66% percent of participants receiving services that fall into thiscategory. Financial aid counseling and assistance was received by 58% ofparticipants. The most common services offered by projects were financialaid counseling (98%), counseling (96%), career awareness and orientation(96%), campus orientation and visits (85%). Most participants who go on topostsecondary activities go to 4-year public universities and the second mostpopular institutions were 2-year public schools.


First GenerationCollege Students:A Review of Needsand EffectiveInterventions

ConsueloArbona

Reviews literature,Talent Search andother programsgeared towardgetting low incomeand minority youth toenroll inpostsecondaryinstitutions to seewhat are the needsof these people andto see whatinterventions work

Some of the findings and recommendations coming out of this review are asfollows. In the studies reviewed, the strongest predictors of dropping out ofhigh school were low academic achievement, being overage for one . sgrade, and low socioeconomic status. Researchers have found that thepredisposition to attend college is already formed by the 9th grade. Earlyinterventions need follow-up to be effective in helping high-risk childrensucceed as adolescents and young adults. The review of practices foundthat three types of services were most important in helping students pursuepostsecondary education: (1) tutoring and enrichment courses; (2) careercounseling; and (3) workshops on college admissions, entrance exams, andfinancial aid process.

High school students, especially those in inner-cityschools, often do not get adequate counseling and guidance, and schoolcounselors do not offer adequate career and postsecondary services to helpstudents plan their high school program to enable them to pursuepostsecondary education.

Learning deficits are cumulative and effects oflow-achievement are progressively more difficult to counteract. Aspirationshave stronger influence on college admissions than in actual attendance forlow-income youth.

A number of studies suggest that improving academicachievement is the most important need of low-income students. She notesthat program evaluations have found the greater the needs of theparticipants, the harder it is to show significant program impacts. However,all students, even low-achieving ones, tend to do better in demanding andrich academic curriculums than in remedial ones. The discussion with regardto TS noted that most low income youth are not likely to attend collegewithout help of programs like Talent Search. There is a need for TalentSearch to provide comprehensive services and continuous instructional andmentoring services to have a significant impact on academic achievement.Talent Search programs need to target their services. There is a need tolook at short-term impacts when evaluating Talent Search. She prefersexperimental method for an evaluation but notes that a comparison groupmethod will also work.


Report on TalentSearchPerformanceCriteria

RussellJackson,ConsueloArbona, andAlvia Branch

Evaluates whatprojects can andshould provide interms of performanceindicators at severallevels (internal,annual, and nationalevaluation).Examines whatprojects can provideand what data ispossible to collectthrough a nationalevaluation.

The report evaluates what projects can and should provide in terms ofperformance indicators at several levels (internal, annual, and nationalevaluation). They note that the characteristics of • contexts* in whichprojects operate need to be captured as well as the • inputs* or resourcesfor program implementation. Most context and input variables can beobtained from the projects. initial proposal, but this may need to be collectedagain to standardize across sites. Also, it is important to collect participantcharacteristics. For consistency, projects should collect detailed informationon services offered that break down by type of service (workshop, smallgroup or individual based), and the number of individual contacts eachproject/counselor makes, the number of participants in each activity, and theaverage number of contacts per participant. They suggest that servicesreceived (and intensity of services) by individual participants be tracked. Theyrecognizes that random assignment is unlikely in a national evaluation butthat comparison groups nonetheless can be used. They note that ifconsistent measures are used, an evaluation of Talent Search is feasible,despite the inter-project variability.

APPENDIX C

SUMMARY OF FIRST TECHNICAL REVIEWPANEL MEETING

FOR THE NATIONAL EVALUATION OF TALENTSEARCH

BACKGROUND

The first meeting of the technical review panel for the National Evaluation of Talent Search washeld January 15, 1999, from 9:15 a.m. to 4:00 p.m., at the Washington, D.C., office of MathematicaPolicy Research, Inc. This memo highlights and summarizes the discussions that took place. Acopy of the agenda follows the summary.

Participants

Technical Review Panel

Paul Beasley, University of South CarolinaAnn Coles, Education Resources InstitutePaula J. Martin, Harlem Center for EducationGary Natriello,Columbia University

U.S. Department ofEducation

Robert Belle, Federal TRIO ProgramsHubert Davis, Federal TRIO ProgramsSandra Furey, Planning and Evaluation ServiceDavid Goodwin, Planning and Evaluation ServiceRicky Takai, Postsecondary and Adult Vocational Education Division

Council for Opportunity in Education

Andrea Reeve

118

Decision Information Resources

Consuelo Arbona

Branch Associates

Alvia Branch

Consultant

Lana Muraskin

Mathematica Policy Research

Margaret CahalanMark DynarskiJulia KimMichael MihaleczDavid MyersTim Silva

INTRODUCTIONS AND OPENING REMARKS

Robert Belle, David Goodwin, and Sandra Furey opened the meeting by talking about ED ' sinterest in this study. They highlighted its importance as the first national evaluation of TalentSearch in roughly 25 years, as well as the potential implications the results could have for informingother education-related initiatives, such as the GEAR-UP program.

After introducing themselves, each participant commented briefly on what he or she hoped thestudy would achieve. Several people expressed a strong interest in having information that will helpprojects to improve services, information on best or effective practices. One panel membercautioned that some project directors will have concerns about whether this study is intended toproduce information (i.e., lack of impacts) that ED or others will use to justify radically changing theprogram or eliminating it altogether. To help gain their support and full cooperation, they will haveto be shown how this study can benefit them. It was noted, however, that linking outcomes toprogram practices is a considerable challenge. Some participants also emphasized the potentialvalue--to both practitioners and policy makers--of information on the middle school component.There was also interest in the study providing basic descriptive information that captures thevariability and complexity of the program and the contexts in which it operates.

Next, Maggie Cahalan gave a brief overview of the study design and its objectives. Tim Silvaand Julia Kim then provided some context for the day ' s discussions by summarizing the highlights

Appendix C: Summary of First Technical Review Panel MeetingFor the National Evaluation of Talent Search

119

of a review they conducted of research literature on Talent Search and roughly 30 recent grantapplications.

CASE STUDIES

Mark Dynarski gave an overview of the case studies to be conducted for the evaluation, and leda discussion among the participants.

Focus/Purpose

The case studies were seen as a potentially good method for exploring issues such as "models ofeffective practices. " A slightly different suggestion was to use the case studies to highlight howprojects operate in various contexts or environments, such as in places relatively rich in pre-collegeservices versus in service-poor environments. A third possibility raised was to select some newly-funded projects, as a way of exploring and highlighting where the field may be headed. Overall,participants approved the concept of maintaining a student-focus, and of trying to describe TalentSearch in the context of everything else in students' lives, both in and out of school. One specificsuggestion for capturing the students ' experience was to interview alumni, to ask what the programhad meant to them as they progressed through the education system and through life moregenerally.

Site Selection and Timing

Some people raised the possibility that the case studies could be done in two waves, with thefirst wave; in the spring, selected randomly and used both to describe projects in general and informthe feasibility study, and with the second wave, in the fall, intended to flesh out certain selectedthemes or describe effective practices, etc. There was limited discussion of key issues such as thenumber of sites that should be chosen randomly and the way in which random selection should takeplace, and the number of projects that should be chosen purposively and how to identify modelsites. (Some participants suggested that grant proposals do not provide a very clear sense of projectdesigns/models/emphases.) However, no decisions were made on these issues. In general, Marchand April were described as the best times to conduct spring site visits; in May, as the end of theschool year nears, projects often have fewer activities going on.


120

PROJECT DIRECTOR SURVEY

Maggie Cahalan distributed a draft compilation of items that could potentially be included inthe project director questionnaire. Participants provided detailed feedback on roughly the first halfof the document; time did not permit a full review. Participants suggested cutting some draft items(such as on the age and languages spoken by project directors), modifying some items (such as onstaff training), and adding a few new items (including one on staff salaries, which is apparently anissue of substantial interest in the field). Some participants encouraged consideration of using otherdatabases (such as the Common Core of Data), rather than the project director survey, to obtaininformation on target schools, target communities, and host institutions. Finally, Maggie invited theparticipants to provide additional feedback, especially on the second half of the draft document,within two weeks.

FEASIBILITY STUDY

Mark Dynarski led a discussion of the key issues surrounding the effort to determine how bestto study the impacts of Talent Search on program participants. Participants with first-handexperience in Talent Search are convinced that the program has a positive effect on the lives ofmany of the students it serves. One participant observed that, given the program ' s design andstructure (a relatively small amount of service to a large number of participants, and withconsiderable between-site variability), an evaluation might expect to find few or small impacts onparticipants. However, the participant commented that an impact study might very usefully focuson how effective the program is in dealing with certain subgroups. Another participant pointed outthe importance of keeping in mind cost (expenditure)-per-student; even if impacts turn out to besmall, we would need to put such a finding in the context of the relatively small amount of moneyspent per student. There was general support for conducting an impact study that examines selectedshort-term outcomes, such as knowledge of financial aid, college-prep course selection, and collegeentrance exam taking.

MATHEMATICA'S RESPONSE AND PLANS

From the perspective of the evaluation team, the meeting was highly productive. Thequestions, comments, and advice we received will be very useful as we proceed with plans for thestudy.

• As for the case studies, the evaluation team has met to make decisions on some of thekey issues summarized above, and will prepare a memo to ED proposing specific plansfor site selection and the timing of the site visits.

• As for the project director survey, we will revise the draft instrument based on feedbackfrom meeting participants and a pair of conference calls planned with project directorsfrom around the country. Indeed, some written feedback has already been receivedfrom panel members.


121

• Finally, as requested by Dr. Belle, we will gladly brief the TRIO Program Office on thepurpose and overall design of the evaluation, as well as on the highlights of our reviewof the literature and grant applications, at a time mutually convenient to him and hisstaff.

FUTURE PANEL ACTIVITIES

We informed the panel members that the next meeting is tentatively planned for late fall orearly winter, by which time the evaluation team is expected to have preliminary results (likely in draftreport form) from the survey and case studies. The primary purpose of the second technical reviewpanel meeting will be to discuss whether and how to conduct an impact study in phase two of theevaluation. (If an impact study is undertaken, the panel will also meet during phase two, to provideadvice on issues related to that effort.) In the meantime, Maggie Cahalan promised to keep thepanel members informed of our activities, and said we may intermittently be sending them variousmaterials to review.


122

NATIONAL EVALUATION OF TALENT SEARCHTECHNICAL REVIEW GROUP

INITIAL MEETING

MEETING AGENDA AND DISCUSSION GUIDE

The discussion guide takes each of the major agenda items and provides key discussion points andbackground information.

January 15, 19999:15 a.m. - 4:00 p.m.


123

AGENDATECHNICAL REVIEW GROUP MEETING

NATIONAL EVALUATION OF TALENT SEARCHPHASE I

January 15, 19999:15 a.m. — 4:00 p.m.

9:15

Coffee, and Bagels

9:30

Introductions —(ED staff, TRG members, Mathematica staff)

9:45

Opening remarks from Department of Education10:00

Overview of the Study Components and Issues• Brief description of tasks and schedule for Phase I and Phase II• Brief discussion of approach, context, and issues• General discussion

10:45

Case Studies• Role of case studies

— Implementation_ study

- Feasibility study• Site selection (note the 32 selected for the grant review/criteria with which

selected; likely candidate for selection/with addition of special selects)• Structure of the visits (who, type of activities)• Protocol design (topics to cover, relationship to survey)• Focus of report and integration with survey report

12:00

Lunch

12:45

Project Director/Coordinator Survey• Talent Search and EOC projects• Role in study

— Implementation

- Feasibility• Topic item development• Review of draft instrument• Data collection plans and issues• Report and analysis plans/integration with case studies

2:15

Break

2:30

Issues of the Impact Study• Alternative designs of other national program impact studies• Possibilities with TS and relation to Congressional Mandate

3:30

Summary Wrap-Up

4:00

Adjourn


124

PROJECT DESIGN OVERVIEW AND DISCUSSION ISSUES

GOALS OF THE STUDY

In legislation providing for TRIO programs, Congress authorizes evaluations as follows:

EVALUATIONS —

(1) IN GENERAL B For the purpose of improving the effectiveness of the programs andprojects assisted under this chapter, the Secretary may make grants to or enter into contracts withinstitutions of higher education and other public and private institutions and organizations toevaluate the effectiveness of the programs and projects assisted under this chapter.

(2) PRACTICES B The evaluations described in paragraph (1) shall identify institutional,community, and program or project practices that are particularly effective in enhancing the accessof low-income individuals and first-generation college students to postsecondary education, thepreparation of the individuals and students for postsecondary education, and the success of theindividuals and students in postsecondary education. Such evaluations shall also investigate theeffectiveness of alternative and innovative methods within Federal TRIO programs of increasingaccess to, and retention of, students in postsecondary education (H.R. 6, Sec. 402H).

The objectives of the planned study correspond to this mandate. The major policy objectives of thestudy are as follows:

• To provide much needed updated information on the operations and services provided byTS and EOC and to identify program improvements and innovative practices

• To contribute to strategic thinking of how best to improve high school graduation rates andaccess to college for disadvantaged students

To understand what role federal policy can play in improving and promoting educationalopportunity for disadvantaged students

Given the outreach quality of the program, one component of the planned study will involve anassessment of whether an outcomes impact study is feasible and how best to approach this design.We will also address how it can be designed to be of most use to projects and policy makersinterested in identifying program improvements.

MAJOR RESEARCH QUESTIONS

The major research questions have been summarized as follows


125

• What are the specific objectives of TS/EOC projects, and how and why do they vary?How are resources organized to achieve objectives? What are the obstacles that impedeattaining objectives?

• To what extent do TS projects emphasize middle school students? How do project effortsregarding these students differ from efforts targeted on high school and older students?

• How does TS complement other school and community-based efforts?

• What are the similarities and differences between TS and EOC projects?

• What TS models/approaches work best and under what circumstances do they do so?

• What are the most important ways TS can supplement the services youth get from publicschools?

• Does ,TS enhance access to college?

MAJOR STUDY COMPONENTS

The major; components of the study are:

1. A descriptive study of the implementation of TS and EOC that involves both aproject survey and student centered case studies of 14 sites. These data collections arescheduled for 1999.

• Project Surveys of TS and EOC. We will conduct a mail and web based survey of allTS and EOC projects. The survey will collect data on project services, objectives, staffing,changes over time, promising innovations, and areas of needed improvement. The surveywill also obtain information on the feasibility of the impact study from the TS projects.

Student-Centered Case Studies. In-depth case studies will also be conducted inselected sites. The studies will focus on students' experiences in TS and examine ways inwhich the programs are able to help students develop the motivation and skills for choosingto attend college. The studies will also look at the relationship of the TS projects to thetarget schools and to other community and school services. The case studies will also beused to inform the impact feasibility study.

• An Impact Study Feasibility Assessment. The feasibility study is intended toascertain whether it will be possible to conduct an impact assessment and how thismight be designed to provide the most useful information to projects.

2. Iffeasible, an impact study ofprogram outcomes for participants. Phase II of thestudy will implement the impact assessment, should the feasibility assessment indicate that


126

it will be meaningful and useful. A preliminary design plan utilizes a matched comparisonapproach and focuses on assessing short-term outcomes for middle and senior highstudents.

PROJECT TASKS

• PHASE I TASKS

• Initial meeting (minutes are in the enclosed draft design document)

• Establish panel of study advisors and hold meetings

• Background review of past studies (see enclosed synthesis)

• Refine Phase I study design (see draft document enclosed)

• Develop data collection instruments

• Prepare OMB package for Phase I

• Select sites for case studies

• Conduct in-depth case Studies

• Conduct coordinator survey

• Analyze data and prepare reports

• PHASE II TASKS

• Refine study design

• Develop data collection instruments

• Prepare OMB package

• Implement impact study

• Analyze data and prepare reports


127

DISCUSSION ITEMS

The following is a listing of major items for discussion at the meeting.

CASE STUDY DISCUSSION POINTS

• What criteria should be used to select the sites? We have a stratified random sample thatwe used for grant review. We propose to select a portion of the 14 from this group and toselect others for special reasons. What method should we use to identify these specialsites(reputation of sites, grant competition scores, size of budget, possibility of experimentaldesign)? What should be the distribution in terms of numbers between random andpurposive selection? What is the relationship of the site selection to the feasibility study?What might cause us to rule out a site for selection?

• What should be the foci of the visits? How can we best to make them student centered?Whom should we interview?

• How can we make the best use of the site visits to inform the feasibility study?

• What is the best format for the final report? How does the decision about this formatinfluence the way the visits are conducted?

PROJECT SURVEY DISCUSSION POINTS

• What is the relationship between the case studies and the surveys? We have proposed to pre-test the survey on the case study sites and in that way to have it completed ahead of time in mostcases.

• How can the project survey best be structured to provide overview of the TS and EOC projectsthat will be of most use to practitioners and policy makers? What differences do there need tobe in the survey instruments for TS and EOC projects?

• How can the survey best be structured to provide input into the feasibility study? Can we use itto find sites that are willing to evaluate alternative service impacts? Can we identify innovativeapproaches?

• How best to ask selected items such as staffing, goals, services? What is the relationship to theinformation collected on the new performance reports?

• What are the major data collection and response rate issues? Burden? Timing? Modes ofcollection?


128

FEASIBILITY STUDY AND IMPACT STUDY DESIGN DISCUSSION POINTS

• What are the issues associated with an overall assessment of program impact vrs looking atimpacts of specific service models or services? Difficulty of measuring impacts.

What are the issues related to using a common definition of program "participation" for animpact study?

• How can we establish a matched comparison or control group?

• Can we capitalize on natural variation among projects or projects willingness to attempt towork with us to assess the impact of alternative services on student outcomes?

• How best to obtain meaningful baseline and follow-up information on participants?

• How to achieve the cooperation of target schools in implementing the impact study?

• How to obtain meaningful measures of service intensity and service type?

• Is it feasible to match study participants with records in federal financial aid files?


APPENDIX D

MINUTES OF PROJECT DIRECTORTELECONFERENCE

MEMORANDUM

MATHEMATICAPolicy Research, Inc.

TO:

Talent Search Project Directors Teleconference Participants

FROM:

Maggie Cahalan

DATE: 6/26/99

SUBJECT: Minutes of the Call

This memo summarizes the Project Directors teleconference for the National Study of TalentSearch. This teleconference was held on June 8' from 1:30- 3:00 (EST). The following personswere present during the call:

Department of EducationSandra Furey

EOC/ Talent Search DirectorsAlba N. Lebron de Ayala (Talent Search)Pamela Boisvert (EOC & Talent Search)Rene Dubay (Talent Search)Nessa Kleinglass (Talent Search)David Megquier (EOC/ Talent Search)Jean Richie (Talent Search)John W. Westby (Talent Search)

Other ParticipantsAndrea Reeve (National TRIO Clearinghouse)Lana Muraskin (Education Research and Policy Consultant)

130

Mathematica Policy ResearchMargaret CahalanMark DynarskiTim SilvaClaire Wilson

The meeting began with introductions of all persons present for the call. Following theintroductions, the Project Director reviewed the main purpose of the call, which was to obtain inputregarding the design of the impact evaluation study for Talent Search. A brief update on the on thecurrent status of the study was also given. During the remainder of the call, each participant had anopportunity to address design issues and expectations regarding the study. The teleconferenceended with a wrap up and summary of the major points discussed during the call.

Current Talent Search Issues

As we began the discussion, each project director gave input on current issues of concern totheir project. Major issues noted included:

• Serving middle school students and implementing a pipeline or pathway model

• Access issues• Maintaining and developing access to students especially related to the schools

participation in school reform and emphasis on standardized test preparation

• Special access issues if state takes over schools if achievement levels not reached

• Concern that a high percentage will not pass the basic skills test needed to graduatedisplacing interest in college access

• Trying to get into the master school schedule and assigned to certain classes on certaindays

• Working with families and understanding how this works

• Use of computers and technology

• Program attendance issues--work programs hinder students' participation in after schoolprograms.

• Concern with increased resources needed to participate in performance reporting

Goals for Impact Study

Next project directors addressed the question of what are the issues that project directors thinkthe national evaluation could most usefully address. The following were among the points made.

• Several project directors stated that they are looking to the study to identify some "bestpractices" and predictors of successful programs.

Appendix D. Minutes of Project Director Teleconference

131

Others pointed out that in addition to looking at overall impact (like rates of going to college)its important to consider short-term outcomes, since they can be easier to track and provide amore immediate index of a program's effectiveness. (Some examples of short- term outcomesmentioned: high school students' enrollment in college preparatory classes, numbers of studentsapplying for financial aid.)

• Different Talent Search programs use different service models. The question was raised as towhether the impact study can inform us about which models provide the most effective service?Alternatively, can we examine the range of practices across different programs to see howvarious programs can learn from each other?

• There is a desire to have the study designed in such a way that its results are most useful topractitioners. This may require a more detailed level of information (i.e., information that will behelpful to those who are actually running the program.) than a typical impact study mightprovide

Design Issues

The discussion then focused on a number of design issues and challenges. These included.

Comparison and Control Groups.

• If a program serves all who can be served, it becomes difficult to create a within schoolcontrol group. Schools are often reluctant to give the kind of cooperation that is needed tocreate a control group.

• Some projects suggested that we compare Talent Search projects with different approachesserving the same types of students and one project director gave an example of two projectsin her area that delivered services differently. She would be interested in knowing which oneworked best under what circumstances.

• Alternatively sometimes there are projects within a school in which only certain sections of aclass are being served. Other sections/classes could serve as a comparison group.

• Another project director suggested that the study use other demographically comparabledistricts and schools that are not currently delivering TRIO services for comparison. Thereare issues with obtaining cooperation of the districts and schools and also with matching.There may be little incentive for them to cooperate unless there is the promise of theirschool receiving such services somewhere down the line.


132

Other Challenges/ Questions:

Defining the participants. This is sometimes hard to do in Talent Search. Some studentswho receive services may not always be defined as a participant.

• Access Issues. It is becoming harder to gain the cooperation of schools in getting accessto students for research.

• Controlling for different levels of service. Some students are served occasionally whileothers receive more consistent or intensive support. This poses a challenge in terms ofevaluating impact, unless we include a measure of level of support received.

• The question was raised: What is the purpose/role of Talent Search/ EOC? Are theysupposed to be large scale Upward Bound Programs or short term interventions?

The issue of achieving overall representativeness or purposive selection of impact studysites was also raised

Summary / Wrap-Up

At the end of the time period the major points were re-capped and a few new issues wereraised, including:

• Placing an emphasis on short term effects rather than long term effects

• Challenges of gaining access to students in the schools

• Considering differences between those who enter Talent Search in high school vs. those whoenter in middle school or between different service models

• Challenges of obtaining control or comparison groups and possibility of matching districtsand schools not being served

• The impact study's dual purpose of assessing overall impact and improving services of eachproject

In addition the possibility of a follow-up call to discuss models of service delivery andcomparison group issues was also raised and participants indicated they would be interested incontinuing to discuss the design.


APPENDIX E

TALENT SEARCH DATABASES IN FIVE STATES

Table E-1: Texas Talent Search Project Databases

According to survey, types of data available in .. .Computer database Hard copy files only SSN included in

database?Type of database

systemYears using DB

systemCould extract

data on 1998-99jrs/srs?

Project 1 DemographicsCollege or postsec.enrollmentContact sheetsServices receivedProject's assessments

SAT scoresACT scoresOther standardized testscoresCareer survey resultsHigh school or postsec.TranscriptsRecommendations orcommendationsCollege or postsec. app'sFinancial aid app'sCollege or postsec.enrollmentFollow-up data on formerparticipants

Yes Blumen 2 years Yes

Project 2 DemographicsRecommendations orcommendationsCollege or postsec. app'sFinancial aid app'sCollege or postsec.enrollmentContact sheetsServices receivedFollow-up data on formerparticipantsProject's assessments

SAT scoresACT scoresOther standardized testscoresCareer survey resultsHigh school or postsec.transcriptsDiagnostic test data

There is no centralized database system maintained by the project.Participant records are maintained by the individual schools.





systemCould extract


Project 3 DemographicsCollege or postsec. app'sFinancial aid app'sCollege or postsec.enrollmentContact sheetsServices receivedProject's assessments

SAT scoresACT scoresOther standardized testscoresCareer survey resultsHigh school or postsec.transcriptsRecommendations orcommendationsFollow-up data on formerparticipants

Yes Corel Paradox Approximately 2years (no morethan 3)

Yes

Project 4 DemographicsCollege or postsec.enrollmentContact sheetsServices receivedFollow-up data on formerparticipantsProject's assessments

SAT ScoresACT ScoresCareer survey resultsHigh school or postsec.transcriptsCollege or postsec. app'sFinancial aid app'sContact sheets


Project 5 DemographicsSAT ScoresACT ScoresOther standardized testscoresHigh school or postsec,transcriptsRecommendations orcommendationsCollege or postsec. app'sFinancial aid app'sCollege or postsecenrollmentContact sheetsServices receivedProject's assessments



According to survey, types of data available in .. .Computer database Hard copy files only SSN included in Type of database Years using DB Could extract

database? system system data on 1998-99jrs/srs?

Project 6

Demographics

Career survey results

Yes

BlumenSAT scoresACT scoresOther standardized testscoresAttitude scale profilesHigh school or postsec.transcriptsRecommendations orcommendations

Project 7 DemographicsSAT scoresACT scoresCollege or postsec.enrollmentContact sheetsServices receivedProject's assessments

Career survey resultsRecommendations andcommendationsCollege or postsec. app'sFinancial aid app's

Yes Blumen

Project 8 DemographicsSAT scoresACT scoresOther standardized testscoresContact sheetsServices receivedFollow-up data on formerparticipants

Career survey resultsHigh school or postsec.TranscriptsRecommendations orcommendationsCollege or postsed. app'sFinancial aid app'sCollege or postsec.enrollment

Yes ACCESS

Using Blumen for Yes4 yrs. – converted2 databases intoBlumen andsome servicesare there, butmostly basicdemographics

2 years

Yes

3 years

Yes





systemCould extract


Project 9 DemographicsFollow-up data on formerparticipants

SAT scoresACT scoresOther standardized testscoresCareer survey resultsHigh school or postsec.TranscriptsRecommendations orcommendationsDiagnostic test dataCollege or postsec. app'sFinancial aid app'sCollege or postsec.enrollmentContact sheetsServices receivedProject's assessments

Yes ACCESS Less than 1 year(currentlytransferring)

Yes

Project 10 DemographicsSAT scoresACT scoresOther standardized testscoresCareer survey resultsHigh school or postsec.transcriptsCollege or postsec. app's.Financial aid app'sCollege or postsec.enrollmentContact sheetsServices receivedFollow-up data on formerparticipantsProject's assessments

Recommendations orcommendations






systemCould extract


Project 11 DemographicsCollege or postsec.enrollmentContact sheetsServices receivedFollow-up data on formerparticipantsProject's assessments

SAT ScoresACT ScoresOther standardized testscoresCareer survey resultsHigh school or postsec.transcriptsRecommendations orcommendationsCollege or postsec. app'sFinancial aid app's

Yes ACCESS 7 years Yes

Table E-2: Indiana Talent Search Project BasesAccording to survey, types of data available in .. .Computer database Hard copy files only SSN included in Type of database Years using DB Could extract

database? system system data on 1998-99j rs/srs?

Project 1 DemographicsSAT scoresACT scoresCollege or postsec.enrollmentServices receivedFollow-up data on formerparticipants

Other standardized testscoresCareer survey resultsHigh school or postsec.transcriptsRecommendations orcommendationsDiagnostic test dataCollege or postsec. app'sFinancial aid app'sContact sheetsProject's assessments

Project 2 DemographicsContact sheetsServices receivedFollow-up data on formerparticipantsProject's assessments

Attitude scale profilesCareer survey resultsHigh school or postsec.TranscriptsDiagnostic test dataFinancial aid app's

Project 3 DemographicsContact sheetsServices receivedProject's assessments

Project director said everydata field is maintained onhard copy form and then putinto the database; so allfields available in bothformats. However, they donot necessarily make apoint of routinely collectingall fields for all students.

SAT scoresACT scoresOther standardized testscoresAttitude scale profilesCareer survey resultsHigh school or postsec.TranscriptsRecommendations orcommendationsCollege or postsec. app'sFinancial aid app'sCollege or postsec.enrollmentFollow-up data on formerparticipants

Custom

(designed toproduce the kindsof informationneeded forannual reports)

2, since fall of1998 whenproject first gotfunded

Yes Yes

Filemaker-Pro, About 7 Yesver. 3, for Macs

Customized Since 1991 Yessystem usingsoftware calledIBM Filing,Writing &ReportingAssistant

There is a placefor it in DB, butkids seldomrepott it, andstaff make nopush to get itYes, for all

Table E-2: Indiana Talent Search Project Bases




systemCould extract


Project 4 DemographicsCollege or postsec.enrollmentContact sheetsServices receivedFollow-up data on formerparticipants

SAT scoresACT scoresOther standardized testscoresCareer survey resultsHigh school or postsec.TranscriptsProject's assessments

Yes, try to getfor as manystudents aspossible.

Current system isAccess; priorsystem was DOS-based D-Base 3+

Data prior to1999-2000 werekept in D-Base3+; started usingAccess this pastyear, butessentially keptsame variables

Yes

Project 5 DemographicsHigh school or postsec.transcriptsCollege or postsec. app'sFinancial aid app'sCollege or postsec.enrollmentProject's assessments

SAT scoresOther standardized testscoresCareer survey resultsRecommendations orcommendationsContact sheetsServices receivedFollow-up data on formerparticipants

Yes, on allstudents.

Blumen 4 years Yes

Project 6 DemographicsServices received

Project staffer said all dataare in both database andhard copy—that they areessentially duplicate recordkeeping systems

Career survey resultsHigh school or postsec.transcriptsRecommendations orcommendationsContact sheetsFollow-up data on formerparticipantsProject's assessments

Yes, for all Blumen 2 years, since fallof 1998 whenproject first gotfunded

Yes

Table E-2: Indiana Talent Search Project Bases




systemCould extract


Project 7 Demographics SAT scoresACT scoresOther standardized testscoresCareer survey resultsHigh school or postsec.transcriptsRecommendations orcommendationsContact sheetsServices receivedFollow-up data on formerparticipantsProject's assessments

Yes, for all Access 5 years, with datamore completefor more recentyears

Yes

Project 8 DemographicsContact sheetsServices received

SAT scoresACT scoresOther standardized testscoresCareer survey resultsHigh school or postsec.transcriptsCollege or postsec. app'sFinancial aid app'sCollege or postsecenrollmentProject's assessments

Yes, for all A new Access DBis currently underdevelopment; oldFoxfire systemwas abandonedfor being non-Y2K compatible

Foxfire was usedfor probably atleast 5 years

Yes

Table E-3: Florida Talent Search Project Databases




systemCould extract


Project 1 DemographicsSAT ScoresACT ScoresCollege or postsec.enrollmentFollow-up data on formerparticipants

Other standardized testscoresCareer survey resultsHigh school or postsec.

'transcriptsRecommendations orcommendationsContact sheetsServices receivedProject's assessments

Yes University ofWashington

Approx. 7-8 years Yes

Project 2 DemographicsSAT scoresACT scoresOther standardized testscoresHigh school or postsec.transcriptsRecommendations orcommendationsCollege or postsec. app'sFinancial aid app'sCollege or postsec,enrollmentContact sheetsServices receivedFollow-up data on formerparticipantsProject's assessments

Yes Blumen Since Fall '98 Yes

Table E-3:_ Florida Talent Search Project Databases




systemCould extract


Project 3 DemographicsSAT scoresACT scoresOther standardized testscoresCareer survey resultsRecommendations orcommendationsFinancial aid app'sCollege or postsec.enrollmentContact sheetsServices receivedFollow-up data on formerparticipants

High school or postsec.transcriptsProject's assessments

Yes FileMaker Pro 4 years Yes

Project 4 DemographicsSAT scoresACT scoresFinancial aid app'sContact sheetsServices received

Career survey resultsHigh school or postsec.transcriptsRecommendations orcommendationsCollege or postsec. app'sCollege or postsec.enrollmentFollow-up data on formerparticipantsProject's assessments

Yes DOS/ ClipperSoftware

Over 10 years Yes

Project 5 Demographics SAT scoresACT scoresOther standardized testscoresCareer survey resultsCollege or postsec. app'sFinancial aid app'sCollege or postsec,enrollmentProject's assessments

Yes Blumen 1 year (newprogram)

No, not for 2years agobecause it is anew program.They couldprovide info. forthis year.





systemCould extract


Project 6 DemographicsSAT scoresACT scoresOther standardized testscoresAttitude scale profilesCareer survey resultsHigh school or postsec.transcriptsRecommendations orcommendationsDiagnostic test dataCollege or postsec. app'sFinancial aid app's

College or postsec.enrollment

Contact sheetsServices receivedFollow-up data on formerparticipantsProject's assessments


Project 7 DemographicsSAT scoresACT scoresOther standardized testscoresCollege or postsec. app'sFinancial aid app'sCollege or postsec.enrollmentContact sheetsServices receivedFollow-up data on formerparticipantsProject's assessments

High school or postsec.transcriptsRecommendations orcommendationsDiagnostic test data

Yes Banner — SCP/Oracle (Access)

3-4 years Yes





systemCould extract


Project 8 Demographics SAT scores Yes In the middle of Converting this Yes, couldACT scoresOther standardized testscoresCareer survey resultsHigh school or postsec.transcriptsRecommendations orcommendationsDiagnostic test dataCollege or postsec. app'sFinancial aid app's

College or postsec.enrollment

Contact sheetsServices receivedProject's assessments

converting toExcel from oldsystem that wasbased on FoxPro.

year's data toExcel; last 5years in old form.

convert old datato new form ifnecessary.

Table E-4: Massachusetts Talent Search Project Databases

According to survey, types of data available in .. .

Computer database Hard copy files only

SSN included indatabase?

Type of databasesystem

Years using DBsystem

Could extractdata on 1998-99

jrs/srs?Project 1 Demographics

College or postsec.applicationsFinancial aid applicationCollege or postsec.enrollmentContact sheetsServices receivedProject's assessments

Career survey results Yes Blumen 3 Yes

Project 2 DemographicsSAT scoresOther standardized testscoresContact sheetsServices received

Financial aid app's Yes for thosewho have aSSN, but for agood portion ofparticipantsthey only havea permanentresidentnumber

Blumen Probably at least5 years; nocurrent staff havebeen there longerthan 1 year, so noone knows forcertain

Yes, theoretically,but new staff arejust being trainedand haven'tmastered thesystem

Project 3 DemographicsCollege or postsec.enrollmentContact sheetsServices received

SAT scoresACT scoresOther standardized testscoresCareer survey resultsHigh school or postsec.transcriptsRecommendations orcommendationsDiagnostic test dataCollege or postsec. app'sFinancial aid app'sFollow-up data on formerparticipantsProject's assessmentsReport cards

No; they useeach student'sschool IDnumber(designated bythe state)

Blumen 4-5 Yes

Table E-4: Massachusetts Talent Search Project Databases

According to survey, types of data available in . . .SSN included in



systemCould extract

data on 1998-99

Computer database Hard copy files only jrs/srs?

Project 4 Demographics Other standardized test Yes -- but it is Blumen Perhaps 5 Director not sure,scoresCareer survey resultsHigh school or postsec.transcriptsRecommendations orcommendationsDiagnostic test dataCollege or postsec. app'sCollege or postsec.enrollmentContact sheetsServices receivedFollow-up data on formerparticipantsProject's assessments

optional forstudents toprovide; onlyabout half doso; for othersthe programcreates aunique 9-digitnumber

and worried thatdata from pastyears notaccurate orcomplete

Table E-5: North Carolina Talent Search Project Databases




systemCould extract


Project 1 DemographicsCollege or postsec.enrollmentServices receivedFollow-up data on formerparticipantsProject's assessments

SAT scoresACT scoresOther standardized testscoresCareer survey resultsHigh school or postsec.transcriptsRecommendations orcommendationsCollege or postsed. app'sFinancial aid app'sContact sheets

Yes ACCESS 2

years Yes

Project 2 DemographicsSAT scoresACT scoresCareer survey resultsDiagnostic test dataContact sheetsServices receivedFollow-up data on formerparticipantsProject's assessments

High school or postsec.TranscriptsRecommendations orcommendationsCollege or postsec. app'sFinancial aid app'sCollege or postsec.Enrollment

Yes Blumen 1 year Yes

Project 3 DemographicsContact sheetsServices receivedProject's assessments

SAT scoresOther standardized testscoresCareer survey resultsHigh school or postsec.TranscriptsDiagnostic test dataCollege or postsec. app'sFinancial aid app'sFollow-up data on formerparticipants






systemCould extract


Project 4 DemographicsCollege or postsec. app'sFinancial aid app'sCollege or postsec.enrollmentContact sheetsServices receivedFollow-up data on formerparticipantsAssessment records

SAT scoresACT scoresOther standardized testscoresCareer survey resultsHigh school or postsec.TranscriptsRecommendations orcommendationsDiagnostic test data

Yes ACCESS Less than 1 year Only the seniors

Project 5 DemographicsContact sheetsServices received

Career survey resultsHigh school or postsec.transcriptsProject's assessments

Yes PSFirst Choice(moving toBlumen in thenext 6 months)

3 years Yes

Project 6 DemographicsServices received

Other standardized testscoresAttitude scale profilesCareer survey resultsHigh school or postsec.transcriptsRecommendations orcommendationsCollege or postsec. app'sFinancial aid app'sContact sheetsProject's assessments

Yes Access 6 years No

Project 7 DemographicsFinancial aid app'sCollege or postsec.enrollmentContact sheetsServices receivedProject's assessments

Career survey results Yes Blumen Less than 1 year No





systemCould extract


Project 8 Demographics Recommendations or Yes Blumen 4 years YesSAT scoresACT scoresOther standardized testscoresCareer survey resultsHigh school or postsec.TranscriptsDiagnostic test dataCollege or postsec. app'sFinancial aid app'sCollege or postsec.enrollmentContact sheetsServices receivedFollow-up data on formerparticipantsProject's assessments

commendations

contract no.: ed-98-co-0073 matheiv atica policy research...

Documents