gao-17-400, k-12 education: education needs to … education . education needs to improve oversight...

K-12 EDUCATION

Education Needs to Improve Oversight of Its 21st Century Program

Report to Congressional Committees

April 2017

GAO-17-400

United States Government Accountability Office

United States Government Accountability Office

Highlights of GAO-17-400, a report to congressional committees

April 2017

K-12 EDUCATION

Education Needs to Improve Oversight of Its 21st Century Program

What GAO Found The Department of Education (Education) awards 21st Century Community Learning Centers (21st Century) grants to states, who in turn, competitively award funds to local organizations, which use them to offer academic enrichment and other activities to improve students’ academic and behavioral outcomes. In their most recent grant competitions, states awarded 21st Century funds to nearly 2,400 organizations—including school districts and community-based organizations—based on a variety of criteria, such as the quality of their proposed program designs. Relevant research we reviewed that compared program participants to those of non-participants suggests that the 21st Century program is effective in improving students’ behavioral outcomes, such as school-day attendance and reduced disciplinary incidents, more often than their academic outcomes. However, because Education’s current 21st Century performance measures primarily focus on students’ reading and math scores on state tests, Education lacks useful data about whether the program is achieving its objectives to improve students’ behavioral outcomes such as attendance and discipline—the areas where the program most frequently has a positive effect. Education officials have not substantially revised the program’s performance measures since 1998, in part because its authorization lapsed from fiscal years 2008 through 2016. Leading practices in performance measurement call for federal agencies to align performance measures with program objectives.

21st Century Community Learning Centers’ Objectives and Performance Measures for Student Outcomes

Education’s technical assistance to states does not adequately address challenges states face in evaluating their 21st Century programs and sustaining them when program funding ends. About a third of states reported in GAO’s 50-state survey that they face challenges in evaluating program performance, such as difficulty designing evaluations that shed light on program effects. Further, the 21st Century program was reauthorized twice, which resulted in significant changes to state requirements for evaluating programs. However, Education has not provided states written guidance on developing and conducting high-quality evaluations since 1999. Federal standards for internal control state that when significant changes occur agencies should periodically review policies for continued relevance and effectiveness in achieving their objectives. Absent written guidance to states on conducting high-quality evaluations, Education may miss opportunities to help states improve their capacity to conduct such evaluations to assure the program is meeting its goals.

View GAO-17-400. For more information, contact Jacqueline M. Nowicki at (617) 788-0580 or [email protected].

Why GAO Did This Study Education’s 21st Century program—funded about $1 billion annually since 2002—supports a broad array of activities outside the school day to improve student outcomes in high-poverty or low-performing K-12 schools. A statement accompanying the Consolidated and Further Continuing Appropriations Act of 2015 included a provision for GAO to review Education programs outside the regular school day.

GAO examined (1) how 21st Century funds are awarded and used, (2) what is known about the effectiveness of these programs, (3) how Education manages and uses program data to inform decision making, and (4) Education's technical assistance for evaluating and sustaining programs. GAO conducted a 50-state survey of program officials, obtaining a 100 percent response rate. GAO also reviewed selected state program evaluations and academic studies on student outcomes, and observed program activities and interviewed officials in four states representing a range of grant size and location.

What GAO Recommends GAO is making four recommendations, including that Education expand its performance measures for behavioral outcomes and provide written guidance to states on conducting high-quality program evaluations. Education neither agreed nor disagreed with the recommendations, and outlined steps it is taking to address them.

http://www.gao.gov/products/GAO-17-400


mailto:[email protected]

Page i GAO-17-400 K-12 Education

Letter 1

Background 4 States Use a Variety of Criteria to Award Sub-grants for 21st

Century Program Centers, Which Use Funds to Provide a Broad Array of Services 8

Research Suggests 21st Century Programs Improve Students’ Behavioral Outcomes More Often Than Academic Outcomes 15

Education Has Not Taken Important Steps to Ensure 21st Century Data Are Useful for Program Decision Making 20

Education’s Technical Assistance Does Not Yet Adequately Address Many States’ Challenges in Evaluating and Sustaining Programs 25

Conclusions 30 Recommendations for Executive Action 31 Agency Comments and Our Evaluation 31

Appendix I Scope and Methodology 35

Appendix II 21st Century Grant Award Amount by States, Fiscal Year 2016 40

Appendix III Comments from the U.S. Department of Education 42

Appendix IV GAO Contact and Staff Acknowledgments 46

Table

Table 1: 21st Century Grant Award Amount by States, Fiscal Year 2016 40

Figures

Figure 1: Overview of Objectives and Activities of the 21st Century Community Learning Center’s Programs 5

Figure 2: Overview of the 21st Century Community Learning Centers Grant Process 6

Contents

Page ii GAO-17-400 K-12 Education

Figure 3: Number of States That Reported Providing Additional Points or Consideration to 21st Century Community Learning Centers Grant Applicants, by Selected Categories 9

Figure 4: Examples of Activities Provided by 21st Century Community Learning Centers’ Programs 12

Figure 5: Most Frequent Funding Sources Used to Supplement 21st Century Community Learning Centers Grants 14

Figure 6: 21st Century Community Learning Centers Objectives and Performance Measures for Student Outcomes, Established by the U.S. Department of Education 21

Abbreviations 21st Century Community Learning Centers 21st Century Department of Education Education Elementary and Secondary Education Act of 1964 ESEA Every Student Succeeds Act ESSA

This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.

GAO-17-400 K-12 Education

441 G St. N.W. Washington, DC 20548

April 26, 2017

The Honorable Roy Blunt Chairman The Honorable Patty Murray Ranking Member Subcommittee on Labor, Health and Human Services, Education, and Related Agencies Committee on Appropriations United States Senate

The Honorable Tom Cole Chairman The Honorable Rosa L. DeLauro Ranking Member Subcommittee on Labor, Health and Human Services, Education, and Related Agencies Committee on Appropriations House of Representatives

The 21st Century Community Learning Centers (21st Century) program is meant to support local communities in providing learning opportunities for K-12 students outside the regular school day, such as before school, after school, and during the summer.1 The program is designed to target services primarily to students who are from low-income families or who attend schools in need of academic assistance. From fiscal years 2002 to 2016, Congress appropriated a total of about $16 billion to the Department of Education (Education) to administer 21st Century funds.2 In 2015, the program was reauthorized by the Every Student Succeeds

1For the purposes of this report, we use the phrase “activities outside the regular school day” to refer to activities “during non-school hours or periods when school is not in session.”

2The program was originally authorized in 1994 and reauthorized in 2002. Pub. L. No. 103-382, § 101, 108 Stat. 3518, 3844 (1994), Pub. L. No. 107-110, § 401, 115 Stat. 1425, 1765 (2002). Funding authorization for the 21st Century program ended in fiscal year 2007. For fiscal years 2008 through 2016, funding for the program was provided through annual appropriations.

Letter


Act (ESSA).3 According to Education, in school year 2013-2014 (the most recent data available at the time we did our work) the program provided funding for about 9,500 centers serving approximately 1.8 million students and 430,000 adult family members.

A statement accompanying the Consolidated and Further Continuing Appropriations Act of 2015 included a provision for GAO to review K-12 education programs that extend learning time beyond the regular school day.4 This report examines (1) how 21st Century funds are awarded and used, (2) what is known about the effectiveness of 21st Century programs, (3) the extent to which Education has effectively managed and used program data to inform decisions about the 21st Century program, and (4) the extent to which Education’s technical assistance has helped states assess and sustain high-quality programs once 21st Century funding ends.

We focused our review on 21st Century afterschool programs because centers have historically provided services more during this time as compared to before school and during the summer. To answer all four research questions, we reviewed Education documents, including annual performance reports, studies, and budget justifications; and federal laws, regulations, and policies. We also interviewed Education officials and other researchers on the 21st Century program. To assess the reliability of federal 21st Century program data underlying Education’s annual performance reports and studies, we conducted electronic data tests, 3Pub. L. No. 114-95, § 4201, 129 Stat. 1802 (2015). ESSA also allows states to use 21st Century funds to support academic activities that are included as part of a program that provides additional program hours during the regular school day. Education stated that this provision will be implemented starting in school year 2017-2018. Prior to fiscal year 2017, authorized activities must be conducted during non-school hours or periods when school is not in session (i.e., before school, after school, or during summer break) and may not be used to extend the school day. For a limited period of time prior to enactment of ESSA, Education approved waivers for some states to exercise flexibility to permit their sub-grantees to conduct authorized 21st Century program activities during the school day if there was an extended learning program in place. Throughout this report, any references to statutory requirements of the 21st Century program are to those that existed prior to the program’s reauthorization by ESSA, as these were the requirements in effect at the time of our review. 4See 160 Cong. Rec. H9307, H9841 (daily ed. Dec. 11, 2014) (explanatory statement), accompanying Pub. L. No. 113-235, 128 Stat. 2130 (2014). Under the same mandate, GAO issued a companion report examining what is known about Education programs that states and school districts can use to support extended learning time. GAO, K-12 Education: Federal Funding for and Characteristics of Public Schools with Extended Learning Time, GAO-16-141 (Washington, D.C.: Nov. 30, 2015).



reviewed relevant documents, and interviewed agency officials. We determined that the data were sufficiently reliable for our purposes. We also conducted a Web-based survey of 21st Century state coordinators from the 50 states and the District of Columbia and received responses from 100 percent of the coordinators.5 We analyzed responses to survey questions as well as additional comments state coordinators provided in response to open-ended questions. In addition, we interviewed state officials responsible for administering the 21st Century program in four states: Idaho, Massachusetts, Rhode Island, and Texas; and we observed activities and interviewed staff at two to four 21st Century centers in each of those states. States were selected to represent a range in the amount of 21st Century grant funding, student demographics such as race and ethnicity, and geographic location. Information collected about these four states is not generalizable but provides insight into 21st Century program operations at the local level.

Additionally, to examine what is known about the effectiveness of 21st Century programs, we reviewed select state-level program evaluations and other studies that reported outcomes for students who participated in the program. To identify relevant state-level evaluations, we took a two-pronged approach. First, we reviewed state evaluations that Education determined in a 2012 report on 21st Century program state evaluation practices (the most recent available Education report on this topic) to have used quasi-experimental methods or statistical controls, such as a comparison group, to account for other plausible influences on the outcomes reported.6 We then interviewed researchers to identify other quasi-experimental state evaluations published since 2012, and we reviewed them for the soundness of their methodology and data. Ultimately, we determined that four state evaluations were appropriate to address our research objective about the 21st Century program’s effectiveness. Second, we conducted a comprehensive literature review of 104 other studies of afterschool programs, including the 21st Century program. We ultimately determined that six studies were sufficiently rigorous and appropriately scoped to include in our review, including three meta-analyses that synthesized a broad array of studies on the outcomes

5Throughout the report, we use “states” to refer to the 50 states and the District of Columbia. 6U.S. Department of Education, Office of Elementary and Secondary Education, Review of 21st Century Community Learning Centers’ Grantee Evaluation Practices Final Report, (Washington, D.C.: Sept. 24, 2012).


of afterschool programs in general, including the 21st Century program.7 Additional information on our research objectives, scope, and methodology is available in appendix I.

We conducted this performance audit from April 2016 to April 2017 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

The 21st Century program is authorized to provide a wide range of activities for K-12 students and their families to:

1. provide opportunities for academic enrichment, including providing tutorial services to help students—particularly students who attend low-performing schools—meet state and local academic standards in core subjects such as reading and mathematics;

2. offer students a broad array of additional services, such as drug and violence prevention, counseling, art, music, recreation, and technology programs that are designed to reinforce and complement instruction in the regular school day; and

3. offer literacy and related educational development opportunities to families of students served.8

Education has established performance objectives for the 21st Century program, including objectives related to student outcomes. These objectives state that 21st Century participants will demonstrate educational and social skills and exhibit positive behavioral changes; and they will improve in outcomes such as academic performance, school attendance, and rates of disciplinary incidents and other adverse behaviors (see fig. 1).

7A meta-analysis is a statistical technique that combines findings from independent studies to develop a conclusion with a statistical power stronger than the analysis of any single study. 820 U.S.C. § 7171(a).

Background


Figure 1: Overview of Objectives and Activities of the 21st Century Community Learning Centers Programs

The 21st Century program is administered by state educational agencies. Education provides state agencies with annual formula grants, which in 2015 ranged from about $6 million to $132 million. The formula that determines the funding amount for a particular state is based in part on the percentage of total students from low-income families enrolled in K-12 public schools and how much that state spends per pupil on education.9 States, in turn, competitively award funds to sub-grantees, which may be school districts or community-based organizations, such as those that focus on youth development. By law, states must award sub-grants of a minimum $50,000 per year for periods of 3 to 5 years.10

920 U.S.C. § 7172(b)(1). 1020 U.S.C. § 7174(g), (h).

21st Century Grant Process


Sub-grantees oversee one or more physical locations, referred to as “centers,” where grant-funded services are provided to participating students and adult family members.11 Centers may be located in schools, churches, community centers’ or other spaces (see fig. 2). They must provide services during non-school hours or periods when school is not in session, such as before school, after school, on weekends, or during summer vacations and school breaks.

Figure 2: Overview of the 21st Century Community Learning Centers Grant Process

In making awards, states must give priority both to applications that propose to serve students who attend schools identified as needing improvement and that are submitted jointly by at least one school district receiving funds under Title I, Part A of the Elementary and Secondary Education Act of 1964 (ESEA), as amended, and at least one community-based organization or other public or private entity.12 In addition, states are authorized to include additional priorities in their sub-grant competitions so long as they are aligned with the statute’s requirements and priorities.

Education’s Office of Academic Improvement, under the Office of Elementary and Secondary Education, is responsible for overseeing states’ implementation of the 21st Century program. The Office of Academic Improvement is also responsible for providing ongoing

11For the purposes of our report, we refer to “community learning centers” as “centers.”

1220 U.S.C. § 7174(i). Title I, Part A (Title I) of ESEA provides formula grants to states for their school districts to improve educational programs in schools with high concentrations of students from low-income families. School districts that are unable to partner with qualified community organizations within reasonable geographic proximity are still entitled to receive priority under the law.

Program Monitoring and Evaluation


technical assistance to states and monitoring state performance through on-site and desk monitoring visits and ongoing communications with state program officials.13 The Office of Academic Improvement conducts on-site monitoring for the 21st Century program for each state grantee every 3 years, or more frequently if needed.

The Office of Academic Improvement is also responsible for reviewing states’ grant applications for, among other things, compliance, including reviewing certain assurances that states are required to provide under ESEA. For example, Education must review states’ assurances that they will grant awards only to eligible entities that propose to serve students who primarily attend schools with at least 40 percent of students from low-income families.14 In addition, states must collect plans from applicants describing how activities funded by the sub-grant will continue after 21st Century funding ends, as well as plans for how centers will address participants’ transportation needs, among other things.15

Education collects program data from states in order to report annually on whether 21st Century programs have met performance targets for their objectives. These data include information on sub-grantee characteristics, center activities, and participants’ demographics and outcomes. States monitor sub-grantee performance and program data submission to Education.

Under ESEA, states are required to conduct ongoing program evaluations to assess the effectiveness of their 21st Century programs and activities and disseminate the findings.16 Education reviews state evaluations as a part of its monitoring process.

13Education may reserve up to 1 percent of the 21st Century program’s annual appropriation to provide technical assistance to entities receiving sub-grants or to conduct a national program evaluation. 20 U.S.C. § 7172(a)(2). 1420 U.S.C. § 7173(a)(3)(A). 1520 U.S.C. § 7173(a)(8)(B), (9). 1620 U.S.C. § 7173(a)(13).


Together states received more than 4,000 21st Century sub-grant applications and funded nearly 60 percent of them (about 2,400) in their most recent sub-grant competitions, according to our survey of all 50 states and the District of Columbia.17 States’ criteria for these competitions vary, and most states score applications on a point system based on a variety of criteria. To help determine whether applicants have the capacity to implement high-quality programs, over half of states award applicants additional points based on the quality of their program design (33 states) and use of evidence-based practices (28 states). To sustain these programs once grant funding ends, about half of the states reported awarding additional points based on the applicants’ level of support from schools and school districts (25 states) and other external organizations (23 states). (See fig. 3.)

17States vary in how recently they held their sub-grant competitions. In our survey, 33 states reported that they held a sub-grant competition in school year 2015-2016. Eleven states reported that their most recent competition was in school year 2014-2015, and the remaining 6 states had not held a competition since school year 2013-2014 or earlier. One state did not respond to this question. We calculated the percentage of applicants funded by dividing the total number accepted by the total number of applicants. One state did not report data needed to calculate the percentage of applicants funded.

States Use a Variety of Criteria to Award Sub-grants for 21st Century Program Centers, Which Use Funds to Provide a Broad Array of Services States Use Varied Criteria, Funding Periods, and Amounts for Sub-grants to 21st Century Program Providers


Figure 3: Number of States That Reported Providing Additional Points or Consideration to 21st Century Community Learning Centers Applicants, by Selected Categories

Additionally, 10 states reported in our survey that they provided additional points to applicants serving schools identified as the lowest performing or with the largest gaps in student achievement based on comparisons by race or socio-economic levels.

To provide adequate time to implement successful programs, a majority of states reported that they offer sub-grantees 21st Century funding for 5 years, the maximum number of years allowed under law.18 Specifically, 39 states said in our survey that they provide 21st Century funding to sub-grantees for 5 years. Education’s 2003 program guidance states that research suggests it takes approximately 5 years of continual revision and improvement for a community to fully implement a successful 21st

1820 U.S.C. § 7174(g).


Century program.19 Further, existing 21st Century sub-grantees are also eligible to re-apply for funding outside the competition period if their grant is close to expiring. Of the applicants that received an award in the most recent competition, about 45 percent (nearly 1,100) had previously received a 21st Century grant award.20

States have provided different levels of funding for sub-grantees and their centers. Fifteen states reported the minimum and maximum amounts sub-grantees allocated to their centers. Specifically, the amount that a sub-grantee’s center received ranged from a minimum of $100,000 to a maximum of about $660,000, with a median amount of $185,000 per center in school year 2015-2016. Officials in Idaho told us the amounts vary because each center has different needs and student numbers. They said their program’s average cost per student was $1,700 per year. They noted that they encourage centers to target students most in need of services instead of serving all students in a school, even if the school primarily serves students from low-income families. Officials in Texas told us that funding amounts to centers can vary based on other state requirements, such as a requirement to employ a full-time program director or coordinator at the sub-grantee or center level.

Sub-grantees have used 21st Century funds to provide a broad array of services to K-12 students. Education’s 2014 annual report on the characteristics of 21st Century programs found that the most commonly provided activities in school year 2012-2013 were those to enrich academics, provide tutoring and homework help, and offer recreational activities.21 According to Education’s 2003 guidance, academic enrichment can include tutoring in core academic subjects and providing extra learning opportunities so that students can practice their academic

19U.S. Department of Education, Office of Elementary and Secondary Education, 21st Century Community Learning Centers Non-Regulatory Guidance (Washington, D.C.: February 2003). 20We calculated the percentage of applicants that had previously received a 21st Century grant by dividing total number accepted who had previously received an award by the total number of applicants who were accepted. Five states did not report data for the total number of applicants. 21Learning Point Associates and Diehl Evaluation and Consulting Services, Inc., An Overview of the 21st Community Learning Centers Characteristic Data: 2012-2013 (Naperville, IL: Learning Point Associates, May 2014). This report uses 21st Century program data from school year 2012-2013.

Sub-grantees Use 21st Century Funds to Provide Academic Enrichment and Recreational Activities


skills through hands-on activities. The 2014 annual report also found that many centers supported building students’ academic skills in reading and math as well as in the arts, science, technology, cultural activities, and health. Further, at least two states gathered information on the time spent on specific activities as part of their state 21st Century program evaluations. In New Jersey, for example, a 2015 state evaluation examined time spent by students on specific 21st Century activities, including academic improvement/remediation, tutoring, academic enrichment, community service, and recreation. Of these, academic improvement/remediation was the activity on which students spent the most time.22 Oregon’s 2012 state evaluation also found centers were most likely to offer weekly activities they categorized as enrichment, homework help, or recreational.23 In terms of subjects targeted by the program, centers were most likely to report in the state evaluation that their centers’ weekly activities focused on reading, math, and arts or music.

Officials in all 13 centers we visited in our four selected states said they offered a mix of academic support and enrichment activities. These activities generally focused on reading and math, science and technology, art and music, and fitness and nutrition. For example, centers we visited used computers and tablets to reinforce math skills or introduce new concepts, such as computer programming, for students who had limited access to technology. Officials at 7 of the 13 centers also said 21st Century activities can fill gaps in a school’s offerings such as providing music or computer instruction that schools may not have the opportunity to provide during the regular school day. (See fig. 4.)

22Matthew Vinson, Samantha Sniegowski, and Feng Liu, New Jersey 21st Century Community Learning Centers Year 2 Report 2013-14 Program Year (Washington, D.C.: American Institutes for Research, November 2015). 23Deborah Moroney et al., Statewide Program Evaluation of 21st CCLC Grants: Oregon Department of Education (Chicago, IL: American Institutes for Research, October 2012).


Figure 4: Examples of Activities Provided by 21st Century Community Learning Centers Programs

21st Century centers serve different grade levels and use different staffing models. Education’s 2014 annual report and officials in the states we visited said that centers were primarily for elementary school students, but some also served middle and high school students. Further, Education’s report found that centers relied primarily on paid staff, but staff characteristics varied widely. For example, the report found that some centers used regular school-day teachers and non-school based staff, such as youth development workers or staff without college degrees. In the 13 centers we visited, officials told us their staff included a mix of certified teachers and paraprofessionals who work during the regular school day, as well as staff from outside the school who work in the program only part-time.


Education’s guidance encourages sub-grantees to identify other sources of related funding, and Education’s monitoring protocols ask sub-grantees to describe how all of these resources will be combined or coordinated to offer a high-quality, sustainable program.24 Eight states in our survey reported that they require sub-grantees to match Education funds with funding from other sources.25 For example, one state required sub-grantees to match 30 percent of their 21st Century grant award, with at least 10 percent required from sources outside the participating school district. Even if a state does not require matching funding, 21st Century applicants for sub-grants must identify federal, state, and local programs they will combine or coordinate with on their 21st Century program to make the most effective use of public resources.26 In our survey, states reported that the two most frequent funding sources sub-grantees used to supplement their 21st Century funding were the local school district (23 states) and other federal government programs (21 states), such as the Department of Health and Human Services’ Child Care and Development Fund and the Department of Agriculture’s Child and Adult Care Food Program.27 (See fig. 5.) Officials at several centers we interviewed also said they provide snacks and meals during the afterschool programs using Department of Agriculture funds. Ten states reported in our survey that sub-grantees frequently use other Education funding, such as Title I, Part A of ESEA, as amended, which provides financial assistance to districts and schools with high numbers or percentages of children from low-income families. In addition, officials in Idaho told us their sub-grantees also use funds from Education’s Migrant Education program, which provides educational and support services to migrant children.

24U.S. Department of Education, Office of Elementary and Secondary Education, 21st Century Community Learning Centers Non-Regulatory Guidance (February 2003).

25If a state has a matching requirement, it cannot require the match to exceed the federal grant amount, and it must be based on a sliding scale that takes into account the relative poverty of the population to be targeted by the sub-grantee and the ability of the sub-grantee to obtain matching funds. 20 U.S.C. § 7174(d)(1), (2). 2620 U.S.C. § 7174(b)(2)(C). 27The Child Care and Development Fund, administered by the Department of Health and Human Services, provides federal funding to states for child care subsidies to low-income parents so they can work, look for jobs, or participate in education and training activities. The Child and Adult Care Food Program, administered by the Department of Agriculture, reimburses child care providers for meals and snacks to children in child care centers and other locations, among other things.

Sub-grantees Leverage Their 21st Century Funding to Obtain Additional Funding from Other Sources


Officials from 46 states reported in our survey that they allow sub-grantees to charge student fees. These states, however, generally had a policy that no child would be turned away due to a family’s inability to pay. Of these states, 27 reported they require sub-grantees to have a sliding scale for fees based on family income.

Figure 5: Most Frequent Funding Sources Used to Supplement 21st Century Community Learning Centers Grants

To encourage sub-grantees to secure funding from alternate sources, most states reported that they gradually reduce 21st Century sub-grants over the funding cycle. Specifically, officials in 37 states reported in our survey that they reduce funding for sub-grantees after an initial period. For example, one state reported that it awards full funding in the first 2 years, but then reduces funding to 80 percent of that amount in the third year, 60 percent in the fourth year, and 40 percent in the fifth year. Further, 15 states reported that they reduce funding levels for sub-


grantees that have received 21st Century grants in previous funding cycles (“repeat sub-grantees”). Officials in Texas told us they fund repeat sub-grantees for 3 years rather than the usual 5 to preserve funds for new recipients. In Massachusetts, repeat sub-grantees can only apply for 50 to 75 percent of their previous grant amount.

Few evaluations of the 21st Century program use methodologies that are appropriate for determining the effect of program participation on student outcomes. Of the 10 studies we included in our review of program effectiveness—4 state evaluations and 6 other studies—that do use such methodologies, 3 state evaluations found a positive relationship between program participation and school-day attendance and/or discipline.28 (See appendix I for more details on our process for selecting studies.) These state evaluations—from New Jersey, Texas, and Washington—examined behavioral outcomes such as school-day attendance and discipline. All three of these evaluations found there was a positive effect on school-day attendance, and two of them found there was also a positive effect on school-day discipline. For example, Washington’s state evaluation found that school-day attendance improved for students in grades 6 through 12 who participated in the program when compared to similar students who had not participated.29 Similarly, Texas’s state evaluation found participating students in grades 4 through 11 had improved school-day attendance. This effect was particularly strong for students who

28Three of the six studies were meta-analyses of afterschool programs in general, including 21st Century programs. 29Neil Naftzger, et al., Washington 21st Century Community Learning Centers Program Evaluation: 2012-13 and 2013-14 (Chicago, IL: American Institutes for Research, October 2015).

Research Suggests 21st Century Programs Improve Students’ Behavioral Outcomes More Often Than Academic Outcomes Research Indicates Improvements in Some Behavioral Outcomes for Participating Students


participated in the program 60 days or more, especially in grades 6 through 11.30 These students had an absentee rate that was more than 20 percent lower than non-participants. Additionally, Texas’s evaluation found that students in a statewide 21st Century pilot program with intensive academics showed improved school-day attendance.31 In New Jersey’s evaluation of students in grades 4 through 8, students in all grades had improved attendance; this improvement was generally larger in the higher grades, particularly 8th grade.32 Further, one of the three meta-analyses of afterschool programs in general also identified similar positive effects on school-day attendance.33 This meta-analysis, published by Education in 2014, synthesized the results of 30 studies, including 6 studies of 21st Century programs.34 It identified positive effects on students’ academic motivation, a broad category of outcomes which included measures such as school-day attendance and homework completion.

Two of the four state evaluations we included in our review also examined school-day disciplinary incidents—which include fighting, bullying, or disruptive conduct that result in a student’s removal from the classroom—and both found that 21st Century program participation may improve outcomes, particularly for students who participated in the program for 60 days or more.35 Specifically, Washington’s state evaluation found an 30Elizabeth Devaney, et al., Texas 21st Century Community Learning Centers 2014-15 Evaluation Report. (Naperville, IL: American Institutes for Research, May 2016). 31The evaluation also assessed the pilot’s effect on participating students’ academic outcomes and found no effect. 32Vinson et al. (2015) 33Yael Kidron and Jim Linsday, The Effects of Increased Learning Time on Student Academic and Nonacademic Outcomes: Findings from a Meta-analytic Review (Washington, D.C.: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Appalachia, 2014). 34Another meta-analysis published in 2014—funded in part through grants from Education and the Department of Health and Human Services—examined afterschool programs that served at-risk students and found no observable effect on school-day attendance. The third meta-analysis of afterschool programs that we reviewed did not examine school-day attendance. Kirstin P. Kremer, et al., “Effects of After-School Programs with At-risk Youth on Attendance and Externalizing Behaviors: A Systematic review and Meta-analysis.” Journal of Youth and Adolescence, vol. 44, no.3 (2014). Sema A. Taheri and Brandon C. Welsh, “After-School Programs for Delinquency Prevention: A Systematic Review and Meta-analysis,” Youth Violence and Juvenile Justice, vol. 14, no. 3 (2016). 35The other two state evaluations that we reviewed did not examine school day discipline.


association between program participation and fewer disciplinary incidents for students in grades 3 through 12 who attended 60 days or more.36 In Texas, student participation in the program for 60 days or more was also associated with fewer disciplinary incidents.37 Separately, Texas’s evaluation also found that the presence of certain program characteristics in the state’s 21st Century academic pilot program were associated with lower rates of disciplinary incidents.38 In particular, the evaluation showed that centers in the pilot that taught students face-to-face rather than via computer were associated with fewer disciplinary incidents. Moreover, this positive association was stronger for centers whose curricula focused on general learning strategies rather than on specific subject area skills.

Regarding the effect of the 21st Century program on reducing delinquency, a 2016 meta-analysis of 12 studies examining afterschool programs in general, including one on programs funded through the 21st Century program, found no significant effect.39 This meta-analysis identified levels of delinquency—incidents such as arrest rates and violent behavior—as an important measure for afterschool programs because many of them focus on reducing such behaviors by providing a safe, supervised environment where children spend less time with potentially delinquent peers. Nevertheless, when the researchers averaged the results of all afterschool programs in their review, they found no significant effect on delinquency rates.

Results of some studies we reviewed demonstrated a positive association between participation in 21st Century programs and improved academic outcomes for selected groups of students or for particular types of activities. For example, in Texas the program was associated with higher math scores for students who participated 60 days or more.40 However, none of the 10 studies in our review observed consistently better scores in either math or reading in program participants’ state assessments. (See appendix I for more details on our process for selecting studies.) 36Naftzger et al. (2015). 37Devaney et al. (2016). 38Devaney et al. (2016). 39Taheri and Welsh (2016). 40 Devaney et al. (2016).

Research Indicates Mixed Effects on Math and Reading Scores for Participating Students


The 10 studies we reviewed identified differing effects of participation in 21st Century programs on students’ math scores. For example, evaluations for Virginia in grades 4 through 12 and for Washington in grades 4 through 8 found the program had no significant effect on math scores.41,42 On the other hand, two other state evaluations indicated that 21st Century programs were associated with improved math scores for certain grade levels. Specifically, New Jersey’s evaluation found a positive association between program participation and higher math scores, but it was not observable for students in 4th or 6th grade.43 Additionally, results from Texas’s evaluation found an association between program participation and increased math scores among middle school students.44

None of the state evaluations we reviewed showed a significant association between participation in a 21st Century program and increased reading scores. In fact, Texas’s evaluation showed lower reading scores among students who participated in the program compared to students who did not participate.45 In particular, participants in grades 4 and 5 had lower reading scores than non-participants. However, the Texas evaluation showed no observable decline in reading scores among students in middle school who attended 30 days or more. Evaluations for Washington, Virginia, and New Jersey found no significant relationship between program participation and increased reading scores for grades 4 through 8.46,47,48 Additionally, a 2014 academic study examined differences in program characteristics among 58 21st Century

41Margie King, et al., Virginia Department of Education Evaluation of 21st Century Community Learning Centers 2014-15. (Memphis, TN: University of Memphis, July 2016). 42Although Washington’s evaluation found that the program had a positive effect on math scores in a prior year, we were unable to obtain information to assess the reliability of Washington’s data from that year. Naftzger et al. (2015) 43Vinson et al. (2015). 44Devaney et al. (2016). 45Devaney et al. (2016). 46Naftzger et al. (2015). 47King et al. (2015). 48 Vinson et al. (2015).

Program Effects on Math Scores

Program Effects on Reading Scores


programs in New York City and found no association with improved reading scores for students.49

Afterschool programs’ inconsistent effects on math and reading scores may be the result of these programs serving students with different needs, according to recent research. In particular, two studies we reviewed indicated that the impact of specific types of activities in 21st Century programs may differ depending on the students being served. Specifically, they used statistical modeling to analyze the relationships between program features and the different educational needs among 21st Century program participants. One study found that programs focusing solely on academic content were associated with larger increases in reading scores for students with limited English proficiency than programs that mixed academic content with other activities; while students overall benefited from both solely academic programs as well as those that mixed academic with other activities.50 The other study found that activities with structured interactions with adults—including opportunities for collaboration and meaningful verbal exchanges—were associated with increased reading scores for middle school students. For elementary students, however, there was no association between these activities and improved reading scores.51

49Lisa M. DeBellis, Participant-Staff Relationship Quality and Academic Outcomes among Elementary and Middle School Children in After-School Programs (Ph.D. dissertation, Fordham University, 2014). 50Carol Ward et al., “Assessing the 21st Century After-School Program and the Educational Gains of LEP Participants: A Contextual Approach,” Journal of Education for Students Placed at Risk (JESPAR), vol. 20, no. 4 (2015). 51Jacob Leos-Urbel, “What Works After School? The Relationship Between After-School Program Quality, Program Attendance, and Academic Outcomes,” Youth & Society, vol. 47, no.5 (2015).

Program Effects for Subgroups of Students


While Education has developed performance measures to align with some 21st Century program objectives—primarily student academic outcomes—it has not aligned its measures with other program objectives related to key student behavioral and socio-emotional outcomes. As previously noted, these objectives describe a goal of program participants demonstrating improvement in three areas—educational, social, and behavioral—with outcomes such as improved academic performance and school attendance, and lower rates of disciplinary incidents and other adverse behaviors.

Education’s current performance measures were established in 1998. They address participating students’ English and math grades and state test scores as well as some behavioral outcomes, including homework completion, class participation, and classroom behavior. However, Education does not measure two other behavioral outcomes that are included in 21st Century program objectives: improved school-day attendance and a decrease in disciplinary incidents, although research has shown positive program effects for these two outcomes more often than for academic outcomes. Some states also recognize the importance of measuring behavioral outcomes associated with these programs, with about half of states (26 states) reporting in our survey that they choose to measure at least one of them.52 The remaining states, however, are not measuring either of these behavioral outcomes.

In addition, Education has not established any performance measures for socio-emotional outcomes, although social skills are also included in program objectives, and socio-emotional learning is an important component of 21st Century implementation across states we visited and

52The remaining 25 states reported they have either not established state-level 21st Century measures or do not measure school-day attendance or disciplinary incidents.

Education Has Not Taken Important Steps to Ensure 21st Century Data Are Useful for Program Decision Making

Education’s Performance Measures and Data Do Not Address Some Key 21st Century Objectives

Alignment of Performance Measures with Program Objectives


surveyed. According to Education’s website, socio-emotional learning involves students’ knowledge and skills necessary to understand and manage emotions and establish positive relationships, among other things. Twenty-seven states reported in our survey that they currently measure or are developing measures for at least one socio-emotional outcome, including student relationships with adults or communication skills.53 Again, the remaining states are not measuring any of these socio-emotional outcomes. At the centers we visited, programs offered various activities geared toward socio-emotional learning, including activities that paired students with adult mentors and developed students’ problem-solving skills. Officials in two states we visited told us that socio-emotional learning is a major component of their 21st Century programs’ philosophy. (See figure 6 for a comparison of 21st Century program objectives and Education’s program performance measures).

Figure 6: 21st Century Community Learning Centers Objectives and Performance Measures for Student Outcomes, Established by the U.S. Department of Education

53These 27 states reported measuring at least one of the following four outcomes: students’ school engagement in learning (e.g., persistence on tasks), relationships with adults (e.g., showing respect and seeking assistance), communication skills, or problem solving. Twenty-three states reported that they have either not established state-level 21st Century measures or do not measure any of these four outcomes. One state reported it has established state-level measures, but it did not respond to the question asking which types of measures it established.


Note: Education measures math scores on state tests only for students attending 21st Century middle and high school centers and reading scores only for students attending elementary school centers. For all remaining measures shown, Education collects data for all students and breaks them out by elementary and middle school/high school.

Several factors contributed to Education’s decision to maintain its current 21st Century performance measures for over 15 years. In particular, Education officials told us they have not substantially revised performance measures since 1998 in part because the program’s authorization lapsed from 2008 through 2016, with ESSA providing funding authorization starting in fiscal year 2017. As a result, the officials said this created uncertainty about potential future program changes. In addition, they said they were reluctant to revise the measures given the costs to states of tracking and reporting on new measures, as well as concerns about maintaining the continuity of data collected over time. Education officials told us they may revise the program’s performance measures when they implement program changes pursuant to ESSA and that in doing so, they will consider including additional behavioral performance measures as well as socio-emotional measures.

Education’s lack of 21st Century performance measures for some key program objectives, however, has limited the usefulness of its performance data. Specifically, without measures for student behavioral and socio-emotional outcomes, Education lacks useful data on the extent to which the program is achieving its stated objectives, especially in areas where the program is likely to have positive effects on student outcomes. While about half of states reported they already measure at least one behavioral or socio-emotional outcome, absent measures for all states, Education will continue to report incomplete program results to Congress and the public. Further, leading practices in performance management call for agencies to use performance information as the basis for decision making.54 These practices also state that aligning performance measures

54GAO, Executive Guide: Effectively Implementing the Government Performance and Results Act, GAO/GGD-96-118 (Washington, D.C.: June 1996).

http://www.gao.gov/products/GAO/GGD-96-118


with program objectives can enhance the use of information for management decision making.55

While Education has taken steps to promote data quality, including data accuracy and completeness, it lacks reasonable assurance that the data submitted by states are accurate. Education officials told us they began using a new online data system in school year 2014-2015, called 21APR, in part to improve data quality. Among other changes, 21APR does not permit users to advance to the next page until the data on the current page are complete. Education officials said that this can help prevent missing data—a feature that was absent in the previous data system. In addition, in 2016 Education provided states with regional training sessions and instructional documents for data entry. However, Education has not independently assessed the accuracy of 21st Century data submitted in 21APR.

Education officials told us they require states to check their data submissions rather than Education conducting these checks because the agency does not have access to sub-grantees’ and centers’ source information. Education’s own Information Quality Guidelines state that data should be processed and edited to help ensure they are accurate.56 Performing accuracy checks would not necessarily require that Education have access to the source data. For example, it could perform basic logic checks between fields—such as verifying that the number of student participants in one grade is fewer than the total number of participants in all grades. Education officials told us they may explore additional checks for accuracy, but they did not provide a timeframe for doing so.

Education officials do review states’ procedures for data quality assurance, although some states have expressed concerns about the quality of data they receive from sub-grantees. Since 2015, when

55GAO, Managing for Results: Enhancing Agency Use of Performance Information for Management Decision Making, GAO-05-927 (Washington, D.C.: Sept. 9, 2005). Further, measures should demonstrate results, address important aspects of program performance, and provide useful information for decision making. GAO, Pipeline Safety: Management of the Office of Pipeline Safety’s Enforcement Program Needs Further Strengthening, GAO-04-801 (Washington, D.C.: July 23, 2004). GAO/GGD/AIMD-99-69. GAO/GGD-96-118. 56U.S. Department of Education, Information Quality Guidelines, accessed February 28, 2017, https://www2.ed.gov/policy/gen/guid/iq/infoqualguide.pdf

Education Does Not Have Reasonable Assurance That 21st Century Program Data Submitted by States Are Accurate



http://www.gao.gov/products/GAO/GGD/AIMD-99-69

http://www.gao.gov/products/GAO/GGD-96-118


Education implemented new monitoring protocols that included a review of states’ data quality procedures, Education officials have monitored 20 states and have not found any states that are noncompliant. However, in our interviews and survey responses, officials from 10 states commented that they had concerns about the quality of program data they collect from sub-grantees. For example, states are unable to upload data to the 21APR system, unlike in Education’s previous data system, and several states expressed concerns that they must enter data manually. One state official commented that this can increase the risk of errors.

Without independently checking accuracy, Education may be unable to detect and address potential data errors in the 21APR system—data that Education eventually needs to use to conduct analyses that inform management decisions, identify sub-grantees needing technical assistance, and contribute to annual performance reports and budget justifications. Without reasonable assurance that data in the 21APR system are of sufficient quality, Education may not be able to effectively use the data for such decision making and reporting going forward, despite the fact that the new system was created, in part, to improve data quality. Further, absent better information on the quality of its data, Education will be unable to communicate any data limitations in its reports to Congress and the public. Our prior work has identified leading practices in performance management, which state that agencies should disclose limitations associated with data used in reporting and indicate what action they plan to take to address the limitations.57

57GAO/GGD/AIMD-99-69. GAO, Results-Oriented Government: GPRA Has Established a Solid Foundation for Achieving Greater Results, GAO-04-38 (Washington, D.C.: Mar. 10, 2004).

http://www.gao.gov/products/GAO/GGD/AIMD-99-69



States in our survey gave mixed reviews on the usefulness of Education’s technical assistance for developing and conducting state-level evaluations. Twenty-nine states reported that they found Education’s technical assistance on conducting state evaluations to be very or moderately useful. On the other hand, 12 states reported they found it only slightly or not at all useful; 8 states reported they had not received any technical assistance on this topic; and one state said it was unaware that Education provided such technical assistance.58

Further, 40 states commented in our survey that they face challenges in evaluating sub-grantee performance that may limit their capacity to conduct high-quality evaluations. For example, six states reported they have difficulty designing evaluations that can determine whether a change in student outcomes results from a student’s participation in a 21st Century program or from other factors, such as interventions during the school day. This difficulty is evidenced by the fact that we identified very few state evaluations whose evidence standards allowed us to include them in our review on program effectiveness, as noted above. In addition, four states reported that they face challenges in defining and establishing performance measures for students who participate in the program, such as measures for behavioral and academic outcomes. For example, one state commented that it is difficult to measure the benefits of enrichment activities on academic performance.

Education also found that states are having difficulty conducting their required evaluations when Education reviewed them as part of its monitoring process. For example, in recent monitoring findings, Education officials said they found that five states did not have plans in place to 58One state did not report any information on this question.

Education’s Technical Assistance Does Not Yet Adequately Address Many States’ Challenges in Evaluating and Sustaining Programs States Reported Mixed Reviews on Education’s Technical Assistance for Evaluating their 21st Century Programs


conduct these evaluations. In addition, according to technical assistance reports provided as a part of the monitoring process, three states had difficulty conducting comprehensive evaluations such as establishing the appropriate scope of work and timeframes for conducting evaluations and developing appropriate performance measurement tools.

According to Education officials, it has not provided written guidance to states on 21st Century program evaluations since 1999, when it provided states non-regulatory guidance in the form of a guide to help them evaluate their programs and use the results to make program improvements.59 However, since 1999, there have been significant changes in the 21st Century program. Specifically, the program was reauthorized twice, resulting in changes to requirements for measuring and evaluating program performance. First, the 2002 reauthorization added new requirements for states to describe in their funding applications how the state will evaluate the effectiveness of programs, including developing performance indicators and measures to evaluate programs and activities.60 It also required sub-grantees to conduct periodic local evaluations to assess progress toward achieving goals.61 Second, as a result of requirements under ESSA, which reauthorized the 21st Century program in 2015, additional changes are planned starting in school year 2017-2018. Specifically, states will have to evaluate their programs in conjunction with new data collection and evaluation planning requirements, including requirements to track student progress over time and to include state standardized test scores and other indicators of student success, such as improved school-day attendance.62

Federal standards for internal control state that when significant changes occur in how an agency achieves its objectives it should periodically review policies and procedures for continued relevance and effectiveness 59N. Pane, I. Mulligan, A. Ginsburg, & A. Lauland, A Guide to Continuous Improvement Management (CIM) for 21st Century Community Learning Centers (Washington, D.C.: U.S. Department of Education, 1999). 60No Child Left Behind Act of 2001, Pub. L. No. 107-110, § 401, 115 Stat. 1425, 1768 (creating 20 U.S.C. § 7173(a)(13)) (2002). 61Pub. L. No. 107-110, § 401, 115 Stat. 1425, 1772 (creating 20 U.S.C. § 7175(b)(2)) (2002). 62 Pub. L. No. 114-95, § 4201, 129 Stat. 1802, 1986-1987 (2015). In addition, ESSA increased the amount that states can set aside for activities including monitoring and evaluation from 3 to 5 percent of their 21st Century funds.


in achieving its objectives or addressing related risks. Further, it states that written policies and procedures can help ensure that necessary actions are taken to address risks to achieving the entity’s objectives.63 In addition, our prior work has identified several leading practices in agencies’ use of evaluations, which state that agencies should promote capacity building to evaluate program activities as a key strategy for supporting objectives.64 Education officials told us they are developing a technical assistance plan to improve the support the agency provides to states to implement the 21st Century program. These officials told us they are considering the topics and areas of concern that they will address including developing and conducting effective evaluations and assessments, and they expect to finalize the list of topics in summer 2017. However, Education officials told us that the agency has not determined what type of technical assistance it will provide to states on conducting evaluations. Absent new written, non-regulatory guidance to states that addresses the areas in which they struggle most, Education may miss opportunities to help states improve their capacity to conduct high-quality evaluations and ultimately improve their programs.

Education’s technical assistance to states does not effectively address the challenges most states face in helping their 21st Century sub-grantees continue to operate their programs once grant funding ends, according to state and sub-grantee officials. In our survey, 35 states reported that their centers often face challenges providing the same level of services to students when the grant funding ends. Officials in 20 states reported that centers in their states generally reduce the level of services or cease to operate once 21st Century funding ends. Two states commented that less than 10 percent of centers had been able to continue operating after the 21st Century grants expired. These concerns were echoed by officials at 10 of the 13 21st Century centers we visited, who told us they had major concerns about sustaining their programs.

Twenty centers that commented they will be unable to sustain operations or maintain the same level of services after 21st Century funding ends had difficulty securing funds from other sources. For example, officials in 63GAO, Standards for Internal Control in the Federal Government, GAO-14-704G (Washington, D.C.: September 2014).

64GAO, Program Evaluation: Strategies to Facilitate Agencies’ Use of Evaluation in Program Management and Policy Making, GAO-13-570 (Washington, D.C.: June 26, 2013).

Education’s Technical Assistance Does Not Effectively Address States’ Difficulty in Sustaining 21st Century Programs

http://www.gao.gov/products/GAO-14-704G




one state commented that it is often difficult for centers to obtain funding from local school districts because district budgets are already strained by supporting school-day programs. Officials in three other states commented that there is no state funding dedicated exclusively to programs outside the regular school day.

By law, states must require sub-grant applicants to submit a preliminary plan for how their programs will continue to operate once grant funding ends.65 Education’s monitoring protocols call for Education officials to ask states if applicants have sustainability plans and how states monitor sub-grantees’ implementation of the plans. A majority of states (42 states) reported in our survey that they require sub-grantees to provide a written sustainability plan, while 6 states reported they do not.66 However, in its monitoring efforts, Education told us it did not identify any states that are currently out of compliance with this requirement.

Education may be missing opportunities in its monitoring efforts to collect information on states’ strategies and practices for program sustainability—information that could be useful for sharing promising practices across states. As a part of its monitoring process, Education officials told us they discuss with state officials how they ensure compliance with the requirement to have such sustainability plans in place, but they do examine the quality of the plans. Officials in 11 states in our survey commented they would benefit from opportunities to collaborate across states. States do appear to have valuable information on sustainability to share with each other. For example, 25 states—in our survey of all 50 states and the District of Columbia—reported providing additional points or credit based on applicants’ ability to leverage external funds and sustain programs when selecting sub-grantees. Further, in the states that collect information on what happens after 21st Century funding ends, most reported centers having some success in finding private or nonprofit funds to help replace Education’s grant funds. States reported that the most common sources of replacement funding were student fees charged to participants (17 states), private foundation funding (14 states), and non-profit funding (e.g., universities and community organizations) (13 states).

6520 U.S.C. § 7174(b)(2)(K). 66Three states did not report data on whether they require sub-grantees to provide a written sustainability plan.


Education officials told us they provide technical assistance on 21st Century program sustainability through interactions such as an annual summer training conference on the program and an online learning portal.67 However, Education’s July 2016 training conference—a forum where 21st Century participants share experiences and best practices—was generally not focused on state practices and policies. The sessions were generally focused on centers’ collaboration with the school or community, centers’ activities (e.g., literacy, science, and math), or how to report performance data. Officials in about a third of states reported in our survey that they did not know about or had no basis to judge Education’s technical assistance on sustainability. Another third of states reported that Education’s technical assistance in this area was slightly or not at all useful. For example, two states reported that the content of Education’s professional development, training, and other assistance is heavily geared toward sub-grantees.

Education officials said they do have other interactions and forums, such as regional meetings, to share information with states. For example, Education started in-depth training in 2016 for state-level officials at four regional meetings for the Midwest, Northwest, South, and East, but the sessions did not cover program sustainability.

Federal standards for internal control state that information should be communicated in a form that enables an agency to achieve its objectives.68 In addition, our prior work has identified leading practices in federal collaboration that have shown that information sharing among grant participants, such as states, is important for effective grants management.69 Education’s efforts to help facilitate information sharing among states can help states identify practices that can be tailored to meet their individual needs and leverage their knowledge to address common challenges in continuing their programs over the long-term.

67You For Youth is an online service hosted by Education that provides professional development and technical assistance on the 21st Century program. See https://y4y.ed.gov/. 68GAO, Standards for Internal Control in the Federal Government, GAO-14-704G (Washington, D.C.: September 2014).

69GAO, Municipalities in Fiscal Crisis: Federal Agencies Monitored Grants and Assisted Grantees, but More Could Be Done to Share Lessons Learned, GAO-15-222 (Washington, D.C.: Mar. 20, 2015); and Grants to State and Local Governments: An Overview of Federal Funding Levels and Selected Challenges, GAO-12-1016 (Washington, D.C.: Sept. 25, 2012).








Unless Education undertakes such efforts for the 21st Century program, students and families who participate may be at greater risk of not receiving the full range program benefits for the longest amount of time.

The 21st Century program is designed to fund programs outside the regular school day to improve academic and behavioral outcomes for K-12 students who are from low-income families or who attend low-performing schools. High-quality research on the effectiveness of this program is very limited due to several factors, including difficulty determining whether a change in students’ outcomes results from their participation in a 21st Century program or from other factors, such as interventions during the school day. Although existing research on effectiveness points to greater positive behavioral effects than academic effects, Education’s current performance measures do not address some key behavioral outcomes. The lack of performances for behavioral outcomes makes it difficult to determine whether the program is achieving some of its stated objectives—especially in areas where research has shown that the program is likely to have the greatest effect on student outcomes. Specifically, Education does not measure socio-emotional outcomes or two other behavioral outcomes included in program objectives: improved school-day attendance and discipline in the classroom. Absent these performance measures, Education is missing an opportunity to assess the full range of benefits of this program.

In addition, Education has not sufficiently assessed the quality of its program data, further limiting its ability to assess program effectiveness. Another factor hindering the agency’s ability to determine program effectiveness is the significant difficulty that states are experiencing in evaluating their programs. Although Education officials told us they are considering topics and areas of concern for additional technical assistance, Education has not yet provided states with sufficient guidance on developing rigorous evaluations. Unless Education takes steps to reasonably ensure the accuracy of data and provides written guidance to help states develop high-quality evaluations, it cannot ensure the program is effectively meeting its goals.

Lastly, states are experiencing substantial difficulty in sustaining their programs after 21st Century funding ends, and many states expressed interest in collaborating with other states to address such challenges. Education is uniquely situated to take the lead in sharing information with states to help them address their sustainability challenges by allowing them to identify state policies and practices that have had some success.

Conclusions


Sharing such information could help states address the challenges they face in continuing their programs over the long-term.

We recommend that the Secretary of Education direct the Office of Academic Improvement to:

1. Expand its performance measures for the 21st Century program to address all program objectives. Specifically, Education should establish performance measures related to key behavioral, including student attendance and disciplinary incidents, and socio-emotional outcomes.

2. Conduct federal-level data checks on the accuracy of 21st Century program data submitted by states. Such checks could test for logical relationships between fields. Education should also publicly disclose and address any data limitations it identifies, as appropriate.

3. Provide written, non-regulatory guidance to states on developing and conducting high-quality 21st Century state evaluations to help address the difficulties states face in measuring program performance and effectiveness.

4. Use the information it collects from its monitoring visits and ongoing interactions with states to share effective practices across states for sustaining their 21st Century programs once program funding ends. This information could be shared using existing mechanisms such as Education’s meetings with 21st Century state coordinators.

We provided a draft of this report to the Department of Education for comment, and its written comments are reproduced in appendix III. Education also provided technical comments that we incorporated in the report as appropriate. Education neither agreed nor disagreed with our recommendations; rather, it generally noted that it will keep our recommendations in mind as it continues to implement changes in the program as a result of ESSA, and outlined steps it will take to address our recommendations.

In response to our recommendation that it expand its performance measures for the 21st Century program to address all program objectives, including those related to behavioral and socio-emotional outcomes, Education said it will keep our recommendation in mind. Specifically, the department stated that it is in the process of re-examining whether additional or revised measures should be developed to align more

Recommendations for Executive Action

Agency Comments and Our Evaluation


significantly with the program’s statutory objectives under ESSA. Education also acknowledged the need to develop measures that provide sufficient information and data on program outcomes. As stated in our draft report, the Department currently measures some student behavioral outcomes; in its response, Education described one measure as: the percentage of all program participants with teacher-reported improvements in behavior. Therefore, we modified our recommendation to focus on measuring the program’s key behavioral outcomes. In its response, Education also expressed concern about collecting data on student attendance and disciplinary measures, noting that it will require effective collaboration between states, districts, and other eligible entities. However, as we stated in our report, about half of states already collect data on at least one of these two measures. Further, as we stated, research has shown that 21st Century programs more often have positive effects on student attendance and reducing disciplinary incidents than on improving students’ academic outcomes. Given these effects, we continue to believe that it is critical for Education to measure student attendance and disciplinary incidents to obtain more complete, accurate information on this program’s effect on student outcomes. In response to our recommendation that it conduct federal-level data checks on the accuracy of 21st Century program data submitted by states, Education commented that it plans to build in additional data checks into the data system beyond its current checks on the data’s completeness. Specifically, Education anticipates that new technology enhancements in the data system will be designed to flag for inconsistencies in data reporting. For example, the system may send a “flag” that participation data is significantly lower or higher than previously reported participation data. Further, Education indicated that it will consider whether auditors performing audits under the Single Audit Act can be asked and guided to do more checks on the accuracy and reliability of 21st Century program data.

Regarding our recommendation to provide written, non-regulatory guidance to states on developing and conducting high-quality 21st Century state evaluations, Education outlined several steps it has taken to assist states in the past. For example, Education said that it provided six states with individualized technical assistance on strategies related to developing statewide evaluations and measures. Education also noted that, to date, it has conducted two webinars on state evaluations and is in the process of including presentations from those webinars on its online learning portal so that states will have easy access to the information. In addition, Education stated that it included presentations on evaluation


strategies in the past during its Summer Institute. Education also said it would consider whether additional guidance for all states was needed. While these are important steps, we do not believe they are sufficient. Twenty-one of 51 states (41 percent of states) reported in our 2016 survey that they found Education’s technical assistance for developing and conducting state-level evaluations only slightly or not at all useful; they had not received any technical assistance on this topic; or they were unaware that Education provided such assistance. Therefore, we continue to believe that Education should prepare written guidance to assist all states in developing and conducting high-quality program evaluations.

Finally, regarding our recommendation to share effective practices across states for sustaining their 21st Century programs once program funding ends, Education stated that it hosts meetings twice a year for 21st Century state coordinators. At these meetings, Education officials share strategies with states related to program sustainability. Education stated that these meetings covered topics such as reducing the amounts of 21st Century grant awards by a percentage each year. However, Education officials told us in February 2017 that these meetings have not focused on topics on program sustainability for several years. We continue to believe that Education should take the lead in sharing information with states to help them address their sustainability challenges by sharing information on state policies and practices that have shown some success. In its comments, Education stated that it has not held regional meetings with states for several years; however, in 2016 it held four regional meetings with states on implementing its new 21APR data system. Therefore, we modified our recommendation to emphasize that such information be shared through these types of existing mechanisms.


We are sending copies of this report to the appropriate congressional committees, the Secretary of Education, and other interested parties. In addition, the report will be available at no charge on GAO’s website at http://www.gao.gov.

If you or your staff should have any questions about this report, please contact me at (617) 788-0580 or [email protected]. Contact points for our Offices of Congressional Relations [email protected] Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV.

Jacqueline M. Nowicki Director, Education, Workforce, and Income Security Issues

http://www.gao.gov/



Appendix I: Scope and Methodology

Page 35 GAO-17-400 K-12 Education

Our study of the 21st Century program was framed around four objectives: (1) how 21st Century funds are awarded and used, (2) what is known about the effectiveness of 21st Century programs, (3) the extent to which Education has effectively managed and used program data to inform decisions about the 21st Century program, and (4) the extent to which Education’s technical assistance has helped states assess and sustain high-quality programs once 21st Century funding ends. We focused our review on 21st Century afterschool programs because centers have historically provided services more during this time as compared to before school and during the summer. To address our four objectives, we used a variety of methods, including a Web-based survey of 21st Century state coordinators; a review of selected state evaluations and academic literature; a review of federal laws, regulations, and agency documents such as annual performance reports and guidance; interviews with federal and other officials; and site visits to four states.

To obtain information about the state sub-grantee award process, program sustainability and technical assistance of the 21st Century program, we conducted a Web-based survey of the 21st Century state coordinator at each state educational agency in all 50 grantee states and the District of Columbia. We conducted the survey from August 2016 through November 2016. The survey covered several topics, including state processes for awarding 21st Century sub-grants to local communities, state-level program evaluations, performance measures, and sub-grantees’ financial sustainability, among other things. We received responses from all 50 states and the District of Columbia for a 100 percent response rate. The survey included an introductory statement specifying that the survey focused on afterschool programs funded by the 21st Century program and collects information about federal-level guidance provided to state officials on this program.

The quality of survey data can be affected by nonsampling error, which includes variations in how respondents interpret questions, respondents’ willingness to offer accurate responses, and data collection and processing errors. To minimize such error, we included the following steps in developing the survey and in collecting and analyzing survey data: in developing the Web survey, we pre-tested draft versions of the instrument with 21st Century program officials in four states to check the clarity of the questions and the flow and layout of the survey. On the basis of the pretests, we made revisions to the survey. Further, using a Web-based survey and allowing 21st Century coordinators to enter their responses directly into an electronic instrument created an automatic


Survey of State Coordinators



record for each state in a data file and eliminated the errors associated with a manual data entry process. In addition, the program used to analyze the survey data was independently verified to ensure the accuracy of this work.

In order to examine what is known about the effectiveness of 21st Century programs, we reviewed select state-level program evaluations and academic studies that reported outcomes for students who participated in the program. To identify relevant research, we took a two-pronged approach. First, we reviewed three state evaluations that Education determined in a 2012 report on 21st Century program grantee evaluation practices (the most recent available Education report on this topic) to have used quasi-experimental methods or statistical controls, such as a comparison group, to account for other plausible influences on the outcomes reported. We retrieved the most recent evaluations from those three states and through interviews with researchers in the field; we identified three other quasi-experimental state evaluations published since 2012. We reviewed each of these state evaluations for the soundness of its methodology and data. In four cases where the published evaluation did not include enough information to complete our review, we contacted the states and the researchers to obtain additional data from their samples. Ultimately, we determined that four of the six state evaluations were appropriate for purposes of our research objective about the 21st Century program’s effectiveness.

Second, we conducted a comprehensive literature review of 104 academic studies of afterschool programs, including the 21st Century program, and we ultimately determined that six of these academic studies were sufficiently rigorous and appropriately scoped to include in our review. To identify academic studies on the effectiveness of 21st Century programs, we conducted a literature search through ProQuest. Our initial search terms were adapted from a 2014 report produced by Education that looked at the effect of 21st Century programs, along with other programs that increased the amount of time that children spent in school.1 Because our review was focused on afterschool programs, rather than

1 Yael Kidron and Jim Linsday, The Effects of Increased Learning Time on Student Academic and Nonacademic Outcomes: Findings from a Meta-analytic Review (Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Appalachia, 2014).

Review of State Evaluations and Academic Literature



the broader range of out of school time or extended learning time programs covered in that study, we only searched for studies of afterschool programs and studies of 21st Century programs particularly.

We identified 104 papers initially and selected 25 of them on the basis of the following criteria:

• Topic relevance–the studies covered 21st Century programs or afterschool programs that measured student outcomes and had some instructional component (i.e. programs were not only a study hall or a sports program).

• Timeframe relevance–studies must have been published since 2012, in order to find studies that may not have been included in Education’s 2014 study, which examined studies published since 1998.

• Publication status–studies were in their final form, not drafts.

• Sample relevance–studies looked at afterschool programs in K-12 settings in the United States. Foreign school systems were excluded.

• Design relevance–studies used experimental or quasi-experimental designs with well-formulated comparison groups or controls; or they used some statistical method to account for other plausible influences on the outcomes of the study.

Next, two analysts reviewed those studies a second time, retaining 13 studies that had a sample size above 100 students in the intervention group and evaluated more than one program or intervention site. The analysts further narrowed this sample to six studies, which we selected because they were focused on 21st Century programs. In addition to reviewing Education’s 2014 meta-analysis, in our initial search of 104 papers, the analysts also identified two additional meta-analyses focused on afterschool programs, which we included in our sample because they were of appropriate quality and scope to assess afterschool program effects. We then conducted detailed reviews of the studies. These reviews entailed an evaluation of each study’s research methodology, including its research design, sampling frame, selection of measures, data quality, limitations, and analytic techniques, as well as a summary of its major findings. We also assessed the extent to which each study was relevant to assessing what is known about the effectiveness of 21st Century programs. Three studies had major research design limitations resulting from the lack of a rigorously formed comparison group. As such, these studies were not able to demonstrate whether the programs were responsible for the effects being measured or whether other factors may



have contributed. After eliminating the three studies with major research design limitations, six studies remained in our review.

For all four objectives, we reviewed agency documents and interviewed agency officials and researchers who had conducted work on the 21st Century program. To examine how states and other entities have used 21st Century funds, we reviewed Education’s 2014 annual characteristic report for the 21st Century program. The reports provide summaries and analyses over time, based on program data collected from states in school years 2001-2002 through 2012-2013 using data from the Profile and Performance Information Collection System. In order to assess the reliability of the data underlying the reports, we reviewed agency documents regarding this data system, spoke with agency officials, and conducted electronic tests of data from the system for select data elements. We determined that the underlying data summarized and analyzed in Education’s 2014 annual characteristic report was sufficiently reliable for our purpose of reporting on program characteristics.

To examine Education’s management of program data, we also reviewed Education documents including budget justifications and documents on the 21st Century data systems such as user guides, technical assistance documents for states, data dictionaries, and a 2013 Education Office of Inspector General report on federal and state oversight for the 21st Century program.

To examine Education’s technical assistance, we reviewed Education’s program guidance to states, monitoring protocols and tools, and annual grantee satisfaction survey results which generally includes Education’s largest grant programs including the 21st Century program. In order to compare the program to established criteria, we reviewed relevant legislation; Office of Management and Budget guidance; federal internal control standards; and past GAO work pertaining to leading practices in performance management, information quality, program evaluation, and federal collaboration for grants management.

We interviewed federal officials from Education’s Office of Academic Improvement, within the Office of Elementary and Secondary Education, regarding data management and use, research and evaluation, and program challenges. In addition, we interviewed stakeholders of the 21st Century program, including contractors and researchers. Finally, we observed Education’s annual Summer Institute, where Education

Review of Agency Documents and Interviews with Agency Officials



provides technical assistance and training to state, sub-grantee, and center officials.

To learn about program activities and challenges, we conducted site visits to four states—Idaho, Massachusetts, Rhode Island, and Texas—to visit 21st Century centers and speak with program staff and state officials. We selected these states for diversity in geographic region, 21st Century formula grant funding levels to states, and student demographics. In each state, we interviewed the 21st Century state coordinator and other officials responsible for the program. In total, we visited 13 centers in the four states. During these visits, we interviewed center staff and observed afterschool program activities.

Information we gathered on our site visits represents only the conditions present in the states and local areas at the time of our visits. Furthermore, our fieldwork focused on in-depth analysis of only a few selected states. On the basis of our site visit information, we cannot generalize our findings beyond the states we visited.

We conducted this performance audit from April 2016 to April 2017 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Site Visits

Appendix II: 21st Century Grant Award Amount by States, Fiscal Year 2016


Table 1: 21st Century Grant Award Amount by States, Fiscal Year 2016

State Educational Agency (SEA) Grant award amount (in dollars) California 132,439,027 Texas 101,389,315 New York 84,279,065 Florida 61,414,141 Illinois 50,808,494 Ohio 43,888,443 Pennsylvania 42,558,875 Georgia 39,347,084 Michigan 38,833,081 North Carolina 32,539,202 Arizona 24,696,549 New Jersey 23,666,114 Louisiana 22,316,104 Tennessee 21,760,677 Indiana 20,236,679 Missouri 18,194,441 Virginia 18,141,534 Kentucky 17,188,889 Alabama 17,135,242 South Carolina 16,787,291 Washington 16,694,742 Massachusetts 16,671,886 Wisconsin 16,137,201 Maryland 15,604,645 Mississippi 14,134,129 Arkansas 12,195,332 Oklahoma 11,926,077 Colorado 11,925,141 Oregon 11,429,471 Minnesota 11,253,198 Nevada 9,133,188 Connecticut 8,966,295 New Mexico 8,392,219 Kansas 8,286,212 Utah 6,982,788




West Virginia 6,849,474 Iowa 6,572,166 Maine 5,643,198 New Hampshire 5,643,198 Rhode Island 5,643,198 Vermont 5,643,198 Delaware 5,643,198 DC 5,643,198 Nebraska 5,643,198 North Dakota 5,643,198 South Dakota 5,643,198 Alaska 5,643,198 Hawaii 5,643,198 Idaho 5,643,198 Montana 5,643,198 Wyoming 5,643,198

Source: GAO analysis of Education documents. GAO 17-400

Appendix III: Comments from the U.S. Department of Education



Appendix IV: GAO Contact and Staff Acknowledgments


Jacqueline M. Nowicki, (617) 788-0580 or [email protected]

In addition to the contact named above, Elizabeth Sirois (Assistant Director), Sheranda Campbell (Analyst-in-Charge), Lucas Alvarez, Ashley Chaifetz, Jamila Kennedy, and Kelsey Kennedy made significant contributions to this report. Assistance, expertise, and guidance were provided by Susan Aschoff, Carl Barden, Deborah Bland, James Bennett, Angie Jacobs, Thomas James, Kirsten Lauber, Ben Licht, Edward Malone, John Mingus, Amy Moran Lowe, Sara Pelton and James Rebbe.

Appendix IV: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

(100498)


The Government Accountability Office, the audit, evaluation, and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability.

The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO’s website (http://www.gao.gov). Each weekday afternoon, GAO posts on its website newly released reports, testimony, and correspondence. To have GAO e-mail you a list of newly posted products, go to http://www.gao.gov and select “E-mail Updates.”

The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s website, http://www.gao.gov/ordering.htm.

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537.

Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information.

Connect with GAO on Facebook, Flickr, LinkedIn, Twitter, and YouTube. Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts. Visit GAO on the web at www.gao.gov and read The Watchblog.

Contact:

Website: http://www.gao.gov/fraudnet/fraudnet.htm E-mail: [email protected] Automated answering system: (800) 424-5454 or (202) 512-7470

Katherine Siggerud, Managing Director, [email protected], (202) 512-4400, U.S. Government Accountability Office, 441 G Street NW, Room 7125, Washington, DC 20548

Chuck Young, Managing Director, [email protected], (202) 512-4800 U.S. Government Accountability Office, 441 G Street NW, Room 7149 Washington, DC 20548

James-Christian Blockwood, Managing Director, [email protected], (202) 512-4707 U.S. Government Accountability Office, 441 G Street NW, Room 7814, Washington, DC 20548

GAO’s Mission

Obtaining Copies of GAO Reports and Testimony Order by Phone

Connect with GAO

To Report Fraud, Waste, and Abuse in Federal Programs

Congressional Relations

Public Affairs

Strategic Planning and External Liaison

Please Print on Recycled Paper.

http://www.gao.gov/

http://www.gao.gov/

http://www.gao.gov/ordering.htm

http://facebook.com/usgao

http://flickr.com/usgao

http://www.linkedin.com/company/us-government?trk=cp_followed_name_us-government

http://twitter.com/usgao

http://youtube.com/usgao

http://www.gao.gov/feeds.html

http://www.gao.gov/subscribe/index.php

http://www.gao.gov/podcast/watchdog.html

http://www.gao.gov/

http://blog.gao.gov/

http://www.gao.gov/fraudnet/fraudnet.htm



