agenda – flp evaluation

Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org

Building an Effective Evaluation of Your Flexible Learning

Program (FLP)

Presenter:Sue Hawks Foster, FLP Education Program Specialist

Georgia Department of [email protected]

(404) 656-2636

1


Federal Regulation Requires Evaluation of the Title I Programs

FLP Evaluation Requirements- Federal, State, and District

Building an Effective Program Evaluation LEA Plan for District FLP Evaluation Performance Based Observation of FLP Most Common Findings FY13 Questions and Answers References

Agenda – FLP Evaluation

2


FEDERAL REQUIREMENTS FOR EVALUATING TITLE I PROGRAMS

3


• Federal regulations requires that a school operating a Title I program annually evaluate the implementation of, and results achieved by, the Title I program

• ESEA Title I program compliance requires written program evaluation procedure and program evaluation summary

Federal Regulation for Title I Programs

4


The Title I program evaluation is an annual review of Title I activities to determine if they are contributing to the desired outcomes:

• Improved student achievement• Effective family engagement• High quality professional development

Federal Requirements for Evaluation

5


Evaluation Components:•Measures efficacy and impact of the Title I FLP program•Documents impact of Title I FLP services on student learning•Informs school and district planning (cycle of continuous improvement)•Is an advocacy tool•Is conducted at end of program year and used for the next years planning


6


Description of:• Data used• Constituents consulted• Process used to arrive at evaluation findings–Previous data– Surveys

• How findings are utilized for planning and improvement


7


Summary of:• Program strengths• Program weaknesses• Subsequent program changes• Reallocation of funds


8


What are the differences and how do they relate to each other?

• Program evaluation– Evaluates Title I program-FLP specifically– Completed after the fact (end of year)– Helps to plan Title I program for next year

including FLP

Needs Assessment & Program Evaluation

9


• Needs assessment– Looks at all school data, including Title I • Schoolwide data in all subject areas • Targeted Assistance data in all subject areas• FLP data

– Completed before beginning of school year– Assesses priority needs in district/schools and

determines where Title I FLP fits in, as well as, other resources

Needs Assessment & Program Evaluation

10


FLP EVALUATION REQUIREMENTS

11


FLP Evaluation Requirements• State Evaluation– Participant Feedback

• Evaluation Question: What is the overall experience of participants with program?

• Data Source: Student and Parent Surveys

– Service Delivery• Evaluation Question: Is the LEAs and programs in compliance

with laws and regulations?• Data Source: Annual monitoring data, Program documentation,

Federal reporting, Public reporting, Technical Assistance, etc.

12


State Evaluation cont’d

• State Evaluation– Effectiveness

• Evaluation Question: Are programs contributing to increased student academic achievement and performance on state education standards?

• Data Source: Student performance on state tests, Pre-post assessment measures of state standards and academic skills targeting by programs, Performance Flag data, and student growth in schools offering FLP

• Evaluation results would be shared with stakeholders and the public and used to inform ongoing program improvement

13



• LEA Evaluation– LEAs will be required to evaluate outcomes

of FLP interventions implemented at schools within their district that have been identified– LEAs will submit their evaluation plan as part

of their overall FLP plan for approval by GaDOE Title I Specialists

14



• LEA Evaluation The LEA must submit the following components of an evaluation plan with their FLP plan in the CLIP for each Priority, Focus, or Alert school within their district required to implement FLP:

– Effectiveness target or overall quantifiable goal of the program– Assessment instrument(s) that will be used to measure this

target/goal– Plan for collecting and maintaining data– Data analysis plan

15


State Evaluation cont’d • What do the LEA evaluation requirements

mean?– Focused, purposeful intervention– Tailor evaluation needs of program to match program goals

and student needs– Expand on broadband state assessments to quantify

benefits of FLP participation– Collect meaningful information for decision making and

continuous program improvement

16


BUILDING AN EFFECTIVE PROGRAM EVALUATION PLAN

17


Building Effective Program Evaluation

• Determining Purpose of Evaluation–Why should we evaluate our program?–What is our program trying to do?–What information do our funders expect?–What questions do we want to answer?

18



• Performance Measures– Measures of effort– Measures of effect

• Strong performance measures should:– Have strong ties to program goals, inputs, and outputs– Be compatible with age and stage of your program– Consider if the data you need are available/accessible– Yield useful information to the program

19



• Assessing Capacity for Evaluation–Who should be involved in the process?–What resources must be in place?–Who will conduct our evaluation?

20



• Selecting Evaluation Design– What type of evaluation should we conduct?• Formative to understand the changes that have

occurred• Summative demonstrate they have achieved the

standard required

– What type of data should we collect?• Qualitative• Quantitative

21



• Selecting Evaluation Design (cont.)–What is our time frame for collecting data?– Causal• Experimental• Quasi-experimental

– Non-experimental• Descriptive• Pre-experimental

22


• Choosing evaluation instruments–Construct –Audience– Format–Reliability–Validity


23


• Analyzing data– How will data be analyzed and shared?– Quantify central evaluation question (and

supporting questions)

• Communicating results– How?–Where?–Who?


24


LEA PLAN FOR DISTRICT FLEXIBLE LEARNING PROGRAM (FLP)

EVALUATION

25


• List the effectiveness target(s) or overall quantifiable goal(s) of your programs

• What are the measureable outcomes that your intervention is designed to improve? Must include at least one academic achievement indicator

LEA Plan for District FLP Evaluation

26


• List the assessment instrument(s) that will be used to measure each listed effectiveness target/goal

• Include a plan for administering assessment instruments and how you will collect and maintain data


27


• When will you administer pre- and post- assessments?

• How will you maintain student assessment information to ensure confidentiality?

• How will you determine program effectiveness based on program goals and measurement instruments listed above?


28


• FLP Program Evaluation – Includes Academic Data and Narrative

• FLP Program Summary Chart• Parent, Staff and Student Summary Results


29


Are graduation rates improving?

Are CRCT scores increasing?

Are EOCT pass rates increasing?

Is our CCRPI increasing?

True Test for FLP

30


PERFORMANCE BASED OBSERVATION (PBO)

31


Performance Based Observation

• FY13 PBO in Review–12 Districts were selected for PBO as a pilot–16 schools were observed• FY14 Review –26 Districts have been selected for review–26 School will be observed

32



• Components of Performance Based Observation(PBO)– Review of Priority, Focus, and Alert School List– Select number of districts to be reviewed based

on the ESEA Flexible Waiver– Selection of schools in district to be monitored

33



• Components of PBO cont’d–Notifications Sent to LEAs• Call to Title I Director• Follow up Letter to LEA – Superintendent,

Title I Director and Principal of selected schools with brief outline of expectations

34



• Components of PBO cont’d• Agenda Developed by Title I Office• Follow up call to LEA by PBO Team

Lead before visit–Addresses and list of staff reviewed –Selection of Teachers/Tutors for

Interview

35



• Actual Visit to LEA– Interview with LEA Staff• Superintendent• Title I Director• School Principal• School FLP Coordinator• School FLP Instructors/Tutors (at least 2)• Other vital staff as needed

36



• Actual Visit to LEA –Classroom Observation -20 minutes per

classroom–Talk with students and staff in classroom–Observe student and FLP staff interactions–Depart Building- Inform building staff we

are leaving the building

37



• Follow-up Letter –Review of the PBO findings for the district–Non-Compliance• If a LEA is found out of compliance the district will

be notified immediately by Program Manger• If corrective action is required it will be provided in

writing and followed up with the FLP Specialist and/or the Program Director for FLP

38


FY13 FLP MOST COMMON FINDINGS

39


FLP’s Most Common Findings

• Models used for FLP– After school programs were used by Middle and

High Schools– Elementary and Middle schools extended the day

or rotated the scheduled day – High schools used Saturday sessions– Summer Sessions were held when extra funds

were available to extend the learning year

40


FLP’s Most Common Findings cont’d

• Technology Usage Increased • Increase Professional Learning• Increased Student Instruction Time• More contact with parents • Reported Increase in Test Scores

41


FLP’s Most Common Findings cont’d• Frustration of operating the program by

some districts because it was so new• Large amounts of carryover because of FLP

funds were not used last year• Special Education Departments not

connecting to FLP in various districts• Lack of staff in some districts to serve

students

42


FLP’s Most Common Findings in Plans• Lack of understanding of FLP purpose• Did not provide performance goals for plans• Determining the needs assessment was not fully

understood to determine subject area• Many did not identify all subgroups within the

schools• Schools did not understand providing data for more

than one year in order to provide a trend

43


FLP’s Most Common Findings in Plans

• Special Education Departments were not really aware of what was going on with Students with Disabilities (SWD) in FLP.

• Student ratio needs to be smaller• Individual Learning Plans needed to be developed and

followed up• Documentation of Multiple Selection Criteria was lacking

in some districts• Lack of understanding Professional Learning

44


QUESTIONS AND ANSWERS

45


References• Afterschool Evaluation 101: How to Evaluate an Expanded Learning

Program, Erin Harris, Harvard Family Research Project, 2011 http://www.hfrp.org/

• Doing What Works Clearinghouse, USED -http://ies.ed.gov/ncee/wwc/

• Preschool for California's Children [email protected]

• Harvard Family Research Project, 2013http://www.hfrp.org/evaluation/overview

• Georgia Department of Education, FLP Report http://www.gadoe.org/School-Improvement/Federal-Programs/Pages/default.aspx

46


Contact Information

Sue Hawks Foster FLP Education Specialist

[email protected]

47