MASN, Evaluation, and PQAs

• MASN Overview– Systems building

• Quality Framework• Training and Technical Assistance

– Site Visits– Quality Action Plans

• Evaluation– Statewide Evaluation– Local/External Evaluation

• Data Collection– Survey data collection– Kids Care Center data– PQA data collection

MASN builds systems across the state that improve, support, and sustain high quality

afterschool programs.

MASN is one of 40+ Statewide Afterschool Networks (SANs) funded by the Mott

Foundation.

MASN Goals

Goal 1 Foster partnerships at the local, state, and national level to support afterschool professionals and programs

Goal 2 Develop public support and garner resources to advance and sustain afterschool programs

Goal 3 Ensure quality by providing supports to afterschool programs that are necessary to meet and exceed the Missouri Quality Standards

Two Avenues to Meet our Goals

• MASN Steering Committee

• Funded Projects

MOARC KCC Project LIFTOFF

MOSAC2 Coordination

Integrated PD System

Funding & Sustainability

Public Awareness Public Policy Quality STEM

MASN Steering Committee

• Meets quarterly (December 10th, February?, May ?)

• Sub-committee meetings are open to everyone– 10 a.m.-noon, 50-75 people participating

• Networking lunch – Noon-1p.m., with sub-committee reports

• Steering Committee Leadership Meeting– 1- 3 p.m., just the Steering Committee members– Note: Training option available during this time.

Sample MASN Quarterly Meeting Schedule

Salon A Columbia Meadows Pines Lewis & Clark Timberlands

10:00-10:15 Full group welcome

10:15-12:00Committee Meetings

Quality Public Awareness

Funding & Sustainability

STEM Public Policy/ Advocacy

12:00-12:45 Full group networking lunch

12:45-1:00 Reports from the morning meetings of the sub-committees

1:00-3:00Business Meetings

MASN Steering Committee

Training Session

MO Girls CSI

Who are the Steering Committee members?

– 4-H– ARCHS– Boys & Girls Clubs– Charter Schools– Child Care Aware® of

Missouri– DESE– DHSS– DSS– Francis Institute– LINC

– Missouri Chamber of Commerce

– MOSAC2

– OPEN Initiative– United for Children– United Way of Greater KC– YMCA– Independent program

representatives that are serving as committee chairs

Quality Framework

Afterschool Logic Model

Afterschool Program Quality

Afterschool Program Quality

Youth Outcomes

Youth Outcomes

College and

Career Readiness/Success

College and

Career Readiness/Success

AS Staff Skills

AS Staff Skills

AS Program Structure

AS Program Structure

Training

Training

Coaching

Coaching

Self-assessment  

Self-assessment  

Weikart Center for Youth Program Quality Youth Program Quality Improvement

(YPQI)

Assess – KCC, Surveys, PQAs

• Youth Survey• Family Survey• Coordinator/Director

Survey• Site Staff Survey• School Administrator

Survey• Community Partner

Survey

Plan – TA Visits & APWD

• Grantee Level Action Plans– Coaching visits

• Site Level Action Plans– Advanced Planning with Data (Y2, Y4)– Coaching visits

Do – Coaching and Training

• Coaching visits• Methods Trainings• Summits • Webinars• Statewide

Conferences

Missouri Afterschool Resource Center (MOARC)

Provides training and technical assistance to 21st CCLC and SAC grantees.

Supports successful implementation of grants and quality improvement process.

Technical Assistance

• 2 Program Visits per grantee– Program Visit Discussion Checklist– Program Level Action Plan

• 2 Site Visits per site (many grants have multiple sites)– Site Visit Discussion Checklist, Focused

Observation, Feedback– Site Level Action Plan

Training

• Advanced Planning with Data– Fall after PQA (Y2, Y4)

• Four clock-hour webinars this Spring– Parent and Family Involvement– Community Partnerships– Marketing Your Program– Afterschool Enrichment

• Methods Trainings– Based on action plans developed at the Advanced Planning

with Data session– This year, one training per grant (Cohort 6 and 7)

• Cohort 8 and SAC may attend if space available

Evaluation

Statewide and Local (Outside) Evaluation

Evaluation Work

• MASN is coordinating the new Statewide and Local Evaluation and processes– Sub contracting with Weikart Center and OSEDA

for survey development, data analysis, and report writing

• Quality Committee - looking at the resources being available for all programs

– Providing training/certification of external evaluators

Coordinated Evaluation

• Local evaluation parallels statewide evaluation– Same data and framework for all grantees

• Same data used for quality improvement efforts and evaluation efforts

PQA/Survey Report Fee

• $1500 per site to 4-H Center for Youth Development– Data:

• PQA(s)• Surveys

– Reports: • PQA(s)• Weikart Center Leading Indicators (Goal 2)• Missouri College & Career Readiness (Goals 1 & 3)

Logic Model

Afterschool Program Quality

Afterschool Program Quality

Youth Outcomes

Youth Outcomes

College and

Career Readiness/Success

College and

Career Readiness/Success

AS Staff Skills

AS Staff Skills

AS Program Structure

AS Program Structure

Training

Training

Coaching

Coaching

Self-assessment  

Self-assessment  

Goals

• Goal 1: Support or increase student achievement and sense of competence in the areas of reading/communication arts, mathematics, and science.

• Goal 2: Develop and maintain a quality program that includes a safe and supportive environment, positive interactions, and meaningful opportunities for engagement.

• Goal 3: Enhance youth’s college and career readiness skills and behaviors, including positive school behaviors, personal and social skills, and commitment to learning.

Data Sources used for the Statewide Evaluation

• Program Quality Assessments (PQAs)• Surveys • Kids Care Center (KCC) Data• DESE Core Data

Program Quality Assessments (PQA)

• Developed by the Weikart Center for Youth Program Quality– School Age Program Quality Assessment (SAPQA)

Walk-Through Method– Youth Program Quality Assessment (YPQA), with

STEM Add-on• On-site assessment by trained, reliable

external assessors

Surveys

• Survey links sent by e-mail March 2015 to grant contact– One per site – coded with Org ID so it is important to

make sure to use the correct links• Paper survey available for parents and Spanish

parent/youth version• Youth survey requires a student identifier and

“opt out” consent• Weekly e-mail with number of completed surveys

https://mopdtoolbox.org/masn/?survey=School&deptcode=11558

Kids Care Center (KCC)

• Statewide database used to collect youth enrollment and activity participation

• KCC training and technical assistance • Data used for Statewide and Local Evaluation

– KCC ID used for the surveys– MOSIS ID - 9 digit number from DESE

• Data is entered into federal system for reporting

KCC Variables

• Afterschool attendance• Grades • Missouri Assessment Program (MAP) test

scores• Teacher surveys

DESE Core Data

• State student-level education database• Core Data variables

– School day attendance– Behavior (suspension)

Statewide Evaluation

Sneak Peak at FY14 Report

Statewide Results

Data received from 185 sites• 9015 youth• 2880 parents• 242 site coordinators• 1280 site staff• 232 school administrators• 330 community partners

Preliminary Statewide Results: Instructional Quality

Supportive En-vironment

Interaction Engagement1

2

3

4

5

4.27

3.77

3.21

Preliminary Statewide Results: Youth Outcomes

Common Instr

ument

Personal

and So

cial S

kills (

Youth)

Personal

and So

cial S

kills (

Parent)

Commitment t

o Learn

ing (Youth

)

Commitment t

o Learn

ing (Pare

nt)12345

3.24

4.42 4.33 4.44 4.29

Preliminary Statewide Results: Parent Outcomes

Benefits of A

fterschool

School-D

ay Li

nkage

s

Strength

ening Fam

ilies S

cale

1

2

3

4

54.09 4.14

3.77

Predicting Grade Maintenance/Increase

• Calculated multiple regressions predicting maintenance/increase of grades in three subject areas.

• All youth scales and demographic variables were entered.• Significant predictors shown below.

Reading (n = 3383) Math (n = 3351) Science (n = 2656)Time 1 reading grade (lower grade more likely)

Time 1 math grade (lower grade more likely)

Time 1 science grade (lower grade more likely)

Race (nonminority more likely) Grade level (younger children more likely) Gender (girls more likely)Grade level (younger children more likely)

Work Habits scale (higher scores more likely) Grade level (younger children more likely)

Work Habits scale (higher scores more likely)

Math Efficacy scale (higher scores more likely)

Work Habits scale (higher scores more likely)

Youth Engagement and Belonging scale (higher scores more likely)

Technology Efficacy scale (higher scores more likely)

Math Efficacy scale (higher scores more likely)

Statewide Evaluation

• Required by Federal Funding• Used to:

– Refine the grant requirements– Determine training and technical assistance needs– Plan for systems level changes or new development that

will support afterschool sites– Use aggregate data to communicate the importance and

impacts of afterschool

Tying it all Together

• Statewide and Local Evaluation

Afterschool Program Quality

Afterschool Program Quality

Youth Outcomes

Youth Outcomes

College and

Career Readiness/Success

College and

Career Readiness/Success

AS Staff Skills

AS Staff Skills

AS Program Structure

AS Program Structure

Training

Training

Coaching

Coaching

Self-assessment  

Self-assessment  

Goal 2Goal 3

Goal 1

Youth Program Quality Improvement (YPQI)

SurveysKCC

Core DataPQAs

Program Level Action PlanSite Level Action Plan

(Advanced Planning with Data)

Coaching VisitsMethods Trainings

SummitsConferences

Other Trainings

Instrument SelectionConsent

SchedulingReports

Coaching

PQA Specifics

Program Quality Assessments

• Developed by the Weikart Center for Youth Development

• Research based, reliable, and valid tool for self-assessment and external evaluation

• Used in many states and localities

l

Light Green- full-state implementation

Dark Green- place-based implementation

Gold- full-state + place-based

YPQI Quality Improvement Systems

Program Quality Assessments

• Four domains– Multiple

scales in a domain

• Multiple items in a scale

Ones, threes, fives, Oh My!!!

• 1 – Not observed on the day of the observation

• 3– Was observed at least once on the day of the

observation – Examples: once, not all youth, some youth

• 5– Was observed multiple times on the day of the

observation– Examples: all youth had opportunities

PQA Protocols

• School Age Walk Through (SAPQA)– K – 6th Grades– From start of programming, through departure– Limit of 3 hours – Welcome, multiple activities, educational offering,

snack, transitions, departure• STEM Program Quality Assessment (STEM PQA)

– 4th – 12th Grades– Two offerings – one STEM, other on same day

PQA Process

• Schedule the assessment• Confirm with program and assessor• Assessment takes place• Data Entry (2-4 weeks)• Quality Control• Report – available in Weikart Center Scores Reporter • Discuss PQA results with Coach at Program and Site Visits• Advanced Planning with Data – set site level action plans

(Sept/Oct)• Methods Trainings

During the Visit• Please tell your staff that someone is coming to observe• Assessors will dress casually

– 4-H Extension nametag/shirt• They will be a “fly on the wall”

– Once the program starts, they will be observing and taking notes so please do not try to hold a conversation with them. They will confirm with the site director and arrive early to hear about the program/schedule for the day.

• They will go from area to area – may not see everything, but that is ok… it is a “snapshot”– 2-3 hours is usually sufficient to have a good feel for the program even if they do not

observe every specific interaction• Assessors will say goodbye to a staff member, but it may not be the site director, if

not readily available• Assessor may ask on the confirmation call or before the assessment begins which

activities you would prefer to have observed if there are multiple offerings (except if older youth STEM)

Participation Agreement

• Key important policies• Sign and return to Anne Reeder

– Today or by December 15th

Questions?