inacol webinar: blended learning program evaluation
DESCRIPTION
This webinar focused on current practices for evaluating program effectiveness, the evaluation tools in use and how blended learning schools analyze multiple sources of data to understand program success. Presenters will lead a discussion of important considerations around the ongoing formative data collected to inform teachers and administrators about what contributes to student success in online courses. The panelists will explore how their programs approach collection of data and what methodology they use to organize and present data for school or district leaders. Speakers: Ernie Silva, Director of External Affairs, SIATech Elizabeth Hessom, Director of Education Services, SIATech Mary Esselman, Deputy Chancellor, Instructional Support & Educational Accountability Education Achievement Authority of MichiganTRANSCRIPT
Approaches to Evaluating Blended Learning Programs
• Mary Esselman, Deputy Chancellor, EAA, Michigan• Ernie Silva, Director of External Affairs, SIATech,
California• Elizabeth Hessom, Director of Education Services,
SIATech
January 2014
Webinar Format• Overview
• Introductions
• Presentation
• Type questions in the chat window
Blended Learning• “a formal education program in which a student
learns at least in part through online delivery of content and instruction with some element of student control over time, place, path, and/or pace and at least in part at a supervised brick-and-mortar location away from home…” - (Horn and Staker, 2013)
Tech-rich = blended
Teaching and Learning• What the student is doing
and where the student is.
What the teacher is doing and where the teacher is.
What and where the content is.
Webinar Focus
• Blended Learning Program Evaluation– What ongoing measures are used to measure
program effectiveness?– What evaluation tools/processes are used by
teachers to determine student and program success?
– What evaluation tools/processes are used by administrators and leadership to determine student and program success?
Today’s Presenters
• Ernie Silva, Director of External Affairs, SIATech, California
• Mary Esselman, Deputy Chancellor, EAA, Michigan
SIATech Overview
• Appropriate Accountability for Drop Out Recovery
Blended Learning &
Competency Based
Job training, mentors
& internshi
ps
= Educated Productive Graduates
+
Adaptive Assessments
+
Our programs are recovering drop-outs. It is that simple.
13,186 standards-aligned diplomas earned since 1998.
Key lessons have been learned in the past 15 years serving a 100% drop-out population.
Key lessons have been learned in the past 15 years serving a 100% drop-out population.
Florida
[N=1241]
California
[N=2110]
Arizona
[N=640]
New Mexico
[N=302]
Arkansas
[N=261]
Average Age on Entry
19.31 19.44 19.81 19.06 18.77
STAR Math EntryG.E. 6.4 [N=1128]
G.E. 6.4 [N=1805]
G.E. 6.0 [N=430]
G.E. 6.1 [N=270]
G.E. 6.0 [N=282]
STAR Reading EntryG.E. 6.9 [N=1133]
G.E. 7.2 [N=1806]
G.E. 6.7 [N=431]
G.E. 7.4 [N=272]
G.E. 6.6 [N=304]
Socioeconomically Disadvantaged 88% 100% 100% 98% 100%
Educational Achievement Authority (EAA) Overview
• Detroit, Michigan
“The mission of the EAA, as a catalyst for change, is to disrupt traditional public schooling and provide a
prototype for next generation learning.”
Year 1 Year 2 Year 3
NGLC Seven Design Principles
Student Centered
High Expectations
Self-Pacing and Mastery Base.d
Blended Instruction
Student Ownership
Financial Sustainability
Scalable
NGLC Seven Design Principles
Student Centered
High Expectations
Self-Pacing and Mastery Base.d
Blended Instruction
Student Ownership
Financial Sustainability
Scalable
EAA – Year 1 – Five Pillars
1.Students are grouped by readiness, not by age or grade.
2.Students create and assume ownership for their respective personalized learning paths and are able to communicate their progress relative to their individualized learning goals.
3.Students are allowed to work at their own pace using a blended delivery system to master rigorous standards aligned to next generation readiness.
4.Students provide evidence of mastery through relevant performance tasks and common assessments.
5.Continuous feedback is provided to students, teachers, administrators and parents.
NGLC Seven Design Principles
Student Centered
High Expectations
Self-Pacing and Mastery Base.d
Blended Instruction
Student Ownership
Financial Sustainability
Scalable
NGLC Seven Design Principles
Student Centered
High Expectations
Self-Pacing and Mastery Base.d
Blended Instruction
Student Ownership
Financial Sustainability
Scalable
EAA – Year 2
• Professional Development was redesigned to follow a blended, personalized delivery model
• Schools began to look at new ways to use time, space, talent and resources.
• PASE, the SCL Village, the Open Math Lab, and the Learning Pods were all created within EAA schools to allow students to move completely at their own pace and to start leveraging time, space, and talent more efficiently and effectively.
• Students began transitioning freely between learning spaces and time. Students are no longer tethered to one classroom and one subject.
NGLC Seven Design Principles
Student Centered
High Expectations
Self-Pacing and Mastery Base.d
Blended Instruction
Student Ownership
Financial Sustainability
Scalable
NGLC Seven Design Principles
Student Centered
High Expectations
Self-Pacing and Mastery Base.d
Blended Instruction
Student Ownership
Financial Sustainability
Scalable
EAA – Year 3:Next Generation High School
• Staffing structures will be redesigned to truly leverage time, talent, and space with a distributed leadership model rather than a traditional principal led school.
• Students will demonstrate mastery of Next Generation Competencies that truly prove they are Next Generation Ready.
• Staffing structures will allow for multiple adults, including full time and part-time teachers, tutors, paras, instructional assistants, mentors and interventions
NGLC Seven Design Principles
Student Centered
High Expectations
Self-Pacing and Mastery Base.d
Blended Instruction
Student Ownership
Financial Sustainability
Scalable
NGLC Seven Design Principles
Student Centered
High Expectations
Self-Pacing and Mastery Base.d
Blended Instruction
Student Ownership
Financial Sustainability
Scalable
“The mission of the EAA, as a catalyst for change, is to disrupt traditional public schooling and provide
a prototype for next generation learning.”
NGLC Seven Design Principles
Student Centered
High Expectations
Self-Pacing and Mastery Base.d
Blended Instruction
Student Ownership
Financial Sustainability
Scalable
NGLC Seven Design Principles
Student Centered
High Expectations
Self-Pacing and Mastery Base.d
Blended Instruction
Student Ownership
Financial Sustainability
Scalable
EAA Year One
Five Pillars:
1.Students are grouped by readiness, not by age or grade.2.Students create and assume ownership for their respective personalized learning paths and are able to communicate their progress relative to their individualized learning goals.3.Students are allowed to work at their own pace using a blended delivery system to master rigorous standards aligned to next generation readiness. 4.Students provide evidence of mastery through relevant performance tasks and common assessments.5.Continuous feedback is provided to students, teachers, administrators and parents.
EAA Year One
Five Pillars:
1.Students are grouped by readiness, not by age or grade.2.Students create and assume ownership for their respective personalized learning paths and are able to communicate their progress relative to their individualized learning goals.3.Students are allowed to work at their own pace using a blended delivery system to master rigorous standards aligned to next generation readiness. 4.Students provide evidence of mastery through relevant performance tasks and common assessments.5.Continuous feedback is provided to students, teachers, administrators and parents.
.
EAA Year Two•Professional Development was redesigned to follow a blended, personalized delivery model
•Schools began to look at new ways to use time, space, talent and resources.
•PASE, the SCL Village, the Open Math Lab, and the Learning Pods were all created within EAA schools to allow students to move completely at their own pace and to start leveraging time, space, and talent more efficiently and effectively.
•Students began transitioning freely between learning spaces and time. Students are no longer tethered to one classroom and one subject.
.
EAA Year Two•Professional Development was redesigned to follow a blended, personalized delivery model
•Schools began to look at new ways to use time, space, talent and resources.
•PASE, the SCL Village, the Open Math Lab, and the Learning Pods were all created within EAA schools to allow students to move completely at their own pace and to start leveraging time, space, and talent more efficiently and effectively.
•Students began transitioning freely between learning spaces and time. Students are no longer tethered to one classroom and one subject.
EAA Year Three
Next Generation High School
•Staffing structures will be redesigned to truly leverage time, talent, and space with a distributed leadership model rather than a traditional principal led school.
•Students will demonstrate mastery of Next Generation Competencies that truly prove they are Next Generation Ready.
•Staffing structures will allow for multiple adult including full time and part-time teachers, tutors, paras, instructional assistants, mentors and interventions
EAA Year Three
Next Generation High School
•Staffing structures will be redesigned to truly leverage time, talent, and space with a distributed leadership model rather than a traditional principal led school.
•Students will demonstrate mastery of Next Generation Competencies that truly prove they are Next Generation Ready.
•Staffing structures will allow for multiple adult including full time and part-time teachers, tutors, paras, instructional assistants, mentors and interventions
SIA Tech Assessment
• Achievement scores on state mandated exams penalize SIATech students, because the majority enter our schools achieving at the 6th grade level in reading & math.
• State tests do not work well for us
• Prescribed time table• Limited administrations• Too hard/too long for too
many
• Achievement scores on state mandated exams penalize SIATech students, because the majority enter our schools achieving at the 6th grade level in reading & math.
• State tests do not work well for us
• Prescribed time table• Limited administrations• Too hard/too long for too
many
SIATech schools needed a different way to demonstrate accountable for student learning!
Advocating for Alternative Accountability
Reaching At Promise Students Association
http://www.rapsa.orgSave the date: November 14-15 San
Diego 3rd Annual Alternative Accountability
Policy Forum
• Adaptive
• Multiple administration times
• Short
• Correlated to state standards
• Reliable
• Measure & report growth and achievement
• Can use results to improve learning and teaching
• Adaptive
• Multiple administration times
• Short
• Correlated to state standards
• Reliable
• Measure & report growth and achievement
• Can use results to improve learning and teaching
Renaissance STAR Math and Reading*
270
299
349
400
Far Below Basic150
Below Basic
Basic
Proficient
Advanced
600
Grade 6 Grade 7
CS
T E
LA
SC
ALE S
CO
RE
305
330
Grade 8
34525 15
• Student learning growth is a statistical measure that estimates how much each student progresses from one test to the next compared to similar students
What is learning growth?
Benefit #1 – Holds schools accountable for the
things they have control over
Benefits of measuring learning growth -
Benefit #2 – Measuring learning growth fosters
student hope and engagement because gains are
based on individual progress – not unrealistic
targetsBenefit #3 – Measuring student learning growth sets
high expectations for every student, because every
student (high, middle, and low achieving) must
increase his or her learning every year
• Interpretation of Individual Student Growth Data
• Individual Student Goal Setting (weekly, biweekly, monthly)
• Student Accountability for Tracking and Adjusting Academic and Learning Progress
• Evaluation of Course Completion and Academic Growth
• Interpretation of Individual Student Growth Data
• Individual Student Goal Setting (weekly, biweekly, monthly)
• Student Accountability for Tracking and Adjusting Academic and Learning Progress
• Evaluation of Course Completion and Academic Growth
Students take ownership of personal learning as schools make efficient use of core processes
Annual Development of SITE Professional Development Plans
• Interpretation of Learning Data to determine general success and gaps in school outcomes.
• Site Goal Setting• Determination of a Site
Professional Development Focus
• Evaluation of Plan Implementation and Results (EOY)
• Interpretation of Learning Data to determine general success and gaps in school outcomes.
• Site Goal Setting• Determination of a Site
Professional Development Focus
• Evaluation of Plan Implementation and Results (EOY)
Annual Development of Individual Staff Member Professional Development Plans
• Interpretation of Learning Data as related to specific content area(s).
• Individual Staff Goal Setting
• Planning Pertinent Training (examples)
• Lesson Plan Development
• Differentiation
• Knowledge Acquisition
• Teaching Strategies
• Data Driven Instruction
• Evaluation of Plan Implementation and Outcomes (EOY)
• Interpretation of Learning Data as related to specific content area(s).
• Individual Staff Goal Setting
• Planning Pertinent Training (examples)
• Lesson Plan Development
• Differentiation
• Knowledge Acquisition
• Teaching Strategies
• Data Driven Instruction
• Evaluation of Plan Implementation and Outcomes (EOY)
• Monthly Content Teacher Conversations Wednesdays 3:30-4:30 pm
Math, ELA, Science, Social Studies, CTSP (4/year)
Sharing of best content practice with a focus on:
- literacy (reading and writing across content areas)
- small group lessons- opening and closing whole group
lessons/activities
• Monthly distribution of active student learning data
• Research into a computer based, individual student reading program (included into 2013-14 budget for all sites)
Multiple Measures Include Gallup Pollof Hope, Engagement and Well Being
Hopethe ideas and energy
we have for the future.
Hope drives:attendance, credits
earned, and high school GPA
Hope predicts: college GPA and
retention
Hope scores are better predictors of college
success than high school GPA, SAT, and
ACT scores.
Ways around a problem
Average student response on a five point scale, where 1=strongly disagree and 5=strongly agree.
Response Average
Possible Student Activities to Impact this Area:During direct instruction problem-solving, ask
students for other ways they would solve the problem or issue.
Remind students of the variety of ways they can get help and support for problem-solving
Help students learn that there are many ways to success and learning (not just innate talent-there are MANY talented but unsuccessful folks)
teach multiple problem solving techniques encourage students to learn how they ‘best learn of
learn best’ provide students with specific strategies to deal with
their ‘blocks’ (e.g. memorizing, planning, organizing, classifying, preparing, …)
Your Reflections / Actions for this Area:
Question Text:
I can find lots of ways around any problem.
(Five point scale)
Question Text:
I can find lots of ways around any problem.
(Five point scale)
HOPE
Engagement the involvement and
enthusiasm for school.
Engagement scores separate
high-performing from low-performing schools
Schoolwork important
ENGAGEMENT
Question Text:
My teachers make me feel my schoolwork is important.(Five point scale)
Question Text:
My teachers make me feel my schoolwork is important.(Five point scale)
Average student response on a five point scale, where 1=strongly disagree and 5=strongly agree.
Response AverageResponse Average
Possible Student Activities to Impact this Area:
Teachers provide frequent, sincere, specific feedback on schoolwork in a variety of ways
Teachers make relevant connections between schoolwork and goal attainment;
Students are provided with opportunities to discuss with teachers and/or mentors how their schoolwork is helping them…
Your Reflections / Actions for this Area:
Well-Being how we think about and experience
our lives.
Well-being is an indicator of how students are currently doing.
Well-being also predicts future success.
A thriving student earns 10% more credits and a 2.9 GPA compared to a struggling or suffering student who completes fewer credits and earns a
2.4 GPA.
Treated with respectWELL-BEING
Question Text:
Were you treated with respect all day yesterday?(Yes or no)
Question Text:
Were you treated with respect all day yesterday?(Yes or no)
Percent who said Yes
Possible Student Activities to Impact this Area:
• ALL staff knows and USES student names• Be and Do what you want to see from students• Explain the reasons/why or why not for
critiques of student work, not just X• Reward and/or acknowledge acts of respect to
build a culture• Demonstrate positive approach to classroom
management and student discipline (preserve dignity)
• Find ways to help students answer their own questions and then honor their success
Your Reflections / Actions for this Area:
EAA Assessment
Program Evaluation Questions
Question 1: To what extent is the student-centered system being executed as planned?
Question 2: To what extent is there sufficient support available to staff, parents, and students who are participating in a student-centered system of education?
Question 3: To what extent is implementation of the student-centered system achieving desired changes in student performance outcomes, socio-emotional well-being and teacher practice?
Question 4: To what extent is the program being scaled in terms of depth, sustainability, spread, shift, and evolution?
4-pronged Assessment System
Individual Growth (Performance Series) Performance Tasks to assess Deeper Learning
(3 pieces of evidence for each Learning Target) Learn (reflection) Practice (Understanding) Apply (real-world connections) Common Assessments
Achievement (State Assessment) Next Generation Ready—Graduation by Defense
Integrated Teaching & Learning
Platform.
Real-time Analytics
• Supports a robust collection of open sourced, licensed, and teacher created resources.
• Learning objects parsed in bite sized chunks.
• Digital assets mapped to standards--local, state, common core.
• Curriculum maps directly linked to the digital assets
• Students have access to choice-based learning pathways.
• Dashboards provide data for students, parents and teachers.
• Data is provided as a prescriptive newsfeed for immediate application.
• Integrated planner, profiles, personalized learning plans with student, teacher, parent, communication tools.
• Peer to Peer support system.
Part 2: Access to Information—monthly, weekly, real-time
Time & Topic Pace/Progress Badges for leadership, scholarship, character,
service personal milestones Student Profile Mastery:
School/Class/Individual by Level/Unit/Standard Mastered/In progress/Not Yet Attempted Digital Portfolio of Evidence
The teaching and learning platform:The student experience…
EDUCATION ACHIEVEMENT AUTHORITY OF MICHIGANEDUCATION ACHIEVEMENT AUTHORITY OF MICHIGAN
Student Progress Page
Creating a personalized learning path
The Teacher Experience: 3-D Classroom Radar Report of Student Productivity—who’s progressing and who is not vs. traditional 2-D, linear gradebook.
Color-coding on students represents self-assessment information
Color-coding on students represents self-assessment information
Click a student to drill down to view real-time personal performance
Click a student to drill down to view real-time personal performance
Classroom Summary ReportTeachers can click to select individual or groups of students to assign supplemental resources and/or send notes.
Teachers can click to select individual or groups of students to assign supplemental resources and/or send notes.
Click a student to drill down to view real-time personal performance
Click a student to drill down to view real-time personal performance
Pie charts indicate mastery of
individual learning targets.Pie charts indicate mastery of
individual learning targets.
Self-assessment resultsSelf-assessment results
Standards Report
Teachers, students and parents can review progress against individual standards as well as access supplemental study resources.
Teachers, students and parents can review progress against individual standards as well as access supplemental study resources.
Teacher Grading and Messaging Center
Item for grading are collected in the to-do list making it easy for teachers to provide grades and feedback.
Item for grading are collected in the to-do list making it easy for teachers to provide grades and feedback.
Just as teachers can monitor real-time student progress they also can monitor
their own progress.
Just as teachers can monitor real-time student progress they also can monitor
their own progress.
Teacher Personal Progress
Lessons Learned - EAA
Part 3: Other Assessments of Progress/Implementation End of Course for non-tested Subjects GRIT Value-added Growth Model for Pay for
Performance S ystem Instructional Practice—scaffolded
walkthroughs Classroom Set-up/Visual Cues Classroom Structures Rituals/Routines Student Ownership/Engagement
Note: We triangulate data across reports/surveys/observations/focus groups
Lessons Learned – SIA Tech
1. A Learning Growth model is imperative for School Evaluation/Monitoring of reengaged students.
2. Graduation Rate Calculations must be appropriate in terms of cohort.
3. Strong advocacy work at the local, state and federal level needs to be a priority.
4. Competency-based education needs engaging, specialized and customized curriculum to truly accelerate learning
5. Staff development and training is crucial to success with a dropout population
6. A robust data system needs to be in place in order to show success in multiple, meaningful ways.
Contact Information
• Mary Esselman, EAA. – [email protected]
• Ernie Silva, SIATech– [email protected]
• Elizabeth Hessom, SIATech– [email protected]