virtual school symposium 2008 1 pre-conference program evaluation in online learning

Virtual School Symposium 2008

1

Pre-Conference Program

Evaluation in Online Learning

2

Pre-Conference on Evaluation in Online Learning

Presenters Talbot Bielefeldt, ISTE Martha Donaldson, Alabama

ACCESS Mickey Revenaugh,

Connections Academy Donna E. Scribner, VHS Inc. Moderator: Tom Clark, TA

Consulting Special Thanks to Brian

Lekander,OII

3


What are your interests? What’s your interest in the topic

of evaluating online learning? What's your role in online

learning? What do YOU hope to learn

about evaluating online learning today?Please write down 1-2

questions that you hope will be answered.

4


Agenda1:00 Overview & Introductions

1:10 Preparing to Evaluate

2:20 Structured Discussion

3:00 Break

3:15 Lessons Learned & Next Steps

4:10 Structured Discussion

4:50 Summing Up

5


Office of Innovation and ImprovementU.S. Department of Education

.

Evaluating Online

Learning: Challenges

and Strategies for

Success

The latest guide in the Innovations in Education series

Office of Innovation & Improvement, U. S. Department of

Education

6

Pre-Conference on Evaluation in Online LearningWhy Did OII Prepare This Guide? Continued skepticism in some

quarters about quality of online learning

Desire for greater accountability in Gov’t funding programs

Concern that innovation often outpaces what we know about educational technology

Evaluators of federally-funded projects are sometimes unfamiliar with online technologies

7

Pre-Conference on Evaluation in Online LearningGuide is Based on Case Studies Alabama ACCESS Algebra I Online (Louisiana) Appleton eSchool (Wisconsin) Arizona Virtual Academy Chicago Public Schools Virtual

High School Digital Learning Commons

(Washington) Maryland Public Television’s

Thinkport

8


Who is this Guide For? Online program administrators

who need to think strategically about evaluation – and how they will use it as their program evolves

Evaluators who are not very experienced with online learning and the challenges it presents

9

Pre-Conference on Evaluation in Online LearningWhat is the Guide’s Focus? Evaluation Challenges Commonly

Encountered in Online Evaluations

Instructive Examples of Responses to these Challenges

10

Pre-Conference on Evaluation in Online LearningEvaluation Challenges Featured Meeting the Needs of Multiple

Stakeholders Building on the Existing Base of

Knowledge Evaluating Multifaceted Online

Resources Finding Appropriate Comparison Groups Interpreting the Impact of Program

Maturity Translating Evaluation Findings into

Action

11

Pre-Conference on Evaluation in Online LearningSome Things OII Has Learned Evaluation of online learning is

inherently political Programs will always be pressed for

information/results before they would like to be

It’s best to be proactive by anticipating the needs of your stakeholders

You won’t always be able to control the spin on your results

It’s helpful to think of evaluation as an ongoing process

12


Where Can I Get the OII Guide?

To view online: http://www.ed.gov/about/pubs/intro/innovations.html

From Ed Pubs:Order online at http://www.edpubs.gov or call 1-877-433-7827 (request order number ED004344P)

13


Office of Innovation and ImprovementU.S. Department of Education

.

Quality and Effectiveness in Online Learning

Issues Brief

Research Committee

North American Council for Online

Learning

14

Pre-Conference on Evaluation in Online LearningNACOL Research Committee Quality and Effectiveness in K –

12 Online Teaching Research Based Practices Practices currently adopted by

Online course providersPractices and Policies for K-12

Online Teaching and LearningOnline Professional Development

Standards Across North AmericaFeatures of Teaching in Virtual

Schools

15

Pre-Conference on Evaluation in Online LearningNACOL Research Committee Online Teacher Support Programs:

Mentoring and Coaching Models Description of the Mentoring/Coaching

relationships in vignettes from perspectives of several virtual high schools

Alabama ACCESS Distance Learning Colorado Online Learning Florida Virtual School Idaho Digital Learning Academy Mississippi Virtual School Missouri Virtual Instructional Program Tennessee: e4TN Virtual High School Global Consortium

16


Introduction Why evaluate online learning? To demonstrate the value or

worth of your program To improve your program over

time To document participant

outcomes related to program goals

To meet stakeholder interests/accountability needs

17


Introduction What happens when evaluation is

neglected? Program set up without clear goals Focus on activities and simple outputs Desired change in participant outcomes

undefined Data essential to studying success

remains undefined, ungathered Focus on anecdotal evidence,

testimonials Stakeholder information needs

neglected Program unable to demonstrate value

or worth

18


Preparing to Evaluate How can you set up your program to

study quality/effectiveness over time? Plan for evaluation early Think about program goals and how to

show you’ve achieved them Thinking at program inception about:

What outcomes are desired?What data need to be collected?What program elements/policies

would support success?

19


Preparing to Evaluate Question posed to panelists

(program managers and evaluators):

Please think back to the start of the online learning program. What did you do to prepare to evaluate quality, effectiveness, and impact?

20


Alabama ACCESS(Alabama Connecting Classrooms, Educators,

and Students Statewide)

Evaluation

Results and IssuesMartha Donaldson ([email protected])Alabama Department of EducationTalbot Bielefeldt ([email protected])International Society for Technology in Education

Preparing to Evaluate

21


ACCESS—Focus on Vision/Mission/ Goal

Vision

The State of Alabama will provide equal access to high quality instruction to improve student achievement through distance learning opportunities.

Mission

To provide an infrastructure that delivers quality learning opportunities for all citizens

Goal

To provide additional educational offerings for all Alabama public high school students


22


Look at Specific Objectives to be Met

To Provide:

Equal Access to High Quality Instruction An Infrastructure That Delivers Quality

Learning Opportunities Greater Equity for all Alabama Public High

School Students Through Cutting-Edge Technology

Wide Range of Courses Available to Relatively few Alabama Students Today (“Advanced Diploma” Courses, Advanced Placement Courses, Additional Course Offerings, Remediation and Supplemental Resources)


23

Pre-Conference on Evaluation in Online LearningACCESS—Don’t Forget the Big

Picture and its Impact on the Evaluation

ACCESS Delivers:Not Just Virtual Online (Asynchronous) CoursesNot Just IVC (Synchronous) Courses

But Utilizes: A unique blended approach to teaching and

learning utilizing 21st century equipment that breaches the disconnect between the traditional and online learning environments by using different delivery models.

Impact for Evaluation:Possibility of Comparing the Effectiveness of Synchronous and Asynchronous Course Delivery


24


ACCESS—Look at Program Growth/Expansion

Enrollments Sites

Summer 2004 - Spring 2005 300 0 Summer 2005 - Spring 2006 2,064 24 Summer 2006 - Spring 2007 7,289 70 Summer 2007 - Spring 2008 18,955 170

As Reported in SREB State Virtual School Reports

(Is the program growing at an acceptable rate?)


25


ACCESS—Established Need for Evaluation—Quality, Effectiveness, Impact

Built Into Program Design Purpose

Provide Evidence that Goals of Initiative are Met

Gauge Satisfaction with Courses Recommend Changes in Procedures and

Resources to Improve and Strengthen Program and Increase its Positive Impact

Determine Effectiveness of Regional Support Centers


26


ACCESS—Preparation for Evaluation

Development of RFP at Beginning of Initiative

Development of Evaluation Plan with ISTE Identification of:

Questions to be AnsweredData that will be NeededData that will not be AvailableEvaluation Methods to be Used

Development of Data Collection Instruments


27


ACCESS—Methods

Interviews: What are the issues? Surveys: How do people feel about

the issues? “State-istics”: How has the system

grown? Outcomes: How well does the system

work to meet its goal of access to quality courses for all students?


28


ACCESS—What Worked

Financial support from the state. Rapid expansion of the infrastructure. Highly rated Regional Support

Centers. Thousands of students take classes

that would be otherwise unavailable. Teaching practices change with

increased technology integration and student-centered pedagogy.


29


ACCESS—Strategy-in-Progress

Seek out teachers with virtual and F2F students.

Compare outcomes for students in similar classes with the same teacher.


30


ACCESS—Areas for Improvement

A percentage of students dissatisfied with various aspects of the system.

Requests for better communication within the system from students, teachers, counselors, and facilitators.

Concerns with student readiness for distance learning.

Technical issues with evaluation.


31


ACCESS—Evaluation Issues

Availability of data (e.g., lack of common end-of-course tests).

Access to data (e.g., Advanced Placement records held by testing vendor).

Technological barriers (e.g., firewalls prevent submission of survey responses).

Research design challenges.


32


ACCESS—Issue: What to Compare?

IVC send/receive sites? (Where sending sites are typically advantaged schools?)

Outcomes for face-to-face/virtual courses? (Where teachers and schools are different?)


33


VHS Global Consortium

Evaluating for Continuous Improvement

Dr. Donna E ScribnerChief Learning [email protected]


34



Mission To develop and deliver standards

based student centered online courses to expand students’ educational opportunities and 21st century skills and to offer professional development to teachers to expand the scope and depth of their instructional skills.


35


VHS Believes that:• Student-centered online courses can

be designed and delivered to students to promote a high quality collaborative learning environment.• in which student exchange and interaction is

a valued component of the instructional process.

• Educational opportunity need not be limited by barriers of time/place/lack of qualified faculty.

• Rather, we believe that high-quality education is possible-today-for all students in all locations.

• Online education offers any school with Internet connectivity a wealth of trained, experienced faculty members qualified in numerous disciplines, for teaching a wide array of courses designed to meet the needs of all students. An innovative, standards-based curriculum delivered online offers diverse, exciting learning choices for students, and the opportunity and skills to participate in a national and global community.


36


• Online teaching should augment rather than replace traditional classroom teaching. • The Virtual High School's online courses are a

proven, flexible solution for schools needing an expanded curriculum, teachers seeking new horizons, parents wanting more involvement with their children's education, and a society grappling with ways to offer opportunity to all its citizens.

• The goals of education are advanced best by putting value and service first. • When schools work together in a collaborative

network such as VHS, they become part of an abundant and generous educational community that promotes the affordable sharing of professional resources

VHS Believes that:

37


VHS Global Consortium 1996 – Technology Innovation

Grant 5-year, $7.4 million US Department of Education

Non-profit; non-degree granting Consortium of schools

575+ member schools 30 states; 39 countries 11,000+ students


38


VHS Global Consortium Program Evaluation

Outsider reviewer Quality, Growth, Program Goals Surveys Superintendents, Principals,

VHS Teachers, VHS Students, VHS Site Coordinators

Published Annually Available to Public via Website

(www.goVHS.org) Quality Benchmark Indicators

(QBIs)


39



QBIs Growth Indicators measured

against Growth Goals Quality Indicators – tie back to

Mission & Beliefs Quality of Courses Quality of Professional Development Quality of Services & Program


40



# of VHS Schools

28 30

87

232183 174

207

278

345

461

544

0

100

200

300

400

500

600

97/98 98/99 99/00 00/01 01/02 02/03 03/04 04/05 05/06 06/07 07/08


41



# of VHS Courses

30 37

94

156134

121

154

184

216241

293

0

50

100

150

200

250

300

350

97/98 98/99 99/00 00/01 01/02 02/03 03/04 04/05 05/06 06/07 07/08


42



# of VHS Students

710 943

2516

3675 3736 4063

5069

6114

7604

911110111

0

2000

4000

6000

8000

10000

12000

97/98 98/99 99/00 00/01 01/02 02/03 03/04 04/05 05/06 06/07 07/08


43



Course Quality Benchmark Indicators

0%

20%

40%

60%

80%

100%

00/01 01/02 02/03 03/04 04/05 05/06 06/07

AP exam pass rate

AP exam take Rate

Course completion rate

Credit recovery rate


44



Course Completion Rate over Enrollments

0%10%20%30%40%50%60%70%80%90%

100%

00/01 01/02 02/03 03/04 04/05 05/06 06/07

0

2000

4000

6000

8000

10000

Course completion rate

VHS Students


45



PD Quality Indicators - First-year Teachers

0%

20%

40%

60%

80%

100%

02/03 03/04 04/05 05/06 06/07 07/08

PD Graduation Rate

PD EffectivenessRate


46



Program Services Quality Benchmark Indicators

65%

70%

75%

80%

85%

90%

95%

02/03 03/04 04/05 05/06 06/07

Membership RetentionRate

Course Seat UtilizationRate


47


Connections Academy Connections Academy programs

are mostly full-time and include K-8 Unique research challenges:

Seeking data from younger children and parents; no additional “program ally” such as site facilitator

Unique research benefits: Address whole learner, gather all demographics, include state test results/NCLB data


48


Connections Academy Built into Connections

Academy program: SIS data within our LMS Data analysis: Data views Log: Teacher communication, action Parent Satisfaction Survey StarTracker: Embedded feedback

on every lesson plus school as a whole

Measurable school and company goals


49

Pre-Conference on Evaluation in Online LearningStructured Discussion: Topic 1 Preparing to Evaluate: How can

you set up your online learning program to study quality/effectiveness over time?

5’ —Choose group topics/join a group

25’—small group discussion 15’—report group findings


50

Pre-Conference on Evaluation in Online LearningStructured Discussion: Topic 1Select a Topic, and Join a Discussion Group a. Program goals: The art and science of

setting measurable goals. b. Outcomes evaluation and logic models:

Will your virtual school achieve the outcomes (changes in knowledge, skills and status) you desire? How will you know you're getting there?

c. Demographic data: Why you must collect demographics, which ones matter, and how to do collect these reliably and unobtrusively

d. User feedback tools: Embedding them in your virtual school.

e. Data analysis: Balancing system-generated and user-generated data in analyzing virtual school performance.


51


Break

See you in about 15 minutes!

52


Lessons Learned/Next Steps What are some lessons learned

about effective practices in evaluating online learning?

What can we learn from the results of an evaluation?

How can we use results to improve the program? To inform stake-holders & decision-makers?

53


Lessons Learned/Next Steps Seven evaluation challenges for

online programs (OII, 2008) Meeting needs of multiple stakeholders Building on limited base of knowledge Evaluating programs that lack uniformity Finding appropriate comparison groups Solving data collection problems Interpreting impact of program maturity Translating evaluation findings into action

54


Lessons Learned/Next Steps Some Lessons Learned (OII,

2008) – Evaluations should: Effectively inform stakeholder groups Share tools and research methods Focus on outcomes, not activities Recruit willing research populations

early Obtain data access, or plan to gather it Move from formative to summative Disseminate timely information to

internal & external decision-makers

55


Lessons Learned/Next Steps What can we learn from an

evaluation? Satisfaction measures Quality & effectiveness measures Changes in knowledge & skills via

participation (participant outcomes) How can we use results to

improve the program? To inform stakeholders & decision-makers?

56


Lessons Learned/Next Steps Question posed to panelists

(program managers and evaluators): Based on your experience using or

conducting evaluations of online learning programs, please share:

1) What are some lessons you’ve learned about effective evaluation practices?

2) How has evaluation helped you improve an online learning program or demonstrate its worth?

57


Connections Academy When Evaluation Pays Off

Early MoVIP K-5 Results: User satisfaction high, teachers make the difference – validation of model

Mississippi K-8 Pilot: Even a small, short pilot can be positively revealing if designed with evaluation in mind

Ongoing Parent Satisfaction Surveys: Overall high rates (90%+) persist, and improvements in “iffy” areas absolutely trackable

Lessons &Next Steps

58


Connections Academy Lessons Learned

No substitute for familiarity: Evaluators need to be equipped to really dig into curriculum and platform

Data transmission is an art: Foster friendships between the evaluators and the program data wonks

Positive results are no guarantee: As in Mississippi example – can’t make up for lack of support

Lessons &Next Steps

Lessons &Next Steps

59


Connections Academy Lessons Learned

Transparency takes some getting used to: Educators are not accustomed to having practice so visible

Evaluation is only half the battle: True continuous improvement takes precision and persistence

Patience is a virtue: User satisfaction and academic results may diverge in beginning, but will converge if students stay

Lessons &Next Steps

60


VHS—Lessons Learned Don’t make decisions on a snap-

shot in time… use longitudinal data and look for trends

Evaluation criteria derives from Objectives; Objectives derive from Mission

It may be interesting but does it inform?

Lessons &Next Steps

61


VHS—Lessons Learned 2Continuous Course

ImprovementStudentProfessional Development

Need for Teacher SupportProgressElluminate SessionsFaculty Advising on graduated

scale

Lessons &Next Steps

62


How has the evaluation helped to improve the ACCESS program?

Year I Findings

Course Changes Needed

Implications/Improvements

Course Revisions Made Increased Use of Voice

Tools, Addition of Improved Speaking Assignments and Examples, and Use of Headphones in Foreign Language Courses

More Detailed Alignment and Gap Analysis Process

Addition of Course Development/Revision Component

Lessons &Next Steps

63


Year I Findings

Need for Better/Increased Communication and Interaction Among Teachers, Students, Facilitators, and Support Center Staff

Implications/Improvements Introduction of Regular

Faculty Meetings via Web Conferencing

Assignment of SDE Liaison to Each Support Center Region

Expansion of Contract for Web Conferencing Capability

Additions to Training Agenda Review of Teacher Pay Issues


Lessons &Next Steps

64


Year I Findings

Need for Additional Professional Development and Training Modules

Implications/Improvements Development of Additional

Training Modules Modification of

Professional Development Plan

Addition of SDE Staff Member to Coordinate Professional Development

Establishment of Teacher Mentoring Plan

Development of C.A.S.T. Site for Teachers


Lessons &Next Steps

65


Year I Findings Need for Assistance

With Scheduling, Registration, and Enrollment Issues

Increased Number of Students Not Prepared/Ready for Assigned Class


Development of Training Module for Counselors on the Registration Process

Expansion of Meetings with Counselors (Regional and State Meetings)

Decision to Develop a New Student Registration Site

Onsite Visits and Individualized Telephone Calls to Assist With Process

Further Look at Course Prerequisites


Lessons &Next Steps

66


Year I Findings

Need for Assistance With Technical Issues


Use of SDE and Regional Support Center Helpdesks

Additional School Visits Addition of Staff at SDE Equipment/Connectivity

Checks by SDE, Support Centers, and Alabama SuperComputer Authority

Increased Communication With School Staff

Identification of Key Areas of Concern


Lessons &Next Steps

67


Year I Findings

Need for Increased Number of Responses on Interviews and Surveys


Identification of Reasons for Poor Response Rate Timing Surveys Used Filters/Blocks at

School Level Content/Clarity

Issues


Lessons &Next Steps

68


Access—Focus of Year 2 Report

Essentials for Student Success in Distance Learning Courses

Ability to Meet Stated Objectives Providing Equitable Access to High School

Courses Leveraging Existing Resources Providing Multimedia and Technology

Tools to Enhance Instruction Planning and Implementing a Successful

Statewide Distance Learning Plan

Lessons &Next Steps

69


Access—Focus of Year 2 Report (Cont.)

Program Growth and Quality of Support to Teachers by Regional Support Centers

Satisfaction Level of Stakeholders Recommendations for Revising

and Enhancing Program Outcomes

Lessons &Next Steps

70


Access—Support Centers Training

Ratings High (Mean Between 3.46 and 3.74 on Scale of 4)

Suggestions for Improvement—Inc. into Training• More Focus on Job-Specific Applications• More Time and Hands-on Practice With Appns

Results From Teacher Surveys 90% Agreed or Strongly Agreed With Positive

Statements About Support High Ratings for Training Content, Scheduling,

and Delivery Open-Ended Items All Positive (Availability,

Helpfulness)

Lessons &Next Steps

71


Access—Satisfaction Surveys • Ratings Generally High (Agree/Strongly Agree all Items)

• Concerns• Lack of Prerequisite Skills by Students• Enrollment Process Cumbersome• Need for More Science Courses• School Technology Inadequate• Need for Hands-on Instruction Specific to Counselors’ Jobs

• Implications/Results• Additional Training/Assistance for Counselors• Continuation of Development of New Student Registration System

• Updating of 2 Science Courses, Addition of 1 More

• Awarding of Additional 21st Century Site Grants

Lessons &Next Steps

72


Access—Recommendations for Future

• Increase Teacher Training, Including Additional Hands-on Time With ACCESS Technologies

• Further Develop Ways to Ensure High Interaction Between Teachers and Students (Such as Moving up of Timeline for Blended Model)

• Increase Role of Support Centers in Providing Training (More Targeted Training for Counselors and Facilitators)

• Continue Annual Audits of Support Centers• Continue emphasis on What Data is Available and

How it Can be Obtained• Ensure That New Student Registration & Data

System is Designed to Collect Needed Data and to Integrate Effectively with LMS for Data Collection and Reporting

Lessons &Next Steps

73

Pre-Conference on Evaluation in Online LearningStructured Discussion: Topic 2 Lessons Learned/Next Steps:

What are some lessons learned about effective evaluation practices, and using evaluation results effectively?

5’ —Choose group topics/join a group

25’—small group discussion 15’—report group findings

Lessons &Next Steps

74

Pre-Conference on Evaluation in Online LearningStructured Discussion: Topic 2Select a Topic, and Join a Discussion Group a. Effective evaluation practices: adjusting

evaluation methods to fit program realities. b. Changing course: Early warnings/interim

evaluations/mid-course corrections – when not to wait til the year end evaluation report.

c. Follow-up: How to build an action plan that addresses your evaluation's findings

d. Program design: Re-designing the program to support ongoing evaluation efforts.

e. Dissemination: Effectively communicating evaluation results to different stakeholders

Lessons &Next Steps

75


Summing Up

Summing up by Moderator Summary thoughts of our

panelists on evaluating online learning

76


Thank You! Thanks for your participation in

our pre-conference today!

Tom Clark (moderator)Talbot Bielefeldt

Martha DonaldsonMickey RevenaughDonna E. Scribner

Thanks to OII & NACOL

virtual school symposium 2008 1 pre-conference program evaluation in online learning

Documents