JENNIFER FREEMAN

Online Course DesignEVALUATION AND REVISION

Online Course Design

• Understand the difference between assessment and evaluation

• Define formative, summative and confirmative evaluation and understand the importance of each

• Explore theories and methods of evaluation

• Create a course evaluation and revision plan

Session Goals

Evaluation: measuring the quality and effectiveness of learning materials and activities

Assessment: measuring students’ learning and achievement of goals and objectives

Evaluation vs. Assessment

• Objectives and alignment• Quality of instructional materials, resources,

strategies• Tools and technology• Testing instruments• Evaluation chart

What Do We Evaluate?

Why?

• Uncover problems early on; fix broken stuff• Discover potential usability/accessibility issues• Examine effectiveness and improve functionality• Dynamic nature of online learning

Formative Evaluation of Instructional Materials

What? When?

• An ongoing process, usually done both during development and while being taught

• Asks the question, “How are we doing?”

Formative Evaluation of Instructional Materials

• Who will use this evaluation information?

• How?

• What should be evaluated?

Formative Evaluation

• Do learning activities and assessments align with the learning objectives?

• Do learning materials meet quality standards?

• Are the technology tools appropriate and working properly?

Formative Evaluation: Questions to Ask

• Course development rubrics• Checklists• Focus group feedback

– consistency and repeatability

• Help desk error logs• Students• Faculty notes

Formative Evaluation: Gathering data

• Checkpoint #1 – syllabus, outline and first lesson• Checkpoint #2 – half of the course, viewed on multiple

platforms• Checkpoint #3 – Entire course proofread/edited• Checkpoint #4 – Entire course tech reviewed• Checkpoint #5 – Final check (previous errors)

• Student survey after first three lessons• Instructor survey after first three lessons• Examination of help desk error logs

Sample Formative Evaluation and Revision Plan

Du

rin

g d

evel

op

men

tC

ou

rse

is L

ive

1. Analysis of problems found2. Assign each issue a priority score3. Establish a threshold4. Prioritized list of change requests5. Assign corrections6. Make note of unaddressed issues

Sample Formative Evaluation and Revision Plan

Why?– Examine effectiveness– Improve functionality– Discover causes for failures– Constant maintenance and improvements

What? When?– Usually done after the completion of each semester– Asks the question, “How did we do?”

Summative Evaluation

• Who will use this evaluation information?

• How?

• What should be evaluated?

Summative Evaluation

• Student success– grades– learning activities , assignments and assessments – time

• Instructor and student satisfaction – learning materials– motivation– tools

• Are program/department needs being met – accreditations, prerequisites for other courses,

competencies– Is the course scalable?

Summative Evaluation: Questions to Ask

• Student grades• Student surveys• Instructor satisfaction surveys• Learner self-assessments• Pretest/posttest comparisons• Assessment item analysis• Focus group feedback • Help desk error logs• Discussion forum and chat archives

Summative Evaluation: Gathering data

• Analyze surveys; identify themes or trends• Analyze assessment results• Analyze help desk logs• Examine course archives• Compile list of issues• Research solutions• Assign priority ratings• Assign tasks and establish deadlines

Sample Summative Evaluation and Revision Plan

Why?

• long-term effectiveness• large-scale issues

What? When?

• after the completion of each semester• “How are we doing now?”

Confirmative Evaluation

• Who will use the evaluation information?

• How?

• What is being evaluated?

Confirmative Evaluation

• program/department needs

• trends in student satisfaction

• scalability/sustainability

• tools and technology

Confirmative Evaluation: Questions to Ask

• Program student surveys

• Departmental administrative opinions

• Faculty peer review of learning materials

• Employer surveys

• Retention data

• Help desk logs

• LMS effectiveness study / survey

Confirmative Evaluation: Gathering data

• cost• server space adequate• maintenance needs being met• vendor/software standards• reliability• security• customization• course structure and presentation

Common LMS Evaluation Criteria

• tracking• assessment • collaboration tools • productivity tools • documentation and training• vendor reputation and position

Common LMS Evaluation Criteria

http://www.edutools.info/static.jsp?pj=4&page=HOME

• is it accessible?

• is it scaleable?

• is it sustainable?

Evaluation Activity

Jennifer Freemanjenni.z.freeman@gmail.com