Quality MattersQuality Matters: : Inter-Institutional Quality Assurance Inter-Institutional Quality Assurance

in Online Learningin Online Learning

Teaching with New Technologies

May 2007

Session Goals

• Provide an overview of the critical materials, processes, and administrative elements of the Quality Matters online course quality assurance program.

• Describe the Peer Course Review process.

• Explore lessons learned in creating a quality assurance program and applications of the QM process.

About About Quality Matters Quality Matters

Peer Course Peer Course ReviewReview

FeedbackFeedback

CourseCourse

Course Meets Course Meets Quality ExpectationsQuality Expectations

Course Course RevisionRevision

Instructional Designers

InstitutionsFaculty Course Developers National Standards &

Research Literature

Rubric

Faculty ReviewersTraining

Quality Matters:Quality Matters: Peer Course Peer Course

Review ProcessReview Process

Factors that Affect Factors that Affect Course QualityCourse Quality

Factors Affecting Course Quality

• Course design QM REVIEWS THIS

• Course delivery (i.e. teaching, faculty performance)

• Course content• Course management system• Institutional infrastructure• Faculty training and readiness• Student engagement and readiness

Underlying Principles of QM• The QM toolset and process are:

– based in national standards of best practice, the research literature and instructional design principles

– designed to promote student learning

– part of a faculty-driven, peer review process

• Process designed to ensure all reviewed courses will eventually meet expectations

• Collegial review process, not an evaluation process

For Our Purposes, Quality Is…

• More than average; more than “good enough”

• An attempt to capture what’s expected in an effective online course at about an 85% level

• Based on research and widely accepted standards

85 %

QM Process Provides

• Institutional toolset and process to meet quality expectations:– Online course design– Student learning– Improved instruction– Assessment and feedback loops– Professional development

What’s In It For Institutions …• External validation

• Strengthen accreditation package

• Raise QA as a priority activity

• Access to a sustainable, replicable, scalable QA process

• Inform online course training & practices

• Provide professional development activities

What’s In It For Faculty …• Improve online courses

• External quality assurance

• Review other courses and gain new ideas for own course; expand professional community

• Participation useful for professional development plan and portfolio

What this process is NOT• Not about an individual instructor

(it’s about the course)

• Not about faculty evaluation

(it’s about course quality)

• Not about “win/lose” or “pass/fail”

(it’s about continuous improvement in a supportive environment)

Design vs. Delivery

The faculty member is integral to both design and delivery.

Course Design … is the forethought and planning that a faculty member puts into the course.

Course Delivery …is the actual teaching of the course, the implementation of the design.

QM is about DESIGN - not delivery or faculty performance

Quality MattersQuality MattersToolset Toolset

The InstructorThe Instructor Worksheet Worksheet

Instructor Worksheet

• Includes info such as:– Institutionally mandated objectives,

materials, practices, policies– Materials outside course pages– Types of interaction used & instructor’s

statement on the appropriateness of interaction in the course

– Additional items that may require review

The Rubric The Rubric

The Rubric

• Eight standards: – Course Overview and Introduction– Learning Objectives – Assessment and Measurement– Resources and Materials– Learner Interaction– Course Technology– Learner Support– Accessibility

Key components must align.

What is Alignment?

Critical course elements

work together to ensure

that students achieve

the desired learning outcomes.

The RubricThe Rubric Essential Standards Essential Standards

(14) (14)

General Standard I:Course Overview and Introduction

1.1: Navigational instructions make the

organization of the course easy to

understand.

1.2: A statement introduces the student to the

course and to the structure of the student

learning, and, in the case of a hybrid

course, clarifies the relationship between

the face-to-face and online components.

General Standard II:Learning Objectives (Competencies)

II.I: The course learning objectives

describe outcomes that are

measurable.

II.2: The module/unit learning objectives

describe outcomes that are measurable and consistent with the

course-level objectives.

General Standard III: Assessment and Measurement

III.1: The types of assessments selected

measure the stated learning objectives

and are consistent with course activities

and resources.

III.2: The course grading policy is stated clearly.

III.3: Specific and descriptive criteria are

provided for the evaluation of students’

work and participation.

General Standard IV: Resources and Materials

IV.I: The instructional materials support

the stated learning objectives.

IV.2: The instructional materials have

sufficient breadth, depth, and

currency for the student to learn the

subject.

General Standard V:Learner Interaction

V.1: The learning activities promote theachievement of stated learning objectives.

V.2: Learning activities foster instructor-student, content-student, and if

appropriate to this course, student-studentinteraction.

V.3: Clear standards are set for instructorresponse and availability (turn-aroundtime for email, grade posting, etc.)

General Standard VI:Course Technology

VI.I: The tools and media support the

learning objectives and are

appropriately chosen to deliver the

content of the course.

General Standard VII: Learner Support

• No 3-point elements

This standard has no “essential” 3-point elements because it’s primarily concerned with academic support services, student support services and technical support services….usually thought to be the primary responsibility of the institution and not the individual instructor.

General Standard VIII:Accessibility

VIII.1: The course acknowledges the importance ofADA requirements.

To meet this standard, the course must include BOTHof these elements:

• The course must be offered using software that is accepted as “ADA compliant.”.

AND• The course should include a brief statement that

clearly tells students how to access ADA services at the institution.

The RubricThe Rubric Scoring Scoring

Rubric ScoringStandards Points Relative Value

14 3 Essential

12 2 Very Important

14 1 Important

TOTALS

40 80

• Team of three (3) reviewers

• One score per standard based on team majority

• Assigned point value; not sliding scale

To Meet Expectations…

A course must achieve:

• “Yes” on all 14 of the 3-point “essential” standards.

• A minimum of 68 out of 80 points

68/80 = 85%

Online & Hybrid Courses

• Rubric designed for application to fully online and hybrid/blended courses

• Same set of standards apply to both

• How we achieve the standards may differ

• For hybrids, focus on pedagogical integration of online and F2F components

RecommendationsRecommendations

Peer Course Review Peer Course Review ProcessProcess

The Peer Review Team

• 3 faculty peer reviewers:– must be experienced online instructors– must attend QM training– one MUST be external to the course’s originating

institution– there must be a subject matter expert (SME)

on the team. NOTE: The SME could also be the external reviewer.

AND • Faculty course developer:

– access to rubric prior to review– involved in pre-review discussions– consulted during review

About the Course

• QM is designed to review “mature” courses (taught at least two semesters)

• QM logo indicates year course met expectations

• Triggers for subsequent reviews:• Faculty request• More than 3 years since original review• New textbook or instructor• Professional or accreditation review pending

About the Review

• On average, a course review takes 7-10 hours; active review period approximately 3 weeks

• Factors affecting review time include

• Reviewer familiarity with the discipline• Reviewer familiarity with the CMS• Reviewer familiarity with the QM review

process• Organization of the course

Review Outcome• If meets expectations:

– Recognized by Quality Matters– Notifications distributed– ID support provided if requested

• If does not yet meet expectations:– ID support provided if requested– Team Chair and ID approve revisions– Course meets expectations

Roles and Responsibilities

Faculty DeveloperPeer Reviewers

Team Chair

QM To DateQM To Date

QM in Transition

• 2003 – August 2006– QM project funded by FIPSE grant money– materials and some services freely available

• August 2006 and beyond– QM project funded by MarylandOnline– Some limited materials will be freely available– Other materials available to individuals and

institutions at nominal fees– Institutional membership affords full access to

materials and services

QM to Date

• Overall Participation– Individuals & programs from 150

institutions across 30 states

• Course Reviews

• Peer Reviewer Rubric Training– 900+ trained

Quality MattersQuality MattersApplicationsApplications

Multiple Uses of QM

Reported Uses of QM System:• Guidelines for initial online course development

• Quality assurance of existing courses

• Ongoing faculty professional development

• Institutional reaccredidation packages

• Formation of distance learning policies & steering committees

Thanks to YOU…Thanks to YOU…Quality Matters! Quality Matters!