2011evaulating online course quality

Evaluating Online Course Quality for Student Learning and Success

WCET 23rd Annual Conference, Denver, Colorado

• Who are we?  Our mission and goals 

• Community of Inquiry Framework and evolution at APUS 

• Use of data for decision making, continuous improvement, ensuring student 

learning and success 

• Continuously improving the quality of courses and programs through quality 

assurance processes 

• Predictive analytics, semantic analyses, PAR framework

• Discussion/comments/questions

OUTLINE

AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE

AMERICAN PUBLIC UNIVERSITY SYSTEM |  EDUCATING THOSE WHO SERVE

WHO WE ARE

American Public University System has been dedicated to educating those who serve since our doors first opened in 1991. The American Public University System (APUS) is a fully online system, encompassing both American Military University and American Public University. We are regionally accredited by the Higher Learning Commission (HLC) of the North Central Association.

“Educating Those Who Serve”

• Founded as American Military University (AMU) in 1991 by James P. Etter, a  Marine Corps Officer 

• In 2002, AMU expanded into the American Public University System (APUS), adding American Public University (APU)

• Nationally Accredited (DETC) in 1995

• Regionally Accredited (NCA) in 2006 

• Serving over 96,000 students worldwide 

Mission:  To provide quality higher education with emphasis on educating the nation’s military and public service communities by offering respected, relevant, accessible and affordable, student‐focused online programs, which prepare them for service and leadership in a diverse, global society.

STRATEGY:

• Focus on quality and affordability 

• Graduate successful alumni who make important contributions to society and their professions

• Enhance America’s competitiveness by meaningfully increasing the population withcollege degrees

• Collaborate with other universities to create solutions to U.S. higher education challenges

AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE

2011: Celebrating 20 Years of Service and Leadership

5

An educational community of inquiry is a group of individuals who collaboratively engage in purposeful critical discourse and reflection to construct personal meaning and confirm mutual understanding.

The Community of Inquiry theoretical framework represents a process of creating a deep and meaningful (collaborative‐constructivist) learning experience through the development of three interdependent elements ‐social, cognitive and teaching presence. 

Over 2 million students across the world have used this validated instrument. 

COMMUNITY OF INQUIRY (COI)

AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE

“It is nothing short of a miracle that modern methods of instruction have not yet entirely strangled the holy curiosity of inquiry” – Albert Einstein

Social presence is “the ability of participants to identify with the community (e.g., course of study), communicate purposefully in a trusting environment, and develop inter‐personal relationships by way of projecting their individual personalities.” (Garrison, 2009) 

Teaching Presence is the design, facilitation, and direction of cognitive and social processes for the purpose of realizing personally meaningful and educationally worthwhile learning outcomes (Anderson, Rourke, Garrison, & Archer, 2001). 

Cognitive Presence is the extent to which learners are able to construct and confirm meaning through sustained reflection and discourse (Garrison, Anderson, & Archer, 2001). 

Garrison, D. R., Anderson, T., & Archer, W. (2000). Critical inquiry in a text‐based environment: Computer conferencing in higher education. The Internet and Higher Education, 2(2‐3), 87‐105.

AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE

• Phil – sample of COI report

Data Driven Culture

AMERICAN PUBLIC UNIVERSITY SYSTEM |  EDUCATING THOSE WHO SERVE

“Quality begins on the inside . . . and then works its way out.” ‐Bob Moawad

• Internal and external benchmarking

• Data tools for stakeholders ‐dashboards, program and course level fact books, data warehouse, student learning assessment reports.  

NATIONALLY BENCHMARKED INSTRUMENTS

9

National Survey of Student 

Engagement

Proficiency Profile Tests

Major Field Tests 

Community of Inquiry Survey

APUS utilizes a number of nationally benchmarked tests and validated instruments to measure student engagement, student satisfaction, and the achievement of student learning outcomes.

AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE

Direct and Indirect Measures of Assessment

AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE

“Experts often possess more data than judgment.” – Colin Powell

Regular Dissemination of Data to Stakeholders: Dashboards

AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE

“In the spirit of science, there really is no such thing as a "failed experiment." Any test that yields valid data is a valid test.” ‐ Adam Savage 

AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE

Regular Dissemination of Data to Stakeholders: Course and Program Level Fact Books

AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE

Student Learning Assessment Report: Alignment of Student Learning Outcomes

AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE

Student Learning Assessment Report: Culminating Experience

AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE

Student Learning Assessment Report: Key Course Embedded Assessments

AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE

Student Learning Assessment Report: Assessment Data

AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE

Student Learning Assessment Report: Checklist

AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE

Student Learning Assessment Report: Changes Based on Data

PROGRAM REVIEW PROCESS

External  ReviewExternal  Review

AnalysisAnalysis

Program Review Meeting

Program Review Meeting

Three Year Plan

Three Year Plan

Follow‐UpFollow‐Up

Data Collection

Data Collection

AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE

“Quality is never an accident; it is always the result of high intention, sincere effort, intelligent direction and skillful execution; it represents the wise choice of many”William A. Foster 

“One of the great mistakes is to judge policies and programs by their intentions rather than their results.” Milton Friedman

Data Aggregation & Analysis

• Course books

• Electronic resources

• Learning strategies

Library and Learning Resources

• Student learning outcomes

• Instructional strategies

• Evaluation procedures 

• Academic rigor

Curriculum Assessment

• Expert reviewer report

• Industry Advisory Council report

External Reviewer Feedback

AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE

AMERICAN PUBLIC UNIVERSITY SYSTEM |  EDUCATING THOSE WHO SERVE

• Analysis of facultycredentials and expertiseto ensure breadth anddiversity

Faculty

• Student demographicinformation

• Enrollment History

• Growth trends

Students

• Curricular Mapping

• Assessment measures 

• Fact books

• Student learning assessment reports

• reports

Learning Outcomes Assessment

Data Aggregation & Analysis

AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE

AMERICAN PUBLIC UNIVERSITY SYSTEM |  EDUCATING THOSE WHO SERVE

• Program benchmarkingwith similar programsand institutions

Program Benchmarking

• Evaluation of findings

• Program recommendations

• Three year proposedstrategic plan

Program Directory Summary

• Dean’s observations 

• Meeting minutes

Review and Findings

Data Aggregation & Analysis

AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE

COURSE REVIEW PROCESS

• Evolution of program review to course review process

• Response to regulation and oversight

• Evaluation of courses as a means to support student success, learning, and 

engagement

• Systematic and comprehensive method to identify the strengths and areas for 

enhancement or improvement for each course offered by APUS

• Aligned with the Community of Inquiry framework

“Everything can be improved.” ‐ C. W. Barron

“Quality is everyone's responsibility.”‐W. Edwards Deming

• Contact hours calculator ‐ Each class evaluated to ensure it had 45 contact hours of learning and an additional 80 – 100 hours of homework associated with the class for a total of 120+ hours for undergrad classes and 150+ hours for grad classes.  

• Contact hour calculator emerged as a result of evaluation in the School of Public Safety and Health

• The template was adopted and refined for implementation across APUS schools. The calculator provides a tool for parties to calculate contact hours while establishing expectations.

ONLINE CONTACT HOURS CALCULATOR

ONLINE CONTACT HOURS CALCULATOR

ONLINE CONTACT HOURS CALCULATOR

ONLINE CONTACT HOURS CALCULATOR

Predictive Modeling

UI / UX

Semantic Analysis

GAP ANALYSIS

GAP ANALYSIS REPORT

Dr. Karan Powell, Provost and Executive Vice‐President

kpowell@apus.edu

Dr. Phil Ice, Vice‐President, Research and Development

pice@apus.edu

Dr. Jennifer Stephens Helm, Vice‐President, Institutional 

Research and Assessment

jhelm@apus.edu

Feel Free to Contact Us!

AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE