is it working? assessing hybrid faculty development and the hybrid classroom andreas brockhaus and...
TRANSCRIPT
Is it working?Assessing hybrid faculty development
and the hybrid classroom
Andreas Brockhaus and Carol Leppa University of Washington Bothell
Hybrid Course Development Institute (HCDI) Main goal: Create a peer-reviewed, new or
redeveloped course syllabus for a hybrid format class
Cohort of 12 faculty Provide faculty incentives (currently $1000) HCDI faculty team: Learning Technologies,
Nursing, Library Designed around Community of Inquiry model
HCDI structure evolution
2010 Structure 10-week model
– 3 face-to-face meetings• 3 hours long
– 1.5 to 2 hours online each week
– Up to 4 weeks between face-to-face meetings
– Ran twice– $500 stipend
2013 Structure 6-week hybrid model
– 6 face-to-face meetings• 1 hour long• Met weekly at the same
time – 2 hours online work weekly– Ran twice– $1000 stipend
Moving from 10 to 6 weeks
Made HCDI shorter and more hybrid Moved most content and lecture online More focus on a couple specific tools Focused mainly on social presence More templates and peer reviews
Results
47% full-time participants
79% full-time participants
29% tenure-track
61% tenure-track
10 weeks 6 weeks
53% completed all requirements
96% completed all requirements
hybrid
How to assess impact
Pre- and post-HCDI participant surveys
Faculty interviews (post-HCDI) Student evaluations
Feedback
Participants found the HCDI useful Being a student was helpful Interacting with colleagues and instructors was invigorating Experiencing a hybrid course and defining hybrid was useful Wanted even more technology focus Challenges teaching hybrid included:
– Technology– Course design– Time out of class
Feedback
“There is no real substitute for an institute of this nature that asks participants to work through the many stages essential to (course) development.”
“Being a student in the class taught me to think like a student.”
Course Evaluations
Do Standard course evaluations reflect hybrid course COI evaluations?
Completed and compared COI and standard course evaluation form X
N=371 cases– 19 class sections Sum 12 thru Spr 13– 8 to 40 evaluations per section
Garrison, Vaughn; Blended Learning in Higher Education (2010)
Compared Factor AnalysisCOI confirmatory Factor Analysis 3 factor model confirmed 60% variance explained
Teaching Presence
Social Presence
Cognitive Presence
Factor analysis Form X 3 factor model 71% variance explained
Effective content learning– Content– Instructor action
Effective Communication– Relative to expectations
Student reflection– Effort and engagement
Combined COI and Form X Factor Analysis 4 factor model – 62% of variance
Teaching Presence/Instruction Cognitive Presence/content Social Presence as separate factor (COI only)
Student self reflection as separate factor (Form X only)
Correlations of items: Form X with COI questions Highly skewed data for both standard and
COI evaluations (any > than .15=“statistically significant”)
Teaching Presence– 7 items with r= .55 to .67
Cognitive Presence– 5 items with r = .50 to .60
Social Presence– 5 items with r = .25 to .39
Global course evaluation scores/Teaching Presence All forms N=1232 Mean(SD)
5-point scale questions
HCDI hybridn=32
Other hybridn=21
All other n=1179
Course as a whole 4.3(.50)
4.1(.73)
4.1(.63)
Course content 4.3(.42)
4.1(.72)
4.1(.60)
Instructor’s contribution
4.5(.46)
4.3(.68)
4.3(.63)
Instructor’s effectiveness
4.4(.48)
4.2(.85)
4.2(.71)
Comb items 1-4 4.4(.45)
4.4(.45)
4.2(.62)
CEI (7-point scale)
5.0(1.10)
5.2(.52)
4.9(1.20)
Social Presence proxy meansN=896 course sections Forms A&X Mean(SD)
7-point scale questions
HCDI alumsN=31
Hybrid NON-HCDIN=20
All other coursesN=845
Sessions engaging 6.4 (.49)
6.1 (.80)
6.2 (.60)
Student participation encouraged
6.5 (.49)
6.4 (.74)
6.4 (.56)
Help available 6.3 (.56)
6.3 (.69)
6.3 (.54)
Learning concepts in course
6.4 (.51)
6.1 (.84)
6.2 (.65)
Applying material in real world
6.4 (.46)
6.2 (.80)
6.3 (.60)
Cognitive Presence proxy means Form X only 101 Sections Mean(SD)
7-point scale questions
HCDI alumsN=13
Hybrid NON-HCDIN=11
All other courses
N=77
Learn conceptual knowledge
6.4(.63)
5.9(.99)
6.2(.71)
Dev appreciation for field
6.5(.54)
6.1(.95)
6.4(.70)
Understand material in field
6.2(.74)
5.9(.94)
6.3(.62)
Solve problems in field
6.3(.59)
5.8(1.0)
6.2(.60)
Gen intellectual development
6.4(.66)
6.0(.99)
6.4(.57)
Global course evaluation questions Pre- Post-HCDI
N=9
Pre-HCDI Mean (SD)
Post-HCDI Mean (SD)
Course as a whole5=highest
4.0 (.51)
4.3 (.60)
Course content 4.0 (.58)
4.2 (.55)
Instructor’s contribution 4.3 (.58)
4.4 (.61)
Instructor effectiveness/content
4.1 (.54)
4.3 (.67)
Combined5=highest
4.1 (.53)
4.3 (.60)
Challenge/Engagement Index7=highest
5.4 (.58)
5.4 (.60)
Social Presence proxy evaluation items
N=9
Pre-HCDI Mean (SD)
Post-HCDI Mean (SD)
Sessions engaging7=highest
6.0 (.42)
6.3 (.70)
Student participation encouraged
6.6 (.33)
6.7 (.27)
Help available 6.4 (.33)
6.5 (.46)
Learning concepts in course 6.2 (.46)
6.5 (.59)
Applying material in real world
6.3 (.35)
6.5 (.50)
Challenges
Identifying and tracking number of hybrid courses
Mixed institutional interest Student evals are outdated
– Inadequate for capturing social presence– Inadequate for capturing online/hybrid info
Questions
Do you have hybrid-specific course evaluations on your campus? If so, how does it differ from other course evaluations. If not is it reasonable to use CoI model to design student surveys?
References sources for COI instrument questions
Garrison, D.R., Cleveland-Innes, M., Fung, T.S. (2010) Exploring causal relationships among teaching, cognitive and social presence: Student perceptions of the community of inquiry framework, Internet and Higher Education, 13, 31-36.
Diaz, S.R., Swan, K., Ice, P., Kupczynski, L. (2010). Student ratings of the importance of survey items, multiplicative factor analysis, and the validity of the community of inquiry survey, Internet and Higher Education, 13, 22-30.
Shea, P. & Bidjerano, T. (2012) Learning presence as a moderator in the community of inquiry model, Computers & Education, 59, 316-326.
Carlon, S., Bennett-Woods, D., Berg, L. et al (2012) The community of inquiry instrument: Validation and results in online health care disciplines. Computers & Education, 59, 215-221.
AppendixTeaching Presence Spearman’s rho = .50 to .65
Form X question The course as a whole
The course content
Instructor’s effectiveness
Instructor explanations
COI Questions Facilitation questions
Facilitation
Design/facilitation
Design/facilitation
AppendixCognitive Presence Spearman’s rho = .50 to .60
Form X questions Learning content
Understanding material in field
Solving problems in field
General intellectual development
COI questions Integration
Trigger events/resolution
Integration/resolution
Integration/resolution
AppendixSocial Presence Spearman’s rho = .20 to .40
Form X questions Class sessions interesting
and engaging
Student participation encouraged
Aware expectations
Apply real world problems
COI questions Affective experience/open
communication
Affective experience/open communication
Affective exp/group cohesion
Affective exp/open communication/group cohesion
AppendixCOI Evaluation questions Teaching Presence
– Design/organization 4 items– Facilitation 6 items
• (teacher-group support)– Direct instruction 3 items
AppendixCOI Evaluation questions Social Presence
• Student-student and student-teacher
– Affective expression 3 items– Open communication 3 items– Group cohesion 3 items
AppendixCOI Evaluation questions Cognitive Presence
– Trigger events 3 items• Activities/assignments
– Exploration 3 items• Breadth of topic discussions
– Integration 3 items• Constructing solutions
– Resolution 3 items• Application