1 when learners are at a distance, the careful design of assessments is particularly important,...
TRANSCRIPT
1
“When learners are at a distance, the careful design of assessments is particularly important, because society somewhat unfairly imposes higher expectations on the assessment of online learners.”Oosterhof, et al. (2008) Assessing Learners Online
2
Donald Staub, Title III Director Carteret Community College
?
?
?
?
?
?
Assessing Distance Learning: Are they learning what we think we’re teaching?
4
The Lineup• DL @ CCC
• Why Worry about DL Assessment?
• How We’re Doing It
• The DL Program
• DL in Programs & Courses
• Possibilities & Challenges
5
DL @ CCC• 33 programs @ CCC
• Enrollment ~1800
• DL is 10+ years old
Spring 2011 399 total sections 261 seated sections 90 Internet (23%) 40 Hybrid (10%) 8 Web-based
5
6
Why worry about assessing DL?
Student Success Persistence(btw: success vs. retention vs. persistence)
“Good assessment guides good PD/ID”
@ the College & Program-Level:
7
@ the Course-Level:
Generalizability(c.f. Oosterhof, et al, 2008)
“…the degree to which learner performances that are observed generalize to performances that are not observed.”
[i.e. are they learning what we think we’re teaching?]
8
So, how are we doing this?
Caveat: We’re not saying this is right or wrong…it’s just the way that we’re currently approaching
it.
9
“Comparability of distance education programs to campus-based programs and courses is ensured by the evaluation of educational effectiveness, including assessments of student learning outcomes, student retention, and student satisfaction.”
(CS 3.3.1)
COC Distance Education Policy Statementhttp://www.sacscoc.org/pdf/Distance%20and%20correspondence%20policy%20final.pdf
10
The (former) Guidelines
11
12
2009-2012 Strategic Plan
We will create a culture where all institutional actions are focused on improving student learning and success and leading to student goal completion.
Q1: Is the distance education program reflected in the institution’s strategic (or master) plan?
PM 16—Success and Withdrawal Rates by Instructional modalities
13
Q2: Is there evidence that outcomes for the program have been identified?
DL-AO #1: Faculty demonstrate the ability to teach in the online environment in a proficient (student centered) manner.
DL-AO #2: Students have more educational options to pursue degrees, diplomas, and certificates due to expanded online course offerings at CCC.
14
AO #1: Faculty demonstrate the ability to teach in the online environment in a proficient (student centered) manner.
Assessment: Quality Assessment Plan, Success/ Persistence Rates, Student Surveys/Evaluations.
AO #2: Students have more educational options to pursue degrees, diplomas, and certificates due to expanded online course offerings at CCC.
Assessment: The number of internet/hybrid sections offered.
Q3: Is there evidence that the effectiveness of the distance education program is regularly assessed and steps taken for improvement of the program?
15
QAP
16
The QAP(Quality Assessment Plan)
• Developed in-house (using models)
• Peer-reviewed by content specialist
• Stipends (T3)
• All current online courses have been reviewed
• All new online courses must be reviewed/ certified
http://web.carteret.edu/keoughp/TitleIII/FinalQAP.xls
17
The QAP Score Card
18
The QAP Score Card: Learning
19
QAP
Success/Persistence
20
Success - Spring 2010
• 75 sections
• seated: 75% success (persist: 87.1%)
• internet: 66% success (persist: 81.7%)
• hybrid: 73% success (persist: 87.8%)
• FA09 internet: 62 sections; 68% success
• FA08 internet: 61 sections; 67% success
• FA07 internet: 59 sections; 66% success
• FA06 internet: 48 sections; 66% success20
21
QAP
Success/Persistence
Program/Course
Evaluations
22
Surveys & Evaluations: DL Program
23
Surveys & Evaluations: DL Program
23
24
Surveys & Evaluations: Course-Level
25
Q3: Is there evidence that the effectiveness of the distance education program is regularly assessed and steps taken for improvement of the program?
Modality Comparison Grade Distributions Program Reviews ILLOs (Gen Ed outcomes)
26
Modality Comparison
27
Modality Comparison
Institutional-level
28
Modality Comparison
Program-level
29
Modality Comparison
Course-level
30
Modality Comparison
Grade Distributions
31
Grade Distributions institution-level
%s
32
Grade Distributions course-level
33
Modality Comparison
Grade Distributions
Program Reviews
34
Evaluate Instructional Modalities
35
Modality Comparison
Grade Distributions
Program Reviews
ILLOs
36
ILLOs (Gen Ed outcomes)
1. Communication
2. Computer Literacy
3. Critical Thinking
4. Humanities & Fine Arts
5. Information Literacy
6. Personal Growth & Responsibility
37
Computer Literacy - CIS 110
Outcome: Students demonstrate proficiency in the use of computer technology including application software, system software, and internet browsers.
Assessment: Final Exam in CIS 110.
38
Computer Literacy - CIS 110 Section analysis
39
Computer Literacy - CIS 110Same instructor teaching one section of each
40
Q4: Has the institution developed student learning competencies for the courses/programs offered by distance education? If these are the same competencies for courses/programs offered by “traditional” methodologies, is assessment identified for distance learning students separate from students taking courses by “traditional” methodologies?
41
“Comparability of distance education programs to campus-based programs and courses is ensured by the evaluation of educational effectiveness, including assessments of student learning outcomes, student retention, and student satisfaction.”
So, are they learning what we think we’re teaching?
42
Yes! … No?…maybe??
• PLLOs
• CLLOs
And the answer is…
43
PLLOs
44
PLLOs (SLOs)
Same instructor teaching one section of each (common assignment from each section)
Business Administration
• ACC 120
• BUS 110
• ECO 251/252
45
PLLO: preparation of financial statements as part of mid-
term Assessed in: ACC 120
45
46
BUS 110
ECO 250
47
PLLOs
CLLOs
48
Course-level assessment - CLLOs
• ACC 120
• ART 111
• EDU 119
• ENG 111
• HIS 111
• PSY 150 & 241
• RCP 114
• SOC 21048
49
Course-level assessment ~ CLLOs
BUS 110 … Students will be able to:
A: Explain Business Ethics and Social Responsibility
B: Describe the major concepts of business, such as Entrepreneurship, Management, Marketing, Human Resource Management, and Financing
C: Describe Business Globalization
50
51
Use of Results: “The on-line section needs some extra material to ‘reinforce’ or explain globalization.”
52
“Students placed more value on the CMS as an effective learning tool when they perceived the CMS as supporting interactivity among student, instructor, peers, content.” (Chou, Peng, Chang, 2010)
53
ART 111
Grade Evaluation ( Percentages and Points )
AssignmentCategory
Number ofAssignments
% of ea.towardFinalGrade
Total pointsgenerated by ea.
individualassignment
Total % ofCategory inFinal Grade
Total Pointsof Category
in Final grade
WeeklyQuizzes
14 2.5% ea. 80 35% 1,120
DiscussionBoard
16 2.5% ea. 80 40% 1,280
Mid TermEssay
1 10% ea. 320 10% 320
Final Exam 1 15% ea. 480 15% 480
Totals 32 100% 3200
54
A - (100 - 90): - You posted 3 or more times and you completed the Assignment with a well thought through response. (The more you post MEANINGFUL responses over 3 the higher your grade. Though welcome and expected, please understand that a One or Two sentence "feel good" post such as (... Great response Michaela!!) or (... I love how you put the meaning and then went down and talked about them. it made it easy to understand!!) will count for very little towards your grade.
54
55
HIS 111
Essays (online) vs. Research Paper (seated) … same rubric
56
RCP 114
57
Other Possibilities
• Downstream assessment• E-ports
57
58
Challenges
59
“I got rid of [the discussion board]…it’s a pain in the ass.”
“I have to pull teeth to get them to post even when they get a score”
“Value of discussion board is suspect at best.”
60
“My first goal is to get the instructors to agree on common CLLOs. Then we’ll work on common assessments.”
“I don’t know” [what texts and assessments the other instructors are using in this multi-section course]
Outcome-Assessment alignment
61
Donald Staub Title III Director
Carteret Community [email protected]
252- 222-6010
http://ccctitle3.wordpress.com
www.carteret.edu