overcoming the teacher’s dilemma (breadth vs. depth)

Overcoming the Teacher’s Overcoming the Teacher’s DilemmaDilemma

(Breadth vs. Depth)(Breadth vs. Depth)The American River College Model The American River College Model of College-Wide Course-Level SLO of College-Wide Course-Level SLO

AssessmentAssessment

**********************Mr. John AubertSLO Assessment Coordinator, Geography

Dr. Yuj ShimizuFaculty Researcher, Psychology

Provide backgroundProvide background Our collegeOur college Our transition to new standardsOur transition to new standards

Share our course-level assessment Share our course-level assessment programprogram Development and implementationDevelopment and implementation Integration with institutional planningIntegration with institutional planning Breadth vs Depth approachBreadth vs Depth approach

Discuss preliminary resultsDiscuss preliminary results

Our Goals TodayOur Goals Today

American River American River College College Background Background

and Contextand Context Located in Sacramento, CA Located in Sacramento, CA (one of four (one of four colleges in the Los Rios CCD)colleges in the Los Rios CCD)

36,646 unduplicated students36,646 unduplicated students 65 academic departments65 academic departments 2,197 individual courses2,197 individual courses 3,473 sections taught (Fall 2009)3,473 sections taught (Fall 2009) 13,318 course level SLOs defined13,318 course level SLOs defined Accreditation site team arrives inAccreditation site team arrives in 5 5

days…days…

The SLO/Assessment The SLO/Assessment Index…Index…

3131The number of times

that the words “assess” and “assessment” appear in the accreditation standards*…

The number of times that the word

“outcome” appears in the

accreditation standards*…

2424

*2006, Accrediting Commission for Community and Junior Colleges

Accrediting

Commission

for

Community

and Junior

Colleges

Accreditation Workshop Today!

AUBERT

American River American River College College Transition to Transition to

New StandardsNew Standards Late 1990s: Blooms taxonomy Late 1990s: Blooms taxonomy utilized in writing course objectivesutilized in writing course objectives

2004: SLO creation/revision 2004: SLO creation/revision embedded in curriculum processembedded in curriculum process

The Outcomes vs Objectives The Outcomes vs Objectives distinction treated as a continuum…distinction treated as a continuum…

OUTCOMES

OBJECTIVESbroad,

overarchingspecific, discrete

American River American River College College Transition to Transition to

New StandardsNew Standardscontinued…continued…

2004-2006: SLO creation/revision continues2004-2006: SLO creation/revision continues 2006-2007: Focus shifts toward assessment2006-2007: Focus shifts toward assessment Spring 2007: Reassigned time allocated to Spring 2007: Reassigned time allocated to

develop assessment programdevelop assessment program Fall 2007: SLO Assessment Coordinator Fall 2007: SLO Assessment Coordinator

appointed (20-40%)appointed (20-40%)

Our Task…Our Task… To develop an SLO Assessment To develop an SLO Assessment

program which would:program which would: Improve teaching and learningImprove teaching and learning Address the needs of a large collegeAddress the needs of a large college Be consistent with our campus Be consistent with our campus

cultureculture Meet accreditation standardsMeet accreditation standards

To survive the process…To survive the process…

Our Guiding Our Guiding Principles…Principles…■Respect faculty workloadRespect faculty workload

■Rely on faculty expertiseRely on faculty expertise■Be flexibleBe flexible■Integrate with existing Integrate with existing processesprocesses■Practice shared governancePractice shared governance■Meet or exceed accreditation Meet or exceed accreditation standards standards

What do they want What do they want from us?from us?■Pervasive, widespread dialoguePervasive, widespread dialogue

■Assessment which is:Assessment which is:■ institutionalized and integratedinstitutionalized and integrated■ systematic and continuous (not systematic and continuous (not

episodic)episodic)

■Data analysis which leads to Data analysis which leads to changechange■Change which improves student Change which improves student learninglearning■Documentation!Documentation!

Our solution…Our solution…■Comprehensive assessment Comprehensive assessment program adopted September program adopted September 20072007

■New standing committee New standing committee formedformed

■Program review streamlinedProgram review streamlined■65 departments divided into 3 65 departments divided into 3

cohortscohorts■Cohort 1 entered 1Cohort 1 entered 1stst year year

(three year cycle)(three year cycle)

■Substantial outreach and Substantial outreach and training !training !

The American River CollegeSix Year

Institutional Planning Cycle

SLO Assessment CohortsSLO Assessment Cohorts

American River CollegeTwo-Part SLO Assessment for

Courses<<< Three-Year Cycle for Departments >>>

Part I

Student Self Assessment

(every course assessed)

Part II

Faculty Designed

Assessment(one course assessed)

YEAR 1

Action Plan

developand

implement

YEAR 2

Action Plan

continue implementatio

n

YEAR 3

Assessment Cycle Assessment Cycle Version 1.0Version 1.0

DENIAL

DEPRESSION

ACCEPTANCE

ANGER

BARGAINING

The ProblemThe Problem: Are students learning : Are students learning what we expect them to?what we expect them to?

The ChallengeThe Challenge: How we can measure : How we can measure this in a this in a sensiblesensible way? way?

SLOs and SLOs and AssessmentAssessment

Our solution: Our solution: Proposed Proposed Model of AssessmentModel of Assessment

To develop a process that would:To develop a process that would: minimize workload minimize workload gather data broadlygather data broadly gather data systematicallygather data systematically be specific to each course’s SLOsbe specific to each course’s SLOs be transparentbe transparent improve teaching and learningimprove teaching and learning

Course Outline of Record

Course SLO’s

Student Self Assessment-Survey Pilot Template

Course SLO’s

Student Self Assessment-Survey Pilot

Positive Ratings!!!

Student Self Assessment Student Self Assessment SurveySurvey Survey piloted Spring 2007.Survey piloted Spring 2007.

Fast and SimpleFast and Simple Results matched teacher predictionsResults matched teacher predictions

2007-2008 (12007-2008 (1stst Cohort): Full term, face Cohort): Full term, face to face courses (43% Return Rate)to face courses (43% Return Rate)

2008-2009(22008-2009(2ndnd Cohort): All courses Cohort): All courses (63% Return Rate)(63% Return Rate)

2009-2010 (32009-2010 (3rdrd Cohort): Now in Cohort): Now in progress (Return Rate ???)progress (Return Rate ???)

Student Self Assessment Student Self Assessment Survey Survey

Adopted Class Climate Software by Adopted Class Climate Software by Scantron to administer surveys, Scantron to administer surveys, store survey data, and to report store survey data, and to report results (centralized approach)results (centralized approach)

Our approach can be easily modified Our approach can be easily modified for a decentralized / departmental for a decentralized / departmental approach approach

Sample Student Self Assessment Survey Sample Student Self Assessment Survey

Survey ResultsSurvey Results

Self assessments/Student ratings Self assessments/Student ratings can be both reliable (can be both reliable (Ross, Rolheiser, Ross, Rolheiser, & Hogaboam-Gray, 2002)& Hogaboam-Gray, 2002) and valid and valid (Cohen, 1981; Fox & Dinur, 2006; Ross, (Cohen, 1981; Fox & Dinur, 2006; Ross,

2006)2006)

Criticisms of Student Self Criticisms of Student Self Report DataReport Data

Known biases in self, course and instructor Known biases in self, course and instructor ratings (ratings (Dunning, Heath, & Suls, in press;Dunning, Heath, & Suls, in press; Frye, Frye, 2005)2005) Self presentation concerns, overconfidenceSelf presentation concerns, overconfidence Class sizeClass size Class format (Lecture vs. Seminar)Class format (Lecture vs. Seminar) Class purpose (elective, major, GE requirement)Class purpose (elective, major, GE requirement) Class level (intro vs. 2Class level (intro vs. 2ndnd year course) year course) Subject matter (Behavioral Science, Humanities, Subject matter (Behavioral Science, Humanities,

Sciences)Sciences) Difficulty of instructor grading Difficulty of instructor grading Liking for instructor Liking for instructor “ “Hotness” and “Ease” (Coladarci & Kornfield, 2007)Hotness” and “Ease” (Coladarci & Kornfield, 2007)

Interpreting Results CorrectlyInterpreting Results Correctly

Look at Relative Ratings, Not Absolute Ratings

SLO 1 →

SLO 2 →

SLO 3 →

SLO 4 →

SLO 5 →

SLO 6 →

SLO 7 →

Typical Instructor Reaction to Typical Instructor Reaction to Student Self Assessment Student Self Assessment

ResultsResults Instructors typically agree and can Instructors typically agree and can

make quick sense of the data:make quick sense of the data: Aligns with time spent on each SLO, Aligns with time spent on each SLO, Realize that they do not teach that SLO Realize that they do not teach that SLO

anymore, anymore, The SLO is at the end and they never get to The SLO is at the end and they never get to

it, it, Realize that the SLO is the most difficult Realize that the SLO is the most difficult

part of the course, part of the course, Written in a way that students would not Written in a way that students would not

understandunderstand

Yeah but…Yeah but…

QuickQuick Broad and systematicBroad and systematic Sensitive to outliers (produced actionable Sensitive to outliers (produced actionable

data)data)

Thus, survey providedThus, survey provided

IndirectIndirect Lacks flexibilityLacks flexibility Does not utilize faculty expertiseDoes not utilize faculty expertise Lacks depthLacks depth

Screening Device

Airport Security

Part 2: Faculty Designed Part 2: Faculty Designed Assessment (Direct Assessment (Direct

Assessment)Assessment)Customized assessment documented on a Customized assessment documented on a

common templatecommon template What course and why?What course and why? What SLO(s)?What SLO(s)? When? When? How broadly? How broadly? How will you assess? (describe tool or rubric)How will you assess? (describe tool or rubric) What is the criteria for successful SLO What is the criteria for successful SLO

achievement? achievement? Who will be administering / scoring?Who will be administering / scoring? Submitted to the SLO Assessment CommitteeSubmitted to the SLO Assessment Committee

American River CollegeTwo-Part SLO Assessment for Courses

Three-Year Cycle for Departments

Part I

Student Self Assessment

(every course assessed)

Part II

Faculty Designed

Assessment(one course assessed)

Action Plan

developand

implement

Action Plan

continue implementatio

n

YEAR 1 YEAR 2 YEAR 3

Year 2: SLO Action PlanYear 2: SLO Action PlanDepartments respond to both parts of assessmentDepartments respond to both parts of assessment

Part 1: Respond to the Student Self-AssessmentPart 1: Respond to the Student Self-Assessment Part 2: Respond to the Faculty Designed AssessmentPart 2: Respond to the Faculty Designed Assessment

Year 2: SLO Action PlanYear 2: SLO Action PlanDetailed InstructionsDetailed Instructions

Action Plan and Instructions approved Action Plan and Instructions approved through an 8 month processthrough an 8 month process

Where we are nowWhere we are now

2/3’s have participated in the college 2/3’s have participated in the college wide SLO Assessment processwide SLO Assessment process

1/ 3 have completed Action Plans1/ 3 have completed Action Plans 63 Separate Actions taken/planned as a 63 Separate Actions taken/planned as a

result of SLO assessmentresult of SLO assessment

Of the 65 Academic departments…

ConclusionConclusion

Thank you!Thank you!

John Aubert, SLO Coordinator, John Aubert, SLO Coordinator, GeographyGeography

aubertj@arc.losrios.edu or 916-484-8637 or 916-484-8637

Yuj Shimizu, Faculty Researcher, Yuj Shimizu, Faculty Researcher, PsychologyPsychology

shimizy@arc.losrios.edu or 916-484-8149 or 916-484-8149

http://inside.arc.losrios.edu/~slo/