Multiple Methods for Assessing Learning Community Outcomes

Maureen Pettitt, Ph.D., Skagit Valley College

Shanda Diehl, Spokane Falls Community College

AIR San Diego May 2005

Session Overview Learning Communities at

SVC & SFCC Rationale Organization

LC Assessment at SVC LC Assessment at SFCC Conclusions

Collaborative Courses at SVC

Collaborative courses are a means of delivering instruction and fostering student learning; they do not have associated credit requirements

Learning Communities (and English links) are required for the transfer degree

Collaborative courses are options for the technical arts degree.

60+ Learning Communities are offered each year at the college.

Rationale for Interdisciplinary Learning at SVC

Advantages (greater retention, student involvement, etc.) were known, but not part of the initial rationale for requirements.

A response to curricular issues: Faculty felt that students did not see connections between and among

disciplines, and needed to engage subjects more fully, to see

education as a dynamic and interconnected process of exploration and discovery

3. CCSSE

2. Student Writing

1. Student Satisfaction Survey

Multiple Assessment Methods

1. Locally-Developed Student Satisfaction Survey

Faculty-developed two-item survey All responses on a scale from “strongly

disagree” to “strongly agree” Current N for LCs= 1364 Faculty are provided their course

results and comments, plus the cumulative for the quarter for all courses

“This learning community has been a valuable learning experience”

3 4 . 5

4 9 . 1

1 0 . 73 . 7 2 . 0

0

1 0

2 0

3 0

4 0

5 0

6 0

S t r o n g lyA g r e e

A g r e e N e u t r a l D i s a g r e e S t r o n g lyD i s g r e e

“It was more valuable to take these classes together than separately”

3 5 . 0 3 4 . 0

1 6 . 71 1 . 0

3 . 3

05

1 01 52 02 53 03 54 0

S t r o n g lyA g r e e

A g r e e N e u t r a l D i s a g r e e S t r o n g lyD i s g r e e

Student Comments LC: “This class has been fun and

not sucky at all. I think I have learned a lot.”

DE LC: “Doing both [topics] in depth was overwhelming at times.”

DE LC: “Art History rocks!”

Student Comments “If they had been

separate, I would have known the what and where, but not the why, and the why is always the most important question.”

2. Student Writing Review of Learning Outcomes

Assessment Plans (LOAPS) indicated that the overarching Gen Ed outcomes were not addressed in any obvious way

Faculty/IR team developed two-year research project to assess student attainment of overarching General Education learning outcomes

Study GoalsAssess students' ability to: Apply a variety of concepts/texts/contexts

and perspectives to solving problems and thinking about issues.

Connect one’s own life experience, ideas, and abilities with those that others bring.

Understand and value the learning process for oneself and for others.

Write, speak, read, and listen effectively. Demonstrate critical thinking skills.

Study Goals

Assess learning outcomes by modality determine "if" and "how" this learning

occurs in the context of "where" (i.e., interdisciplinary courses, stand-alone courses, distance education courses)

Approach Identified courses being taught in

Learning Communities, stand-alone, and DE that could be “matched” over a two-year period

Faculty agreed to participate and give students course credit

Reviewed work done at SVC with Bill Moore (MID), Alverno College interviews, etc.

Developed a set of questions for students beginning-, mid-, and end-course

Faculty Participation Winter 2002: Developed 20-

item list -- “Learning That We Value” – collapsed to 5 items

Spring 2002: Analyzed student writing from Winter quarter to “pilot” the framework

Summer 2002 & 2003: Stipends to analyze student writing; discuss and report results

Beginning of Course Questions

1. What are your learning expectations for this course?

2. How will you know that your learning expectations are being met?

3. What value, if any, do you expect this learning will have for you?

Mid-Course Questions

1. Are your learning expectations being met?

2. Have you learned things that you hadn't anticipated? If so, please describe.

3. Do you find your learning experience in this class is any different from high school or other community colleges that you know about?

End of Course Questions

1. Have your learning expectations been met?2. Have you learned things that you hadn't

anticipated? If so, please describe.3. What do you think are the most important

aspects of your experiences in this course that account for your learning?

4. What have you learned in this course that will matter to you five years from now?

Findings Student writing elicited adequate depth and

breadth of comments that demonstrated application of a variety of concepts, texts,

contexts to solving problems/thinking about issues

understanding/valuing learning process Fewer, but adequate, comments

demonstrating self-other connection and critical thinking

Least useful for demonstrating writing, speaking, reading and listening skills

Students in Learning Communities “After taking this course I feel that I can

make connections to various things, such as history, influences, people, and culture. This course taught me the value of making connections and things from my own perspective.”

“I think I will be more likely to make connections between subjects, both similarities and differences.”

Students in Learning Communities “I do believe that I may think in a

way of applying what I am learning to something else.”

“The link between history and music and the connection to the present day versus the past was important writing. The weekly seminar papers really forced me to look at the world differently.”

Students in Learning Communities “By combining course topics you

get the ‘bigger picture’ and are able to sort of apply what we are learning better. By applying a subject or topic to another subject or topic you have to comprehend what you are learning and apply it to other things.”

Comparisons Comparing student responses in

the stand-alone courses with students in collaborative courses—same courses with the same instructor(s)--students in stand-alone courses: were less likely to write about these

connections or about learning, and tended to focus on personal

growth, liking faculty, etc.

Lots more to do….

Qualitative assessments of this kind are very ambitious projects--review, coding and analysis very time consuming

3. Community College Survey of Student Engagement

CCSSE Factors Analyzed Based on Learning Community Attendance

Active & Collaborative Learning Student Effort Student-Faculty Interaction College Contribution to

Knowledge, Skills & Personal Development

Mental Activities

Mental Activities Similar in structure to Bloom’s

Taxonomy Prompt: “During the current school

year, to what extent has your coursework emphasized the following mental activities…”

Response Options: Range from 1 “Very Little” to 4 “Very Much”

Groupings SVC 1 = SVC students who have already taken a

learning community or linked course SVC 2 = SVC students who have not taken, but are

planning on taking a learning community or linked course

SVC 3 = SVC students who have not taken and do not plan on taking a learning community or linked course

Consortium = all students in the Northwest consortium of colleges, excluding those from Skagit Valley College and Douglas College

All = all students who participated in the CCSSE Survey, based on 93 colleges

Memorizing

2 . 7 2 2 . 7 1

2 . 8 02 . 8 3

2 . 8 0

2 . 5 0

2 . 6 0

2 . 7 0

2 . 8 0

2 . 9 0

3 . 0 0

B lo o m ' s T a x o n o m y : M e m o r iz in g

S V C 1 S V C 2 S V C 3 C o n s o r t iu m A ll

”memorizing facts, ideas, or methods from your courses and readings so you can repeat them in pretty much the same form.”

Analyzing

3 . 0 4

2 . 9 2

2 . 7 9

2 . 9 2

2 . 7 9

2 . 5 0

2 . 6 0

2 . 7 0

2 . 8 0

2 . 9 0

3 . 0 0

B l o o m 's T a x o n o m y : A n a l y z i n g

S V C 1 S V C 2 S V C 3 C o n s o r t iu m A ll

“mental activities: analyzing the basic elements of an idea, experience, or theory.”

Synthesizing

2 . 8 6

2 . 7 8

2 . 6 2

2 . 8 0

2 . 6 7

2 . 5 0

2 . 6 0

2 . 7 0

2 . 8 0

2 . 9 0

3 . 0 0

B l o o m 's T a x o n o m y : S y n t h e s i z i n g

S V C 1 S V C 2 S V C 3 C o n s o r t iu m A ll

“synthesizing and organizing ideas, information, or experiences in new ways.”

Making Judgments

2 . 7 4

2 . 6 0

2 . 4 9

2 . 6 7

2 . 5 2

2 . 3 0

2 . 4 0

2 . 5 0

2 . 6 0

2 . 7 0

2 . 8 0

B l o o m 's T a x o n o m y : M a k i n gj u d g m e n t s

S V C 1 S V C 2 S V C 3 C o n s o r t iu m A ll

“making judgments about the value or soundness of information, arguments, or methods.”

Applying Theories or Concepts

2 . 7 7

2 . 6 8

2 . 6 0

2 . 7 1

2 . 6 3

2 . 5 0

2 . 6 0

2 . 7 0

2 . 8 0

2 . 9 0

3 . 0 0

B l o o m 's T a x o n o m y : A p p l y i n gT h e o r i e s

S V C 1 S V C 2 S V C 3 C o n s o r t iu m A ll

“applying theories or concepts to practical problems or in new situations.”

Using Info for a New Skill

2 . 8 1

2 . 7 5 2 . 7 3

2 . 7 82 . 7 5

2 . 5 0

2 . 6 0

2 . 7 0

2 . 8 0

2 . 9 0

3 . 0 0

B l o o m 's T a x o n o m y : U s i n g i n f o r m a t i o n

S V C 1 S V C 2 S V C 3 C o n s o r t iu m A ll

“using information you have read or heard to perform a new skill.”

Students who had taken Learning Communities were:

Significantly more likely to have: Discussed grades with an instructor Worked with classmates outside of class

to prepare class assignments Had discussions, conversations, and

contacts that encourage multiculturalism and global awareness.

Used advising and computer labs

SFCC Learning Communities Learning community is a package of

interdisciplinary courses where students and faculty are consistently involved

Learning community provides strong network of relationships, both intellectual and emotional

Purpose: Improve Student Success in Community College System

Benefits and Costs Benefits

Improve Student Outcomes – congruent with college mission

See interconnectedness of disciplines Establish relationships with students and

faculty Student-centered

Costs Higher expense Efficiency measures

Methodology

Conducted Study in Fall 2003 Cohorts: 2000-01, 2001-02, and

2002-03 full-time, day-time, academic degree-seeking freshmen taking state-supported classes

Separated cohorts into two groups: Learning Community participation and NO Learning Community participation

Methodology (continued) Compared between the two groups

Cumulative Grade Point Average (T-Test) Percent of Credits Earned versus Attempted

(T-Test) Retention – came back the subsequent

academic year (Chi-Square) Analyzed each cohort separately Supplemented Analysis with Student

Satisfaction Surveys of Learning Community Participation

Findings Average

GPA Percent Credits

Earned Versus Attempted

Came Back Subsequent Academic

Year Learning Community n=210

2.64* 78%* 72%* 2000 – 2001

No Learning Community n=994

2.43* 70%* 63%*

Learning Community n=199

2.42 72%* 65% 2001 – 2002

No Learning Community n=1,081

2.44 68%* 61%

Learning Community n=172

2.44 68% 62% 2002 – 2003

No Learning Community n= 1,151

2.48 72% 62%

* Differences were statistically significant

Results Ambiguous Participation in learning communities

make a difference in 2000-01, make a slight difference in 2001-02 but no difference at all in 2002-03

Study was replicated for late 90s cohorts and results were consistent with the 2000-01 results

What does this mean?

Thought Process Fall 2003: Hypothesis- not enough

time had elapsed to get full cumulative GPA, credits earned versus attempted and only Fall 2003 was considered as the time to come back for the 2002-03 cohort.

Study put on hold.

Spring 2005 Replicated the study, reran all

cohort years, and added 2003-04 cohort using the same criteria.

Found that 2003-04 showed same results as 2002-03. No difference was detected in student outcomes by learning community participation in the 2003-04 freshman cohort.

Revised Study Is there a change in the learning community

offerings that have led to these findings? Yes! The change in learning community

offerings was offering more developmental education (remedial classes) as a part of the learning community. The theory was that students who form strong intellectual and emotional connections would do better. Writing Reading Math

Changes in Learning Communities

Took Developmental Education in

Learning Community

Learning Community and

Took Developmental

Education Outside of Learning Community

College Level and Learning

Community Participation

2000-01 Freshmen Cohort

11% 59% 30%

2001-02 Freshmen Cohort

18% 50% 32%

2002-03 Freshmen Cohort

26% 49% 25%

2003-04 Freshmen Cohort

28% 36% 36%

So . . . 5 Groups Now Group 1:  Took developmental education in

learning community as a freshman Group 2:  Took developmental education AND

had no participation in a learning community as a freshman

Group 3:  College Level students (NO developmental education) AND participated in a learning community as a freshman

Group 4:  College Level students (NO developmental education) AND did NOT participate in a learning community as a freshman

Group 5:  Participated in a Learning Community AND took developmental education NOT in a learning community as a freshman

Methodology Analysis of individual cohort years ANOVA tests were used to

determine differences between the 5 groups in cumulative GPA and percent credits earned versus credits attempted

Chi-Square Analysis was used to detect differences in retention

Average GPA*

Percent Credits Earned Versus

Attempted*

Came Back Subsequent Academic

Year*

Group 1: Took developmental education in learning community as a freshman

2.17 66% 56%

Group 2: Took developmental education AND had NO participation in a learning community as a freshman

2.46 70% 60%

Group 3: College Level students (NO developmental education) AND participated in a learning community as a freshman

2.47 75% 42%

Group 4: College Level students (NO developmental education) AND did NOT participate in a learning community as a freshman

2.60 76% 46%

Group 5: Participated in a learning community AND took developmental education NOT in a learning community as a freshman

2.54 74% 66%

* Differences are statistically significant

Results Developmental Education seems to be the

key factor: Of all developmental education freshmen included

in the study, those who take their developmental education courses outside of the learning community AND participate in a learning community have the best outcomes

Students in learning communities with the best outcomes are those students who participate in learning communities but take developmental education courses outside learning community

What Does This Mean? SFCC purpose of learning communities is

to improve student success in community college system

Yet, we find that this is not the case, due in part to changes in learning communities to incorporate developmental education offerings

These findings will change the content and themes of learning communities at SFCC

Re-examine delivery of developmental education courses

SVC and SFCC The goal and purpose of learning

communities shape the evaluation and assessment methodology.

The study findings shape the delivery of learning communities at each college.

Different approaches yield different results.

Use the results to improve programs and student success