Data on Student Learning

Office of AssessmentUniversity of Kentucky

Sources of Data on UK Learning

Perspective Direct Measures Indirect Measures

Program-level Program Reviews &Accreditation Responses

Institutional-level

(External/Nationally Comparable)

CLACAAPWabash National

Study

Wabash National Study

Wabash National Study, Spring 2007 Good Practice Scales

Scale 1 - Good Teaching and High Quality Interactions with Faculty

Strong Moderate

Weak

Faculty interest in teaching and student development

41% 47% 13%

Prompt Feedback 20% 69% 12%

Quality of no classroom interactions with faculty

22% 69% 10%

Teaching clarity and organization 46% 54% 1%

Wabash National Study, Spring 2007 Good Practice Scales

Scale 2 - Academic Challenge and High Expectations

Strong Moderate

Weak

Academic challenge and effort 29% 71% 0%

Frequency of higher-order exams and assignments

19% 73% 9%

Challenging classes and high faculty expectations

14% 77% 9%

Integrating ideas, information, and experiences

27% 73% 0%

Wabash National Study, Spring 2007 Good Practice Scales

Scale 3 – Diversity Experience Scale

Strong

Moderate Weak

Diversity experiences 3% 78% 20%

Meaningful discussion with diverse peers 15% 39% 47%

Wabash National Study: Overall Scale 1 – Good Teaching and High Quality Interactions with Faculty

Strong Moderate Weak

Small Institutions 44% 56% 0%

Large Institutions 28% 72% 1%

University of Kentucky 23% 75% 2%

Scale 2 – Academic Challenge and High Expectations

Small Institutions26% 74% 0%

Large Institutions18% 82% 0%

University of Kentucky 13% 87% 0%

Scale 3 – Diversity Experience Scale

Small Institutions5% 84% 12%

Large Institutions3% 83% 14%

University of Kentucky 3% 73% 24%Small Institutions N = 17; Large Institutions N = 9

Narrative Summary of Wabash ResultsNational Results: Initial Participating Institutions

(N=19)

Largest positive change was student growth in moral reasoning◦ 10% over the first two semesters of college.

CAAP Critical Thinking test revealed less than 1% positive change

Small but consistent declines were observed in attitudes about diversity◦ Results reveal a consistent downward pattern across all measures of

diversity.

A moderate, consistent decline observed in:◦ academic motivation and interest in academic subject matter◦ community involvement◦ professional success.

Student perspectives on teaching and instructional practices: ◦ Good Teaching questions had a significant positive impact on 24 of the

29 outcomes◦ Academic Challenge questions had a significant positive effect on 25 of

the 29 outcomes◦ Diversity Experiences questions had a significant positive impact on 17

of the 29 outcomes

Narrative Summary of Wabash Results

Outcomes Measures (fall 2006 to spring 2007)

◦ UK students registered minimal change on most Outcomes Measures

◦ UK students did show a significant increase in moral reasoning

◦ UK students registered a sharp decrease in academic motivation, a component of the Inclination to Inquire outcome

Collegiate Learning Assessment (CLA)

Skills Measured

Performance Task and Analytical Writing Tasks◦ Critical thinking ◦ Problem solving◦ Analytic reasoning◦ Written communication skills

CLA Performance TaskRequires students to use an integrated set of critical

thinking, analytic reasoning, problem solving, and written communication skills to answer several open-ended questions about a hypothetical but realistic situation.

Provides a document library that includes a range of information sources.

Students use the materials in preparing essay answers

Requires students to present their ideas clearly, including justifying their points of view.

CLA Analytic Writing TaskStudents write answers to two types of essay

prompts:◦ “Make-an-Argument” question asks them to

support or reject a position on some issue ◦ “Critique-an-Argument” question asks them to

evaluate the validity of an argument made by someone else

Both tasks measure a student’s skill in articulating complex ideas, examining claims and evidence, supporting ideas with relevant reasons and examples, sustaining a coherent discussion, and using standard written English

Collegiate Learning Assessment (CLA)

Fall 2007◦ Freshman tested (longitudinal cohort), n =

303Spring 2008

◦ 102 Seniors were testedSpring 2009

◦ Longitudinal cohort re-tested◦ N = 174 students

176 institutions were included in the comparisons

Collegiate Learning Assessment (CLA)

Fall 2007 Spring 2009Student Count

Mean score

Std Deviation

Mean score

Std Deviation

Effect Size Performance Level

Performance Task

173 1209 202 1224 166 0.07 Below Expected

Analytic Writing Task

168 1129 144 1170 179 0.29 At Expected

Make-an-Argument

168 1122 173 1151 217 0.17 At Expected

Critique-an-Argument

171 1134 181 1186 193 0.29 At Expected

Total Score

168 1172 145 1199 149 0.19 At Expected

Collegiate Learning Assessment (CLA)

Note: An effect size of 0.50 is considered “large.”

Perfo

rman

ce T

ask

Analyt

ic W

riting

Task

Mak

e-an

-Arg

umen

t

Critique

-an-

Argum

ent

Tota

l CLA

Sco

re0

0.05

0.1

0.15

0.2

0.25

0.3

0.07

0.29

0.17

0.29

0.19

Eff

ect

Siz

e

Collegiate Learning Assessment (CLA)

CLA Measure Effect Size

Performance Task

Analytic Writing Task

Make an Argument

Critique an Argument

Total Score

Collegiate Assessment of Academic Proficiency (CAAP) – MathDescriptive Analysis N = 308 Level:

◦ 79.41% Freshman College:

◦ 33.27% College of Arts & Sciences

◦ 27.12% Undeclared

◦ 9.15% Engineering Transfer vs. Traditional

◦ 14.05% Transfer

◦ 85.95% Traditional Residence

◦ 82.03% In-state

◦ 17.97% Out-of-State

Collegiate Assessment of Academic Proficiency (CAAP) – Math Results

UK National

Mean 58.5 58.5

Standard Deviation 3.8 4

N 308 21,678

Overall Mathematics Test Scores of UK Students – Spring 2009

CAAP – Math Results

UK National

Mean 15.7 15.6

Standard Deviation 2.4 2.4

N 308 21,678

Basic Algebra Scores – Spring 2009

College Algebra Scores – Spring 2009

UK National

Mean 15.1 15.4

Standard Deviation 2.8 2.7

N 308 21,678

CAAP – Critical ThinkingSpring 2010 Summary Results

Count Percentage

Average Score

National Average

Score

Freshman 128 22% 62 62

Sophomore 252 43% 63 62

Junior 124 21% 63 62

Senior 61 10% 63 62

Other/No Response

27 4% 63 62

UK Average 592 100% 63 62

National N= 9,531

CAAP – Critical ThinkingSpring 2010

Tried my

Best

Gave Moderate Effort

Gave Little Effort

Gave No Effort

No Respon

se

Freshman 17% 34% 10% 2% 36%

Sophomore 19% 35% 12% 7% 26%

Junior 23% 48% 9% 2% 22%

Senior 34% 28% 7% 5% 26%

UK Average

21% 36% 11% 4% 27%

UK Student Perceptions of Effort

Program Review/Accreditation Analysis

Need Faculty

Need Resources

Curriculum Improvement

Improve Facilities

Graduate Teaching Experience

Improve Practicum and/or Internship

Faculty/Student Interaction

Faculty Training

Assessment Related

96

247

232

140

15

31

31

17

31

Program Reviews and/or Accreditation Documents from 2002-present