assessing student achievement in the core curriculum office of assessment and program improvement...

Assessing Student Assessing Student Achievement in theAchievement in the

Core CurriculumCore Curriculum

Office of Assessment and Program ImprovementOffice of Assessment and Program Improvement

Report to the University CommunityReport to the University Community

Fall 2003Fall 2003

2

Office of the ProvostOffice of the Provost

The Assessment Process

Define specific learning outcomes that should generally be evident in students who complete a program of study

Identify sources of evidence about how well students are typically achieving these outcomes

Set performance targets as criteria for program success

Conduct studies and use the results to improve achievement

3


Assessing a Major vs.Assessing the Core

C O R E

What any AU student

should know

and be able to do

no matter what major

M A J O R

What an AU student

majoring in a given subject

should know

and be able to do

Defined by specialists

Defined Cooperatively

4

Office of the ProvostOffice of the Provost Working Assumptions

We can collect reasonably good evidence about how well students are achieving the desired “core outcomes” for an AU graduate, but it is difficult to know what led to this achievement

Assessing students’ core competencies is more feasible than ssessing students’ core competencies is more feasible than determining the exact contribution of the Core Curriculum itselfdetermining the exact contribution of the Core Curriculum itself

Using several methods and measurement points will provide fuller information than relying on any one “snapshot” method

The “core outcomes” are everybody’s businessThe “core outcomes” are everybody’s business

5


Core Curriculum Oversight Committee

Created together with the Core Curriculum

• Original charge was to oversee implementation of Core

• Began assessment in 1995 with reviews of social science and Great Books courses

• Reviews focused on teaching – syllabi, assignments, textbooks, grade distributions

• Assessment best practice now focuses on student learning

6

Office of the ProvostOffice of the Provost Assessment Process

Define intended outcomes

7


From Broad Purpose to Specific Plans

In 2001 the Core Curriculum Oversight Committee

Studied the broad purpose statement for the Core Curriculum

Formulated 12 more specific intended learning outcomes

Selected 4 outcomes for assessment in 2002 and 2003

8


Starting Pointsfor Assessment

Read critically and analytically

Communicate in writing at a sophisticated level

Comprehend the basic concepts of algebra

Gather, interpret, and synthesize information in accordance with contemporary scholarly standards

Students who have completed the Core Curriculum will be able to

9

Office of the ProvostOffice of the Provost Future Assessment

Scientific Reasoning*

Art Appreciation

History, Culture, Values

Social Sciences

Oral Communication

Awareness of Issues

Critical Thinking

Problem Solving

Outcomes for Future Study

10



Identify means of assessment

11

Office of the ProvostOffice of the Provost Two Methods Used

Local methods to estimate the abilities of students who were still taking Core courses

Analysis of papers and exams Pre- and post-testing

ACT’s Collegiate Assessment of Academic Proficiency (CAAP) tests to estimate the abilities of students who had fulfilled all Core Curriculum requirements

Writing, Reading, Math, Critical Thinking, Science Reasoning

12

Office of the ProvostOffice of the Provost Assessment Methods

S T A N D A R D

ACT CAAP Tests

Project SAILS

L O C A L

Review of work samples

Pre- and Post-Testing

Taking Core Fulfilled Core

13




Set targets for program success

14


Standards Chosen:Local Methods

Critical Reading• 70% of portfolios reviewed should rate satisfactory or better

when evaluated by two independent judges

Written Communication• 70% of portfolios reviewed should rate satisfactory or better

when evaluated by two independent judges

College Algebra• Improvement should be evident on post-test

Information Literacy• Both standards – 70% satisfactory and post-test improvement

15


Standards Chosen:CAAP Test

AU students who have completed the Core Curriculum should score at or above the 70th percentile nationally on each CAAP module

No more than 20% of AU students tested should perform below the 50th percentile nationally on any CAAP module

National comparison group was sophomores at about 80 public 4-year colleges

16





Collect and analyze assessment data

Outcome 1Critical Reading

Students who have completed the Students who have completed the Core Curriculum will be able to read Core Curriculum will be able to read critically and analytically at a critically and analytically at a sophisticated levelsophisticated level

18

Office of the ProvostOffice of the Provost Critical Reading

Analysis of papers and exams from Great Books II showed

Students were able to find, remember, and interpret specific details from their readings

They were able to discuss themes found in their readings

Assessment could not determine whether students were able to discuss formal literary features of the works they had read or place them in their historical and social contexts

19


Critical Reading

75

86

3833

90 88

49

61

0

100

Details Themes Forms Contexts

%

Papers Exams Can't Tell

Percent of Great Books II Portfolios Rated Excellent/Satisfactory

20


In formal papers for Great Books II, students met but did not exceed the performance target for three aspects writing about texts that may imply critical reading ability

Formulating a thesis about a text they had read

Using textual evidence to support that thesis

Reasoning soundly to connect evidence and thesis

21


Critical Reading

71 73 73

0

100

Thesis Evidence Reasoning

%

Percent Rated Excellent or Satisfactory

GB2 Essays rated Excellent or Satisfactory

22


Results from the two CAAP tests for Critical Reading were mixed

Reading• Median score was at the 73rd percentile nationally• But 22% of those tested scored below the 50th percentile

Critical Thinking• Median score was at the 82nd percentile nationally• Only 11% scored below the 50th percentile

23


Critical Reading

73

82

11

22

0

100

Reading Critical Thinking

%

Median Score as National Percentile Rank % Below 50th Percentile

Median Percentile Scores and “Failure Rates”

CAAP Reading and Critical Thinking Tests

Outcome 2Written Communication

Students who have completed the Students who have completed the Core Curriculum will be able to Core Curriculum will be able to communicate in writing at a communicate in writing at a sophisticated levelsophisticated level

25

Office of the ProvostOffice of the Provost Written Communication

Analysis of argument papers from Composition II showed

Students were clearly able

• to formulate a thesis• to use language that was appropriate to the writing situation

Students had more trouble

• Supporting their thesis with sound evidence• Organizing their arguments effectively

26


Written Communication

90

73

89

66

0

100

Thesis Evidence Organization Language

%

Rated Satisfactory or Better

Four Aspects of Rhetorical Effectiveness

ENGL1120 Papers with Satisfactory Ratings

27

Office of the ProvostOffice of the Provost Written Communication

The two CAAP tests for writing were given in 2002 only

Students “maxed out” the Essay Writing test – nearly all scored at or above the 70th national percentile

Scores on the multiple-choice Writing Skills test were also strongly positive

It is not clear that there is good alignment between this test and the goals of Composition I and II

28


Written Communication

86

97

167

0

100

Writing Skills (2002 only) Essay Writing (2002 only)

%



CAAP Writing Skills and Essay Writing Tests

Outcome 3College Algebra

Students who have completed the Students who have completed the Core Curriculum will comprehend Core Curriculum will comprehend the basic concepts of algebrathe basic concepts of algebra

30

Office of the ProvostOffice of the Provost College Algebra

Local Assessment (2003 Only)

Students took Math Placement Exam at Camp War Eagle

Some items from this exam were embedded in the final exams for pre-calculus Core courses (not including MATH 1100)

On average, the 916 students tested scored 30% higher on these items on the final exam than they had done on the placement test

31


Results for the CAAP Mathematics test must be treated separately for 2002 and 2003 because of a change in the test design

2002 version (4 calculus items)

students met the performance targets for the algebra subscore

2003 version (no calculus items)

students met the performance targets for the whole test

For both years, students’ performance just cleared the bar of the 70th national percentile, but concerns remain

32


College Algebra

7673

5

18

0

100

Algebra Subscore (2002) Mathematics Test (2003)

%


Median Percentile Scores for CAAP

Mathematics Test

33


The CAAP results are less positive when students are grouped

Those who had taken Math courses beyond the Core met performance targets

. . . But . . .

Those who had taken only one Math course failed to meet performance targets

34


College Algebra

64

82

24

11

0

100

"More Math" (2003) "Core Math" (2003)

%



“More Math” vs. “Core Math” Students

Experimental OutcomeScience Reasoning

36

Office of the ProvostOffice of the Provost Science Reasoning

The CAAP results for Science Reasoning (not formally assessed in either year) resemble those for Mathematics

Those who had taken only one Math course scored poorly in Science Reasoning

Those who had taken Math courses in addition to the Core did much better in Science Reasoning

37


Science Reasoning

72.5

87

70

14

4

20

0

100

All Students "More Math" Students "Core Math" Students

%



CAAP Science Reasoning Test

Outcome 4Information Literacy

Students who have completed the Students who have completed the Core Curriculum will be able to Core Curriculum will be able to gather, synthesize, and interpret gather, synthesize, and interpret information in accordance with information in accordance with contemporary scholarly standardscontemporary scholarly standards

39

Office of the ProvostOffice of the Provost Information Literacy

Before receiving bibliographic instruction (BI), ENGL 1120 students were tested on how to conduct library searches

Sample: “You wish to find current articles dealing with drug use on U.S. campuses. Your best “place” to begin?

Students were retested after BI and had higher scores

AU Bibliographic Instruction Quiz (Summer 2001)

48

61

0

100

Before BI After BI

% Correct

40

Office of the ProvostOffice of the Provost Information Literacy

165 UNIV1000 students used the online Texas Information Literacy Tutorial (TILT) and took the accompanying quizzes

The overall score on the TILT was 96%

Their lowest score was on an item that tested whether they thought the library’s whole collection was available online

Texas Information Literacy Test

96

86

0

100

Total Availability Question

% Correct

41


Information LiteracyProject SAILS

In 2003, 247 ENGL1120 students took part in Project SAILS, a national information literacy test recognized by the Association of Research Libraries

Multiple choice questions yielded information about 4 key information literacy skills

On each skill, AU students did somewhat better than the average for all 10 participating institutions

Project SAILS

50

66

5057

0

100

AssessNeeds

FindInformation

EvaluateInformation

UnderstandIssues

% Correct

42


Information LiteracyWork Samples

91

47

666863

0

100

Appropriateness Balance* Integration Citation* Reference List*

%

Rated Satisfactory or Better (* = 2003 only)

Aspects of Using Sources Effectively

ENGL1120 Papers with Satisfactory Ratings

43





Collect and analyze assessment data

Act on the results to improve student achievement

44

Office of the ProvostOffice of the Provost Planned Actions

Remedy focused deficiencies by additional instruction and practice in appropriate courses – e.g., documenting sources

Guide instruction, assessment, and improvement by clarifying some broad intended outcomes – e.g., critical reading ability

Continue to get good evidence about our students’ core competencies through dialogue and collaboration

Benefit from experience with four outcomes to guide assessment planning for the intended learning outcomes that have not been studied so far

Further InformationFurther Information

Office of Assessment and Program ImprovementOffice of Assessment and Program ImprovementDrew Clark, DirectorDrew Clark, Director

209 Samford Hall209 Samford Hall844-5802844-5802

[email protected]://www.auburn.edu/academic/provost/assessment


mailto:[email protected]

http://www.auburn.edu/academic/provost/assessment

assessing student achievement in the core curriculum office of assessment and program improvement...

Documents

provost office

desired core outcomes

implementation of core

means of assessment

achievement slide

assessment process

intended outcomes

student learning slide