effectiveness asap (as simple as possible) david eubanks dir. planning, assessment, and information...

Effectiveness ASAP

(As Simple As Possible)

David EubanksDir. Planning, Assessment, and Information Services

Coker College

Coker College

• Hartsville, South Carolina

• Independent Liberal Arts college

• About 1200 students

• Traditional and Non-traditional programs

• On and off-campus locations

• Reaffirmed 2005

My background

• Math PhD from Southern Illinois

• Institutional researcher

• Institutional effectiveness chair

• SACS liaison

• Library and IT responsibilities

Why isn’t IE easy?

People!

Being Realistic

MoreWork

High Motivation

Less Work

Anti-Motivation

No Motivation

Motivation Strategies

• Cultivate existing motivation

• Create new motivation– Incentives– Punishments– Temporary– May have unpleasant side effects

Making it Easier

• Make the project simple to understand– Use familiar language or gently introduce new

language

• Use electronic systems to minimize effort

• Don’t duplicate effort– Use existing information where possible

Case Study: Writing Assessment

How can we gather authentic samples?

Situation

Student → email → instructor

• Emails get lost

• Papers vanish after term is over

Solution

Student → portfolio → instructor

• Dependable repository saves trouble

• Papers are saved for later assessment

• Adds a new form of communication

Case Study: Documentation

Keeping track of documents is time-consuming, so many people don’t bother.

This is a problem in creating a compliance certification, for example.

Searching

Search results

Use in Certification

Case Study: Building Plans

“Solution”: give staff and faculty the SACS Principles and ask them to make it so!

Section 3.3.1

The institution identifies expected outcomes for its educational programs and its administrative and educational support services; assesses whether it achieves these outcomes; and provides evidence of improvement based on analysis of those results.

What does it mean?

… provides evidence of improvement based on analysis of those results.

It sounds like:

1. Set a goal

2. Measure attainment of the goal and do some analysis

3. Provide evidence that the goal is now being attained better

The missing links

1. Set a goal.

2. Measure attainment of the goal and do some analysis.

* Take actions that may lead to improvement.

** Measure goal attainment again, and compare to the first time you did it.

3. Provide evidence that the goal is now being attained better.

“Ideal” Example

1. Goal: reduce attrition

2. Measurement: currently the 5 yr grad rate is 60%

* Take action: raise admission standards

** Measure again: now the 5 yr rate is 62%

3. Evidence: before and after rates constitute evidence of improvement

What’s more important?

• Taking informed ACTION?– Practical—we can act– Immediate—we can act NOW

• Show evidence of IMPROVEMENT?– May be impractical– Takes time

Standardizing 3.3.1

1. General Objective

2. Outcomes statement

3. Assessment method

4. Results and analysis

5. Actions and improvements

Which do people care more about?

Case Study: General Ed

How can we increase motivation and decrease new requirements in assessing general education objectives?

Assessing the Elephant

Longitudinal

Core Skills Over Four Years

1.0

1.2

1.4

1.6

1.8

2.0

2.2

1 2 3 4 5 6 7 8

Semester

Assessed

Level

Writing

Speaking

Distribution

2005-6 FACS Average Distribution

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

Performance level

Fra

ctio

n o

f cl

ass

Fresh

Soph

Jr

Sr

GPA

FACS Composite by GPA

0.5

0.7

0.9

1.1

1.3

1.5

1.7

1.9

03FA 04SP 04FA 05SP 05FA 06SP

Semester

FA

CS

Co

mp

osi

te A

vera

ge

GPA > 3

2 < GPA < 3

GPA < 2