Interpreting Data & “Closing the Loop”

Margaret Kasimatis, PhDVP for Academic Planning & Effectiveness

ARTICULATEMISSION/

GOALS

IDENTIFYSPECIFIC

OUTCOMES

DETERMINEPRACTICES USED

TO ACHIEVEOUTCOMES

GATHEREVIDENCE

REVIEW &INTERPRETRESULTS

RECOMMENDACTIONS

Location in Assessment Cycle

Step #1: Organize Your Data

• Assemble the following in one place:– All data or possible sources of data– List of learning outcomes/research questions– Curriculum, experiences, practices

• Map data sources to outcomes– Outcomes x Measures Map

• Map results to outcomes– Present in “user-friendly” way

LEARNING OUTCOMES

ASSESSMENT MEASURES

Alumni Survey

National Survey of Student Engagement

(NSSE)

Evaluation of Clinic

Presentations

Evaluation of First-Year

Presentations

Rubric Evaluation of Upper-level Coursework

Ability to apply knowledge in math, science, and engineering X X

Ability to design a system, component, or process to meet desired needs X

Ability to identify, formulate, andsolve engineering problems X X X

Ability to communicate effectively X X X X XAbility to use techniques, skills, and modern engineering tools necessary for engineering practice

X X

Ability to function on multidisciplinary teams X X X

Recognition of need for and abilityto engage in lifelong learning X X

Understanding of professional andethical responsibility X X

Broad education necessary tounderstand the impact of engineering solutions in a global & societal context

X X

Mapping Data Sources to Outcomes:Engineering Example

• It’s best to present results one outcome at a time– It’s not recommended to present results primarily

by measure

• Make sure to present the data in a user-friendly way (i.e., in a way that faculty in your department are comfortable with)

Presenting Results

NSSE: Data on Communication Skills

1

2

3

4

Experience contributed to abilityto write clearly and effectively

Experience contributed to abilityto speak clearly and effectively

Mea

n R

esp

on

se (

1-4

scal

e)

HMC Senior Engineers

AITU Seniors

Liberal Arts Seniors

0

10

20

30

40

50

60

To what extent was an emphasis onwriting skills present while you were at

HMC?

To what extent was an emphasis on oralcommuniction skills present while you

were at HMC?

Pe

rce

nt

Ind

ica

tin

g "

Ve

ry M

uc

h"

Biology

Chemistry

Comp Sci

Engineering

Math

Physics

Alumni Survey: Data on Communication Skills Rubric Evaluation of Student Presentations

1

2

3

4

First-Year Presentation Clinic Presentation

DeliveryLanguageOrganizationContent

Rubric Evaluation of Student Work: Writing Style

1

2

3

4

5

Course #1 Course #2

Mea

n R

ub

ric

Sco

re o

n W

riti

ng

Sty

le (

1-5

scal

e)

ACADEMIC CHALLENGE: LMU Jesuit Master’s NSSETotal

How often did you work harder than you thought you could to meet an instructor's standards or expectations?(Percent of students who said ‘Often’ or ‘Very Often)

FY 62% 55% 54% 52%

SR 54% 59% 60% 56%

How much time did you spend preparing for class (e.g. studying, reading, writing, doing homework, rehearsing,

other academic activities)? (Percent of students who reported spending 11 or more hours per week)

FY 65% 68% 49% 56%

SR 40% 58% 52% 54%

How often did you come to class without completing readings or assignments? (Percent of students who said ‘Often’ or ‘Very Often’)

FY 21% 17% 17% 19%

SR 27% 23% 20% 23%

Describe the extent in the current year to which exams challenged you to do your best work. (mean response on a 7-point scale)

FY 5.47 5.56 5.33 5.42

SR 5.33 5.46 5.43 5.40

Step #2: Interpret the Data

• What type of criterion?

• What is “significant?”

• Are the findings reliable/valid?

What type of criterion?

• Norm-referenced– Need appropriate comparison group

• Avoid percentile rankings

– Good for more nebulous findings

• Absolute standard– Usually more appropriate for

performance-based measures

What is significant?

• Important to test when making comparisons– Numbers that look different may not really

be different

• However, just because a difference is significant doesn’t mean it’s important

Are findings reliable/valid?

• Can we believe student self-reports?

• Are standardized measures more valid?

• Single measure vs. multiple measures

Step #3: Make Recommendations

• Start by considering– Where in the curriculum are outcomes

addressed? And at what level?• Refer to (or create) curriculum map, or

• Inventory for particular outcome

– How? What practices/techniques are used?

– Where are the gaps?

– What can be changed?• In the short-term? In the long-term?

Step #3: Make Recommendations

• Possible types of recommendations:– Changes to pedagogy– Changes to curriculum/programming– Allocation of resources

• Keep in mind that you can’t fix everything at once – so start small

• It’s usually better to modify than add

• Be as specific/concrete as possible

PROPOSED ACTION IMPLEMENTATION

Document (with grades) the importance of presentation clarity

Essay questions added to exams

Engineering Curriculum Committee developed writing rubric to be used to evaluate project report

Make common writing style guidelines available to faculty and students

Department selected Hacker, A Writer’s

Reference

Hacker, A Writer’s Reference required in E4

Hold Clinic team members

accountable for peer review

of team reports

Clinic advisors distributed peer review sheets to

Clinic team members. Students completed the

peer review sheets when they reviewed fellow

team members’ portions of Clinic report .

Step #4: Implement & Document Recommend Actions

Example

Example of Assessment Loop

• Step #1: Articulate Goals/Outcomes– Goal: Graduates will possess strong

communication skills• Outcome: Graduates will be able to write clearly and

effectively

• Step #2: Determine practices used to achieve outcome– Survey of required writing experiences in Core

and academic programs

Sem # BIO CHEM CS ENG MATH PHYS H/SS

One 4 papers

Two Report in E4 Tech report in Phys 28

4 papers

Three 1-2 papers in Bio 101

3 lab reports

Lab notebook & paper in Chem 53

Tech report

in Phys 53

Variable

Amount

Of

Writing

Four 4 lab reports

1-2 papers in Bio 108

Essays/paper

in Chem 52

Lab notebook Exp abstracts

Design doc,

Report, &

User guide

in CS 121

Tech report

in Phys 54

Term paper

in Phys 52

Five 1-2 papers in

Bio 109

3 lab reports

3 reports + paper in 109

Exp abstracts

Abstracts in CS 110

Proposal &

mid-year report OR

Six 1-2 papers

in Bio 113

Seminar paper

Project paper

in Chem 114

Essay in

CS 131

Clinic report Tech report in Phys 134

IE Paper IE Paper IE Paper IE Paper IE Paper IE Paper

Seven Proposal

3 Lab reports

Letter of Intro

Prog report

Proposal +

midyear

report

Proposal +

midyear

report

Seminar

Research

PaperEight Thesis Thesis Clinic

reportClinic report Thesis or

Clin. report

Thesis or

Clin. report

Example of Assessment Loop, cont’d

• Step #3: Conduct measurements– Higher Education Research Institute (HERI)

College Student Survey (CSS)– National Survey of Student Engagement– HMC Alumni Survey

• Step #4: Review & interpret results– Faculty committees

• Assessment• Curriculum• Teaching & Learning

– Full faculty - workshop

Example of Assessment Loop, cont’d

Writing SkillsFrequency data from HERI, NSSE & Alumni Survey

0

10

20

30

40

50

60

70

80

"Much stronger writingskills than when entering

college"

"Institution contributedvery much to ability to

write clearly andeffectively"

"To what extent was anemphasis on writing skillspresent while you were at

HMC?" (5)

HMC Seniors

4-Yr Non-Sectarian

1959-1987Cohort

1988-2001CohortHMC

Seniors

Liberal Arts

AITU Seniors

Seniors’ Perceptions of Writing Skills by Major (from NSSE)

1

1.5

2

2.5

3

3.5

4

Biology CS Engineering Mathematics PhysicalSciencesT

o w

hat

ext

ent

did

exp

erie

nce

s at

you

r in

stit

uti

on c

ontr

ibu

te

to y

our

abil

ity

to w

rite

cle

arly

an

d e

ffec

tive

ly?

Alumni Perceptions of Writing Emphasis by Major (alumni graduating in last 10 years)

0

10

20

30

40

50

60

70

Biology Chemistry CS Engineering Math Physics

Wri

tin

g em

ph

asis

was

"ve

ry m

uch

" p

rese

nt

Sem # BIO CHEM CS ENG MATH PHYS H/SS

One 4 papers

Two Report in E4 Tech report in Phys 28

4 papers

Three 1-2 papers in Bio 101

3 lab reports

Lab notebook & paper in Chem 53

Tech report

in Phys 53

Variable

Amount

Of

Writing

Four 4 lab reports

1-2 papers in Bio 108

Essays/paper

in Chem 52

Lab notebook Exp abstracts

Design doc,

Report, &

User guide

in CS 121

Tech report

in Phys 54

Term paper

in Phys 52

Five 1-2 papers in

Bio 109

3 lab reports

3 reports + paper in 109

Exp abstracts

Abstracts in CS 110

Proposal &

mid-year report OR

Six 1-2 papers

in Bio 113

Seminar paper

Project paper

in Chem 114

Essay in

CS 131

Clinic report Tech report in Phys 134

IE Paper IE Paper IE Paper IE Paper IE Paper IE Paper

Seven Proposal

3 Lab reports

Letter of Intro

Prog report

Proposal +

midyear

report

Proposal +

midyear

report

Seminar

Research

PaperEight Thesis Thesis Clinic

reportClinic report Thesis or

Clin. report

Thesis or

Clin. report

• Step #5: Recommend actions– Universal writing handbook– Faculty workshop on writing instruction– Revision to Core Curriculum

Example of Assessment Loop, cont’d

MULTIPLE CURRICULAR OBJECTIVESIN HORIZONTALLY INTEGRATED

CORE CURRICULUM

Content• disciplinary knowledge • discipline-related techniques

Skills

• oral and written communication • critical thinking• teamwork and collaboration• project management• leadership

Context

• relationship of science or technology with contemporary society

• individual and comparative explorations of cultural identity

QUESTIONS?