hill and serra, 2015 aacu, gen ed assessment …...critical thinking, and reflective judgment, and...

Revisiting e-Assessment: Revisiting e Assessment: Leveraging online platforms for outcomes assessment in the general education

Jennifer Hill, Ed.D.

Matt Serra Ph DMatt Serra, Ph.D.

Office of Assessment, Trinity College

Duke University

assessment.aas.duke.edu

1

We argue that assessment occurs most We argue that assessment occurs most authentically in the context of a course, degree program, or co-curricular learning

i C ll h ld i t i experience. Colleges should persist in promoting authentic, embedded assessment in their courses and programs. p g

[ and ]

Given that stakeholder groups also demand generalizable evidence of learning across the gen ed curriculum learning across the gen. ed. curriculum and co-curriculum, we suggest a parallel effort at the college level, to enrich and

l i l l b ti f l iexplain local observations of learning.

4

We argue that assessment occurs most We argue that assessment occurs most authentically in the context of a course, degree program, or co-curricular learning

i C ll h ld i t i experience. Colleges should persist in promoting authentic, embedded assessment in their courses and programs. p g

[ and ]

Given that stakeholder groups also demand generalizable evidence of learning across the gen ed curriculum learning across the gen. ed. curriculum and co-curriculum, we suggest a parallel effort at the college level, to enrich and

l i l l b ti f l i

How do we do this at Duke?explain local observations of learning. Duke?

5

Gen Ed assessment initiative

Our purposes are

( ) t th b li d t t i i t i i (1) to gather baseline data pertaining to incoming students’ levels of ethical reasoning, global perspectives, critical thinking, and reflective judgment, and

(2) to follow the development of student competencies, broadly defined, over time.

6


Our purposes are



Project design allows (a) wide engagement among incoming and exiting students

7


Our purposes are



Project design allows (a) wide engagement among incoming and exiting students, (b) collection of data from three separate nationally-normed instruments, three separate nationally normed instruments,

8


Our purposes are



Project design allows (a) wide engagement among incoming and exiting students, (b) collection of data from three separate nationally-normed instruments, and (c) three separate nationally normed instruments, and (c) efficiencies of administration, data analysis, and reporting.

9

Gen Ed assessment initiative (the front end)

What

To whom

When

How

10


What: Administration of several assessment instruments i t t d t ’ t i l ti t D k E h t t prior to students’ matriculation at Duke. Each targets a

general education student learning outcome.

To whom To whom

When

H How

11


What

To whom: Voluntary sample of incoming first year (non-transfer) students, A&S and Engineering

When

How

12


What

To whom

When: Project started in summer 2010 and continues jannually. Early communications start in April, with periodic reminders in July and August. Data collection ends at the start of first year student orientation.

How

13


What

To whom

When

How: Deploy three online, externally hosted instruments via a central project homepage. Data are returned to us b tby our partners.

14


15


16


Defining Issues Test (DIT-2)ethicaldevelopment.ua.edu

Global Perspectives Inventory (GPI)gpi.central.edu

Reasoning about Current Issues Test (RCI)reflectivejudgment.org

Critical thinking Assessment Test (CAT)tntech.edu/cat/home

17


Publicity: Blurb in “Blue Book”. Email from Director of Academic Advising Center Class Facebook page Academic Advising Center. Class Facebook page posting. Direct email from our office.

Incentives

Consent

Return rates

Challenges

Findings Findings

Next steps18


Publicity

Incentives: Drawing among eligible students. Awards and award values vary. “Hierarchy” of awards.

Consent

Return rates

Challenges

Findingsg

Next steps19


Publicity

Incentives

Consent: Consent statement is embedded on initial project homepage, and on each instrument’s website.

Return rates

Challenges

Findingsg

Next steps20


Publicity

Incentives

Consent

Return rates: Surprisingly high in year 1, fluctuating slightly over time. Lower among graduating cohort.

Challenges

Findingsg

Next steps21


Publicity

Incentives

Consent

Return rates

Challenges: Monitoring completions on externally hosted b R t d i i t ti t i di t web apps. Remote administration prevents immediate

distribution of incentives. Maintaining high return rates.

Findings

Next steps

22


Selected findings from pre-tests (matriculates 2010-2014):

Our voluntary samples of study participants are large and tend to be representative of their respective incoming classes classes.

Results at the pre-test tend to be stable across incoming classes. classes.

Individual students’ results and Duke’s overall results are emailed to them as early as possible. Content of the mailing is confidential, secure, contextualized, and written in accessible language.

23

Gen Ed assessment initiativeResponse rates by instrument, and by matriculating cohort

Overall59% Overall70%

80%90%

100%

d st

uden

ts

Overall48% Overall

42% Overall40% Overall

36%

30%40%50%60%

nt o

f inv

ited

DIT-260%

DIT-246%

DIT-243%

DIT-241%

DIT-238%

GPI66%

GPI54%

GPI47%

GPI42%

GPI33%

RCI52%

RCI45%

RCI37%

RCI38%

RCI36%

0%10%20%

Incoming 2010 Incoming 2011 Incoming 2012 Incoming 2013 Incoming 2014

Perc

en

24

g g g g g

DIT-2 GPI RCI Overall

GPIGPIGlobal Perspectives InventoryGlobal Perspectives Inventory

25

Gen Ed assessment initiative: GPI

1.0

Observed movement in GPI results (pre-test to post-test)Repeat measures cohort (N=119), graduating class of 2014

0.50.60.70.80.91.0

0.00.10.20.30.4

* * * * * al al al y n

Ove

rall G

PI

Cog

nitiv

e

og. K

now

ing

. Kno

wle

dge

ntra

pers

onal

Intra

pers

ona

Iden

tity

Intra

pers

ona

Affe

ct*

Inte

rper

son a

erpe

rson

alRe

spon

sibilit

y

terp

erso

nal

al In

tera

ctio

n

Co

Cog

. In

Inte

Soci

al

Int

Soci

a

26

* Represents significance at p < 0.001

Gen Ed assessment initiative: GPIst


By duration of foreign language coursework

0.5

0.6

0.7

est t

o po

st-te

s By duration of foreign language coursework

0.3

0.4

ovem

ent,

pre-

te

Exactly twoclasses (N=15)*

Exactly one class (N=22)*

0.0

0.1

0.2

3 50 3 60 3 70 3 80 3 90 4 00gnitu

de o

f mo

Non-completion of FL requirement (N=21)

classes (N=15) class (N=22)

Exactly threeclasses (N=61)*

27

3.50 3.60 3.70 3.80 3.90 4.00

Mag

Mean overall GPI score at pre-test* Represents significance at p < 0.01

Gen Ed assessment initiative: GPI

0.7test


By curricular and co-curricular experiences

0.5

0.6

e-te

st to

pos

t-

Participation in exactly

0 2

0.3

0.4

mov

emen

t, pr

e p y2 SL courses (N=10)*

Participation in study abroad (N=68)*

0.0

0.1

0.2

Mag

nitu

de o

f m Participation in 1 or more DukeEngage experiences (N=36)*

Participation in exactly 1 SL course (N=43)*

28

3.50 3.60 3.70 3.80 3.90 4.00M


Gen Ed assessment initiative: GPIte

st


By division of the students' first major

0.5

0.6

0.7

e-te

st to

pos

t-t

S i l S i (N 41)*

0 2

0.3

0.4

mov

emen

t, pr

e Social Sciences (N=41)*

Humanities (N=21)* Other (N=3)

0.0

0.1

0.2

3.50 3.60 3.70 3.80 3.90 4.00Mag

nitu

de o

f m

Engineering (N=14)Natural Sciences (N=40)

29

3.50 3.60 3.70 3.80 3.90 4.00M


DIT 2DIT-2Defining Issues TestDefining Issues Test

30

Gen Ed assessment initiative: DIT

3 0nd. s

core

,

Observed movement in DIT-2 results (pre-test to post-test)Repeat measures cohort (N=101), graduating class of 2014

By curricular and co-curricular experiences

1.52.02.53.0

nge

in T

ype

Into

Pos

t-tes

t

0.00.51.0

01)*

FL

27)*

FL

=9)*

L 49)*

udy

* ctly

)* ct

ly

0)itude

of c

han

Pre-

test

ticip

ants

(N=1

0

Com

plet

ion

of

requ

irem

ent

with

1 c

lass

(N=2

Com

plet

ion

of F

requ

irem

ent

th 2

cla

sses

(N=

ompl

etio

n of

Fre

quire

men

t h

3 cl

asse

s (N

=4

ticip

atio

n in

Stu

Abr

oad

(N=5

5)*

cipa

tion

in e

xac

L co

urse

(N=3

1)

cipa

tion

in e

xac

L co

urse

s (N

=10

Mag

ni

All

par C w C

wit C

wit h Pa

rt A

Parti

c1

SL

Parti

c2

SL

* Represents significance at p < 0.05

32

Gen Ed assessment initiative: DIT

e,

Observed movement in DIT-2 results (pre-test to post-test)Repeat measures cohort (N=101), graduating class of 2014

By division of first major

2 0

2.5

3.0

pe In

d. s

cor

-test

y j

1.0

1.5

2.0

chan

ge in

Ty

-test

to P

ost-

S i l S i (N 31)*E i i (N 18)

0.0

0.5

1 0 2 0 3 0 4 0 5 0 6 0 7 0Mag

nitu

de o

f cPr

e- Social Sciences (N=31)*

Humanities (N=16)

Engineering (N=18)

Natural Sciences (N=34)*

33

1.0 2.0 3.0 4.0 5.0 6.0 7.0M

Mean overall DIT-2 Type Indicator score at pre-test* Represents significance at p < 0.05

RCIRCIReasoning about Current Issues testReasoning about Current Issues test

34


Next steps:

Disseminate program-level findings to stakeholders; facilitate informed discussion and improvement of pedagogy and curricula.

Continue reviewing pre-test/baseline data as cohorts matriculate.

Longitudinal and additional cross sectional analyses continue Longitudinal and additional cross-sectional analyses continue. Review study findings in the context of other assessment endeavors underway at Duke.

Continue to explore other instruments (e g quantitative Continue to explore other instruments (e.g., quantitative reasoning), rotating instruments as assessment needs evolve.

37

hill and serra, 2015 aacu, gen ed assessment …...critical thinking, and reflective judgment, and...

Documents