Transformative AssessmentTransformative Assessment

Joan K. LippincottCoalition for Networked Information

Gary BrownWashington State University

Transformative AssessmentTransformative Assessmentat

Washington State UniversityWashington State University

Gary Brown, DirectorThe Center for Teaching, Learning, & TechnologyThe Center for Teaching, Learning, & Technologybrowng@wsu.edu

Transformative Assessment PracticesTransformative Assessment Practiceshttp://www.educause.edu/TransformativeAssessment/2599

• A program initiated by:– EDUCAUSE/National Learning Infrastructure Initiative

(NLII)– Coalition for Networked Information (CNI)– The Teaching, Learning, & Technology Group (TLT

Group)– The Center for Teaching, Learning, & Technology

at Washington State University• Elicit new ideas about assessment• Help institutions put these ideas into practice• Map assessment to assessment

Dimensions of Transformation

1. Purpose of Assessment• Outcomes not coverage (Learning not Teaching)

• Alignment

2. Data• Learning & Learning Skills

3. Application• Systematic

• Integrative

• Closes the loop

4. Dissemination• Embrace new collaborations & new constituencies

AdministrativeAdministrative

ProgressiveProgressive

TransformativeTransformative

The purposeful gathering and use of data and data analysis to ensure that the application of findings and the dissemination of results will help align, substantially change and enrich the student learning experience.

Transformative Assessment is…

What is Transformative Assessment?

Not TransformativeNot Transformative Transformative

4. Student Evaluations Learning Process Evaluations

5. Tests Outcomes & Collective Rating

3. Selectivity Measures Equity Measures

2. Old Accreditation New Accreditation

Trans-course Collaborations1. Technology

TechnologyTechnology

Enrollments in WSU Online Learning SpacesEnrollments in WSU Online Learning Spaces

0

10000

20000

30000

40000

50000

60000

70000

1998 1999 2000 2001 2002 2003 2004

Pedagogy & Technology

Good to Great (Collins, 2001,Harper Business)

• “Technology and technology-driven change has virtually nothing to do with igniting a transformation from good to great.”

• “Yet, paradoxically, great organizations are pioneers in the application of carefully selected technologies.”

Transformative…Transformative… Trans-Course CollaborationTrans-Course Collaboration

(using technology)(using technology)

Online Decision ScienceOnline Decision Science

• Saved a dairy plant $250,000• Saved the plant• Received $95,000 from company executives

to complete a study on call-center efficiency.– And incentives from the company resulted in

more employees enrolling in the course. • Managed pantry inventory for a house

of teens

In A Human Development ClassIn A Human Development Class• “This information [researched &

reported by students] was better than text books that are obsolete as soon as they are printed.”

• “The students in this course provided up to date pertinent information about current issues and how to affect change.”

• “The next group of students who take this class will write their own text book.”

NOTNOT

Transformative…Transformative… Old AccreditationOld Accreditation

The Good Old Days

My Class Your Class His Class Her Class Another Class

1st Year Content

2nd Year Content

3rd Year Content

4th Year Content

Capstone Content

The Good Old Days

My Class Your Class His Class Her Class Another Class

1st Year Content

2nd Year Content

3rd Year Content

4th Year Content

Capstone Content

Transformative…Transformative… Accreditation Accreditation

Focus onFocus on

Closing the LoopClosing the Loop

Outcomes—A Science ProgramOutcomes—A Science Program

• Faculty assessed competency of 60 papers collected from 11 different courses of their program.

• Maintained inter-rater reliability of 75%

Why Worry About Consensus?Why Worry About Consensus?

• Accreditation– High standards are not the same as

standardization…• Legal

– Faculty content expertise trumps faculty exam design expertise.

• Curricular coherence– No single course will meet all

program/institutional outcomes.

0

1

2

3

4

5

6

Graduate Students Upper Division Lower Division

A Science Program OutcomesA Science Program Outcomes Fall PilotFall Pilot

2003

Professional Competency

Mastery

Developing

Emerging

Crisis and Opportunity

Closing the Loop

• Faculty agreed:–Assignments need to be revised to align

with the shared OUTCOMES.–Jointly create & embed assignments for

target points in curricula…– Include accreditors, professional

advisory board partners, and upper-level students in the process• Use an ePortfolio….

Round OneRound One(Of a Continuous Process)

“Working together to design assignments is unusual because faculty are accustomed to working independently.  However, collaboration should enable improvement of individual assignments, thus the entire curriculum shall benefit.”

—Stephanie Clark, FSHN

NOT NOT

Transformative…Transformative… Selectivity MeasuresSelectivity Measures

An Inland Northwest Land GrantGOAL #1 Offer the Best Undergraduate Experience in a

Research UniversitySub goal 1

Attract, recruit, and retain a diverse high quality student body.

Required actions Implement recruitment and admissions strategies that reach and serve high ability students from high schools and community colleges.

What DoesWhat Does

Institutional Selectivity Institutional Selectivity Tell Us AboutTell Us About Educational Quality? Educational Quality? “National magazines that purport to

identify the nation’s ‘best’ colleges are essentially ranking institutions by their selectivity, not by the likelihood of their exposing students to the most effective educational practices.”

“Given the challenges facing the nation and the higher education system, it’s time for some straight talk about the deleterious grip that selectivity has had on our perceptions of what constitutes collegiate quality. It’s bad public policy and educationally indefensible.”

“It would be much more productive to focus on developing indicators that more accurately represent what happens to students during college and to make this information available in a responsible way so that prospective students, policymakers, and institutional leaders can use it to make decisions that can improve student learning.”

George D. Kuh and Ernest T. Pascarella (Change, September/October 2004 pp.53-58)

The Center for Urban EducationEstela Mara Bensimon, Director, CUE

Equity Frame Color-conscious, awareness of racialized practices, personal and collective responsibility/accountability for outcomes, focus on outcomes

Diversity Frame Visibility of racial and ethnic diversity, diversity as an institutional property, cross-racial relationships, color blind, focus on access

Deficit Frame Underperformance is person-centered & attributed to membership in a particular group; “at risk,” lack of motivation, lack of socialization, missionary attitude

Mental ModelsDifferent Questions

Diversity Questions Equity Questions

In what ways can we attract more students of color to our campus?

In what ways can we improve the educational outcomes of historically underrepresented students?

In what ways can we increase students’ understanding of cultural, racial, and ethnic differences?

In what ways can we—as faculty members, deans, counselors—be more accountable for the outcomes of historically underrepresented students?

NOT…NOT… Traditional Student EvaluationsTraditional Student Evaluations

Traditional Student EvaluationsAdministrative Progressive Transformative

Purpose

Data

Application

Dissemination

Rank Instructors

Student Satisfaction

RankInstructors

Between Chair & Faculty

The GOLDGOLD StandardParticipants Who Used Survey Data to Inform Change….

257

106

5516 2

0

100

200

300

Invited faculty Faculty who tookinstructor survey

Faculty w/ studentresponses

Faculty w/ >50%student response

rate

Faculty who usedthe data

Transformative…Transformative… Learning Focused Evaluations

Student Profile

CourseQuestions

Learning Environment

Critical Engagement

Relevance

College Questions

Department Questions

The TLT Group BeTA Project:The TLT Group BeTA Project:

Student Evaluations

Peer Review

Faculty DevelopmentSeminar Series

Validation

Outcomes Development(Rubrics)

Student Evaluations

Validation

Peer Reviews&

Focus Groups

0123456

Your Goals Your Results

Reporting Results

Evaluation as Community Practice

Administrative Progressive Transformative

Purpose

Data

Application

Dissemination

Community of Evidence

DeepenOutcomes

Public Seminars

Trans-Institutional& Community

SUNSHINE

LOGO

Bringing it TogetherBringing it Together

• Goal, Assessment, Assignment & Curricula Alignment

• Collective Community Engagement:– Activities, projects are co-created– Performance criteria are shared with students

• Technology in Community Context – ePortfolio– Blog-based Reflection

NOT…NOT… TestsTests

Transformative…Transformative… Outcomes Outcomes

& & Collective JudgmentCollective Judgment

Assignment Assessment & the Echo EffectAssignment Assessment & the Echo Effect

0

1

2

3

4

5

6

Faculty Assignment Student Critical Thinking

Mastery

Developing

Emerging

From the Messy NSSEGeorge Kuh

• “Students typically don’t exceed their own academic expectations.”

• But “Students will go beyond what they think they can do under certain conditions, especially when teachers expect, challenge, and support them.”

• “Are we willing to make the effort that such practices demand of us?”

Final ThoughtFinal Thought

Transformation is not an event…

IT’s a Lifestyle!

Ten Lessons from TAPTen Lessons from TAP

Joan K. Lippincott

Coalition for Networked Information

TAP ProgramTAP Program

• Focus Sessions

• READI Branch

• Rubric

• Case studies

• Report

What Did We Learn?What Did We Learn?

• Not many institutions are implementing transformative assessment programs

• Most institutions understand the need for better assessment practices

• Higher education is subject to pressures for accountability

1: Why Assess?1: Why Assess?

• Improve teaching and learning• Provide evidence of the value of the investment

in technology for teaching and learning– Produce students who can contribute to the

knowledge economy– Contribute to community development– Extend benefits of higher education to more sectors of

the society

2: Understand What 2: Understand What You Want to AssessYou Want to Assess

• Locate and review institutional documents

• Choose large issues that have importance for your institution

• Align the assessment goals and objectives with institutional goals and objectives

Institutional GoalsInstitutional Goals

Align Goals and AssessmentAlign Goals and Assessment

3: Work Towards an Institutional 3: Work Towards an Institutional Culture of AssessmentCulture of Assessment

• Assessment is embedded in major institutional processes such as strategic planning, budgeting, curriculum review

• Studies inform others’ work across the institution

• Incentives are available

4: Seek Support of Top Administrators

• Need for vision and mission related to teaching, learning, and technology

• Need for resources

• Need for action based on assessment findings

5: Work with Assessment Experts5: Work with Assessment Experts

• Campus– Institutional Research or Planning– Undergraduate Education– Center for Teaching and Learning

• Consultants– Flashlight Program, TLT Group

6: Assemble Stakeholders6: Assemble Stakeholders

• Academic administrators

• Faculty senate

• Center for Teaching Excellence

• IT/Library/Instructional Technology

• Assessment Experts

• “Risk-takers and consensus-seekers” (Carmean)

7: Consider Qualitative and 7: Consider Qualitative and Quantitative MeasuresQuantitative Measures

• “Triangulation”

• The power of stories

“This summer, I worked for a biomedical engineeringCompany… and I used what I learned from the class… I had a problem to solve, which required an immense amount of background research, which had to be organized in an effective manner. I knew exactly how to acquire reputable resources, document them, and put them together in a literature review. I used every aspect of the Library Studies class, and my employers were very impressed with my ability…”

Student quoted in Snavely and Wright, 2003

8: Take Advantage of Others’ 8: Take Advantage of Others’ Tools and ExperiencesTools and Experiences

• National Survey of Student Engagement (NSSE)

• AAHE Assessment Forum

• TLT Group Flashlight Project

• ACRL Information Literacy Standards

9: Be Persistent9: Be Persistent

“Transformative assessment practice lies in ruthless inquiry and a determined commitment of resources.”

Colleen Carmean, “Assessment Boot Camp,” 2003

10: Use What You Have Learned 10: Use What You Have Learned to Improve Your Institutionto Improve Your Institution

• Use findings to systematically inform and reshape teaching and learning practice and to promote a culture of evidence

NLII Key Theme:NLII Key Theme:Alignment & AssessmentAlignment & Assessment

http://www.educause.edu/Alignment/934