1©2003-2005 Subjective Metrics, Inc.05.11.2005

waypointonline assessment and structured peer review

COAS: Drexel University

Andrew J. McCannvisiting professor of english, drexel university

founder and president, subjective metrics, inc.www.gowaypoint.com

ajmccann@subjectivemetrics.com

215.713.9393

2©2003-2005 Subjective Metrics, Inc.05.11.2005

Agenda

• Background

• Introduce waypoint

– Brief overview

• Drexel applications

• Live Demonstration

– Evaluate

– Manage

– Libraries

3©2003-2005 Subjective Metrics, Inc.05.11.2005

Problem: Feedback

• Handwritten and/or manually typed

• “Lost”

• Little accountability

• Consistency issues

• No data

• Needed: not AI – but a technological tool

4©2003-2005 Subjective Metrics, Inc.05.11.2005

Waypoint: Evaluation Tool

• Encourages pre-written feedback:

clearer, more detailed explanations

of key concepts

• Facilitates sharing of assessments

amongst instructors

• Quantifies evaluations by skill

• Archives all feedback

Web-based:

• cross-platform

• backed up

• zero maintenance

5©2003-2005 Subjective Metrics, Inc.05.11.2005

Waypoint Process

6©2003-2005 Subjective Metrics, Inc.05.11.2005

Develop Assessment

7©2003-2005 Subjective Metrics, Inc.05.11.2005

Develop Assessment

8©2003-2005 Subjective Metrics, Inc.05.11.2005

Receive Paper or Exam(Hard copy shown here)

9©2003-2005 Subjective Metrics, Inc.05.11.2005

Receive Paper or Exam(Hard copy shown here)

10©2003-2005 Subjective Metrics, Inc.05.11.2005

Evaluate Against Skills

11©2003-2005 Subjective Metrics, Inc.05.11.2005

Evaluate Against Skills

12©2003-2005 Subjective Metrics, Inc.05.11.2005

Respond to Student

13©2003-2005 Subjective Metrics, Inc.05.11.2005

Respond to Student

14©2003-2005 Subjective Metrics, Inc.05.11.2005

Snapshot Analysis

15©2003-2005 Subjective Metrics, Inc.05.11.2005

Snapshot Analysis

16©2003-2005 Subjective Metrics, Inc.05.11.2005

Longitudinal Cohort Analysis

Organizational Behavior: Research Competency1 to 4 scale; 110 students evaluated

0

10

20

30

40

50

60

70

1 2 3 4

# T

eam

s Feb '04Mar '04

Jun '04

17©2003-2005 Subjective Metrics, Inc.05.11.2005

Process Summary

Organizational Behavior: Research Competency1 to 4 scale; 110 students evaluated

0

10

20

30

40

50

60

70

1 2 3 4

# T

eam

s Feb '04Mar '04

Jun '04

18©2003-2005 Subjective Metrics, Inc.05.11.2005

Drexel Applications

• tDec Humanities (Dr. Valarie Arms)– Evaluation of all major writing assignments (600 students)

• Writing Center (Harriet Millan)– Customized assessments for WITs

• College of Engineering (Kevin Scoles & Adam Fontecchio)– Evaluation of lab reports with WITs and TAs

• College of Business (Frank Linnehan)– Structured response to student writing & accreditation data

generation

• Engineering Management (Mike Scheuerman)– Peer review

19©2003-2005 Subjective Metrics, Inc.05.11.2005

Demonstration

• www.gowaypoint.com• Evaluate

– “Final Report”• Peer Review

– “Final Report”• Manage

– Quantification– Sorting– Data analysis

20©2003-2005 Subjective Metrics, Inc.05.11.2005

Additional Features

• Self-assessment

• Collaborative assessment

– Multiple instructors can contribute to evaluation

21©2003-2005 Subjective Metrics, Inc.05.11.2005

Questions and Discussion

22©2003-2005 Subjective Metrics, Inc.05.11.2005

Testimonials

• “I've been doing this for twenty years and have my own system of grading and commentary…and found little need to improve. And then Waypoint came along. Somehow--and I really can't even explain it--but my grading time has been cut down by more than half and my students are actually thanking me for the in-depth commentary.”– Professor Ken Bingham

• “Usually I have trouble criticizing a peer’s paper if I’m not given certain criteria to judge. During this peer review, I was actually focused and excited about judging a peer’s paper.”– Erin Williams, COE 2008

• “If this peer review program was available in high school, I would have probably done a lot better in English.” – Ed Itaas, COE 2008

23©2003-2005 Subjective Metrics, Inc.05.11.2005

Testimonials

• “An innovative teacher-friendly, student-friendly, efficient approach to grading writing—the most creative and time-saving method for evaluating writing I have ever found or used.”

– Gayle, a high school English teacher with 35 years’ experience

• “I don’t know how I ran a writing program for three years without it.”

– Harriet Millan, Director of the University Writing Program

24©2003-2005 Subjective Metrics, Inc.05.11.2005

Quantitative Data

• Anonymous survey of Peer Review process

– 114 freshman engineers: 51 use waypoint, 63 the ‘old’ method

1-5 Scale (1 – strongly disagree, 5 – strongly agree)

wp (N=51)

Mean

‘old’ (N=63)

Mean

P value

(t-test, 95% CL)

1. I received clear and helpful criticism of my draft.

3.90 3.52 p<0.025

2. Evaluating other students’ papers helped me better understand the assignment and the play.

3.86 3.48 p<0.02

3. I made significant changes to my first draft based on peer review feedback.

3.81 3.25 p<0.005

4. I made significant changes to my first draft independent of peer review feedback.

3.49 3.43 p=0.38