capstone engineering design learning and assessment

Engineering Education Research Center, Washington State University http://eerc.wsu.edu

Capstone Engineering Design Learning and Assessment

Denny Davis, PhD, PE Washington State University

Engineering Education SeminarPurdue UniversitySeptember 18, 2008


Acknowledgements

• Project Leadership Team– Denny Davis, Howard Davis, Michael Trevisan,

Shane Brown, Washington State University– Steven Beyerlein, Jay McCormack, University of Idaho– Phillip Thompson, Seattle University– Olakunle Harrison, Tuskegee University

• Project Consultants– Susannah Howe, Smith College– Patricia Brackin, Rose-Hulman Institute of Technology– Paul Leiffer, LeTourneau University– Durward Sobek, Montana State University– Jerine Pegg, University of Idaho

• Funding– NSF DUE 0717561: Capstone Engineering Design Assessment:

Development, Testing, and Adoption Research


Project Goal and Objectives

• Goal – Develop an integrated system for effective,

sustainable assessment of capstone engineering design outcomes

• Objectives– Develop an assessment system suitable for broad

adoption in capstone engineering design courses– Document effectiveness of the assessment

system to measure student achievement– Investigate factors that affect assessment

adoption by the capstone engineering design community


Guiding Research Questions

Research Questions: 1. To what extent can assessments

measure desired performances in learner and solution development?

2. How can assessments be integrated effectively into capstone design courses?

3. How can adoption of assessments be encouraged in capstone design courses?


What Outcomes?


Areas of Performance

DesignProcesses

SolutionAssets

SolutionSolutionDevelopmentDevelopment

LearnerLearnerDevelopmentDevelopment

Team-workProfessional

Development

2.2.InformationInformationGatheringGathering

4.4.IdeaIdea

GenerationGeneration

3. 3. ProblemProblem

DefinitionDefinition5.5.

IdeaIdeaEvaluationEvaluation

6.6.IdeaIdea

RefinementRefinement

7.7.ImplementationImplementation

1.1.RecognitionRecognitionof Challengeof Challenge

Iteration/reflection


Learner Development

• Professional Development– Individuals performing and improving

individual skills and attributes essential to engineering design

• Teamwork– Teams developing and implementing

collective processes that support team productivity in design


Solution Development

• Design Processes– Practices implemented that effectively

and efficiently facilitate the production of valuable project assets

• Solution Assets– Results from a design project that meet

needs and deliver satisfaction and value to key project stakeholders


Assessment Framework


Interpretation

Model

Observation

Profile ofProfessional

• Roles• Behaviors

PerformanceCriteria• Learners• Solutions

Course Context• Project Mix

• Professor Preparation • Infrastructure/Resources

• Role in Program

Sampling• Student Sample

• Knowledge Sample• Time of Sample

Measures• Outcomes

• Levels• Metrics

Tasks• Individual Tasks

• Team Tasks

Reporting• Learning• Grading

• Improvement

Expectations• Students• Faculty• Clients

• Administrators• Accreditors• Employers

Scoring• Training

• Reliability

Profile ofLearner

• Background• Skill Set

• Motivation

Capstone Course Assessment Framework

*NRC, Knowing What Students Know.

Assessment Triangle*


Performance Criteria: Learner

• Professional Development– Individuals document professional development

aligned with their personal and project needs, professional behaviors, and ways of a reflective practitioner.

• Teamwork– Teams demonstrate high productivity, synergistic

individual and joint contributions, a supportive team climate, and well-developed team processes.


Performance Criteria: Solution

• Design Processes– Designers resourcefully iterate among problem

scoping, concept generation, and solution realization activities to co-develop problem understanding and a responsive design solution.

• Solution Assets– Designers deliver and effectively defend

solutions that satisfy stakeholder needs for functionality, financial benefit, implementation feasibility, and impacts on society.


Purposes of Assessment

• Measure Achievement– Guide changes in instruction– Gather data for grading– Document student achievement in course– Study learning processes

• Facilitate Learning– Guide learners’ effort to greater learning– Teach self- and peer-assessment skills– Establish reflective practitioner mindset


Assessment Design


Assessing Reflective Practice

• Instructional Activities– Assign reflections on performances– Assign similar reflections multiple times

• Evidence– Quality of reflections improve over time– Reflective assignments reveal gains from natural

reflective practice

Reflectionon Goals

Reflectionon Progress

Reflection onAchievement


Capstone Course Context

Problem ScopingPhase

Concept GenerationPhase

Solution RealizationPhase

Project Timeline

Solution Development

Learner Development?? ?? ???? ?? ??


Structure of Assignments

Self-Rating• Importance• Level

Set Target• Describe it• Action plan

Progress• A strength• An opportunity

Formative

Self-Rating• Importance• Level

Strengths• Describe them• Explain causes

Extension• Envision it• Define impacts

Summative


Structure of Feedback

StudentResponse

Self-Rating

Narrative• Descriptions• Analysis• Extension

Factor 1:

Factor 2:

Level1

Level3

Level5

Level2

Level4

PerformanceMetric

X

X

Narrative Feedback

• Comments and suggestions


Assessing Learner Development




Professional Development Processes

Team Contract

[T]Feedback

Project Timeline

Team Member

Citizenship [I]

Feedback

Team Development Processes

Act

iviti

esTe

am-

focu

sIn

divi

dual

-fo

cus

Team Member

Citizenship [I]

Feedback

Teamwork Achieved [I] Assessment

Assessment Notation: [I] = individual assignment [T] = team assignment

Growth Planning

[I] Feedback

Team Processes

[I] Feedback

Growth Progress [I] Feedback

Growth Achieved [I] AssessmentProfessional

Practices [I] Feedback


Professional Development Assessments

Technical Interpersonal Individual

□ Analyzing information □ Communicating □ Practicing self-growth

□ Solving problems □ Collaborating □ Being a high achiever

□ Designing products □ Relating inclusively □ Adapting to change

□ Researching questions □ Leading others □ Serving professionally

GrowthPlanning

ProfessionalPractices

GrowthProgress

GrowthAchieved


Teamwork Assessments

Team Relationship Joint Achievement Member Contribution Team Information

□ Building an inclusive climate

□ Establishing shared team goals

□ Allocating responsibility

□ Achieving in-team communication

□ Gaining buy-in, interdependence

□ Managing tasks to achieve team goals

□ Achieving quality work from members

□ Stakeholder communication

□ Resolving conflicts

□ Producing consensus

□ Facilitating team member growth

□ Building shared knowledge assets

TeamContract

TeamProcesses

Team MemberCitizenship

TeamworkAchieved


Assessing Solution Development




implementationdesign

Solution Realization Processes

Problem Scoping

Processes [I]Feedback

Project Timeline

Defined Problem [T] /

Design Reflection [I] Assessments

Concept Generation Processes

[I] Feedback

Selected Concept [T] /


Problem Scoping Processes

Concept Generation Processes

Act

iviti

esFe

edba

ckEv

alua

tion

Solution Realization

Processes [I] Feedback

Proposed Solution [T] /


Solution Realization Processes

[I] Feedback

Proposed Solution [T] /


Assessment Notation: [I] = individual assignment [T] = team assignment


Design Processes/Solution Assets

Design Phase– Problem scoping– Concept generation– Solution realization

[Design Phase]Processes

DesignReflection

[Asset]

[Design Phase]Processes

DesignReflection

[Asset]

Asset– Defined problem– Selected concept– Proposed solution


Assessment Evaluation


Assessment Validity and Reliability

• Scoring Reliability– Inter-rater reliability

• Content Validity– Instructor and practitioner content

analysis

• Value to Users– Value gained by students and instructors


Web-Based Implementation


Why Web-Based Implementation?

• Practicality– Reduce paperwork of assessment process– Enable automated processing of data– Facilitate testing of assessments

• Effectiveness– More rapid feedback to students (learning)– Better view of team member performances– Supports reflection on progress and learning


Web-based System Concept

InstructorMakes Assignment

(what, when)

Students Complete Assignment

(ratings, explanations)

Instructor PreparesFeedback

(ratings, comments)

Student RetrievesFeedback

from peers (anonymous)from instructor

Responses Saved and Compiled

(secure database)

1

3

42

Instructor ReceivesSummary Data

(ratings, comparisons)

5


Make Assignment


Student Coaching of Members

What makes it strong?How does it benefit the team?


Scoring of Student Work

Comments or suggestions


Summary

Measure desired performances – Individual and team development– Design process and solution development– Reflective practices

Integration into capstone design– Assignments for instruction (formative)– Assignments for assessment (summative)

Adoption of assessments– Web-based implementation– Reference-based performance scoring– Testing underway


Questions ? ?

ContactDenny Davis, Washington State University

[email protected] or [email protected]


Scoring: Executive Summary

1Novice

2Beginner

3Intern

4Competent

5Expert

Problem or Opportunity

Uninformed; too broad, narrow, or

off-target

Vague understanding; questionable

details

Fair understanding; some reputable

detail

Good understanding;

substantive good detail

Superb understanding;

extensive proven detail

Solution Envisioned

Very vague idea; no vision for applicability

General idea; simple vision for

usefulness

Good idea; reasonable vision

for usefulness

Feasible solution; good vision for

usefulness

Superb solution; clearly useful,

feasible

Benefits of Solution

Unlikely or very limited benefits

possible

Small benefits; very narrow beneficiaries

Moderate benefits; narrow

beneficiaries

Good benefits; multiple varied beneficiaries

Many varied benefits; many beneficiaries

Writing Quality

Many errors; not understandable

Several errors; unclear; not interesting

Few errors; clear; somewhat

interesting

Very few errors; clear; very interesting

Error-free; clear; highly attractive,

compelling


Scoring: Solution Specifications1

Novice2

Beginner3

Intern4

Competent5

Expert

Quality

ClarityIll-defined; not measurable

Vague; general expectations

Ok description, not quantitative

Clear targets, some measurable

Specific measurable performance

targets

RefinementNo additions or improvements

Very minor tweaks; minor improvement

One or more notable

improvements

Several revisions cause

improvement

Several revisions; some

transformative

Com

pleteness

FunctionalityIgnores most

important needsAddresses few

important needs

Addresses several important

needs

Addresses most vital needs, some

others

Addresses all vital, many other

needs

FinancialIgnores financial

needs, opportunities

Vaguely mentions cost limitations

Sets cost limits for project budget

Sets project budget and ROI

target

Sets budget, ROI; targets

opportunities

FeasibilityIgnores most

important issuesAddresses few

important issues

Addresses several important

issues

Addresses most vital issues, some

others

Addresses all vital, many other

issues

Social ImpactIgnores major social & safety

issues

Vaguely mentions social, safety

issues

Defines some social, safety requirements

Cites important codes for

compliance

Embraces all relevant codes &

standards


Scoring: Proposed Solution

Proof of Performance

Functions considered Very few, little breadth Several, good breadth in types

Many, insightful breadth & measures

AnalysisNo analysis; opinions

onlySome analysis;

simplistic methodsExtensive analysis; best

methods

Strength of evidenceNo evidence to support

claimsGood evidence for some

claimsDefensible evidence for

important claims

Proof of Profitability

Development costsMajor concern; not

definedMinor concern: need

clarificationAttractive; sound; fully

justified

Cost/benefit ratioMajor concern; not

definedMinor concern: need

clarificationAttractive; sound; fully

justified

Market potentialLacks potential to fit a

marketSome potential to reach

a marketLikely to serve a growing market

Proof of Feasibility

ProducibilityMajor concern; known

problemsMinor concern: possible

problemsAttractive; methods fully

documented

UsabilityMajor concern; known


problemsAttractive; usability fully

documented

ServiceabilityMajor concern; known


problemsAttractive; servicing fully

documented

Proof of Impact

EnvironmentPossible serious negative impacts

Possible minor negative impacts

Certainly only positive impacts

Legal/politicalPossible serious negative impacts



Health & safetyPossible serious negative impacts



capstone engineering design learning and assessment

Documents