11

Inc.

Choosing Assessment Methods

Gloria Rogers

Cop

yrig

ht

201

0 by

AB

ET,

Gloria Rogers

Wewillbeginpromptlyat2:00EST

2

Overview9Where does assessment fit into a

Continuous Improvement process?9Wh t di t d i di t ?

Inc.

9What are direct and indirect measures?9Review of assessment methods9Making the right choice of methods9Assessment method truisms9H d t diff f

Cop

yrig

ht

201

0 by

AB

ET,

9How does assessment differ from program educational objectives to learning outcomes

2Program(Student)

MissionProgramEducationalObjectives

Assess/E l t

g ( )Outcomes

FeedbackforContinuousImprovement

PerformanceIndicators

EducationalPractices/Strategies

Evaluate

Constituent/Stakeholder

AssessmentforContinuousImprovement

Assessment:Collection,Analysis

ofEvidence

Evaluation:Interpretationof

Evidence

GloriaRogers ABET,Inc

Program Educational

Student Outcomes

Performance Indicators

S d illEducational Objective

Graduates will be effective life-long learners including

demonstrating the professional and ethical responsibilities

Students will demonstrate:

Appreciation for and ability to pursue life-long learning

Understand professional ethical responsibilities

G.Rogers--ABET, Inc.

3Program Educational

Student Outcomes

1)Demonstrate knowledge

Performance Indicators

Educational Objective

1)Demonstrate knowledge of professional code of ethics.

2)Evaluate the ethical dimensions of a problem in the discipline.

Graduates will be effective life-long learners including

demonstrating the professional and ethical responsibilities

Understand ethical responsibilities

G.Rogers--ABET, Inc.

6

Direct MeasuresDirect measures provide for the direct examination or observation of student

Inc.

examination or observation of student knowledge or skills against measurable learning outcomes

Cop

yrig

ht

201

0 by

AB

ET,

47

Indirect Measures

Indirect measures are those that

Inc.

ascertain the opinion or self-report of the extent or value of learning experiences.

Cop

yrig

ht

201

0 by

AB

ET,

AssessmentMethods

Written surveys and ti i Portfoliosquestionnaires

Exit and other interviews

Standardized exams Locally developed

Portfolios Simulations Performance

Appraisal External

e aminery pexams Archival records Focus groups

examiner Oral exams Behavioral

observations

5Written Surveys andQuestionnaires

9Asking individuals to share their ti b t th ( th iperceptions about the program (e.g., their

own or others skills/attitudes/behavior, or program/course qualities and attributes) Most common indirect measure Usually locally developed but also some y y p

national surveys that allow for comparisons (e.g., National Survey of Student Engagement, Educational Benchmarking, Inc.)

10

Exit and Other Interviews9Asking individuals to share their

perceptions about the program (e g their

Inc.

perceptions about the program (e.g., their own skills/attitudes, skills and attitudes of others, or program qualities) in a face-to-face dialog with an interviewer Generally indirect measure

Cop

yrig

ht

201

0 by

AB

ET,

Interview could be crafted to include elements of direct measures

611

Standardized Exams9Subject-specific examinations, generally

group administered mostly multiple choice

Inc.

group administered mostly multiple choice, objective tests, usually purchased from a private vendor Direct measure of student learning Provide ability to make comparisons with

th

Cop

yrig

ht

201

0 by

AB

ET,

other programs Need to be confident that it is relevant to the

program for which it is used

Locally developed exams9Objective (includes true-false, fill-in-the

blank, matching, and multiple choice g pquestion) and/or subjective (open-endedrequire students to write) tests designed by faculty of the program Most common at classroom level

Direct measure of student learning Direct measure of student learning Can be specific to performance indicators for

the learning outcomes Can be difficult to get faculty agreement on

questions related to outcomes

713

Focus Groups

9Group discussions conducted by a trained d t ith ti i t t id tif

Inc.

moderator with participants to identify trends/patterns in perceptions Indirect method that can provide valuable

information about student perceptions and experiences

Cop

yrig

ht

201

0 by

AB

ET,

Can be used to provide insights about student responses on other assessments

Results cannot be generalized to entire cohort

14

Archival Records9Biographical, academic, or other file data

available from the college or other

Inc.

gagencies and institutions Identify data already available (data audit) Build upon data collection efforts that have

already occurredConstitutes non intrusive measurement not

Cop

yrig

ht

201

0 by

AB

ET,

Constitutes non-intrusive measurement, not requiring additional time or effort from students or other groups

8Portfolios9Collections of student work which is archived

and rated for level of attainment using scoring g grubrics. The design of a portfolio is dependent upon how the scoring results are going to be used. Direct measure of student learning Possible to measure more than one learning oss b e to easu e o e t a o e ea g

outcome at one time (e.g., writing and use of technology)

Course management systems often support portfolio development

16

Simulations (Competency-Based Measure)

9A persons abilities are measured in a In

c.

situation that approximates a real world world setting Direct measure of student learning Need well defined outcomes with appropriate

tasks

Cop

yrig

ht

201

0 by

AB

ET,

tasks Can be designed for individuals and groups of

students

917

Performance Appraisals9Systematic measurement of the

demonstration of acquired skills through di t b ti

Inc.

direct observation Provides a direct measure of students

abilities to apply what has been learned Internships and co-op experiences provide a

good setting for data collection

Cop

yrig

ht

201

0 by

AB

ET,

Need to be focused data collection process Those who are in a position to make judgment Well constructed instrument for data collection

18

External Examiner9Using an expert in the field from outside the

program (usually from a similar program at

Inc.

another institution) to conduct, evaluate, or supplement assessment of your students Generally a direct measure of student learning (if they

assess against specific competencies) Outsiders can see attributes to which insiders have

grown accustomed

Cop

yrig

ht

201

0 by

AB

ET,

grown accustomed Evaluators may have skills, knowledge, or resources

not otherwise available

10

19

Oral Exams9An assessment of student knowledge

levels through a face-to-face dialogue

Inc.

levels through a face to face dialogue between the student and examinerusually faculty Direct measure of student learning Content and style can be geared to specific learning

outcomes and characteristics of the program,

Cop

yrig

ht

201

0 by

AB

ET,

p g ,curriculum, etc.

May not be allowed by institution who have concerns about pressure on students

20

Behavioral Observations9Measuring the frequency, duration,

relationships, etc. of student actions, usually in a

Inc.

natural setting with non-interactive methods (e.g., formal or informal observations in a classroom). Direct measure of student behavior Observations are most often made be an

Cop

yrig

ht

201

0 by

AB

ET,

individual and can be augmented by audio or videotape.

Requires experienced observers

11

21

Direct Indirect Exit and other interviews Standardized exams Locally developed exams

Written surveys and questionnaires

Inc.

Locally developed exams Portfolios Simulations Performance Appraisal External examiner

O l

questionnaires Exit and other

interviews Archival records Focus groups

Cop

yrig

ht

201

0 by

AB

ET,

Oral exams Behavioral observations

22

Validity9relevance - the assessment option

measures the educational outcome as directly as possible

Inc.

directly as possible 9accuracy - the option measures the

educational outcome as correctly as possible 9utility - the option provides formative and

summative results with clear implications

Cop

yrig

ht

201

0 by

AB

ET,

summative results with clear implicationsfor educational program evaluation and improvement

12

23

Bottom Lines9All assessment options have advantages

and disadvantages

Inc.

9Ideal method means those that are best fit between program needs, satisfactory validity, and affordability (time, effort, and money)9Crucial to use multi method/multi source

Cop

yrig

ht

201

0 by

AB

ET,

9Crucial to use multi-method/multi-source approach to maximize validity and reduce bias of any one approach

TRIANGULATIONTRIANGULATION

PortfoliosTruthTruth

13

TRIANGULATION*TRIANGULATION*

TruthTruth

Portfolios

*Joseph *Joseph HoeyHoey--Savannah College of Art and DesignSavannah College of Art and Design

26

Assessment Method Truisms9There will always be more than one way

to measure any learning outcomeIn

c.

9No single method is good for measuring a wide variety of different student abilities9There is generally an inverse relationship

between the quality of measurement methods and their expediency

Cop

yrig

ht

201

0 by

AB

ET,

p y9It is important to pilot test to see if a

method is appropriate for your program

14

27

Assessing Program EducationalObjectives

9Similar to program learning outcomes butt th

Inc.

not the same 9What are some of the differences?

Degree of specificity Role of constituents

Cop

yrig

ht

201

0 by

AB

ET,

Types of measurements possible Cycles of data collection

27

28

Sampling

9For program assessment, sampling is acceptable and even desirable for

Inc.

acceptable and even desirable for programs of sufficient size. Sample is representative of all students

Cop

yrig

ht

201

0 by

AB

ET,

15

29

Data collection

Inc.

Data collection

Evaluation & design of

improvements

Implement improvements

& Data Collection

Yr 1 Yr 2 Yr 3 Yr 4 Yr

Define Outcomes/ Map

Curr.

Cop

yrig

ht

201

0 by

AB

ET,

Collection

30

Learning Outcomes: 08-09 09-10 10-11 11-12 12-13 13-14

A recognition of ethical and professional responsibilities

An understanding of how contemporary issues shape and are shaped by mathematics science &

In

c.

shaped by mathematics, science, & engineering

An ability to recognize the role of professionals in the global society

An understanding of diverse lt l d h i ti t diti

Cop

yrig

ht

201

0 by

AB

ET,

cultural and humanistic traditions

An ability to work effectively in teams

An ability to communicate effectively in oral, written, graphical and visual forms

16

31

Inc.

Cop

yrig

ht

201

0 by

AB

ET,

32

Inc.

Cop

yrig

ht

201

0 by

AB

ET,

17

33

Th k f i i i !

Inc.

Thank you for your participation! Please fill out the closing evaluation. All webinars can be ordered online:

www.abet.org/webinar.shtml

Cop

yrig

ht

201

0 by

AB

ET,