Making Our Way from Anecdote to Evidence

Presentation forGeorgia College & State University

Caroline R. Noyes, Ph.D.Assistant Director, Office of Assessment

2

BACKGROUND AND INTRODUCTION

3

Four Purposes of Evaluation (Mark, Henry, & Julnes ,2000)

1. Assessment of merit and worth: the development of warranted judgments, at the individual and societal level, of the value of a policy or program.

2. Program and organizational development: the effort to use information to directly modify and enhance program operations.

3. Oversight and compliance: the assessment of the extent to which a program follows the directives of statutes, regulations, rules, mandated standards or any other formal expectations.

4. Knowledge development: the discovery or testing of general theories, propositions, and hypotheses in the contexts of policies and programs.

4

Program Mission

Program Goals

Student Learning

Outcomes

5

Program Mission

History: The department has as its mission through the BA in History to provide students with a basic knowledge of selected areas of history; to teach students to think critically in analyzing historical issues, to write clearly and cogently on historical topics, and to conduct historical research; and to provide students with a basic understanding of history as a discipline.

Theatre:We seek to provide the best education possible in theatre and dance for our students, preparing them for careers in the professional and educational arenas both in and out of the performing arts. We cultivate imagination, passion, discipline, and collaboration in the creation of arts and artists. We emphasize interdisciplinary connections and a global perspective as essential components of a liberal arts education.

6

Sample Program Goals

a. Graduates will be able to communicate their knowledge effectively, both orally and in writing.

b. Graduates will demonstrate knowledge of the major authors, periods, movements, and issues in drama and theater.

c. Graduates will understand and apply basic research methods in psychology, including research design, data analysis, and interpretation.

7

Properties of Effective Student Learning Outcomes

Have a clear purpose Use action words Describe meaningful learning Are easily understandable Represent high level learning Result in observable behaviors/products

8 8

Formulating Student Learning Outcomes

Method Inventory of program content Rank most important items Categorize by outcome type Draft outcome statement Revise using criteria for quality outcomes

Use action verbs Should be observable (either directly or indirectly

inferred)

9

Bloom’s Taxonomy of the Cognitive Domain

Synthesis

Analysis

Application

Comprehension

Knowledge

10

Action Verbs Accompanying Bloom’s Taxonomy

Knowledge Comprehension Application Analysis Synthesis EvaluationDescribe Compare Apply Analyze Arrange Argue

Find Describe Classify Appraise Assemble Assess

List Distinguish Complete Categorize Collect Choose

Locate Discuss Construct Compare Combine Conclude

Name Explain Demonstrate Contrast Comply Decide

Recall Express Dramatize Debate Compose Evaluate

Record Identify Employ Diagram Construct Interpret

Relate Outline Illustrate Differentiate Create Judge

Repeat Predict Interpret Distinguish Design Justify

State Recognize Operate Examine Devise Measure

Tell Restate Practice Experiment Formulate Rate

Underline Tell Schedule Inspect Imagine Recommend

Write Translate Sketch Inventory Manage Revise

Solve Investigate Organize Support

Use Question Plan Value

Separate Predict Verify

11

Bloom’s Taxonomy of the Affective DomainOrganization

Valuing

Responding

Receiving

12

Action Verbs Accompanying Bloom’s Taxonomyhttp://www.abet.org/_TrainingCD/data/references/Assessment%20Links/Goals_revised_Blooms.pdf

Receiving Responding Valuing Organization CharacterizationAsks Answers Completes Adheres Acts

Chooses Assists Describes Alters Discriminates

Describes Complies Differentiates Arranges Displays

Follows Conforms Explains Defends Influences

Gives Discusses Follows Explains Listens

Identifies Helps Initiates Generalizes Modifies

Names Performs Joins Identifies Performs

Selects Presents Justifies Integrates Practices

Replies Selects Reads Modifies Proposes

Uses Tells Reports Organizes Qualifies

Selects Prepares Questions

Shares Relates Serves

Studies Synthesizes Solves

Works Uses

Verifies

13

Student Learning Outcomes: Oral Communication

Graduates will be able to communicate their knowledge effectively, both orally and in writing.

Students will:

Knowledge support arguments with relevant and adequate evidence

Skills answer questions directly and accurately

Dispositions convey enthusiasm for their topic

14

Student Learning Outcomes: Written Communication

Graduates will be able to communicate their knowledge effectively, both orally and in writing.

Students will:

Knowledge respond to essay questions on exams with clear, well-organized presentations focused on the topic and relatively free of errors

Skills recognize well-written texts and advise peers on how to improve their written work

Dispositions develop a personal voice and sense of style in writing

15

Student Learning Outcomes: Research Methods (Psychology)

Graduates will understand and apply basic research methods in psychology, including research design, data analysis, and interpretation

Students will:

Knowledge articulate strengths and limitations of various research designs

Skills formulate testable hypotheses, based on operational definitions of variables

Dispositions recognize that theoretical and socio-cultural contexts as well as personal biases may shape research questions, design, data collection, analysis, and interpretation

16

WORKING ON LEARNING OUTCOMES IN GROUPS

17

GC&SU Core Outcomes

• Communicate clearly in written English, demonstrating comprehension, analysis, and critical interrogation of a variety of texts. (A: Essential Skills; English)

• Use description, analysis, and synthesis of data, ideas, or information appropriate to the purpose. (A: Essential Skills: English)

• Express and interpret mathematical information, concepts, and thoughts in verbal, numeric, graphical, and symbolic form, and to shift among these different modes when solving problems. (A: Essential Skills; Math)

• Use appropriate technology in the location, evaluation, analysis, synthesis, and application of information in problem-solving situations (A: Essential Skills; hybrid of Math and English)

18

FINDING THE EVIDENCE

19

Things to Consider:

What’s already in place? Exams, assignments, or projects? Common exams, assignments, or projects? Writing samples (common question) Capstone experience products

Look for logical assessment points in the curriculum At program entrance Program courses required for the major At program conclusion

The evidence you collect depends on the questions you want to answer

20

Kinds of Evidence, Part 1

Direct EvidenceMeasure student learning outcomesdirectly

Embedded questions/tasks Pre-post testing Evaluation of projects/work Standardized tests (e.g., MFT) Portfolios Presentations

Indirect EvidenceMeasure opinions or thoughtsabout students’ (or alumni) own knowledge, skills, attitudes,learning experiences, etc. Also includes external evaluations. Alumni surveys Exit surveys Employer surveys Focus Groups

21

Kinds of Evidence, Part 2

Quantitative Evidence Ratings of student

skills/performance Scores and pass rates on tests

(licensure/certification) Scores on locally-designed

tests (final exams, qualifiers, comprehensives) when accompanied by test “blueprints”

Classroom response systems (clickers)

Qualitative Evidence Observations of student

behavior w/ systematic notes Summaries of discussion

threads Student reflections Portfolios evaluation Written work, performances,

or presentations scored with a rubric

Analysis of comments on surveys

22

Student Learning Outcome: answer questions directly and accurately

Course Point of Assessment

Assessment Method

Results

Introduction to Psychology Oral presentation on 40 STCP chapter

Presentation rubric (professor)

Research Methods Oral presentation of research project

Presentation rubric (professor and peers)

Psychology Seminar Classroom guest lecture on historical figure in psychology

Presentation rubric (professor)

23

Oral Presentation Rubric (http://www1.uprh.edu/cruzmigu/OralRubric.pdf)

4 3 2 1Subject Knowledge

Student demonstrates full knowledge by answering all class questions with explanations and elaboration

Student is at ease with expected answers to all questions, without elaboration

Student is uncomfortable with information and is able to answer only rudimentary questions

Student does not have grasp of information; student cannot answer questions about subject.

24

Student Learning Outcome: articulate strengths and limitations of various research designs

Course Point of Assessment

Assessment Method

Results

Introduction to Psychology 1st Test Multiple Choice or Matching Question

Developmental Psychology 1st Test

Final Exam

Application short answer

Essay questionResearch Methods a) 1st Test

b) Research Proposal

Application question

Rationale for research method selection

25

IDENTIFYING SOURCES OF EVIDENCE IN GROUPS

26

SHARING YOUR FINDINGS

27

At Last: Reporting your Findings

Analyze the data…identify the results

Integrate results from various assessment methods

What conclusions can you draw from that evidence?

What recommendations arise from that data?

What actions should/will you take based on the recommendations?

28

Student Learning Outcome: answer questions directly and accurately

Course Point of Assessment

Assessment Method

Results

Introduction to Psychology Oral presentation on 40 STCP chapter

Presentation rubric (professor)

Spring 2010: n=55, avg. score was 1.9

Research Methods Oral presentation of research project

Presentation rubric (professor and peers)

Fall 2009: n=25, avg. score was 2.6 (avg. peer rating was 3.1)

Psychology Seminar Classroom guest lecture on historical figure in psychology

Presentation rubric (professor)

Spring 2010: n=25, avg. score was 2.9

In general, as students progress though the major, there is improvement in students’ ability to answer questions at the end of their oral presentation, but students may benefit from more practice in elective classes. Increase use of peer ratings to increase student critical reflection on this task.

29

Closing the Loop, Option 1

Outcome

Practice

30

Closing the Loop, Option 2

Practice

Outcome

31

Conclusions

The “systematic collection, review, and use of information about educational programs undertaken for the purpose of improving learning” allows us to address three fundamental questions:

1. What have our students learned?2. How well have our students learned that information?3. How successful have we been at what we are trying to

accomplish?

32

Contact Information

Caroline R. Noyes, Ph.DAssistant Director, Office of AssessmentGeorgia Institute of Technology(404) 894-0510cnoyes@gatech.edu