Making Our Way from Anecdote to EvidenceMaking Our Way from Anecdote to Evidence
Presentation forGeorgia College & State University
Caroline R. Noyes, Ph.D.Assistant Director, Office of Assessment
3
Four Purposes of Evaluation (Mark, Henry, & Julnes ,2000)
Four Purposes of Evaluation (Mark, Henry, & Julnes ,2000)
1. Assessment of merit and worth: the development of warranted judgments, at the individual and societal level, of the value of a policy or program.
2. Program and organizational development: the effort to use information to directly modify and enhance program operations.
3. Oversight and compliance: the assessment of the extent to which a program follows the directives of statutes, regulations, rules, mandated standards or any other formal expectations.
4. Knowledge development: the discovery or testing of general theories, propositions, and hypotheses in the contexts of policies and programs.
5
Program MissionProgram Mission
History: The department has as its mission through the BA in History to provide students with a basic knowledge of selected areas of history; to teach students to think critically in analyzing historical issues, to write clearly and cogently on historical topics, and to conduct historical research; and to provide students with a basic understanding of history as a discipline.
Theatre:We seek to provide the best education possible in theatre and dance for our students, preparing them for careers in the professional and educational arenas both in and out of the performing arts. We cultivate imagination, passion, discipline, and collaboration in the creation of arts and artists. We emphasize interdisciplinary connections and a global perspective as essential components of a liberal arts education.
6
Sample Program GoalsSample Program Goals
a. Graduates will be able to communicate their knowledge effectively, both orally and in writing.
b. Graduates will demonstrate knowledge of the major authors, periods, movements, and issues in drama and theater.
c. Graduates will understand and apply basic research methods in psychology, including research design, data analysis, and interpretation.
7
Properties of Effective Student Learning Outcomes
Properties of Effective Student Learning Outcomes
Have a clear purpose Use action words Describe meaningful learning Are easily understandable Represent high level learning Result in observable behaviors/products
8 8
Formulating Student Learning OutcomesFormulating Student Learning Outcomes
Method Inventory of program content Rank most important items Categorize by outcome type Draft outcome statement Revise using criteria for quality outcomes
Use action verbs Should be observable (either directly or indirectly
inferred)
9
Bloom’s Taxonomy of the Cognitive DomainBloom’s Taxonomy of the Cognitive Domain
Synthesis
Analysis
Application
Comprehension
Knowledge
10
Action Verbs Accompanying Bloom’s TaxonomyAction Verbs Accompanying Bloom’s Taxonomy
Knowledge Comprehension Application Analysis Synthesis Evaluation
Describe Compare Apply Analyze Arrange Argue
Find Describe Classify Appraise Assemble Assess
List Distinguish Complete Categorize Collect Choose
Locate Discuss Construct Compare Combine Conclude
Name Explain Demonstrate Contrast Comply Decide
Recall Express Dramatize Debate Compose Evaluate
Record Identify Employ Diagram Construct Interpret
Relate Outline Illustrate Differentiate Create Judge
Repeat Predict Interpret Distinguish Design Justify
State Recognize Operate Examine Devise Measure
Tell Restate Practice Experiment Formulate Rate
Underline Tell Schedule Inspect Imagine Recommend
Write Translate Sketch Inventory Manage Revise
Solve Investigate Organize Support
Use Question Plan Value
Separate Predict Verify
11
Bloom’s Taxonomy of the Affective DomainBloom’s Taxonomy of the Affective DomainOrganization
Valuing
Responding
Receiving
12
Action Verbs Accompanying Bloom’s Taxonomyhttp://www.abet.org/_TrainingCD/data/references/Assessment%20Links/Goals_revised_Blooms.pdf
Action Verbs Accompanying Bloom’s Taxonomyhttp://www.abet.org/_TrainingCD/data/references/Assessment%20Links/Goals_revised_Blooms.pdf
Receiving Responding Valuing Organization Characterization
Asks Answers Completes Adheres Acts
Chooses Assists Describes Alters Discriminates
Describes Complies Differentiates Arranges Displays
Follows Conforms Explains Defends Influences
Gives Discusses Follows Explains Listens
Identifies Helps Initiates Generalizes Modifies
Names Performs Joins Identifies Performs
Selects Presents Justifies Integrates Practices
Replies Selects Reads Modifies Proposes
Uses Tells Reports Organizes Qualifies
Selects Prepares Questions
Shares Relates Serves
Studies Synthesizes Solves
Works Uses
Verifies
13
Student Learning Outcomes: Oral Communication
Student Learning Outcomes: Oral Communication
Graduates will be able to communicate their knowledge effectively, both orally and in writing.
Students will:
Knowledge support arguments with relevant and adequate evidence
Skills answer questions directly and accurately
Dispositions convey enthusiasm for their topic
14
Student Learning Outcomes: Written Communication
Student Learning Outcomes: Written Communication
Graduates will be able to communicate their knowledge effectively, both orally and in writing.
Students will:
Knowledge respond to essay questions on exams with clear, well-organized presentations focused on the topic and relatively free of errors
Skills recognize well-written texts and advise peers on how to improve their written work
Dispositions develop a personal voice and sense of style in writing
15
Student Learning Outcomes: Research Methods (Psychology)
Student Learning Outcomes: Research Methods (Psychology)
Graduates will understand and apply basic research methods in psychology, including research design, data analysis, and interpretation
Students will:
Knowledge articulate strengths and limitations of various research designs
Skills formulate testable hypotheses, based on operational definitions of variables
Dispositions recognize that theoretical and socio-cultural contexts as well as personal biases may shape research questions, design, data collection, analysis, and interpretation
17
GC&SU Core OutcomesGC&SU Core Outcomes
• Communicate clearly in written English, demonstrating comprehension, analysis, and critical interrogation of a variety of texts. (A: Essential Skills; English)
• Use description, analysis, and synthesis of data, ideas, or information appropriate to the purpose. (A: Essential Skills: English)
• Express and interpret mathematical information, concepts, and thoughts in verbal, numeric, graphical, and symbolic form, and to shift among these different modes when solving problems. (A: Essential Skills; Math)
• Use appropriate technology in the location, evaluation, analysis, synthesis, and application of information in problem-solving situations (A: Essential Skills; hybrid of Math and English)
19
Things to Consider:Things to Consider:
What’s already in place? Exams, assignments, or projects? Common exams, assignments, or projects? Writing samples (common question) Capstone experience products
Look for logical assessment points in the curriculum At program entrance Program courses required for the major At program conclusion
The evidence you collect depends on the questions you want to answer
20
Kinds of Evidence, Part 1Kinds of Evidence, Part 1
Direct EvidenceMeasure student learning outcomesdirectly
Embedded questions/tasks Pre-post testing Evaluation of projects/work Standardized tests (e.g., MFT) Portfolios Presentations
Indirect EvidenceMeasure opinions or thoughtsabout students’ (or alumni) own knowledge, skills, attitudes,learning experiences, etc. Also includes external evaluations. Alumni surveys Exit surveys Employer surveys Focus Groups
21
Kinds of Evidence, Part 2Kinds of Evidence, Part 2
Quantitative Evidence Ratings of student
skills/performance Scores and pass rates on tests
(licensure/certification) Scores on locally-designed
tests (final exams, qualifiers, comprehensives) when accompanied by test “blueprints”
Classroom response systems (clickers)
Qualitative Evidence Observations of student
behavior w/ systematic notes Summaries of discussion
threads Student reflections Portfolios evaluation Written work, performances,
or presentations scored with a rubric
Analysis of comments on surveys
22
Student Learning Outcome: answer questions directly and accuratelyStudent Learning Outcome: answer questions directly and accurately
Course Point of Assessment
Assessment Method
Results
Introduction to Psychology Oral presentation on 40 STCP chapter
Presentation rubric (professor)
Research Methods Oral presentation of research project
Presentation rubric (professor and peers)
Psychology Seminar Classroom guest lecture on historical figure in psychology
Presentation rubric (professor)
23
Oral Presentation Rubric (http://www1.uprh.edu/cruzmigu/OralRubric.pdf)Oral Presentation Rubric (http://www1.uprh.edu/cruzmigu/OralRubric.pdf)
4 3 2 1
Subject Knowledge
Student demonstrates full knowledge by answering all class questions with explanations and elaboration
Student is at ease with expected answers to all questions, without elaboration
Student is uncomfortable with information and is able to answer only rudimentary questions
Student does not have grasp of information; student cannot answer questions about subject.
24
Student Learning Outcome: articulate strengths and limitations of various research designsStudent Learning Outcome: articulate strengths and limitations of various research designs
Course Point of Assessment
Assessment Method
Results
Introduction to Psychology 1st Test Multiple Choice or Matching Question
Developmental Psychology 1st Test
Final Exam
Application short answer
Essay question
Research Methods a) 1st Test
b) Research Proposal
Application question
Rationale for research method selection
27
At Last: Reporting your FindingsAt Last: Reporting your Findings
Analyze the data…identify the results
Integrate results from various assessment methods
What conclusions can you draw from that evidence?
What recommendations arise from that data?
What actions should/will you take based on the recommendations?
28
Student Learning Outcome: answer questions directly and accuratelyStudent Learning Outcome: answer questions directly and accurately
Course Point of Assessment
Assessment Method
Results
Introduction to Psychology Oral presentation on 40 STCP chapter
Presentation rubric (professor)
Spring 2010: n=55, avg. score was 1.9
Research Methods Oral presentation of research project
Presentation rubric (professor and peers)
Fall 2009: n=25, avg. score was 2.6 (avg. peer rating was 3.1)
Psychology Seminar Classroom guest lecture on historical figure in psychology
Presentation rubric (professor)
Spring 2010: n=25, avg. score was 2.9
In general, as students progress though the major, there is improvement in students’ ability to answer questions at the end of their oral presentation, but students may benefit from more practice in elective classes. Increase use of peer ratings to increase student critical reflection on this task.
31
ConclusionsConclusions
The “systematic collection, review, and use of information about educational programs undertaken for the purpose of improving learning” allows us to address three fundamental questions:
1. What have our students learned?2. How well have our students learned that information?3. How successful have we been at what we are trying to
accomplish?
32
Contact InformationContact Information
Caroline R. Noyes, Ph.DAssistant Director, Office of AssessmentGeorgia Institute of Technology(404) [email protected]