assessment in college teaching ursula waln, director of student learning assessment central new...
TRANSCRIPT
Assessment in College Teaching
Ursula Waln, Director of Student Learning Assessment
Central New Mexico Community College
Overview
Grading, Assessment & the Purpose of these Slides
Grading and Assessment Go Hand-in-Hand
Grading• Summarizing student
performance symbolically• Percentages correct• Points earned• Letter grades
• A holistic evaluation of work
• Used to communicate student success relative to criteria and/or other students
Assessment• Analyzing student learning
• What students learned well• What they didn’t learn so well• The factors that influenced the
learning
• A multifaceted evaluation of student progress
• Used to identify ways to improve learning
• Does not have to involve grades
The Purpose of these Slides
• These slides aim provide an overview of three techniques instructors can use to get the most out of their course-level assessment efforts:
1. Classroom Assessment Techniques (CATs)• For formative assessment
2. Item Analysis • For objective evaluation
3. Descriptive Rubrics• For subjective evaluation
Click the links in the list to skip
ahead to a section.
Classroom Assessment Techniques
For Formative Assessment
A Comprehensive, Authoritative Resource
• Angelo, T. A., & Cross, K. P. (1993). Classroom assessment techniques: A handbook for college teachers (2nd ed.). San Francisco, CA: Jossey-Bass.• Describes 50 commonly used classroom assessment techniques (CATs) • Emphasizes the importance of having clear learning goals• Promotes planned, intentional use to gauge student progress• Encourages discussing results with the students
• To promote learning• To teach students to monitor their own learning progress
• Encourages the use of insights gained to redirect instruction
• Examples of CATs are briefly described in the following 10 slides. Click to go to:
Item Analysis Descriptive
Rubrics
Prior Knowledge, Recall & Understanding
• Misconception/Preconception Check • Having students write answers to questions designed to uncover
prior knowledge or beliefs that may impede learning
• Empty Outlines • Providing students with an empty or partially completed outline and
having them fill it in
• Memory Matrix • Giving students a table with column and row headings and having
them fill in the intersecting cells with relevant details, match the categories, etc.
Click to go to: Item Analysis
Descriptive Rubrics
Skill in Analysis & Critical Thinking
• Categorizing Grid• Giving students a table with row headings and having students match
by category and write in corresponding items from a separate list
• Content, Form, and Function Outlines• Having students outline the what, how, and why related to a concept
• Analytic Memos• Having students write a one- or two-page analysis of a problem or
issue as if they were writing to an employer, client, stakeholder, politician, etc.
Click to go to: Item Analysis
Descriptive Rubrics
Skill in Synthesis & Creative Thinking
• Approximate Analogies• Having students complete the analogy A is to B as ___ is to ___, with
A and B provided
• Concept Maps• Having students illustrate relationships between concepts by
creating a visual layout bubbles and arrows connecting words and/or phrases
• Annotated Portfolios• Having students create portfolios presenting a limited number of
works related to the specific course, a narrative, and maybe supporting documentation
Click to go to: Item Analysis
Descriptive Rubrics
Skill in Problem Solving
• Problem Recognition Tasks• Presenting students with a few examples of common problem types
and then asking them to identify the particular type of problem each represents
• What’s the Principle?• Presenting students with a few examples of common problem types
and then asking them to state the principle that best applies to each problem
• Documented Problem Solutions• Having students not only show their work, but also explain next to it
in writing how they worked the problem out (“show and tell”)Click to go to: Item Analysis
Descriptive Rubrics
Skill in Application & Performance
• Directed Paraphrasing• Having students paraphrase part of a lesson for a specific audience
and purpose
• Application Cards• Handing out an index card (or slip of scratch paper) and having
students write down at least one ‘real-world’ application for what they have learned
• Paper or Project Prospectus• Having students create a brief, structured plan for a paper or
project, anticipating and identifying the elements to be developed
Click to go to: Item Analysis
Descriptive Rubrics
Awareness of Attitudes & Values
• Profiles of Admirable Individuals• Having students write a brief, focused profile of an individual – in a
field related to the course – whose values, skills, or actions they admire
• Everyday Ethical Dilemmas• Presenting students with a case study that poses an ethical dilemma –
related to the course – and having them write anonymous responses
• Course-Related Self-Confidence Surveys• Having students write responses to a few questions aimed at
measuring their self-confidence in relation to a specific skill or ability
Click to go to: Item Analysis
Descriptive Rubrics
Self-Awareness as Learners
• Focused Autobiographical Sketches• Having students write one to two pages about a single, successful
learning experience in their past relevant to the learning in the course
• Interest/Knowledge/Skills Checklists• Giving students a checklist of the course topics and/or skills and having
them rate their level of interest, skill, and/or knowledge for each
• Goal Ranking and Matching• Having students write down a few goals they hope to achieve – in
relation to the course/ program – and rank those goals; then comparing student goals to instructor/program goals to help students better understand what the course/program is about
Click to go to: Item Analysis
Descriptive Rubrics
Course-Related Study Skills & Behaviors
• Productive Study-Time Logs• Having students record how much time they spend studying, when
they study, and/or how productively they study
• Punctuated Lectures• Stopping periodically during lectures and having students reflect
upon and then write briefly about their listening behavior just prior and how it helped or hindered their learning
• Process Analysis• Having student keep a record of the step they take in carrying out
an assignment and then reflect on how well their approach worked
Click to go to: Item Analysis
Descriptive Rubrics
Reactions to Instruction
• Teacher-Designed Feedback Forms• Having students respond anonymously to 3 to 7 questions in multiple-
choice, Likert scale, or short-answer formats to get course-specific feedback
• Group Instructional Feedback Technique• Having someone else (other than the instructor) poll students on what
works, what doesn’t, and what could be done to improve the course
• Classroom Assessment Quality Circles• Involving groups of students in conducting structured, ongoing
assessment of course materials, activities, and assignments and suggesting ways to improve student learning
Click to go to: Item Analysis
Descriptive Rubrics
Reactions to Class Activities & Materials
• Group-Work Evaluations• Having students answer questions to evaluate team dynamics and
learning experiences following cooperative learning activities
• Reading Rating Sheets• Having students rate their own reading behaviors and/or the
interest, relevance, etc., of a reading assignment
• Exam Evaluations• Having students provide feedback that reflects on the degree to
which an exam (and preparing for it) helped them to learn the material, how fair they think the exam is as an assessment of their learning, etc.
Click to go to: Item Analysis
Descriptive Rubrics
Item Analysis
For Objective Evaluation
Item Analysis
• Looks at frequency of correct responses (or behaviors) in connection with overall performance
• Used to examine item reliability • How consistently a question or performance criterion discriminates
between high and low performers
• Can be useful in improving validity of measures
• Can help instructors decide whether to eliminate certain items from the grade calculations
• Can reveal specific strengths and gaps in student learning
Click to go to: CATs
Descriptive Rubrics
How Item Analysis Works
• Groups students by the highest, mid-range, and lowest overall scores and examines item responses by group
• Assumes that higher-scoring students have a higher probability of getting any given item correct than do lower-scoring students• May have studied and/or practiced more and understood the material
better• May have greater test-taking savvy, less anxiety, etc.
• Produces a calculation for each item• Do it yourself to easily calculate a group difference or discrimination index• Use EAC Outcomes (a Blackboard plug-in made available to all CNM faculty
by the Nursing program) to generate a point-biserial correlation coefficient
• Gives the instructor a way to analyze performance on each item Click to go to: CATs
Descriptive Rubrics
One Way to Do Item Analysis by HandShared by Linda Suskie at the NMHEAR Conference, 2015
Item
Tally of those in Top 27%
who missed item*
Tally of those in
the Middle 46% who missed
item
Tally of those in
the Lower 27% who missed item*
Total % Who
Missed Item
Group Difference(# in Lower minus # in
Top)
1 ||||| ||||| ||||| ||||| ||
||||| ||||| ||||| ||
34% 17
2 ||||| || ||||| ||||| ||||| |||||
||||| ||||| ||||| ||||
40% 12
3 ||| | || 5% -14 ||||| |||| ||||| |||||
|17% 11
* You can use whatever portion you want for the top and lower groups, but they need to be equal. Using 27% is accepted convention (Truman Kelley, 1939).
An
unreliable
question
Good
discriminatio
n
Another Way to Do Item Analysis by HandRasch Item Discrimination Index (D)
N=31 because the upper and lower group each contain 31 students (115 students tested)
Item
# in Upper Group who answered correctly
(#UG)
Portion of UG who
answered correctly
(pUG)
# in Lower Group who answered correctly
(#LG)
Portion of LG who
answered correctly
(pLG)
Discrimination Index (D)D = pUGpLG
orD
1 31 1.00 (100%) 14 0.45 (45%) 0.552 24 0.77 (77%) 12 0.39 (39%) 0.38
3 28 0.90 (90%) 29 0.93 (93%) -0.034 31 1.00 (100%) 20 0.65 (65%) 0.35
A discrimination index of 0.4 or greater is generally regarded as high and anything less than 0.2 as low (R.L. Ebel, 1954).
An
unreliable
question
Good
discriminatio
n
The Same Thing but Less ComplicatedRasch Item Discrimination Index (D)
N in Upper and Lower Groups is 31 (27% of 115 students)
Item
# in Upper Group who answered correctly
(#UG)
# in Lower Group who answered correctly
(#LG)
Discrimination Index (D)
D
1 31 14 0.552 24 12 0.38
3 28 29 -0.034 31 20 0.35
It isn’t necessary to calculate the portions of correct responses in each group if you use the formula shown here.
This is really easy to do!
= 0.55 = 0.38 = -0.03 = 0.35
N=.27
Example of an EAC Outcomes Report
A point-biserial correlation is the Pearson correlation between responses to a particular item and scores on the total test (with or without that item).
Correlation coefficients range from -1 to 1.
This is available to CNM faculty through Blackboard course tools.
An
unreliable
question
Good
discriminatio
n
Identifying Key Questions
• A key (a.k.a. signature) question is one that provides information about student learning in relation to a specific instructional objective (or student learning outcome statement).
• The item analysis methods shown in the preceding slides can help you identify and improve the reliability of key questions.• A low level of discrimination may indicate a need to tweak the
wording.• Improving discrimination value also improves question validity.• The more valid an assessment measure, the more useful it is in
gauging student learning. Click to go to: CATs
Descriptive Rubrics
Detailed Multiple-Choice Item Analysis
• The detailed item analysis method shown on the next slide is for use with key multiple-choice items.
• This type of analysis can provide clues to the nature of students’ misunderstanding, provided:• The item is a valid measure of the instructional objective• Incorrect options (distractors) are written to be diagnostic (i.e., to
reveal misconceptions or breakdowns in understanding)
Click to go to: CATs
Descriptive Rubrics
Example of a Detailed Item Analysis
Item 2 of 4. The correct option is E. (115 students tested)
Item Response Pattern A B C D E Row
TotalUpper 27%
|| ||||| ||||| ||||| ||||| ||||| |||| 31
6.5% 16% 77.5%
Middle 46%
||| ||||| ||||| |||| || | ||||| ||||| ||||| ||||| ||||| ||||| |||
53
6% 26% 4% 2% 62%
Lower 27%
||||| ||||| || ||||| || ||||| ||||| || 31 16% 23% 16% 6% 39%
Grand Total
10 26 7 3 69 1158.5% 23% 6% 2.5% 60%
These results suggest that distractor B might provide the greatest clue about breakdown in students’ understanding, followed by distractor A, then C.
Descriptive Rubrics
For Subjective Evaluation
Rubric: Just Another Word for Scoring Guide
• A rubric is any scoring guide that lists specific criteria, such as a checklist or a rating scale.• Checklists are used for objective evaluation (did it or did not do it).• Rating scales are used for subjective evaluation (gradations of
quality).
• Descriptive rubrics are rating scales that contain descriptions of what constitutes each level of performance.• Maybe call them descriptive scoring guides if you don’t like the
word rubric.• Most people who talk about rubrics are referring to descriptive
rubrics, not checklists or rating scales.Click to go to:
CATs Item Analysis
The Purpose of Descriptive Rubrics
• Descriptive rubrics are used to lend objectivity to evaluations that are inherently subjective, e.g.:• Grading of artwork, papers, performances, projects, speeches, etc.• Assessing overall student progress toward specific learning outcomes
(course and/or program level)• Monitoring developmental levels of individuals as they progress
through a program (‘developmental rubrics’).• Conducting employee performance evaluations.• Assessing group progress toward a goal.
• When used by multiple evaluators, descriptive rubrics can minimize differences in rater thresholds (especially if normed). Click to go to:
CATs Item Analysis
Why Use Descriptive Rubrics in Class?
• In giving assignments, descriptive rubrics can help clarify the instructor’s expectations and grading criteria for students.• Students can ask more informed questions about the assignment.• A clear sense of what is expected can inspire students to achieve more.• The rubric helps explain to students why they received the grade they did.
• Descriptive rubrics help instructors remain fair and consistent in their scoring of student work (more so than rating scales).• Scoring is easier and faster when descriptions clearly distinguish levels.• The effects of scoring fatigue (e.g., grading more generously toward the
bottom of a stack due to disappointed expectations) are minimized.
Click to go to: CATs
Item Analysis
Why Use Descriptive Rubrics for Assessment?
• Clearly identifying benchmark levels of performance and describing what learning looks like at each level establishes a solid framework for interpreting multiple measures of performance.• Student performance on different types of assignments and at
different points in the learning process can be interpreted for analysis using a descriptive rubric as a central reference.
• With rubrics that describe what goal achievement looks like, instructors can more readily identify and assess the strength of connections between: • Course assignments and course goals• Course assignments and program goals
Click to go to: CATs
Item Analysis
Two Common Types of Descriptive Rubrics
Holistic• Each level of performance
has just one comprehensive description.
• Descriptions may be organized in columns or rows.
• Useful for quick and general assessment and feedback.
Analytic• Each level of performance
has descriptions for each of the performance criteria.
• Descriptions are organized in a matrix.
• Useful for detailed assessment and feedback.
Click to go to: CATs
Item Analysis
Example of a Holistic Rubric
Performance Levels
Descriptions
Proficient(10 points)
Ideas are expressed clearly and succinctly. Arguments are developed logically and with sensitivity to audience and context. Original and interesting concepts and/or unique perspectives are introduced.
Intermediate(6 points)
Ideas are clearly expressed but not fully developed or supported by logic and may lack originality, interest, and/or consideration of alternative points of view.
Emerging(3 points)
Expression of ideas is either undeveloped or significantly hindered by errors in logic, grammatical and/or mechanical errors, and/or over-reliance on jargon and/or idioms.
Example of an Analytic RubricDelve, Mintz, and Stewart’s (1990) Service Learning Model
Developmental Variables
Phase 1Exploration
Phase 2Clarification
Phase 3Realization
Phase 4Activation
Phase 5Internalization
InterventionMode
Group Group (beginning to identify with group)
Group that shares focus or independently
Group that shares focus or independently
Individual
Setting Minimal community interaction—Prefers on-campus activities
Trying many types of contact
Direct contact with community
Direct contact with community—intense focus on issue or cause
Frequent and committed involvement
CommitmentFrequency
One Time Several Activities or Sites
Consistently at One Site
Consistently at One Site or with one issue
Consistently at One Site or focused on particular issues
Duration Short Term Long Term Commitment to Group
Long Term Commitment to Activity, Site, or Issue
Lifelong Commitment to Issue (beginnings of Civic Responsibility)
Lifelong Commitment to Social Justice
BehaviorNeeds
Participate in Incentive Activities
Identify with Group Camaraderie
Commit to Activity, Site, or Issue
Advocate for Issue(s) Promote Values in self and others
Outcomes Feeling Good Belonging to a Group Understanding Activity, Site, or Issue
Changing Lifestyle Living One’s Values
BalanceChallenges
Becoming InvolvedConcern about new environments
Choosing from Multiple Opportunities/Group Process
Confronting Diversity and Breaking from Group
Questioning Authority/Adjusting to Peer Pressure
Living Consistently with Values
Supports Activities are Non-threatening and Structured
Group Setting, Identification and Activities are Structured
Reflective-Supervisors, Coordinators, Faculty, and Other Volunteers
Reflective-Partners, Clients, and Other Volunteers
Community—Have Achieved a Considerable Inner Support System
Another Example of an Analytic RubricAAC&U Ethical Reasoning Value Rubric
Capstone4
Milestones3 2
Benchmark1
Ethical Self-Awareness
Student discusses in detail/analyzes both core beliefs and the origins of the core beliefs and discussion has greater depth and clarity.
Student discusses in detail/analyzes both core beliefs and the origins of the core beliefs.
Student states both core beliefs and the origins of the core beliefs.
Student states either their core beliefs or articulates the origins of the core beliefs but not both.
Understanding Different Ethical Perspectives/Concepts
Student names the theory or theories, can present the gist of said theory or theories, and accurately explains the details of the theory or theories used.
Student can name the major theory or theories she/he uses, can present the gist of said theory or theories, and attempts to explain the details of the theory or theories used, but has some inaccuracies.
Student can name the major theory she/he uses, and is only able to present the gist of the named theory.
Student only names the major theory she/he uses.
Ethical Issue Recognition
Student can recognize ethical issues when presented in a complex, multilayered (gray) context AND can recognize cross-relationships among the issues.
Student can recognize ethical issues when issues are presented in a complex, multilayered (gray) context OR can grasp cross-relationships among the issues.
Student can recognize basic and obvious ethical issues and grasp (incompletely) the complexities or interrelationships among the issues.
Student can recognize basic and obvious ethical issues but fails to grasp complexity or interrelationships.
Application of Ethical Perspectives/Concepts
Student can independently apply ethical perspectives/concepts to an ethical question, accurately, and is able to consider full implications of the application.
Student can independently apply ethical perspectives/concepts to an ethical question, accurately, but does not consider the specific implications of the application.
Student can apply ethical perspectives/concepts to an ethical question, independently (to a new example) and the application is inaccurate.
Student can apply ethical perspectives/concepts to an ethical question with support (using examples, in a class, in a group, or a fixed-choice setting) but is unable to apply ethical perspectives/concepts independently (to a new example.).
Evaluation of Different Ethical Perspectives/Concepts
Student states a position and can state the objections to, assumptions and implications of and can reasonably defend against the objections to, assumptions and implications of different ethical perspectives/concepts, and the student's defense is adequate and effective.
Student states a position and can state the objections to, assumptions and implications of, and respond to the objections to, assumptions and implications of different ethical perspectives/concepts, but the student's response is inadequate.
Student states a position and can state the objections to, assumptions and implications of different ethical perspectives/concepts but does not respond to them (and ultimately objections, assumptions, and implications are compartmentalized by student and do not affect student's position.)
Student states a position but cannot state the objections to and assumptions and limitations of the different perspectives/concepts.
There are No Rules for Developing Rubrics
• Form typically follows function, so how one sets up a descriptive rubric is usually determined by how one plans to use it.
• Performance levels are usually column headings but can function just as wall as row headings.
• Performance levels can be arranged in ascending or descending order, and one can include as many levels as one wants.
• Descriptions can focus only on positive manifestations or include references to missing or negative characteristics.
• Some use grid lines while others do not.
Click to go to: CATs
Item Analysis
Rubric
Art
ifact
An
aly
ses
Interviews
Observatio
ns
Surv
eys
Written Tests
Descriptive rubrics can help pull together results from multiple measures for a more comprehensive picture of student learning.
Using the Model
• To pull together multiple measures for an overall assessment of student learning: • Take a random sample from each assignment and re-score those using the
rubric (instead of the grading criteria), or rate the students as a group based on overall performance on each assignment.
• Then, combine the results, weighting their relative importance based on:• At what stage in the learning process the results were obtained• How well you think students understood the assignment or testing process• How closely the learning measured relates to the instructional objectives • Factors that could have biased the results• Your own observations, knowledge of the situations, and professional
judgment
Click to go to: CATs
Item Analysis
“
”
Remember that when you do assessment, whether in the department, the general education program, or at the institutional level, you are not trying to achieve the perfect research design; you are trying to gather enough data to provide a reasonable basis for action. You are looking for something to work on.
Woolvard, B. E. (2010). Assessment clear and simple: Apractical guide for institutions, departments, and general education. San Fransisco, CA: Jossey-Bass.