what “counts” as evidence of student learning in program assessment? sarah zappe research...
TRANSCRIPT
![Page 1: What “Counts” as Evidence of Student Learning in Program Assessment? Sarah Zappe Research Assistant Testing and Assessment Specialist Schreyer Institute](https://reader036.vdocuments.site/reader036/viewer/2022062620/551a8c0f5503466b3a8b4ceb/html5/thumbnails/1.jpg)
What “Counts” as Evidence of Student Learning in Program
Assessment?Sarah Zappe
Research AssistantTesting and Assessment Specialist
Schreyer Institute for Teaching Excellence
![Page 2: What “Counts” as Evidence of Student Learning in Program Assessment? Sarah Zappe Research Assistant Testing and Assessment Specialist Schreyer Institute](https://reader036.vdocuments.site/reader036/viewer/2022062620/551a8c0f5503466b3a8b4ceb/html5/thumbnails/2.jpg)
Workshop Goals
To provide information and guidance on the processes of:
Identifying sources of evidence of student learning
Mapping evidence to program outcomes
Developing reports for stakeholders
![Page 3: What “Counts” as Evidence of Student Learning in Program Assessment? Sarah Zappe Research Assistant Testing and Assessment Specialist Schreyer Institute](https://reader036.vdocuments.site/reader036/viewer/2022062620/551a8c0f5503466b3a8b4ceb/html5/thumbnails/3.jpg)
Definition of Assessment“Assessment is an ongoing process aimed at understanding and improving student learning. It involves making expectations explicit and public; setting appropriate criteria and high standards for learning quality; systematically gathering, analyzing, and interpreting evidence to determine how well performance matches those expectations and standards, and using the resulting information to document, explain, and improve performance.”
Angelo, T. A. (1995)
![Page 4: What “Counts” as Evidence of Student Learning in Program Assessment? Sarah Zappe Research Assistant Testing and Assessment Specialist Schreyer Institute](https://reader036.vdocuments.site/reader036/viewer/2022062620/551a8c0f5503466b3a8b4ceb/html5/thumbnails/4.jpg)
Enhance teaching and learning/
Inform stakeholders
Interpret Evidence
Gather Evidence
Assessment Loop
Goals and
Outcomes
Maki (2001)
![Page 5: What “Counts” as Evidence of Student Learning in Program Assessment? Sarah Zappe Research Assistant Testing and Assessment Specialist Schreyer Institute](https://reader036.vdocuments.site/reader036/viewer/2022062620/551a8c0f5503466b3a8b4ceb/html5/thumbnails/5.jpg)
Student Learning Outcomes
Measurable and specific goals for what we want our students to know, feel, or be able to do following the program Knowledge, skills, and attitudes
Drives all other stages of assessment
![Page 6: What “Counts” as Evidence of Student Learning in Program Assessment? Sarah Zappe Research Assistant Testing and Assessment Specialist Schreyer Institute](https://reader036.vdocuments.site/reader036/viewer/2022062620/551a8c0f5503466b3a8b4ceb/html5/thumbnails/6.jpg)
University Guidelines for the Internal Review of Academic
Programs
1. Background, purpose, and goals2. Specify evaluation areas3. Data collection plan4. Data collection and analysis5. Recommendations
![Page 7: What “Counts” as Evidence of Student Learning in Program Assessment? Sarah Zappe Research Assistant Testing and Assessment Specialist Schreyer Institute](https://reader036.vdocuments.site/reader036/viewer/2022062620/551a8c0f5503466b3a8b4ceb/html5/thumbnails/7.jpg)
Do we already have data that provides evidence of student
learning?
Probably, let’s see… Direct evidence of student learning
Measures of student performance that demonstrate actual learning
What did students learn and NOT learn? Indirect evidence of student learning
Measures of perception or demographic indicators that imply learning has occurred
![Page 8: What “Counts” as Evidence of Student Learning in Program Assessment? Sarah Zappe Research Assistant Testing and Assessment Specialist Schreyer Institute](https://reader036.vdocuments.site/reader036/viewer/2022062620/551a8c0f5503466b3a8b4ceb/html5/thumbnails/8.jpg)
Direct Measures of Student Learning
Capstone projects, senior theses, exhibits
Portfolios Standardized tests Concept inventories Employer/internship ratings of
students’ performance
Middle States Commission, (2003)
![Page 9: What “Counts” as Evidence of Student Learning in Program Assessment? Sarah Zappe Research Assistant Testing and Assessment Specialist Schreyer Institute](https://reader036.vdocuments.site/reader036/viewer/2022062620/551a8c0f5503466b3a8b4ceb/html5/thumbnails/9.jpg)
Limitations of Direct Evidence
No evidence of why students have learned or not learned
Does not indicate “value-added” Did students already have the
knowledge or skills before completing the program?
![Page 10: What “Counts” as Evidence of Student Learning in Program Assessment? Sarah Zappe Research Assistant Testing and Assessment Specialist Schreyer Institute](https://reader036.vdocuments.site/reader036/viewer/2022062620/551a8c0f5503466b3a8b4ceb/html5/thumbnails/10.jpg)
Indirect Measures of Student Learning
Focus groups/interviews Employer surveys Alumni surveys Registration/course enrollment
information Department or program review data Job placement indicators Graduate school placement rates Comparisons with other institutions
Middle States Commission, (2003)
![Page 11: What “Counts” as Evidence of Student Learning in Program Assessment? Sarah Zappe Research Assistant Testing and Assessment Specialist Schreyer Institute](https://reader036.vdocuments.site/reader036/viewer/2022062620/551a8c0f5503466b3a8b4ceb/html5/thumbnails/11.jpg)
Limitations of Indirect Evidence
Do not evaluate student learning per se
Should not be the only means of assessing outcomes
![Page 12: What “Counts” as Evidence of Student Learning in Program Assessment? Sarah Zappe Research Assistant Testing and Assessment Specialist Schreyer Institute](https://reader036.vdocuments.site/reader036/viewer/2022062620/551a8c0f5503466b3a8b4ceb/html5/thumbnails/12.jpg)
Does all evidence need to be quantitative?
No… In fact, good practice in assessment
suggests collecting multiple types of information
Both direct and indirect Both qualitative and quantitative
![Page 13: What “Counts” as Evidence of Student Learning in Program Assessment? Sarah Zappe Research Assistant Testing and Assessment Specialist Schreyer Institute](https://reader036.vdocuments.site/reader036/viewer/2022062620/551a8c0f5503466b3a8b4ceb/html5/thumbnails/13.jpg)
Quantitative Evidence Represented numerically Examples
Scores on tests Survey scales
Advantages Ease of collection Ease of analysis Ease making calculations and comparisons (across
time or between groups) Generalizability
Limitations Often doesn’t answer the question of “why”
![Page 14: What “Counts” as Evidence of Student Learning in Program Assessment? Sarah Zappe Research Assistant Testing and Assessment Specialist Schreyer Institute](https://reader036.vdocuments.site/reader036/viewer/2022062620/551a8c0f5503466b3a8b4ceb/html5/thumbnails/14.jpg)
Qualitative Evidence Data represented in narrative or prose format Examples
Interviews Focus groups Open-ended questions on surveys
Advantages Provides very “rich” information
Limitations More difficult to analyze and to make direct comparisons Not generalizable Methods of ensuring reliability are difficult and time-
consuming
![Page 15: What “Counts” as Evidence of Student Learning in Program Assessment? Sarah Zappe Research Assistant Testing and Assessment Specialist Schreyer Institute](https://reader036.vdocuments.site/reader036/viewer/2022062620/551a8c0f5503466b3a8b4ceb/html5/thumbnails/15.jpg)
Brainstorm Activity Brainstorm existing types of
evidence for your program Direct evidence Indirect evidence
What is missing but should be collected?
Discuss these with your table
![Page 16: What “Counts” as Evidence of Student Learning in Program Assessment? Sarah Zappe Research Assistant Testing and Assessment Specialist Schreyer Institute](https://reader036.vdocuments.site/reader036/viewer/2022062620/551a8c0f5503466b3a8b4ceb/html5/thumbnails/16.jpg)
Isn’t sampling somehow cheating?
No, but… Sampling should be representative of
population Population– students in your program
Sample should embody important characteristics of population
Stratified random sample Avoid convenience or accidental
sampling
![Page 17: What “Counts” as Evidence of Student Learning in Program Assessment? Sarah Zappe Research Assistant Testing and Assessment Specialist Schreyer Institute](https://reader036.vdocuments.site/reader036/viewer/2022062620/551a8c0f5503466b3a8b4ceb/html5/thumbnails/17.jpg)
Do Grades Count as Evidence?
Yes! But… Only if they are linked to learning goals
Score/grade alone does not express the content of what students have learned
Need to define what each score means Match course assessment to outcomes Syllabi Test blueprints
![Page 18: What “Counts” as Evidence of Student Learning in Program Assessment? Sarah Zappe Research Assistant Testing and Assessment Specialist Schreyer Institute](https://reader036.vdocuments.site/reader036/viewer/2022062620/551a8c0f5503466b3a8b4ceb/html5/thumbnails/18.jpg)
Do Grades Count (Cont.)
“If the grades of individual students can be traced directly to their respective competencies in a course, the learning achievements of those students are being assessed in a meaningful fashion.”
Middle States, 2003
![Page 19: What “Counts” as Evidence of Student Learning in Program Assessment? Sarah Zappe Research Assistant Testing and Assessment Specialist Schreyer Institute](https://reader036.vdocuments.site/reader036/viewer/2022062620/551a8c0f5503466b3a8b4ceb/html5/thumbnails/19.jpg)
Embedded Course Assessment
Questions or problems relevant to outcomes are embedded within course assessment
Examples Specific course projects Capstone projects Test and blueprints matched to outcomes
Advantages: No extra time for student or faculty Student motivation is greater Provides both formative and summative data
![Page 20: What “Counts” as Evidence of Student Learning in Program Assessment? Sarah Zappe Research Assistant Testing and Assessment Specialist Schreyer Institute](https://reader036.vdocuments.site/reader036/viewer/2022062620/551a8c0f5503466b3a8b4ceb/html5/thumbnails/20.jpg)
Linking Outcomes
Course Assessment
Program
Assessment
Institutional Assessment
Bakersfield College (2006)
![Page 21: What “Counts” as Evidence of Student Learning in Program Assessment? Sarah Zappe Research Assistant Testing and Assessment Specialist Schreyer Institute](https://reader036.vdocuments.site/reader036/viewer/2022062620/551a8c0f5503466b3a8b4ceb/html5/thumbnails/21.jpg)
Activity: Aligning Courses to Program Outcomes
Using the matrix provided, identify sources of evidence and match to your outcomes. Evidence embedded in courses Other evidence
![Page 22: What “Counts” as Evidence of Student Learning in Program Assessment? Sarah Zappe Research Assistant Testing and Assessment Specialist Schreyer Institute](https://reader036.vdocuments.site/reader036/viewer/2022062620/551a8c0f5503466b3a8b4ceb/html5/thumbnails/22.jpg)
How should we decide what to present in our report?
Consider the stakeholders External stakeholders Internal audience
Consider a short and a long form Get feedback Sample assessment report
![Page 23: What “Counts” as Evidence of Student Learning in Program Assessment? Sarah Zappe Research Assistant Testing and Assessment Specialist Schreyer Institute](https://reader036.vdocuments.site/reader036/viewer/2022062620/551a8c0f5503466b3a8b4ceb/html5/thumbnails/23.jpg)
Where can we get help if we need it?
Schreyer Institute for Teaching Excellence http://www.schreyerinstitute.psu.edu
Office of Institutional Planning and Assessment http://www.psu.edu/president/pia/ind
ex.htm
![Page 24: What “Counts” as Evidence of Student Learning in Program Assessment? Sarah Zappe Research Assistant Testing and Assessment Specialist Schreyer Institute](https://reader036.vdocuments.site/reader036/viewer/2022062620/551a8c0f5503466b3a8b4ceb/html5/thumbnails/24.jpg)
7 Common Misperceptions about Assessment
1. We’re doing just fine without it.2. We’re already doing it.3. We’re far too busy to do it.4. The most important things can’t be
measured.5. We’d need more staff and money.6. They’ll use the results against us.7. No one will care about or use what we
find.Angel (2005)
![Page 25: What “Counts” as Evidence of Student Learning in Program Assessment? Sarah Zappe Research Assistant Testing and Assessment Specialist Schreyer Institute](https://reader036.vdocuments.site/reader036/viewer/2022062620/551a8c0f5503466b3a8b4ceb/html5/thumbnails/25.jpg)
Mini-Evaluation of Session Please complete the mini-
evaluation form provided so that we can work on improving OUR efforts!
Thank you for your time!