Download - Using IDEA for Faculty Evaluation
![Page 1: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/1.jpg)
Using IDEA for Faculty Evaluation
Shelley A. Chapman, PhDTexas A & M University February 2013
![Page 2: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/2.jpg)
Plan for this Session “Teaching Effectiveness” What it is Uniqueness of IDEA Conditions for the Good Use of IDEA 3-Phase Process for Faculty
Evaluation Using Reports to Improve Teaching
![Page 3: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/3.jpg)
Most SurveysHow well do the instructor’s methods resemble those of a “model” teacher?
How well do students rate their progress on the types of learning the instructor targeted?
Teaching Effectiveness
![Page 4: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/4.jpg)
What makes IDEA unique?1. Focus on Student
Learning 2. Focus on Instructor’s
Purpose3. Adjustments for
Extraneous Influences4. Validity and Reliability5. Comparison Data6. Flexibility
![Page 5: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/5.jpg)
Conditions for Good Use
The instrument o Focuses on learning o Provides suggested
action steps
![Page 6: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/6.jpg)
Conditions for Good Use
The Facultyo Trust the processo Value student
feedbacko Are motivated to
make improvements
![Page 7: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/7.jpg)
Conditions for Good Use
Campus Cultureo Teaching excellence - high
priorityo Resources to improve -
providedo Student ratings -
appropriate weight
![Page 8: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/8.jpg)
Conditions for Good Use
The Evaluation Processo 30-50% of evaluation of teachingo 6-8 classes, more if small (<10)oNot over-interpreted
(3-5 performance categories)
![Page 9: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/9.jpg)
Underlying Philosophy of IDEA
Teaching effectiveness is determined primarily by students’ progress on the types of learning the instructor targets.
![Page 10: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/10.jpg)
Faculty Information Form
![Page 11: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/11.jpg)
Diagnostic Report Overview
1. How did students rate their learning experience?
2. What contextual factors impacted those ratings?
3. How do my scores compare to: IDEA, discipline, and institution?
4. What might I do to facilitate better learning for my students next time?
![Page 12: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/12.jpg)
Wisdom
Knowledge
InformationData
What the Report Can Provide
Calculation of Scores
Context: Variables and Comparisons
Suggested Action Steps
![Page 13: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/13.jpg)
Using IDEA
As Part of a Faculty Evaluation Process
![Page 14: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/14.jpg)
Multiple forms of Assessment
Student Ratings
External PerspectiveArtifacts
BalancedPlan for SummativeEvaluation
![Page 15: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/15.jpg)
Evidence of Good Teaching
Artifacts
• Syllabi• Graphic Organizers• Assignments and project
descriptions• Rubrics• Written Teaching
Philosophy/Reflections• Samples of Student Work• CATs and results
![Page 16: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/16.jpg)
Evidence of Good Teaching
• Classroom Observation• Classroom Visitation• Invited Presentations• Alumni Surveys• Focus Groups of
Graduating Students
External Perspectiv
e
![Page 17: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/17.jpg)
Classroom ObservationsTime What Happened What Was Said
![Page 18: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/18.jpg)
Classroom ObservationsTime What Happened What Was Said
8:05
8:10
8:15
Instructor shut door
Students are shuffling papers, opening books.
Student comes in late
Several students raise hands
Female in first row is called on
Instructor (I): OK, Class. Let’s begin. Make sure you turned in your homework as you came in.
Today we will begin our discussion on the brain. Turn in your textbooks to chapter 5.
Is your brain more like a computer or a jungle? Who would like to respond first?
Student (S) My brain is a jungle! I am so unorganized! (class laughs)…
![Page 19: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/19.jpg)
Flow of Communication MapInstructor
M
F
F
F
M M M M M
M M
M
F F F F
F
F
F
F F
![Page 20: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/20.jpg)
Evidence of Good Teaching
• Administer Appropriately• Collect 6-8 Reports (more if class size is <10)• 30-50% of Overall Evaluation
Student
Ratings
![Page 21: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/21.jpg)
Evidence of Good Teaching
• Student Comments-formative • Be mindful of standard error of
measurement (±.3)• Use 3-5 Performance Categories
Student
Ratings
![Page 22: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/22.jpg)
Three-Phase Process for Faculty Evaluation
Set Expectations
Collect Data
Use Data
![Page 23: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/23.jpg)
Three-Phase Process for Faculty Evaluation
Set Expectations
Collect Data
Use Data
I. Set ExpectationsWhat does this entail regarding IDEA?
![Page 24: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/24.jpg)
Criterion-Referenced Standards
Criteriono Use averages on 5-
point scaleo Recognize that
some objectives are more difficult to achieve
o “Authenticate” objectives
Page 1
![Page 25: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/25.jpg)
Norm-Referenced Standards
Use Converted Averages
o IDEAo Disciplineo Institutio
n
![Page 26: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/26.jpg)
Comparison Information: Converted Averages
![Page 27: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/27.jpg)
T-Score Distribution
40%
Similar
Lower20%10%
Much Lower
10%Much Higher
20%Higher
Gray Band
![Page 28: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/28.jpg)
Comparison Scores
![Page 29: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/29.jpg)
Create Categories of Performance
Below Acceptable Standards
Marginal, Needs Improvement
MeetsExpectations
ExceedsExpectations
Outstanding
Does Not Meet Expectations Meets Expectations Exceeds Expectations
![Page 30: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/30.jpg)
Performance Categories: EXAMPLECriterionAverage Rating
Effectiveness Category
NormativeT-Score
Below 3.0 Below acceptable standards Below 38
3.0-3.4 Marginal, improvement needed
38-44
3.5-3.9 Meets expectations 45-54
4.0-4.4 Exceeds expectations 55-62
4.5 or higher Outstanding 63 or higher
![Page 31: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/31.jpg)
Three-Phase Process for Faculty Evaluation
Set Expectations
Collect Data
Use Data
II. Collect DataWhat do you look for regarding IDEA?
![Page 32: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/32.jpg)
Improving Online Response Rates
• Create value for student feedback
• Monitor and Communicate through multiple modalities:• Twitter• Facebook• Other
• Prepare Students• Talk about it• Syllabus
![Page 33: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/33.jpg)
Example: Course Syllabus
Objective 3: Learning to apply course material (to improve thinking, problem solving, and decisions)
Students will be able to apply the methods, processes, and principles of earth science to understanding natural phenomenaStudents will think more critically about the earth and environment
Objective 8: Developing skill in expressing myself orally or in writing
Students will be able to present scientific results in written and oral forms
IDEA Center Learning Objective
Course Learning Outcomes
![Page 34: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/34.jpg)
For Personnel Decisions
Pages 1 and 2 What were students’ perceptions of the
course and their learning?
![Page 35: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/35.jpg)
Things toConsider…
Were the appropriate objectives selected?
• How many?• Do they match the course?
• How might you “authenticate” the objectives selected?
![Page 36: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/36.jpg)
FIF: Selecting Objectives• 3-5 as “Essential” or
“Important”o Is it a significant part of the
course?o Do you do something
specific to help students accomplish the objective?
o Does the student’s progress on the objective influence his or her grade?Be true to your course.
![Page 37: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/37.jpg)
Things to Consider…
What were the students’ perceptions of their course and their learning?
![Page 38: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/38.jpg)
How Did Students Rate Their Learning?
![Page 39: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/39.jpg)
Your Average (5-point Scale)
Raw Adj.
A. Progress on Relevant Objectives1
Four objectives were selected as relevant (Important or Essential—see page 2)
4.1 4.3
1If you are comparing Progress on Relevant Objectives from one instructor to another, use the converted average.
How Did Students Rate Their Learning?
![Page 40: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/40.jpg)
ProgressOnRelevantObjectives
4
4.3 + 4.34.14.23.6 5
![Page 41: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/41.jpg)
Summary Evaluation: Five-Point Scale
Report Page 1
Your Average Score
(5-point scale)
Raw Adj.
A. Progress on Relevant ObjectivesFour objectives were selected as relevant (Important or Essential—see page 2)
4.1 4.3
Overall Ratings B. Excellent Teacher 4.7 4.9 C. Excellent Course 4.1 4.4
D. Average of B & C 4.4 4.7
Summary Evaluation(Average of A & D) 4.3 4.5
50%
25%25%
![Page 42: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/42.jpg)
Understanding Adjusted Scores
![Page 43: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/43.jpg)
Work Habits
(Item 43)
Student Motivation (Item 39)
HighHigh Avg.
Avg.LowAvg.
Low
High 4.48 4.38 4.28 4.13 4.04
High Avg. 4.38 4.29 4.14 3.96 3.76
Average 4.28 4.14 4.01 3.83 3.64
Low Avg. 4.15 4.05 3.88 3.70 3.51
Low 4.11 3.96 3.78 3.58 3.38
Impact of Extraneous Factors
Gaining Factual Knowledge – Average Progress Ratings
Technical Report 12, page 40
![Page 44: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/44.jpg)
Work Habits
(Item 43)
Student Motivation (Item 39)
HighHigh Avg.
Avg.LowAvg.
Low
High 4.48 4.38
High Avg. 4.38 4.29
Average 4.01
Low Avg. 3.70 3.51
Low 3.58 3.38
Impact of Extraneous Factors
Gaining Factual Knowledge – Average Progress Ratings
Technical Report 12, page 40
![Page 45: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/45.jpg)
Raw or Adjusted Scores
Purpose Raw or Adjusted?How much did students learn? Raw
What were the instructor’s contributions to learning?
Adjusted
How do faculty compare?
Adjusted
![Page 46: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/46.jpg)
Doraw
scores meet or exceed
Expectations?*
Are adjusted
scores lower or higher than
rawscores?
Use adjusted
scoresUse
raw scores
Lower Yes
Higher
When to Use Adjusted Scores
*Expectations defined by your unit.
No
![Page 47: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/47.jpg)
Three-Phase Process for Faculty Evaluation
Set Expectations
Collect Data
Use Data
III. Use DataWhich data will you use and how?
![Page 48: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/48.jpg)
.
IDEA Faculty Worksheet
Keep track of reports Look for longitudinal trends Use for promotion and tenure
Created by Pam Milloy, Grand View University
Available from The IDEA Center Website
![Page 49: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/49.jpg)
Using the DataSummative (pp.1-
2) Criterion or Norm-
referenced Adjusted or raw Categories of
Performance 30-50% of Teaching
Evaluation 6-8 Classes (more if
small)
Formative (p.3) Identify areas to
improve Access applicable
resources from IDEA website
Read and have conversations
Implement new ideas
![Page 50: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/50.jpg)
Reflective Practice
Collect Feedback
Interpret Results
Read & Learn
Reflect & Discuss
Improve
IDEA resources that are keyed to reports
Talk with colleagues
Try new ideas Online, Paper
What the reports sayand what they mean
![Page 51: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/51.jpg)
Research
Relationship of Learning Objectives to Teaching
Methods
![Page 52: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/52.jpg)
Information Improvement
![Page 53: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/53.jpg)
Information Improvement
![Page 54: Using IDEA for Faculty Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062520/568161d6550346895dd1db26/html5/thumbnails/54.jpg)
Questions?