cecilia chan, ir planning analyst paul freebairn, ed.d , director of assessment

Instructor Characteristics Correlated Positively with

Overall Instructor Ratings at a Small, Ethnically-Diverse University

Cecilia Chan, IR Planning AnalystPaul Freebairn, Ed.D, Director of AssessmentRonald M. Miller, Ph.D, Associate Professor

2009 AIR Forum, AtlantaJune 2, 2009

BYU-Hawaii

• 2400 students• Mission to serve Asia and the Pacific• 50% are international students• Students represent 70 different countries• At least two-thirds of our students speak two

or more languages • Average class size: 25• Student/faculty ratio: 17 to 1

Expanded Study

• Sample size (859 to 2118)• Year (2007 only; now 2007 to April 2009)• Trends (similar)• Effect size and data impact

Majority of the findings remain similar to those of 2007

Overview

• Introduction• General Trends• Best Predictors• Findings by Faculty Demographic

Characteristics• Findings by Course Characteristics• Automated Neural Network Analysis • Conclusions

Previous Studies onStudent evaluations of teaching (SET)

• Extensively researched since 1970’s; no general consensus.

• Students’ perceptions of their grades appear to be related to the evaluations they give both for the class and the instructor. Both a leniency and a reciprocity effect have been found (Clayson, 2009)

• Relatively valid against a variety of indicators hypothesized as potential biases, such as class size, grading leniency, workload and prior student interest (Wachtel, 1998)

• The same instructor gets higher ratings when giving higher grades or teaching smaller classes (Greenwald and Gillmore, 1997)

Previous Studies onStudent evaluations of teaching (SET)

• Negatively related to age and years of teaching experience (Feldman, 1983)

• Provides valid information on instructor effectiveness; little evidence of bias, but typically shows substantial correlations with student achievement as measured by examination performance (d’Apollonia & Abrami, 1997)

• No one has given a widely accepted definition of what “good” teaching is; universally agreeable criterion of teaching effectiveness has not been established (Kulik, 2001)

Objective

To investigate the relationships between various characteristics of faculty members, and their course and faculty ratings for possible course improvement

Overview of Instructors

• 2118 individual classes evaluated from Jan ‘07 to April ‘09

• Majority of classes taught by teachers who were • from colleges of Business, Computing & Government (26%)

or Language, Culture and Arts (26%)• non-tenured faculty status (41%)• employed for fewer than 5 years (44%) • male (68%)

Overview of Courses

• Courses were mainly• 200 level (26%)• had fewer than 20 students evaluate the course

(73%)• major courses (52%)

Primary Dependent Variables

• Overall Course *Rating• Overall Faculty *Rating

*7-point Scale (very poor, poor, fair, good, very good, excellent, exceptional)

Independent Variables• Faculty demographics

• College, department, faculty status, years teaching at BYUH, type and gender

• Course characteristics • Class level, number SETs, course department and class

type• Evaluation Form includes

• Two overall questions• 13 questions about the course• 18 questions about the instructor

Sample Questions

• Course• “Course objectives are clear.”• “Assigned workload is appropriate for credit

hours.”• Instructor

• “Gives clear examples and explanations.”• “Motivates me by his/her example to want to

learn about the subject.”

Findings

General Findings• The overall faculty rating is slightly higher than the overall

course rating• High Correlation between the overall course rating and overall

faculty rating (r=0.92, p<.00)

2007 2008 20093.00

4.00

5.00

6.00

7.00

5.75 5.84 5.825.55 5.65 5.62

Overall Ratings by Evaluation Year

Overall Faculty Overall Course

Regression Analyses

• Conducted two multiple regression analyses• Examined the relationship between the overall

ratings and specific questions for course and instructor

Regression AnalysisOverall Faculty

• Eight of the “instructor” questions were significant in predicting overall instructor rating (listed in descending order of significance)

• The regression model explains 87% of the variance (Adjusted R²= .87, p<.0000)

Most Significant PredictorsOverall Faculty

Question Beta

3.7 Clearly explains difficult concepts, ideas, or theories. 0.26

3.11 Motivates me by his/her example to want to learn about the subject. 0.21

3.4 Makes good use of class time. 0.11

3.6 Makes helpful evaluation of my work (e.g. papers, exams) 0.10

3.12 Has produced new knowledge, skills and awareness in me. 0.09

Non-Significant PredictorsOverall Faculty

• 3.2 Is enthusiastic about the subject.• 3.8 Responds respectfully to student questions

and viewpoints.• 3.10 Is available to students during regular and

reasonable office hours.• 3.14 Seldom misses class.• 6c Help students prepare to live effectively in

society.

Regression AnalysisOverall Course

• Nine of the “course” questions were significant in predicting the overall course rating (listed in descending order of significance):

• The regression model explains 85% of the variance (Adjusted R²= .85, p<.0000)

Most Significant PredictorsOverall CourseQuestion Beta

2.13 Course as a whole has produced new knowledge, skills and awareness in me.

0.25

2.4 Course content is relevant and useful. 0.20

2.2 Course is well organized. 0.16

2.1 Course objectives are clear. 0.14

2.10 Exams are good measures of my knowledge, understanding and ability to perform.

0.12

Non-Significant PredictorsOverall Course

• 2.3 Student responsibilities are clearly defined.

• 2.5 Assigned workload is appropriate for credit hours.

• 2.11 Grading procedure is fair and impartial• 2.12 Assignments are appropriately

distributed throughout the semester.

Findings by Demographics

Analyses of Variance (ANOVA)• Analyses of Variance compares specific faculty and

course characteristics• Significant differences were found among faculty:

• gender, status, position, years of teaching at BYUH, department, and college

• Significant differences were found among course:• level, type, size and department

Findings by Demographics:Faculty

Overall Ratings by Faculty Gender

Male Female 3

4

5

6

7

5.895.615.67

5.45


Mean

Overall Ratings by Faculty Status

NCS SI CFS3

4

5

6

7

5.88 5.74 5.715.66 5.56 5.54


Mean

Overall Ratings by Years Teaching at BYUH

<6 6 to 10 11 to 15 16 to 20 21 to 303

4

5

6

7

5.87 5.775.57

5.77 5.765.66 5.58 5.44 5.51 5.59


Mean

Overall Ratings by Faculty Type

SERVICE ASSOC ASST STAF-ADM LECTURER SI VISIT PROF3

4

5

6

7

6.185.94 5.84 5.77 5.70 5.68 5.68 5.51

6.025.72 5.64 5.56 5.48 5.52 5.48 5.38


Mean

Overall Ratings by Faculty College

LCA BCG HD MS3

4

5

6

7

5.88 5.79 5.78 5.685.69 5.58 5.645.46


Mean

Overall Ratings by Faculty Department

(Highest)

REL POSC FAR ICS PSYC3

4

5

6

7

6.11 5.99 5.99 5.99 5.905.985.73 5.84 5.79 5.72


Mean

Overall Ratings by Faculty Department

(Lowest)

ACCTF ELT BIOL BSC MATH3

4

5

6

7

5.67 5.65 5.65 5.56 5.445.51 5.45 5.36 5.26 5.20


Mean

Findings by Demographics: Course

Overall Ratingsby Course Department

(Highest)• Overall Faculty

• Courses with a smaller number of evaluations or that are more discussion-based tend to receive higher overall faculty ratings

• Language classes also tend to be rated higher• Overall Course

• Beginner level courses tend to have lower overall course ratings

• Language classes do not have high overall course ratings

Overall Faculty by Course Department

(Lowest)

IBM MATH TONG CHEM GEOL MAOR ECON SOC3

4

5

6

7

5.55 5.51 5.48 5.41 5.37 5.355.07

4.60

Overall Faculty

Mean

Overall Ratingsby Course Department

• Course type (like major vs GE) and number of student evaluating the class were found to have no confounding effect

MAJ PRESKIL GE3

4

5

6

7

5.795.64

5.17

5.585.33

4.89


Mean

Overall Ratingsby Course Type (Math Only)

Overall Ratingsby No. of Evaluation

(Fine Arts Only)

5 to 9 10 to 14 15 to 19 20 to 30 31 and above3

4

5

6

7

5.83 5.866.05

5.53

5.955.94 6.026.36

5.73

6.12


Mean

Overall Ratings by Class Level

0 1 2 3 43

4

5

6

7

5.63 5.74 5.79 5.85 5.95

5.40 5.55 5.58 5.66 5.73


Mean

Overall Ratings by Class Type

REL MAJ PRESKIL ELECT MIN GE EIL3

4

5

6

7

6.105.84 5.89 5.88 5.81

5.65 5.635.97

5.65 5.62 5.58 5.50 5.46 5.41


Mean

Overall Ratingsby Number of Evaluation

5 to 9 10 to 14 15 to 19 20 to 30 31 and above3

4

5

6

7

5.95 5.82 5.73 5.71 5.755.74 5.61 5.55 5.52 5.56


Mean

Predicting Overall Ratings

• A model with predictive validity of 94.3% was generated

• Below are the best predictors in this model:• 3.11 Motivates me by his/her example to want to learn

about the subject.• 3.12 Has produced new knowledge, skills and

awareness in me.• 3.7 Clearly explains difficult concepts, ideas, or

theories.• 3.5 Gives clear examples and explanations.

Predicting Overall Ratings

• Below are the least predictive variables in this model• 2.10 Exams are good measures of my knowledge,

understanding and ability to perform.• 2.12 Assignments are appropriately distributed throughout

the semester. • 2.5 Assigned workload is appropriate for credit hours.

• 2.9 Exams are clearly worded.• 3.10 Is available to students during regular and reasonable

office hours.

Conclusions

• Significant predictors• Questions emphasize learning-centered activities

• Non-significant predictors• Questions are more faculty-interaction and

housekeeping questions• Findings suggest that level of course difficulty has

no effect on the overall ratings in the regressions

Conclusions

• Male instructors tend to have higher overall ratings than female instructors

• Instructors with non-continuing status are more likely to have higher ratings

• There is a non-linear relationship between the overall ratings and years of teaching at BYUH

Conclusions

• Faculty Type trends are consistent with faculty status and years of teaching at BYU-Hawaii trends

• College of Languages, Cultures and Arts instructors have the highest overall rating than the other three colleges

• Instructors of a more discussion-based department tend to have higher overall ratings

Conclusions

• Instructors of a more science-based department tend to have lower overall ratings

• Courses with a smaller number of evaluation or that are more discussion-based tend to receive higher overall faculty ratings

• Language learning classes also tend to be rated higher.

Contact Information

Paul H. [email protected]

Cecilia Yiu [email protected]

Paper Available online

mailto:[email protected]

mailto:[email protected]

References• Ambady, N., & Rosenthal, R. (1993). Half a minute: Predicting teacher

evaluations from thin slices of nonverbal behavior and physical attractiveness. Journal of Personality and Social Psychology, 64(3), 431-441.

• Arreola, R.A. (2000). Developing a comprehensive faculty evaluation system (2nd ed.). Bolton, MA: Anker Publishing Company, Inc.

• Cashin, W.E. (1988). Student ratings of teaching: A summary of the research (IDEA Paper No. 20). Manhattan: Kansas State University, Center for Faculty Evaluation and Development.

• Cashin, W. E., & Downey, R. G. (1992). Using global student rating items for summative evaluation. Journal of Educational Psychology, 84(4), 563-572.

• Cashin, W.E. (1995). Student ratings of teaching: The research revisited (IDEA Paper No. 32). Manhattan: Kansas State University, Center for Faculty Evaluation and Development.

References• Clayson, D.E. (2009). Student evaluations of teaching: Are they related to

what students learn? A meta- analysis and review of the literature. Journal of Marketing Education, 31(1), 16-30.

• Costin, F., Greenough, W. T., & Menges, R. J. (1971). Student ratings of college teaching: reliability, validity, and usefulness. Review of Educational Research, 41(5), 511-535.

• Crader, K.W., & Butler, J.K. Jr. (1996). Validity of students’ teaching evaluation scores: The Wimberly-Faulkner-Moxley Questionnaire. Educational and Psychological Measurement, 56(2), 304-314.

• Cranton, P. A., & Smith, R. A. (1986). A New look at the effect of course characteristics on student ratings of instruction. American Educational Research Journal, 23(1), 117-128.

• d’Apollonia, S., & Abrami, P.C. (1997). Navigating student ratings of instruction. American Psychologist, 52, 1198-1208.

References• Darling-Hammond, L., Wise, A. E., & Pease, S. R. (1983). Teacher

evaluation in the organizational context: A review of the literature. Review of Educational Research, 53(3), 285-328.

• Feldman, K. A. (1978). Course characteristics and college students' ratings of their teachers -What we know and what we don't. Research in Higher Education, 9, 199-242.

• Feldman, K.A. (1983). The seniority and instructional experience of college teachers as related to the evaluations they receive from their students. Research in Higher Education, 18, 3-124.

• Fernandez, J., Mateo, M.A., & Muniz, J. (1998). Is there a relationship between class size and student ratings of teaching quality? Educational and Psychological Measurement, 58(4), 596-604.

• Gilbert, L. A. (1985). Dimensions of same-gender student-faculty role-model relationships. Sex Roles, 12(1-2), 111-123.

References• Greenwald, A. G., & Gillrnore, G. M. (1997). No pain, no gain? The

importance of measuring course workload in student ratings of instruction. Journal of Educational Psychology, 89(4), 743-751.

• Greenwald, A.G. (1997). Validity concerns and usefulness of student ratings of instruction. American Psychologist, 52(11), 1182-1186.

• Kulik, J.A. (2001). Student ratings: validity, utility, and controversy. New Directions for Institutional Research, 109, 9-25.

• Marsh, H.W., & Roche, L.A. (1997). Making students’ evaluations of teaching effectiveness effective: The critical issues of validity, bias and utility. American Psychologist, 52, 1187-1197.

• Marsh, H. W. (1980). The influence of student, course, and instructor characteristics in evaluations of university teaching. American Educational Research Journal, 17(2), 219-237.

References• Marsh, H.W. (1983). Multidimensional ratings of teaching effectiveness by

students from different academic settings and their relation to student/course/instructor characteristics. Journal of Educational Psychology, 75(1), 150-166.

• Marsh, H. W. (1984). Students' evaluations of university teaching: Dimensionality, reliability, validity, potential biases, and utility. Journal of Educational Psychology, 76(5), 707-754.

• McKeachie, W.J. (1997). Student ratings: The validity of use. American Psychologist, 52, 1218-1225.

• Theall, M., Abrami, P., & Mets, L. (Eds.). (2001, Spring). The student ratings debate: Are they valid? How can we best use them? New Directions for Institutional Research, 109.

• Wachtel, H. K. (1998). Student evaluation of college teaching effectiveness: A brief review. Assessment & Evaluation in Higher Education, 23(2), 191-212.

References• Wright, P. S., Horn, S. P., & Sanders, W. L. (1997). Teacher and classroom

context effects on student achievement: Implications for teacher evaluation. Journal of Personnel Evaluation in Education, 11, 57-67.

Summary of Best Predictors• Best Overall Course Predictors

• 3.7 Clearly explains difficult concepts, ideas, or theories.• 3.11 motivates me by his/her example to want to learn

about the subject.• 3.4 Makes good use of class time.

• Best Overall Course Predictors • 2.13 Course as a whole has produced new knowledge,

skills and awareness in me.• 2.4 Course content is relevant and useful.• 2.2 Course is well organized.

Summary of Predictors (Non-Significant)

• Overall Faculty Predictors • 3.10 Is available to students during regular and reasonable

office hours.• 3.14 Seldom misses class.• 6c Help students prepare to live effectively in society.

• Overall Course Predictors • 2.5 Assigned workload is appropriate for credit hours.• 2.11 Grading procedure is fair and impartial• 2.12 Assignments are appropriately distributed throughout

the semester.

cecilia chan, ir planning analyst paul freebairn, ed.d , director of assessment

Documents