1

Neal Schmitt Michigan State University

Presented at College Board, ETS, AERA Conference

December 10, 2010

Combining Cognitive and Non-Cognitive Measures: Expanding the Domain of

College Performance and its Prediction

2

Acknowledgements Jessica Fandre Tim Pleskac Alyssa Friede Abigail Quinn Michael Gillespie Lauren Ramsay Anna Imus Smriti Shivpuri Brian Kim Ruchi Sinha Stephanie Merritt Tae-Young Yoo Fred Oswald Mark Zorzie Matt Reeder Juliya Golubovich College Board (Wayne Camara and Krista

Mattern)

3

Outline History and Background “Job analysis” or conceptualization Instrumentation (description of noncognitive

measures used and outcomes examined) Validity data Subgroup Differences and Implications Faking issues Acceptability Research (profiles, fit, goal orientation, dif) Limitations and future research

4

Developing Alternative Measures

of Student PotentialMotivation for our work Broaden the scope of student outcomes and

capabilities considered in college admissions.

Reduce adverse impact.

Test the feasibility of developing “noncognitive” measures that are

valid practical in terms of time and effort required to assess less susceptible to faking

5

Identify a broader domain of college student performance:

Review university mission statements and department objectives

Interview with university staff responsible for student life at Michigan State University

Review of the education literature on student outcomes

Our systematic search resulted in 12 dimensions of student performance…

Developing Alternative Measures of Student Potential (Oswald et al, 2004,

JAP)

6

1. Knowledge and mastery of general principles

2. Continuous learning, intellectual interest and curiosity

3. Artistic and cultural appreciation

4. Appreciation for diversity

5. Leadership

6. Interpersonal skills

7. Social responsibility and citizenship

8. Physical and psychological health

9. Career orientation

10. Adaptability and life skills

11. Perseverance

12. Ethics and integrity

Dimensions of College Student

Performanceintellectual

interpersonal

intrapersonal

7

Two “Noncognitive” Measures

1. Situational judgment inventory A situation is presented along with several

alternative courses of action. The respondent is asked to indicate what she/he

would be most likely and least likely to do.

2. Biodata Short, multiple choice reports of

past experience/background and/or interests/preferences.

8

Situational Judgment Inventory (SJI)

Developed situational judgment items for each of the 12 performance dimensions

Student generated critical incidents (CIs) for each dimension

Translated CIs to stems for each item

Other students generated solutions to these questions

Researchers edited the options

Re-sorting back into 12 dimensions

3 Answer Keys: (see Motowidlo, Dunnette, & Carter, 1990) Expert student (junior and senior students) scoring Resident Advisor scoring African American key

9

Sample SJI Item for Leadership

You are assigned to a group to work on a particular project. When you sit down together as a group, no one says anything.

a) -1 Look at them until someone eventually says something

b) Start the conversation yourself by introducing yourself

c) +1 Get to know everyone first and see what they are thinking about the project to make sure the project’s goals are clear to everyone

d) Try to start working on the project by asking everyone’s opinion about the nature of the project

e) You would take the leadership role by assigning people to do things or ask questions to get things rolling

10

Sample SJI Item forInterpersonal Skills

You and some other students in your dorm area feel that a small group of students are highly disruptive during times when you would like to study or sleep. What would you do?

a) Talk to the resident assistant about it, as that is one of the responsibilities of their job.

b) -1 That’s part of life in the dorms. Let it go.c) Bring it up at the next floor meeting.d) +1 Politely talk to the disruptive students and ask them to

be more considerate. If the problem persists, talk to the resident assistant.

e) Wear earplugs or headphones when necessary.f) Develop and implement appropriate rules to address the

problem.

Sample Item for Knowledge

You decided early in the term to do a paper on a topic very interesting to you. However, you have found it difficult to find information on your topic, your job has taken more time than you wanted, and you have had more work in your other courses than you anticipated. Now it seems like you may have to engage in several "all-nighters" to complete your paper on time. What would you do?

a. Seek help from other students who may have had a similar experience. b. (-1)Pick a topic that can be completed quicker. An “A” is an “A”. c. (1)Set up a schedule on which you can complete all of the other work

you need to do, spend as much time on the paper as possible, and meet with the instructor to discuss what you have so far and get suggestions.

d. Do whatever it takes to complete the paper, including “all-nighters”. e. Talk to the instructor about the situation and ask for advice. f. Make the paper a priority, but take into account how much the paper is

worth in the class.

11

12

Biodata MeasureDeveloped biodata items for each dimension

Reviewed biodata item pools and adapted items related to each major performance dimension.

Resorted items into dimensions. Asked a pilot sample to respond to open-ended

versions of quantitative response options to determine appropriate scale anchors.

We used a rational scoring approach to these items, but also developed empirical keys against 3 criteria

13

Sample Biodata Items for Leadership

1. The number of high school clubs and organized activities (such as band, sports, newspapers, etc.) in which I took a leadership role was:

a) 4 or moreb) 3c) 2d) 1e) I did not take a leadership role

2. How often do you talk your friends into doing what you want to do during the evening?

a) most of the timeb) sometimes (about half the time)c) occasionally (about as often as others in my groupd) seldom or infrequentlye) never

14

Sample Biodata Items for Multicultural Appreciation

1. How often have you participated in social service or charity organizations?

a) Four or more timesb) Three timesc) Two timesd) Oncee) Never

2. If given a choice at a restaurant, would you order any food with which you are unfamiliar?

a) Never, I would always order foods that I know and enjoyb) Sometimes I might try a new food if someone else ordered itc) Occasionally I will order something new provided I can also

order familiar food at the same timed) If given a chance, I will always order a new food and try it

15

Sample Biodata Items for Social Responsibility

1. In the past year how many times have you considered the environment when purchasing a product (for example hairspray, or a car?)

a) Neverb) Oncec) Twiced) Three or four timese) Five times or more

2. How often do you work with not for profit groups?a) Neverb) Not very oftenc) Sometimesd) Oftene) Always

16

Outcomes Examined Self Ratings on behaviorally anchored

rating scales built around the 12 dimensions

Self rated class attendance University archives (grades) Organizational citizenship behavior Deviance Continuation in school and graduation

17

Interpersonal skills—Self Rated Performance Outcome 1 2 3 4 5 6 7 | | | | | | | Before you make your rating, please read these two examples: Example 1 Your roommate, usually a tidy person, has recently experienced some personal difficulties . As a result, the roommate has become quite distracted and has left much of the household responsibilities to you. You have talked to the roommate about your concerns, and empathetically requested that the roommate resume his/her share of the responsibilities as soon as possible. A month passes and you are still doing too much of the roommate’s work. What do you expect you would do? Example 2 You have been standing in line for the restroom for some time after a campus event, and someone cut s into the line ahead of you. What do you expect you would do?

Unsatisfactory Fulfills

Expectations Exceptional

Exceptional

You talk with the roommate again and explain that you are suffering as a result of the roommate’s behavior. You attempt to come up with a mutually acceptable plan of action.

Unsatisfactory

You ask to change rooms.

Unsatisfactory

You comment loudly to someone nearby how rude it is that people cut in line.

Fulfills Expectations

You tell the person that there is a line.

Exceptional

You calmly and politely inform the person that there is a line and ask that they move to the back.

Fulfills Expectations

You do his/her share of the work, and put anything of the roommate’s that affects you in the roommate’s area of the room.

Definition: Communicating and dealing well with others, whether in informal social situations or more formal school-related situations. Being aware of the social dynamics of a situation and responding appropriately.

18

Validity Data: College GPA 2001-02 2003-04 2004-8 2009-10 (N=614) (N=568) (N>1900) (N>550)

Knowledge .22 .22 .26 .22 Learning .05 -.02 .13 .10 Art Appr. .01 -.03 .18 .16 Mltcult.Appr. .07 -.04 .11 .12 Lead .14 -.01 .09 .07 Int. Skl. .04 Soc. Resp. .08 .07 .13 .08 Health .23 .14 .11 .04 Cr. Ornt. -.02 -.06 -.14 -.11 Adapt .21 .13 .05 .01 Perser .15 .07 .07 .10 Ethics .14 .22 .17 .13 SJI .16 .11 .22 .09 HSGPA .39 .53 .29 ACT/SAT .33 .34 .53 .44

19

Validity Data: Class Absences

2001-02 2005-06 2008 2009-10 (N>630) (N>900) (N >556) (N>600)

Knowledge -.18 -.15 -.14 -.03 Learning .00 -.07 -.06 .04 Art Appr. .07 -.03 -.08 -.10 Mltcult.Appr. .02 -.04 -.10 -.06 Lead -.03 -.06 -.02 -.04 Int. Skl. -.09 Soc. Resp. -.09 -.08 -.08 -.11 Health -.23 -.17 -.12 -.05 Cr. Ornt. -.06 -.05 -.08 -.05 Adapt -.15 -.10 -.10 .07 Perser -.20 -.18 -.16 -.01 Ethics -.31 -.17 -.24 -.11 SJI -.27 -.16 -.17 -.14 HSGPA -.04 -.01 -.02 ACT/SAT .11 .17 .14 .07

20

Validity Data: Self Report-BARS

2001-02 2003-04 2008 2009-10(N=614) (N=568) (N>547) (N>600)

Knowledge .33 .46 .21 .32 Learning .27 .37 .24 .39 Art Appr. .51 .33 .20 .33 Mltcult.Appr. .38 .34 .29 .30 Lead .43 .34 .29 .38 Int. Skl. .15 Soc. Resp. .35 .32 .26 .30 Health .28 .16 .26 .29 Cr. Ornt. .22 .36 .22 .31 Adapt .24 .26 .28 .30 Perser .34 .46 .36 .37 Ethics .11 .39 .23 .22 SJI .53 .54 .23 .25 HSGPA .08 .08 .02 ACT/SAT -.01 -.01 .01 .08

21

Incremental Validity: College GPA

01-02 03-04 08 09-10Step 1: Act/SAT, HSGPA ∆R2 .103 .179 .398 .200Step 2: Biodata, SJI ∆R2 .089 .070 .029 .069 Adjusted R .438 .499 .419 .480N 610 331 1155 296

Noncog. Vars. Significant Know Know Lrng. Cr. Ornt. Lrng. Lrng. Health Know Health Ethics Cr.Ornt. Adapt SJI SJI

22

Incremental Validity: Absenteeism and BARS

05-06 08 follow-upAbsenteeism BARS Absenteeism BARS

Step 1: Act/SAT, HSGPA ∆R2 .028 .019 .033 .008Step 2: Biodata,

SJI ∆R2 .060 .215 .116 .240

Adjusted R .297 .484 .386 .497N 800 801 556 547Noncog. Vars. Health Health Lead Multic

Ethics Ethics Health Health SJI SJI Ethics Persv Adapt SJT Ethics

Cr.Ornt.Persev.

23

Subgroup Differences in Standardized Units (04sample)

Male-Female Cauc-Afr.Am. Cauc-Hisp.Am.SJT -.15 .13 .10Know .00 .36 .23 Cont.Lrn. .10 .05 .00Artistic -.17 .31 .01Multi.Appr. -.16 -.08 -.46Leader -.15 .14 .10Respons. -.25 .16 .01 Health .52 .44 .30Car.Ornt. -.17 -.55 -.08Adapt. .00 .11 .16Persev. -.23 -.19 .00Ethics -.19 .22 .29HSGPA -.04 -.61 .42SAT/ACT .43 -1.46 1.01

24

Percent of subgroups admitted under various strategies

Hispanic Asian African Caucasian

Cog Cog+ Cog Cog+ Cog Cog+ Cog Cog+

Top 15% 4.3 6.4 17.8 14.9 .9 4.1 77.0 74.6

Top 50% 4.0 4.6 10.5 10.1 8.3 10.0 77.1 75.3

Top 85% 4.5 4.7 7.6 7.7 18.4 18.7 69.5 69.0

All 3.7 9.0 19.4 67.8

Cog=equally weighted composite of HSGPA and SAT/ACT

Cog+=equally weighted composite of HSGPA, SAT/ACT, and Non-cognitive measures.

25

Average Cumulative GPA for Subgroup Members who graduated at various levels of selectivity Hispanic Asian African Caucasian N Cog N Cog+ N Cog N Cog+ N Cog N Cog+ N Cog N Cog+ Top15 4 3.67 6 3.57 23 3.88 18 3.84 0 1 3.66 208 3.88 197 3.87 Top50 17 3.51 17 3 .42 55 3.65 54 3.63 15 3.46 15 3.38 651 3.66 638 3.66

Top85 23 3.44 22 3.47 64 3.62 64 3.62 49 3.05 47 3.08 929 3.52 905 3.54

26

Proportion of Subgroups Graduating in Four Years under Different Levels of Selectivity

Hispanic Asian African Caucasian

Cog Cog+ Cog Cog+ Cog Cog+ Cog Cog+Top15 93 90 89 91 100 92 72 70Top50 79 78 72 75 87 87 59 60Top85 77 76 68 68 79 78 51 52

27

Admits versus Applicants: Standardized mean differences (d)

2004 2006 Knowledge .55 .54 Continuous Learning .43 .49 Artistic Appreciation .35 .30 Multicultural Appreciation .48 .40 Leadership .43 .35 Social Responsibility .46 .46 Health .26 .30 Career Orientation .25 .21 Adaptability .22 .25 Perseverance .39 .32 Ethics .62 .53 Situational Judgment .50 .52

28

Conclusions on Faking Research (represents summary of a series of studies Applicants score higher than current students

(d=.2 to .6) Coaching has a significant impact on the degree

of score inflation Elaboration can minimize score inflation, but…

feasibility is an issue and its effects do not appear to generalize to nonelaborated items

Elaboration has no impact on validity Warnings do not appear to have much effect,

but…generalizability to an applicant situation has not been evaluated.

Are they any less fakable than essays or less inflated than letters of recommendation???

29

How often in the past year have you programmed in AJMR?

never once twice three or four times five times or more

How often, in the past three years, have you operated a rhetaguard?

never once twice three or four times five times or more

Sample Bogus Items: Carelessness

Profiling Subgroups of Students based on HSGPA, SAT/ACT and Noncognitive Variables

30

31

32

33

34

35

36

37

Outcomes and Implications There were predicted outcome differences (GPA,

absenteeism, satisfaction) across profile groups Highly motivated, career-oriented group is most

likely to respond to remedial efforts particularly if they relate to their career objectives

Marginal group without career objectives is a high risk group. Career counseling and intensive remediation may be necessary

Efforts to broaden the scope of interests of the high ability, culturally limited may be desirable in some universities

Students in the high ability well rounded group might be identified as potential student leaders and peer mentors/tutors.

38

Overall Conclusions We can develop valid noncognitive measures that

relate to GPA and other important student outcomes

Faking of the biodata and SJI remains a problem. Reactions to their use are not significantly

different than reactions to the ACT/SAT Subgroup differences are minimal and certainly

much less than those we find for cognitive ability measures

There may be useful other ways to employ these instruments; that is, to identify subgroups for whom interventions designed to retain them will be needed.

39

Thank you for your attention!