iii. assessment a. introduction 1. why we measure? 2. what we measure

III. AssessmentA. Introduction

1. Why we measure?2. What we measure3. The case of abstract concepts

B. The OSS example1. Leadership2. The procedures

C. Modes of measurement1. Self-reports

a. empirical approachb. rational approachc. factor analysis

2. The remaining modesa. definitionsb. examples

D. Assessing the Assessment1. Reliability

a. definitionb. measurement

2. Validitya. criterionb. contentc. construct

AssessmentAssessment

Why we measure?Research: to test hypotheses and facilitate classification

Practical problems: personnel selection, career counseling, treatment of individuals with psychological disorders

Each (more or less) requires:Specifying the outcome

Identifying the relevant characteristics

Developing assessment procedures

Communicating results

AssessmentAssessmentWhat do we measure?

Measure one or more attributes or characteristics (variables)

Examples: Height, weight, duration of temper tantrum

Reduce these attributes to numbers

Different types of scales

Different properties

Few scales in psychology are ratio scales

Abstract concepts?

Agreeableness

Sensation Seeking

Leadership Potential

Measuring ConceptsMeasuring Concepts

Process of conceptualization: working to define exactly what is meant by a given term

Generate rough description

Agreeableness

Start pointing to particular examplesMultiple examples

From broad domains

Begin with idea

Agreeableness is ….

Specific examples

Specific indicators constitute the measure

An example: The OSS assessmentAn example: The OSS assessmentOffice of Strategic Services (OSS)

War-time agency (WWII)Tactical and covert operationsNeeded to recruit and train agents

Identified 7 major variables1. Motivation 2. Energy and initiative

5. Social relations4. Emotional stability3. Effective intelligence

6. Leadership7. Security

Measuring LeadershipMeasuring Leadership

Assessment guided by principle of convergence: gather evidence from multiple viewpoints and look for agreement

Multiple sources

Leadership: ability to take the initiative in social situations, to plan and organize action, and in so doing evoke cooperation

Multiple methods

Used 7 different procedures

Important points to take from OSS Important points to take from OSS assessmentassessment

1.1. First identified relevant personality variables, then First identified relevant personality variables, then designed way to assess.designed way to assess.

ConceptualizationConceptualizationThen specific indicators (operationalize)Then specific indicators (operationalize)

2.2. Measured each variable in different situations and with Measured each variable in different situations and with different methodsdifferent methods

3.3. Used multiple observers. Looked for convergence.Used multiple observers. Looked for convergence.

= important points about measurement= important points about measurement Conceptualization & operationalizationConceptualization & operationalization Reliability & validityReliability & validity

Methods for Measuring PersonalityMethods for Measuring PersonalityMode 1 Mode 2 Mode 3 Mode 4 Mode 5 Mode 6 Mode 7

Description Trait inventories

State experiences

Ability tests Informed ratings

Impartial ratings

Behavioral measures

Physiological measures

Source of Data Self Self Self Other Other Instrument Instrument

Time Frame Past Current Current Past Current Current Current

How Source Functions Interprets Perceives

Solves or decides Interprets

Interprets or transcribes

Records mechanically

Records mechanically

Nature of Task

Describe yourself

Describe your experience

Answer correctly

Describe this person

Describe this person

Record behavioral response

Record physiological activity

Type of Variable Ordinal Ordinal

Ordinal or interval Ordinal Ordinal

Interval or ratio

Interval or ratio

Typical Variables

Traits; attitudes; beliefs

Preferences; judgments; feelings

Knowledge; skills; abilities

Traits; complex behaviors

Traits; behaviors

Simple behavioral responses

Cortical or autonomic arousal

Typical Examples

BFI; MMPI; NEO-PI-R

PANAS; perceptual judgments

Rod and frame; IQ Peer ratings

Clinical interview

Reaction time

EEGs; HR; SCL; BP

Methods for Measuring PersonalityMethods for Measuring Personality

Mode 1: Self-report trait inventories

Typical Variables: Traits

Attitudes

Beliefs

Potential limitation of self-report Potential limitation of self-report methodsmethods

1.1. Misconstrue meaningMisconstrue meaning2.2. Hurry to finish or be carelessHurry to finish or be careless3.3. Deliberately distort answersDeliberately distort answers4.4. Lack insight into selfLack insight into self5.5. Try to present self favorablyTry to present self favorably

Methods for constructing self-report Methods for constructing self-report measuresmeasures

1.1. Empirical approachesEmpirical approaches2.2. Rational/theoretical approachesRational/theoretical approaches3.3. Factor analysis Factor analysis

Methods for Constructing Self-ReportsMethods for Constructing Self-Reports

1. Empirical Approach: Select items that distinguish between those who possess the trait in great magnitude and those who possess the trait only in small magnitude if at all.

No assumptions about nature of psychiatric d/os

Empirically derived

Example: MMPI

Empirical Approach Example: MMPIEmpirical Approach Example: MMPI

Began with 1,000 item pool taken from textbooks, interviews, and existing tests

Administered these items to two groups:

Groups of psychiatric patients

“Normals”: 724 visitors to U of M hospitals

Chose items that distinguished between “normals” and one patient groupEnded up with 566 items

Three validity scales and 10 clinical scales

Empirical Approach: MMPIEmpirical Approach: MMPI

Control group Control group (724 normals)(724 normals)

Criterion group Criterion group (50 (50 schizophrenics)schizophrenics)

Difference Difference between between groupsgroups

Scoring Scoring weights for weights for Sc scaleSc scale

ItemsItems T FT F T FT F T FT F T FT F

I like mechanics I like mechanics magazinesmagazines

I hear strange I hear strange things when I’m things when I’m alonealone

I get all the I get all the sympathy I shouldsympathy I should

50 5050 50

5 955 95

80 2080 20

51 4951 49

35 6535 65

50 5050 50

+1 -1+1 -1

+30 -30 +30 -30

-30 +30 -30 +30

0 00 0

+1 0+1 0

0 +10 +1

Illustration of item selection

Empirical Approach: MMPIEmpirical Approach: MMPICriticisms:

Too long

Some items are offensive

Poor test-retest reliability

Poor discrimination between psychiatric groups

Fails to get at basic dimensions of personality

Inefficient

Fails to assess normal range of personality

Methods for Constructing Self-ReportsMethods for Constructing Self-Reports2. Rational/Theoretical Approach: Relies on theory to determine the how and what of assessment

Theory may dictate HOW to measure something

Psychoanalytic theory projective tests

Projective hypothesis: If an individual is confronted with an ambiguous stimulus, the interpretation that the individual provides is to some extent a "projection" of the individual's internal characteristics.

Microsoft PowerPoint Presentation

Projective TestsProjective TestsOften contrasted with objective testsProjective tests are:

UnstructuredQuestions asked

Responses available

Disguised

Scored subjectively

Two classic examples:Rorschach Inkblot Test

Thematic Apperception Test

Rorschach Inkblot TestRorschach Inkblot TestWhat might this be?

Method for Constructing Self-reportMethod for Constructing Self-report

Rational/theoretical approach continuedRational/theoretical approach continued Projective tests example of theory driving Projective tests example of theory driving

HOW to measureHOW to measure

Theory may also dictate WHAT to measureTheory may also dictate WHAT to measure

Rational approach to scale construction: Rational approach to scale construction: exampleexample

Rational ApproachRational Approach

1.1. Idea: what it is you want to measure?Idea: what it is you want to measure?2.2. Literature search: what have others Literature search: what have others

done? What is similar and different?done? What is similar and different?3.3. Operationalize the construct – precise Operationalize the construct – precise

definition of variable of interestdefinition of variable of interest4.4. Create initial sample poolCreate initial sample pool5.5. Adminster itemsAdminster items

Rational Approach cont.Rational Approach cont.4) Create initial item pool

First guiding principle: “The items of the pool should be chosen so as to sample all possible contents which might comprise the putative trait according to all known alternative theories of the trait” (Loevinger, 1957)

Implications of first guiding principle:

a. Pool should be broader and more comprehensive than own view

b. Pool will include content that will later be shown to be irrelevant

Rational ApproachRational Approach4) Create initial item pool (continued)

Second guiding principle: ensure that there is an adequate sample of items within each of the major content areas comprising the broadly conceptualized domain

Implications of second guiding principleCreate formal subscales for each content area

Proportion of items dedicated to each content area should reflect importance of that area in target construct

UPPS Urgency example Microsoft PowerPoint Presentation

Rational ApproachRational Approach7) Decide on the status of the scale

Good enough proceed with construct validation

Not good enough return to Step 4

Rational ApproachRational ApproachTips for writing good items

Language: simple, straightforward, appropriate level

Avoid trendy expressions

Ensure variability in responding

Avoid double-barreled items

Decide on scale format

Methods for Constructing Self-ReportsMethods for Constructing Self-Reports

Factor Analytic Approach: All the characteristics found in personality test scales can be reduced to a few common factors that describe the underlying and fundamental aspects of human personality.

Factor AnalysisStatistical tool for data reduction

Highly correlated items are measuring the same thing

Scales/subscales chosen based on results

Microsoft PowerPoint Presentation

Example: How people cope with Example: How people cope with stressstress

How much did you do ______ during your most recent How much did you do ______ during your most recent stressful event? Or rate each item:stressful event? Or rate each item:

1.1. Took action quickly, before things could get out of Took action quickly, before things could get out of hand.hand.

2.2. Refused to believe it was real.Refused to believe it was real. 3.3. Did something concrete to make the situation Did something concrete to make the situation

better.better. 4.4. Tried to convince myself that it wasn’t happening.Tried to convince myself that it wasn’t happening. 5.5. Went on thinking that things were just like they Went on thinking that things were just like they

were.were. 6.6. Changed or grew as a person in a new way.Changed or grew as a person in a new way. 7.7. Tried to look at the bright side.Tried to look at the bright side.

Hypothetical correlation matrixHypothetical correlation matrix

ItemItem 11 22 33 44 55 66 77

11 ** .1.1 .75.75 -.05-.05 .03.03 .12.12 00

22 ** -.19-.19 .52.52 .61.61 -.07-.07 -.08-.08

33 ** .17.17 00 .11.11 .08.08

44 ** .71.71 .09.09 .04.04

55 ** .16.16 .09.09

66 ** .59.59

77 **

Hypothetical Factor LoadingsHypothetical Factor Loadings

FactorFactor AA BB CC

Item 1Item 1 .62.62 .15.15 .01.01

Item 2Item 2 .03.03 -.08-.08 .49.49

Item 3Item 3 .54.54 .04.04 -.20-.20

Item 4Item 4 .10.10 .11.11 .56.56

Item 5Item 5 .07.07 .08.08 .45.45

Item 6Item 6 -.02-.02 .66.66 .12.12

Item 7Item 7 .22.22 .48.48 .06.06

ImplicationsImplications

Garbage in, garbage out; brilliance in, Garbage in, garbage out; brilliance in, brilliance outbrilliance out

Missing info. may mean missing factorsMissing info. may mean missing factorsE.g. Escape: smoking, drinking, eating, E.g. Escape: smoking, drinking, eating, daydreaming about vacation, etc.daydreaming about vacation, etc.

Importance of names of factorsImportance of names of factors

Factor Analytic ApproachFactor Analytic Approach1. 2. 4. 6. 7. 9. 11. 12.

1. talkative

2. finds fault (r) -.07

4. depressed -.13 -.21

6. reserved (r) .37 .05 -.16

7. helpful -.02 .21 -.12 -.05

9. relaxed (r) .00 -.14 .37 -.02 -.07

11. full of energy .38 .10 -.33 .27 .12 -.24

12. starts quarrels (r) -.13 .33 -.16 -.07 .26 -.07 -.02

14. tense .04 -.22 .29 -.06 -.03 .41 -.11 -.12

Methods for Measuring PersonalityMethods for Measuring PersonalityMode 1: Trait Inventories (just discussed)

Mode 2: State InventoriesSource: Self

Time Frame: Current

Function of Source: Perceives

Task: Describe Your Experience

Typical Variables: Preferences

Judgments

Feelings


Mode 3: Ability Tests

Source: Self

Time Frame: Current

Function of Source: Solves or decides

Task: Answer Correctly

Typical Variables: Knowledge

Skills

Abilities


Mode 4: Informed Ratings

Source: Other

Time Frame: Past

Function of Source: Interprets

Task: Describe This Person


Behaviors


Mode 5: Impartial Ratings

Source: Other

Time Frame: Current

Function of Source: Interprets or transcribes

Task: Describe This Person


Behaviors


Mode 6: Behavioral Measures

Source: Instrument

Time Frame: Current

Function of Source: Records Mechanically

Task: Record Behavioral Response

Typical Variables: Reaction times

Responses


Mode 7: Physiological Measures

Source: Instrument

Time Frame: Current

Function of Source: Records Mechanically

Task: Record Physiological Activity

Typical Variables: Cortical Arousal

Autonomic Arousal

Sexual Arousal

iii. assessment a. introduction 1. why we measure? 2. what we measure

Documents